eXist Developer's Blog

Besides fixing critical bugs in the storage backend, the 1.2.4 release mainly improves the memory consumption of queries on large document sets. Major changes include:

• new node set implementation, which is much more memory efficient compared to previous approaches. The old implementation consumed a lot of memory when used with larger sets of documents. Obviously this had a negative effect on overall performance.

• reduce memory consumption of documents constructed during a query: if you have a query which creates thousands of small XML fragments, each of those fragments used to have its own document context with its own name pool and various fields which may have never been needed. Large parts of the document context are now shared between fragments and we make more use of lazy initialization, thus reducing the memory consumption of in-memory fragments dramatically (in my tests, I could save up to 100mb memory when creating a few thousand XML fragments in one query).

• fixed fatal btree bugs leading to index corruptions (which usually caused an ArrayIndexOutOfBounds exception). The bugs were more likely to occur when indexing large string keys, but they may also have happened in other situations. The failure damaged the index and rendered the db unusable (though it could be repaired).

• fixed concurrency issues leading to ArrayIndexOutOfBounds or NoSuchElement exception when querying for attributes

• memory leak: we observed that the xerces XML parser builds some internal data structures when validating a document, which are unfortunately not properly cleared afterwards. This is a major problem since eXist pools the XML parser instances. To work around those issues, eXist will no longer pool XML parsers which were used on larger documents.

• using full text and ngram indexes at the same time caused eXist to hang in an endless loop

The release is now available for download.

Note: all releases in the 1.2 branch are bug fix releases and can be considered stable. They only contain hand-selected changes which were ported back from the main development version.

As reported by users, the 1.2.2 and 1.2.3 releases shipped with a bad memory configuration: in the main configuration file (conf.xml), the cacheSize parameter was set to 256M:

However, Java is started with only 128M max. memory, so using 256M for caches will sooner or later result in eXist hitting the wall.

The problem here is that the effects of an OutOfMemory error are somehow unpredictable and may lead to unnoticed corruptions in the database. Java doesn't show many warnings before it runs out of memory. All you usually get is a message on stderr.

In general, the cacheSize parameter in conf.xml should never be set to more than 1/3 of the maximum memory available to Java. Please adjust cacheSize accordingly or increase Java's max memory (usually set through the -Xmx parameter which has to be passed on the java command line - see bin/functions.d/eXist-settings.sh or bin/startup.bat).

We still had some issues with the installer in the 1.2.2 release. This was a major problem for some users who redistribute eXist with their own application. Version 1.2.3 has been uploaded to solve those issues.

It also fixes the consistency checker, which was introduced with 1.2.1 and unfortunately triggered a false alarm in some cases.

Updating is not really necessary unless you had problems with the installer or rely on the consistency check service. The 1.2 branch is maintained separately from the development branch. This allows us to release selected bug fixes and improvements much more frequently.

Google has extended the deadline for student applications until Monday, April 7, 2008. We hope this will convince a few more people to apply. So far we received 6 proposals. However, I think that a detailed application which concentrates on concrete development steps will still have a realistic chance to be accepted.

eXist is participating in the Google Summer of Code again. Student applications can be submitted until March 26! The list of proposals along with the timeline can be found here:

http://www.exist-db.org/gsoc/2008/summer.html

Some of our project proposals may sound a bit ambiguous, but please don't be scared: we are here to help ;-) We chose those projects because they represent more or less separate work packages which can be handled without knowing the entire eXist code base.

The list includes, for example, a "remote debugging interface" for XQuery. We certainly don't expect someone to write a full-blown debugger. What we need is a well-defined debugging interface on the server and a simple command-line prototype, which demonstrates the interface.

Or take "index-support for order-by, distinct-values and aggregate functions": eXist's new indexing architecture makes it possible to implement the required functionality as an index plugin. Not too difficult. The challenge though will be to make it efficient.

I have not been there myself, but as we heard from those who attended XML 2007, there were quite a few talks mentioning eXist:

Erik Bruchez: XForms and the eXist XML database: a perfect couple

Dan McCreary: Using XForms and eXist to Manage Metadata

Kurt Cagle: Lightweight XML

Mark Birbeck: XForms, XHTML, and RDFa for Internet-Facing Applications

Mark Birbeck: XForms on the Desktop using Sidewinder

The slides for the first two talks are available online. We also heard Norm Walsh mentioned an XProc implementation in XQuery based on eXist (I guess this must be Jim Fuller's XProcXQ). XProc is an interesting new standard and I would love to see a simple implementation which can be easily integrated with eXist.

We just reached another milestone in our struggle to make eXist 100% conformant with the XQuery specs: 99.4%! Details can be found on the official W3C XQuery Test Suite pages.

Recent changes were mostly related to namespace handling, though we also had a number of small fixes to the XQuery parser, including whitespace processing and ordering declarations.

This site is now going public as we are starting to switch the links on the eXist homepage to point to http://atomic.exist-db.org. We will try to move all valuable contents from the old wiki into here. This has to be done manually though as the old server is definitely dead. Well, it's a good chance to re-read and evaluate all the old stuff.

The new site will be in a "private beta" mode for now, which means that only selected users can edit entries. We will open registration for other users once we are sure the system is running stable enough.