The GCX XQuery Engine – Benchmark Results for GCX v1.0β

G(arbage) C(collected) X(query) Engine – An open source in-memory XQuery engine

The GCX engine is an in-memory XQuery engine designed for memory-efficient XQuery evaluation against large XML documents. The C++-prototype, which was released in v1.0β, supports a powerful fragment of the XQuery language. The following experiments are part of our publication at ICDE 2007.

Experiments with the XMark Benchmark

We measured the performance of the GCX egnine v1.0β on benchmark data from XMark – An XML Benchmark Project. To this end, we generated XML documents of sizes between 10MB and 200MB with the XMark data generator xmlgen. As the GCX engine does not yet support the full XQuery standard, we modified the XMark benchmarks as follows:

• The GCX engine does not yet support XML attributes. Consequently, all attributes in the XML documents were rewritten to subelements. For instance, an opening tag <book id="1"> is rewritten to <book><book_id>1</book_id> (a 1MB sample XML document is available here).
• The GCX engine v1.0β does not support aggregation (the support for aggregation has been added in v2.0!). Hence, we slightly modified selected XMark queries, as shown in the XMark Queries section. Each query is related to the same-numbered XMark query, but due to the rewriting some of them yield different results than their original counterparts.

In our experiments, all XQuery engines evaluated the same rewritten queries on exactly the same input XML documents.

Reference Implementations

The GCX engine has two main characteristics: It is an in-memory XQuery engine and it is geared towards streaming XQuery evaluation. With this in mind, we chose the following reference implementations.

• The FluXQuery engine (programming laguage: Java) is probably the most natural choice for a reference implementation: it is also a main memory XQuery engine geared towards XML stream processing and it implements a similar XQuery fragment as the GCX engine v1.0β. Moreover, the FluXQuery engine is able to exploit schema information which come from a Document Type Definition (DTD). Consequently, the FluXQuery engine was provided with the XMark DTD in our experiments.
• The in-memory query engines Galax v0.6.8 (programming language: Objective CAML), Qizx/open v1.1 (programming language: Java) and Saxon v8.7.1 (programming laguage: Java) implement the full XQuery standard. While Galax has not been designed with XML stream processing in mind, it is often consulted in XQuery benchmarks and – for this reason – also included here. Note that the static projection of Galax could not be made to work in our experiments.
• Finally, we chose MonetDB v4.12.0/XQuery v0.12.0, a mature XML database system. As a secondary-storage implementation, MonetDB can make use of index structures to speed up query evaluation, which is not done by streaming in-memory XQuery engines. On the other side, MonetDB XQuery stores the entire data physically before query evaluation. To account for the fact that the GCX engine and the other main memory engines read the complete input XML document for each query evaluation, in each run we forced the MonetDB server to reload the complete XML document (and include this document loading time in our time measurements).

Execution Platform

We ran our experiments on a 3GHz CPU Intel Pentium IV with 2GB RAM, running SuSe Linux 10.0. All Java-based systems were executed using J2RE v1.4.2.

The focus of the benchmarks was primarily on main memory consumption, but we also consider query execution time. Time is given either in seconds (abbreviated with "s") (e.g. 1.59s means 1 second and 59 millisecond) or in minutes (abbreviated with "m") (e.g. 02:07m means 2 minutes and 7 seconds). Memory consumption is given in megabytes (abbreviated with "MB") or gigabytes (abbreviated with "GB"). The main memory consumption was measured with the Linux "top" command. For each system and query we set a timeout of one hour. For each system and size of the input XML document, we measured the high watermark of non-swapped memory consumption, and the total query evaluation time. "Not available" (abbreviated with "n/a") indicates that the query could not be expressed in the language supported by the specific engine, while a dash (abbreviated with "–") denotes failure, e.g. caused by segmentation faults. With the Java-based engines, we could observe that due to effects caused by automatic memory management and the Java Virtual Machine, memory consumption often increased with the XML document size even though the buffer size remained constant (e.g. in case of the FluXQuery engine).

Benchmark Results

The table below summarizes the runtime and memory consumption of the GCX engine (v1.0β) compared to the other XQuery engines. Note that the benchmark results are not permanently kept up to date. The behavior (needed runtime and memory consumption) of the current version (v2.1) of the GCX engine might differ from the benchmark results given in the following table.

XMark Queries

Note: All following (XMark) queries were taken from XMark – An XML Benchmark Project and modified to match the GCX v1.0β supported XQuery fragment.

XMark Q1

<query1> {
  for $site in /site return
    for $people in $site/people return
      for $person in $people/person return
        if ($person/person_id="person0")
          then <result> {$person/name} </result>
          else ()
} </query1>

XMark Q6

<query6> {
  for $site in //site return
    for $regions in $site/regions return
      $regions//item
} </query6>

XMark Q8

<query8> {
  for $site in /site return
    for $people in $site/people return
      for $person in $people/person return
        <item> {
          (
            <person> {$person/name} </person>,
            <items_bought> {
              for $site2 in /site return
                for $cas in $site2/closed_auctions return
                  for $ca in $cas/closed_auction return
                    for $buyer in $ca/buyer return
                      if ($buyer/buyer_person=$person/person_id)
                        then <result> {$ca} </result>
                        else ()
              } </items_bought>
          )
        } </item>
} </query8>

XMark Q13

<query13> {
  for $site in /site return
    for $regions in $site/regions return
      for $australia in $regions/australia return
        for $item in $australia/item return
          <item> {
            (
              <name> {$item/name} </name>,
              <desc> {$item/description} </desc>
            )
          } </item>
} </query13>

XMark Q20

<query20> {
  for $site in /site return
    for $people in $site/people return
      for $person in $people/person return
        if (fn:not(fn:exists($person/person_income)))
          then $person
          else ()
} </query20>