The SP²Bench SPARQL Performance Benchmark

Abstract

Recently, the SPARQL query language for RDF has reached the W3C recommendation status. Addressing the needs of this new standard, the database community is currently exploring efficient storage techniques for RDF data and evaluation strategies for SPARQL queries. Unfortunately, the evaluation of virtually all existing approaches relies on customized, user-defined experiments. This makes it particularly difficult to (1) compare optimization strategies, (2) estimate their generality, and (3) predict their benefits for real-world scenarios. In order to overcome these deficiencies we have developed a comprehensive benchmark for SPARQL, called SP²Bench.

Our benchmark comprises a data-generator for arbitrarily large documents, which builds upon the well-known DBLP scenario, and thus comes close to a real-world application scenario. The benchmark queries implement meaningful requests on top of this data, thereby testing typical SPARQL operator constellations and RDF access patterns. With this focus, our benchmark allows to easiliy detect deficiencies in current SPARQL implementations and can be used to tune existing engines.

Resources

On this site, we provide the following resources.

• Publications
  • The SP²Bench Technical Report, including a detailed description of the SP²Bench framework and benchmark results for selected engines
  • A short version of the Technical Report above has been accepted for publication at ICDE'09 [ .pdf ]
  • An experimental study of RDF Data Management approaches carried out on top of the SP²Bench framework has been carried out in our ISWC'08 publication [ .pdf ]
• Benchmark specification, description, data generator, and other material
  • A short description of the DBLP RDF data that is created by the SP²Bench data generator
  • The SP²Bench benchmark queries
  • A download section, where you find both the current version of the data generator and the benchmark queries as text files
  • SQL-translations of the benchmark queries as used for the experiments of our ISWC publication [ .pdf ]

The ICDE paper and the Technical Report linked above highlight design decisions and provide detailed information on data generation, the DBLP characteristics that are mirrored by our data generator, an in-depth discussion of the specific challenges that the benchmark queries impose to SPARQL engines, and benchmark results for selected SPARQL engines.

Project Members

• Contact: Michael Schmidt (Freiburg University Database Group)
• Thomas Hornung (Freiburg University Database Group)
• Norbert Küchlin (Freiburg University Database Group)
• Georg Lausen (Freiburg University Database Group)
• Christoph Pinkel (MTC Infomedia OHG)

References and Related Projects

• The Berlin SPARQL Benchmark is another nice benchmark project for SPARQL, designed by Chris Bizer and Andreas Schultz [ home ]
• W3C RDF(S) Specifications [ RDF | RDF Semantics | RDF Schema ]
• W3C SPARQL Specification [ SPARQL ]
• Original DBLP Dataset [ home ]
• D2R Server Publishing the DBLP Bibliography Database in RDF [ home ]
• Ontologies that are used by our data generator [ FOAF | SWRC | Dublin Core ]