Benchmark story · v4 · VOS 7 vs QLever
SPARQL benchmark

Virtuoso Open Source (VOS Develop) vs QLever (with NUMA)

I

Who wins where🔗

For each workload family, the geometric-mean time of both engines on the same queries, with the faster one named. Speed-up shows how many times as long the slower engine needs; Times shows the times themselves. Group by workload family or by result type, and pick any row to isolate that group everywhere.

II

How fast is a typical query?🔗

Each curve shows what share of queries has finished by a given time. A curve that sits left and high is quicker across the board; one that stops short of the top never finished some queries.

III

Every query, one tile🔗

Each tile is a query, coloured by its fastest engine. Saturation is the margin: pale is a photo finish, solid is a 20× blowout. Pick a tile to race it.

IV

Why the numbers look like this🔗

Each explainer carries a tag from the reading guide above: architecture (how the engines are built), method (how the numbers were taken). Result callouts and the race panel link here, and each explainer lists the measured queries it covers.

The full data🔗
Sortable, filtered by everything above.

Reading this objectively🔗

  • This edition compares VOS 7 and QLever on DBLP data with Sparqloscope queries, run on the same machine. Times are in seconds, measured warm and reported as the best of several rounds. Differences under 5% in the few-millisecond range are noise.
  • The test machine:
    • Processors: 2 × Intel Xeon E5-2630, each with 6 cores at 2.30 GHz. In total, 12 cores and 24 threads.
    • Memory: 192 GB, from 3 × Corsair XMS3 DIMM kits of 64 GB each (DDR3-1333). Each socket has its own memory, and reaching the other socket’s memory costs about twice as much as local memory.
    • Operating system: CentOS release 6.2 (Final).
    • File system: ext2, on SSDs.
  • All runs used the same conditions: started under numactl --interleave=all, then warmup.sql, then the benchmark. The next point explains the numactl setting and the memory note above.
  • What the NUMA items mean for these results:
    • NUMA (non-uniform memory access) means that how long a memory read takes depends on where the memory is. On this machine with two processor sockets, each processor reads its own socket’s memory fastest and the other socket’s memory more slowly, here about twice as slowly.
    • What numactl --interleave=all does. It spreads the memory a program uses evenly across both sockets, instead of placing it mostly next to whichever thread asked for it first.
    • For the comparison. Every thread of every engine sees about the same mix of near and far memory. No engine was given the fast placement or left with the slow one, so the comparison treats both engines alike.
    • For the absolute times. Interleaving trades the best case for a predictable average. On a machine with one socket, or with memory placed near each thread, the times could be lower, and the effect could differ between engines. This report did not measure that.
  • VOS configuration (INI settings): 
    [Flags]
enable_mt_txn                   = 1
enable_mt_transact              = 1
qp_thread_min_usec              = 100
enable_exact_p_stat             = 1
hash_join_enable                = 2
enable_g_in_sec                 = 1
tn_max_memory                   = 10000000000
enable_sparql_count_distinct_dt = 1
enable_sparql_sample_min        = 1
  • Geometric means compare only queries every shown engine completed. Timeouts are excluded there and drawn as unfinished on the curves.
  • Speed-up callouts ignore pairs where both engines finish under 50 ms, where ratios are mostly timer resolution.