OpenLink Community Blog

More on RDF and Vertical Storage

Mon, 11 Jun 2007 08:35:00 GMT

We actually did the experiment I mentioned a couple of posts back, about storing RDF triples column-wise.

The test loads 4.8 million triples of LUBM data and reads the whole set on one index and then checks if it finds the same row on another index.

Reading GSPO and checking OGPS takes 27 seconds. Doing the same with column wise bitmap indices on S, G, P and O takes 86 seconds.   The latter checks the existence of the row by AND'ing 4 bitmap indices and the former checks its existence by a single lookup in a multi-part index whose last part is a bitmap. The result is approximately what one would expect. The bitmap AND could be optimized a bit, dropping the time to maybe 70 seconds.

Now speaking of compression, it is true that column storage will work better. For example the G and P columns will compress to pretty much nothing. On a row layout they compress too but not to nothing since even if a value is not unique you have to store the place where the value is if you want to read rows in constant time per row.

What is nice with the 4 bitmaps is that no combination of search conditions is penalized. But the trick of using bitmaps for self-join is lost: You can't evaluate {?s a Person . ?s name "Mary"} by and'ing the S bitmaps for persons and for subjects named "Mary".

The 4 bitmap indices are remarkably compact, though. 8840 pages all together.
We could probably get the G, S, P, O columns in 3000 pages or so, using very little compression.
The OGPS index is   5169 pages and the GSPO index is 21243 pages.

None of the figures have any compression, except what a bitmap naturally produces.

Now we have figured out a modified row layout which will about double working set with the same memory and keep things in rows. We will try that. The GSPO index will be about 10000 pages and OGPS will be about 4500. We do not expect much impact on search or insert times.

We looked at using gzip for database pages. They go to between 1/4 to 1/3 page.   But this does not improve working set and having variable length pages generates all kinds of special cases you don’tt want. So we will improve working set first and deal with somewhat compressed data in the execution engine.
After that, maybe gzip will cut the size to 1/2 or so but that will be good for disk only. And it does not so much matter how much you transfer but how many seeks you do.

Still, column-wise storage will likely win for size. So if the working set is much larger than memory this may have an edge. To keep all bases covered we will eventually add this as an option.

Announcing Virtuoso Open-Source Edition v5.0.0

Thu, 12 Apr 2007 13:48:34 GMT

All, OpenLink Software are pleased to announce a new release of Virtuoso, Open-Source Edition, version 5.0.0. This version includes:

Significant rewrite of database engine resulting in 50%-100% improvement on single CPU and in some cases up to 300% on multiprocessor CPUs by decreasing resource-contention between threads and other optimizations.
Radical expansion of RDF support including

In-built middleware (called the Sponger) for transforming non-RDF into RDF "on the fly" (e.g. producing Triples from Microformats, REST-style Web Services, and (X)HTML etc.)
Full Text Indexing of Literal Objects in Triple Patterns (via Filter or magic bif:contains predicate applied to Literal Objects)
Basic Inferencing (Subclass and Subproperty Support)
SPARQL Aggregate Functions
SPARQL Update Language Support (Updates, Inserts, Deletions in SPARQL)
Improved Support of XML Schema Type System (including the use of XML Schema Complex Types as Objects of bif:xcontains predicate)

Enhancements to the in-built SPARQL to SQL Compiler's Cost Optimizer
Performance Optimizations to RDF VIEWs (SQL to RDF Mapping)
Various bug-fixes

NOTE: Databases created with earlier versions of Virtuoso will be automatically upgraded to Virtuoso 5.0 but after upgrade will not be readable with older Virtuoso versions. For more information please see: Virtuoso Open Source Edition: Home Page: http://virtuoso.openlinksw.com/wiki/main/ Download Page: http://virtuoso.openlinksw.com/wiki/main/Main/VOSDownload OpenLink Data Spaces: Home Page: http://virtuoso.openlinksw.com/wiki/main/Main/OdsIndex SPARQL Usage Examples (re. SIOC, FOAF, AtomOWL, SKOS): http://virtuoso.openlinksw.com/wiki/main/Main/ODSSIOCRef Interactive SPARQL Demo: http://demo.openlinksw.com/isparql/ OpenLink AJAX Toolkit (OAT): Project Page: http://sourceforge.net/projects/oat Live Demonstration: http://demo.openlinksw.com/DAV/JS/oat/index.html

Technorati Tags: database, databases, open-source, OpenLink, RDBMS, RDF, semantic web, Semantic Web, SPARQL, virtuoso

Announcing Virtuoso Open-Source Edition v5.0.0

Thu, 12 Apr 2007 10:30:29 GMT

All, OpenLink Software are pleased to announce a new release of Virtuoso , Open-Source Edition, version 5.0.0. This version includes:

Significant rewrite of database engine resulting in 50%-100% improvement on single CPU and in some cases up to 300% on multiprocessor CPUs by decreasing resource-contention between threads and other optimizations.
Radical expansion of RDF support including

In-built middleware (called the Sponger) for transforming non-RDF into RDF "on the fly" (e.g. producing Triples from Microformats, REST-style Web Services, and (X)HTML etc.)
Full Text Indexing of Literal Objects in Triple Patterns (via Filter or magic bif:contains predicate applied to Literal Objects)
Basic Inferencing (Subclass and Subproperty Support)
SPARQL Aggregate Functions
SPARQL Update Language Support (Updates, Inserts, Deletions in SPARQL)
Improved Support of XML Schema Type System (including the use of XML Schema Complex Types as Objects of bif:xcontains predicate)
Enhancements to the in-built SPARQL to SQL Compiler's Cost Optimizer
Performance Optimizations to RDF VIEWs (SQL to RDF Mapping)

Various bug-fixes

Home Page: http://virtuoso.openlinksw.com/wiki/main/
Download Page: http://virtuoso.openlinksw.com/wiki/main/Main/VOSDownload

OpenLink Data Spaces:

Home Page: http://virtuoso.openlinksw.com/wiki/main/Main/OdsIndex
SPARQL Usage Examples (re. SIOC, FOAF, AtomOWL, SKOS): http://virtuoso.openlinksw.com/wiki/main/Main/ODSSIOCRef
Interactive SPARQL Demo: http://demo.openlinksw.com/isparql/

OpenLink AJAX Too lkit (OAT):

Project Page: http://sourceforge.net/projects/oat
Live Demonstration: http://demo.openlinksw.com/DAV/JS/oat/index.html

Technorati Tags: database, databases, open-source, OpenLink, SPARQL, RDBMS, RDF, semantic web, Semantic Web, virtuoso

Virtuoso Open Source 5.0 Release Imminent

Fri, 16 Mar 2007 09:55:29 GMT

We are a couple of days from releasing the Virtuoso Open Source 5.0 cut. This will make the technology that we are showing with Dbpedia and the various OpenLink web sites available to the public.

The updates involve:

Significant database engine improvements, as discussed in previous posts.
Tons of RDF related bug fixes.
Text index extension to SPARQL
New SQL data type capturing the whole XML Schema scalar type system used in RDF.

Soon to follow are:

Basic inference for RDF, including type and property subsumption.
Whole new disk IO system with much better disk locality.

Existing databases will be automatically upgraded when started with the new Virtuoso 5.0 server. Note that after upgrade, the RDF data is not backward compatible.

We will be rolling out more Virtuoso hosted semantic web content in the Linking Open Data project, part of our participation in the Semantic Web Education and Outreach activity at W3C.