OpenLink Virtuoso (Product Blog)

Virtuoso update

Mon, 24 Sep 2007 15:03:10 GMT

I have been occupied with getting the Virtuoso cluster to fuifull functionality. It now does insert/delete/update and select over partitions spread over multiple instances and can send co-located joins and other co-located query fragments over to partner nodes. Of course, the more functionality is transferred per message, the better.

Most of the code is definitely written. Now it is a question of testing and adding optimizations here and there, also detecting distributed deadlocks.

I'll go to the SABRE conference in Leipzig this week, where we present an update on Virtuoso. This is an updated verversion of the Virtuoso intro paper, now also at the Virtuoso web site.

In another month, I'll be at the RDF Relational Mapping workshop in Cambridge, MA. There will be more updates on our results before then.

More on RDF and Vertical Storage

Mon, 11 Jun 2007 08:35:00 GMT

We actually did the experiment I mentioned a couple of posts back, about storing RDF triples column-wise.

The test loads 4.8 million triples of LUBM data and reads the whole set on one index and then checks if it finds the same row on another index.

Reading GSPO and checking OGPS takes 27 seconds. Doing the same with column wise bitmap indices on S, G, P and O takes 86 seconds.   The latter checks the existence of the row by AND'ing 4 bitmap indices and the former checks its existence by a single lookup in a multi-part index whose last part is a bitmap. The result is approximately what one would expect. The bitmap AND could be optimized a bit, dropping the time to maybe 70 seconds.

Now speaking of compression, it is true that column storage will work better. For example the G and P columns will compress to pretty much nothing. On a row layout they compress too but not to nothing since even if a value is not unique you have to store the place where the value is if you want to read rows in constant time per row.

What is nice with the 4 bitmaps is that no combination of search conditions is penalized. But the trick of using bitmaps for self-join is lost: You can't evaluate {?s a Person . ?s name "Mary"} by and'ing the S bitmaps for persons and for subjects named "Mary".

The 4 bitmap indices are remarkably compact, though. 8840 pages all together.
We could probably get the G, S, P, O columns in 3000 pages or so, using very little compression.
The OGPS index is   5169 pages and the GSPO index is 21243 pages.

None of the figures have any compression, except what a bitmap naturally produces.

Now we have figured out a modified row layout which will about double working set with the same memory and keep things in rows. We will try that. The GSPO index will be about 10000 pages and OGPS will be about 4500. We do not expect much impact on search or insert times.

We looked at using gzip for database pages. They go to between 1/4 to 1/3 page.   But this does not improve working set and having variable length pages generates all kinds of special cases you don’tt want. So we will improve working set first and deal with somewhat compressed data in the execution engine.
After that, maybe gzip will cut the size to 1/2 or so but that will be good for disk only. And it does not so much matter how much you transfer but how many seeks you do.

Still, column-wise storage will likely win for size. So if the working set is much larger than memory this may have an edge. To keep all bases covered we will eventually add this as an option.

Virtuoso Open Source 5.0 Release Imminent

Fri, 16 Mar 2007 09:55:29 GMT

We are a couple of days from releasing the Virtuoso Open Source 5.0 cut. This will make the technology that we are showing with Dbpedia and the various OpenLink web sites available to the public.

The updates involve:

Significant database engine improvements, as discussed in previous posts.
Tons of RDF related bug fixes.
Text index extension to SPARQL
New SQL data type capturing the whole XML Schema scalar type system used in RDF.

Soon to follow are:

Basic inference for RDF, including type and property subsumption.
Whole new disk IO system with much better disk locality.

Existing databases will be automatically upgraded when started with the new Virtuoso 5.0 server. Note that after upgrade, the RDF data is not backward compatible.

We will be rolling out more Virtuoso hosted semantic web content in the Linking Open Data project, part of our participation in the Semantic Web Education and Outreach activity at W3C.

Virtuoso Open Source 5.0 Release Imminent

Fri, 16 Mar 2007 09:55:29 GMT

We are a couple of days from releasing the Virtuoso Open Source 5.0 cut. This will make the technology that we are showing with Dbpedia and the various OpenLink web sites available to the public.

The updates involve:

Significant database engine improvements, as discussed in previous posts.
Tons of RDF related bug fixes.
Text index extension to SPARQL
New SQL data type capturing the whole XML Schema scalar type system used in RDF.

Soon to follow are:

Basic inference for RDF, including type and property subsumption.
Whole new disk IO system with much better disk locality.

Existing databases will be automatically upgraded when started with the new Virtuoso 5.0 server. Note that after upgrade, the RDF data is not backward compatible.

We will be rolling out more Virtuoso hosted semantic web content in the Linking Open Data project, part of our participation in the Semantic Web Education and Outreach activity at W3C.

Recent Virtuoso Developments

Tue, 19 Sep 2006 11:45:29 GMT

We have been extensively working on virtual database refinements. There aremany SQL cost model adjustments to better model distributed queries and wenow support direct access to Oracle and Informix statistics system tables.Thus, when you attach a table from one or the other, you automatically getup to date statistics. This helps Virtuoso optimize distributed queries.Also the documentation is updated as concerns these, with a new section ondistributed query optimization.

On the applications side, we have been keeping up with the SIOC RDF ontologydevelopments. All ODS applications now make their data available as SIOCgraphs for download and SPARQL query access.

What is most exciting however is our advance in mapping relational data intoRDF. We now have a mapping language that makes arbitrary legacy data in Virtuoso or elsewhere in the relational world RDF queriable. We will putout a white paper on this in a few days.

Also we have some innovations in mind for optimizing the physical storage ofRDF triples. We keep experimenting, now with our sights set to the highend of triple storage, towards billion triple data sets. We areexperimenting with a new more space efficient index structure for betterworking set behavior. Next week will yield the first results.

SQL Database Databases Virtuoso Programming Semantic Web SPARQL

Recent Virtuoso Developments

Tue, 19 Sep 2006 11:45:29 GMT

On the applications side, we have been keeping up with the SIOC RDF ontologydevelopments. All ODS applications now make their data available as SIOCgraphs for download and SPARQL query access.

SQL Database Databases Virtuoso Programming Semantic Web SPARQL