OpenLink Virtuoso (Product Blog) http://www.openlinksw.com/blog/vdb/blog/ A great place to track Virtuoso's rapid evolution. kidehen@openlinksw.com 2009-11-23T10:44:53Z More on RDF and Vertical Storage 2007-06-11T08:35:00Z We actually did the experiment I mentioned a couple of posts back, about storing RDF triples column-wise. The test loads 4.8 million triples of LUBM data and reads the whole set on one index and then checks if it finds the same row on another index. Reading GSPO and checking OGPS takes 27 seconds.  Doing the same with column wise bitmap indices on S, G, P and O takes 86 seconds.   The latter checks the existence of the row by AND'ing 4 bitmap indices and the former checks its existence by a single lookup in a multi-part index whose last part is a bitmap.  The result is approximately what one would expect.  The bitmap AND could be optimized a bit, dropping the time to maybe 70 seconds.  Now speaking of compression, it is true that column storage will work better.  For example the G and P columns will compress to pretty much nothing.  On a row layout they compress too but not to nothing since even if a value is not unique you have to store the place where the value is if you want to read rows in constant time per row. What is nice with the 4 bitmaps is that no combination of search conditions is penalized.  But the trick of using bitmaps for self-join is lost:  You can't evaluate {?s a Person . ?s name "Mary"} by and'ing the S bitmaps for persons and for subjects named "Mary". The 4 bitmap indices are remarkably compact, though. 8840 pages all together.We could probably get the G, S, P, O columns in 3000 pages or so, using very little  compression.The OGPS index is   5169 pages and the GSPO index is 21243 pages. None of the figures have any compression, except what a bitmap naturally produces. Now we have figured out a modified row layout which will about double working set with the same memory and keep things in rows.  We will try that.  The GSPO index will be about  10000 pages and OGPS will be about 4500.  We do not expect much impact on search or insert times. We looked at using gzip for database pages.  They go to between 1/4 to 1/3 page.   But this does not improve working set and having variable length pages generates all kinds of special cases you don’tt want.  So we will improve working set first and deal with somewhat compressed data in the execution engine. After that, maybe gzip will cut the size to 1/2 or so but  that will be good for disk only.  And it does not so much matter how much you transfer but how many seeks you do. Still, column-wise storage will likely win for size.  So if the working set is much larger than memory this may have an edge.  To keep all bases covered we will eventually add this as an option.  Semantic Web |Database |Databases |Virtuoso |SPARQL|RDF
The test loads 4.8 million triples of LUBM data and reads the whole set on one index and then checks if it finds the same row on another index.

Reading GSPO and checking OGPS takes 27 seconds.  Doing the same with column wise bitmap indices on S, G, P and O takes 86 seconds.   The latter checks the existence of the row by AND'ing 4 bitmap indices and the former checks its existence by a single lookup in a multi-part index whose last part is a bitmap.  The result is approximately what one would expect.  The bitmap AND could be optimized a bit, dropping the time to maybe 70 seconds. 

Now speaking of compression, it is true that column storage will work better.  For example the G and P columns will compress to pretty much nothing.  On a row layout they compress too but not to nothing since even if a value is not unique you have to store the place where the value is if you want to read rows in constant time per row.

What is nice with the 4 bitmaps is that no combination of search conditions is penalized.  But the trick of using bitmaps for self-join is lost:  You can't evaluate {?s a Person . ?s name "Mary"} by and'ing the S bitmaps for persons and for subjects named "Mary".

The 4 bitmap indices are remarkably compact, though. 8840 pages all together.
We could probably get the G, S, P, O columns in 3000 pages or so, using very little  compression.
The OGPS index is   5169 pages and the GSPO index is 21243 pages.

None of the figures have any compression, except what a bitmap naturally produces.

Now we have figured out a modified row layout which will about double working set with the same memory and keep things in rows.  We will try that.  The GSPO index will be about  10000 pages and OGPS will be about 4500.  We do not expect much impact on search or insert times.

We looked at using gzip for database pages.  They go to between 1/4 to 1/3 page.   But this does not improve working set and having variable length pages generates all kinds of special cases you don’tt want.  So we will improve working set first and deal with somewhat compressed data in the execution engine.
After that, maybe gzip will cut the size to 1/2 or so but  that will be good for disk only.  And it does not so much matter how much you transfer but how many seeks you do.

Still, column-wise storage will likely win for size.  So if the working set is much larger than memory this may have an edge.  To keep all bases covered we will eventually add this as an option.
 

| | | ||


]]> Announcing Virtuoso Open-Source Edition v5.0.0 2007-04-12T13:48:34Z All, OpenLink Software are pleased to announce a new release of Virtuoso, Open-Source Edition, version 5.0.0. This version includes: Significant rewrite of database engine resulting in 50%-100% improvement on single CPU and in some cases up to 300% on multiprocessor CPUs by decreasing resource-contention between threads and other optimizations. Radical expansion of RDF support including In-built middleware (called the Sponger) for transforming non-RDF into RDF "on the fly" (e.g. producing Triples from Microformats, REST-style Web Services, and (X)HTML etc.) Full Text Indexing of Literal Objects in Triple Patterns (via Filter or magic bif:contains predicate applied to Literal Objects) Basic Inferencing (Subclass and Subproperty Support) SPARQL Aggregate Functions SPARQL Update Language Support (Updates, Inserts, Deletions in SPARQL) Improved Support of XML Schema Type System (including the use of XML Schema Complex Types as Objects of bif:xcontains predicate) Enhancements to the in-built SPARQL to SQL Compiler's Cost Optimizer Performance Optimizations to RDF VIEWs (SQL to RDF Mapping) Various bug-fixes NOTE: Databases created with earlier versions of Virtuoso will be automatically upgraded to Virtuoso 5.0 but after upgrade will not be readable with older Virtuoso versions. For more information please see: Virtuoso Open Source Edition: Home Page: http://virtuoso.openlinksw.com/wiki/main/ Download Page: http://virtuoso.openlinksw.com/wiki/main/Main/VOSDownload OpenLink Data Spaces: Home Page: http://virtuoso.openlinksw.com/wiki/main/Main/OdsIndex SPARQL Usage Examples (re. SIOC, FOAF, AtomOWL, SKOS): http://virtuoso.openlinksw.com/wiki/main/Main/ODSSIOCRef Interactive SPARQL Demo: http://demo.openlinksw.com/isparql/ OpenLink AJAX Toolkit (OAT): Project Page: http://sourceforge.net/projects/oat Live Demonstration: http://demo.openlinksw.com/DAV/JS/oat/index.html Technorati Tags: database, databases, open-source, OpenLink, RDBMS, RDF, semantic web, Semantic Web, SPARQL, virtuoso Virtuoso, Open-Source Edition, version 5.0.0. This version includes:
  • Significant rewrite of database engine resulting in 50%-100% improvement on single CPU and in some cases up to 300% on multiprocessor CPUs by decreasing resource-contention between threads and other optimizations.
  • Radical expansion of RDF support including
  • In-built middleware (called the Sponger) for transforming non-RDF into RDF "on the fly" (e.g. producing Triples from Microformats, REST-style Web Services, and (X)HTML etc.)
  • Full Text Indexing of Literal Objects in Triple Patterns (via Filter or magic bif:contains predicate applied to Literal Objects)
  • Basic Inferencing (Subclass and Subproperty Support)
  • SPARQL Aggregate Functions
  • SPARQL Update Language Support (Updates, Inserts, Deletions in SPARQL)
  • Improved Support of XML Schema Type System (including the use of XML Schema Complex Types as Objects of bif:xcontains predicate)
  • Enhancements to the in-built SPARQL to SQL Compiler's Cost Optimizer
  • Performance Optimizations to RDF VIEWs (SQL to RDF Mapping)
  • Various bug-fixes
NOTE: Databases created with earlier versions of Virtuoso will be automatically upgraded to Virtuoso 5.0 but after upgrade will not be readable with older Virtuoso versions. For more information please see: Virtuoso Open Source Edition: Home Page: http://virtuoso.openlinksw.com/wiki/main/ Download Page: http://virtuoso.openlinksw.com/wiki/main/Main/VOSDownload OpenLink Data Spaces: Home Page: http://virtuoso.openlinksw.com/wiki/main/Main/OdsIndex SPARQL Usage Examples (re. SIOC, FOAF, AtomOWL, SKOS): http://virtuoso.openlinksw.com/wiki/main/Main/ODSSIOCRef Interactive SPARQL Demo: http://demo.openlinksw.com/isparql/ OpenLink AJAX Toolkit (OAT): Project Page: http://sourceforge.net/projects/oat Live Demonstration: http://demo.openlinksw.com/DAV/JS/oat/index.html

Technorati Tags: , , , , , , , , ,

]]> Virtuoso Open Source 5.0 Release Imminent 2007-03-16T09:55:29Z Virtuoso Open Source 5.0 Release Imminent We are a couple of days from releasing the Virtuoso Open Source 5.0 cut. This will make the technology that we are showing with Dbpedia and the various OpenLink web sites available to the public. The updates involve: Significant database engine improvements, as discussed in previous posts. Tons of RDF related bug fixes. Text index extension to SPARQL New SQL data type capturing the whole XML Schema scalar type system used in RDF. Soon to follow are: Basic inference for RDF, including type and property subsumption. Whole new disk IO system with much better disk locality. Existing databases will be automatically upgraded when started with the new Virtuoso 5.0 server. Note that after upgrade, the RDF data is not backward compatible. We will be rolling out more Virtuoso hosted semantic web content in the Linking Open Data project, part of our participation in the Semantic Web Education and Outreach activity at W3C. Semantic Web |Database |Databases |Virtuoso |SPARQL|RDF
Virtuoso Open Source 5.0 Release Imminent

We are a couple of days from releasing the Virtuoso Open Source 5.0 cut. This will make the technology that we are showing with Dbpedia and the various OpenLink web sites available to the public.

The updates involve:

  • Significant database engine improvements, as discussed in previous posts.
  • Tons of RDF related bug fixes.
  • Text index extension to SPARQL
  • New SQL data type capturing the whole XML Schema scalar type system used in RDF.

Soon to follow are:

  • Basic inference for RDF, including type and property subsumption.
  • Whole new disk IO system with much better disk locality.

Existing databases will be automatically upgraded when started with the new Virtuoso 5.0 server. Note that after upgrade, the RDF data is not backward compatible.

We will be rolling out more Virtuoso hosted semantic web content in the Linking Open Data project, part of our participation in the Semantic Web Education and Outreach activity at W3C.

| | | ||

]]> Virtuoso Open Source 5.0 Release Imminent 2007-03-16T09:55:29Z Virtuoso Open Source 5.0 Release Imminent We are a couple of days from releasing the Virtuoso Open Source 5.0 cut. This will make the technology that we are showing with Dbpedia and the various OpenLink web sites available to the public. The updates involve: Significant database engine improvements, as discussed in previous posts. Tons of RDF related bug fixes. Text index extension to SPARQL New SQL data type capturing the whole XML Schema scalar type system used in RDF. Soon to follow are: Basic inference for RDF, including type and property subsumption. Whole new disk IO system with much better disk locality. Existing databases will be automatically upgraded when started with the new Virtuoso 5.0 server. Note that after upgrade, the RDF data is not backward compatible. We will be rolling out more Virtuoso hosted semantic web content in the Linking Open Data project, part of our participation in the Semantic Web Education and Outreach activity at W3C. Semantic Web |Database |Databases |Virtuoso |SPARQL|RDF
Virtuoso Open Source 5.0 Release Imminent

We are a couple of days from releasing the Virtuoso Open Source 5.0 cut. This will make the technology that we are showing with Dbpedia and the various OpenLink web sites available to the public.

The updates involve:

  • Significant database engine improvements, as discussed in previous posts.
  • Tons of RDF related bug fixes.
  • Text index extension to SPARQL
  • New SQL data type capturing the whole XML Schema scalar type system used in RDF.

Soon to follow are:

  • Basic inference for RDF, including type and property subsumption.
  • Whole new disk IO system with much better disk locality.

Existing databases will be automatically upgraded when started with the new Virtuoso 5.0 server. Note that after upgrade, the RDF data is not backward compatible.

We will be rolling out more Virtuoso hosted semantic web content in the Linking Open Data project, part of our participation in the Semantic Web Education and Outreach activity at W3C.

| | | ||

]]> Recent Virtuoso Developments 2006-09-19T11:45:29Z Recent Virtuoso Developments We have been extensively working on virtual database refinements.  There aremany SQL cost model adjustments to better model  distributed queries and wenow support direct access to Oracle and Informix statistics system tables.Thus, when you attach  a table from one or the other, you automatically getup to date statistics.  This helps Virtuoso optimize distributed  queries.Also the documentation is updated as concerns these, with a new section ondistributed query optimization. On the applications side, we have been keeping up with the SIOC RDF ontologydevelopments.  All ODS applications now make  their data available as SIOCgraphs for download and SPARQL query access. What is most exciting however is our advance in mapping relational data intoRDF.  We now have a mapping language that makes  arbitrary legacy data in Virtuoso or elsewhere in the relational world RDF queriable.  We will putout a white paper on  this in a few days. Also we have some innovations in mind for optimizing the physical storage ofRDF triples.  We keep experimenting, now with  our sights set to the highend of triple storage, towards billion triple data sets.  We areexperimenting with a new more space efficient index structure  for betterworking set behavior.  Next week will yield the first results. SQL Database Databases Virtuoso Programming Semantic Web SPARQL
Recent Virtuoso Developments

We have been extensively working on virtual database refinements.  There aremany SQL cost model adjustments to better model  distributed queries and wenow support direct access to Oracle and Informix statistics system tables.Thus, when you attach  a table from one or the other, you automatically getup to date statistics.  This helps Virtuoso optimize distributed  queries.Also the documentation is updated as concerns these, with a new section ondistributed query optimization.

On the applications side, we have been keeping up with the SIOC RDF ontologydevelopments.  All ODS applications now make  their data available as SIOCgraphs for download and SPARQL query access.

What is most exciting however is our advance in mapping relational data intoRDF.  We now have a mapping language that makes  arbitrary legacy data in Virtuoso or elsewhere in the relational world RDF queriable.  We will putout a white paper on  this in a few days.

Also we have some innovations in mind for optimizing the physical storage ofRDF triples.  We keep experimenting, now with  our sights set to the highend of triple storage, towards billion triple data sets.  We areexperimenting with a new more space efficient index structure  for betterworking set behavior.  Next week will yield the first results.

]]> Recent Virtuoso Developments 2006-09-19T11:45:29Z Recent Virtuoso Developments We have been extensively working on virtual database refinements.  There aremany SQL cost model adjustments to better model  distributed queries and wenow support direct access to Oracle and Informix statistics system tables.Thus, when you attach  a table from one or the other, you automatically getup to date statistics.  This helps Virtuoso optimize distributed  queries.Also the documentation is updated as concerns these, with a new section ondistributed query optimization. On the applications side, we have been keeping up with the SIOC RDF ontologydevelopments.  All ODS applications now make  their data available as SIOCgraphs for download and SPARQL query access. What is most exciting however is our advance in mapping relational data intoRDF.  We now have a mapping language that makes  arbitrary legacy data in Virtuoso or elsewhere in the relational world RDF queriable.  We will putout a white paper on  this in a few days. Also we have some innovations in mind for optimizing the physical storage ofRDF triples.  We keep experimenting, now with  our sights set to the highend of triple storage, towards billion triple data sets.  We areexperimenting with a new more space efficient index structure  for betterworking set behavior.  Next week will yield the first results. SQL Database Databases Virtuoso Programming Semantic Web SPARQL
Recent Virtuoso Developments

We have been extensively working on virtual database refinements.  There aremany SQL cost model adjustments to better model  distributed queries and wenow support direct access to Oracle and Informix statistics system tables.Thus, when you attach  a table from one or the other, you automatically getup to date statistics.  This helps Virtuoso optimize distributed  queries.Also the documentation is updated as concerns these, with a new section ondistributed query optimization.

On the applications side, we have been keeping up with the SIOC RDF ontologydevelopments.  All ODS applications now make  their data available as SIOCgraphs for download and SPARQL query access.

What is most exciting however is our advance in mapping relational data intoRDF.  We now have a mapping language that makes  arbitrary legacy data in Virtuoso or elsewhere in the relational world RDF queriable.  We will putout a white paper on  this in a few days.

Also we have some innovations in mind for optimizing the physical storage ofRDF triples.  We keep experimenting, now with  our sights set to the highend of triple storage, towards billion triple data sets.  We areexperimenting with a new more space efficient index structure  for betterworking set behavior.  Next week will yield the first results.

]]>