http://www.openlinksw.com/weblog/dav/dav-blog-1/
OpenLink Community Blog
A Collection of blogs by OpenLink Staff
kidehen@openlinksw.com
kidehen@openlinksw.com
2009-11-23T13:03:28Z
Virtuoso Universal Server 05.12.3041
http://www.openlinksw.com/weblog/public/images/vbloglogo.gif
Virtuoso update
http://www.openlinksw.com/blog/vdb/blog/?date=2007-09-24#1263
2007-09-24T15:03:10Z
2007-09-24T11:03:10.000003-04:00
<div><div style="display:none;">Virtuoso update</div> <font face="Arial"> <font size="2">I have been occupied with getting the Virtuoso cluster to fuifull functionality. It now does insert/delete/update and select over partitions spread over multiple instances and can send co-located joins and other co-located query fragments over to partner nodes. <span class="093250614-24092007"> Of course, the more functionality is transferred per message, the better.</span> </font> </font> <div> <font face="Arial" size="2"><br />Most of the code is definitely written. Now it is a question of testing and adding optimizations here and there, also detecting distributed deadlocks.</font> </div> <div> <font face="Arial" size="2"><br />I'll go to the <a href="http://www.sabre-conference.com/">SABRE</a> conference in Leipzig this week, where we present an update on Virtuoso. This is an updated verversion of the Virtuoso intro paper, now also at the Virtuoso web site.</font> </div> <div> </div> <div> <font face="Arial" size="2">In another month, I'll be at the RDF Relational Mapping workshop in Cambridge, MA. There will be more updates on our results before then.</font> </div> <a href="index.vspx?tag=Database" rel="tag" style="display:none;">Database</a><a href="index.vspx?tag=Databases" rel="tag" style="display:none;">Databases</a><a href="index.vspx?tag=Virtuoso" rel="tag" style="display:none;">Virtuoso</a><a href="index.vspx?tag=Scalability" rel="tag" style="display:none;">Scalability</a><a href="index.vspx?tag=Clustering" rel="tag" style="display:none;">Clustering</a></div>
Virtuoso update
http://www.openlinksw.com/weblog/oerling/?date=2007-09-24#1262
2007-09-24T14:30:28Z
2007-09-24T11:03:10-04:00
<font face="Arial"> <font size="2">I have been occupied with getting the Virtuoso cluster to fuifull functionality. It now does insert/delete/update and select over partitions spread over multiple instances and can send co-located joins and other co-located query fragments over to partner nodes. <span class="093250614-24092007"> Of course, the more functionality is transferred per message, the better.</span> </font> </font> <div> <font face="Arial" size="2"><br />Most of the code is definitely written. Now it is a question of testing and adding optimizations here and there, also detecting distributed deadlocks.</font> </div> <div> <font face="Arial" size="2"><br />I'll go to the <a href="http://www.sabre-conference.com/">SABRE</a> conference in Leipzig this week, where we present an update on Virtuoso. This is an updated verversion of the Virtuoso intro paper, now also at the Virtuoso web site.</font> </div> <div> </div> <div> <font face="Arial" size="2">In another month, I'll be at the RDF Relational Mapping workshop in Cambridge, MA. There will be more updates on our results before then.</font> </div> <a href="index.vspx?tag=Database" rel="tag" style="display:none;">Database</a><a href="index.vspx?tag=Databases" rel="tag" style="display:none;">Databases</a><a href="index.vspx?tag=Virtuoso" rel="tag" style="display:none;">Virtuoso</a><a href="index.vspx?tag=Scalability" rel="tag" style="display:none;">Scalability</a><a href="index.vspx?tag=Clustering" rel="tag" style="display:none;">Clustering</a>
More on RDF and Vertical Storage
http://www.openlinksw.com/blog/vdb/blog/?date=2007-06-11#1223
2007-06-11T08:35:00Z
2007-06-11T04:36:17.000003-04:00
We actually did the experiment I mentioned a couple of posts back, about storing RDF triples column-wise. <br /> <br />The test loads 4.8 million triples of LUBM data and reads the whole set on one index and then checks if it finds the same row on another index.<br /> <br />Reading GSPO and checking OGPS takes 27 seconds.<span> </span>Doing the same with column wise bitmap indices on S, G, P and O takes 86 seconds.<span> </span>The latter checks the existence of the row by AND'ing 4 bitmap indices and the former checks its existence by a single lookup in a multi-part index whose last part is a bitmap.<span> </span>The result is approximately what one would expect.<span> </span>The bitmap AND could be optimized a bit, dropping the time to maybe 70 seconds.<span> </span> <br /> <br />Now speaking of compression, it is true that column storage will work better.<span> </span>For example the G and P columns will compress to pretty much nothing.<span> </span>On a row layout they compress too but not to nothing since even if a value is not unique you have to store the place where the value is if you want to read rows in constant time per row. <br /> <br />What is nice with the 4 bitmaps is that no combination of search conditions is penalized.<span> </span>But the trick of using bitmaps for self-join is lost:<span> </span>You can't evaluate {?s a Person . ?s name "Mary"} by and'ing the S bitmaps for persons and for subjects named "Mary". <br /> <br />The 4 bitmap indices are remarkably compact, though. 8840 pages all together.<br />We could probably get the G, S, P, O columns in 3000 pages or so, using very little<span> </span>compression.<br />The OGPS index is <span> </span>5169 pages and the GSPO index is 21243 pages. <br /> <br />None of the figures have any compression, except what a bitmap naturally produces. <br /> <br />Now we have figured out a modified row layout which will about double working set with the same memory and keep things in rows.<span> </span>We will try that.<span> </span>The GSPO index will be about<span> </span>10000 pages and OGPS will be about 4500.<span> </span>We do not expect much impact on search or insert times.<br /> <br />We looked at using gzip for database pages.<span> </span>They go to between 1/4 to 1/3 page.<span> </span>But this does not improve working set and having variable length pages generates all kinds of special cases you don’tt want.<span> </span>So we will improve working set first and deal with somewhat compressed data in the execution engine. <br />After that, maybe gzip will cut the size to 1/2 or so but<span> </span>that will be good for disk only.<span> </span>And it does not so much matter how much you transfer but how many seeks you do.<br /> <br />Still, column-wise storage will likely win for size.<span> </span>So if the working set is much larger than memory this may have an edge.<span> </span>To keep all bases covered we will eventually add this as an option.<br /> <br /> <p> <a href="http://www.technorati.com/tags/semantic%20web" rel="tag">Semantic Web</a> |<a href="http://www.technorati.com/tags/database" rel="tag">Database</a> |<a href="http://www.technorati.com/tags/databases" rel="tag">Databases</a> |<a href="http://www.technorati.com/tags/virtuoso" rel="tag">Virtuoso</a> |<a href="http://www.technorati.com/tags/sparql" rel="tag">SPARQL</a>|<a href="http://www.technorati.com/tag/rdf" rel="tag">RDF</a> </p> <br />
Virtuoso Open Source 5.0 Release Imminent
http://www.openlinksw.com/blog/vdb/blog/?date=2007-03-16#1160
2007-03-16T09:55:29Z
2007-03-16T05:55:29-04:00
<div> <div style="display:none;">Virtuoso Open Source 5.0 Release Imminent</div> <p>We are a couple of days from releasing the Virtuoso Open Source 5.0 cut. This will make the technology that we are showing with Dbpedia and the various OpenLink web sites available to the public.</p> <p>The updates involve:</p> <ul> <li>Significant database engine improvements, as discussed in previous posts.</li> <li>Tons of RDF related bug fixes.</li> <li>Text index extension to SPARQL</li> <li>New SQL data type capturing the whole XML Schema scalar type system used in RDF.</li> </ul> <p>Soon to follow are:</p> <ul> <li>Basic inference for RDF, including type and property subsumption.</li> <li>Whole new disk IO system with much better disk locality.</li> </ul> <p>Existing databases will be automatically upgraded when started with the new Virtuoso 5.0 server. Note that after upgrade, the RDF data is not backward compatible.</p> <p>We will be rolling out more Virtuoso hosted semantic web content in the <a href="http://esw.w3.org/topic/TaskForces/CommunityProjects/LinkingOpenData/">Linking Open Data project</a>, part of our participation in the Semantic Web Education and Outreach activity at W3C.</p> <p> <a href="http://www.technorati.com/tags/semantic%20web" rel="tag">Semantic Web</a> |<a href="http://www.technorati.com/tags/database" rel="tag">Database</a> |<a href="http://www.technorati.com/tags/databases" rel="tag">Databases</a> |<a href="http://www.technorati.com/tags/virtuoso" rel="tag">Virtuoso</a> |<a href="http://www.technorati.com/tags/sparql" rel="tag">SPARQL</a>|<a href="http://www.technorati.com/tag/rdf" rel="tag">RDF</a> </p> </div>
Virtuoso Open Source 5.0 Release Imminent
http://www.openlinksw.com/blog/vdb/blog/?date=2007-03-16#1160
2007-03-16T09:55:29Z
2007-03-16T05:55:29-04:00
<div> <div style="display:none;">Virtuoso Open Source 5.0 Release Imminent</div> <p>We are a couple of days from releasing the Virtuoso Open Source 5.0 cut. This will make the technology that we are showing with Dbpedia and the various OpenLink web sites available to the public.</p> <p>The updates involve:</p> <ul> <li>Significant database engine improvements, as discussed in previous posts.</li> <li>Tons of RDF related bug fixes.</li> <li>Text index extension to SPARQL</li> <li>New SQL data type capturing the whole XML Schema scalar type system used in RDF.</li> </ul> <p>Soon to follow are:</p> <ul> <li>Basic inference for RDF, including type and property subsumption.</li> <li>Whole new disk IO system with much better disk locality.</li> </ul> <p>Existing databases will be automatically upgraded when started with the new Virtuoso 5.0 server. Note that after upgrade, the RDF data is not backward compatible.</p> <p>We will be rolling out more Virtuoso hosted semantic web content in the <a href="http://esw.w3.org/topic/TaskForces/CommunityProjects/LinkingOpenData/">Linking Open Data project</a>, part of our participation in the Semantic Web Education and Outreach activity at W3C.</p> <p> <a href="http://www.technorati.com/tags/semantic%20web" rel="tag">Semantic Web</a> |<a href="http://www.technorati.com/tags/database" rel="tag">Database</a> |<a href="http://www.technorati.com/tags/databases" rel="tag">Databases</a> |<a href="http://www.technorati.com/tags/virtuoso" rel="tag">Virtuoso</a> |<a href="http://www.technorati.com/tags/sparql" rel="tag">SPARQL</a>|<a href="http://www.technorati.com/tag/rdf" rel="tag">RDF</a> </p> </div>
Recent Virtuoso Developments
http://www.openlinksw.com/blog/vdb/blog/?date=2006-09-19#1044
2006-09-19T11:45:29Z
2006-09-19T07:45:29.000003-04:00
<div> <div style="display:none;">Recent Virtuoso Developments</div> <p>We have been extensively working on virtual database refinements. There aremany SQL cost model adjustments to better model distributed queries and wenow support direct access to Oracle and Informix statistics system tables.Thus, when you attach a table from one or the other, you automatically getup to date statistics. This helps <a href="http://virtuoso.openlinksw.com/wiki/main/">Virtuoso</a> optimize distributed queries.Also the documentation is updated as concerns these, with a new section ondistributed query optimization.</p> <p>On the applications side, we have been keeping up with the SIOC RDF ontologydevelopments. All ODS applications now make their data available as SIOCgraphs for download and SPARQL query access.</p> <p>What is most exciting however is our advance in mapping relational data intoRDF. We now have a mapping language that makes arbitrary legacy data in <a href="http://virtuoso.openlinksw.com/wiki/main/">Virtuoso</a> or elsewhere in the relational world RDF queriable. We will putout a white paper on this in a few days.</p> <p>Also we have some innovations in mind for optimizing the physical storage ofRDF triples. We keep experimenting, now with our sights set to the highend of triple storage, towards billion triple data sets. We areexperimenting with a new more space efficient index structure for betterworking set behavior. Next week will yield the first results.</p> <a href="http://www.technorati.com/tags/sql" rel="tag">SQL</a> <a href="http://www.technorati.com/tags/database" rel="tag">Database</a> <a href="http://www.technorati.com/tags/databases" rel="tag">Databases</a> <a href="http://www.technorati.com/tags/virtuoso" rel="tag">Virtuoso</a> <a href="http://www.technorati.com/tags/programming" rel="tag">Programming</a> <a href="http://www.technorati.com/tags/semantic%20web" rel="tag">Semantic Web</a> <a href="http://www.technorati.com/tags/sparql" rel="tag">SPARQL</a> </div>
Recent Virtuoso Developments
http://www.openlinksw.com/blog/vdb/blog/?date=2006-09-19#1044
2006-09-19T11:45:29Z
2006-09-19T07:45:29.000003-04:00
<div> <div style="display:none;">Recent Virtuoso Developments</div> <p>We have been extensively working on virtual database refinements. There aremany SQL cost model adjustments to better model distributed queries and wenow support direct access to Oracle and Informix statistics system tables.Thus, when you attach a table from one or the other, you automatically getup to date statistics. This helps <a href="http://virtuoso.openlinksw.com/wiki/main/">Virtuoso</a> optimize distributed queries.Also the documentation is updated as concerns these, with a new section ondistributed query optimization.</p> <p>On the applications side, we have been keeping up with the SIOC RDF ontologydevelopments. All ODS applications now make their data available as SIOCgraphs for download and SPARQL query access.</p> <p>What is most exciting however is our advance in mapping relational data intoRDF. We now have a mapping language that makes arbitrary legacy data in <a href="http://virtuoso.openlinksw.com/wiki/main/">Virtuoso</a> or elsewhere in the relational world RDF queriable. We will putout a white paper on this in a few days.</p> <p>Also we have some innovations in mind for optimizing the physical storage ofRDF triples. We keep experimenting, now with our sights set to the highend of triple storage, towards billion triple data sets. We areexperimenting with a new more space efficient index structure for betterworking set behavior. Next week will yield the first results.</p> <a href="http://www.technorati.com/tags/sql" rel="tag">SQL</a> <a href="http://www.technorati.com/tags/database" rel="tag">Database</a> <a href="http://www.technorati.com/tags/databases" rel="tag">Databases</a> <a href="http://www.technorati.com/tags/virtuoso" rel="tag">Virtuoso</a> <a href="http://www.technorati.com/tags/programming" rel="tag">Programming</a> <a href="http://www.technorati.com/tags/semantic%20web" rel="tag">Semantic Web</a> <a href="http://www.technorati.com/tags/sparql" rel="tag">SPARQL</a> </div>
Recent Virtuoso Developments
http://www.openlinksw.com/weblog/oerling/?date=2006-09-19#1043
2006-09-19T10:59:30Z
2006-09-19T07:45:29-04:00
<p>We have been extensively working on virtual database refinements. There aremany SQL cost model adjustments to better model distributed queries and wenow support direct access to Oracle and Informix statistics system tables.Thus, when you attach a table from one or the other, you automatically getup to date statistics. This helps <a href="http://virtuoso.openlinksw.com/wiki/main/">Virtuoso</a> optimize distributed queries.Also the documentation is updated as concerns these, with a new section ondistributed query optimization.</p> <p>On the applications side, we have been keeping up with the SIOC RDF ontologydevelopments. All ODS applications now make their data available as SIOCgraphs for download and SPARQL query access.</p> <p>What is most exciting however is our advance in mapping relational data intoRDF. We now have a mapping language that makes arbitrary legacy data in <a href="http://virtuoso.openlinksw.com/wiki/main/">Virtuoso</a> or elsewhere in the relational world RDF queriable. We will putout a white paper on this in a few days.</p> <p>Also we have some innovations in mind for optimizing the physical storage ofRDF triples. We keep experimenting, now with our sights set to the highend of triple storage, towards billion triple data sets. We areexperimenting with a new more space efficient index structure for betterworking set behavior. Next week will yield the first results.</p> <a href="http://www.technorati.com/tags/sql" rel="tag">SQL</a> <a href="http://www.technorati.com/tags/database" rel="tag">Database</a> <a href="http://www.technorati.com/tags/databases" rel="tag">Databases</a> <a href="http://www.technorati.com/tags/virtuoso" rel="tag">Virtuoso</a> <a href="http://www.technorati.com/tags/programming" rel="tag">Programming</a> <a href="http://www.technorati.com/tags/semantic%20web" rel="tag">Semantic Web</a> <a href="http://www.technorati.com/tags/sparql" rel="tag">SPARQL</a>
Recent Virtuoso Developments
http://www.openlinksw.com/weblog/oerling/?date=2006-09-19#1043
2006-09-19T10:59:30Z
2006-09-19T07:45:29-04:00
<p>We have been extensively working on virtual database refinements. There aremany SQL cost model adjustments to better model distributed queries and wenow support direct access to Oracle and Informix statistics system tables.Thus, when you attach a table from one or the other, you automatically getup to date statistics. This helps <a href="http://virtuoso.openlinksw.com/wiki/main/">Virtuoso</a> optimize distributed queries.Also the documentation is updated as concerns these, with a new section ondistributed query optimization.</p> <p>On the applications side, we have been keeping up with the SIOC RDF ontologydevelopments. All ODS applications now make their data available as SIOCgraphs for download and SPARQL query access.</p> <p>What is most exciting however is our advance in mapping relational data intoRDF. We now have a mapping language that makes arbitrary legacy data in <a href="http://virtuoso.openlinksw.com/wiki/main/">Virtuoso</a> or elsewhere in the relational world RDF queriable. We will putout a white paper on this in a few days.</p> <p>Also we have some innovations in mind for optimizing the physical storage ofRDF triples. We keep experimenting, now with our sights set to the highend of triple storage, towards billion triple data sets. We areexperimenting with a new more space efficient index structure for betterworking set behavior. Next week will yield the first results.</p> <a href="http://www.technorati.com/tags/sql" rel="tag">SQL</a> <a href="http://www.technorati.com/tags/database" rel="tag">Database</a> <a href="http://www.technorati.com/tags/databases" rel="tag">Databases</a> <a href="http://www.technorati.com/tags/virtuoso" rel="tag">Virtuoso</a> <a href="http://www.technorati.com/tags/programming" rel="tag">Programming</a> <a href="http://www.technorati.com/tags/semantic%20web" rel="tag">Semantic Web</a> <a href="http://www.technorati.com/tags/sparql" rel="tag">SPARQL</a>