http://www.openlinksw.com:443/weblog/dav/dav-blog-1/
OpenLink Community Blog
A Collection of blogs by OpenLink Staff
kidehen@openlinksw.com
kidehen@openlinksw.com
2026-09-06T19:32:00Z
Virtuoso Universal Server 08.03.3334
http://www.openlinksw.com:443/weblog/public/images/vbloglogo.gif
New Semantic Publishing Benchmark Record
http://www.openlinksw.com:443/blog/vdb/blog/?date=2016-01-11#1867
2016-01-11T15:22:22Z
<p>There is a new <a href="http://ldbcouncil.org/developer/spb" id="link-id0x7fd87214c348">SPB (Semantic Publishing Benchmark)</a> 256 Mtriple record with <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x7fd870d26a48">Virtuoso</a>.</p> <p>As before, the result has been measured with the <a href="https://github.com/v7fasttrack/virtuoso-opensource/tree/feature/analytics" id="link-id0x7fd870d26be8">feature/analytics</a> branch of the <a href="https://github.com/v7fasttrack/virtuoso-opensource/" id="link-id0x7fd870d26d58">v7fasttrack</a> open source distribution, and it will soon be available as a preconfigured Amazon EC2 image. The updated <a href="http://www.openlinksw.com/weblogs/oerling/?id=1849" id="link-id0x7fd87217aec8">benchmarks AMI</a> with this version of the software will be out there within the next week, to be announced on this blog.</p> <h2>On the Cost of RDF Query Optimization</h2> <p>RDF query optimization is harder than the relational equivalent; first, because there are more joins, hence an NP complete explosion of plan search space, and second, because cardinality estimation is harder and usually less reliable. The work on characteristic sets, pioneered by <a href="https://www.linkedin.com/pub/thomas-neumann/2/35/189" id="link-id0x7fd870c3e918">Thomas Neumann</a> in <a href="https://domino.mpi-inf.mpg.de/intranet/ag5/ag5publ.nsf/0/AD3DBAFA6FB90DD2C1257593002FF3DF/$file/rdf3x.pdf" id="link-id0x7fd870c3eae8">RDF3X</a>, uses regularities in structure for treating properties usually occurring in the same subject as columns of a table. The same idea is applied for tuning physical representation in <a href="http://homepages.cwi.nl/~duc/papers/emergentschema_www15.pdf" id="link-id0x7fd8721c1ea8">the joint Virtuoso / MonetDB work published at WWW 2015</a>.</p> <p>The Virtuoso results discussed here, however, are all based on a single RDF quad table with Virtuoso's default index configuration.</p> <p>Introducing query plan caching raises the Virtuoso score from 80 qps to 144 qps at the 256 Mtriple scale. The SPB queries are not extremely complex; lookups with many more triple patterns exist in actual workloads, e.g., <a href="http://dbpedia.org/resource/OpenPHACTS" id="link-id0x7fd8721313e8">Open PHACTS</a>. In such applications, query optimization indeed dominates execution times. In SPB, data volumes touched by queries grow near linearly with data scale. At the 256 Mtriple scale, nearly half of CPU cycles are spent deciding a query plan. Below are the CPU cycles for execution and compilation per query type, sorted by descending sum of the times, scaled to milliseconds per execution. These are taken from a one minute sample of running at full throughput.</p> <p>Test system is the same used before in the <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x7fd872131738">TPC-H series</a>: dual Xeon E5-2630 Sandy Bridge, 2 x 6 cores x 2 threads, 2.3GHz, 192 GB RAM.</p> <p>We measure the compile and execute times, with and without using hash join. When considering hash join, the throughput is 80 qps. When not considering hash join, the throughput is 110 qps. With query plan caching, the throughput is 145 qps whether or not hash join is considered. Using hash join is not significant for the workload but considering its use in query optimization leads to significant extra work.</p> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <td> <h3>With hash join</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Compile</th> <th>Execute</th> <th>Total</th> <th>Query</th> </tr> <tr> <td style="text-align:right;"> <code><i> 3156 ms </i></code> </td> <td style="text-align:right;"> <code><i> 1181 ms </i></code> </td> <td style="text-align:right;"> <code><i> 4337 ms </i></code> </td> <td style="text-align:right;"> <code><i> Total </i></code></td> </tr> <tr> <td style="text-align:right;"> <code> 1327 ms </code> </td> <td style="text-align:right;"> <code> 28 ms </code> </td> <td style="text-align:right;"> <code> 1355 ms </code> </td> <td style="text-align:right;"> <code> query 01 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 444 ms </code> </td> <td style="text-align:right;"> <code> 460 ms </code> </td> <td style="text-align:right;"> <code> 904 ms </code> </td> <td style="text-align:right;"> <code> query 08 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 466 ms </code> </td> <td style="text-align:right;"> <code> 54 ms </code> </td> <td style="text-align:right;"> <code> 520 ms </code> </td> <td style="text-align:right;"> <code> query 06 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 123 ms </code> </td> <td style="text-align:right;"> <code> 268 ms </code> </td> <td style="text-align:right;"> <code> 391 ms </code> </td> <td style="text-align:right;"> <code> query 05 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 257 ms </code> </td> <td style="text-align:right;"> <code> 5 ms </code> </td> <td style="text-align:right;"> <code> 262 ms </code> </td> <td style="text-align:right;"> <code> query 11 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 191 ms </code> </td> <td style="text-align:right;"> <code> 59 ms </code> </td> <td style="text-align:right;"> <code> 250 ms </code> </td> <td style="text-align:right;"> <code> query 10 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 9 ms </code> </td> <td style="text-align:right;"> <code> 179 ms </code> </td> <td style="text-align:right;"> <code> 188 ms </code> </td> <td style="text-align:right;"> <code> query 04 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 114 ms </code> </td> <td style="text-align:right;"> <code> 26 ms </code> </td> <td style="text-align:right;"> <code> 140 ms </code> </td> <td style="text-align:right;"> <code> query 07 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 46 ms </code> </td> <td style="text-align:right;"> <code> 62 ms </code> </td> <td style="text-align:right;"> <code> 108 ms </code> </td> <td style="text-align:right;"> <code> query 09 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 71 ms </code> </td> <td style="text-align:right;"> <code> 25 ms </code> </td> <td style="text-align:right;"> <code> 96 ms </code> </td> <td style="text-align:right;"> <code> query 12 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 61 ms </code> </td> <td style="text-align:right;"> <code> 13 ms </code> </td> <td style="text-align:right;"> <code> 74 ms </code> </td> <td style="text-align:right;"> <code> query 03 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 47 ms </code> </td> <td style="text-align:right;"> <code> 2 ms </code> </td> <td style="text-align:right;"> <code> 49 ms </code> </td> <td style="text-align:right;"> <code> query 02 </code></td> </tr> </table> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> <h3>Without hash join</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Compile</th> <th>Execute</th> <th>Total</th> <th>Query</th> </tr> <tr> <td style="text-align:right;"> <code><i> 1816 ms </i></code> </td> <td style="text-align:right;"> <code><i> 1019 ms </i></code> </td> <td style="text-align:right;"> <code><i> 2835 ms </i></code> </td> <td style="text-align:right;"> <code><i> Total </i></code></td> </tr> <tr> <td style="text-align:right;"> <code> 197 ms </code> </td> <td style="text-align:right;"> <code> 466 ms </code> </td> <td style="text-align:right;"> <code> 663 ms </code> </td> <td style="text-align:right;"> <code> query 08 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 609 ms </code> </td> <td style="text-align:right;"> <code> 32 ms </code> </td> <td style="text-align:right;"> <code> 641 ms </code> </td> <td style="text-align:right;"> <code> query 01 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 188 ms </code> </td> <td style="text-align:right;"> <code> 293 ms </code> </td> <td style="text-align:right;"> <code> 481 ms </code> </td> <td style="text-align:right;"> <code> query 05 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 275 ms </code> </td> <td style="text-align:right;"> <code> 61 ms </code> </td> <td style="text-align:right;"> <code> 336 ms </code> </td> <td style="text-align:right;"> <code> query 09 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 163 ms </code> </td> <td style="text-align:right;"> <code> 10 ms </code> </td> <td style="text-align:right;"> <code> 173 ms </code> </td> <td style="text-align:right;"> <code> query 03 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 128 ms </code> </td> <td style="text-align:right;"> <code> 38 ms </code> </td> <td style="text-align:right;"> <code> 166 ms </code> </td> <td style="text-align:right;"> <code> query 10 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 102 ms </code> </td> <td style="text-align:right;"> <code> 5 ms </code> </td> <td style="text-align:right;"> <code> 107 ms </code> </td> <td style="text-align:right;"> <code> query 11 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 63 ms </code> </td> <td style="text-align:right;"> <code> 27 ms </code> </td> <td style="text-align:right;"> <code> 90 ms </code> </td> <td style="text-align:right;"> <code> query 12 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 24 ms </code> </td> <td style="text-align:right;"> <code> 57 ms </code> </td> <td style="text-align:right;"> <code> 81 ms </code> </td> <td style="text-align:right;"> <code> query 06 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 47 ms </code> </td> <td style="text-align:right;"> <code> 1 ms </code> </td> <td style="text-align:right;"> <code> 48 ms </code> </td> <td style="text-align:right;"> <code> query 02 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 15 ms </code> </td> <td style="text-align:right;"> <code> 24 ms </code> </td> <td style="text-align:right;"> <code> 39 ms </code> </td> <td style="text-align:right;"> <code> query 07 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 5 ms </code> </td> <td style="text-align:right;"> <code> 5 ms </code> </td> <td style="text-align:right;"> <code> 10 ms </code> </td> <td style="text-align:right;"> <code> query 04 </code></td> </tr> </table> </td> </tr> </table> <p>Considering hash join always slows down compilation, and sometimes improves and sometimes worsens execution. Some improvement in cost-model and plan-space traversal-order is possible, but altogether removing compilation via caching is better still. The results are as expected, since a lookup workload such as SPB has little use for hash join by nature.</p> <p>The rationale for considering hash join in the first place is that analytical workloads rely heavily on this. A good <a href="http://www.tpc.org/tpch/" id="link-id0x7fd8721ce268">TPC-H</a> score is simply unfeasible without this as <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x7fd8720f3f48">previously discussed on this blog</a>. If RDF is to be a serious contender beyond serving lookups, then hash join is indispensable. The decision for using this however depends on accurate cardinality estimates on either side of the join.</p> <p>Previous work (e.g., papers from <a href="http://www.forth.gr/" id="link-id0x7fd8720f8548">FORTH</a> around <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x7fd8721bfe38">MonetDB</a>) advocates doing away with a cost model altogether, since one is hard and unreliable with RDF anyway. The idea is not without its attraction but will lead to missing out of analytics or to relying on query hints for hash join.</p> <p>The present Virtuoso thinking is that going to rule based optimization is not the preferred solution, but rather using characteristic sets for reducing triples into wider tables, which also cuts down on plan search space and increases reliability of cost estimation.</p> <p>When looking at execution alone, we see that actual database operations are low in the profile, with memory management taking the top 19%. This is due to <code>CONSTRUCT</code> queries allocating small blocks for returning graphs, which is entirely avoidable.</p>
2016-01-11T15:25:17.969354
New Semantic Publishing Benchmark Record
http://www.openlinksw.com:443/weblog/oerling/?date=2016-01-11#1866
2016-01-11T15:22:09Z
<p>There is a new <a href="http://ldbcouncil.org/developer/spb" id="link-id0x7fd87214c348">SPB (Semantic Publishing Benchmark)</a> 256 Mtriple record with <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x7fd870d26a48">Virtuoso</a>.</p> <p>As before, the result has been measured with the <a href="https://github.com/v7fasttrack/virtuoso-opensource/tree/feature/analytics" id="link-id0x7fd870d26be8">feature/analytics</a> branch of the <a href="https://github.com/v7fasttrack/virtuoso-opensource/" id="link-id0x7fd870d26d58">v7fasttrack</a> open source distribution, and it will soon be available as a preconfigured Amazon EC2 image. The updated <a href="http://www.openlinksw.com/weblogs/oerling/?id=1849" id="link-id0x7fd87217aec8">benchmarks AMI</a> with this version of the software will be out there within the next week, to be announced on this blog.</p> <h2>On the Cost of RDF Query Optimization</h2> <p>RDF query optimization is harder than the relational equivalent; first, because there are more joins, hence an NP complete explosion of plan search space, and second, because cardinality estimation is harder and usually less reliable. The work on characteristic sets, pioneered by <a href="https://www.linkedin.com/pub/thomas-neumann/2/35/189" id="link-id0x7fd870c3e918">Thomas Neumann</a> in <a href="https://domino.mpi-inf.mpg.de/intranet/ag5/ag5publ.nsf/0/AD3DBAFA6FB90DD2C1257593002FF3DF/$file/rdf3x.pdf" id="link-id0x7fd870c3eae8">RDF3X</a>, uses regularities in structure for treating properties usually occurring in the same subject as columns of a table. The same idea is applied for tuning physical representation in <a href="http://homepages.cwi.nl/~duc/papers/emergentschema_www15.pdf" id="link-id0x7fd8721c1ea8">the joint Virtuoso / MonetDB work published at WWW 2015</a>.</p> <p>The Virtuoso results discussed here, however, are all based on a single RDF quad table with Virtuoso's default index configuration.</p> <p>Introducing query plan caching raises the Virtuoso score from 80 qps to 144 qps at the 256 Mtriple scale. The SPB queries are not extremely complex; lookups with many more triple patterns exist in actual workloads, e.g., <a href="http://dbpedia.org/resource/OpenPHACTS" id="link-id0x7fd8721313e8">Open PHACTS</a>. In such applications, query optimization indeed dominates execution times. In SPB, data volumes touched by queries grow near linearly with data scale. At the 256 Mtriple scale, nearly half of CPU cycles are spent deciding a query plan. Below are the CPU cycles for execution and compilation per query type, sorted by descending sum of the times, scaled to milliseconds per execution. These are taken from a one minute sample of running at full throughput.</p> <p>Test system is the same used before in the <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x7fd872131738">TPC-H series</a>: dual Xeon E5-2630 Sandy Bridge, 2 x 6 cores x 2 threads, 2.3GHz, 192 GB RAM.</p> <p>We measure the compile and execute times, with and without using hash join. When considering hash join, the throughput is 80 qps. When not considering hash join, the throughput is 110 qps. With query plan caching, the throughput is 145 qps whether or not hash join is considered. Using hash join is not significant for the workload but considering its use in query optimization leads to significant extra work.</p> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <td> <h3>With hash join</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Compile</th> <th>Execute</th> <th>Total</th> <th>Query</th> </tr> <tr> <td style="text-align:right;"> <code><i> 3156 ms </i></code> </td> <td style="text-align:right;"> <code><i> 1181 ms </i></code> </td> <td style="text-align:right;"> <code><i> 4337 ms </i></code> </td> <td style="text-align:right;"> <code><i> Total </i></code></td> </tr> <tr> <td style="text-align:right;"> <code> 1327 ms </code> </td> <td style="text-align:right;"> <code> 28 ms </code> </td> <td style="text-align:right;"> <code> 1355 ms </code> </td> <td style="text-align:right;"> <code> query 01 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 444 ms </code> </td> <td style="text-align:right;"> <code> 460 ms </code> </td> <td style="text-align:right;"> <code> 904 ms </code> </td> <td style="text-align:right;"> <code> query 08 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 466 ms </code> </td> <td style="text-align:right;"> <code> 54 ms </code> </td> <td style="text-align:right;"> <code> 520 ms </code> </td> <td style="text-align:right;"> <code> query 06 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 123 ms </code> </td> <td style="text-align:right;"> <code> 268 ms </code> </td> <td style="text-align:right;"> <code> 391 ms </code> </td> <td style="text-align:right;"> <code> query 05 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 257 ms </code> </td> <td style="text-align:right;"> <code> 5 ms </code> </td> <td style="text-align:right;"> <code> 262 ms </code> </td> <td style="text-align:right;"> <code> query 11 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 191 ms </code> </td> <td style="text-align:right;"> <code> 59 ms </code> </td> <td style="text-align:right;"> <code> 250 ms </code> </td> <td style="text-align:right;"> <code> query 10 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 9 ms </code> </td> <td style="text-align:right;"> <code> 179 ms </code> </td> <td style="text-align:right;"> <code> 188 ms </code> </td> <td style="text-align:right;"> <code> query 04 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 114 ms </code> </td> <td style="text-align:right;"> <code> 26 ms </code> </td> <td style="text-align:right;"> <code> 140 ms </code> </td> <td style="text-align:right;"> <code> query 07 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 46 ms </code> </td> <td style="text-align:right;"> <code> 62 ms </code> </td> <td style="text-align:right;"> <code> 108 ms </code> </td> <td style="text-align:right;"> <code> query 09 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 71 ms </code> </td> <td style="text-align:right;"> <code> 25 ms </code> </td> <td style="text-align:right;"> <code> 96 ms </code> </td> <td style="text-align:right;"> <code> query 12 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 61 ms </code> </td> <td style="text-align:right;"> <code> 13 ms </code> </td> <td style="text-align:right;"> <code> 74 ms </code> </td> <td style="text-align:right;"> <code> query 03 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 47 ms </code> </td> <td style="text-align:right;"> <code> 2 ms </code> </td> <td style="text-align:right;"> <code> 49 ms </code> </td> <td style="text-align:right;"> <code> query 02 </code></td> </tr> </table> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> <h3>Without hash join</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Compile</th> <th>Execute</th> <th>Total</th> <th>Query</th> </tr> <tr> <td style="text-align:right;"> <code><i> 1816 ms </i></code> </td> <td style="text-align:right;"> <code><i> 1019 ms </i></code> </td> <td style="text-align:right;"> <code><i> 2835 ms </i></code> </td> <td style="text-align:right;"> <code><i> Total </i></code></td> </tr> <tr> <td style="text-align:right;"> <code> 197 ms </code> </td> <td style="text-align:right;"> <code> 466 ms </code> </td> <td style="text-align:right;"> <code> 663 ms </code> </td> <td style="text-align:right;"> <code> query 08 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 609 ms </code> </td> <td style="text-align:right;"> <code> 32 ms </code> </td> <td style="text-align:right;"> <code> 641 ms </code> </td> <td style="text-align:right;"> <code> query 01 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 188 ms </code> </td> <td style="text-align:right;"> <code> 293 ms </code> </td> <td style="text-align:right;"> <code> 481 ms </code> </td> <td style="text-align:right;"> <code> query 05 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 275 ms </code> </td> <td style="text-align:right;"> <code> 61 ms </code> </td> <td style="text-align:right;"> <code> 336 ms </code> </td> <td style="text-align:right;"> <code> query 09 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 163 ms </code> </td> <td style="text-align:right;"> <code> 10 ms </code> </td> <td style="text-align:right;"> <code> 173 ms </code> </td> <td style="text-align:right;"> <code> query 03 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 128 ms </code> </td> <td style="text-align:right;"> <code> 38 ms </code> </td> <td style="text-align:right;"> <code> 166 ms </code> </td> <td style="text-align:right;"> <code> query 10 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 102 ms </code> </td> <td style="text-align:right;"> <code> 5 ms </code> </td> <td style="text-align:right;"> <code> 107 ms </code> </td> <td style="text-align:right;"> <code> query 11 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 63 ms </code> </td> <td style="text-align:right;"> <code> 27 ms </code> </td> <td style="text-align:right;"> <code> 90 ms </code> </td> <td style="text-align:right;"> <code> query 12 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 24 ms </code> </td> <td style="text-align:right;"> <code> 57 ms </code> </td> <td style="text-align:right;"> <code> 81 ms </code> </td> <td style="text-align:right;"> <code> query 06 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 47 ms </code> </td> <td style="text-align:right;"> <code> 1 ms </code> </td> <td style="text-align:right;"> <code> 48 ms </code> </td> <td style="text-align:right;"> <code> query 02 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 15 ms </code> </td> <td style="text-align:right;"> <code> 24 ms </code> </td> <td style="text-align:right;"> <code> 39 ms </code> </td> <td style="text-align:right;"> <code> query 07 </code></td> </tr> <tr> <td style="text-align:right;"> <code> 5 ms </code> </td> <td style="text-align:right;"> <code> 5 ms </code> </td> <td style="text-align:right;"> <code> 10 ms </code> </td> <td style="text-align:right;"> <code> query 04 </code></td> </tr> </table> </td> </tr> </table> <p>Considering hash join always slows down compilation, and sometimes improves and sometimes worsens execution. Some improvement in cost-model and plan-space traversal-order is possible, but altogether removing compilation via caching is better still. The results are as expected, since a lookup workload such as SPB has little use for hash join by nature.</p> <p>The rationale for considering hash join in the first place is that analytical workloads rely heavily on this. A good <a href="http://www.tpc.org/tpch/" id="link-id0x7fd8721ce268">TPC-H</a> score is simply unfeasible without this as <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x7fd8720f3f48">previously discussed on this blog</a>. If RDF is to be a serious contender beyond serving lookups, then hash join is indispensable. The decision for using this however depends on accurate cardinality estimates on either side of the join.</p> <p>Previous work (e.g., papers from <a href="http://www.forth.gr/" id="link-id0x7fd8720f8548">FORTH</a> around <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x7fd8721bfe38">MonetDB</a>) advocates doing away with a cost model altogether, since one is hard and unreliable with RDF anyway. The idea is not without its attraction but will lead to missing out of analytics or to relying on query hints for hash join.</p> <p>The present Virtuoso thinking is that going to rule based optimization is not the preferred solution, but rather using characteristic sets for reducing triples into wider tables, which also cuts down on plan search space and increases reliability of cost estimation.</p> <p>When looking at execution alone, we see that actual database operations are low in the profile, with memory management taking the top 19%. This is due to <code>CONSTRUCT</code> queries allocating small blocks for returning graphs, which is entirely avoidable.</p>
2016-01-11T15:25:01.739223
DBpedia Usage Report, August 2015
http://www.openlinksw.com:443/blog/vdb/blog/?date=2015-08-11#1865
2015-08-11T16:59:23Z
<p>We recently published the latest <a href="http://bit.ly/1IL35Xu" id="link-id0x2aabbfb1e308">DBpedia Usage Report</a>, covering v3.3 (released July, 2009) to v3.10 (sometimes called "DBpedia 2014"; released September, 2014).</p> <p>The new report has usage data through July 31, 2015, and brought a few surprises to our eyes. What do you think?</p>
2015-08-11T12:59:23.150522-04:00
DBpedia Usage Report, August 2015
http://www.openlinksw.com:443/weblog/oerling/?date=2015-08-11#1864
2015-08-11T16:58:21Z
<p>We recently published the latest <a href="http://bit.ly/1IL35Xu" id="link-id0x2aabbfb1e308">DBpedia Usage Report</a>, covering v3.3 (released July, 2009) to v3.10 (sometimes called "DBpedia 2014"; released September, 2014).</p> <p>The new report has usage data through July 31, 2015, and brought a few surprises to our eyes. What do you think?</p>
2015-08-11T12:58:21.429167-04:00
Big Data, Part 2: Virtuoso Meets Impala
http://www.openlinksw.com:443/blog/vdb/blog/?date=2015-07-15#1863
2015-07-15T20:17:55Z
<p>In this article we will look at <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aac0250bde8">Virtuoso</a> vs. <a href="https://dbpedia.org/resource/Cloudera_Impala" id="link-id0x2aac01e3c6d8">Impala</a> with 100G TPC-H on two R3.8 EC2 instances. We get a single user win for Virtuoso by a factor of 136, and a five user win by a factor of 55. The details and analysis follow.</p> <p>The load setup is the same as ever, with copying from CSV files attached as external tables into <a href="http://www.cloudera.com/content/cloudera/en/documentation/cloudera-impala/v2-0-x/topics/impala_parquet.html" id="link-id0x2aac014886d8">Parquet tables</a>. We get <code>lineitem</code> split over 88 Parquet files, which should provide enough parallelism for the platform. The Impala documentation states that there can be up to one thread per file, and here we wish to see maximum parallelism for a single query stream. We use the schema from the <a href="https://github.com/cloudera/impala" id="link-id0x2aac01cb0518">Impala github</a> checkout, with <code>string</code> for <code>string</code> and <code>date</code> columns, and <code>decimal</code> for <code>numbers</code>. We suppose the authors know what works best.</p> <p>The execution behavior is surprising. Sometimes we get full platform utilization, but quite often only 200% CPU per box. The query plan for Q1, for example, says 2 cores per box. This makes no sense, as the same plan fully well knows the table cardinality. The settings for scanner threads and cores to use (in <code>impala-shell</code>) can be changed, but the behavior does not seem to change.</p> <p>Following are the run times for one query stream.</p> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Query</th> <th>Virtuoso</th> <th>Impala</th> <th>Notes</th> </tr> <tr> <th style="text-align:center;"> — </th> <td style="text-align:right;"> <code> 332 s </code> </td> <td style="text-align:right;"> <code> 841 s </code> </td> <td style="text-align:left;"> Data Load </td> </tr> <tr> <th style="text-align:center;"> Q1 </th> <td style="text-align:right;"> <code> 1.098 s </code> </td> <td style="text-align:right;"> <code> 164.61 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q2 </th> <td style="text-align:right;"> <code> 0.187 s </code> </td> <td style="text-align:right;"> <code> 24.19 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q3 </th> <td style="text-align:right;"> <code> 0.761 s </code> </td> <td style="text-align:right;"> <code> 105.70 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q4 </th> <td style="text-align:right;"> <code> 0.205 s </code> </td> <td style="text-align:right;"> <code> 179.67 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q5 </th> <td style="text-align:right;"> <code> 0.808 s </code> </td> <td style="text-align:right;"> <code> 84.51 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q6 </th> <td style="text-align:right;"> <code> 2.403 s </code> </td> <td style="text-align:right;"> <code> 4.43 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q7 </th> <td style="text-align:right;"> <code> 0.59 s </code> </td> <td style="text-align:right;"> <code> 270.88 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q8 </th> <td style="text-align:right;"> <code> 0.775 s </code> </td> <td style="text-align:right;"> <code> 51.89 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q9 </th> <td style="text-align:right;"> <code> 1.836 s </code> </td> <td style="text-align:right;"> <code> 177.72 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q10 </th> <td style="text-align:right;"> <code> 3.165 s </code> </td> <td style="text-align:right;"> <code> 39.85 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q11 </th> <td style="text-align:right;"> <code> 1.37 s </code> </td> <td style="text-align:right;"> <code> 22.56 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q12 </th> <td style="text-align:right;"> <code> 0.356 s </code> </td> <td style="text-align:right;"> <code> 17.03 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q13 </th> <td style="text-align:right;"> <code> 2.233 s </code> </td> <td style="text-align:right;"> <code> 103.67 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q14 </th> <td style="text-align:right;"> <code> 0.488 s </code> </td> <td style="text-align:right;"> <code> 10.86 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q15 </th> <td style="text-align:right;"> <code> 0.72 s </code> </td> <td style="text-align:right;"> <code> 11.49 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q16 </th> <td style="text-align:right;"> <code> 0.814 s </code> </td> <td style="text-align:right;"> <code> 23.93 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q17 </th> <td style="text-align:right;"> <code> 0.681 s </code> </td> <td style="text-align:right;"> <code> 276.06 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q18 </th> <td style="text-align:right;"> <code> 1.324 s </code> </td> <td style="text-align:right;"> <code> 267.13 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q19 </th> <td style="text-align:right;"> <code> 0.417 s </code> </td> <td style="text-align:right;"> <code> 368.80 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q20 </th> <td style="text-align:right;"> <code> 0.792 s </code> </td> <td style="text-align:right;"> <code> 60.45 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q21 </th> <td style="text-align:right;"> <code> 0.720 s </code> </td> <td style="text-align:right;"> <code> 418.09 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q22 </th> <td style="text-align:right;"> <code> 0.155 s </code> </td> <td style="text-align:right;"> <code> 40.59 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Total </th> <td style="text-align:right;"> <code> 20 s </code> </td> <td style="text-align:right;"> <code> 2724 s </code> </td> <td style="text-align:left;"> </td> </tr> </table> <p>Because the platform utilization was often low, we made a second experiment running the same queries in five parallel sessions. We show the average execution time for each query. We then compare this with the Virtuoso throughput run average times. We permute the single query stream used in the first tests in 5 different orders, as per the TPC-H spec. The results are not entirely comparable, because Virtuoso is doing the refreshes in parallel. According to Impala documentation, there is no random delete operation, so the refreshes cannot be implemented.</p> <p>Just to establish a baseline, we do <code>SELECT COUNT (*) FROM lineitem</code>. This takes 20s when run by itself. When run in five parallel sessions, the fastest terminates in 64s and the slowest in 69s. Looking at <code>top</code>, the platform utilization is indeed about 5x more in CPU%, but the concurrency does not add much to throughput. This is odd, considering that there is no synchronization requirement worth mentioning between the operations.</p> <p>Following are the average times for each query in the 5 stream experiment.</p> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Query</th> <th>Virtuoso</th> <th>Impala</th> <th>Notes</th> </tr> <tr> <th style="text-align:center;"> Q1 </th> <td style="text-align:right;"> <code> 1.95 s </code> </td> <td style="text-align:right;"> <code> 191.81 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q2 </th> <td style="text-align:right;"> <code> 0.70 s </code> </td> <td style="text-align:right;"> <code> 40.40 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q3 </th> <td style="text-align:right;"> <code> 2.01 s </code> </td> <td style="text-align:right;"> <code> 95.67 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q4 </th> <td style="text-align:right;"> <code> 0.71 s </code> </td> <td style="text-align:right;"> <code> 345.11 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q5 </th> <td style="text-align:right;"> <code> 2.93 s </code> </td> <td style="text-align:right;"> <code> 112.29 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q6 </th> <td style="text-align:right;"> <code> 4.76 s </code> </td> <td style="text-align:right;"> <code> 14.41 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q7 </th> <td style="text-align:right;"> <code> 2.08 s </code> </td> <td style="text-align:right;"> <code> 329.25 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q8 </th> <td style="text-align:right;"> <code> 3.00 s </code> </td> <td style="text-align:right;"> <code> 98.91 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q9 </th> <td style="text-align:right;"> <code> 5.58 s </code> </td> <td style="text-align:right;"> <code> 250.88 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q10 </th> <td style="text-align:right;"> <code> 8.23 s </code> </td> <td style="text-align:right;"> <code> 55.23 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q11 </th> <td style="text-align:right;"> <code> 4.26 s </code> </td> <td style="text-align:right;"> <code> 27.84 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q12 </th> <td style="text-align:right;"> <code> 1.74 s </code> </td> <td style="text-align:right;"> <code> 37.66 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q13 </th> <td style="text-align:right;"> <code> 6.07 s </code> </td> <td style="text-align:right;"> <code> 147.69 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q14 </th> <td style="text-align:right;"> <code> 1.73 s </code> </td> <td style="text-align:right;"> <code> 23.91 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q15 </th> <td style="text-align:right;"> <code> 2.27 s </code> </td> <td style="text-align:right;"> <code> 23.79 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q16 </th> <td style="text-align:right;"> <code> 2.41 s </code> </td> <td style="text-align:right;"> <code> 34.76 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q17 </th> <td style="text-align:right;"> <code> 3.92 s </code> </td> <td style="text-align:right;"> <code> 362.43 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q18 </th> <td style="text-align:right;"> <code> 3.02 s </code> </td> <td style="text-align:right;"> <code> 348.08 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q19 </th> <td style="text-align:right;"> <code> 2.27 s </code> </td> <td style="text-align:right;"> <code> 443.94 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q20 </th> <td style="text-align:right;"> <code> 3.05 s </code> </td> <td style="text-align:right;"> <code> 92.50 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q21 </th> <td style="text-align:right;"> <code> 2.00 s </code> </td> <td style="text-align:right;"> <code> 623.69 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q22 </th> <td style="text-align:right;"> <code> 0.37 s </code> </td> <td style="text-align:right;"> <code> 61.36 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Total for <br /> Slowest Stream </th> <td style="text-align:right;"> <code> 67 s </code> </td> <td style="text-align:right;"> <code> 3740 s </code> </td> <td style="text-align:left;"> </td> </tr> </table> <p>There are 4 queries in Impala that terminated with an error (<code>memory limit exceeded</code>). These were two Q21s, one Q19, one Q4. One stream executed without errors, so this stream is reported as the slowest stream. Q21 will, in the absence of indexed access, do a hash build side of half of <code>lineitem</code>, which explains running out of memory. Virtuoso does Q21 mostly by index.</p> <p>Looking at the 5 streams, we see CPU between 1000% and 2000% on either box. This looks about 5x more than the 250% per box that we were seeing with, for instance, Q1. The process sizes for <code>impalad</code> are over 160G, certainly enough to have the working set in memory. <code>iostat</code> also does not show any <code>I</code>, so we seem to be running from memory, as intended.</p> <p>We observe that Impala does not store tables in any specific order. Therefore a merge join of <code>orders</code> and <code>lineitem</code> is not possible. Thus we always get a hash join with a potentially large build side, e.g., half of <code>orders</code> and half of <code>lineitem</code> in Q21, and all <code>orders</code> in Q9. This explains in part why these take so long. <a href="http://www.tpc.org/tpcds/" id="link-id0x2aac01486778">TPC-DS</a> does not pose this particular problem though, as there are no tables in the DS schema where the primary key of one would be the prefix of that of another.</p> <p>However, the <code>lineitem/orders</code> join does not explain the scores on Q1, Q20, or Q19. A simple hash join of <code>lineitem</code> and <code>part</code> was about 90s, with a replicated <code>part</code> hash table. In the profile, the hash probe was 74s, which seems excessive. One would have to single-step through the hash probe to find out what actually happens. Maybe there are prohibitive numbers of collisions, which would throw off the results across the board. We would have to ask the Impala community about this.</p> <p>Anyway, <i><b>Impala experts out there are invited to set the record straight.</b></i> We have attached the results and the output of the Impala <code>profile</code> statement for each query for the single stream run. <code><a href="http://www.openlinksw.com/weblog/oerling/media/impala_stream0.zip" title="impala_stream0.zip" alt="impala_stream0.zip" id="link-id0x2aac0170e038">impala_stream0.zip</a></code> contains the evidence for the single-stream run; <code><a href="http://www.openlinksw.com/weblog/oerling/media/impala-stream1-5.zip" title="impala-stream1-5.zip" alt="impala-stream1-5.zip" id="link-id0x2aac0170e258">impala-stream1-5.zip</a></code> holds the 5-stream run.</p> <p>To be more Big Data-like, we should probably run with significantly larger data than memory; for example, 3T in 0.5T RAM. At EC2, we could do this with 2 I3.8 instances (6.4T SSD each). With Virtuoso, we'd be done in 8 hours or so, counting 2x for the I/O and 30x for the greater scale (the 100G experiment goes in 8 minutes or so, all included). With Impala, we could be running for weeks, so at the very least we'd like to do this with an Impala expert, to make sure things are done right and will not have to be retried. Some of the hash joins would have to be done in multiple passes and with partitioning.</p> <p>In subsequent articles, we will look at other players in this space, and possibly some other benchmarks, like the TPC-DS subset that <a href="http://www.actian.com/" id="link-id0x2aac0045dc78">Actian</a> uses to beat Impala.</p>
2015-07-15T16:17:55.789042-04:00
Big Data, Part 2: Virtuoso Meets Impala
http://www.openlinksw.com:443/weblog/oerling/?date=2015-07-15#1862
2015-07-15T20:12:27Z
<p>In this article we will look at <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aac0250bde8">Virtuoso</a> vs. <a href="https://dbpedia.org/resource/Cloudera_Impala" id="link-id0x2aac01e3c6d8">Impala</a> with 100G TPC-H on two R3.8 EC2 instances. We get a single user win for Virtuoso by a factor of 136, and a five user win by a factor of 55. The details and analysis follow.</p> <p>The load setup is the same as ever, with copying from CSV files attached as external tables into <a href="http://www.cloudera.com/content/cloudera/en/documentation/cloudera-impala/v2-0-x/topics/impala_parquet.html" id="link-id0x2aac014886d8">Parquet tables</a>. We get <code>lineitem</code> split over 88 Parquet files, which should provide enough parallelism for the platform. The Impala documentation states that there can be up to one thread per file, and here we wish to see maximum parallelism for a single query stream. We use the schema from the <a href="https://github.com/cloudera/impala" id="link-id0x2aac01cb0518">Impala github</a> checkout, with <code>string</code> for <code>string</code> and <code>date</code> columns, and <code>decimal</code> for <code>numbers</code>. We suppose the authors know what works best.</p> <p>The execution behavior is surprising. Sometimes we get full platform utilization, but quite often only 200% CPU per box. The query plan for Q1, for example, says 2 cores per box. This makes no sense, as the same plan fully well knows the table cardinality. The settings for scanner threads and cores to use (in <code>impala-shell</code>) can be changed, but the behavior does not seem to change.</p> <p>Following are the run times for one query stream.</p> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Query</th> <th>Virtuoso</th> <th>Impala</th> <th>Notes</th> </tr> <tr> <th style="text-align:center;"> — </th> <td style="text-align:right;"> <code> 332 s </code> </td> <td style="text-align:right;"> <code> 841 s </code> </td> <td style="text-align:left;"> Data Load </td> </tr> <tr> <th style="text-align:center;"> Q1 </th> <td style="text-align:right;"> <code> 1.098 s </code> </td> <td style="text-align:right;"> <code> 164.61 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q2 </th> <td style="text-align:right;"> <code> 0.187 s </code> </td> <td style="text-align:right;"> <code> 24.19 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q3 </th> <td style="text-align:right;"> <code> 0.761 s </code> </td> <td style="text-align:right;"> <code> 105.70 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q4 </th> <td style="text-align:right;"> <code> 0.205 s </code> </td> <td style="text-align:right;"> <code> 179.67 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q5 </th> <td style="text-align:right;"> <code> 0.808 s </code> </td> <td style="text-align:right;"> <code> 84.51 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q6 </th> <td style="text-align:right;"> <code> 2.403 s </code> </td> <td style="text-align:right;"> <code> 4.43 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q7 </th> <td style="text-align:right;"> <code> 0.59 s </code> </td> <td style="text-align:right;"> <code> 270.88 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q8 </th> <td style="text-align:right;"> <code> 0.775 s </code> </td> <td style="text-align:right;"> <code> 51.89 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q9 </th> <td style="text-align:right;"> <code> 1.836 s </code> </td> <td style="text-align:right;"> <code> 177.72 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q10 </th> <td style="text-align:right;"> <code> 3.165 s </code> </td> <td style="text-align:right;"> <code> 39.85 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q11 </th> <td style="text-align:right;"> <code> 1.37 s </code> </td> <td style="text-align:right;"> <code> 22.56 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q12 </th> <td style="text-align:right;"> <code> 0.356 s </code> </td> <td style="text-align:right;"> <code> 17.03 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q13 </th> <td style="text-align:right;"> <code> 2.233 s </code> </td> <td style="text-align:right;"> <code> 103.67 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q14 </th> <td style="text-align:right;"> <code> 0.488 s </code> </td> <td style="text-align:right;"> <code> 10.86 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q15 </th> <td style="text-align:right;"> <code> 0.72 s </code> </td> <td style="text-align:right;"> <code> 11.49 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q16 </th> <td style="text-align:right;"> <code> 0.814 s </code> </td> <td style="text-align:right;"> <code> 23.93 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q17 </th> <td style="text-align:right;"> <code> 0.681 s </code> </td> <td style="text-align:right;"> <code> 276.06 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q18 </th> <td style="text-align:right;"> <code> 1.324 s </code> </td> <td style="text-align:right;"> <code> 267.13 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q19 </th> <td style="text-align:right;"> <code> 0.417 s </code> </td> <td style="text-align:right;"> <code> 368.80 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q20 </th> <td style="text-align:right;"> <code> 0.792 s </code> </td> <td style="text-align:right;"> <code> 60.45 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q21 </th> <td style="text-align:right;"> <code> 0.720 s </code> </td> <td style="text-align:right;"> <code> 418.09 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q22 </th> <td style="text-align:right;"> <code> 0.155 s </code> </td> <td style="text-align:right;"> <code> 40.59 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Total </th> <td style="text-align:right;"> <code> 20 s </code> </td> <td style="text-align:right;"> <code> 2724 s </code> </td> <td style="text-align:left;"> </td> </tr> </table> <p>Because the platform utilization was often low, we made a second experiment running the same queries in five parallel sessions. We show the average execution time for each query. We then compare this with the Virtuoso throughput run average times. We permute the single query stream used in the first tests in 5 different orders, as per the TPC-H spec. The results are not entirely comparable, because Virtuoso is doing the refreshes in parallel. According to Impala documentation, there is no random delete operation, so the refreshes cannot be implemented.</p> <p>Just to establish a baseline, we do <code>SELECT COUNT (*) FROM lineitem</code>. This takes 20s when run by itself. When run in five parallel sessions, the fastest terminates in 64s and the slowest in 69s. Looking at <code>top</code>, the platform utilization is indeed about 5x more in CPU%, but the concurrency does not add much to throughput. This is odd, considering that there is no synchronization requirement worth mentioning between the operations.</p> <p>Following are the average times for each query in the 5 stream experiment.</p> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Query</th> <th>Virtuoso</th> <th>Impala</th> <th>Notes</th> </tr> <tr> <th style="text-align:center;"> Q1 </th> <td style="text-align:right;"> <code> 1.95 s </code> </td> <td style="text-align:right;"> <code> 191.81 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q2 </th> <td style="text-align:right;"> <code> 0.70 s </code> </td> <td style="text-align:right;"> <code> 40.40 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q3 </th> <td style="text-align:right;"> <code> 2.01 s </code> </td> <td style="text-align:right;"> <code> 95.67 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q4 </th> <td style="text-align:right;"> <code> 0.71 s </code> </td> <td style="text-align:right;"> <code> 345.11 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q5 </th> <td style="text-align:right;"> <code> 2.93 s </code> </td> <td style="text-align:right;"> <code> 112.29 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q6 </th> <td style="text-align:right;"> <code> 4.76 s </code> </td> <td style="text-align:right;"> <code> 14.41 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q7 </th> <td style="text-align:right;"> <code> 2.08 s </code> </td> <td style="text-align:right;"> <code> 329.25 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q8 </th> <td style="text-align:right;"> <code> 3.00 s </code> </td> <td style="text-align:right;"> <code> 98.91 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q9 </th> <td style="text-align:right;"> <code> 5.58 s </code> </td> <td style="text-align:right;"> <code> 250.88 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q10 </th> <td style="text-align:right;"> <code> 8.23 s </code> </td> <td style="text-align:right;"> <code> 55.23 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q11 </th> <td style="text-align:right;"> <code> 4.26 s </code> </td> <td style="text-align:right;"> <code> 27.84 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q12 </th> <td style="text-align:right;"> <code> 1.74 s </code> </td> <td style="text-align:right;"> <code> 37.66 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q13 </th> <td style="text-align:right;"> <code> 6.07 s </code> </td> <td style="text-align:right;"> <code> 147.69 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q14 </th> <td style="text-align:right;"> <code> 1.73 s </code> </td> <td style="text-align:right;"> <code> 23.91 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q15 </th> <td style="text-align:right;"> <code> 2.27 s </code> </td> <td style="text-align:right;"> <code> 23.79 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q16 </th> <td style="text-align:right;"> <code> 2.41 s </code> </td> <td style="text-align:right;"> <code> 34.76 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q17 </th> <td style="text-align:right;"> <code> 3.92 s </code> </td> <td style="text-align:right;"> <code> 362.43 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q18 </th> <td style="text-align:right;"> <code> 3.02 s </code> </td> <td style="text-align:right;"> <code> 348.08 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q19 </th> <td style="text-align:right;"> <code> 2.27 s </code> </td> <td style="text-align:right;"> <code> 443.94 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q20 </th> <td style="text-align:right;"> <code> 3.05 s </code> </td> <td style="text-align:right;"> <code> 92.50 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q21 </th> <td style="text-align:right;"> <code> 2.00 s </code> </td> <td style="text-align:right;"> <code> 623.69 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q22 </th> <td style="text-align:right;"> <code> 0.37 s </code> </td> <td style="text-align:right;"> <code> 61.36 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Total for <br /> Slowest Stream </th> <td style="text-align:right;"> <code> 67 s </code> </td> <td style="text-align:right;"> <code> 3740 s </code> </td> <td style="text-align:left;"> </td> </tr> </table> <p>There are 4 queries in Impala that terminated with an error (<code>memory limit exceeded</code>). These were two Q21s, one Q19, one Q4. One stream executed without errors, so this stream is reported as the slowest stream. Q21 will, in the absence of indexed access, do a hash build side of half of <code>lineitem</code>, which explains running out of memory. Virtuoso does Q21 mostly by index.</p> <p>Looking at the 5 streams, we see CPU between 1000% and 2000% on either box. This looks about 5x more than the 250% per box that we were seeing with, for instance, Q1. The process sizes for <code>impalad</code> are over 160G, certainly enough to have the working set in memory. <code>iostat</code> also does not show any <code>I</code>, so we seem to be running from memory, as intended.</p> <p>We observe that Impala does not store tables in any specific order. Therefore a merge join of <code>orders</code> and <code>lineitem</code> is not possible. Thus we always get a hash join with a potentially large build side, e.g., half of <code>orders</code> and half of <code>lineitem</code> in Q21, and all <code>orders</code> in Q9. This explains in part why these take so long. <a href="http://www.tpc.org/tpcds/" id="link-id0x2aac01486778">TPC-DS</a> does not pose this particular problem though, as there are no tables in the DS schema where the primary key of one would be the prefix of that of another.</p> <p>However, the <code>lineitem/orders</code> join does not explain the scores on Q1, Q20, or Q19. A simple hash join of <code>lineitem</code> and <code>part</code> was about 90s, with a replicated <code>part</code> hash table. In the profile, the hash probe was 74s, which seems excessive. One would have to single-step through the hash probe to find out what actually happens. Maybe there are prohibitive numbers of collisions, which would throw off the results across the board. We would have to ask the Impala community about this.</p> <p>Anyway, <i><b>Impala experts out there are invited to set the record straight.</b></i> We have attached the results and the output of the Impala <code>profile</code> statement for each query for the single stream run. <code><a href="http://www.openlinksw.com/weblog/oerling/media/impala_stream0.zip" title="impala_stream0.zip" alt="impala_stream0.zip" id="link-id0x2aac0170e038">impala_stream0.zip</a></code> contains the evidence for the single-stream run; <code><a href="http://www.openlinksw.com/weblog/oerling/media/impala-stream1-5.zip" title="impala-stream1-5.zip" alt="impala-stream1-5.zip" id="link-id0x2aac0170e258">impala-stream1-5.zip</a></code> holds the 5-stream run.</p> <p>To be more Big Data-like, we should probably run with significantly larger data than memory; for example, 3T in 0.5T RAM. At EC2, we could do this with 2 I3.8 instances (6.4T SSD each). With Virtuoso, we'd be done in 8 hours or so, counting 2x for the I/O and 30x for the greater scale (the 100G experiment goes in 8 minutes or so, all included). With Impala, we could be running for weeks, so at the very least we'd like to do this with an Impala expert, to make sure things are done right and will not have to be retried. Some of the hash joins would have to be done in multiple passes and with partitioning.</p> <p>In subsequent articles, we will look at other players in this space, and possibly some other benchmarks, like the TPC-DS subset that <a href="http://www.actian.com/" id="link-id0x2aac0045dc78">Actian</a> uses to beat Impala.</p>
2015-07-15T16:12:27.639584-04:00
Vectored Execution in Column/Row Stores
http://www.openlinksw.com:443/blog/vdb/blog/?date=2015-07-13#1861
2015-07-13T17:49:10Z
<p>This article discusses the relationship between vectored execution and column- and row-wise data representations. <a href="http://dbpedia.org/resource/Column-oriented_DBMS" id="link-id0x2aab8a00d208">Column stores</a> are traditionally considered to be good for big scans but poor at indexed access. This is not necessarily so, though. We take <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aab8a00d398">TPC-H Q9</a> as a starting point, working with different row- and column-wise data representations and index choices. The goal of the article is to provide a primer on the performance implications of different physical designs.</p> <p>All the experiments are against the TPC-H 100G dataset hosted in Virtuoso on the test system used before in the <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aab8bcec9d8">TPC-H series</a>: dual Xeon E5-2630, 2x6 cores x 2 threads, 2.3GHz, 192 GB RAM. The Virtuoso version corresponds to the <a href="https://github.com/v7fasttrack/virtuoso-opensource/tree/feature/analytics" id="link-id0x2aab8bcecbd8">feature/analytics branch</a> in the <a href="https://github.com/v7fasttrack/virtuoso-opensource/" id="link-id0x2aab8bceccb8">v7fasttrack github project</a>. All run times are from memory, and queries generally run at full platform, 24 concurrent threads.</p> <p>We note that RDF stores and graph databases usually do not have secondary indices with multiple key parts. However, these do predominantly index-based access as opposed to big scans and hash joins. To explore the impact of this, we have decomposed the tables into projections with a single dependent column, which approximates a triple store or a vertically-decomposed graph database like <a href="http://dbpedia.org/resource/DEX_(Graph_database)" id="link-id0x2aab8bced038">Sparksee</a>.</p> <p>So, in these experiments, we store the relevant data four times over, as follows:</p> <ul> <li> <p>100G TPC-H dataset in the column-wise schema as discussed in the TPC-H series, now complemented with indices on <code>l_partkey</code> and on <code>l_partkey, l_suppkey</code> </p> </li> <li> <p>The same in row-wise data representation </p> </li> <li> <p>Column-wise tables with a single dependent column for <code>l_partkey, l_suppkey, l_extendedprice, l_quantity, l_discount, ps_supplycost, s_nationkey, p_name</code>. These all have the original tables primary key, e.g., <code>l_orderkey, l_linenumber</code> for the <code>l_ prefixed tables</code> </p> </li> <li> <p>The same with row-wise tables</p> </li> </ul> <p>The column-wise structures are in the <code>DB</code> qualifier, and the row-wise are in the <code>R</code> qualifier. There is a summary of space consumption at the end of the article. This is relevant for scalability, since even if row-wise structures can be faster for scattered random access, they will fit less data in RAM, typically 2 to 3x less. Thus, if "faster" rows cause the working set not to fit, "slower" columns will still win.</p> <p>As a starting point, we know that the best Q9 is the one in the Virtuoso TPC-H implementation which is described in <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aab8a78d208">Part 10 of the TPC-H blog series</a>. This is a scan of <code>lineitem</code> with a selective hash join followed ordered index access of <code>orders</code>, then hash joins against the smaller tables. There are special tricks to keep the hash tables small by propagating restrictions from the probe side to the build side.</p> <p>The query texts are <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/" id="link-id0x2aab8bc59538">available here</a>, along with the table declarations and scripts for populating the single-column projections. <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/rs.sql" id="link-id0x2aab8bc59778">rs.sql</a></code> makes the tables and indices, <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/rsload.sql" id="link-id0x2aab8bc59918">rsload.sql</a></code> copies the data from the TPC-H tables.</p> <p>The business question is to calculate the profit from sale of selected <code>parts</code> grouped by <code>year</code> and <code>country</code> of the <code>supplier</code>. This touches most of the tables, aggregates over 1/17 of all sales, and touches at least every page of the tables concerned, if not every row.</p> <blockquote> <code><pre>SELECT n_name AS nation, EXTRACT(year FROM o_orderdate) AS o_year, SUM (l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity) AS sum_profit FROM lineitem, part, partsupp, orders, supplier, nation WHERE s_suppkey = l_suppkey AND ps_suppkey = l_suppkey AND ps_partkey = l_partkey AND p_partkey = l_partkey AND o_orderkey = l_orderkey AND s_nationkey = n_nationkey AND p_name LIKE '%green%' GROUP BY nation, o_year ORDER BY nation, o_year DESC </pre> </code> </blockquote> <h2>Query Variants</h2> <p>The query variants discussed here are:</p> <ol> <li> <p>Hash based, the best plan -- <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/9h.sql" id="link-id0x2aab8bd04788">9h.sql</a></code> </p> </li> <li> <p>Index based with multicolumn rows, with <code>lineitem</code> index on <code>l_partkey</code> -- <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/9i.sql" id="link-id0x2aab8bd04bb8">9i.sql</a>, <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/9ir.sql" id="link-id0x2aab88894bf8">9ir.sql</a></code> </p> </li> <li> <p>Index based with multicolumn rows, <code>lineitem</code> index on <code>l_partkey, l_suppkey</code> -- <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/9ip.sql" id="link-id0x2aab88894fc8">9ip.sql</a>, <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/9ipr.sql" id="link-id0x2aab888950c8">9ipr.sql</a></code> </p> </li> <li> <p>Index based with one table per dependent column, index on <code>l_partkey</code> -- <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/9p.sql" id="link-id0x2aab8a4e42c8">9p.sql</a></code> </p> </li> <li> <p>index based with one table per dependent column, with materialized <code>l_partkey, l_suppkey</code> -> <code>l_orderkey, l_minenumber</code> -- <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/9pp.sql" id="link-id0x2aab8a4e4708">9pp.sql</a>, <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/9ppr.sql" id="link-id0x2aab8a4e4808">9ppr.sql</a></code> </p> </li> </ol> <p>These are done against row- and column-wise data representations with 3 different vectorization settings. The dynamic vector size starts at 10,000 values in a vector, and adaptively upgrades this to 1,000,000 if it finds that index access is too sparse. Accessing rows close to each other is more efficient than widely scattered rows in vectored index access, so using a larger vector will likely cause a denser, hence more efficient, access pattern.</p> <p>The 10K vector size corresponds to running with a fixed vector size. The Vector 1 sets vector size to 1, effectively running a tuple at a time, which corresponds to a non-vectorized engine.</p> <p>We note that <code>lineitem</code> and its single column projections contain 600M rows. So, a vector of 10K values will hit, on the average, every 60,000th row. A vector of 1,000,000 will thus hit every 600th. This is when doing random lookups that are in no specific order, e.g., getting <code>lineitems</code> by a secondary index on <code>l_partkey</code>.</p> <h3>1 — Hash-based plan</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Vector</th> <th>Dynamic</th> <th>10k</th> <th>1</th> </tr> <tr> <th style="text-align:left;">Column-wise </th> <td style="text-align:right;"><code>4.1 s</code> </td> <td style="text-align:right;"><code>4.1 s</code> </td> <td style="text-align:right;"><code>145 s</code> </td> </tr> <tr> <th style="text-align:left;">Row-wise </th> <td style="text-align:right;"><code>25.6 s</code> </td> <td style="text-align:right;"><code>25.9 s</code> </td> <td style="text-align:right;"><code>45.4 s</code> </td> </tr> </table> <p>Dynamic vector size has no effect here, as there is no indexed access that would gain from more locality. The column store is much faster because of less memory access (just scan the <code>l_partkey</code> column, and filter this with a Bloom filter; and then hash table lookup to pick only items with the desired <code>part</code>). The other columns are accessed only for the matching rows. The hash lookup is vectored since there are hundreds of compressed <code>l_partkey</code> values available at each time. The row store does the hash lookup row by row, hence losing cache locality and instruction-level parallelism.</p> <p>Without vectorization, we have a situation where the <code>lineitem</code> scan emits one row at a time. Restarting the scan with the column store takes much longer, since 5 buffers have to be located and pinned instead of one for the row store. The row store is thus slowed down less, but it too suffers almost a factor of 2 from interpretation overhead.</p> <h3>2 — Index-based, <code>lineitem</code> indexed on <code>l_partkey</code> </h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Vector</th> <th>Dynamic</th> <th>10k</th> <th>1</th> </tr> <tr> <th style="text-align:left;">Column-wise </th> <td style="text-align:right;"><code> 30.4 s </code> </td> <td style="text-align:right;"><code> 62.3 s </code> </td> <td style="text-align:right;"><code> 321 s</code> </td> </tr> <tr> <th style="text-align:left;">Row-wise </th> <td style="text-align:right;"><code> 31.8 s</code> </td> <td style="text-align:right;"><code> 27.7 s</code> </td> <td style="text-align:right;"><code> 122 s </code> </td> </tr> </table> <p>Here the plan scans <code>part</code>, then <code>partsupp</code>, which shares ordering with <code>part</code>; both are ordered on <code>partkey</code>. Then <code>lineitem</code> is fetched by a secondary index on <code>l_partkey</code>. This produces <code>l_orderkey, l_lineitem</code>, which are used to get the <code>l_suppkey</code>. We then check if the <code>l_suppkey</code> matches the <code>ps_suppkey</code> from <code>partsupp</code>, which drops 3/4 of the rows. The next join is on <code>orders</code>, which shares ordering with <code>lineitem</code>; both are ordered on <code>orderkey</code>.</p> <p>There is a narrow win for columns with dynamic vector size. When access becomes scattered, rows win by 2.5x, because there is only one page to access instead of 1 + 3 for columns. This is compensated for if the next item is found on the same page, which happens if the access pattern is denser. </p> <h3>3 — Index-based, <code>lineitem</code> indexed on <code>L_partkey, l_suppkey</code> </h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Vector</th> <th>Dynamic</th> <th>10k</th> <th>1</th> </tr> <tr> <th style="text-align:left;">Column-wise </th> <td style="text-align:right;"><code> 16.9 s</code> </td> <td style="text-align:right;"><code> 47.2 s </code> </td> <td style="text-align:right;"><code> 151 s </code> </td> </tr> <tr> <th style="text-align:left;">Row-wise </th> <td style="text-align:right;"><code> 22.4 s</code> </td> <td style="text-align:right;"><code> 20.7 s</code> </td> <td style="text-align:right;"><code> 89 s </code> </td> </tr> </table> <p>This is similar to the previous, except that now only <code>lineitems</code> that match <code>ps_partkey, ps_suppkey</code> are accessed, as the secondary index has two columns. Access is more local. Columns thus win more with dynamic vector size.</p> <h3>4 — Decomposed, index on <code>l_partkey</code> </h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Vector</th> <th>Dynamic</th> <th>10k</th> <th>1</th> </tr> <tr> <th style="text-align:left;">Column-wise </th> <td style="text-align:right;"><code> 35.7 s </code> </td> <td style="text-align:right;"><code> 170 s </code> </td> <td style="text-align:right;"><code> 601 s </code> </td> </tr> <tr> <th style="text-align:left;">Row-wise </th> <td style="text-align:right;"><code> 44.5 s </code> </td> <td style="text-align:right;"><code> 56.2 s </code> </td> <td style="text-align:right;"><code> 130 s </code> </td> </tr> </table> <p>Now, each of the <code>l_extendedprice, l_discount, l_quantity</code> and <code>l_suppkey</code> is a separate index lookup. The times are slightly higher but the dynamic is the same.</p> <p>The non-vectored columns case is hit the hardest.</p> <h3>5 — Decomposed, index on <code>l_partkey, l_suppkey</code> </h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Vector</th> <th>Dynamic</th> <th>10k</th> <th>1</th> </tr> <tr> <th style="text-align:left;">Column-wise </th> <td style="text-align:right;"><code> 19.6 s </code> </td> <td style="text-align:right;"><code> 111 s </code> </td> <td style="text-align:right;"><code> 257 s </code> </td> </tr> <tr> <th style="text-align:left;">Row-wise </th> <td style="text-align:right;"><code> 32.0 s </code> </td> <td style="text-align:right;"><code> 37 s </code> </td> <td style="text-align:right;"><code> 74.9 s </code> </td> </tr> </table> <p>Again, we see the same dynamic as with a multicolumn table. Columns win slightly more at long vector sizes because of overall better index performance in the presence of locality.</p> <h2>Space Utilization </h2> <p>The following tables list the space consumption in megabytes of allocated pages. Unallocated space in database files is not counted.</p> <p>The row-wise table also contains entries for column-wise structures (<code>DB.*</code>) since these have a row-wise sparse index. The size of this is however negligible, under 1% of the column-wise structures.</p> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align: center;">Row-Wise</th> <th> </th> <th style="text-align: center;"> Column-Wise</th> </tr> <tr> <td style="vertical-align:top;"> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>MB</th> <th>structure</th> </tr> <tr> <td style="text-align:right;"><code>73515</code> </td> <td style="text-align:left;"><code>R.DBA.LINEITEM</code> </td> </tr> <tr> <td style="text-align:right;"><code>14768</code> </td> <td style="text-align:left;"><code>R.DBA.ORDERS</code> </td> </tr> <tr> <td style="text-align:right;"><code>11728</code> </td> <td style="text-align:left;"><code>R.DBA.PARTSUPP</code> </td> </tr> <tr> <td style="text-align:right;"><code>10161</code></td> <td style="text-align:left;"><code>r_lpk_pk</code> </td> </tr> <tr> <td style="text-align:right;"><code>10003</code></td> <td style="text-align:left;"><code>r_l_pksk</code> </td> </tr> <tr> <td style="text-align:right;"><code>9908</code></td> <td style="text-align:left;"><code>R.DBA.l_partkey</code> </td> </tr> <tr> <td style="text-align:right;"><code>8761</code></td> <td style="text-align:left;"><code>R.DBA.l_extendedprice</code> </td> </tr> <tr> <td style="text-align:right;"><code>8745</code></td> <td style="text-align:left;"><code>R.DBA.l_discount</code> </td> </tr> <tr> <td style="text-align:right;"><code>8738</code></td> <td style="text-align:left;"><code>r_l_pk</code> </td> </tr> <tr> <td style="text-align:right;"><code>8713</code></td> <td style="text-align:left;"><code>R.DBA.l_suppkey</code> </td> </tr> <tr> <td style="text-align:right;"><code>6267</code></td> <td style="text-align:left;"><code>R.DBA.l_quantity</code> </td> </tr> <tr> <td style="text-align:right;"><code>2223</code></td> <td style="text-align:left;"><code>R.DBA.CUSTOMER</code> </td> </tr> <tr> <td style="text-align:right;"><code>2180</code></td> <td style="text-align:left;"><code>R.DBA.o_orderdate</code> </td> </tr> <tr> <td style="text-align:right;"><code>2041</code></td> <td style="text-align:left;"><code>r_O_CK</code> </td> </tr> <tr> <td style="text-align:right;"><code>1911</code></td> <td style="text-align:left;"><code>R.DBA.PART</code> </td> </tr> <tr> <td style="text-align:right;"><code>1281</code></td> <td style="text-align:left;"><code>R.DBA.ps_supplycost</code> </td> </tr> <tr> <td style="text-align:right;"><code>811</code></td> <td style="text-align:left;"><code>R.DBA.p_name</code> </td> </tr> <tr> <td style="text-align:right;"><code>127</code></td> <td style="text-align:left;"><code>R.DBA.SUPPLIER</code> </td> </tr> <tr> <td style="text-align:right;"><code>88</code></td> <td style="text-align:left;"><code>DB.DBA.LINEITEM</code> </td> </tr> <tr> <td style="text-align:right;"><code>24</code></td> <td style="text-align:left;"><code>DB.DBA.ORDERS</code> </td> </tr> <tr> <td style="text-align:right;"><code>11</code></td> <td style="text-align:left;"><code>DB.DBA.PARTSUPP</code> </td> </tr> <tr> <td style="text-align:right;"><code>9</code></td> <td style="text-align:left;"><code>R.DBA.s_nationkey</code> </td> </tr> <tr> <td style="text-align:right;"><code>5</code></td> <td style="text-align:left;"><code>l_pksk</code> </td> </tr> <tr> <td style="text-align:right;"><code>4</code></td> <td style="text-align:left;"><code>DB.DBA.l_partkey</code> </td> </tr> <tr> <td style="text-align:right;"><code>4</code></td> <td style="text-align:left;"><code>lpk_pk</code> </td> </tr> <tr> <td style="text-align:right;"><code>4</code></td> <td style="text-align:left;"><code>DB.DBA.l_extendedprice</code> </td> </tr> <tr> <td style="text-align:right;"><code>3</code></td> <td style="text-align:left;"><code>l_pk</code> </td> </tr> <tr> <td style="text-align:right;"><code>3</code></td> <td style="text-align:left;"><code>DB.DBA.l_suppkey</code> </td> </tr> <tr> <td style="text-align:right;"><code>2</code></td> <td style="text-align:left;"><code>DB.DBA.CUSTOMER</code> </td> </tr> <tr> <td style="text-align:right;"><code>2</code></td> <td style="text-align:left;"><code>DB.DBA.l_quantity</code> </td> </tr> <tr> <td style="text-align:right;"><code>1</code></td> <td style="text-align:left;"><code>DB.DBA.PART</code> </td> </tr> <tr> <td style="text-align:right;"><code>1</code></td> <td style="text-align:left;"><code>O_CK</code> </td> </tr> <tr> <td style="text-align:right;"><code>1</code></td> <td style="text-align:left;"><code>DB.DBA.l_discount</code> </td> </tr> </table> </td> <td> </td> <td style="vertical-align:top;"> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>MB</th> <th>structure</th> </tr> <tr> <td style="text-align:right;"><code>36482</code> </td> <td style="text-align:left;"><code>DB.DBA.LINEITEM</code> </td> </tr> <tr> <td style="text-align:right;"><code>13087</code></td> <td style="text-align:left;"><code>DB.DBA.ORDERS</code> </td> </tr> <tr> <td style="text-align:right;"><code>11587</code></td> <td style="text-align:left;"><code>DB.DBA.PARTSUPP</code> </td> </tr> <tr> <td style="text-align:right;"><code>5181</code></td> <td style="text-align:left;"><code>DB.DBA.l_extendedprice</code> </td> </tr> <tr> <td style="text-align:right;"><code>4431</code></td> <td style="text-align:left;"><code>l_pksk</code> </td> </tr> <tr> <td style="text-align:right;"><code>3072</code></td> <td style="text-align:left;"><code>DB.DBA.l_partkey</code> </td> </tr> <tr> <td style="text-align:right;"><code>2958</code></td> <td style="text-align:left;"><code>lpk_pk</code> </td> </tr> <tr> <td style="text-align:right;"><code>2918</code></td> <td style="text-align:left;"><code>l_pk</code> </td> </tr> <tr> <td style="text-align:right;"><code>2835</code></td> <td style="text-align:left;"><code>DB.DBA.l_suppkey</code> </td> </tr> <tr> <td style="text-align:right;"><code>2067</code></td> <td style="text-align:left;"><code>DB.DBA.CUSTOMER</code> </td> </tr> <tr> <td style="text-align:right;"><code>1618</code></td> <td style="text-align:left;"><code>DB.DBA.PART</code> </td> </tr> <tr> <td style="text-align:right;"><code>1156</code></td> <td style="text-align:left;"><code>DB.DBA.l_quantity</code> </td> </tr> <tr> <td style="text-align:right;"><code>961</code></td> <td style="text-align:left;"><code>DB.DBA.ps_supplycost</code> </td> </tr> <tr> <td style="text-align:right;"><code>814</code></td> <td style="text-align:left;"><code>O_CK</code> </td> </tr> <tr> <td style="text-align:right;"><code>798</code></td> <td style="text-align:left;"><code>DB.DBA.l_discount</code> </td> </tr> <tr> <td style="text-align:right;"><code>724</code></td> <td style="text-align:left;"><code>DB.DBA.p_name</code> </td> </tr> <tr> <td style="text-align:right;"><code>436</code></td> <td style="text-align:left;"><code>DB.DBA.o_orderdate</code> </td> </tr> <tr> <td style="text-align:right;"><code>126</code></td> <td style="text-align:left;"><code>DB.DBA.SUPPLIER</code> </td> </tr> <tr> <td style="text-align:right;"><code>1</code></td> <td style="text-align:left;"><code>DB.DBA.s_nationkey</code> </td> </tr> </table> </td> </tr> </table> <p>In both cases, the large tables are on top, but the column-wise case takes only half the space due to compression. </p> <p>We note that the single column projections are smaller column-wise. The <code>l_extendedprice</code> is not very compressible hence column-wise takes much more space than <code>l_quantity</code>; the row-wise difference is less. Since the leading key parts <code>l_orderkey, l_linenumber</code> are ordered and very compressible, the column-wise structures are in all cases noticeably more compact.</p> <p>The same applies to the multipart index <code>l_pksk</code> and <code>r_l_pksk</code> (<code>l_partkey, l_suppkey, l_orderkey, l_linenumber</code>) in column- and row-wise representations.</p> <p>Note that <code>STRING</code> columns (e.g., <code>l_comment</code>) are not compressed. If they were, the overall space ratio would be even more to the advantage of the column store.</p> <h2>Conclusions </h2> <p>Column stores and vectorization inextricably belong together. Column-wise compression yields great gains also for indices, since sorted data is easy to compress. Also for non-sorted data, adaptive use of dictionaries, run lengths, etc., produce great space savings. Columns also win with indexed access if there is locality.</p> <p>Row stores have less dependence on locality, but they also will win by a factor of 3 from dropping interpretation overhead and exploiting join locality.</p> <p>For point lookups, columns lose by 2+x but considering their better space efficiency, they will still win if space savings prevent going to secondary storage. For bulk random access, like in graph analytics, columns will win because of being able to operate on a large vector of keys to fetch.</p> <p>For many workloads, from TPC-H to LDBC social network, multi-part keys are a necessary component of physical design for performance if indexed access predominates. Triple stores and most graph databases do not have such and are therefore at a disadvantage. Self-joining, like in RDF or other vertically decomposed structures, can cost up to a factor of 10-20 over a column-wise multicolumn table. This depends however on the density of access.</p> <p>For analytical workloads, where the dominant join pattern is the scan with selective hash join, column stores are unbeatable, as per common wisdom. There are good physical reasons for this and the row store even with well implemented vectorization loses by a factor of 5.</p> <p>For decomposed structures, like RDF quads or single column projections of tables, column stores are relatively more advantageous because the key columns are extensively repeated, and these compress better with columns than with rows. In all the RDF workloads we have tried, columns never lose, but there is often a draw between rows and columns for lookup workloads. The longer the query, the more columns win.</p>
2015-07-13T13:56:51.535359-04:00
Vectored Execution in Column/Row Stores
http://www.openlinksw.com:443/weblog/oerling/?date=2015-07-13#1860
2015-07-13T17:46:03Z
<p>This article discusses the relationship between vectored execution and column- and row-wise data representations. <a href="http://dbpedia.org/resource/Column-oriented_DBMS" id="link-id0x2aab8a00d208">Column stores</a> are traditionally considered to be good for big scans but poor at indexed access. This is not necessarily so, though. We take <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aab8a00d398">TPC-H Q9</a> as a starting point, working with different row- and column-wise data representations and index choices. The goal of the article is to provide a primer on the performance implications of different physical designs.</p> <p>All the experiments are against the TPC-H 100G dataset hosted in Virtuoso on the test system used before in the <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aab8bcec9d8">TPC-H series</a>: dual Xeon E5-2630, 2x6 cores x 2 threads, 2.3GHz, 192 GB RAM. The Virtuoso version corresponds to the <a href="https://github.com/v7fasttrack/virtuoso-opensource/tree/feature/analytics" id="link-id0x2aab8bcecbd8">feature/analytics branch</a> in the <a href="https://github.com/v7fasttrack/virtuoso-opensource/" id="link-id0x2aab8bceccb8">v7fasttrack github project</a>. All run times are from memory, and queries generally run at full platform, 24 concurrent threads.</p> <p>We note that RDF stores and graph databases usually do not have secondary indices with multiple key parts. However, these do predominantly index-based access as opposed to big scans and hash joins. To explore the impact of this, we have decomposed the tables into projections with a single dependent column, which approximates a triple store or a vertically-decomposed graph database like <a href="http://dbpedia.org/resource/DEX_(Graph_database)" id="link-id0x2aab8bced038">Sparksee</a>.</p> <p>So, in these experiments, we store the relevant data four times over, as follows:</p> <ul> <li> <p>100G TPC-H dataset in the column-wise schema as discussed in the TPC-H series, now complemented with indices on <code>l_partkey</code> and on <code>l_partkey, l_suppkey</code> </p> </li> <li> <p>The same in row-wise data representation </p> </li> <li> <p>Column-wise tables with a single dependent column for <code>l_partkey, l_suppkey, l_extendedprice, l_quantity, l_discount, ps_supplycost, s_nationkey, p_name</code>. These all have the original tables primary key, e.g., <code>l_orderkey, l_linenumber</code> for the <code>l_ prefixed tables</code> </p> </li> <li> <p>The same with row-wise tables</p> </li> </ul> <p>The column-wise structures are in the <code>DB</code> qualifier, and the row-wise are in the <code>R</code> qualifier. There is a summary of space consumption at the end of the article. This is relevant for scalability, since even if row-wise structures can be faster for scattered random access, they will fit less data in RAM, typically 2 to 3x less. Thus, if "faster" rows cause the working set not to fit, "slower" columns will still win.</p> <p>As a starting point, we know that the best Q9 is the one in the Virtuoso TPC-H implementation which is described in <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aab8a78d208">Part 10 of the TPC-H blog series</a>. This is a scan of <code>lineitem</code> with a selective hash join followed ordered index access of <code>orders</code>, then hash joins against the smaller tables. There are special tricks to keep the hash tables small by propagating restrictions from the probe side to the build side.</p> <p>The query texts are <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/" id="link-id0x2aab8bc59538">available here</a>, along with the table declarations and scripts for populating the single-column projections. <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/rs.sql" id="link-id0x2aab8bc59778">rs.sql</a></code> makes the tables and indices, <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/rsload.sql" id="link-id0x2aab8bc59918">rsload.sql</a></code> copies the data from the TPC-H tables.</p> <p>The business question is to calculate the profit from sale of selected <code>parts</code> grouped by <code>year</code> and <code>country</code> of the <code>supplier</code>. This touches most of the tables, aggregates over 1/17 of all sales, and touches at least every page of the tables concerned, if not every row.</p> <blockquote> <code><pre>SELECT n_name AS nation, EXTRACT(year FROM o_orderdate) AS o_year, SUM (l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity) AS sum_profit FROM lineitem, part, partsupp, orders, supplier, nation WHERE s_suppkey = l_suppkey AND ps_suppkey = l_suppkey AND ps_partkey = l_partkey AND p_partkey = l_partkey AND o_orderkey = l_orderkey AND s_nationkey = n_nationkey AND p_name LIKE '%green%' GROUP BY nation, o_year ORDER BY nation, o_year DESC </pre> </code> </blockquote> <h2>Query Variants</h2> <p>The query variants discussed here are:</p> <ol> <li> <p>Hash based, the best plan -- <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/9h.sql" id="link-id0x2aab8bd04788">9h.sql</a></code> </p> </li> <li> <p>Index based with multicolumn rows, with <code>lineitem</code> index on <code>l_partkey</code> -- <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/9i.sql" id="link-id0x2aab8bd04bb8">9i.sql</a>, <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/9ir.sql" id="link-id0x2aab88894bf8">9ir.sql</a></code> </p> </li> <li> <p>Index based with multicolumn rows, <code>lineitem</code> index on <code>l_partkey, l_suppkey</code> -- <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/9ip.sql" id="link-id0x2aab88894fc8">9ip.sql</a>, <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/9ipr.sql" id="link-id0x2aab888950c8">9ipr.sql</a></code> </p> </li> <li> <p>Index based with one table per dependent column, index on <code>l_partkey</code> -- <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/9p.sql" id="link-id0x2aab8a4e42c8">9p.sql</a></code> </p> </li> <li> <p>index based with one table per dependent column, with materialized <code>l_partkey, l_suppkey</code> -> <code>l_orderkey, l_minenumber</code> -- <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/9pp.sql" id="link-id0x2aab8a4e4708">9pp.sql</a>, <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150713VectoredExecution/9ppr.sql" id="link-id0x2aab8a4e4808">9ppr.sql</a></code> </p> </li> </ol> <p>These are done against row- and column-wise data representations with 3 different vectorization settings. The dynamic vector size starts at 10,000 values in a vector, and adaptively upgrades this to 1,000,000 if it finds that index access is too sparse. Accessing rows close to each other is more efficient than widely scattered rows in vectored index access, so using a larger vector will likely cause a denser, hence more efficient, access pattern.</p> <p>The 10K vector size corresponds to running with a fixed vector size. The Vector 1 sets vector size to 1, effectively running a tuple at a time, which corresponds to a non-vectorized engine.</p> <p>We note that <code>lineitem</code> and its single column projections contain 600M rows. So, a vector of 10K values will hit, on the average, every 60,000th row. A vector of 1,000,000 will thus hit every 600th. This is when doing random lookups that are in no specific order, e.g., getting <code>lineitems</code> by a secondary index on <code>l_partkey</code>.</p> <h3>1 — Hash-based plan</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Vector</th> <th>Dynamic</th> <th>10k</th> <th>1</th> </tr> <tr> <th style="text-align:left;">Column-wise </th> <td style="text-align:right;"><code>4.1 s</code> </td> <td style="text-align:right;"><code>4.1 s</code> </td> <td style="text-align:right;"><code>145 s</code> </td> </tr> <tr> <th style="text-align:left;">Row-wise </th> <td style="text-align:right;"><code>25.6 s</code> </td> <td style="text-align:right;"><code>25.9 s</code> </td> <td style="text-align:right;"><code>45.4 s</code> </td> </tr> </table> <p>Dynamic vector size has no effect here, as there is no indexed access that would gain from more locality. The column store is much faster because of less memory access (just scan the <code>l_partkey</code> column, and filter this with a Bloom filter; and then hash table lookup to pick only items with the desired <code>part</code>). The other columns are accessed only for the matching rows. The hash lookup is vectored since there are hundreds of compressed <code>l_partkey</code> values available at each time. The row store does the hash lookup row by row, hence losing cache locality and instruction-level parallelism.</p> <p>Without vectorization, we have a situation where the <code>lineitem</code> scan emits one row at a time. Restarting the scan with the column store takes much longer, since 5 buffers have to be located and pinned instead of one for the row store. The row store is thus slowed down less, but it too suffers almost a factor of 2 from interpretation overhead.</p> <h3>2 — Index-based, <code>lineitem</code> indexed on <code>l_partkey</code> </h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Vector</th> <th>Dynamic</th> <th>10k</th> <th>1</th> </tr> <tr> <th style="text-align:left;">Column-wise </th> <td style="text-align:right;"><code> 30.4 s </code> </td> <td style="text-align:right;"><code> 62.3 s </code> </td> <td style="text-align:right;"><code> 321 s</code> </td> </tr> <tr> <th style="text-align:left;">Row-wise </th> <td style="text-align:right;"><code> 31.8 s</code> </td> <td style="text-align:right;"><code> 27.7 s</code> </td> <td style="text-align:right;"><code> 122 s </code> </td> </tr> </table> <p>Here the plan scans <code>part</code>, then <code>partsupp</code>, which shares ordering with <code>part</code>; both are ordered on <code>partkey</code>. Then <code>lineitem</code> is fetched by a secondary index on <code>l_partkey</code>. This produces <code>l_orderkey, l_lineitem</code>, which are used to get the <code>l_suppkey</code>. We then check if the <code>l_suppkey</code> matches the <code>ps_suppkey</code> from <code>partsupp</code>, which drops 3/4 of the rows. The next join is on <code>orders</code>, which shares ordering with <code>lineitem</code>; both are ordered on <code>orderkey</code>.</p> <p>There is a narrow win for columns with dynamic vector size. When access becomes scattered, rows win by 2.5x, because there is only one page to access instead of 1 + 3 for columns. This is compensated for if the next item is found on the same page, which happens if the access pattern is denser. </p> <h3>3 — Index-based, <code>lineitem</code> indexed on <code>L_partkey, l_suppkey</code> </h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Vector</th> <th>Dynamic</th> <th>10k</th> <th>1</th> </tr> <tr> <th style="text-align:left;">Column-wise </th> <td style="text-align:right;"><code> 16.9 s</code> </td> <td style="text-align:right;"><code> 47.2 s </code> </td> <td style="text-align:right;"><code> 151 s </code> </td> </tr> <tr> <th style="text-align:left;">Row-wise </th> <td style="text-align:right;"><code> 22.4 s</code> </td> <td style="text-align:right;"><code> 20.7 s</code> </td> <td style="text-align:right;"><code> 89 s </code> </td> </tr> </table> <p>This is similar to the previous, except that now only <code>lineitems</code> that match <code>ps_partkey, ps_suppkey</code> are accessed, as the secondary index has two columns. Access is more local. Columns thus win more with dynamic vector size.</p> <h3>4 — Decomposed, index on <code>l_partkey</code> </h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Vector</th> <th>Dynamic</th> <th>10k</th> <th>1</th> </tr> <tr> <th style="text-align:left;">Column-wise </th> <td style="text-align:right;"><code> 35.7 s </code> </td> <td style="text-align:right;"><code> 170 s </code> </td> <td style="text-align:right;"><code> 601 s </code> </td> </tr> <tr> <th style="text-align:left;">Row-wise </th> <td style="text-align:right;"><code> 44.5 s </code> </td> <td style="text-align:right;"><code> 56.2 s </code> </td> <td style="text-align:right;"><code> 130 s </code> </td> </tr> </table> <p>Now, each of the <code>l_extendedprice, l_discount, l_quantity</code> and <code>l_suppkey</code> is a separate index lookup. The times are slightly higher but the dynamic is the same.</p> <p>The non-vectored columns case is hit the hardest.</p> <h3>5 — Decomposed, index on <code>l_partkey, l_suppkey</code> </h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Vector</th> <th>Dynamic</th> <th>10k</th> <th>1</th> </tr> <tr> <th style="text-align:left;">Column-wise </th> <td style="text-align:right;"><code> 19.6 s </code> </td> <td style="text-align:right;"><code> 111 s </code> </td> <td style="text-align:right;"><code> 257 s </code> </td> </tr> <tr> <th style="text-align:left;">Row-wise </th> <td style="text-align:right;"><code> 32.0 s </code> </td> <td style="text-align:right;"><code> 37 s </code> </td> <td style="text-align:right;"><code> 74.9 s </code> </td> </tr> </table> <p>Again, we see the same dynamic as with a multicolumn table. Columns win slightly more at long vector sizes because of overall better index performance in the presence of locality.</p> <h2>Space Utilization </h2> <p>The following tables list the space consumption in megabytes of allocated pages. Unallocated space in database files is not counted.</p> <p>The row-wise table also contains entries for column-wise structures (<code>DB.*</code>) since these have a row-wise sparse index. The size of this is however negligible, under 1% of the column-wise structures.</p> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align: center;">Row-Wise</th> <th> </th> <th style="text-align: center;"> Column-Wise</th> </tr> <tr> <td style="vertical-align:top;"> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>MB</th> <th>structure</th> </tr> <tr> <td style="text-align:right;"><code>73515</code> </td> <td style="text-align:left;"><code>R.DBA.LINEITEM</code> </td> </tr> <tr> <td style="text-align:right;"><code>14768</code> </td> <td style="text-align:left;"><code>R.DBA.ORDERS</code> </td> </tr> <tr> <td style="text-align:right;"><code>11728</code> </td> <td style="text-align:left;"><code>R.DBA.PARTSUPP</code> </td> </tr> <tr> <td style="text-align:right;"><code>10161</code></td> <td style="text-align:left;"><code>r_lpk_pk</code> </td> </tr> <tr> <td style="text-align:right;"><code>10003</code></td> <td style="text-align:left;"><code>r_l_pksk</code> </td> </tr> <tr> <td style="text-align:right;"><code>9908</code></td> <td style="text-align:left;"><code>R.DBA.l_partkey</code> </td> </tr> <tr> <td style="text-align:right;"><code>8761</code></td> <td style="text-align:left;"><code>R.DBA.l_extendedprice</code> </td> </tr> <tr> <td style="text-align:right;"><code>8745</code></td> <td style="text-align:left;"><code>R.DBA.l_discount</code> </td> </tr> <tr> <td style="text-align:right;"><code>8738</code></td> <td style="text-align:left;"><code>r_l_pk</code> </td> </tr> <tr> <td style="text-align:right;"><code>8713</code></td> <td style="text-align:left;"><code>R.DBA.l_suppkey</code> </td> </tr> <tr> <td style="text-align:right;"><code>6267</code></td> <td style="text-align:left;"><code>R.DBA.l_quantity</code> </td> </tr> <tr> <td style="text-align:right;"><code>2223</code></td> <td style="text-align:left;"><code>R.DBA.CUSTOMER</code> </td> </tr> <tr> <td style="text-align:right;"><code>2180</code></td> <td style="text-align:left;"><code>R.DBA.o_orderdate</code> </td> </tr> <tr> <td style="text-align:right;"><code>2041</code></td> <td style="text-align:left;"><code>r_O_CK</code> </td> </tr> <tr> <td style="text-align:right;"><code>1911</code></td> <td style="text-align:left;"><code>R.DBA.PART</code> </td> </tr> <tr> <td style="text-align:right;"><code>1281</code></td> <td style="text-align:left;"><code>R.DBA.ps_supplycost</code> </td> </tr> <tr> <td style="text-align:right;"><code>811</code></td> <td style="text-align:left;"><code>R.DBA.p_name</code> </td> </tr> <tr> <td style="text-align:right;"><code>127</code></td> <td style="text-align:left;"><code>R.DBA.SUPPLIER</code> </td> </tr> <tr> <td style="text-align:right;"><code>88</code></td> <td style="text-align:left;"><code>DB.DBA.LINEITEM</code> </td> </tr> <tr> <td style="text-align:right;"><code>24</code></td> <td style="text-align:left;"><code>DB.DBA.ORDERS</code> </td> </tr> <tr> <td style="text-align:right;"><code>11</code></td> <td style="text-align:left;"><code>DB.DBA.PARTSUPP</code> </td> </tr> <tr> <td style="text-align:right;"><code>9</code></td> <td style="text-align:left;"><code>R.DBA.s_nationkey</code> </td> </tr> <tr> <td style="text-align:right;"><code>5</code></td> <td style="text-align:left;"><code>l_pksk</code> </td> </tr> <tr> <td style="text-align:right;"><code>4</code></td> <td style="text-align:left;"><code>DB.DBA.l_partkey</code> </td> </tr> <tr> <td style="text-align:right;"><code>4</code></td> <td style="text-align:left;"><code>lpk_pk</code> </td> </tr> <tr> <td style="text-align:right;"><code>4</code></td> <td style="text-align:left;"><code>DB.DBA.l_extendedprice</code> </td> </tr> <tr> <td style="text-align:right;"><code>3</code></td> <td style="text-align:left;"><code>l_pk</code> </td> </tr> <tr> <td style="text-align:right;"><code>3</code></td> <td style="text-align:left;"><code>DB.DBA.l_suppkey</code> </td> </tr> <tr> <td style="text-align:right;"><code>2</code></td> <td style="text-align:left;"><code>DB.DBA.CUSTOMER</code> </td> </tr> <tr> <td style="text-align:right;"><code>2</code></td> <td style="text-align:left;"><code>DB.DBA.l_quantity</code> </td> </tr> <tr> <td style="text-align:right;"><code>1</code></td> <td style="text-align:left;"><code>DB.DBA.PART</code> </td> </tr> <tr> <td style="text-align:right;"><code>1</code></td> <td style="text-align:left;"><code>O_CK</code> </td> </tr> <tr> <td style="text-align:right;"><code>1</code></td> <td style="text-align:left;"><code>DB.DBA.l_discount</code> </td> </tr> </table> </td> <td> </td> <td style="vertical-align:top;"> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>MB</th> <th>structure</th> </tr> <tr> <td style="text-align:right;"><code>36482</code> </td> <td style="text-align:left;"><code>DB.DBA.LINEITEM</code> </td> </tr> <tr> <td style="text-align:right;"><code>13087</code></td> <td style="text-align:left;"><code>DB.DBA.ORDERS</code> </td> </tr> <tr> <td style="text-align:right;"><code>11587</code></td> <td style="text-align:left;"><code>DB.DBA.PARTSUPP</code> </td> </tr> <tr> <td style="text-align:right;"><code>5181</code></td> <td style="text-align:left;"><code>DB.DBA.l_extendedprice</code> </td> </tr> <tr> <td style="text-align:right;"><code>4431</code></td> <td style="text-align:left;"><code>l_pksk</code> </td> </tr> <tr> <td style="text-align:right;"><code>3072</code></td> <td style="text-align:left;"><code>DB.DBA.l_partkey</code> </td> </tr> <tr> <td style="text-align:right;"><code>2958</code></td> <td style="text-align:left;"><code>lpk_pk</code> </td> </tr> <tr> <td style="text-align:right;"><code>2918</code></td> <td style="text-align:left;"><code>l_pk</code> </td> </tr> <tr> <td style="text-align:right;"><code>2835</code></td> <td style="text-align:left;"><code>DB.DBA.l_suppkey</code> </td> </tr> <tr> <td style="text-align:right;"><code>2067</code></td> <td style="text-align:left;"><code>DB.DBA.CUSTOMER</code> </td> </tr> <tr> <td style="text-align:right;"><code>1618</code></td> <td style="text-align:left;"><code>DB.DBA.PART</code> </td> </tr> <tr> <td style="text-align:right;"><code>1156</code></td> <td style="text-align:left;"><code>DB.DBA.l_quantity</code> </td> </tr> <tr> <td style="text-align:right;"><code>961</code></td> <td style="text-align:left;"><code>DB.DBA.ps_supplycost</code> </td> </tr> <tr> <td style="text-align:right;"><code>814</code></td> <td style="text-align:left;"><code>O_CK</code> </td> </tr> <tr> <td style="text-align:right;"><code>798</code></td> <td style="text-align:left;"><code>DB.DBA.l_discount</code> </td> </tr> <tr> <td style="text-align:right;"><code>724</code></td> <td style="text-align:left;"><code>DB.DBA.p_name</code> </td> </tr> <tr> <td style="text-align:right;"><code>436</code></td> <td style="text-align:left;"><code>DB.DBA.o_orderdate</code> </td> </tr> <tr> <td style="text-align:right;"><code>126</code></td> <td style="text-align:left;"><code>DB.DBA.SUPPLIER</code> </td> </tr> <tr> <td style="text-align:right;"><code>1</code></td> <td style="text-align:left;"><code>DB.DBA.s_nationkey</code> </td> </tr> </table> </td> </tr> </table> <p>In both cases, the large tables are on top, but the column-wise case takes only half the space due to compression. </p> <p>We note that the single column projections are smaller column-wise. The <code>l_extendedprice</code> is not very compressible hence column-wise takes much more space than <code>l_quantity</code>; the row-wise difference is less. Since the leading key parts <code>l_orderkey, l_linenumber</code> are ordered and very compressible, the column-wise structures are in all cases noticeably more compact.</p> <p>The same applies to the multipart index <code>l_pksk</code> and <code>r_l_pksk</code> (<code>l_partkey, l_suppkey, l_orderkey, l_linenumber</code>) in column- and row-wise representations.</p> <p>Note that <code>STRING</code> columns (e.g., <code>l_comment</code>) are not compressed. If they were, the overall space ratio would be even more to the advantage of the column store.</p> <h2>Conclusions </h2> <p>Column stores and vectorization inextricably belong together. Column-wise compression yields great gains also for indices, since sorted data is easy to compress. Also for non-sorted data, adaptive use of dictionaries, run lengths, etc., produce great space savings. Columns also win with indexed access if there is locality.</p> <p>Row stores have less dependence on locality, but they also will win by a factor of 3 from dropping interpretation overhead and exploiting join locality.</p> <p>For point lookups, columns lose by 2+x but considering their better space efficiency, they will still win if space savings prevent going to secondary storage. For bulk random access, like in graph analytics, columns will win because of being able to operate on a large vector of keys to fetch.</p> <p>For many workloads, from TPC-H to LDBC social network, multi-part keys are a necessary component of physical design for performance if indexed access predominates. Triple stores and most graph databases do not have such and are therefore at a disadvantage. Self-joining, like in RDF or other vertically decomposed structures, can cost up to a factor of 10-20 over a column-wise multicolumn table. This depends however on the density of access.</p> <p>For analytical workloads, where the dominant join pattern is the scan with selective hash join, column stores are unbeatable, as per common wisdom. There are good physical reasons for this and the row store even with well implemented vectorization loses by a factor of 5.</p> <p>For decomposed structures, like RDF quads or single column projections of tables, column stores are relatively more advantageous because the key columns are extensively repeated, and these compress better with columns than with rows. In all the RDF workloads we have tried, columns never lose, but there is often a draw between rows and columns for lookup workloads. The longer the query, the more columns win.</p>
2015-07-13T13:56:22.323548-04:00
Virtuoso at SIGMOD 2015
http://www.openlinksw.com:443/blog/vdb/blog/?date=2015-07-13#1859
2015-07-13T16:52:53Z
<p>Two papers presented at <a href="http://dbpedia.org/resource/SIGMOD" id="link-id0x10f5b4448">SIGMOD</a> <a href="http://www.sigmod2015.org/" id="link-id0x10f84b6e8">2015</a> have been added to the <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtuosoScienceLibrary" id="link-id0x10f84b8c8">Virtuoso Science Library</a>.</p> <ul> <li> <p> <b>Orri Erling (OpenLink Software); Alex Averbuch (Neo Technology); Josep Larriba-Pey (Sparsity Technologies); Hassan Chafi (Oracle Labs); Andrey Gubichev (TU Munich); Arnau Prat-Pérez (Universitat Politècnica de Catalunya); Minh-Duc Pham (VU University Amsterdam); Peter Boncz (CWI): <a href="http://dl.acm.org/authorize.cfm?key=N97179" id="link-id0x10f7147b8">The LDBC Social Network Benchmark: Interactive Workload</a>. <a href="http://www.sigmod2015.org/toc_sigmod.shtml" id="link-id0x10f7148d8">Proceedings of SIGMOD 2015, Melbourne</a>.</b> </p> <p>This paper is an overview of the challenges posed in the LDBC social network benchmark, from data generation to the interactive workload.</p> </li> <li> <p> <b>Mihai Capotă (Delft University of Technology), Tim Hegeman (Delft University of Technology), Alexandru Iosup (Delft University of Technology), Arnau Prat-Pérez (Universitat Politècnica de Catalunya), Orri Erling (OpenLink Software), Peter Boncz (CWI): <a href="http://dl.acm.org/authorize.cfm?key=N97204" id="link-id0x110864288">Graphalytics: A Big Data Benchmark for Graph-Processing Platforms</a>. <a href="http://www.sigmod2015.org/toc_grades.html" id="link-id0x1108643a8">Sigmod GRADES 2015</a>.</b> </p> <p>This paper discusses the future evolution of the LDBC Social Network Benchmark and gives a preview of Virtuoso graph traversal performance.</p> </li> </ul>
2015-07-13T12:52:53.615634-04:00
Virtuoso at SIGMOD 2015
http://www.openlinksw.com:443/weblog/oerling/?date=2015-07-13#1858
2015-07-13T16:52:45Z
<p>Two papers presented at <a href="http://dbpedia.org/resource/SIGMOD" id="link-id0x10f5b4448">SIGMOD</a> <a href="http://www.sigmod2015.org/" id="link-id0x10f84b6e8">2015</a> have been added to the <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtuosoScienceLibrary" id="link-id0x10f84b8c8">Virtuoso Science Library</a>.</p> <ul> <li> <p> <b>Orri Erling (OpenLink Software); Alex Averbuch (Neo Technology); Josep Larriba-Pey (Sparsity Technologies); Hassan Chafi (Oracle Labs); Andrey Gubichev (TU Munich); Arnau Prat-Pérez (Universitat Politècnica de Catalunya); Minh-Duc Pham (VU University Amsterdam); Peter Boncz (CWI): <a href="http://dl.acm.org/authorize.cfm?key=N97179" id="link-id0x10f7147b8">The LDBC Social Network Benchmark: Interactive Workload</a>. <a href="http://www.sigmod2015.org/toc_sigmod.shtml" id="link-id0x10f7148d8">Proceedings of SIGMOD 2015, Melbourne</a>.</b> </p> <p>This paper is an overview of the challenges posed in the LDBC social network benchmark, from data generation to the interactive workload.</p> </li> <li> <p> <b>Mihai Capotă (Delft University of Technology), Tim Hegeman (Delft University of Technology), Alexandru Iosup (Delft University of Technology), Arnau Prat-Pérez (Universitat Politècnica de Catalunya), Orri Erling (OpenLink Software), Peter Boncz (CWI): <a href="http://dl.acm.org/authorize.cfm?key=N97204" id="link-id0x110864288">Graphalytics: A Big Data Benchmark for Graph-Processing Platforms</a>. <a href="http://www.sigmod2015.org/toc_grades.html" id="link-id0x1108643a8">Sigmod GRADES 2015</a>.</b> </p> <p>This paper discusses the future evolution of the LDBC Social Network Benchmark and gives a preview of Virtuoso graph traversal performance.</p> </li> </ul>
2015-07-13T12:52:45.067773-04:00
Big Data, Part 1: Virtuoso Meets Hive
http://www.openlinksw.com:443/blog/vdb/blog/?date=2015-07-13#1857
2015-07-13T16:17:45Z
<p>In this series, we will look at <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aab3b3c1df8">Virtuoso</a> and some of the big data technologies out there. <a href="http://dbpedia.org/resource/SQL" id="link-id0x2aab3b3c1f08">SQL</a> on <a href="http://dbpedia.org/resource/Apache_Hadoop" id="link-id0x2aab3b3c2028">Hadoop</a> is of interest, as well as <a href="http://dbpedia.org/resource/NoSQL" id="link-id0x2aab3b3c2148">NoSQL</a> technologies.</p> <p>We begin at the beginning, with <a href="http://dbpedia.org/resource/Apache_Hive" id="link-id0x2aab3b3c2168">Hive</a>, the grand-daddy of SQL on Hadoop.</p> <p>The test platform is two Amazon R3.8 <a href="http://dbpedia.org/resource/Amazon_Machine_Image" id="link-id0x2aab62f44bf8">AMI</a> instances. We compared Hive with the Virtuoso 100G TPC-H experiment on the same platform, <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aab62f44d78">published earlier on this blog</a>. The runs follow a bulk load in both cases, with all data served from memory. The platform has 2x244GB RAM with only 40GB or so of working set.</p> <p>The Virtuoso version and settings are as in the <a href="http://www.openlinksw.com/weblogs/oerling/?id=1849" id="link-id0x2aab3be6df88">Virtuoso Cluster test AMI</a>.</p> <p>The Hive version is <a href="http://hortonworks.com/blog/announcing-apache-hive-0-14/" id="link-id0x2aab3be6e148">0.14</a> from the <a href="http://dbpedia.org/resource/Hortonworks" id="link-id0x2aab3be6e298">Hortonworks</a> <a href="http://hortonworks.com/hdp/downloads/" id="link-id0x2aab3b149c48">HDP 2.2 distribution>. The Hive schema and query formulations are the ones from <a href="https://github.com/hortonworks/hive-testbench" id="link-id0x2aab3b1498c8"><code>hive-testbench</code> on GitHub</a>. The Hive configuration parameters are as set by <a href="https://cwiki.apache.org/confluence/display/AMBARI/Installation+Guide+for+Ambari+2.0.1" id="link-id0x2aab3b149aa8">Ambari 2.0.1</a>. These are different from the ones in <code>hive-testbench</code>, but the Ambari choices offer higher performance on the platform. We did run statistics with Hive and did not specify any settings not in the <code>hive-testbench</code>. Thus we suppose the query plans were as good as Hive will make them. Platform utilization was even across both machines, and varied between 30% and 100% of the 2 x 32 hardware threads.</a> </p> <p>Load time with Hive was 742 seconds against 232 seconds with Virtuoso. In both cases, this was a copy from 32 <a href="http://dbpedia.org/resource/Comma-separated_values" id="link-id0x2aab3be6e328">CSV</a> files into native database format; for Hive, this is <a href="https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC" id="link-id0x2aab3b149868">ORC (Optimized Row Columnar)</a>. In Virtuoso, there is one index, (<code>o_custkey</code>); in Hive, there are no indices.</p> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Query</th> <th>Virtuoso</th> <th>Hive</th> <th>Notes</th> </tr> <tr> <th style="text-align:center;"> — </th> <td style="text-align:right;"> <code> 332 s </code> </td> <td style="text-align:right;"> <code> 742 s </code> </td> <td style="text-align:left;"> Data Load </td> </tr> <tr> <th style="text-align:center;"> Q1 </th> <td style="text-align:right;"> <code> 1.098 s </code> </td> <td style="text-align:right;"> <code> 296.636 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q2 </th> <td style="text-align:right;"> <code> 0.187 s </code> </td> <td style="text-align:right;"> <code> >3600 s </code> </td> <td style="text-align:left;"> Hive Timeout </td> </tr> <tr> <th style="text-align:center;"> Q3 </th> <td style="text-align:right;"> <code> 0.761 s </code> </td> <td style="text-align:right;"> <code> 98.652 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q4 </th> <td style="text-align:right;"> <code> 0.205 s </code> </td> <td style="text-align:right;"> <code> 147.867 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q5 </th> <td style="text-align:right;"> <code> 0.808 s </code> </td> <td style="text-align:right;"> <code> 114.782 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q6 </th> <td style="text-align:right;"> <code> 2.403 s </code> </td> <td style="text-align:right;"> <code> 71.789 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q7 </th> <td style="text-align:right;"> <code> 0.59 s </code> </td> <td style="text-align:right;"> <code> 394.201 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q8 </th> <td style="text-align:right;"> <code> 0.775 s </code> </td> <td style="text-align:right;"> <code> >3600 s </code> </td> <td style="text-align:left;"> Hive Timeout </td> </tr> <tr> <th style="text-align:center;"> Q9 </th> <td style="text-align:right;"> <code> 1.836 s </code> </td> <td style="text-align:right;"> <code> >3600 s </code> </td> <td style="text-align:left;"> Hive Timeout </td> </tr> <tr> <th style="text-align:center;"> Q10 </th> <td style="text-align:right;"> <code> 3.165 s </code> </td> <td style="text-align:right;"> <code> 179.646 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q11 </th> <td style="text-align:right;"> <code> 1.37 s </code> </td> <td style="text-align:right;"> <code> 43.094 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q12 </th> <td style="text-align:right;"> <code> 0.356 s </code> </td> <td style="text-align:right;"> <code> 101.193 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q13 </th> <td style="text-align:right;"> <code> 2.233 s </code> </td> <td style="text-align:right;"> <code> 208.476 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q14 </th> <td style="text-align:right;"> <code> 0.488 s </code> </td> <td style="text-align:right;"> <code> 89.047 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q15 </th> <td style="text-align:right;"> <code> 0.72 s </code> </td> <td style="text-align:right;"> <code> 136.431 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q16 </th> <td style="text-align:right;"> <code> 0.814 s </code> </td> <td style="text-align:right;"> <code> 105.652 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q17 </th> <td style="text-align:right;"> <code> 0.681 s </code> </td> <td style="text-align:right;"> <code> 255.848 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q18 </th> <td style="text-align:right;"> <code> 1.324 s </code> </td> <td style="text-align:right;"> <code> 337.921 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q19 </th> <td style="text-align:right;"> <code> 0.417 s </code> </td> <td style="text-align:right;"> <code> >3600 s </code> </td> <td style="text-align:left;"> Hive Timeout </td> </tr> <tr> <th style="text-align:center;"> Q20 </th> <td style="text-align:right;"> <code> 0.792 s </code> </td> <td style="text-align:right;"> <code> 193.965 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q21 </th> <td style="text-align:right;"> <code> 0.720 s </code> </td> <td style="text-align:right;"> <code> 670.718 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q22 </th> <td style="text-align:right;"> <code> 0.155 s </code> </td> <td style="text-align:right;"> <code> 68.462 s </code> </td> <td style="text-align:left;"> </td> </tr> </table> <p>Hive does relatively best on bulk load. This is understandable since this is a sequential read of many files in parallel with just compression to do.</p> <p>Hive's query times are obviously affected by not having a persistent memory image of the data, as this is always streamed from the storage files into other files as <a href="http://dbpedia.org/resource/MapReduce" id="link-id0x2aab3b57ad38">MapReduce</a> intermediate results. This seems to be an operator-at-a-time business as opposed to Virtuoso's vectorized streaming.</p> <p>The queries that would do partitioned <a href="http://dbpedia.org/resource/Hash_join" id="link-id0x2aab3bec82d8">hash joins</a> (e.g., Q9) did not finish under an hour in Hive, so we do not have a good metric of a cross-partition hash join.</p> <p>One could argue that one should benchmark Hive only in disk-bound circumstances. We may yet get to this.</p> <p>Our next stop will probably be <a href="https://en.wikipedia.org/wiki/Cloudera_Impala" id="link-id0x2aab3b4f7bc8">Impala</a>, which ought to do much better than Hive, as it dose not have the MapReduce overheads.</p> <p> <i><b>If you are a Hive expert</b> and believe that Hive should have done much better, please let us know how to improve the Hive scores, and we will retry.</i> </p>
2015-07-13T12:17:45.682053-04:00
Big Data, Part 1: Virtuoso Meets Hive
http://www.openlinksw.com:443/weblog/oerling/?date=2015-07-13#1856
2015-07-13T16:16:31Z
<p>In this series, we will look at <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aab3b3c1df8">Virtuoso</a> and some of the big data technologies out there. <a href="http://dbpedia.org/resource/SQL" id="link-id0x2aab3b3c1f08">SQL</a> on <a href="http://dbpedia.org/resource/Apache_Hadoop" id="link-id0x2aab3b3c2028">Hadoop</a> is of interest, as well as <a href="http://dbpedia.org/resource/NoSQL" id="link-id0x2aab3b3c2148">NoSQL</a> technologies.</p> <p>We begin at the beginning, with <a href="http://dbpedia.org/resource/Apache_Hive" id="link-id0x2aab3b3c2168">Hive</a>, the grand-daddy of SQL on Hadoop.</p> <p>The test platform is two Amazon R3.8 <a href="http://dbpedia.org/resource/Amazon_Machine_Image" id="link-id0x2aab62f44bf8">AMI</a> instances. We compared Hive with the Virtuoso 100G TPC-H experiment on the same platform, <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aab62f44d78">published earlier on this blog</a>. The runs follow a bulk load in both cases, with all data served from memory. The platform has 2x244GB RAM with only 40GB or so of working set.</p> <p>The Virtuoso version and settings are as in the <a href="http://www.openlinksw.com/weblogs/oerling/?id=1849" id="link-id0x2aab3be6df88">Virtuoso Cluster test AMI</a>.</p> <p>The Hive version is <a href="http://hortonworks.com/blog/announcing-apache-hive-0-14/" id="link-id0x2aab3be6e148">0.14</a> from the <a href="http://dbpedia.org/resource/Hortonworks" id="link-id0x2aab3be6e298">Hortonworks</a> <a href="http://hortonworks.com/hdp/downloads/" id="link-id0x2aab3b149c48">HDP 2.2 distribution>. The Hive schema and query formulations are the ones from <a href="https://github.com/hortonworks/hive-testbench" id="link-id0x2aab3b1498c8"><code>hive-testbench</code> on GitHub</a>. The Hive configuration parameters are as set by <a href="https://cwiki.apache.org/confluence/display/AMBARI/Installation+Guide+for+Ambari+2.0.1" id="link-id0x2aab3b149aa8">Ambari 2.0.1</a>. These are different from the ones in <code>hive-testbench</code>, but the Ambari choices offer higher performance on the platform. We did run statistics with Hive and did not specify any settings not in the <code>hive-testbench</code>. Thus we suppose the query plans were as good as Hive will make them. Platform utilization was even across both machines, and varied between 30% and 100% of the 2 x 32 hardware threads.</a> </p> <p>Load time with Hive was 742 seconds against 232 seconds with Virtuoso. In both cases, this was a copy from 32 <a href="http://dbpedia.org/resource/Comma-separated_values" id="link-id0x2aab3be6e328">CSV</a> files into native database format; for Hive, this is <a href="https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC" id="link-id0x2aab3b149868">ORC (Optimized Row Columnar)</a>. In Virtuoso, there is one index, (<code>o_custkey</code>); in Hive, there are no indices.</p> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Query</th> <th>Virtuoso</th> <th>Hive</th> <th>Notes</th> </tr> <tr> <th style="text-align:center;"> — </th> <td style="text-align:right;"> <code> 332 s </code> </td> <td style="text-align:right;"> <code> 742 s </code> </td> <td style="text-align:left;"> Data Load </td> </tr> <tr> <th style="text-align:center;"> Q1 </th> <td style="text-align:right;"> <code> 1.098 s </code> </td> <td style="text-align:right;"> <code> 296.636 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q2 </th> <td style="text-align:right;"> <code> 0.187 s </code> </td> <td style="text-align:right;"> <code> >3600 s </code> </td> <td style="text-align:left;"> Hive Timeout </td> </tr> <tr> <th style="text-align:center;"> Q3 </th> <td style="text-align:right;"> <code> 0.761 s </code> </td> <td style="text-align:right;"> <code> 98.652 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q4 </th> <td style="text-align:right;"> <code> 0.205 s </code> </td> <td style="text-align:right;"> <code> 147.867 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q5 </th> <td style="text-align:right;"> <code> 0.808 s </code> </td> <td style="text-align:right;"> <code> 114.782 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q6 </th> <td style="text-align:right;"> <code> 2.403 s </code> </td> <td style="text-align:right;"> <code> 71.789 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q7 </th> <td style="text-align:right;"> <code> 0.59 s </code> </td> <td style="text-align:right;"> <code> 394.201 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q8 </th> <td style="text-align:right;"> <code> 0.775 s </code> </td> <td style="text-align:right;"> <code> >3600 s </code> </td> <td style="text-align:left;"> Hive Timeout </td> </tr> <tr> <th style="text-align:center;"> Q9 </th> <td style="text-align:right;"> <code> 1.836 s </code> </td> <td style="text-align:right;"> <code> >3600 s </code> </td> <td style="text-align:left;"> Hive Timeout </td> </tr> <tr> <th style="text-align:center;"> Q10 </th> <td style="text-align:right;"> <code> 3.165 s </code> </td> <td style="text-align:right;"> <code> 179.646 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q11 </th> <td style="text-align:right;"> <code> 1.37 s </code> </td> <td style="text-align:right;"> <code> 43.094 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q12 </th> <td style="text-align:right;"> <code> 0.356 s </code> </td> <td style="text-align:right;"> <code> 101.193 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q13 </th> <td style="text-align:right;"> <code> 2.233 s </code> </td> <td style="text-align:right;"> <code> 208.476 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q14 </th> <td style="text-align:right;"> <code> 0.488 s </code> </td> <td style="text-align:right;"> <code> 89.047 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q15 </th> <td style="text-align:right;"> <code> 0.72 s </code> </td> <td style="text-align:right;"> <code> 136.431 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q16 </th> <td style="text-align:right;"> <code> 0.814 s </code> </td> <td style="text-align:right;"> <code> 105.652 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q17 </th> <td style="text-align:right;"> <code> 0.681 s </code> </td> <td style="text-align:right;"> <code> 255.848 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q18 </th> <td style="text-align:right;"> <code> 1.324 s </code> </td> <td style="text-align:right;"> <code> 337.921 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q19 </th> <td style="text-align:right;"> <code> 0.417 s </code> </td> <td style="text-align:right;"> <code> >3600 s </code> </td> <td style="text-align:left;"> Hive Timeout </td> </tr> <tr> <th style="text-align:center;"> Q20 </th> <td style="text-align:right;"> <code> 0.792 s </code> </td> <td style="text-align:right;"> <code> 193.965 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q21 </th> <td style="text-align:right;"> <code> 0.720 s </code> </td> <td style="text-align:right;"> <code> 670.718 s </code> </td> <td style="text-align:left;"> </td> </tr> <tr> <th style="text-align:center;"> Q22 </th> <td style="text-align:right;"> <code> 0.155 s </code> </td> <td style="text-align:right;"> <code> 68.462 s </code> </td> <td style="text-align:left;"> </td> </tr> </table> <p>Hive does relatively best on bulk load. This is understandable since this is a sequential read of many files in parallel with just compression to do.</p> <p>Hive's query times are obviously affected by not having a persistent memory image of the data, as this is always streamed from the storage files into other files as <a href="http://dbpedia.org/resource/MapReduce" id="link-id0x2aab3b57ad38">MapReduce</a> intermediate results. This seems to be an operator-at-a-time business as opposed to Virtuoso's vectorized streaming.</p> <p>The queries that would do partitioned <a href="http://dbpedia.org/resource/Hash_join" id="link-id0x2aab3bec82d8">hash joins</a> (e.g., Q9) did not finish under an hour in Hive, so we do not have a good metric of a cross-partition hash join.</p> <p>One could argue that one should benchmark Hive only in disk-bound circumstances. We may yet get to this.</p> <p>Our next stop will probably be <a href="https://en.wikipedia.org/wiki/Cloudera_Impala" id="link-id0x2aab3b4f7bc8">Impala</a>, which ought to do much better than Hive, as it dose not have the MapReduce overheads.</p> <p> <i><b>If you are a Hive expert</b> and believe that Hive should have done much better, please let us know how to improve the Hive scores, and we will retry.</i> </p>
2015-07-13T12:16:31.280965-04:00
Rethink Big and Europe?s Position in Big Data
http://www.openlinksw.com:443/blog/vdb/blog/?date=2015-06-29#1855
2015-06-29T19:36:16Z
<p>I will here take a break from core database and talk a bit about <a href="http://dbpedia.org/page/European_Union" id="link-id0x2aab76cf53a8">EU</a> policies for research funding.</p> <p>I had lunch with <a href="http://homepages.cwi.nl/~manegold/" id="link-id0x2aab74e7ff38">Stefan Manegold</a> of <a href="http://www.cwi.nl/" id="link-id0x2aab76fdf4a8">CWI</a> last week, where we talked about where European research should go. Stefan is involved in <i><a href="http://www.rethinkbig-project.eu/" id="link-id0x2aab767d4d88">RETHINK big</a>,</i> a European research project for compiling policy advice regarding big data for EC funding agencies. As part of this, he is interviewing various stakeholders such as end user organizations and developers of technology.</p> <p> <i>RETHINK big</i> wants to come up with a research agenda primarily for hardware, anything from faster networks to greener data centers. CWI represents software expertise in the consortium.</p> <p>So, we went through a regular questionnaire about how we see the landscape. I will summarize this below, as this is anyway informative.</p> <h3>Core competence</h3> <p>My own core competence is in core database functionality, specifically in high performance query processing, scale-out, and managing schema-less data. Most of the <a href="http://dbpedia.org/page/Virtuoso_Universal_Server" id="link-id0x2aab766406e8">Virtuoso</a> installed base is in the <a href="http://dbpedia.org/page/Resource_Description_Framework" id="link-id0x2aab74871c28">RDF</a> space, but most potential applications are in fact outside of this niche.</p> <h3>User challenges</h3> <p>The life sciences vertical is the one in which I have the most application insight, from going to <a href="http://dbpedia.org/resource/OpenPHACTS" id="link-id0x2aab765598c8">Open PHACTS</a> meetings and holding extensive conversations with domain specialists. We have users in many other verticals, from manufacturing to financial services, but there I do not have as much exposure to the actual applications.</p> <p>Having said this, the challenges throughout tend to be in diversity of data. Every researcher has their <a href="http://dbpedia.org/page/MySQL" id="link-id0x2aab76bff778">MySQL</a> database or <a href="http://dbpedia.org/page/Spreadsheet" id="link-id0x2aab75ceeaa8">spreadsheet</a>, and there may not even be a top level catalogue of everything. Data formats are diverse. Some people use <a href="http://dbpedia.org/resource/Linked_data" id="link-id0x2aab74e675e8">linked data</a> (most commonly RDF) as a top level metadata format. The application data, such as gene sequences or microarray assays, reside in their native file formats and there is little point in RDF-izing these.</p> <p>There are also public data resources that are published in RDF serializations as vendor-neutral, self-describing format. Having everything as triples, without <i>a priori</i> schema, makes things easier to integrate and in some cases easier to describe and query.</p> <p>So, the challenge is in the labor intensive nature of data integration. Data comes with different levels of quantity and quality, from hand-curated to <a href="http://dbpedia.org/resource/Natural_language_processing" id="link-id0x2aab76dfe0c8">NLP</a> extractions. Querying in the single- or double-digit terabyte range with RDF is quite possible, as we have shown many times on this blog, but most use cases do not even go that far. Anyway, what we see on the field is primarily a data diversity game. The scenario is data integration; the technology we provide is database. The data transformation proper, data cleansing, units of measure, entity de-duplication, and such core data-integration functions are performed using diverse, user-specific means.</p> <p> <a href="https://ch.linkedin.com/in/jervenbolleman" id="link-id0x2aab754e5e38">Jerven Bolleman</a> of the <a href="https://www.linkedin.com/company/sib-swiss-institute-of-bioinformatics" id="link-id0x2aab75d891d8">Swiss Institute of Bioinformatics</a> is a user of ours with whom we have long standing discussions on the virtues of federated data and querying. I advised Stefan to go talk to him; he has fresh views about the volume challenges with unexpected usage patterns. Designing for performance is tough if the usage pattern is out of the blue, like correlating air humidity on the day of measurement with the presence of some genomic patterns. Building a warehouse just for that might not be the preferred choice, so the problem field is not exhausted. Generally, I’d go for warehousing though.</p> <h3>What technology would you like to have? Network or power efficiency?</h3> <p>OK. Even a fast network is a network. A set of processes on a single shared-memory box is also a kind of network. InfiniBand is maybe half the throughput and 3x the latency of single threaded interprocess communication within one box. The operative word is latency. Making large systems always involves a network or something very much like one in large scale-up scenarios.</p> <p>On the software side, next to nobody understands latency and contention; yet these are the one core factor in any pursuit of scalability. Because of this situation, paradigms like <a href="http://dbpedia.org/page/MapReduce" id="link-id0x2aab76bea288">MapReduce</a> and <a href="http://dbpedia.org/page/Bulk_synchronous_parallel" id="link-id0x2aab75907c48">bulk synchronous parallel (BSP)</a> processing have become popular because these take the communication out of the program flow, so the programmer cannot muck this up, as otherwise would happen with the inevitability of destiny. Of course, our beloved SQL or declarative query in general does give scalability in many tasks without programmer participation. <a href="http://dbpedia.org/resource/Datalog" id="link-id0x2aab75e63cc8">Datalog</a> has also been used as a means of shipping computation around, as in the the work of <a href="http://www.linkedin.com/in/joehellerstein" id="link-id0x2aab75483c38">Hellerstein</a>.</p> <p>There are no easy solutions. We have built scale-out conscious, vectorized extensions to SQL procedures where one can express complex parallel, distributed flows, but people do not use or understand these. These are very useful, even indispensable, but only on the inside, not as a programmer-facing construct. MapReduce and BSP are the limit of what a development culture will absorb. MapReduce and BSP do not hide the fact of distributed processing. What about things that do? Parallel, partitioned extensions to <a href="http://dbpedia.org/resource/Fortran" id="link-id0x2aab76254d78">Fortran</a> <a href="http://dbpedia.org/page/Array_programming" id="link-id0x2aab75db6098">arrays</a>? <a href="http://dbpedia.org/page/Functional_programming" id="link-id0x2aab75ffb7b8">Functional languages</a>? I think that all the obvious aids to parallel/distributed programming have been conceived of. No silver bullet; just hard work. And above all the discernment of what paradigm fits what problem. Since these are always changing, there is no finite set of rules, and no substitute for understanding and insight, and the latter are vanishingly scarce. "<a href="http://dissoiblogoi.blogspot.com/2005/11/aristotles-paradigmatism.html" id="link-id0x2aab764f6738">Paradigmatism</a>," i.e., the belief that one particular programming model is a panacea outside of its original niche, is a common source of complexity and inefficiency. This is a common form of enthusiastic naïveté. </p> <p>If you look at power efficiency, the clusters that are the easiest to program consist of relatively few high power machines and a fast network. A typical node size is 16+ cores and 256G or more RAM. Amazon has these in entirely workable configurations, as <a href="http://www.openlinksw.com/weblogs/oerling/?id=1843" id="link-id0x2aab76396be8">documented earlier on this blog</a>. The leading edge in power efficiency is in larger number of smaller units, which makes life again harder. This exacerbates latency and forces one to partition the data more often, whereas one can play with replication of key parts of data more freely if the node size is larger.</p> <p>One very specific item where research might help without having to rebuild the hardware stack would be better, lower-latency exposure of networks to software. Lightweight threads and user-space access, bypassing slow protocol stacks, etc. <a href="https://en.wikipedia.org/wiki/Message_Passing_Interface" id="link-id0x2aab754d0208">MPI</a> has some of this, but maybe more could be done.</p> <p>So, I will take a cluster of such 16-core, 256GB machines on a faster network, over a cluster of 1024 x 4G mobile phones connected via USB. Very selfish and unecological, but one has to stay alive and life is tough enough as is.</p> <h3>Are there pressures to adapt business models based on big data?</h3> <p>The transition from <a href="http://dbpedia.org/page/Capital_expenditure" id="link-id0x2aab767114e8">capex</a> to <a href="http://dbpedia.org/page/Operating_expense" id="link-id0x2aab76254b28">opex</a> may be approaching maturity, as there have been workable cloud configurations for the past couple of years. The <a href="http://dbpedia.org/page/Amazon_Elastic_Compute_Cloud" id="link-id0x2aab75235fa8">EC2</a> from way back, with at best a 4 core 16G VM and a horrible network for $2/hr, is long gone. It remains the case that 4 months of 24x7 rent in the cloud equals the purchase price of physical hardware. So, for this to be economical long-term at scale, the average utilization should be about 10% of the peak, and peaks should not be on for more than 10% of the time.</p> <p>So, database software should be rented by the hour. A 100-150% markup for the $2.80 a large EC2 instance costs would be reasonable. Consider that 70% of the cost in TPC benchmarks is database software.</p> <p>There will be different pricing models combining different up-front and per-usage costs, just as there are for clouds now. If the platform business goes that way and the market accepts this, then systems software will follow. Price/performance quotes should probably be expressed as speed/price/hour instead of speed/price.</p> <p>The above is rather uncontroversial but there is no harm restating these facts. Reinforce often.</p> <h3>Well, the question is raised, what should Europe do that would have tangible impact in the next 5 years?</h3> <p>This is a harder question. There is some European business in wide area and mobile infrastructures. Competing against <a href="http://dbpedia.org/resource/Huawei" id="link-id0x2aab76c8a288">Huawei</a> will keep them busy. <a href="http://dbpedia.org/page/Intel" id="link-id0x2aab752b4178">Intel</a> and <a href="http://dbpedia.org/page/Mellanox_Technologies" id="link-id0x2aab753a84b8">Mellanox</a> will continue making faster networks regardless of European policies. Intel will continue building denser compute nodes, e.g., integrated Knight’s Corner with dual IB network and 16G fast RAM on chip. Clouds will continue making these available on demand once the technology is in mass production.</p> <p>What’s the next big innovation? <a href="http://dbpedia.org/page/Neuromorphic_engineering" id="link-id0x2aab76156358">Neuromorphic computing</a>? <a href="http://dbpedia.org/page/Quantum_computer" id="link-id0x2aab74efc458">Quantum computing</a>? Maybe. For now, I’d just do more engineering along the core competence discussed above, with emphasis on good marketing and scalable execution. By this I mean trained people who know something about deployment. There is a huge training gap. In the would-be "Age of Data," knowledge of how things actually work and scale is near-absent. I have offered to do some courses on this to partners and public alike, but I need somebody to drive this show; I have other things to do.</p> <p>I have been to many, many project review meetings, mostly as a project partner but also as reviewer. For the past year, the <a href="http://dbpedia.org/resource/European_Commission" id="link-id0x2aab7628cb88">EC</a> has used an innovation questionnaire at the end of the meetings. It is quite vague, and I don’t think it delivers much actionable intelligence.</p> <p>What would deliver this would be a venture capital type activity, with well-developed networks and active participation in developing a business. The EC is not now set up to perform this role, though. But the EC is a fairly large and wealthy entity, so it could invest some money via this type of channel. Also there should be higher individual incentives and rewards for speed and excellence. Getting the next Horizon 2020 research grant may be good, but better exists. The grants are competitive enough and the calls are not bad; they follow the times.</p> <p>In the projects I have seen, productization does get some attention, e.g., the <a href="http://stack.lod2.eu/blog/" id="link-id0x2aab768be028">LOD2 stack</a>, but it is not something that is really ongoing or with dedicated commercial backing. It may also be that there is no market to justify such dedicated backing. Much of the RDF work has been "me, too" — let’s do what the real database and data integration people do, but let’s just do this with triples. Innovation? Well, I took the best of the real DB world and adapted this to RDF, which did produce a competent piece of work with broad applicability, extending outside RDF. Is there better than this? Well, some of the data integration work (e.g., <a href="http://svn.aksw.org/papers/2011/WWW_LIMES/public.pdf" id="link-id0x2aab76423568">LIMES</a>) is not bad, and it might be picked up by some of the players that do this sort of thing in the broader world, e.g., <a href="http://dbpedia.org/resource/Informatica" id="link-id0x2aab753c3e68">Informatica</a>, the DI suites of big DB vendors, <a href="https://www.crunchbase.com/organization/tamr" id="link-id0x2aab76fc6e78">Tamr</a>, etc. I would not know if this in fact adds value to the non-RDF equivalents; I do not know the field well enough, but there could be a possibility.</p> <p>The recent emphasis for benchmarking, spearheaded by <a href="https://www.linkedin.com/in/stefanobertolo" id="link-id0x2aab76c45018">Stefano Bertolo</a> is good, as exemplified by the <a href="http://ldbcouncil.org/industry/organization/origins" id="link-id0x2aab75f4e288">LDBC FP7</a>. There should probably be one or two projects of this sort going at all times. These make challenges known and are an effective means of guiding research, with a large multiplier: Once a benchmark gets adopted, infinitely more work goes into solving the problem than in stating it in the first place.</p> <p>The aims and calls are good. The execution by projects is variable. For 1% of excellence, there apparently must be 99% of so-and-so, but this is just a fact of life and not specific to this context. The projects are rather diffuse. There is not a single outcome that gets all the effort. In this, the level of engagement of participants is less and focus is much more scattered than in startups. A really hungry, go-getter mood is mostly absent. I am a believer in core competence. Well, most people will agree that core competence is nice. But the projects I have seen do not drive for it hard enough.</p> <p>It is hard to say exactly what kinds of incentives could be offered to encourage truly exceptional work. The American startup scene does offer high rewards and something of this could be transplanted into the EC project world. I would not know exactly what form this could take, though.</p>
2015-06-29T15:36:16.622570-04:00
Rethink Big and Europe?s Position in Big Data
http://www.openlinksw.com:443/weblog/oerling/?date=2015-06-29#1854
2015-06-29T19:36:03Z
<p>I will here take a break from core database and talk a bit about <a href="http://dbpedia.org/page/European_Union" id="link-id0x2aab76cf53a8">EU</a> policies for research funding.</p> <p>I had lunch with <a href="http://homepages.cwi.nl/~manegold/" id="link-id0x2aab74e7ff38">Stefan Manegold</a> of <a href="http://www.cwi.nl/" id="link-id0x2aab76fdf4a8">CWI</a> last week, where we talked about where European research should go. Stefan is involved in <i><a href="http://www.rethinkbig-project.eu/" id="link-id0x2aab767d4d88">RETHINK big</a>,</i> a European research project for compiling policy advice regarding big data for EC funding agencies. As part of this, he is interviewing various stakeholders such as end user organizations and developers of technology.</p> <p> <i>RETHINK big</i> wants to come up with a research agenda primarily for hardware, anything from faster networks to greener data centers. CWI represents software expertise in the consortium.</p> <p>So, we went through a regular questionnaire about how we see the landscape. I will summarize this below, as this is anyway informative.</p> <h3>Core competence</h3> <p>My own core competence is in core database functionality, specifically in high performance query processing, scale-out, and managing schema-less data. Most of the <a href="http://dbpedia.org/page/Virtuoso_Universal_Server" id="link-id0x2aab766406e8">Virtuoso</a> installed base is in the <a href="http://dbpedia.org/page/Resource_Description_Framework" id="link-id0x2aab74871c28">RDF</a> space, but most potential applications are in fact outside of this niche.</p> <h3>User challenges</h3> <p>The life sciences vertical is the one in which I have the most application insight, from going to <a href="http://dbpedia.org/resource/OpenPHACTS" id="link-id0x2aab765598c8">Open PHACTS</a> meetings and holding extensive conversations with domain specialists. We have users in many other verticals, from manufacturing to financial services, but there I do not have as much exposure to the actual applications.</p> <p>Having said this, the challenges throughout tend to be in diversity of data. Every researcher has their <a href="http://dbpedia.org/page/MySQL" id="link-id0x2aab76bff778">MySQL</a> database or <a href="http://dbpedia.org/page/Spreadsheet" id="link-id0x2aab75ceeaa8">spreadsheet</a>, and there may not even be a top level catalogue of everything. Data formats are diverse. Some people use <a href="http://dbpedia.org/resource/Linked_data" id="link-id0x2aab74e675e8">linked data</a> (most commonly RDF) as a top level metadata format. The application data, such as gene sequences or microarray assays, reside in their native file formats and there is little point in RDF-izing these.</p> <p>There are also public data resources that are published in RDF serializations as vendor-neutral, self-describing format. Having everything as triples, without <i>a priori</i> schema, makes things easier to integrate and in some cases easier to describe and query.</p> <p>So, the challenge is in the labor intensive nature of data integration. Data comes with different levels of quantity and quality, from hand-curated to <a href="http://dbpedia.org/resource/Natural_language_processing" id="link-id0x2aab76dfe0c8">NLP</a> extractions. Querying in the single- or double-digit terabyte range with RDF is quite possible, as we have shown many times on this blog, but most use cases do not even go that far. Anyway, what we see on the field is primarily a data diversity game. The scenario is data integration; the technology we provide is database. The data transformation proper, data cleansing, units of measure, entity de-duplication, and such core data-integration functions are performed using diverse, user-specific means.</p> <p> <a href="https://ch.linkedin.com/in/jervenbolleman" id="link-id0x2aab754e5e38">Jerven Bolleman</a> of the <a href="https://www.linkedin.com/company/sib-swiss-institute-of-bioinformatics" id="link-id0x2aab75d891d8">Swiss Institute of Bioinformatics</a> is a user of ours with whom we have long standing discussions on the virtues of federated data and querying. I advised Stefan to go talk to him; he has fresh views about the volume challenges with unexpected usage patterns. Designing for performance is tough if the usage pattern is out of the blue, like correlating air humidity on the day of measurement with the presence of some genomic patterns. Building a warehouse just for that might not be the preferred choice, so the problem field is not exhausted. Generally, I’d go for warehousing though.</p> <h3>What technology would you like to have? Network or power efficiency?</h3> <p>OK. Even a fast network is a network. A set of processes on a single shared-memory box is also a kind of network. InfiniBand is maybe half the throughput and 3x the latency of single threaded interprocess communication within one box. The operative word is latency. Making large systems always involves a network or something very much like one in large scale-up scenarios.</p> <p>On the software side, next to nobody understands latency and contention; yet these are the one core factor in any pursuit of scalability. Because of this situation, paradigms like <a href="http://dbpedia.org/page/MapReduce" id="link-id0x2aab76bea288">MapReduce</a> and <a href="http://dbpedia.org/page/Bulk_synchronous_parallel" id="link-id0x2aab75907c48">bulk synchronous parallel (BSP)</a> processing have become popular because these take the communication out of the program flow, so the programmer cannot muck this up, as otherwise would happen with the inevitability of destiny. Of course, our beloved SQL or declarative query in general does give scalability in many tasks without programmer participation. <a href="http://dbpedia.org/resource/Datalog" id="link-id0x2aab75e63cc8">Datalog</a> has also been used as a means of shipping computation around, as in the the work of <a href="http://www.linkedin.com/in/joehellerstein" id="link-id0x2aab75483c38">Hellerstein</a>.</p> <p>There are no easy solutions. We have built scale-out conscious, vectorized extensions to SQL procedures where one can express complex parallel, distributed flows, but people do not use or understand these. These are very useful, even indispensable, but only on the inside, not as a programmer-facing construct. MapReduce and BSP are the limit of what a development culture will absorb. MapReduce and BSP do not hide the fact of distributed processing. What about things that do? Parallel, partitioned extensions to <a href="http://dbpedia.org/resource/Fortran" id="link-id0x2aab76254d78">Fortran</a> <a href="http://dbpedia.org/page/Array_programming" id="link-id0x2aab75db6098">arrays</a>? <a href="http://dbpedia.org/page/Functional_programming" id="link-id0x2aab75ffb7b8">Functional languages</a>? I think that all the obvious aids to parallel/distributed programming have been conceived of. No silver bullet; just hard work. And above all the discernment of what paradigm fits what problem. Since these are always changing, there is no finite set of rules, and no substitute for understanding and insight, and the latter are vanishingly scarce. "<a href="http://dissoiblogoi.blogspot.com/2005/11/aristotles-paradigmatism.html" id="link-id0x2aab764f6738">Paradigmatism</a>," i.e., the belief that one particular programming model is a panacea outside of its original niche, is a common source of complexity and inefficiency. This is a common form of enthusiastic naïveté. </p> <p>If you look at power efficiency, the clusters that are the easiest to program consist of relatively few high power machines and a fast network. A typical node size is 16+ cores and 256G or more RAM. Amazon has these in entirely workable configurations, as <a href="http://www.openlinksw.com/weblogs/oerling/?id=1843" id="link-id0x2aab76396be8">documented earlier on this blog</a>. The leading edge in power efficiency is in larger number of smaller units, which makes life again harder. This exacerbates latency and forces one to partition the data more often, whereas one can play with replication of key parts of data more freely if the node size is larger.</p> <p>One very specific item where research might help without having to rebuild the hardware stack would be better, lower-latency exposure of networks to software. Lightweight threads and user-space access, bypassing slow protocol stacks, etc. <a href="https://en.wikipedia.org/wiki/Message_Passing_Interface" id="link-id0x2aab754d0208">MPI</a> has some of this, but maybe more could be done.</p> <p>So, I will take a cluster of such 16-core, 256GB machines on a faster network, over a cluster of 1024 x 4G mobile phones connected via USB. Very selfish and unecological, but one has to stay alive and life is tough enough as is.</p> <h3>Are there pressures to adapt business models based on big data?</h3> <p>The transition from <a href="http://dbpedia.org/page/Capital_expenditure" id="link-id0x2aab767114e8">capex</a> to <a href="http://dbpedia.org/page/Operating_expense" id="link-id0x2aab76254b28">opex</a> may be approaching maturity, as there have been workable cloud configurations for the past couple of years. The <a href="http://dbpedia.org/page/Amazon_Elastic_Compute_Cloud" id="link-id0x2aab75235fa8">EC2</a> from way back, with at best a 4 core 16G VM and a horrible network for $2/hr, is long gone. It remains the case that 4 months of 24x7 rent in the cloud equals the purchase price of physical hardware. So, for this to be economical long-term at scale, the average utilization should be about 10% of the peak, and peaks should not be on for more than 10% of the time.</p> <p>So, database software should be rented by the hour. A 100-150% markup for the $2.80 a large EC2 instance costs would be reasonable. Consider that 70% of the cost in TPC benchmarks is database software.</p> <p>There will be different pricing models combining different up-front and per-usage costs, just as there are for clouds now. If the platform business goes that way and the market accepts this, then systems software will follow. Price/performance quotes should probably be expressed as speed/price/hour instead of speed/price.</p> <p>The above is rather uncontroversial but there is no harm restating these facts. Reinforce often.</p> <h3>Well, the question is raised, what should Europe do that would have tangible impact in the next 5 years?</h3> <p>This is a harder question. There is some European business in wide area and mobile infrastructures. Competing against <a href="http://dbpedia.org/resource/Huawei" id="link-id0x2aab76c8a288">Huawei</a> will keep them busy. <a href="http://dbpedia.org/page/Intel" id="link-id0x2aab752b4178">Intel</a> and <a href="http://dbpedia.org/page/Mellanox_Technologies" id="link-id0x2aab753a84b8">Mellanox</a> will continue making faster networks regardless of European policies. Intel will continue building denser compute nodes, e.g., integrated Knight’s Corner with dual IB network and 16G fast RAM on chip. Clouds will continue making these available on demand once the technology is in mass production.</p> <p>What’s the next big innovation? <a href="http://dbpedia.org/page/Neuromorphic_engineering" id="link-id0x2aab76156358">Neuromorphic computing</a>? <a href="http://dbpedia.org/page/Quantum_computer" id="link-id0x2aab74efc458">Quantum computing</a>? Maybe. For now, I’d just do more engineering along the core competence discussed above, with emphasis on good marketing and scalable execution. By this I mean trained people who know something about deployment. There is a huge training gap. In the would-be "Age of Data," knowledge of how things actually work and scale is near-absent. I have offered to do some courses on this to partners and public alike, but I need somebody to drive this show; I have other things to do.</p> <p>I have been to many, many project review meetings, mostly as a project partner but also as reviewer. For the past year, the <a href="http://dbpedia.org/resource/European_Commission" id="link-id0x2aab7628cb88">EC</a> has used an innovation questionnaire at the end of the meetings. It is quite vague, and I don’t think it delivers much actionable intelligence.</p> <p>What would deliver this would be a venture capital type activity, with well-developed networks and active participation in developing a business. The EC is not now set up to perform this role, though. But the EC is a fairly large and wealthy entity, so it could invest some money via this type of channel. Also there should be higher individual incentives and rewards for speed and excellence. Getting the next Horizon 2020 research grant may be good, but better exists. The grants are competitive enough and the calls are not bad; they follow the times.</p> <p>In the projects I have seen, productization does get some attention, e.g., the <a href="http://stack.lod2.eu/blog/" id="link-id0x2aab768be028">LOD2 stack</a>, but it is not something that is really ongoing or with dedicated commercial backing. It may also be that there is no market to justify such dedicated backing. Much of the RDF work has been "me, too" — let’s do what the real database and data integration people do, but let’s just do this with triples. Innovation? Well, I took the best of the real DB world and adapted this to RDF, which did produce a competent piece of work with broad applicability, extending outside RDF. Is there better than this? Well, some of the data integration work (e.g., <a href="http://svn.aksw.org/papers/2011/WWW_LIMES/public.pdf" id="link-id0x2aab76423568">LIMES</a>) is not bad, and it might be picked up by some of the players that do this sort of thing in the broader world, e.g., <a href="http://dbpedia.org/resource/Informatica" id="link-id0x2aab753c3e68">Informatica</a>, the DI suites of big DB vendors, <a href="https://www.crunchbase.com/organization/tamr" id="link-id0x2aab76fc6e78">Tamr</a>, etc. I would not know if this in fact adds value to the non-RDF equivalents; I do not know the field well enough, but there could be a possibility.</p> <p>The recent emphasis for benchmarking, spearheaded by <a href="https://www.linkedin.com/in/stefanobertolo" id="link-id0x2aab76c45018">Stefano Bertolo</a> is good, as exemplified by the <a href="http://ldbcouncil.org/industry/organization/origins" id="link-id0x2aab75f4e288">LDBC FP7</a>. There should probably be one or two projects of this sort going at all times. These make challenges known and are an effective means of guiding research, with a large multiplier: Once a benchmark gets adopted, infinitely more work goes into solving the problem than in stating it in the first place.</p> <p>The aims and calls are good. The execution by projects is variable. For 1% of excellence, there apparently must be 99% of so-and-so, but this is just a fact of life and not specific to this context. The projects are rather diffuse. There is not a single outcome that gets all the effort. In this, the level of engagement of participants is less and focus is much more scattered than in startups. A really hungry, go-getter mood is mostly absent. I am a believer in core competence. Well, most people will agree that core competence is nice. But the projects I have seen do not drive for it hard enough.</p> <p>It is hard to say exactly what kinds of incentives could be offered to encourage truly exceptional work. The American startup scene does offer high rewards and something of this could be transplanted into the EC project world. I would not know exactly what form this could take, though.</p>
2015-06-29T15:36:03.387872-04:00
Virtuoso updated to version 7.2.1
http://www.openlinksw.com:443/blog/vdb/blog/?date=2015-06-29#1853
2015-06-29T19:21:25Z
<p>We're pleased to announce that Virtuoso 7.2.1 is now available, and includes various enhancements and bug fixes. Important additions include new support for <code>xsd:boolean</code> and <code>TIMEZONE</code>-less <code>DATETIME</code> & <code>xsd:dateTime</code>; and significantly improved compatibility with the Jena and Sesame Frameworks.</p> <p>New product features as of June 24, 2015, v7.2.1, include:</p> <ul> <li> <p>Virtuoso Engine</p> <ul> <li>Added support for <code>TIMEZONE</code>-less <code>xsd:dateTime</code> and <code>DATETIME</code> </li> <li>Added support for <code>xsd:boolean</code> </li> <li>Added new text index functions </li> <li>Added better handling of HTTP status codes on SPARQL graph protocol endpoint </li> <li>Added new cache for compiled regular expressions </li> <li>Added support for expression in <code>TOP/SKIP</code> </li> </ul> </li> <li> <p>SPARQL</p> <ul> <li>Added support for SPARQL <code>GROUPING SETS</code> </li> <li>Added support for SPARQL 1.1 <code>EBV</code> (Efficient Boolean Value) </li> <li>Added support for <code>define input:with-fallback-graph_uri </code> </li> <li>Added support for <code>define input:target-fallback-graph-uri</code> </li> </ul> </li> <li> <p>Jena & Sesame Compatibility</p> <ul> <li>Added support for using <code>rdf_insert_triple_c()</code> to insert BNode data </li> <li>Added support for returning <code>xsd:boolean</code> as <code>true/false</code> rather than <code>1/0</code> </li> <li>Added support for <code>maxQueryTimeout</code> in Sesame2 provider </li> </ul> </li> <li> <p>JDBC Driver</p> <ul> <li>Added new methods <code>setLogFileName</code> and <code>getLogFileName</code> </li> <li>Added new attribute "<code>logFileName</code>" to <code>VirtuosoDataSources</code> for logging support </li> </ul> </li> <li> <p>Faceted Browser</p> <ul> <li>Added support for emitting HTML5+Microdata instead of RDFa as default HTML page </li> <li>Added query optimizations </li> <li>Added new footer icons to /describe page </li> </ul> </li> <li> <p>Conductor and DAV</p> <ul> <li>Added support for VAD dependency tree </li> <li>Added support for default vdirs when creating new listeners </li> <li>Added support for private RDF graphs </li> <li>Added support for LDP in DAV API </li> <li>Added option to create shared folder if not present </li> <li>Added option to enable/disable DET graphs binding </li> <li>Added option to set content length threshold for asynchronous sponging </li> <li>Added folder option related to <code>.TTL</code> redirection </li> <li>Added functions to edit turtle files </li> <li>Added popup dialog to search for unknown prefixes </li> <li>Added registry option to add missing prefixes for <code>.TTL</code> files </li> </ul> </li> </ul> More details of the additions, fixes, and other changes in this update of both Open Source and Commercial Editions, may be found on <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VOSNews#2015-06-24%20-%20Virtuoso%20Open-Source%20Edition%207.2.1%20Released" id="link-id0x2aab74ec1248">the Virtuoso News page</a>. Additional Information: <ul> <li> <p>Virtuoso Commercial Edition</p> <ul> <li> <a href="http://virtuoso.openlinksw.com/" id="link-id0x2aab75e9ead8">Home Page</a> </li> <li> <a href="http://virtuoso.openlinksw.com/download/" id="link-id0x2aab75e9ebe8">Download Page</a> </li> </ul> </li> <li> <p>Virtuoso Open Source Edition </p> <ul> <li> <a href="https://github.com/openlink/virtuoso-opensource/tree/develop/7" id="link-id0x2aab75e9eef8">Development Branch</a> </li> <li> <a href="https://github.com/openlink/virtuoso-opensource/tree/stable/7" id="link-id0x2aab75e9f048">Stable Branch</a> </li> </ul> </li> </ul>
2015-06-29T15:21:25.985492-04:00
Announcement: UDA Release 7.0 Lite Edition ODBC Driver for Oracle
http://www.openlinksw.com:443/weblog/uda/135/?date=2015-06-23#1852
2015-06-23T20:14:43Z
<p>Today, we've updated the <b><a href="http://uda.openlinksw.com/odbc-oracle-st/" id="link-id0x2aac17a529d8">Lite Edition ODBC Driver for Oracle</a></b>.</p> <p>Installation and configuration takes only minutes, by following the documentation which remains available anytime, specifically for this driver on <a href="http://wikis.openlinksw.com/dataspace/owiki/wiki/UdaWikiWeb/PreinstallOracleLiteWin32" id="link-id0x2aac17a1eaf8">Windows</a>. </p> <p>Release 7.0 licenses are also <a href="http://uda.openlinksw.com/offers/offers.vsp?data_access=http%3A%2F%2Fwww.openlinksw.com%2Fontology%2Fsoftware%23DataAccessODBC&dbms=http%3A%2F%2Fwww.openlinksw.com%2Fontology%2Fsoftware%23Oracle&format=http%3A%2F%2Fdata.openlinksw.com%2Foplweb%2Fproduct_format%2Fst%23this&version=7.x" id="link-id0x2aac17c00a78">available for immediate purchase</a>.</p> <h2>Client Platform Support</h2> <p>Release 7.0 installers are <a href="http://uda.openlinksw.com/odbc-oracle-st/" id="link-id0x2aac171b9928">available for immediate download</a> for Windows. Builds for Mac, Linux, and other Unix-like OS will be available soon; please <a href="http://www.openlinksw.com/contact/" id="link-id0x2aabdbc32078">contact us if you have urgent need</a>.</p> <p>Release 7.0 supports all 32-bit and 64-bit ODBC client tools and applications, both GUI and command-line, on —</p> <table style="align:center;width:45%;"> <tr> <th style="width:45%;vertical-align:top;">Windows and Windows Server <br /> on x86 and x86_64</th> </tr> <tr> <td style="width:45%;vertical-align:top;"> <ul> <li> <p>Windows 8.x (x86, x86_64)</p> </li> <li> <p>Windows 7.x (x86, x86_64)</p> </li> <li> <p>Windows Vista (x86, x86_64)</p> </li> <li> <p>Windows XP (x86, x86_64)</p> </li> <li> <p>Windows Server 2012 R2 (x86_64)</p> </li> <li> <p>Windows Server 2012 (x86_64)</p> </li> <li> <p>Windows Server 2008 R2 (x86_64)</p> </li> <li> <p>Windows Server 2008 (x86, x86_64)</p> </li> <li> <p>Windows Server 2003 R2 (x86, x86_64)</p> </li> <li> <p>Windows Server 2003 (x86, x86_64)</p> </li> </ul> </td> </tr> </table> <h2>DBMS Version Support</h2> <p>The Release 7.0 Lite Edition ODBC Driver supports virtually every version of Oracle in current use, including —</p> <table style="align:center;width:95%;"> <tr> <td style="width:30%;vertical-align:top;"> <ul> <li> <p>Oracle 12c Release 1 (12.1.x)</p> </li> <li> <p>Oracle 11g Release 2 (11.2.x)</p> </li> <li> <p>Oracle 11g Release 1 (11.1.x)</p> </li> <li> <p>Oracle 10g Release 2 (10.2.x)</p> </li> <li> <p>Oracle 10g Release 1 (10.1.x)</p> </li> <li> <p>Oracle 9i Release 2 (9.2.x)</p> </li> </ul> </td> </tr> </table> <h2> Changes since Release 6.x </h2> <h3>Additions</h3> <ul> <li> <p>Support for Oracle 12c</p> </li> <li>Support for Windows 8 and Windows Server 2012 <p></p> </li> </ul> <h3>Fixes</h3> <ul> <li> <p>Enhanced support for Oracle 11g</p> </li> </ul>
2015-06-23T16:56:02.257317-04:00
Announcement: UDA Release 7.0 Express Edition ODBC Driver for Oracle
http://www.openlinksw.com:443/weblog/uda/135/?date=2015-06-23#1851
2015-06-23T20:14:36Z
<p>Today, we've updated the <b><a href="http://uda.openlinksw.com/odbc-oracle-ee/" id="link-id0x2aac15bc16d8">Express Edition ODBC Driver for Oracle</a></b>.</p> <p>Installation and configuration takes only minutes, by following the documentation which remains available anytime, specifically for this driver on <a href="http://wikis.openlinksw.com/dataspace/owiki/wiki/UdaWikiWeb/PreInstallOracleExpressOSX" id="link-id0x2aac17257ea8">OS X</a> and <a href="http://wikis.openlinksw.com/dataspace/owiki/wiki/UdaWikiWeb/PreInstallOracleExpressWin32" id="link-id0x2aac1432de48">Windows</a>. </p> <p>Release 7.0 licenses are also <a href="http://uda.openlinksw.com/offers/offers.vsp?data_access=http%3A%2F%2Fwww.openlinksw.com%2Fontology%2Fsoftware%23DataAccessODBC&dbms=http%3A%2F%2Fwww.openlinksw.com%2Fontology%2Fsoftware%23Oracle&format=http%3A%2F%2Fdata.openlinksw.com%2Foplweb%2Fproduct_format%2Fexpress%23this&version=7.x" id="link-id0x2aac15531168">available for immediate purchase</a>.</p> <h2>Client Platform Support</h2> <p>Release 7.0 installers are <a href="http://uda.openlinksw.com/odbc-oracle-ee/" id="link-id0x2aac16f812e8">available for immediate download</a> for Mac and Windows. (Express Edition is not typically produced for Linux and other Unix-like OS will be available soon; please <a href="http://www.openlinksw.com/contact/" id="link-id0x2aabdbc32078">contact us if you have specific need</a>.)</p> <p>Release 7.0 supports all 32-bit and 64-bit ODBC client tools and applications, both GUI and command-line, on —</p> <table style="align:center;width:95%;"> <tr> <th style="width:45%;vertical-align:top;">OS X and OS X Server <br /> on x86 and x86_64</th> <th style="width:45%;vertical-align:top;">Windows and Windows Server <br /> on x86 and x86_64</th> </tr> <tr> <td style="width:45%;vertical-align:top;"> <ul> <li> <p> Yosemite (10.10.x) (x86_64)</p> </li> <li> <p>Mavericks (10.9.x) (x86_64)</p> </li> <li> <p>Mountain Lion (10.8.x) (x86_64)</p> </li> <li> <p>Lion (10.7.x) (x86_64)</p> </li> </ul> </td> <td style="width:45%;vertical-align:top;"> <ul> <li> <p>Windows 8.x (x86, x86_64)</p> </li> <li> <p>Windows 7.x (x86, x86_64)</p> </li> <li> <p>Windows Vista (x86, x86_64)</p> </li> <li> <p>Windows XP (x86, x86_64)</p> </li> <li> <p>Windows Server 2012 R2 (x86_64)</p> </li> <li> <p>Windows Server 2012 (x86_64)</p> </li> <li> <p>Windows Server 2008 R2 (x86_64)</p> </li> <li> <p>Windows Server 2008 (x86, x86_64)</p> </li> <li> <p>Windows Server 2003 R2 (x86, x86_64)</p> </li> <li> <p>Windows Server 2003 (x86, x86_64)</p> </li> </ul> </td> </tr> </table> <h2>DBMS Version Support</h2> <p>The Release 7.0 Express Edition ODBC Driver supports virtually every version of Oracle in current use, including —</p> <table style="align:center;width:95%;"> <tr> <td style="width:30%;vertical-align:top;"> <ul> <li> <p>Oracle 12c Release 1 (12.1.x)</p> </li> <li> <p>Oracle 11g Release 2 (11.2.x)</p> </li> <li> <p>Oracle 11g Release 1 (11.1.x)</p> </li> <li> <p>Oracle 10g Release 2 (10.2.x)</p> </li> <li> <p>Oracle 10g Release 1 (10.1.x)</p> </li> <li> <p>Oracle 9i Release 2 (9.2.x)</p> </li> </ul> </td> </tr> </table> <h2> Changes since Release 6.x </h2> <h3>Additions</h3> <ul> <li> <p>Support for Oracle 12c</p> </li> <li>Support for OS X Yosemite, Windows 8, and Windows Server 2012 <p></p> </li> </ul> <h3>Fixes</h3> <ul> <li> <p>Enhanced support for Oracle 11g</p> </li> <li>Enhanced support for OS X Mavericks <p></p> </li> </ul>
2015-06-26T10:41:25.982191-04:00
Virtuoso Elastic Cluster Benchmarks AMI on Amazon EC2
http://www.openlinksw.com:443/blog/vdb/blog/?date=2015-06-16#1850
2015-06-16T21:53:53Z
<p>We have another new Amazon machine image, this time for deploying your own Virtuoso Elastic Cluster on the cloud. The <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x10cd1dc38">previous post</a> gave a summary of running TPC-H on this image. This post is about what the AMI consists of and how to set it up.</p> <p> <i><b>Note:</b> This AMI is running a pre-release build of Virtuoso 7.5, Commercial Edition. Features are subject to change, and this build is not licensed for any use other than the AMI-based benchmarking described herein.</i> </p> <p>There are two preconfigured cluster setups; one is for two (2) machines/instances and one is for four (4). Generation and loading of TPC-H data, as well as the benchmark run itself, is preconfigured, so you can do it by entering just a few commands. The whole sequence of doing a terabyte (1000G) scale TPC-H takes under two hours, with 30 minutes to generate the data, 35 minutes to load, and 35 minutes to do three benchmark runs. The 100G scale is several times faster still.</p> <p>To experiment with this AMI, you will need a set of license files, one per machine/instance, which <a href="http://www.openlinksw.com/contact/" id="link-id0x10ca15da8">our Sales Team can provide</a>.</p> <p>Detailed instructions are on the AMI, in <code>/home/ec2-user/<a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150616ElasticClusterSetup/cluster_instructions.txt" title="cluster_instructions.txt" alt="Detailed Cluster Setup Instructions" id="link-id0x2aab90d25058">cluster_instructions.txt</a></code>, but the basic steps to get up and running are as follows:</p> <ol> <li> <p>Instantiate machine image <b>ami-811becea)</b> (AMI ID is subject to change; you should be able to find the latest by searching for "OpenLink Virtuoso Benchmarks" in "Community AMIs"; this one is short-named <code>virtuoso-bench-cl</code>) with two or four (2 or 4) R3.8xlarge instances within one virtual private cluster and placement group. Make sure the VPC security is set to allow all connections.</p> </li> <li> <p>Log in to the first, and fill in the configuration file with the internal IP addresses of all machines instantiated in step 1.</p> </li> <li> <p>Distribute the license files to the instances, and start the OpenLink License Manager on each machine.</p> </li> <li> <p>Run 3 shell commands to set up the file systems and the Virtuoso configuration files.</p> </li> <li> <p>If you do not plan to run one of these benchmarks, you can simply start and work with the Virtuoso cluster now. It is ready for use with an empty database.</p> </li> <li> <p>Before running one of these benchmark, generate the appropriate dataset with the <code>dbgen.sh</code> command.</p> </li> <li> <p>Bulk load the data with <code>load.sh</code>.</p> </li> <li> <p>Run the benchmark with <code>run.sh</code>.</p> </li> </ol> <p>Right now the cluster benchmarks are limited to TPC-H but cluster versions of the LDBC Social Network and Semantic Publishing benchmarks will follow soon.</p>
2015-06-17T10:13:21.091311-04:00
Virtuoso Elastic Cluster Benchmarks AMI on Amazon EC2
http://www.openlinksw.com:443/weblog/oerling/?date=2015-06-16#1849
2015-06-16T21:53:29Z
<p>We have another new Amazon machine image, this time for deploying your own Virtuoso Elastic Cluster on the cloud. The <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x10cd1dc38">previous post</a> gave a summary of running TPC-H on this image. This post is about what the AMI consists of and how to set it up.</p> <p> <i><b>Note:</b> This AMI is running a pre-release build of Virtuoso 7.5, Commercial Edition. Features are subject to change, and this build is not licensed for any use other than the AMI-based benchmarking described herein.</i> </p> <p>There are two preconfigured cluster setups; one is for two (2) machines/instances and one is for four (4). Generation and loading of TPC-H data, as well as the benchmark run itself, is preconfigured, so you can do it by entering just a few commands. The whole sequence of doing a terabyte (1000G) scale TPC-H takes under two hours, with 30 minutes to generate the data, 35 minutes to load, and 35 minutes to do three benchmark runs. The 100G scale is several times faster still.</p> <p>To experiment with this AMI, you will need a set of license files, one per machine/instance, which <a href="http://www.openlinksw.com/contact/" id="link-id0x10ca15da8">our Sales Team can provide</a>.</p> <p>Detailed instructions are on the AMI, in <code>/home/ec2-user/<a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150616ElasticClusterSetup/cluster_instructions.txt" title="cluster_instructions.txt" alt="Detailed Cluster Setup Instructions" id="link-id0x2aab9a4158b8">cluster_instructions.txt</a></code>, but the basic steps to get up and running are as follows:</p> <ol> <li> <p>Instantiate machine image <b>ami-811becea)</b> (AMI ID is subject to change; you should be able to find the latest by searching for "OpenLink Virtuoso Benchmarks" in "Community AMIs"; this one is short-named <code>virtuoso-bench-cl</code>) with two or four (2 or 4) R3.8xlarge instances within one virtual private cluster and placement group. Make sure the VPC security is set to allow all connections.</p> </li> <li> <p>Log in to the first, and fill in the configuration file with the internal IP addresses of all machines instantiated in step 1.</p> </li> <li> <p>Distribute the license files to the instances, and start the OpenLink License Manager on each machine.</p> </li> <li> <p>Run 3 shell commands to set up the file systems and the Virtuoso configuration files.</p> </li> <li> <p>If you do not plan to run one of these benchmarks, you can simply start and work with the Virtuoso cluster now. It is ready for use with an empty database.</p> </li> <li> <p>Before running one of these benchmark, generate the appropriate dataset with the <code>dbgen.sh</code> command.</p> </li> <li> <p>Bulk load the data with <code>load.sh</code>.</p> </li> <li> <p>Run the benchmark with <code>run.sh</code>.</p> </li> </ol> <p>Right now the cluster benchmarks are limited to TPC-H but cluster versions of the LDBC Social Network and Semantic Publishing benchmarks will follow soon.</p>
2015-06-17T10:13:37.325753-04:00
Announcement: UDA Release 7.0 Lite Edition ODBC Driver for Sybase and Microsoft SQL Server
http://www.openlinksw.com:443/weblog/uda/135/?date=2015-06-16#1848
2015-06-16T21:43:12Z
<p>In coming months, we'll be gradually shipping Release 7.0 of all our UDA drivers. This post will be the first of many, describing some of the fixes, changes, and improvements in each driver as they are made available.</p> <p>Today, we have the <b><a href="http://uda.openlinksw.com/odbc-sqlserver-st/" id="link-id0x2aabdba80608">Lite Edition ODBC Drivers for Sybase and Microsoft SQL Server</a></b>.</p> <p>Installation and configuration takes only minutes, by following the documentation which remains available anytime, specifically for this driver on <a href="http://wikis.openlinksw.com/dataspace/owiki/wiki/UdaWikiWeb/PreinstallSQLLiteOSX" id="link-id0x2aabdbc32318">OS X</a> and <a href="http://wikis.openlinksw.com/dataspace/owiki/wiki/UdaWikiWeb/PreinstallSQLLiteWin32" id="link-id0x2aabdbc32498">Windows</a>. </p> <p>Release 7.0 licenses are also <a href="http://uda.openlinksw.com/offers/offers.vsp?data_access=http%3A%2F%2Fwww.openlinksw.com%2Fontology%2Fsoftware%23DataAccessODBC&dbms=http%3A%2F%2Fwww.openlinksw.com%2Fontology%2Fsoftware%23SQLServer&format=http%3A%2F%2Fdata.openlinksw.com%2Foplweb%2Fproduct_format%2Fst%23this&version=7.x" id="link-id0x2aabdbc32738">available for immediate purchase</a>.</p> <h2>Client Platform Support</h2> <p>Release 7.0 installers are <a href="http://uda.openlinksw.com/odbc-sqlserver-st/" id="link-id0x2aabdba80798">available for immediate download</a> for Mac and Windows. Builds for Linux and other Unix-like OS will be available soon; please <a href="http://www.openlinksw.com/contact/" id="link-id0x2aabdbc32078">contact us if you have urgent need</a>.</p> <p>Release 7.0 supports all 32-bit and 64-bit ODBC client tools and applications, both GUI and command-line, on —</p> <table style="align:center;width:95%;"> <tr> <th style="width:45%;vertical-align:top;">OS X and OS X Server <br /> on x86 and x86_64</th> <th style="width:45%;vertical-align:top;">Windows and Windows Server <br /> on x86 and x86_64</th> </tr> <tr> <td style="width:45%;vertical-align:top;"> <ul> <li> <p> Yosemite (10.10.x) (x86_64)</p> </li> <li> <p>Mavericks (10.9.x) (x86_64)</p> </li> <li> <p>Mountain Lion (10.8.x) (x86_64)</p> </li> <li> <p>Lion (10.7.x) (x86_64)</p> </li> </ul> </td> <td style="width:45%;vertical-align:top;"> <ul> <li> <p>Windows 8.x (x86, x86_64)</p> </li> <li> <p>Windows 7.x (x86, x86_64)</p> </li> <li> <p>Windows Vista (x86, x86_64)</p> </li> <li> <p>Windows XP (x86, x86_64)</p> </li> <li> <p>Windows Server 2012 R2 (x86_64)</p> </li> <li> <p>Windows Server 2012 (x86_64)</p> </li> <li> <p>Windows Server 2008 R2 (x86_64)</p> </li> <li> <p>Windows Server 2008 (x86, x86_64)</p> </li> <li> <p>Windows Server 2003 R2 (x86, x86_64)</p> </li> <li> <p>Windows Server 2003 (x86, x86_64)</p> </li> </ul> </td> </tr> </table> <h2>DBMS Version Support</h2> <p>The Release 7.0 Lite Edition ODBC Driver supports virtually every version of Microsoft SQL Server and Sybase Adaptive Server in current use, including —</p> <table style="align:center;width:95%;"> <tr> <td style="width:30%;vertical-align:top;"> <ul> <li> <p>Microsoft SQL Server 6.5</p> </li> <li> <p>Microsoft SQL Server 7.0</p> </li> <li> <p>Microsoft SQL Server 2000</p> </li> <li> <p>Microsoft SQL Server 2005</p> </li> <li> <p>Microsoft SQL Server 2008</p> </li> <li> <p>Microsoft SQL Server 2012</p> </li> <li> <p>Microsoft SQL Server 2014 </p> </li> <li> <p>Microsoft SQL Azure</p> </li> </ul> </td> <td style="width:30%;vertical-align:top;"> <ul> <li> <p>Sybase SQL Server 4.x</p> </li> <li> <p>Sybase SQL Server 10.x</p> </li> <li> <p>Sybase SQL Server 11.x</p> </li> <li> <p>Sybase Adaptive Server Enterprise (ASE) 11.x</p> </li> <li> <p>Sybase Adaptive Server Enterprise (ASE) 12.x</p> </li> <li> <p>Sybase Adaptive Server Enterprise (ASE) 15.x</p> </li> </ul> </td> <td style="width:30%;vertical-align:top;"> <ul> <li> <p>Sybase SQL Anywhere 6.x</p> </li> <li> <p>Sybase Adaptive Server Anywhere (ASA) 7.x</p> </li> <li> <p>Sybase Adaptive Server Anywhere (ASA) 8.x</p> </li> <li> <p>Sybase Adaptive Server Anywhere (ASA) 9.x</p> </li> <li> <p>Sybase SQL Anywhere 10.x</p> </li> <li> <p>Sybase SQL Anywhere 11.x</p> </li> </ul> </td> </tr> </table> <h2> Changes since Release 6.x </h2> <h3>Additions</h3> <ul> <li> <p>added support for <a href="https://msdn.microsoft.com/en-us/library/cc280604%28v=sql.100%29.aspx" id="link-id0x2aabdbbc6278"><code>SPARSE</code> columns</a> in <code>SQLColumns()</code> call</p> </li> <ul> <li> <p>added DSN options <code><nowiki>SHOWSPARSECOLS / ShowSparseCols</nowiki></code> and Multi-Tier connect option <code>-X</code> )</p> </li> <li> <p>details, based on test table:</p> <blockquote> <pre> <code><nowiki> CREATE TABLE tbl_sparse_test ( col1 INT SPARSE , col2 INT , col3 XML COLUMN_SET FOR ALL_SPARSE_COLUMNS ) </nowiki> </code> </pre></blockquote> <ul> <li> <p>wildcard query will return only <code>col2</code> and <code>col3</code>; will not include <code>SPARSE</code> columns. This is standard SQL Server behavior, and it cannot be changed. </p> <blockquote> <pre> <code><nowiki> SELECT * FROM tbl_sparse_test ; </nowiki> </code> </pre></blockquote> <p>To include <code>SPARSE</code> columns in results, they must be explicitly <code>SELECTed</code> </p> <blockquote> <pre> <code><nowiki> SELECT col1, col2, col3 FROM tbl_sparse_test ; </nowiki> </code> </pre></blockquote> </li> <li> <p>By default, calls to <code>SQLColumns()</code> don't return Sparse Columns. To receive full columns list:</p> <ul> <li> <p>via our Lite Edition ODBC driver — </p> <ol> <li> <p>open connection with <code>SHOWSPARSECOLS</code> in DSN connection string, e.g., <code><nowiki>"DSN=TdsSQL;UID=sa;PWD=sa;SHOWSPARSECOLS=Y;"</nowiki></code> </p> </li> <li> <p> <code><nowiki>SQLColumns (hstmt, NULL, 0, NULL, 0, L"tbl_sparse_test", SQL_NTS, NULL, 0 );</nowiki> </code> </p> </li> </ol> </li> <li> <p>via the Microsoft ODBC driver — </p> <ol> <li> <p> <code><nowiki>SQLSetStmtAttr (hstmt, SQL_SOPT_SS_NAME_SCOPE, (SQLPOINTER)SQL_SS_NAME_SCOPE_EXTENDED, SQL_IS_SMALLINT);</nowiki> </code> </p> </li> <li> <p> <code><nowiki>SQLColumns (hstmt, L"tempdb", SQL_NTS, L"dbo", SQL_NTS, L"tbl_sparse_test", SQL_NTS, NULL, 0 );</nowiki> </code> </p> </li> </ol> </li> </ul> </li> </ul> </li> </ul> <li> <p>added support for new SQL Server datatypes such as <a href="https://msdn.microsoft.com/en-us/library/bb677335.aspx" id="link-id0x2aab7f6655a8"><code>datetime2</code></a> </p> </li> <li> <p>added support for <a href="https://msdn.microsoft.com/en-us/library/dd304783.aspx" id="link-id0x107149af8"><code>NBCROW</code> token</a> </p> </li> <li> <p>added support for Sybase 15 </p> <ul> <li> <p>added support for <a href="http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.infocenter.dc38151.1540/doc/html/jwi1287495761977.html" id="link-id0x2aac180e7f08"><code>BIGDATETIME</code> and <code>BIGTIME</code></a> </p> </li> <li> <p>added support for <code>UNITEXT</code> </p> </li> <li> <p>added support for <code><a href="http://infocenter.sybase.com/help/topic/com.sybase.infocenter.dc38151.1540/doc/html/san1278452908740.html" id="link-id0x2aac185b5d48">UNSIGNED BIGINT</a></code> </p> </li> </ul> </li> </ul> <h3>Fixes</h3> <ul> <li> <p>fixed issue with SQL Server <a href="https://msdn.microsoft.com/en-us/library/ms177603.aspx" id="link-id0x2aabdbb8e848"><code>BIT</code> datatype</a> </p> </li> <li> <p>fixed memory overwrite error, when DB procedure is called with <code><nowiki>SQL_PARAM_OUTPUT</nowiki></code> parameter of <code>CHAR/VARCHAR/LONGVARCHAR</code> </p> </li> <li> <p>fixed issue with <code>VARBINARY</code> datatype and DB procedures</p> </li> <li> <p>fixed issue with converting <code>TIMESTAMP</code> to <code>CHAR/WCHAR</code> </p> </li> <li> <p>fixed datatype info in <code><nowiki>SQLGetTypeInfo</nowiki></code> -- new Sybase and MSSQL datatypes were added</p> </li> <li> <p>fixed database catalog and query metadata info for Sybase 15's <code>UNSIGNED INT, UNSIGNED SMALLINT, BIGINT, SYSNAME, LONGSYSNAME</code> </p> </li> </ul>
2015-06-23T16:18:08.671938-04:00
Why Do I Need To Pay For ODBC, JDBC, ADO.NET, OLE-DB Drivers? (revisited)
http://www.openlinksw.com:443/weblog/uda/135/?date=2015-06-11#1847
2015-06-11T21:18:05Z
<h2>Situation Analysis</h2> <p>It's the year 2015, and the fundamental issues associated with the utility of data access drivers remain confusing. Basically, we remain uncertain about the value-to-compensation alignment of ODBC (Open Database Connectivity), JDBC (Java Database Connectivity), and ADO.NET drivers/providers.</p> <table style="border-spacing: 10px; margin-left: auto; margin-right: auto; padding: 10px; width: 95%;"> <tr> <th style="text-align: center; width: 45%;">ODBC</th> <th style="text-align: center; width: 45%;">JDBC</th> </tr> <tr> <td style="text-align: center; width: 45%;"> <a href="https://s-media-cache-ak0.pinimg.com/originals/ef/b2/7e/efb27e54ef098c18a93aa8a074d5817d.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;" id="link-id0x1066f4c38"><img alt="Image" border="0" src="https://s-media-cache-ak0.pinimg.com/originals/ef/b2/7e/efb27e54ef098c18a93aa8a074d5817d.png" style="max-width: 400px; width: 100%" /></a> </td> <td style="text-align: center; width: 45%;"> <a href="https://s-media-cache-ak0.pinimg.com/originals/75/11/a8/7511a81c2e4ec244473cc514d997bd0a.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;" id="link-id0x10619cce8"><img alt="Image" border="0" src="https://s-media-cache-ak0.pinimg.com/originals/75/11/a8/7511a81c2e4ec244473cc514d997bd0a.png" style="max-width: 400px; width: 100%" /></a> </td> </tr> <tr> <th style="text-align: center; width: 45%;" colspan="2">ADO.NET</th> </tr> <tr> <td style="text-align: center; width: 45%;" colspan="2"> <a href="https://s-media-cache-ak0.pinimg.com/originals/70/67/69/706769ab0ff453e94c50f38c956beeab.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;" id="link-id0x108c5e078"><img alt="Image" border="0" src="https://s-media-cache-ak0.pinimg.com/originals/70/67/69/706769ab0ff453e94c50f38c956beeab.jpg" style="max-width: 400px; width: 100%" /></a></td> </tr> </table> <h3>Why do we pay for anything?</h3> <p>After allowing for consumer irrationality [<a href="http://www.theatlantic.com/business/archive/2013/01/the-irrational-consumer-why-economics-is-dead-wrong-about-how-we-make-choices/267255/" id="link-id0x10711d0f8">1</a>], the basis of any payment is fundamentally tied to the monetization of opportunity costs. Essentially, we pay for one thing to alleviate the (usually higher) costs of something else.</p> <p>The rest of this post focuses on highlighting the real pains associated with the $0.00 value misconception associated with Data Access Drivers: ODBC, JDBC, ADO.NET, OLE-DB, etc.</p> <h2>Real Costs of Data Access Drivers</h2> <p>In the most basic sense, there are some fundamental aspects of data access that are complex to implement and rarely implemented (if at all) by free drivers. The list includes:</p> <ul> <li> <b>Escape Syntaxes for Dates and Functions</b> -- abstraction for dates and function signatures at the application level (i.e., the same abstraction works across all compliant databases)</li> <li> <b>Metadata Calls which enable smarter ODBC compliant applications</b> -- this feature is typically missing in Drivers and abused by the Clients (consumers); i.e., clients are made DBMS specific by testing for specific DBMS names, rather than utilizing feature and functionality metadata returned by drivers</li> <li> <b>Scrollable Cursors</b> -- this is how you deal with change sensitivity and paging through large datasets; some drivers actually fake support and get away with it due to shortage of applications to test proper cursor types (Static, Forward-Only, Key-Set, Dynamic, and Mixed models).</li> </ul> <p>Beyond actual driver sophistication, in regards to key feature implementations, let's up the ante by veering into the area of data security. At the most basic level, It's extremely important to understand that all data access drivers provide read-write access to your databases; thus, it's imperative that data access drivers address the following:</p> <ul> <li>Read-Only or Read-Write Access modalities scoped to specific <ul> <li>users</li> <li>user groups</li> <li>target databases</li> <li>data access standards (e.g., ODBC, JDBC, ADO.NET) </li> <li>client applications</li> <li>client host operating systems</li> <li>client host IP addresses</li> </ul> </li> <li>Any combination of the above as part of a configurable collection of data access rules (or policies).</li> </ul> <p>Once you're done with security, you then have the thorny issue of data access and data flow management. In a nutshell, your driver needs to be able to handle:</p> <ul> <li> <i>Protection against Cartesian-product-based network flooding</i> (e.g., user queries an enormous table without knowing or understanding back-end implications)</li> <li> <i>Enabling or Disabling of key DBMS engine data access optimization features</i> (e.g. SQL RDBMS-specific extensions exposed via Environment Variables or SQL-command-based settings)</li> <li> <i>Conditional Connection Pooling</i> across various dimensions such as Users, User Groups, Applications, Host Operating Systems, IP Addresses</li> <li> <i>Encryption of Data</i> in Transit</li> </ul> <p>Once you've dealt with Security and Data Flow, you then have to address the enforcement of these settings across a myriad of ODBC compliant host, which is where Zeroconfig and centralized data access administration comes into play i.e., configure once (locally) and enforce globally.</p> <h2>OpenLink Universal Data Access (UDA) Drivers </h2> <p>When OpenLink Software entered the ODBC Driver Market segment (circa 1992), the issues above were the fundamental basis of our Multi-Tier Drivers. Although the marketplace highlighted our drivers for high performance, stability, and specification adherence -- to all of which we remain committed -- our fundamental engineering focus has always been skewed towards configurable data security, platform independence, and scalability.</p> <p>Every item of concern outlined in the section above is addressed by security features built into our Multi-Tier Drivers [<a href="http://uda.openlinksw.com/odbc/mt/" id="link-id0x106b2c0c8">2</a>][<a href="http://uda.openlinksw.com/jdbc/mt/" id="link-id0x108454e28">3</a>][<a href="http://uda.openlinksw.com/dotnet/" id="link-id0x108454f28">4</a>]. These features all leverage the fact that our multi-tier drivers include a sophisticated DB session rules book that enables construction and enforcement of user attribute (user name, application, client operating system, IP address, target database etc.) based rules which are applied to all database sessions (single or pooled). </p> <table style="border-spacing: 10px; margin-left: auto; margin-right: auto; padding: 10px; width: 95%;"> <tr> <td style="text-align: center; width: 45%;"> <a href="https://s-media-cache-ak0.pinimg.com/originals/ed/95/14/ed9514178ecf89849402af1c48942412.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;" id="link-id0x107378c98"><img alt="Image" border="0" src="https://s-media-cache-ak0.pinimg.com/originals/ed/95/14/ed9514178ecf89849402af1c48942412.png" style="max-width: 400px; width: 100%" /></a> <div class="p2"></div></td> </tr> </table> <p>Today, in the year 2015, the security issues that pervade Data Access, whether via Native SQL RDBMS Drivers, or ODBC, JDBC, and ADO.NET Drivers/Providers, have only increased, courtesy of ubiquitous computing -- facilitated by the Internet & Web, across desktop and mobile device dimensions. Paradoxically, there remains a fundamental illusion that all Data Access Drivers are made the same; i.e., they simply provide you with the ability to connect to SQL RDBMS back-ends, for the industry standard price of $0.00, without consequence -- thereby skewing the very nature of SQL RDBMS data access and its security and privacy implications. </p> <p>I hope that this post brings some clarity to a very serious security and general configuration management issues associated with Data Access Drivers. Free ODBC Drivers offer nothing; that's why they cost $0.00. When dealing with real issues associated with Open Data Access, you must have a handle on the inevitable issues of data security and privacy. </p> <h2>Links</h2> <ol> <li> <a href="http://www.theatlantic.com/business/archive/2013/01/the-irrational-consumer-why-economics-is-dead-wrong-about-how-we-make-choices/267255/" id="link-id0x1082b54b8">The Irrational Consumer: Why Economics Is Dead Wrong About How We Make Choices</a> </li> <li> <a href="http://uda.openlinksw.com/odbc/mt/" id="link-id0x1082b55b8">OpenLink Multi-Tier ODBC Drivers</a> </li> <li> <a href="http://uda.openlinksw.com/jdbc/mt/" id="link-id0x1067e1e68">OpenLink Multi-Tier JDBC Drivers</a> </li> <li> <a href="http://uda.openlinksw.com/dotnet/mt/" id="link-id0x1067e1fd8">OpenLink Multi-Tier ADO.NET Providers</a> </li> </ol> <h2>Related</h2> <ul> <li> <a href="http://www.dba-oracle.com/t_auditing4_horror_stories_inside_jobs.htm" id="link-id0x107044328"><span class="s1">Oracle Security Auditing Horror Stories</span> </a><span class="s1"> -- showcases social dimension of security that exploits literal identifiers used by SQL RDBMS products in regards to user identity</span> </li> <li> <a href="http://searchoracle.techtarget.com/news/2240015728/Database-security-strategies-need-to-grow-up-in-2010?" id="link-id0x1062ad718">Database Security Strategies Need to Grow Up in 2010</a> </li> <li> <a href="http://walkingoncoals.blogspot.com/2009/12/whos-data-is-it-part-1.html" id="link-id0x10772cc78"><span class="s1">Whose Data is it? Part 1</span> </a> </li> <li> <span class="s1"><a href="http://walkingoncoals.blogspot.com/2009/12/whos-data-is-it-part-2.html" id="link-id0x10876fc48">Whose Data is it? Part 2</a> </span> </li> </ul>
2015-07-17T14:00:52.056449-04:00
In Hoc Signo Vinces (part 21 of n): Running TPC-H on Virtuoso Elastic Cluster on Amazon EC2
http://www.openlinksw.com:443/blog/vdb/blog/?date=2015-06-10#1846
2015-06-10T16:04:52Z
<p>We have made an Amazon EC2 deployment of <a href="http://virtuoso.openlinksw.com/features-comparison-matrix/#cluster" id="link-id0x2aabd14557d8">Virtuoso 7 Commercial Edition, configured to use the Elastic Cluster Module</a> with TPC-H preconfigured, similar to the <a href="http://www.openlinksw.com/weblogs/oerling/?id=1843" id="link-id0x2aabd235a378">recently published OpenLink Virtuoso Benchmark AMI</a> running the Open Source Edition. The details of the new Elastic Cluster AMI and steps to use it will be published in a forthcoming post. Here we will simply look at results of running TPC-H 100G scale on two machines, and 1000G scale on four machines. This shows how Virtuoso provides great performance on a cloud platform. The extremely fast bulk load — 33 minutes for a terabyte! — means that you can get straight to work even with on-demand infrastructure.</p> <p>In the following, the Amazon instance type is R3.8xlarge, each with dual Xeon E5-2670 v2, 244G RAM, and 2 x 300G SSD. The image is made from the Amazon Linux with built-in network optimization. We first tried a RedHat image without network optimization and had considerable trouble with the interconnect. Using network-optimized Amazon Linux images inside a virtual private cloud has resolved all these problems.</p> <p>The network optimized 10GE interconnect at Amazon offers throughput close to the QDR InfiniBand running TCP-IP; thus the Amazon platform is suitable for running cluster databases. The execution that we have seen is not seriously network bound.</p> <h3>100G on 2 machines, with a total of 32 cores, 64 threads, 488 GB RAM, 4 x 300 GB SSD</h3> <b>Load time:</b> 3m 52s <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Run</th> <th>Power</th> <th>Throughput</th> <th>Composite</th> </tr> <tr> <th style="text-align:center;"><code>1</code> </th> <td style="text-align:right;"><code>523,554.3</code> </td> <td style="text-align:right;"><code>590,692.6</code> </td> <td style="text-align:right;"><code>556,111.2</code> </td> </tr> <tr> <th style="text-align:center;"><code>2</code> </th> <td style="text-align:right;"><code>565,353.3</code> </td> <td style="text-align:right;"><code>642,503.0</code> </td> <td style="text-align:right;"><code>602,694.9</code> </td> </tr> </table> <h3>1000G on 4 machines, with a total of 64 cores, 128 threads, 976 GB RAM, 8 x 300 GB SSD</h3> <b>Load time:</b> 32m 47s <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Run</th> <th>Power</th> <th>Throughput</th> <th>Composite</th> </tr> <tr> <th style="text-align:center;"><code>1</code> </th> <td style="text-align:right;"><code>592,013.9</code> </td> <td style="text-align:right;"><code>754,107.6</code> </td> <td style="text-align:right;"><code>668,163.3</code> </td> </tr> <tr> <th style="text-align:center;"><code>2</code> </th> <td style="text-align:right;"><code>896,564.1</code> </td> <td style="text-align:right;"><code>828,265.4</code> </td> <td style="text-align:right;"><code>861,738.4</code> </td> </tr> <tr> <th style="text-align:center;"><code>3</code> </th> <td style="text-align:right;"><code>883,736.9</code> </td> <td style="text-align:right;"><code>829,609.0</code> </td> <td style="text-align:right;"><code>856,245.3</code> </td> </tr> </table> <p>For the larger scale we did 3 sets of power + throughput tests to measure consistency of performance. By the TPC-H rules, the worst (first) score should be reported. Even after bulk load, this is markedly less than the next power score due to working set effects. This is seen to a lesser degree with the first throughput score also.</p> <p>The numerical quantities summaries are available <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150610TPCHonClusterAMI/report.zip" id="link-id0x2aabafe59058">in a report.zip file</a>, or individually --</p> <ul> <li> <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150610TPCHonClusterAMI/report-100-1.txt" style="wikiautogen" id="link-id0x2aac0d449ff8">report-100-1.txt</a> </code> </li> <li> <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150610TPCHonClusterAMI/report-100-2.txt" style="wikiautogen" id="link-id0x2aab628ae658">report-100-2.txt</a> </code> </li> <li> <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150610TPCHonClusterAMI/report-1000-1.txt" style="wikiautogen" id="link-id0x2aab637b1078">report-1000-1.txt</a> </code> </li> <li> <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150610TPCHonClusterAMI/report-1000-2.txt" style="wikiautogen" id="link-id0x2aac0db30728">report-1000-2.txt</a> </code> </li> <li> <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150610TPCHonClusterAMI/report-1000-3.txt" style="wikiautogen" id="link-id0x2aac0d462b08">report-1000-3.txt</a> </code> </li> </ul> <p>Subsequent posts will explain how to deploy Virtuoso Elastic Clusters on AWS.</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aab52abc0b8"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aab53935178"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aab537ea348"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aab516a28b8"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aab514b53d8"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aaba8e12aa8"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aab537ea308"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aaba97e18d8"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aab52a6add8"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aaba9725b78"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aab51109e68"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aaba97e1898"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aab51a706a8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aaba97e18b8"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aaba9725b98"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x2aab52abc098"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aab5130e068"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aab52979f88"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aab53015408"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aab53015408"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2<i> (this post)</i> </li> </ul>
2015-06-10T12:49:08.293219-04:00
In Hoc Signo Vinces (part 21 of n): Running TPC-H on Virtuoso Elastic Cluster on Amazon EC2
http://www.openlinksw.com:443/weblog/oerling/?date=2015-06-10#1845
2015-06-10T16:03:13Z
<p>We have made an Amazon EC2 deployment of <a href="http://virtuoso.openlinksw.com/features-comparison-matrix/#cluster" id="link-id0x2aabd14557d8">Virtuoso 7 Commercial Edition, configured to use the Elastic Cluster Module</a> with TPC-H preconfigured, similar to the <a href="http://www.openlinksw.com/weblogs/oerling/?id=1843" id="link-id0x2aabd235a378">recently published OpenLink Virtuoso Benchmark AMI</a> running the Open Source Edition. The details of the new Elastic Cluster AMI and steps to use it will be published in a forthcoming post. Here we will simply look at results of running TPC-H 100G scale on two machines, and 1000G scale on four machines. This shows how Virtuoso provides great performance on a cloud platform. The extremely fast bulk load — 33 minutes for a terabyte! — means that you can get straight to work even with on-demand infrastructure.</p> <p>In the following, the Amazon instance type is R3.8xlarge, each with dual Xeon E5-2670 v2, 244G RAM, and 2 x 300G SSD. The image is made from the Amazon Linux with built-in network optimization. We first tried a RedHat image without network optimization and had considerable trouble with the interconnect. Using network-optimized Amazon Linux images inside a virtual private cloud has resolved all these problems.</p> <p>The network optimized 10GE interconnect at Amazon offers throughput close to the QDR InfiniBand running TCP-IP; thus the Amazon platform is suitable for running cluster databases. The execution that we have seen is not seriously network bound.</p> <h3>100G on 2 machines, with a total of 32 cores, 64 threads, 488 GB RAM, 4 x 300 GB SSD</h3> <b>Load time:</b> 3m 52s <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Run</th> <th>Power</th> <th>Throughput</th> <th>Composite</th> </tr> <tr> <th style="text-align:center;"><code>1</code> </th> <td style="text-align:right;"><code>523,554.3</code> </td> <td style="text-align:right;"><code>590,692.6</code> </td> <td style="text-align:right;"><code>556,111.2</code> </td> </tr> <tr> <th style="text-align:center;"><code>2</code> </th> <td style="text-align:right;"><code>565,353.3</code> </td> <td style="text-align:right;"><code>642,503.0</code> </td> <td style="text-align:right;"><code>602,694.9</code> </td> </tr> </table> <h3>1000G on 4 machines, with a total of 64 cores, 128 threads, 976 GB RAM, 8 x 300 GB SSD</h3> <b>Load time:</b> 32m 47s <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>Run</th> <th>Power</th> <th>Throughput</th> <th>Composite</th> </tr> <tr> <th style="text-align:center;"><code>1</code> </th> <td style="text-align:right;"><code>592,013.9</code> </td> <td style="text-align:right;"><code>754,107.6</code> </td> <td style="text-align:right;"><code>668,163.3</code> </td> </tr> <tr> <th style="text-align:center;"><code>2</code> </th> <td style="text-align:right;"><code>896,564.1</code> </td> <td style="text-align:right;"><code>828,265.4</code> </td> <td style="text-align:right;"><code>861,738.4</code> </td> </tr> <tr> <th style="text-align:center;"><code>3</code> </th> <td style="text-align:right;"><code>883,736.9</code> </td> <td style="text-align:right;"><code>829,609.0</code> </td> <td style="text-align:right;"><code>856,245.3</code> </td> </tr> </table> <p>For the larger scale we did 3 sets of power + throughput tests to measure consistency of performance. By the TPC-H rules, the worst (first) score should be reported. Even after bulk load, this is markedly less than the next power score due to working set effects. This is seen to a lesser degree with the first throughput score also.</p> <p>The numerical quantities summaries are available <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150610TPCHonClusterAMI/report.zip" id="link-id0x2aabafe59058">in a report.zip file</a>, or individually --</p> <ul> <li> <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150610TPCHonClusterAMI/report-100-1.txt" style="wikiautogen" id="link-id0x2aab623aa418">report-100-1.txt</a> </code> </li> <li> <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150610TPCHonClusterAMI/report-100-2.txt" style="wikiautogen" id="link-id0x2aac0dc9e6d8">report-100-2.txt</a> </code> </li> <li> <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150610TPCHonClusterAMI/report-1000-1.txt" style="wikiautogen" id="link-id0x2aab63598398">report-1000-1.txt</a> </code> </li> <li> <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150610TPCHonClusterAMI/report-1000-2.txt" style="wikiautogen" id="link-id0x2aac0c70f3a8">report-1000-2.txt</a> </code> </li> <li> <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20150610TPCHonClusterAMI/report-1000-3.txt" style="wikiautogen" id="link-id0x2aac0d939808">report-1000-3.txt</a> </code> </li> </ul> <p>Subsequent posts will explain how to deploy Virtuoso Elastic Clusters on AWS.</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aab52abc0b8"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aab53935178"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aab537ea348"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aab516a28b8"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aab514b53d8"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aaba8e12aa8"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aab537ea308"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aaba97e18d8"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aab52a6add8"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aaba9725b78"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aab51109e68"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aaba97e1898"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aab51a706a8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aaba97e18b8"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aaba9725b98"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x2aab52abc098"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aab5130e068"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aab52979f88"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aab53015408"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aab53015408"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2<i> (this post)</i> </li> </ul>
2015-06-10T12:49:39.451863-04:00
Introducing the OpenLink Virtuoso Benchmarks AMI on Amazon EC2
http://www.openlinksw.com:443/blog/vdb/blog/?date=2015-06-09#1844
2015-06-09T15:51:35Z
<p>The <b>OpenLink Virtuoso Benchmarks</b> AMI is an Amazon EC2 machine image with the latest Virtuoso open source technology preconfigured to run —</p> <ul> <li> <p> <i><a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0xf6c2bae8">TPC-H</a> </i>, the classic of SQL data warehousing</p> </li> <li> <p> <i><a href="http://www.openlinksw.com/weblog/oerling/?id=1834" id="link-id0xfdfb2aa8">LDBC SNB</a>,</i> the new Social Network Benchmark from the Linked Data Benchmark Council</p> </li> <li> <p> <i><a href="http://www.openlinksw.com/weblog/oerling/?id=1831" id="link-id0x104b01368">LDBC SPB</a>,</i> the RDF/SPARQL Semantic Publishing Benchmark from LDBC</p> </li> </ul> <p>This package is ideal for technology evaluators and developers interested in getting the most performance out of Virtuoso. This is also an all-in-one solution to any questions about reproducing claimed benchmark results. All necessary tools for building and running are included; thus any developer can use this model installation as a starting point. The benchmark drivers are preconfigured with appropriate settings, and benchmark qualification tests can be run with a single command.</p> <p>The Benchmarks AMI includes a precompiled, preconfigured checkout of the <a href="https://github.com/v7fasttrack/virtuoso-opensource/" id="link-id0x1051c7358">v7fasttrack github repository</a>, checkouts of the github repositories of the benchmarks, and a number of running directories with all configuration files preset and optimized. The image is intended to be instantiated on a R3.8xlarge Amazon instance with 244G RAM, dual Xeon E5-2670 v2, and 600G SSD.</p> <p>Benchmark datasets and preloaded database files can be downloaded from S3 when large, and generated as needed on the instance when small. As an alternative, the instance is also set up to do all phases of data generation and database bulk load.</p> <p>The following benchmark setups are included:</p> <ul> <li>TPC-H 100G</li> <li>TPC-H 300G</li> <li>LDBC SNB Validation </li> <li>LDBC SNB Interactive 100G</li> <li>LDBC SNB Interactive 300G (SF3)</li> <li>LDBC SPB Validation </li> <li>LDBC SPB Basic 256 Mtriples (SF5)</li> <li>LDBC SPB Basic 1 Gtriple</li> </ul> <p>The AMI will be expanded as new benchmarks are introduced, for example, the <i>LDBC Social Network Business Intelligence</i> or <i>Graph Analytics.</i> </p> <p>To get started: </p> <ol> <li> <p>Instantiate machine image <b>ami-eb789280</b> (AMI ID is subject to change; you should be able to find the latest by searching for "OpenLink Virtuoso Benchmarks" in "Community AMIs"; this one is short-named <code>virtuoso-bench-6</code>) with a R3.8xlarge instance. </p> </li> <li> <p>Connect via <code>ssh</code>.</p> </li> <li> <p>See the <b><a href="https://github.com/v7fasttrack/virtuoso-opensource/blob/feature/analytics/binsrc/samples/ldami/ec2-user/README" id="link-id0xfca2bab8">README</a></b> (also found in the <b><code>ec2-user</code></b>'s home directory) for full instructions on getting up and running.</p> </li> </ol>
2015-06-18T14:56:20.063339-04:00
Introducing the OpenLink Virtuoso Benchmarks AMI on Amazon EC2
http://www.openlinksw.com:443/weblog/oerling/?date=2015-06-09#1843
2015-06-09T15:51:22Z
<p>The <b>OpenLink Virtuoso Benchmarks</b> AMI is an Amazon EC2 machine image with the latest Virtuoso open source technology preconfigured to run —</p> <ul> <li> <p> <i><a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0xf6c2bae8">TPC-H</a> </i>, the classic of SQL data warehousing</p> </li> <li> <p> <i><a href="http://www.openlinksw.com/weblog/oerling/?id=1834" id="link-id0xfdfb2aa8">LDBC SNB</a>,</i> the new Social Network Benchmark from the Linked Data Benchmark Council</p> </li> <li> <p> <i><a href="http://www.openlinksw.com/weblog/oerling/?id=1831" id="link-id0x104b01368">LDBC SPB</a>,</i> the RDF/SPARQL Semantic Publishing Benchmark from LDBC</p> </li> </ul> <p>This package is ideal for technology evaluators and developers interested in getting the most performance out of Virtuoso. This is also an all-in-one solution to any questions about reproducing claimed benchmark results. All necessary tools for building and running are included; thus any developer can use this model installation as a starting point. The benchmark drivers are preconfigured with appropriate settings, and benchmark qualification tests can be run with a single command.</p> <p>The Benchmarks AMI includes a precompiled, preconfigured checkout of the <a href="https://github.com/v7fasttrack/virtuoso-opensource/" id="link-id0x1051c7358">v7fasttrack github repository</a>, checkouts of the github repositories of the benchmarks, and a number of running directories with all configuration files preset and optimized. The image is intended to be instantiated on a R3.8xlarge Amazon instance with 244G RAM, dual Xeon E5-2670 v2, and 600G SSD.</p> <p>Benchmark datasets and preloaded database files can be downloaded from S3 when large, and generated as needed on the instance when small. As an alternative, the instance is also set up to do all phases of data generation and database bulk load.</p> <p>The following benchmark setups are included:</p> <ul> <li>TPC-H 100G</li> <li>TPC-H 300G</li> <li>LDBC SNB Validation </li> <li>LDBC SNB Interactive 100G</li> <li>LDBC SNB Interactive 300G (SF3)</li> <li>LDBC SPB Validation </li> <li>LDBC SPB Basic 256 Mtriples (SF5)</li> <li>LDBC SPB Basic 1 Gtriple</li> </ul> <p>The AMI will be expanded as new benchmarks are introduced, for example, the <i>LDBC Social Network Business Intelligence</i> or <i>Graph Analytics.</i> </p> <p>To get started: </p> <ol> <li> <p>Instantiate machine image <b>ami-eb789280</b> (AMI ID is subject to change; you should be able to find the latest by searching for "OpenLink Virtuoso Benchmarks" in "Community AMIs"; this one is short-named <code>virtuoso-bench-6</code>) with a R3.8xlarge instance. </p> </li> <li> <p>Connect via <code>ssh</code>.</p> </li> <li> <p>See the <b><a href="https://github.com/v7fasttrack/virtuoso-opensource/blob/feature/analytics/binsrc/samples/ldami/ec2-user/README" id="link-id0xfca2bab8">README</a></b> (also found in the <b><code>ec2-user</code></b>'s home directory) for full instructions on getting up and running.</p> </li> </ol>
2015-06-18T14:55:56.642858-04:00
SNB Interactive, Part 3: Choke Points and Initial Run on Virtuoso
http://www.openlinksw.com:443/blog/vdb/blog/?date=2015-06-09#1842
2015-06-09T15:35:27Z
<p>In this post we will look at running the <a href="http://ldbcouncil.org/developer/snb" id="link-id0x2aab62c37078">LDBC SNB</a> on <a href="http://virtuoso.openlinksw.com/" id="link-id0x2aab629da5c8">Virtuoso</a>.</p> <p>First, let's recap what the benchmark is about: </p> <ol> <li>fairly frequent short updates, with no update contention worth mentioning</li> <li>short random lookups</li> <li>medium complex queries centered around a person's social environment</li> </ol> <p>The updates exist so as to invalidate strategies that rely too heavily on precomputation. The short lookups exist for the sake of realism; after all, an online social application does lookups for the most part. The medium complex queries are to challenge the DBMS.</p> <p>The DBMS challenges have to do firstly with query optimization, and secondly with execution with a lot of non-local random access patterns. Query optimization is not a requirement, <i>per se,</i> since imperative implementations are allowed, but we will see that these are no more free of the laws of nature than the declarative ones.</p> <p>The workload is arbitrarily parallel, so intra-query parallelization is not particularly useful, if also not harmful. There are latency constraints on operations which strongly encourage implementations to stay within a predictable time envelope regardless of specific query parameters. The parameters are a combination of person and date range, and sometimes tags or countries. The hardest queries have the potential to access all content created by people within 2 steps of a central person, so possibly thousands of people, times 2000 posts per person, times up to 4 tags per post. We are talking in the millions of key lookups, aiming for sub-second single-threaded execution.</p> <p>The test system is the same as used in the <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aab62e7f368">TPC-H series</a>: dual Xeon E5-2630, 2x6 cores x 2 threads, 2.3GHz, 192 GB RAM. The software is the <a href="https://github.com/v7fasttrack/virtuoso-opensource/tree/feature/analytics" id="link-id0x2aab62bf6de8">feature/analytics branch</a> of <a href="https://github.com/v7fasttrack/virtuoso-opensource/" id="link-id0x2aab623b1358">v7fasttrack, available from www.github.com</a>.</p> <p>The dataset is the SNB 300G set, with:</p> <blockquote> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <td style="text-align:right;"><code>1,136,127</code> </td> <td> <b><code>persons</code> </b> </td> </tr> <tr> <td style="text-align:right;"><code>125,249,604</code> </td> <td> <b><code>knows</code> </b> edges</td> </tr> <tr> <td style="text-align:right;"><code>847,886,644</code> </td> <td> <b><code>posts</code> </b>, including replies</td> </tr> <tr> <td style="text-align:right;"><code>1,145,893,841</code> </td> <td> <b><code>tags</code> </b> of posts or replies</td> </tr> <tr> <td style="text-align:right;"><code>1,140,226,235</code> </td> <td> <b><code>likes</code> </b> of posts or replies</td> </tr> </table> </blockquote> <p>As an initial step, we run the benchmark as fast as it will go. We use 32 threads on the driver side for 24 hardware threads.</p> <p>Below are the numerical quantities for a 400K operation run after 150K operations worth of warmup.</p> <blockquote> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <td style="text-align:right;"><b>Duration:</b> </td> <td><code>10:41.251</code> </td> </tr> <tr> <td style="text-align:right;"><b>Throughput:</b> </td> <td><code>623.71 (op/s)</code> </td> </tr> </table> </blockquote> <p>The statistics that matter are detailed below, with operations ranked in order of descending client-side wait-time. All times are in milliseconds.</p> <blockquote> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>% of total</th> <th>total_wait</th> <th>name</th> <th>count</th> <th>mean</th> <th>min</th> <th>max</th> </tr> <tr> <td style="text-align:right;"><code>20 %</code> </td> <td style="text-align:right;"><code>4,231,130</code> </td> <td><code>LdbcQuery5 </code> </td> <td style="text-align:right;"><code> 656</code> </td> <td style="text-align:right;"><code>6,449.89 </code> </td> <td style="text-align:right;"><code> 245</code> </td> <td style="text-align:right;"><code>10,311</code> </td> </tr> <tr> <td style="text-align:right;"><code>11 %</code> </td> <td style="text-align:right;"><code>2,272,954</code> </td> <td><code>LdbcQuery8 </code> </td> <td style="text-align:right;"><code>18,354</code> </td> <td style="text-align:right;"><code> 123.84 </code> </td> <td style="text-align:right;"><code> 14</code> </td> <td style="text-align:right;"><code> 2,240</code> </td> </tr> <tr> <td style="text-align:right;"><code>10 %</code> </td> <td style="text-align:right;"><code>2,200,718</code> </td> <td><code>LdbcQuery3 </code> </td> <td style="text-align:right;"><code> 388</code> </td> <td style="text-align:right;"><code>5,671.95 </code> </td> <td style="text-align:right;"><code> 468</code> </td> <td style="text-align:right;"><code>17,368</code> </td> </tr> <tr> <td style="text-align:right;"><code> 7.3 %</code> </td> <td style="text-align:right;"><code>1,561,382</code> </td> <td><code>LdbcQuery14 </code> </td> <td style="text-align:right;"><code> 1,124</code> </td> <td style="text-align:right;"><code>1,389.13 </code> </td> <td style="text-align:right;"><code> 4</code> </td> <td style="text-align:right;"><code> 5,724</code> </td> </tr> <tr> <td style="text-align:right;"><code> 6.7 %</code> </td> <td style="text-align:right;"><code>1,441,575</code> </td> <td><code>LdbcQuery12 </code> </td> <td style="text-align:right;"><code> 1,252</code> </td> <td style="text-align:right;"><code>1,151.42 </code> </td> <td style="text-align:right;"><code> 15</code> </td> <td style="text-align:right;"><code> 3,273</code> </td> </tr> <tr> <td style="text-align:right;"><code> 6.5 %</code> </td> <td style="text-align:right;"><code>1,396,932</code> </td> <td><code>LdbcQuery10 </code> </td> <td style="text-align:right;"><code> 1,252</code> </td> <td style="text-align:right;"><code>1,115.76 </code> </td> <td style="text-align:right;"><code> 13</code> </td> <td style="text-align:right;"><code> 4,743</code> </td> </tr> <tr> <td style="text-align:right;"><code> 5 %</code> </td> <td style="text-align:right;"><code>1,064,457</code> </td> <td><code>LdbcShortQuery3PersonFriends </code> </td> <td style="text-align:right;"><code>46,285</code> </td> <td style="text-align:right;"><code> 22.9979 </code> </td> <td style="text-align:right;"><code> 0</code> </td> <td style="text-align:right;"><code> 2,287</code> </td> </tr> <tr> <td style="text-align:right;"><code> 4.9 %</code> </td> <td style="text-align:right;"><code>1,047,536</code> </td> <td><code>LdbcShortQuery2PersonPosts </code> </td> <td style="text-align:right;"><code>46,285</code> </td> <td style="text-align:right;"><code> 22.6323 </code> </td> <td style="text-align:right;"><code> 0</code> </td> <td style="text-align:right;"><code> 2,156</code> </td> </tr> <tr> <td style="text-align:right;"><code> 4.1 %</code> </td> <td style="text-align:right;"><code> 885,102</code> </td> <td><code>LdbcQuery6 </code> </td> <td style="text-align:right;"><code> 1,721</code> </td> <td style="text-align:right;"><code> 514.295 </code> </td> <td style="text-align:right;"><code> 8</code> </td> <td style="text-align:right;"><code> 5,227</code> </td> </tr> <tr> <td style="text-align:right;"><code> 3.3 %</code> </td> <td style="text-align:right;"><code> 707,901</code> </td> <td><code>LdbcQuery1 </code> </td> <td style="text-align:right;"><code> 2,117</code> </td> <td style="text-align:right;"><code> 334.389 </code> </td> <td style="text-align:right;"><code> 28</code> </td> <td style="text-align:right;"><code> 3,467</code> </td> </tr> <tr> <td style="text-align:right;"><code> 2.4 %</code> </td> <td style="text-align:right;"><code> 521,738</code> </td> <td><code>LdbcQuery4 </code> </td> <td style="text-align:right;"><code> 1,530</code> </td> <td style="text-align:right;"><code> 341.005 </code> </td> <td style="text-align:right;"><code> 49</code> </td> <td style="text-align:right;"><code> 2,774</code> </td> </tr> <tr> <td style="text-align:right;"><code> 2.1 %</code> </td> <td style="text-align:right;"><code> 440,197</code> </td> <td><code>LdbcShortQuery4MessageContent </code> </td> <td style="text-align:right;"><code>46,302</code> </td> <td style="text-align:right;"><code> 9.50708</code> </td> <td style="text-align:right;"><code> 0</code> </td> <td style="text-align:right;"><code> 2,015</code> </td> </tr> <tr> <td style="text-align:right;"><code> 1.9 %</code> </td> <td style="text-align:right;"><code> 407,450</code> </td> <td><code>LdbcUpdate5AddForumMembership </code> </td> <td style="text-align:right;"><code>14,338</code> </td> <td style="text-align:right;"><code> 28.4175 </code> </td> <td style="text-align:right;"><code> 0</code> </td> <td style="text-align:right;"><code> 2,008</code> </td> </tr> <tr> <td style="text-align:right;"><code> 1.9 %</code> </td> <td style="text-align:right;"><code> 405,243</code> </td> <td><code>LdbcShortQuery7MessageReplies </code> </td> <td style="text-align:right;"><code>46,302</code> </td> <td style="text-align:right;"><code> 8.75217</code> </td> <td style="text-align:right;"><code> 0</code> </td> <td style="text-align:right;"><code> 2,112</code> </td> </tr> <tr> <td style="text-align:right;"><code> 1.9 %</code> </td> <td style="text-align:right;"><code> 404,002</code> </td> <td><code>LdbcShortQuery6MessageForum </code> </td> <td style="text-align:right;"><code>46,302</code> </td> <td style="text-align:right;"><code> 8.72537</code> </td> <td style="text-align:right;"><code> 0</code> </td> <td style="text-align:right;"><code> 1,968</code> </td> </tr> <tr> <td style="text-align:right;"><code> 1.8 %</code> </td> <td style="text-align:right;"><code> 387,044</code> </td> <td><code>LdbcUpdate3AddCommentLike </code> </td> <td style="text-align:right;"><code>12,659</code> </td> <td style="text-align:right;"><code> 30.5746 </code> </td> <td style="text-align:right;"><code> 0</code> </td> <td style="text-align:right;"><code> 2,060</code> </td> </tr> <tr> <td style="text-align:right;"><code> 1.7 %</code> </td> <td style="text-align:right;"><code> 361,290</code> </td> <td><code>LdbcShortQuery1PersonProfile </code> </td> <td style="text-align:right;"><code>46,285</code> </td> <td style="text-align:right;"><code> 7.80577</code> </td> <td style="text-align:right;"><code> 0</code> </td> <td style="text-align:right;"><code> 2,015</code> </td> </tr> <tr> <td style="text-align:right;"><code> 1.6 %</code> </td> <td style="text-align:right;"><code> 334,409</code> </td> <td><code>LdbcShortQuery5MessageCreator </code> </td> <td style="text-align:right;"><code>46,302</code> </td> <td style="text-align:right;"><code> 7.22234</code> </td> <td style="text-align:right;"><code> 0</code> </td> <td style="text-align:right;"><code> 2,055</code> </td> </tr> <tr> <td style="text-align:right;"><code> 1 %</code> </td> <td style="text-align:right;"><code> 220,740</code> </td> <td><code>LdbcQuery2 </code> </td> <td style="text-align:right;"><code> 1,488</code> </td> <td style="text-align:right;"><code> 148.347 </code> </td> <td style="text-align:right;"><code> 2</code> </td> <td style="text-align:right;"><code> 2,504</code> </td> </tr> <tr> <td style="text-align:right;"><code> 0.96 %</code> </td> <td style="text-align:right;"><code> 205,910</code> </td> <td><code>LdbcQuery7 </code> </td> <td style="text-align:right;"><code> 1,721</code> </td> <td style="text-align:right;"><code> 119.646 </code> </td> <td style="text-align:right;"><code> 11</code> </td> <td style="text-align:right;"><code> 2,295</code> </td> </tr> <tr> <td style="text-align:right;"><code> 0.93 %</code> </td> <td style="text-align:right;"><code> 198,971</code> </td> <td><code>LdbcUpdate2AddPostLike </code> </td> <td style="text-align:right;"><code> 5,974</code> </td> <td style="text-align:right;"><code> 33.3062 </code> </td> <td style="text-align:right;"><code> 0</code> </td> <td style="text-align:right;"><code> 1,987</code> </td> </tr> <tr> <td style="text-align:right;"><code> 0.88 %</code> </td> <td style="text-align:right;"><code> 189,871</code> </td> <td><code>LdbcQuery11 </code> </td> <td style="text-align:right;"><code> 2,294</code> </td> <td style="text-align:right;"><code> 82.7685 </code> </td> <td style="text-align:right;"><code> 4</code> </td> <td style="text-align:right;"><code> 2,219</code> </td> </tr> <tr> <td style="text-align:right;"><code> 0.85 %</code> </td> <td style="text-align:right;"><code> 182,964</code> </td> <td><code>LdbcQuery13 </code> </td> <td style="text-align:right;"><code> 2,898</code> </td> <td style="text-align:right;"><code> 63.1346 </code> </td> <td style="text-align:right;"><code> 1</code> </td> <td style="text-align:right;"><code> 2,201</code> </td> </tr> <tr> <td style="text-align:right;"><code> 0.74 %</code> </td> <td style="text-align:right;"><code> 158,188</code> </td> <td><code>LdbcQuery9 </code> </td> <td style="text-align:right;"><code> 78</code> </td> <td style="text-align:right;"><code>2,028.05 </code> </td> <td style="text-align:right;"><code>1,108</code> </td> <td style="text-align:right;"><code> 4,183</code> </td> </tr> <tr> <td style="text-align:right;"><code> 0.67 %</code> </td> <td style="text-align:right;"><code> 143,457</code> </td> <td><code>LdbcUpdate7AddComment </code> </td> <td style="text-align:right;"><code> 3,986</code> </td> <td style="text-align:right;"><code> 35.9902 </code> </td> <td style="text-align:right;"><code> 1</code> </td> <td style="text-align:right;"><code> 1,912</code> </td> </tr> <tr> <td style="text-align:right;"><code> 0.26 %</code> </td> <td style="text-align:right;"><code> 54,947</code> </td> <td><code>LdbcUpdate8AddFriendship </code> </td> <td style="text-align:right;"><code> 571</code> </td> <td style="text-align:right;"><code> 96.2294 </code> </td> <td style="text-align:right;"><code> 1</code> </td> <td style="text-align:right;"><code> 988</code> </td> </tr> <tr> <td style="text-align:right;"><code> 0.2 %</code> </td> <td style="text-align:right;"><code> 43,451</code> </td> <td><code>LdbcUpdate6AddPost </code> </td> <td style="text-align:right;"><code> 1,386</code> </td> <td style="text-align:right;"><code> 31.3499 </code> </td> <td style="text-align:right;"><code> 1</code> </td> <td style="text-align:right;"><code> 2,060</code> </td> </tr> <tr> <td style="text-align:right;"><code> 0.0086%</code> </td> <td style="text-align:right;"><code> 1,848</code> </td> <td><code>LdbcUpdate4AddForum </code> </td> <td style="text-align:right;"><code> 103</code> </td> <td style="text-align:right;"><code> 17.9417 </code> </td> <td style="text-align:right;"><code> 1</code> </td> <td style="text-align:right;"><code> 65</code> </td> </tr> <tr> <td style="text-align:right;"><code> 0.0002%</code> </td> <td style="text-align:right;"><code> 44</code> </td> <td><code>LdbcUpdate1AddPerson </code> </td> <td style="text-align:right;"><code> 2</code> </td> <td style="text-align:right;"><code> 22 </code> </td> <td style="text-align:right;"><code> 10</code> </td> <td style="text-align:right;"><code> 34</code> </td> </tr> </table> </blockquote> <p>At this point we have in-depth knowledge of the choke points the benchmark stresses, and we can give a first assessment of whether the design meets its objectives for setting an agenda for the coming years of graph database development.</p> <p>The implementation is well optimized in general but still has maybe 30% room for improvement. We note that this is based on a compressed column store. One could think that alternative data representations, like in-memory graphs of structs and pointers between them, are better for the task. This is not necessarily so; at the least, a compressed column store is much more space efficient. Space efficiency is the root of cost efficiency, since as soon as the working set is not in memory, a random access workload is badly hit.</p> <p>The set of choke points (technical challenges) actually revealed by the benchmark is so far as follows:</p> <ul> <li> <p> <b>Cardinality estimation under heavy data skew —</b> Many queries take a <code>tag</code> or a <code>country</code> as a parameter. The cardinalities associated with <code>tags</code> vary from 29M <code>posts</code> for the most common to 1 for the least common. Q6 has a common <code>tag</code> (in top few hundred) half the time and a random, most often very infrequent, one the rest of the time. A declarative implementation must recognize the cardinality implications from the literal and plan accordingly. An imperative one would have to count. Missing this makes Q6 take about 40% of the time instead of 4.1% when adapting.</p> </li> <li> <p> <b>Covering indices —</b> Being able to make multi-column indices that duplicate some columns from the table often saves an entire table lookup. For example, an index on <code>post</code> by <code>author</code> can also contain the <code>post</code>'s <code>creation date</code>.</p> </li> <li> <p> <b>Multi-hop graph traversal —</b> Most queries access a two-hop environment starting at a <code>person</code>. Two queries look for shortest paths of unbounded length. For the two-hop case, it makes almost no difference whether this is done as a union or a special graph traversal operator. For shortest paths, this simply must be built into the engine; doing this client-side incurs prohibitive overheads. A bidirectional shortest path operation is a requirement for the benchmark.</p> </li> <li> <p> <b>Top <i>K</i> —</b> Most queries returning <code>posts</code> order results by descending <code>date</code>. Once there are at least <i>k</i> results, anything older than the <i>k</i>th can be dropped, adding a <code>date</code> selection as early as possible in the query. This interacts with vectored execution, so that starting with a short vector size more rapidly produces an initial top <i>k</i>.</p> </li> <li> <p> <b>Late projection —</b> Many queries access several columns and touch millions of rows but only return a few. The columns that are not used in sorting or selection can be retrieved only for the rows that are actually returned. This is especially useful with a column store, as this removes many large columns (e.g., text of a <code>post</code>) from the working set.</p> </li> <li> <p> <b>Materialization —</b> Q14 accesses an expensive-to-compute edge weight, the number of <code>post-reply</code> pairs between two <code>people</code>. Keeping this precomputed drops Q14 from the top place. Other materialization would be possible, for example Q2 (top 20 <code>posts</code> by friends), but since Q2 is just 1% of the load, there is no need. One could of course argue that this should be 20x more frequent, in which case there could be a point to this.</p> </li> <li> <p> <b>Concurrency control —</b> Read-write contention is rare, as updates are randomly spread over the database. However, some pages get read very frequently, e.g., some middle level index pages in the <code>post</code> table. Keeping a count of reading threads requires a mutex, and there is significant contention on this. Since the hot set can be one page, adding more mutexes does not always help. However, hash partitioning the index into many independent trees (as in the case of a cluster) helps for this. There is also contention on a mutex for assigning threads to client requests, as there are large numbers of short operations.</p> </li> </ul> <p>In subsequent posts, we will look at specific queries, what they in fact do, and what their theoretical performance limits would be. In this way we will have a precise understanding of which way SNB can steer the graph DB community.</p> <h3> SNB Interactive Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1834" id="link-id0x2aab62c9ee38"> SNB Interactive, Part 1: What is SNB Interactive Really About?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1836" id="link-id0x2aab62e64818"> SNB Interactive, Part 2: Modeling Choices</a> </li> <li> SNB Interactive, Part 3: Choke Points and Initial Run on Virtuoso<i> (this post)</i> </li> </ul>
2015-06-09T11:35:27.248425-04:00
SNB Interactive, Part 3: Choke Points and Initial Run on Virtuoso
http://www.openlinksw.com:443/weblog/oerling/?date=2015-06-09#1841
2015-06-09T15:24:58Z
<p>In this post we will look at running the <a href="http://ldbcouncil.org/developer/snb" id="link-id0x2aab62c37078">LDBC SNB</a> on <a href="http://virtuoso.openlinksw.com/" id="link-id0x2aab629da5c8">Virtuoso</a>.</p> <p>First, let's recap what the benchmark is about: </p> <ol> <li>fairly frequent short updates, with no update contention worth mentioning</li> <li>short random lookups</li> <li>medium complex queries centered around a person's social environment</li> </ol> <p>The updates exist so as to invalidate strategies that rely too heavily on precomputation. The short lookups exist for the sake of realism; after all, an online social application does lookups for the most part. The medium complex queries are to challenge the DBMS.</p> <p>The DBMS challenges have to do firstly with query optimization, and secondly with execution with a lot of non-local random access patterns. Query optimization is not a requirement, <i>per se,</i> since imperative implementations are allowed, but we will see that these are no more free of the laws of nature than the declarative ones.</p> <p>The workload is arbitrarily parallel, so intra-query parallelization is not particularly useful, if also not harmful. There are latency constraints on operations which strongly encourage implementations to stay within a predictable time envelope regardless of specific query parameters. The parameters are a combination of person and date range, and sometimes tags or countries. The hardest queries have the potential to access all content created by people within 2 steps of a central person, so possibly thousands of people, times 2000 posts per person, times up to 4 tags per post. We are talking in the millions of key lookups, aiming for sub-second single-threaded execution.</p> <p>The test system is the same as used in the <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aab62e7f368">TPC-H series</a>: dual Xeon E5-2630, 2x6 cores x 2 threads, 2.3GHz, 192 GB RAM. The software is the <a href="https://github.com/v7fasttrack/virtuoso-opensource/tree/feature/analytics" id="link-id0x2aab62bf6de8">feature/analytics branch</a> of <a href="https://github.com/v7fasttrack/virtuoso-opensource/" id="link-id0x2aab623b1358">v7fasttrack, available from www.github.com</a>.</p> <p>The dataset is the SNB 300G set, with:</p> <blockquote> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <td style="text-align:right;"><code>1,136,127</code> </td> <td> <b><code>persons</code> </b> </td> </tr> <tr> <td style="text-align:right;"><code>125,249,604</code> </td> <td> <b><code>knows</code> </b> edges</td> </tr> <tr> <td style="text-align:right;"><code>847,886,644</code> </td> <td> <b><code>posts</code> </b>, including replies</td> </tr> <tr> <td style="text-align:right;"><code>1,145,893,841</code> </td> <td> <b><code>tags</code> </b> of posts or replies</td> </tr> <tr> <td style="text-align:right;"><code>1,140,226,235</code> </td> <td> <b><code>likes</code> </b> of posts or replies</td> </tr> </table> </blockquote> <p>As an initial step, we run the benchmark as fast as it will go. We use 32 threads on the driver side for 24 hardware threads.</p> <p>Below are the numerical quantities for a 400K operation run after 150K operations worth of warmup.</p> <blockquote> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <td style="text-align:right;"><b>Duration:</b> </td> <td><code>10:41.251</code> </td> </tr> <tr> <td style="text-align:right;"><b>Throughput:</b> </td> <td><code>623.71 (op/s)</code> </td> </tr> </table> </blockquote> <p>The statistics that matter are detailed below, with operations ranked in order of descending client-side wait-time. All times are in milliseconds.</p> <blockquote> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr style="text-align:center;"> <th>% of total</th> <th>total_wait</th> <th>name</th> <th>count</th> <th>mean</th> <th>min</th> <th>max</th> </tr> <tr> <td style="text-align:right;"><code>20 %</code> </td> <td style="text-align:right;"><code>4,231,130</code> </td> <td><code>LdbcQuery5 </code> </td> <td style="text-align:right;"><code> 656</code> </td> <td style="text-align:right;"><code>6,449.89 </code> </td> <td style="text-align:right;"><code> 245</code> </td> <td style="text-align:right;"><code>10,311</code> </td> </tr> <tr> <td style="text-align:right;"><code>11 %</code> </td> <td style="text-align:right;"><code>2,272,954</code> </td> <td><code>LdbcQuery8 </code> </td> <td style="text-align:right;"><code>18,354</code> </td> <td style="text-align:right;"><code> 123.84 </code> </td> <td style="text-align:right;"><code> 14</code> </td> <td style="text-align:right;"><code> 2,240</code> </td> </tr> <tr> <td style="text-align:right;"><code>10 %</code> </td> <td style="text-align:right;"><code>2,200,718</code> </td> <td><code>LdbcQuery3 </code> </td> <td style="text-align:right;"><code> 388</code> </td> <td style="text-align:right;"><code>5,671.95 </code> </td> <td style="text-align:right;"><code> 468</code> </td> <td style="text-align:right;"><code>17,368</code> </td> </tr> <tr> <td style="text-align:right;"><code> 7.3 %</code> </td> <td style="text-align:right;"><code>1,561,382</code> </td> <td><code>LdbcQuery14 </code> </td> <td style="text-align:right;"><code> 1,124</code> </td> <td style="text-align:right;"><code>1,389.13 </code> </td> <td style="text-align:right;"><code> 4</code> </td> <td style="text-align:right;"><code> 5,724</code> </td> </tr> <tr> <td style="text-align:right;"><code> 6.7 %</code> </td> <td style="text-align:right;"><code>1,441,575</code> </td> <td><code>LdbcQuery12 </code> </td> <td style="text-align:right;"><code> 1,252</code> </td> <td style="text-align:right;"><code>1,151.42 </code> </td> <td style="text-align:right;"><code> 15</code> </td> <td style="text-align:right;"><code> 3,273</code> </td> </tr> <tr> <td style="text-align:right;"><code> 6.5 %</code> </td> <td style="text-align:right;"><code>1,396,932</code> </td> <td><code>LdbcQuery10 </code> </td> <td style="text-align:right;"><code> 1,252</code> </td> <td style="text-align:right;"><code>1,115.76 </code> </td> <td style="text-align:right;"><code> 13</code> </td> <td style="text-align:right;"><code> 4,743</code> </td> </tr> <tr> <td style="text-align:right;"><code> 5 %</code> </td> <td style="text-align:right;"><code>1,064,457</code> </td> <td><code>LdbcShortQuery3PersonFriends </code> </td> <td style="text-align:right;"><code>46,285</code> </td> <td style="text-align:right;"><code> 22.9979 </code> </td> <td style="text-align:right;"><code> 0</code> </td> <td style="text-align:right;"><code> 2,287</code> </td> </tr> <tr> <td style="text-align:right;"><code> 4.9 %</code> </td> <td style="text-align:right;"><code>1,047,536</code> </td> <td><code>LdbcShortQuery2PersonPosts </code> </td> <td style="text-align:right;"><code>46,285</code> </td> <td style="text-align:right;"><code> 22.6323 </code> </td> <td style="text-align:right;"><code> 0</code> </td> <td style="text-align:right;"><code> 2,156</code> </td> </tr> <tr> <td style="text-align:right;"><code> 4.1 %</code> </td> <td style="text-align:right;"><code> 885,102</code> </td> <td><code>LdbcQuery6 </code> </td> <td style="text-align:right;"><code> 1,721</code> </td> <td style="text-align:right;"><code> 514.295 </code> </td> <td style="text-align:right;"><code> 8</code> </td> <td style="text-align:right;"><code> 5,227</code> </td> </tr> <tr> <td style="text-align:right;"><code> 3.3 %</code> </td> <td style="text-align:right;"><code> 707,901</code> </td> <td><code>LdbcQuery1 </code> </td> <td style="text-align:right;"><code> 2,117</code> </td> <td style="text-align:right;"><code> 334.389 </code> </td> <td style="text-align:right;"><code> 28</code> </td> <td style="text-align:right;"><code> 3,467</code> </td> </tr> <tr> <td style="text-align:right;"><code> 2.4 %</code> </td> <td style="text-align:right;"><code> 521,738</code> </td> <td><code>LdbcQuery4 </code> </td> <td style="text-align:right;"><code> 1,530</code> </td> <td style="text-align:right;"><code> 341.005 </code> </td> <td style="text-align:right;"><code> 49</code> </td> <td style="text-align:right;"><code> 2,774</code> </td> </tr> <tr> <td style="text-align:right;"><code> 2.1 %</code> </td> <td style="text-align:right;"><code> 440,197</code> </td> <td><code>LdbcShortQuery4MessageContent </code> </td> <td style="text-align:right;"><code>46,302</code> </td> <td style="text-align:right;"><code> 9.50708</code> </td> <td style="text-align:right;"><code> 0</code> </td> <td style="text-align:right;"><code> 2,015</code> </td> </tr> <tr> <td style="text-align:right;"><code> 1.9 %</code> </td> <td style="text-align:right;"><code> 407,450</code> </td> <td><code>LdbcUpdate5AddForumMembership </code> </td> <td style="text-align:right;"><code>14,338</code> </td> <td style="text-align:right;"><code> 28.4175 </code> </td> <td style="text-align:right;"><code> 0</code> </td> <td style="text-align:right;"><code> 2,008</code> </td> </tr> <tr> <td style="text-align:right;"><code> 1.9 %</code> </td> <td style="text-align:right;"><code> 405,243</code> </td> <td><code>LdbcShortQuery7MessageReplies </code> </td> <td style="text-align:right;"><code>46,302</code> </td> <td style="text-align:right;"><code> 8.75217</code> </td> <td style="text-align:right;"><code> 0</code> </td> <td style="text-align:right;"><code> 2,112</code> </td> </tr> <tr> <td style="text-align:right;"><code> 1.9 %</code> </td> <td style="text-align:right;"><code> 404,002</code> </td> <td><code>LdbcShortQuery6MessageForum </code> </td> <td style="text-align:right;"><code>46,302</code> </td> <td style="text-align:right;"><code> 8.72537</code> </td> <td style="text-align:right;"><code> 0</code> </td> <td style="text-align:right;"><code> 1,968</code> </td> </tr> <tr> <td style="text-align:right;"><code> 1.8 %</code> </td> <td style="text-align:right;"><code> 387,044</code> </td> <td><code>LdbcUpdate3AddCommentLike </code> </td> <td style="text-align:right;"><code>12,659</code> </td> <td style="text-align:right;"><code> 30.5746 </code> </td> <td style="text-align:right;"><code> 0</code> </td> <td style="text-align:right;"><code> 2,060</code> </td> </tr> <tr> <td style="text-align:right;"><code> 1.7 %</code> </td> <td style="text-align:right;"><code> 361,290</code> </td> <td><code>LdbcShortQuery1PersonProfile </code> </td> <td style="text-align:right;"><code>46,285</code> </td> <td style="text-align:right;"><code> 7.80577</code> </td> <td style="text-align:right;"><code> 0</code> </td> <td style="text-align:right;"><code> 2,015</code> </td> </tr> <tr> <td style="text-align:right;"><code> 1.6 %</code> </td> <td style="text-align:right;"><code> 334,409</code> </td> <td><code>LdbcShortQuery5MessageCreator </code> </td> <td style="text-align:right;"><code>46,302</code> </td> <td style="text-align:right;"><code> 7.22234</code> </td> <td style="text-align:right;"><code> 0</code> </td> <td style="text-align:right;"><code> 2,055</code> </td> </tr> <tr> <td style="text-align:right;"><code> 1 %</code> </td> <td style="text-align:right;"><code> 220,740</code> </td> <td><code>LdbcQuery2 </code> </td> <td style="text-align:right;"><code> 1,488</code> </td> <td style="text-align:right;"><code> 148.347 </code> </td> <td style="text-align:right;"><code> 2</code> </td> <td style="text-align:right;"><code> 2,504</code> </td> </tr> <tr> <td style="text-align:right;"><code> 0.96 %</code> </td> <td style="text-align:right;"><code> 205,910</code> </td> <td><code>LdbcQuery7 </code> </td> <td style="text-align:right;"><code> 1,721</code> </td> <td style="text-align:right;"><code> 119.646 </code> </td> <td style="text-align:right;"><code> 11</code> </td> <td style="text-align:right;"><code> 2,295</code> </td> </tr> <tr> <td style="text-align:right;"><code> 0.93 %</code> </td> <td style="text-align:right;"><code> 198,971</code> </td> <td><code>LdbcUpdate2AddPostLike </code> </td> <td style="text-align:right;"><code> 5,974</code> </td> <td style="text-align:right;"><code> 33.3062 </code> </td> <td style="text-align:right;"><code> 0</code> </td> <td style="text-align:right;"><code> 1,987</code> </td> </tr> <tr> <td style="text-align:right;"><code> 0.88 %</code> </td> <td style="text-align:right;"><code> 189,871</code> </td> <td><code>LdbcQuery11 </code> </td> <td style="text-align:right;"><code> 2,294</code> </td> <td style="text-align:right;"><code> 82.7685 </code> </td> <td style="text-align:right;"><code> 4</code> </td> <td style="text-align:right;"><code> 2,219</code> </td> </tr> <tr> <td style="text-align:right;"><code> 0.85 %</code> </td> <td style="text-align:right;"><code> 182,964</code> </td> <td><code>LdbcQuery13 </code> </td> <td style="text-align:right;"><code> 2,898</code> </td> <td style="text-align:right;"><code> 63.1346 </code> </td> <td style="text-align:right;"><code> 1</code> </td> <td style="text-align:right;"><code> 2,201</code> </td> </tr> <tr> <td style="text-align:right;"><code> 0.74 %</code> </td> <td style="text-align:right;"><code> 158,188</code> </td> <td><code>LdbcQuery9 </code> </td> <td style="text-align:right;"><code> 78</code> </td> <td style="text-align:right;"><code>2,028.05 </code> </td> <td style="text-align:right;"><code>1,108</code> </td> <td style="text-align:right;"><code> 4,183</code> </td> </tr> <tr> <td style="text-align:right;"><code> 0.67 %</code> </td> <td style="text-align:right;"><code> 143,457</code> </td> <td><code>LdbcUpdate7AddComment </code> </td> <td style="text-align:right;"><code> 3,986</code> </td> <td style="text-align:right;"><code> 35.9902 </code> </td> <td style="text-align:right;"><code> 1</code> </td> <td style="text-align:right;"><code> 1,912</code> </td> </tr> <tr> <td style="text-align:right;"><code> 0.26 %</code> </td> <td style="text-align:right;"><code> 54,947</code> </td> <td><code>LdbcUpdate8AddFriendship </code> </td> <td style="text-align:right;"><code> 571</code> </td> <td style="text-align:right;"><code> 96.2294 </code> </td> <td style="text-align:right;"><code> 1</code> </td> <td style="text-align:right;"><code> 988</code> </td> </tr> <tr> <td style="text-align:right;"><code> 0.2 %</code> </td> <td style="text-align:right;"><code> 43,451</code> </td> <td><code>LdbcUpdate6AddPost </code> </td> <td style="text-align:right;"><code> 1,386</code> </td> <td style="text-align:right;"><code> 31.3499 </code> </td> <td style="text-align:right;"><code> 1</code> </td> <td style="text-align:right;"><code> 2,060</code> </td> </tr> <tr> <td style="text-align:right;"><code> 0.0086%</code> </td> <td style="text-align:right;"><code> 1,848</code> </td> <td><code>LdbcUpdate4AddForum </code> </td> <td style="text-align:right;"><code> 103</code> </td> <td style="text-align:right;"><code> 17.9417 </code> </td> <td style="text-align:right;"><code> 1</code> </td> <td style="text-align:right;"><code> 65</code> </td> </tr> <tr> <td style="text-align:right;"><code> 0.0002%</code> </td> <td style="text-align:right;"><code> 44</code> </td> <td><code>LdbcUpdate1AddPerson </code> </td> <td style="text-align:right;"><code> 2</code> </td> <td style="text-align:right;"><code> 22 </code> </td> <td style="text-align:right;"><code> 10</code> </td> <td style="text-align:right;"><code> 34</code> </td> </tr> </table> </blockquote> <p>At this point we have in-depth knowledge of the choke points the benchmark stresses, and we can give a first assessment of whether the design meets its objectives for setting an agenda for the coming years of graph database development.</p> <p>The implementation is well optimized in general but still has maybe 30% room for improvement. We note that this is based on a compressed column store. One could think that alternative data representations, like in-memory graphs of structs and pointers between them, are better for the task. This is not necessarily so; at the least, a compressed column store is much more space efficient. Space efficiency is the root of cost efficiency, since as soon as the working set is not in memory, a random access workload is badly hit.</p> <p>The set of choke points (technical challenges) actually revealed by the benchmark is so far as follows:</p> <ul> <li> <p> <b>Cardinality estimation under heavy data skew —</b> Many queries take a <code>tag</code> or a <code>country</code> as a parameter. The cardinalities associated with <code>tags</code> vary from 29M <code>posts</code> for the most common to 1 for the least common. Q6 has a common <code>tag</code> (in top few hundred) half the time and a random, most often very infrequent, one the rest of the time. A declarative implementation must recognize the cardinality implications from the literal and plan accordingly. An imperative one would have to count. Missing this makes Q6 take about 40% of the time instead of 4.1% when adapting.</p> </li> <li> <p> <b>Covering indices —</b> Being able to make multi-column indices that duplicate some columns from the table often saves an entire table lookup. For example, an index on <code>post</code> by <code>author</code> can also contain the <code>post</code>'s <code>creation date</code>.</p> </li> <li> <p> <b>Multi-hop graph traversal —</b> Most queries access a two-hop environment starting at a <code>person</code>. Two queries look for shortest paths of unbounded length. For the two-hop case, it makes almost no difference whether this is done as a union or a special graph traversal operator. For shortest paths, this simply must be built into the engine; doing this client-side incurs prohibitive overheads. A bidirectional shortest path operation is a requirement for the benchmark.</p> </li> <li> <p> <b>Top <i>K</i> —</b> Most queries returning <code>posts</code> order results by descending <code>date</code>. Once there are at least <i>k</i> results, anything older than the <i>k</i>th can be dropped, adding a <code>date</code> selection as early as possible in the query. This interacts with vectored execution, so that starting with a short vector size more rapidly produces an initial top <i>k</i>.</p> </li> <li> <p> <b>Late projection —</b> Many queries access several columns and touch millions of rows but only return a few. The columns that are not used in sorting or selection can be retrieved only for the rows that are actually returned. This is especially useful with a column store, as this removes many large columns (e.g., text of a <code>post</code>) from the working set.</p> </li> <li> <p> <b>Materialization —</b> Q14 accesses an expensive-to-compute edge weight, the number of <code>post-reply</code> pairs between two <code>people</code>. Keeping this precomputed drops Q14 from the top place. Other materialization would be possible, for example Q2 (top 20 <code>posts</code> by friends), but since Q2 is just 1% of the load, there is no need. One could of course argue that this should be 20x more frequent, in which case there could be a point to this.</p> </li> <li> <p> <b>Concurrency control —</b> Read-write contention is rare, as updates are randomly spread over the database. However, some pages get read very frequently, e.g., some middle level index pages in the <code>post</code> table. Keeping a count of reading threads requires a mutex, and there is significant contention on this. Since the hot set can be one page, adding more mutexes does not always help. However, hash partitioning the index into many independent trees (as in the case of a cluster) helps for this. There is also contention on a mutex for assigning threads to client requests, as there are large numbers of short operations.</p> </li> </ul> <p>In subsequent posts, we will look at specific queries, what they in fact do, and what their theoretical performance limits would be. In this way we will have a precise understanding of which way SNB can steer the graph DB community.</p> <h3> SNB Interactive Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1834" id="link-id0x2aab62c9ee38"> SNB Interactive, Part 1: What is SNB Interactive Really About?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1836" id="link-id0x2aab62e64818"> SNB Interactive, Part 2: Modeling Choices</a> </li> <li> SNB Interactive, Part 3: Choke Points and Initial Run on Virtuoso<i> (this post)</i> </li> </ul>
2015-06-09T11:24:58.727972-04:00
Life after sun.jdbc.odbc.JdbcOdbcDriver
http://www.openlinksw.com:443/weblog/uda/135/?date=2015-06-04#1840
2015-06-04T21:12:19Z
<h2>In the beginning...</h2> <p>When Sun originally released Java 1.0, there were no JDBC drivers -- there wasn't even a JDBC.</p> <p>Data access came in Java 2.0, as <a href="http://www.oracle.com/technetwork/java/overview-141217.html" id="link-id0x2aab5b257518">JDBC 1.0</a>, but there were very few JDBC drivers from any source, as would be expected with any new technology -- but the ODBC ecosystem (itself then at only v2.0) was going strong.</p> <p>Sun recognized that Java wouldn't have as much uptake without a functional data access solution -- so they produced and bundled the original Type 1 JDBC-ODBC Bridge Driver, <code>sun.jdbc.odbc.JdbcOdbcDriver</code>, but from the very beginning, they warned that users "should use the JDBC-ODBC Bridge only for experimental prototyping or when you have no other driver available."</p> <p>That bundled JDBC-ODBC Bridge was (and always remained) single-threaded, and though it received some other updates along the way, it only ever supported a subset of JDBC 2.0 and later. Sun (and later Oracle) recommended that users employ "a pure Java JDBC technology-enabled driver, type 3 or 4, in order to get all of the benefits of the Java programming language and the JDBC API."</p> <h2>Where does OpenLink Software come in?</h2> <p>Even in the early days of JDBC, we saw that there would not always be an available JDBC driver for a given target data source -- but the numbers of ODBC drivers were rapidly increasing, supporting every major and many minor DBMS and other data sources. We saw a need for an enterprise-grade, non-experimental Bridge solution, with full support for the JDBC API.</p> <p>We delivered this first as our <a href="http://uda.openlinksw.com/jdbc-odbc-mt/" id="link-id0x2aab5bf439b8">Type 3 Multi-Tier solution</a>, bridging from JDBC in one environment (typically a UNIX-like OS) to ODBC in another (most often, Microsoft Windows). </p> <p style="text-align:center"> <a href="http://uda.openlinksw.com/images/jdbcodbcmt.gif" target="_blank" id="link-id0x7fe32e904c08"><img src="http://uda.openlinksw.com/images/jdbcodbcmt.gif" alt="Type 3 Enterprise Edition (Multi-Tier) Architecture Diagram" title="Type 3 Enterprise Edition (Multi-Tier) Architecture Diagram" style="max-width: 300px; height: auto" /> </a> <br /> Type 3 Enterprise Edition (Multi-Tier) Architecture Diagram <br /> <i>(click to enlarge)</i> </p> Soon afterward, we released our first <a href="http://uda.openlinksw.com/jdbc-odbc-st/" id="link-id0x2aab5b257628">Type 1 Single-Tier solution</a>, first for Windows, but quickly also supporting Linux, Mac OS X, Solaris, AIX, HP-UX, and other UNIX-like OS, through our own <a href="http://www.iodbc.org/dataspace/iodbc/wiki/iODBC/" id="link-id0x2aab5b0cbc48">iODBC driver manager</a> -- which was itself helping to expand the ODBC ecosystem far beyond its birthplace on Windows. <p style="text-align:center"> <a href="http://uda.openlinksw.com/images/jdbcodbcst.gif" target="_blank" id="link-id0x7fe32e385fe8"><img src="http://uda.openlinksw.com/images/jdbcodbcst.gif" alt="Type 1 Lite Edition (Single-Tier) Architecture Diagram" title="Type 1 Lite Edition (Single-Tier) Architecture Diagram" style="max-width: 300px; height: auto" /> </a> <br /> Type 1 Lite Edition (Single-Tier) Architecture Diagram <br /> <i>(click to enlarge)</i> </p> <h2>Why is Java 8 so special?</h2> <p>Sun long warned that the JRE-bundled Bridge was transitional, and Oracle confirmed immediately upon acquisition that it would "be removed in JDK 8. In addition, Oracle does not support the JDBC-ODBC Bridge." Java 8 is now in full release, and indeed, the venerable <code>sun.jdbc.odbc.JdbcOdbcDriver</code> is no longer present, as evidenced by the scary looking error --</p> <blockquote> <code><pre>java.lang.ClassNotFoundException: sun.jdbc.odbc.JdbcOdbcDriver at java.net.URLClassLoader$1.run(URLClassLoader.java:372) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:360) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:30 at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:259) </pre> </code> </blockquote> <p>Any Java users or applications relying on ODBC connections and also needing the security and other improvements found in Java 8 are left high and dry... Or would be, but for OpenLink Software.</p> <p>Our JDBC-to-ODBC Bridge, in both <a href="http://uda.openlinksw.com/jdbc-odbc-st/" id="link-id0x2aab5b0cc1f8">Type 1</a> and <a href="http://uda.openlinksw.com/jdbc-odbc-mt/" id="link-id0x2aab5b0cc308">Type 3</a> forms, has been available and regularly updated since its original release for JDBC 1. Fully multi-threaded since Java Runtime Environments (JREs) could handle such, we have also kept pace with the JDBC API -- now at <a href="http://docs.oracle.com/javase/8/docs/technotes/guides/jdbc/jdbc_42.html" id="link-id0x2aab5b0cc548">JDBC 4.2, in 2015's Java 8 a/k/a JDK/JVM/JRE 1.8</a> -- and maintained compatibility with the also-evolving <a href="https://msdn.microsoft.com/library/ee388580.aspx" id="link-id0x2aab5b0cc6a8">ODBC API, now at 3.8</a>.</p> <p>Especially important for the modern world, our solutions support both 64-bit and 32-bit environments, including both 64-bit JVMs and ODBC drivers, and our <a href="http://uda.openlinksw.com/jdbc-odbc-mt/" id="link-id0x7fe35a5e2168">Type 3 solutions</a> can even bridge between these, whether you have a 64-bit JVM and need to connect to a 32-bit ODBC driver, or you have a 32-bit JVM and need to connect to a 64-bit ODBC driver.</p> <p>As always, our solutions are <a href="http://download.openlinksw.com/solwiz/" id="link-id0x2aab5b0cc0e8">available for immediate download</a>, with a free two-week trial license provided alongside. We encourage pre-purchase installation, configuration, and testing, with support provided through our <a href="http://boards.openlinksw.com/support/index.php" id="link-id0x2aab5b0cc9a8">web-based Support Forums</a> and even <a href="http://support.openlinksw.com/support/online-support.vsp" id="link-id0x2aab5b0ccaa8">free up-and-running Support Cases</a>. Once you've confirmed the driver works for you, <a href="http://uda.openlinksw.com/offers/" id="link-id0x2aab5b255fc8">entry level and special offer licenses</a> may be purchased online or through our Sales Team; these as well as custom license configurations or <a href="http://www.openlinksw.com/partners/" id="link-id0x2aab5b256168">partnership (IBP, ISV, VAR, OEM, etc.) arrangements</a> are always available by <a href="http://www.openlinksw.com/contact/" id="link-id0x2aab5b256288">direct contact</a>.</p>
2016-05-12T22:39:03.406510
The Virtuoso Science Library
http://www.openlinksw.com:443/blog/vdb/blog/?date=2015-06-03#1839
2015-06-03T16:51:13Z
<p>There is a lot of scientific material on Virtuoso, but it has not been presented all together in any one place. So I am making here a compilation of the best resources with a paragraph of introduction on each. Some of these are project deliverables from projects under the EU FP7 programme; some are peer-reviewed publications.</p> <p>For the future, <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtuosoScienceLibrary" id="link-id0x2aab62129648">an updated version of this list</a> may be found on the <a href="http://virtuoso.openlinksw.com/" id="link-id0x2aab614e7508">main Virtuoso site</a>.</p> <h2>European Project Deliverables</h2> <ul> <li> <p> <b><a href="http://svn.aksw.org/projects/GeoKnow/Public/D2.6.1_Prototype_of_Built-in_Complex_Geo_Problem_Solving.pdf" id="link-id0x2aab62a51908">GeoKnow D 2.6.1</a>: Graph Analytics in the DBMS</b> (2015-01-05)</p> <p>This introduces the idea of unbundling basic cluster DBMS functionality like cross partition joins and partitioned group by to form a graph processing framework collocated with the data.</p> </li> <li> <p> <b><a href="http://svn.aksw.org/projects/GeoKnow/Public/D2.4.1_Geospatial_Clustering.pdf" id="link-id0x2aab61577e18">GeoKnow D2.4.1</a>: Geospatial Clustering and Characteristic Sets</b> (2015-01-06)</p> <p>This presents experimental results of structure-aware RDF applied to geospatial data. The regularly structured part of the data goes in tables; the rest is triples/quads. Furthermore, for the first time in the RDF space, physical storage location is correlated to properties of entities, in this case geo location, so that geospatially adjacent items are also likely adjacent in the physical data representation.</p> </li> <li> <p> <b><a href="http://lod2.eu/Deliverable/D2.1.5.html" id="link-id0x2aab617997f8">LOD2 D2.1.5</a>: 500 billion triple BSBM</b> (2014-08-18)</p> <p>This presents experimental results on lookup and BI workloads on Virtuoso cluster with 12 nodes, for a total of 3T RAM and 192 cores. This also discusses bulk load, at up to 6M triples/s and specifics of query optimization in scale-out settings.</p> </li> <li> <p> <b><a href="http://lod2.eu/Deliverable/D2.6.html" id="link-id0x2aab629cf8c8">LOD2 D2.6</a>: Parallel Programming in SQL</b> (2012-08-12)</p> <p>This discusses ways of making SQL procedures partitioning-aware, so that one can, map-reduce style, send parallel chunks of computation to each partition of the data.</p> </li> </ul> <h2>Publications</h2> <h3>2015</h3> <ul> <li> <p> <b>Minh-Duc, Pham, Linnea, P., Erling, O., and Boncz, P.A. "<a href="http://homepages.cwi.nl/~duc/papers/emergentschema_www15.pdf" id="link-id0x2aab61eb10a8">Deriving an Emergent Relational Schema from RDF Data</a>," WWW, 2015.</b> </p> <p>This paper shows how RDF is in fact structured and how this structure can be reconstructed. This reconstruction then serves to create a physical schema, reintroducing all the benefits of physical design to the schema-last world. Experiments with Virtuoso show marked gains in query speed and data compactness.</p> </li> </ul> <h3>2014</h3> <ul> <li> <p> <b>Peter A. Boncz, Orri Erling, Minh-Duc Pham: <a href="http://oai.cwi.nl/oai/asset/21394/21394B.pdf" id="link-id0x2aab62eb22f8">Experiences with Virtuoso Cluster RDF Column Store</a>. Linked Data Management 2014: 239-259</b> </p> <p>This book chapter gives an in-depth look at the performance dynamics of Virtuoso scale out.</p> </li> </ul> <h3>2013</h3> <ul> <li> <p> <b>P. A. Boncz, T. Neumann, and O. Erling. <a href="http://oai.cwi.nl/oai/asset/21424/21424B.pdf" id="link-id0x2aab621bc728">TPC-H Analyzed: Hidden Messages and Lessons Learned from an Influential Benchmark</a>. Proceedings of the TPC Technology Conference on Performance Evaluation & Benchmarking TPCTC, 2013.</b> </p> <p>This is a summary of all factors that make up analytics performance by those who know. The Virtuoso <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aab61dceb68">TPC-H blog series</a> is a further development and commentary on these same truths.</p> </li> </ul> <h3>2012</h3> <ul> <li> <p> <b>Orri Erling: <a href="http://sites.computer.org/debull/A12mar/vicol.pdf" id="link-id0x2aab61fac308">Virtuoso, a Hybrid RDBMS/Graph Column Store</a>. IEEE Data Eng. Bull. (DEBU) 35(1):3-8 (2012)</b> </p> <p>This paper introduces the Virtuoso column store architecture and design choices. One design is made to serve both random updates and lookups as well as the big scans where column stores traditionally excel. Examples are given from both TPC-H and the schema-less RDF world.</p> </li> <li> <p> <b>Minh-Duc Pham, Peter A. Boncz, Orri Erling: <a href="http://oai.cwi.nl/oai/asset/19919/19919D.pdf" id="link-id0x2aab617b0238">S3G2: A Scalable Structure-Correlated Social Graph Generator</a>. TPCTC 2012:156-172</b> </p> <p>This paper presents the basis of the social network benchmarking technology later used in the LDBC benchmarks.</p> </li> </ul> <h3>2011</h3> <ul> <li> <p> <b>Christian Bizer, Peter A. Boncz, Michael L. Brodie, Orri Erling: "<a href="http://www.sigmod.org/publications/sigmod-record/1112/pdfs/10.report.bizer.pdf" id="link-id0x2aab616b2678">The Meaningful Use of Big Data: Four Perspectives – Four Challenges</a>." SIGMOD Record (SIGMOD) 40(4):56-60 (2011)</b> </p> <p>This is an anthology of views by industry thought leaders on what semantics could or ought to contribute to the practice of data management.</p> </li> </ul> <h3>2009</h3> <ul> <li> <p> <b>Orri Erling, Ivan Mikhailov: <a href="http://www.openlinksw.com/weblog/oerling/lodw2.pdf" id="link-id0x2aab62629058">Faceted Views over Large-Scale Linked Data</a>. LDOW 2009</b> </p> <p>This paper introduces anytime query answering as an enabling technology for open-ended querying of large data on public service end points. While not every query can be run to completion, partial results can most often be returned within a constrained time window.</p> </li> <li> <p> <b>Orri Erling, Ivan Mikhailov: <a href="http://www.researchgate.net/publication/234196284_Virtuoso_RDF_Support_in_a_Native_RDBMS" id="link-id0x2aab6150f208">Virtuoso: RDF Support in a Native RDBMS</a>. Semantic Web Information Management 2009:501-519</b> </p> <p>This is a general presentation of how a SQL engine needs to be adapted to serve a run-time typed and schema-less workload.</p> </li> </ul> <h3>2008</h3> <ul> <li> <p> <b>Orri Erling, Ivan Mikhailov: <a href="http://www.openlinksw.com/weblog/oerling/RDFAndMapped_BI.pdf" id="link-id0x2aab62b107d8">Integrating Open Sources and Relational Data with SPARQL</a>. ESWC 2008:838-842</b> </p> <p>This paper introduces the still challenging RDF-H benchmark, an RDF translation of the classic TPC-H. Running this over SPARQL to SQL mapping is considered.</p> </li> </ul> <h3>2007</h3> <ul> <li> <p> <b>Orri Erling, Ivan Mikhailov: <a href="http://ceur-ws.org/Vol-301/Paper_5_Erling.pdf" id="link-id0x2aab616f18a8">RDF Support in the Virtuoso DBMS</a>. CSSW 2007:59-68</b> </p> <p>This is an initial discussion of RDF support in Virtuoso. Most specifics are by now different but this can give a historical perspective.</p> </li> </ul>
2015-06-03T12:53:50.259637-04:00
The Virtuoso Science Library
http://www.openlinksw.com:443/weblog/oerling/?date=2015-06-03#1838
2015-06-03T16:51:04Z
<p>There is a lot of scientific material on Virtuoso, but it has not been presented all together in any one place. So I am making here a compilation of the best resources with a paragraph of introduction on each. Some of these are project deliverables from projects under the EU FP7 programme; some are peer-reviewed publications.</p> <p>For the future, <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtuosoScienceLibrary" id="link-id0x2aab62129648">an updated version of this list</a> may be found on the <a href="http://virtuoso.openlinksw.com/" id="link-id0x2aab614e7508">main Virtuoso site</a>.</p> <h2>European Project Deliverables</h2> <ul> <li> <p> <b><a href="http://svn.aksw.org/projects/GeoKnow/Public/D2.6.1_Prototype_of_Built-in_Complex_Geo_Problem_Solving.pdf" id="link-id0x2aab62a51908">GeoKnow D 2.6.1</a>: Graph Analytics in the DBMS</b> (2015-01-05)</p> <p>This introduces the idea of unbundling basic cluster DBMS functionality like cross partition joins and partitioned group by to form a graph processing framework collocated with the data.</p> </li> <li> <p> <b><a href="http://svn.aksw.org/projects/GeoKnow/Public/D2.4.1_Geospatial_Clustering.pdf" id="link-id0x2aab61577e18">GeoKnow D2.4.1</a>: Geospatial Clustering and Characteristic Sets</b> (2015-01-06)</p> <p>This presents experimental results of structure-aware RDF applied to geospatial data. The regularly structured part of the data goes in tables; the rest is triples/quads. Furthermore, for the first time in the RDF space, physical storage location is correlated to properties of entities, in this case geo location, so that geospatially adjacent items are also likely adjacent in the physical data representation.</p> </li> <li> <p> <b><a href="http://lod2.eu/Deliverable/D2.1.5.html" id="link-id0x2aab617997f8">LOD2 D2.1.5</a>: 500 billion triple BSBM</b> (2014-08-18)</p> <p>This presents experimental results on lookup and BI workloads on Virtuoso cluster with 12 nodes, for a total of 3T RAM and 192 cores. This also discusses bulk load, at up to 6M triples/s and specifics of query optimization in scale-out settings.</p> </li> <li> <p> <b><a href="http://lod2.eu/Deliverable/D2.6.html" id="link-id0x2aab629cf8c8">LOD2 D2.6</a>: Parallel Programming in SQL</b> (2012-08-12)</p> <p>This discusses ways of making SQL procedures partitioning-aware, so that one can, map-reduce style, send parallel chunks of computation to each partition of the data.</p> </li> </ul> <h2>Publications</h2> <h3>2015</h3> <ul> <li> <p> <b>Minh-Duc, Pham, Linnea, P., Erling, O., and Boncz, P.A. "<a href="http://homepages.cwi.nl/~duc/papers/emergentschema_www15.pdf" id="link-id0x2aab61eb10a8">Deriving an Emergent Relational Schema from RDF Data</a>," WWW, 2015.</b> </p> <p>This paper shows how RDF is in fact structured and how this structure can be reconstructed. This reconstruction then serves to create a physical schema, reintroducing all the benefits of physical design to the schema-last world. Experiments with Virtuoso show marked gains in query speed and data compactness.</p> </li> </ul> <h3>2014</h3> <ul> <li> <p> <b>Peter A. Boncz, Orri Erling, Minh-Duc Pham: <a href="http://oai.cwi.nl/oai/asset/21394/21394B.pdf" id="link-id0x2aab62eb22f8">Experiences with Virtuoso Cluster RDF Column Store</a>. Linked Data Management 2014: 239-259</b> </p> <p>This book chapter gives an in-depth look at the performance dynamics of Virtuoso scale out.</p> </li> </ul> <h3>2013</h3> <ul> <li> <p> <b>P. A. Boncz, T. Neumann, and O. Erling. <a href="http://oai.cwi.nl/oai/asset/21424/21424B.pdf" id="link-id0x2aab621bc728">TPC-H Analyzed: Hidden Messages and Lessons Learned from an Influential Benchmark</a>. Proceedings of the TPC Technology Conference on Performance Evaluation & Benchmarking TPCTC, 2013.</b> </p> <p>This is a summary of all factors that make up analytics performance by those who know. The Virtuoso <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aab61dceb68">TPC-H blog series</a> is a further development and commentary on these same truths.</p> </li> </ul> <h3>2012</h3> <ul> <li> <p> <b>Orri Erling: <a href="http://sites.computer.org/debull/A12mar/vicol.pdf" id="link-id0x2aab61fac308">Virtuoso, a Hybrid RDBMS/Graph Column Store</a>. IEEE Data Eng. Bull. (DEBU) 35(1):3-8 (2012)</b> </p> <p>This paper introduces the Virtuoso column store architecture and design choices. One design is made to serve both random updates and lookups as well as the big scans where column stores traditionally excel. Examples are given from both TPC-H and the schema-less RDF world.</p> </li> <li> <p> <b>Minh-Duc Pham, Peter A. Boncz, Orri Erling: <a href="http://oai.cwi.nl/oai/asset/19919/19919D.pdf" id="link-id0x2aab617b0238">S3G2: A Scalable Structure-Correlated Social Graph Generator</a>. TPCTC 2012:156-172</b> </p> <p>This paper presents the basis of the social network benchmarking technology later used in the LDBC benchmarks.</p> </li> </ul> <h3>2011</h3> <ul> <li> <p> <b>Christian Bizer, Peter A. Boncz, Michael L. Brodie, Orri Erling: "<a href="http://www.sigmod.org/publications/sigmod-record/1112/pdfs/10.report.bizer.pdf" id="link-id0x2aab616b2678">The Meaningful Use of Big Data: Four Perspectives – Four Challenges</a>." SIGMOD Record (SIGMOD) 40(4):56-60 (2011)</b> </p> <p>This is an anthology of views by industry thought leaders on what semantics could or ought to contribute to the practice of data management.</p> </li> </ul> <h3>2009</h3> <ul> <li> <p> <b>Orri Erling, Ivan Mikhailov: <a href="http://www.openlinksw.com/weblog/oerling/lodw2.pdf" id="link-id0x2aab62629058">Faceted Views over Large-Scale Linked Data</a>. LDOW 2009</b> </p> <p>This paper introduces anytime query answering as an enabling technology for open-ended querying of large data on public service end points. While not every query can be run to completion, partial results can most often be returned within a constrained time window.</p> </li> <li> <p> <b>Orri Erling, Ivan Mikhailov: <a href="http://www.researchgate.net/publication/234196284_Virtuoso_RDF_Support_in_a_Native_RDBMS" id="link-id0x2aab6150f208">Virtuoso: RDF Support in a Native RDBMS</a>. Semantic Web Information Management 2009:501-519</b> </p> <p>This is a general presentation of how a SQL engine needs to be adapted to serve a run-time typed and schema-less workload.</p> </li> </ul> <h3>2008</h3> <ul> <li> <p> <b>Orri Erling, Ivan Mikhailov: <a href="http://www.openlinksw.com/weblog/oerling/RDFAndMapped_BI.pdf" id="link-id0x2aab62b107d8">Integrating Open Sources and Relational Data with SPARQL</a>. ESWC 2008:838-842</b> </p> <p>This paper introduces the still challenging RDF-H benchmark, an RDF translation of the classic TPC-H. Running this over SPARQL to SQL mapping is considered.</p> </li> </ul> <h3>2007</h3> <ul> <li> <p> <b>Orri Erling, Ivan Mikhailov: <a href="http://ceur-ws.org/Vol-301/Paper_5_Erling.pdf" id="link-id0x2aab616f18a8">RDF Support in the Virtuoso DBMS</a>. CSSW 2007:59-68</b> </p> <p>This is an initial discussion of RDF support in Virtuoso. Most specifics are by now different but this can give a historical perspective.</p> </li> </ul>
2015-06-03T12:53:36.943110-04:00
SNB Interactive, Part 2: Modeling Choices
http://www.openlinksw.com:443/blog/vdb/blog/?date=2015-05-14#1837
2015-05-14T15:37:59Z
<p> <a href="http://ldbcouncil.org/benchmarks/snb" id="link-id0x2aab76e5ac28">SNB Interactive</a> is the wild frontier, with very few rules. This is necessary, among other reasons, because there is no standard property graph data model, and because the contestants support a broad mix of programming models, ranging from in-process APIs to declarative query.</p> <p>In the case of <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aab76e5af48">Virtuoso</a>, we have played with <a href="http://dbpedia.org/resource/SQL" id="link-id0x2aab76e5b088">SQL</a> and <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x2aab77e969e8">SPARQL</a> implementations. For a fixed schema and well known workload, SQL will always win. The reason is that SQL allows materialization of multi-part indices and data orderings that make sense for the application. In other words, there is transparency into physical design. An RDF/SPARQL-based application may also have physical design by means of structure-aware storage, but this is more complex and here we are just concerned with speed and having things work precisely as we intend.</p> <h2>Schema Design</h2> <p>SNB has a regular schema described by a <a href="https://en.wikipedia.org/wiki/Unified_Modeling_Language" id="link-id0x2aab77e96da8">UML</a> diagram. This has a number of relationships, of which some have attributes. There are no heterogenous sets, i.e., no need for run-time typed attributes or graph edges with the same label but heterogenous end-points. Translation into SQL or SPARQL is straightforward. Edges with attributes (e.g., the <code><a href="http://xmlns.com/foaf/spec/#term_knows" id="link-id0x2aab77e97098">foaf:knows</a></code> relation between people) would end up represented as a subject with the end points and the effective date as properties. The relational implementation has a two-part primary key and the effective date as a dependent column. A native property graph database would use an edge with an extra property for this, as such are typically supported.</p> <p>The only table-level choice has to do with whether <code>posts</code> and <code>comments</code> are kept in the same or different data structures. The Virtuoso schema uses a single table for both, with nullable columns for the properties that occur only in one. This makes the queries more concise. There are cases where only non-reply <code>posts</code> of a given <code>author</code> are accessed. This is supported by having two <code>author</code> foreign key columns each with its own index. There is a single nullable foreign key from the reply to the post/comment being replied to.</p> <p>The workload has some frequent access paths that need to be supported by index. Some queries reward placing extra columns in indices. For example, a common pattern is accessing the most recent posts of an author or a group of authors. There, having a composite key of <code>ps_creatorid, ps_creationdate, ps_postid</code> pays off since the <code>top-k</code> on <code>creationdate</code> can be pushed down into the index without needing a reference to the table.</p> <p>The implementation is free to choose data types for attributes, particularly <code>datetimes</code>. The Virtuoso implementation adopts the practice of the <a href="http://dbpedia.org/resource/DEX_(Graph_database)" id="link-id0x2aab77e97d88">Sparksee</a> and <a href="http://dbpedia.org/resource/Neo4j" id="link-id0x2aab77e97ed8">Neo4j</a> implementations and represents this is a count of milliseconds since epoch. This is less confusing, faster to compare, and more compact than a native datetime datatype that may or may not have timezones, etc. Using a built-in datetime seems to be nearly always a bad idea. A dimension table or a number for a time dimension avoids the ambiguities of a calendar or at least makes these explicit.</p> <p>The benchmark allows procedurally maintained materializations of intermediate results for use by queries as long as these are maintained transaction-by-transaction. For example, each person could have the 20 newest posts by their immediate contacts precomputed. This would reduce Q2 "top of the wall" to a single lookup. This does not however appear to be worthwhile. The Virtuoso implementation does do one such materialization for Q14: A connection weight is calculated for every pair of persons that know each other. This is related to the count of replies by either to content generated by the other. If there does not exist a single reply in either direction, the weight is taken to be 0. This weight is precomputed after bulk load and subsequently maintained each time a reply is added. The table for this is the only row-wise structure in the schema and represents a half-matrix of connected people, i.e., <code>person1, person2 -> weight</code>. <code>Person1</code> is by convention the one with the smaller <code>p_personid</code>. Note that comparing IDs in this way is useful but not normally supported by SPARQL/RDF systems. SPARQL would end up comparing strings of URIs with disastrous performance implications unless an implementation-specific trick were used.</p> <p>In the next installment, we will analyze an actual run.</p> <h3> SNB Interactive Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1834" id="link-id0x2aab62d2ef08"> SNB Interactive, Part 1: What is SNB Interactive Really About?</a> </li> <li> SNB Interactive, Part 2: Modeling Choices<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1841" id="link-id0x2aab62df6248"> SNB Interactive, Part 3: Choke Points and Initial Run on Virtuoso</a> </li> </ul>
2015-06-09T11:36:52.327195-04:00
SNB Interactive, Part 2: Modeling Choices
http://www.openlinksw.com:443/weblog/oerling/?date=2015-05-14#1836
2015-05-14T15:37:50Z
<p> <a href="http://ldbcouncil.org/benchmarks/snb" id="link-id0x2aab76e5ac28">SNB Interactive</a> is the wild frontier, with very few rules. This is necessary, among other reasons, because there is no standard property graph data model, and because the contestants support a broad mix of programming models, ranging from in-process APIs to declarative query.</p> <p>In the case of <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aab76e5af48">Virtuoso</a>, we have played with <a href="http://dbpedia.org/resource/SQL" id="link-id0x2aab76e5b088">SQL</a> and <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x2aab77e969e8">SPARQL</a> implementations. For a fixed schema and well known workload, SQL will always win. The reason is that SQL allows materialization of multi-part indices and data orderings that make sense for the application. In other words, there is transparency into physical design. An RDF/SPARQL-based application may also have physical design by means of structure-aware storage, but this is more complex and here we are just concerned with speed and having things work precisely as we intend.</p> <h2>Schema Design</h2> <p>SNB has a regular schema described by a <a href="https://en.wikipedia.org/wiki/Unified_Modeling_Language" id="link-id0x2aab77e96da8">UML</a> diagram. This has a number of relationships, of which some have attributes. There are no heterogenous sets, i.e., no need for run-time typed attributes or graph edges with the same label but heterogenous end-points. Translation into SQL or SPARQL is straightforward. Edges with attributes (e.g., the <code><a href="http://xmlns.com/foaf/spec/#term_knows" id="link-id0x2aab77e97098">foaf:knows</a></code> relation between people) would end up represented as a subject with the end points and the effective date as properties. The relational implementation has a two-part primary key and the effective date as a dependent column. A native property graph database would use an edge with an extra property for this, as such are typically supported.</p> <p>The only table-level choice has to do with whether <code>posts</code> and <code>comments</code> are kept in the same or different data structures. The Virtuoso schema uses a single table for both, with nullable columns for the properties that occur only in one. This makes the queries more concise. There are cases where only non-reply <code>posts</code> of a given <code>author</code> are accessed. This is supported by having two <code>author</code> foreign key columns each with its own index. There is a single nullable foreign key from the reply to the post/comment being replied to.</p> <p>The workload has some frequent access paths that need to be supported by index. Some queries reward placing extra columns in indices. For example, a common pattern is accessing the most recent posts of an author or a group of authors. There, having a composite key of <code>ps_creatorid, ps_creationdate, ps_postid</code> pays off since the <code>top-k</code> on <code>creationdate</code> can be pushed down into the index without needing a reference to the table.</p> <p>The implementation is free to choose data types for attributes, particularly <code>datetimes</code>. The Virtuoso implementation adopts the practice of the <a href="http://dbpedia.org/resource/DEX_(Graph_database)" id="link-id0x2aab77e97d88">Sparksee</a> and <a href="http://dbpedia.org/resource/Neo4j" id="link-id0x2aab77e97ed8">Neo4j</a> implementations and represents this is a count of milliseconds since epoch. This is less confusing, faster to compare, and more compact than a native datetime datatype that may or may not have timezones, etc. Using a built-in datetime seems to be nearly always a bad idea. A dimension table or a number for a time dimension avoids the ambiguities of a calendar or at least makes these explicit.</p> <p>The benchmark allows procedurally maintained materializations of intermediate results for use by queries as long as these are maintained transaction-by-transaction. For example, each person could have the 20 newest posts by their immediate contacts precomputed. This would reduce Q2 "top of the wall" to a single lookup. This does not however appear to be worthwhile. The Virtuoso implementation does do one such materialization for Q14: A connection weight is calculated for every pair of persons that know each other. This is related to the count of replies by either to content generated by the other. If there does not exist a single reply in either direction, the weight is taken to be 0. This weight is precomputed after bulk load and subsequently maintained each time a reply is added. The table for this is the only row-wise structure in the schema and represents a half-matrix of connected people, i.e., <code>person1, person2 -> weight</code>. <code>Person1</code> is by convention the one with the smaller <code>p_personid</code>. Note that comparing IDs in this way is useful but not normally supported by SPARQL/RDF systems. SPARQL would end up comparing strings of URIs with disastrous performance implications unless an implementation-specific trick were used.</p> <p>In <a href="http://www.openlinksw.com/weblog/oerling/?id=1841" id="link-id0x2aab6315d2b8">the next installment</a>, we will analyze an actual run.</p> <h3> SNB Interactive Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1834" id="link-id0x2aab62d2ef08"> SNB Interactive, Part 1: What is SNB Interactive Really About?</a> </li> <li> SNB Interactive, Part 2: Modeling Choices<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1841" id="link-id0x2aab62df6248"> SNB Interactive, Part 3: Choke Points and Initial Run on Virtuoso</a> </li> </ul>
2015-06-09T11:26:49.994102-04:00
SNB Interactive, Part 1: What is SNB Interactive Really About?
http://www.openlinksw.com:443/blog/vdb/blog/?date=2015-05-14#1835
2015-05-14T15:37:41Z
<p>This is the first in a series of blog posts analyzing the Interactive workload of the <a href="http://ldbcouncil.org/benchmarks/snb" id="link-id0x2aab77caacf8">LDBC Social Network Benchmark</a>. This is written from the dual perspective of participating in the benchmark design, and of building the <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aab76f21928">OpenLink Virtuoso</a> implementation of same.</p> <p>With two implementations of SNB Interactive at four different scales, we can take a first look at what the benchmark is really about. The hallmark of a benchmark implementation is that its performance characteristics are understood; even if these do not represent the maximum of the attainable, there are no glaring mistakes; and the implementation represents a reasonable best effort by those who ought to know such, namely the system vendors.</p> <p>The essence of a benchmark is a set of trick questions or "choke points," as <a href="http://ldbcouncil.org/" id="link-id0x2aab77ca9c88">LDBC</a> calls them. A number of these were planned from the start. It is then the role of experience to tell whether addressing these is really the key to winning the race. Unforeseen ones will also surface.</p> <p>So far, we see that SNB confronts the implementor with choices in the following areas:</p> <ul> <li> <p> <b>Data model —</b> Tabular relational (commonly known as SQL), graph relational (including RDF), property graph, etc.</p> </li> <li> <p> <b>Physical storage model —</b> Row-wise vs. column-wise, for instance.</p> </li> <li> <p> <b>Ordering of materialized data —</b> Sorted projections, composite keys, replicating columns in auxiliary data structures, etc.</p> </li> <li> <p> <b>Persistence of intermediate results — </b> Materialized views, triggers, precomputed temporary tables, etc.</p> </li> <li> <p> <b>Query optimization —</b> join order/type, interesting physical data orderings, late projection, top k, etc.</p> </li> <li> <p> <b>Parameters vs. literals —</b> Sometimes different parameter values result in different optimal query plans.</p> </li> <li> <p> <b>Predictable, uniform latency —</b> Measurement rules stipulate the the <a href="http://dbpedia.org/resource/System_under_test" id="link-id0x2aab77de7a58">SUT (system under test)</a> must not fall behind the simulated workload.</p> </li> <li> <p> <b>Durability —</b> How to make data durable while maintaining steady throughput, e.g., logging, checkpointing, etc.</p> </li> </ul> <p>In the process of making a benchmark implementation, one naturally encounters questions about the validity, reasonability, and rationale of the benchmark definition itself. Additionally, even though the benchmark might not directly measure certain aspects of a system, making an implementation will take a system past its usual envelope and highlight some operational aspects.</p> <ul> <li> <p> <b>Data generation —</b> Generating a mid-size dataset takes time, e.g., 8 hours for 300G. In a cloud situation, keeping the dataset in <a href="http://dbpedia.org/resource/Amazon_S3" id="link-id0x2aab77c11b18">S3</a> or similar is necessary; re-generating every time is not an option.</p> </li> <li> <p> <b>Query mix —</b> Are the relative frequencies of the operations reasonable? What bias does this introduce?</p> </li> <li> <p> <b>Uniformity of parameters —</b> Due to non-uniform data distributions in the dataset, there is easily a 100x difference between "fast" and "slow" cases of a single query template. How long does one need to run to balance these fluctuations?</p> </li> <li> <p> <b>Working set —</b> Experience shows that there is a large difference between almost-warm and steady-state of working set. This can be a factor of 1.5 in throughput.</p> </li> <li> <p> <b>Reasonability of latency constraints —</b> In the present case, a qualifying run must have no more than 5% of all query executions starting over 1 second late. Each execution is scheduled beforehand and done at the intended time. If the SUT does not keep up, it will have all available threads busy and must finish some work before accepting new work, so some queries will start late. Is this a good criterion for measuring consistency of response time? There are some obvious possibilities for abuse.</p> </li> <li> <p> <b>Ease of benchmark implementation/execution —</b> Perfection is open-ended and optimization possibilities infinite, albeit with diminishing returns. Still, getting started should not be too hard. Since systems will be highly diverse, testing that these in fact do the same thing is important. The SNB validation suite is good for this and, given publicly available reference implementations, the effort of getting started is not unreasonable.</p> </li> <li> <p> <b>Ease of adjustment —</b> Since a qualifying run must meet latency constraints while going as fast as possible, setting the performance target involves trial and error. Does the tooling make this easy?</p> </li> <li> <p> <b>Reasonability of durability rule —</b> Right now, one is not required to do checkpoints but must report the time to roll forward from the last checkpoint or initial state. Inspiring vendors to build faster recovery is certainly good, but we are not through with all the implications. What about redundant clusters?</p> </li> </ul> <p>The following posts will look at the above in light of actual experience.</p> <h3> SNB Interactive Series</h3> <ul> <li> SNB Interactive, Part 1: What is SNB Interactive Really About?<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1836" id="link-id0x2aab6357ab98"> SNB Interactive, Part 2: Modeling Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1841" id="link-id0x2aab6357ada8"> SNB Interactive, Part 3: Choke Points and Initial Run on Virtuoso</a> </li> </ul>
2015-06-09T11:36:14.812596-04:00
SNB Interactive, Part 1: What is SNB Interactive Really About?
http://www.openlinksw.com:443/weblog/oerling/?date=2015-05-14#1834
2015-05-14T15:37:01Z
<p>This is the first in a series of blog posts analyzing the Interactive workload of the <a href="http://ldbcouncil.org/benchmarks/snb" id="link-id0x2aab77caacf8">LDBC Social Network Benchmark</a>. This is written from the dual perspective of participating in the benchmark design, and of building the <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aab76f21928">OpenLink Virtuoso</a> implementation of same.</p> <p>With two implementations of SNB Interactive at four different scales, we can take a first look at what the benchmark is really about. The hallmark of a benchmark implementation is that its performance characteristics are understood; even if these do not represent the maximum of the attainable, there are no glaring mistakes; and the implementation represents a reasonable best effort by those who ought to know such, namely the system vendors.</p> <p>The essence of a benchmark is a set of trick questions or "choke points," as <a href="http://ldbcouncil.org/" id="link-id0x2aab77ca9c88">LDBC</a> calls them. A number of these were planned from the start. It is then the role of experience to tell whether addressing these is really the key to winning the race. Unforeseen ones will also surface.</p> <p>So far, we see that SNB confronts the implementor with choices in the following areas:</p> <ul> <li> <p> <b>Data model —</b> Tabular relational (commonly known as SQL), graph relational (including RDF), property graph, etc.</p> </li> <li> <p> <b>Physical storage model —</b> Row-wise vs. column-wise, for instance.</p> </li> <li> <p> <b>Ordering of materialized data —</b> Sorted projections, composite keys, replicating columns in auxiliary data structures, etc.</p> </li> <li> <p> <b>Persistence of intermediate results — </b> Materialized views, triggers, precomputed temporary tables, etc.</p> </li> <li> <p> <b>Query optimization —</b> join order/type, interesting physical data orderings, late projection, top k, etc.</p> </li> <li> <p> <b>Parameters vs. literals —</b> Sometimes different parameter values result in different optimal query plans.</p> </li> <li> <p> <b>Predictable, uniform latency —</b> Measurement rules stipulate the the <a href="http://dbpedia.org/resource/System_under_test" id="link-id0x2aab77de7a58">SUT (system under test)</a> must not fall behind the simulated workload.</p> </li> <li> <p> <b>Durability —</b> How to make data durable while maintaining steady throughput, e.g., logging, checkpointing, etc.</p> </li> </ul> <p>In the process of making a benchmark implementation, one naturally encounters questions about the validity, reasonability, and rationale of the benchmark definition itself. Additionally, even though the benchmark might not directly measure certain aspects of a system, making an implementation will take a system past its usual envelope and highlight some operational aspects.</p> <ul> <li> <p> <b>Data generation —</b> Generating a mid-size dataset takes time, e.g., 8 hours for 300G. In a cloud situation, keeping the dataset in <a href="http://dbpedia.org/resource/Amazon_S3" id="link-id0x2aab77c11b18">S3</a> or similar is necessary; re-generating every time is not an option.</p> </li> <li> <p> <b>Query mix —</b> Are the relative frequencies of the operations reasonable? What bias does this introduce?</p> </li> <li> <p> <b>Uniformity of parameters —</b> Due to non-uniform data distributions in the dataset, there is easily a 100x difference between "fast" and "slow" cases of a single query template. How long does one need to run to balance these fluctuations?</p> </li> <li> <p> <b>Working set —</b> Experience shows that there is a large difference between almost-warm and steady-state of working set. This can be a factor of 1.5 in throughput.</p> </li> <li> <p> <b>Reasonability of latency constraints —</b> In the present case, a qualifying run must have no more than 5% of all query executions starting over 1 second late. Each execution is scheduled beforehand and done at the intended time. If the SUT does not keep up, it will have all available threads busy and must finish some work before accepting new work, so some queries will start late. Is this a good criterion for measuring consistency of response time? There are some obvious possibilities for abuse.</p> </li> <li> <p> <b>Ease of benchmark implementation/execution —</b> Perfection is open-ended and optimization possibilities infinite, albeit with diminishing returns. Still, getting started should not be too hard. Since systems will be highly diverse, testing that these in fact do the same thing is important. The SNB validation suite is good for this and, given publicly available reference implementations, the effort of getting started is not unreasonable.</p> </li> <li> <p> <b>Ease of adjustment —</b> Since a qualifying run must meet latency constraints while going as fast as possible, setting the performance target involves trial and error. Does the tooling make this easy?</p> </li> <li> <p> <b>Reasonability of durability rule —</b> Right now, one is not required to do checkpoints but must report the time to roll forward from the last checkpoint or initial state. Inspiring vendors to build faster recovery is certainly good, but we are not through with all the implications. What about redundant clusters?</p> </li> </ul> <p>The following posts will look at the above in light of actual experience.</p> <h3> SNB Interactive Series</h3> <ul> <li> SNB Interactive, Part 1: What is SNB Interactive Really About?<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1836" id="link-id0x2aab6357ab98"> SNB Interactive, Part 2: Modeling Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1841" id="link-id0x2aab6357ada8"> SNB Interactive, Part 3: Choke Points and Initial Run on Virtuoso</a> </li> </ul>
2015-06-09T11:30:09.916724-04:00
DBpedia Usage Report, January 2015
http://www.openlinksw.com:443/blog/vdb/blog/?date=2015-01-07#1833
2015-01-07T20:12:47Z
<p>We've just published the latest <a href="http://bit.ly/1DymR8p" id="link-id0x2aab663027d8">DBpedia Usage Report</a>, covering v3.3 (released July, 2009) to v3.9 (released September, 2013); v3.10 (sometimes called "DBpedia 2014"; released September, 2014) will be included in the next report.</p> <p>We think you'll find some interesting details in the statistics. There are also some important notes about Virtuoso configuration options and other sneaky technical issues that can surprise you (as they did us!) when exposing an ad-hoc query server to the world.</p>
2015-08-11T12:59:11.440347-04:00
DBpedia Usage Report, January 2015
http://www.openlinksw.com:443/weblog/oerling/?date=2015-01-07#1832
2015-01-07T20:12:37Z
<p>We've just published the latest <a href="http://bit.ly/1DymR8p" id="link-id0x2aab663027d8">DBpedia Usage Report</a>, covering v3.3 (released July, 2009) to v3.9 (released September, 2013); v3.10 (sometimes called "DBpedia 2014"; released September, 2014) will be included in the next report.</p> <p>We think you'll find some interesting details in the statistics. There are also some important notes about Virtuoso configuration options and other sneaky technical issues that can surprise you (as they did us!) when exposing an ad-hoc query server to the world.</p>
2015-08-11T12:59:03.643766-04:00
LDBC: Making Semantic Publishing Execution Rules
http://www.openlinksw.com:443/weblog/oerling/?date=2014-11-13#1831
2014-11-13T21:19:42Z
<p> <a href="http://ldbcouncil.org/" id="link-id0x2aab2c2b02f8">LDBC</a> <a href="http://ldbcouncil.org/developer/spb" id="link-id0x2aab2c2b0438">SPB (Semantic Publishing Benchmark)</a> is based on the BBC Linked Data use case. Thus the data modeling and transaction mix reflect the BBC's actual utilization of RDF. But a benchmark is not only a condensation of current best practice. The BBC Linked Data is deployed on <a href="http://www.ontotext.com/products/ontotext-graphdb-owlim/" id="link-id0x2aab2c2b0698">Ontotext GraphDB</a> (formerly known as OWLIM).</p> <p>So, in SPB we wanted to address substantially more complex queries than the lookups than the BBC linked data deployment primarily serves. Diverse dataset summaries, timelines, and faceted search qualified by keywords and/or geography, are examples of online user experience that SPB needs to cover.</p> <p>SPB is not an analytical workload, per se, but we still find that the queries fall broadly in two categories: </p> <ul> <li>Some queries are centered on a particular search or entity. The data touched by the query size does not grow at the same rate as the dataset. </li> <li>Some queries cover whole cross sections of the dataset, e.g., find the most popular tags across the whole database.</li> </ul> These different classes of questions need to be separated in a metric, otherwise the short lookup dominates at small scales, and the large query at large scales. <p>Another guiding factor of SPB was the BBC's and others' express wish to cover operational aspects such as online backups, replication, and fail-over in a benchmark. True, most online installations have to deal with these, yet these things are as good as absent from present benchmark practice. We will look at these aspects in a different article; for now, I will just discuss the matter of workload mix and metric.</p> <p>Normally, the lookup and analytics workloads are divided into different benchmarks. Here, we will try something different. There are three things the benchmark does:</p> <ul> <li> <p> <b>Updates</b> - These sometimes insert a graph, sometimes delete and re-insert the same graph, sometimes just delete a graph. These are logarithmic to data size.</p> </li> <li> <p> <b>Short queries</b> - These are lookups that most often touch on recent data and can drive page impressions. These are roughly logarithmic to data scale.</p> </li> <li> <p> <b>Analytics</b> - These cover a large fraction of the dataset and are roughly linear to data size.</p> </li> </ul> <p>A test sponsor can decide on the query mix within certain bounds. A qualifying run must sustain a minimum, scale-dependent update throughput and must execute a scale-dependent number of analytical query mixes, or run for a scale-dependent duration. The minimum update rate, the minimum number of analytics mixes and the minimum duration all grow logarithmically to data size.</p> <p>Within these limits, the test sponsor can decide how to mix the workloads. Publishing several results emphasizing different aspects is also possible. A given system may be especially good at one aspect, leading the test sponsor to accentuate this.</p> <p>The benchmark has been developed and tested at small scales, between 50 and 150M triples. Next we need to see how it actually scales. There we expect to see how the two query sets behave differently. One effect that we see right away when loading data is that creating the full text index on the literals is in fact the longest running part. For a SF 32 ( 1.6 billion triples) SPB database we have the following space consumption figures:</p> <ul> <li>46,886 MB of RDF literal text</li> <li>23,924 MB of full text index for RDF literals</li> <li>23,598 MB of URI strings</li> <li>21,981 MB of quads, stored column-wise with default index scheme</li> </ul> <p>Clearly, applying column-wise compression to the strings is the best move for increasing scalability. The literals are individually short, so literal per literal compression will do little or nothing but applying this by the column is known to get a 2x size reduction with Google Snappy.</p> <p>The full text index does not get much from column store techniques, as it already consists of words followed by space efficient lists of word positions. The above numbers are measured with Virtuoso column store, with quads column-wise and the rest row-wise. Each number includes the table(s) and any extra indices associated to them.</p> <p>Let's now look at a full run at unit scale, i.e., 50M triples.</p> <p>The run rules stipulate a minimum of 7 updates per second. The updates are comparatively fast, so we set the update rate to 70 updates per second. This is seen not to take too much CPU. We run 2 threads of updates, 20 of short queries, and 2 of long queries. The minimum run time for the unit scale is 10 minutes, so we do 10 analytical mixes, as this is expected to take a little over 10 minutes. The run stops by itself when the last of the analytical mixes finishes.</p> <p>The interactive driver reports:</p> <blockquote> <code><pre> Seconds run : 2,144 Editorial: 2 agents 68,164 inserts (avg : 46 ms, min : 5 ms, max : 3002 ms) 8,440 updates (avg : 72 ms, min : 15 ms, max : 2471 ms) 8,539 deletes (avg : 37 ms, min : 4 ms, max : 2531 ms) 85,143 operations (68,164 CW Inserts (98 errors), 8,440 CW Updates ( 0 errors), 8,539 CW Deletions ( 0 errors)) 39.7122 average operations per second Aggregation: 20 agents 4120 Q1 queries (avg : 789 ms, min : 197 ms, max : 6,767 ms, 0 errors) 4121 Q2 queries (avg : 85 ms, min : 26 ms, max : 3,058 ms, 0 errors) 4124 Q3 queries (avg : 67 ms, min : 5 ms, max : 3,031 ms, 0 errors) 4118 Q5 queries (avg : 354 ms, min : 3 ms, max : 8,172 ms, 0 errors) 4117 Q8 queries (avg : 975 ms, min : 25 ms, max : 7,368 ms, 0 errors) 4119 Q11 queries (avg : 221 ms, min : 75 ms, max : 3,129 ms, 0 errors) 4122 Q12 queries (avg : 131 ms, min : 45 ms, max : 1,130 ms, 0 errors) 4115 Q17 queries (avg : 5,321 ms, min : 35 ms, max : 13,144 ms, 0 errors) 4119 Q18 queries (avg : 987 ms, min : 138 ms, max : 6,738 ms, 0 errors) 4121 Q24 queries (avg : 917 ms, min : 33 ms, max : 3,653 ms, 0 errors) 4122 Q25 queries (avg : 451 ms, min : 70 ms, max : 3,695 ms, 0 errors) 22.5239 average queries per second. Pool 0, queries [ Q1 Q2 Q3 Q5 Q8 Q11 Q12 Q17 Q18 Q24 Q25 ] 45,318 total retrieval queries (0 timed-out) 22.5239 average queries per second </pre> </code> </blockquote> <p>The analytical driver reports:</p> <blockquote> <code><pre> Aggregation: 2 agents 14 Q4 queries (avg : 9,984 ms, min : 4,832 ms, max : 17,957 ms, 0 errors) 12 Q6 queries (avg : 4,173 ms, min : 46 ms, max : 7,843 ms, 0 errors) 13 Q7 queries (avg : 1,855 ms, min : 1,295 ms, max : 2,415 ms, 0 errors) 13 Q9 queries (avg : 561 ms, min : 446 ms, max : 662 ms, 0 errors) 14 Q10 queries (avg : 2,641 ms, min : 1,652 ms, max : 4,238 ms, 0 errors) 12 Q13 queries (avg : 595 ms, min : 373 ms, max : 1,167 ms, 0 errors) 12 Q14 queries (avg : 65,362 ms, min : 6,127 ms, max : 136,346 ms, 2 errors) 13 Q15 queries (avg : 45,737 ms, min : 12,698 ms, max : 59,935 ms, 0 errors) 13 Q16 queries (avg : 30,939 ms, min : 10,224 ms, max : 38,161 ms, 0 errors) 13 Q19 queries (avg : 310 ms, min : 26 ms, max : 1,733 ms, 0 errors) 12 Q20 queries (avg : 13,821 ms, min : 11,092 ms, max : 15,435 ms, 0 errors) 13 Q21 queries (avg : 36,611 ms, min : 14,164 ms, max : 70,954 ms, 0 errors) 13 Q22 queries (avg : 42,048 ms, min : 7,106 ms, max : 74,296 ms, 0 errors) 13 Q23 queries (avg : 48,474 ms, min : 18,574 ms, max : 93,656 ms, 0 errors) 0.0862 average queries per second. Pool 0, queries [ Q4 Q6 Q7 Q9 Q10 Q13 Q14 Q15 Q16 Q19 Q20 Q21 Q22 Q23 ] 180 total retrieval queries (2 timed-out) 0.0862 average queries per second </pre> </code> </blockquote> <p>The metric would be 22.52 qi/s , 310 qa/h, 39.7 u/s @ 50Mt (SF 1)</p> <p>The SUT is dual Xeon E5-2630, all in memory. The platform utilization is steadily above 2000% CPU (over 20/24 hardware threads busy on the DBMS). The DBMS is Virtuoso Open Source (<a href="https://github.com/v7fasttrack/virtuoso-opensource/" id="link-id0x2aab2c370168">v7fasttrack at github.com</a>, <a href="https://github.com/v7fasttrack/virtuoso-opensource/tree/feature/analytics" id="link-id0x2aab2c370308">feature/analytics</a> branch).</p> <p>The minimum update rate of 7/s was sustained, but fell short of the target of 70/s. In this run, most demand was put on the interactive queries. Different thread allocations would give different ratios of the metric components. The analytics mix, for example, is about 3x faster without other concurrent activity.</p> <p>Is this good or bad? I would say that this is possible but better can certainly be accomplished.</p> <p>The initial observation is that Q17 is the worst of the interactive lot. 3x better is easily accomplished by avoiding a basic stupidity. The query does the evil deed of checking for a substring in a URI. This is done in the wrong place and accounts for most of the time. The query is meant to test geo retrieval but ends up doing something quite different. Optimizing this right would by itself almost double the interactive score. There are some timeouts in the analytical run, which as such disqualifies the run. This is not a fully compliant result, but is close enough to give an idea of the dynamics. So we see that the experiment is definitely feasible, is reasonably defined, and that the dynamics seen make sense.</p> <p>As an initial comment of the workload mix, I'd say that interactive should have a few more very short point-lookups, to stress compilation times and give a higher absolute score of queries per second.</p> <p>Adjustments to the mix will depend on what we find out about scaling. As with SNB, it is likely that the workload will shift a little so this result might not be comparable with future ones.</p> <p>In the next SPB article, we will look closer at performance dynamics and choke points and will have an initial impression on scaling the workload.</p>
2014-11-13T16:19:42.399180-05:00
LDBC: Creating a Metric for SNB
http://www.openlinksw.com:443/weblog/oerling/?date=2014-11-13#1830
2014-11-13T21:09:00Z
<p>In the <a href="http://ldbcouncil.org/blog/making-it-interactive" id="link-id0x2aaad4d40f58"><i>Making It Interactive</i> post on the LDBC blog</a>, we were talking about composing an interactive <a href="http://ldbcouncil.org/developer/snb" id="link-id0x2aaad4d410a8">Social Network Benchmark (SNB)</a> metric. Now we will look at what this looks like in practice.</p> <p>A benchmark is known by its primary metric. An actual benchmark implementation may deal with endless complexity but the whole point of the exercise is to reduce this all to an extremely compact form, optimally a number or two.</p> <p>For SNB, we suggest clicks per second Interactive at scale (cpsI@ so many GB) as the primary metric. To each scale of the dataset corresponds a rate of update in the dataset's timeline (simulation time). When running the benchmark, the events in simulation time are transposed to a timeline in real time.</p> <p>Another way of expressing the metric is therefore acceleration factor at scale. In this example, we run a 300 GB database at an acceleration of 1.64; i.e., in the present example, we did 97 minutes of simulation time in 58 minutes of real time.</p> <p>Another key component of a benchmark is the full disclosure report (FDR). This is expected to enable any interested party to reproduce the experiment.</p> <p>The system under test (SUT) is Virtuoso running an SQL implementation of the workload at 300 GB (SF = 300). This run gives an idea of what an official report will look like but is not one yet. The implementation differs from the present specification in the following:</p> <ul> <li> <p>The SNB test driver is not used. Instead, the workload is read from the file system by stored procedures on the SUT. This is done to circumvent latencies in update scheduling in the test driver which would result in the SUT not reaching full platform utilization.</p> </li> <li> <p>The workload is extended by 2 short lookups, i.e., person profile view and post detail view. These are very short and serve to give the test more of an online flavor.</p> </li> <li> <p>The short queries appear in the report as multiple entries. This should not be the case. This inflates the clicks per second number but does not significantly affect the acceleration factor.</p> </li> </ul> <p>As a caveat, this metric will not be comparable with future ones.</p> <p>Aside from the composition of the report, the interesting point is that with the present workload, a 300 GB database keeps up with the simulation timeline on a commodity server, also when running updates. The query frequencies and run times are in <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20141111LdbcCreatingAMetricForSnb" id="link-id0x2aaad6fc6528">the full report</a>. We also produced a graphic showing the evolution of the throughput over a run of one hour -- </p> <p align="center"> <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20141111LdbcCreatingAMetricForSnb/ldbc-snb-qpm.png" id="link-id0x2aaad4cbdb68"><img src="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20141111LdbcCreatingAMetricForSnb/ldbc-snb-qpm.png" alt="ldbc-snb-qpm.png" width="640px" /> </a> <br /> <i>(click to embiggen)</i> </p> <p>We see steady throughput except for some slower minutes which correspond to database checkpoints. (A checkpoint, sometimes called a log checkpoint, is the operation which makes a database state durable outside of the transaction log.) If we run updates only at full platform, we get an acceleration of about 300x in memory for 20 minutes, then 10 minutes of nothing happening while the database is being checkpointed. This is measured with 6 2TB magnetic disks. Such a behavior is incompatible with an interactive workload. But with a checkpoint every 10 minutes and updates mixed with queries, checkpointing the database does not lead to impossible latencies. Thus, we do not get the TPC-C syndrome which requires tens of disks or several SSDs per core to run.</p> <p>This is a good thing for the benchmark, as we do not want to require unusual I/O systems for competition. Such a requirement would simply encourage people to ignore the specification for the point and would limit the number of qualifying results.</p> <p> <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20141111LdbcCreatingAMetricForSnb" id="link-id0x2aaad4cbe258">The full report</a> contains the details. This is also a template for later "real" FDRs. The supporting files are divided into test implementation and system configuration. With these materials plus the data generator, one should be able to repeat the results using a Virtuoso Open Source cut from <a href="https://github.com/v7fasttrack/virtuoso-opensource/" id="link-id0x2aaad7642ab8">v7fasttrack at github.com</a>, <a href="https://github.com/v7fasttrack/virtuoso-opensource/tree/feature/analytics" id="link-id0x2aaad7642c28">feature/analytics branch</a>.</p> <p>In later posts we will analyze the results a bit more and see how much improvement potential we find. The next SNB article will be about the business intelligence and graph analytics areas of SNB.</p>
2014-11-13T16:09:00.580195-05:00
On Universality and Core Competence
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-10-22#1829
2014-10-22T17:24:14Z
<p>I will here develop some ideas on the platform of <a href="http://www.cs.vu.nl/en/research/artificial-intelligence/kr/staff/boncz/index.asp" id="link-id0x2aab794ab9f8">Peter Boncz</a>'s inaugural lecture mentioned in <a href="http://www.openlinksw.com/weblog/oerling/?id=1826" id="link-id0x2aab7921f328">the previous post</a>. This is a high-level look at where the leading edge of analytics will be, now that the column store is mainstream.</p> <p>Peter's description of his domain was roughly as follows, summarized from memory:</p> <blockquote> <i> <p>The new chair is for data analysis and engines for this purpose. The data analysis engine includes the analytical DBMS but is a broader category. For example, the diverse parts of the big data chain (including preprocessing, noise elimination, feature extraction, natural language extraction, graph analytics, and so forth) fall under this category, and most of these things are usually not done in a DBMS. For anything that is big, the main challenge remains one of performance and time to solution. These things are being done, and will increasingly be done, on a platform with heterogenous features, e.g., CPU/GPU clusters, possibly custom hardware like FPGAs, etc. This is driven by factors of cost and energy efficiency. Different processing stages will sometimes be distributed over a wide area, as for example in instrument networks and any network infrastructure, which is wide area by definition.</p> <p>The design space of database and all that is around it is huge, and any exhaustive exploration is impossible. Development times are long, and a platform might take ten years to be mature. This is ill compatible with academic funding cycles. However, we should not leave all the research in this to industry, as industry maximizes profit, not innovation or absolute performance. Architecting data systems has aspects of an art. Consider the parallel with architecture of buildings: There are considerations of function, compatibility with environment, cost, restrictions arising from the materials at hand, and so forth. How a specific design will work cannot be known without experiment. The experiments themselves must be designed to make sense. This is not an exact science with clear-cut procedures and exact metrics of success.</p> </i> </blockquote> <p>This is the gist of Peter's description of our art. Peter's successes, best exemplified by <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x10ff99dc8">MonetDB</a> and <a href="http://dbpedia.org/resource/Vectorwise" id="link-id0x11200e6d8">Vectorwise</a>, arise from focus over a special problem area and from developing and systematically applying specific insights to a specific problem. This process led to the emergence of the column store, which is now a mainstream thing. The DBMS that does not do columns is by now behind the times.</p> <p>Needless to say, I am a great believer in core competence. Not every core competence is exactly the same. But a core competence needs to be broad enough so that its integral mastery and consistent application can produce a unit of value valuable in itself. What and how broad this is varies a great deal. Typically such a unit of value is something that is behind a "natural interface." This defies exhaustive definition but the examples below may give a hint. Looking at value chains and all diverse things in them that have a price tag may be another guideline.</p> <p>There is a sort of <a href="http://dbpedia.org/resource/Hegelian_dialectic" id="link-id0x10dffc548">Hegelian dialectic</a> to technology trends: At the start, it was generally believed that a <a href="http://dbpedia.org/resource/Database" id="link-id0xff8f1508">DBMS</a> would be universal like the <a href="http://dbpedia.org/resource/Operating_system" id="link-id0x112383048">operating system</a> itself, with a few products with very similar functionality covering the whole field. The antithesis came with <a href="http://dbpedia.org/resource/Michael_Stonebraker" id="link-id0x1124d3718">Michael Stonebraker</a> declaring that one size no longer fit all. Since then the <a href="http://dbpedia.org/resource/Online_transaction_processing" id="link-id0x10ff77b18">transactional (OLTP)</a> and <a href="http://dbpedia.org/resource/Online_analytical_processing" id="link-id0x1113877a8">analytical (OLAP)</a> sides are clearly divided. The eventual synthesis may be in the air, with pioneering work like <a href="http://hyper-db.de/" id="link-id0x10f62b958">HyPer</a> led by <a href="https://www.linkedin.com/pub/thomas-neumann/2/35/189" id="link-id0x11094d628">Thomas Neumann</a> of <a href="http://dbpedia.org/resource/Technische_Universit%C3%A4t_M%C3%BCnchen" id="link-id0x110a34158">TU München</a>. Peter, following his <a href="http://www.cwi.nl/brief/2013/humboldt-research-award-peter-boncz" id="link-id0x112a93358">Humbolt prize</a>, has spent a couple of days a week in Thomas's group, and I have joined him there a few times. The key to eventually bridging the gap would be compilation and adaptivity. If the workload is compiled on demand, then the right data structures could always be at hand.</p> <p>This might be the start of a shift similar to the column store turning the DBMS on its side, so to say.</p> <p>In the mainstream of software engineering, objects, abstractions and interfaces are held to be a value almost in and of themselves. Our science, that of performance, stands in apparent opposition to at least any naive application of the paradigm of objects and interfaces. Interfaces have a cost, and boxes limit transparency into performance. So inlining and merging distinct (in principle) processing phases is necessary for performance. Vectoring is one take on this: An interface that is crossed just a few times is much less harmful than one crossed a billion times. Using compilation, or at least type-and-data-structure-specific variants of operators and switching their application based on run-time observed behaviors, is another aspect of this.</p> <p>Information systems thus take on more attributes of nature, i.e., more interconnectedness and adaptive behaviors.</p> <p>Something quite universal might emerge from the highly problem-specific technology of the column store. The big scan, selective hash join plus aggregation, has been explored in slightly different ways by all of HyPer, Vectorwise, and <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x10ffbeb88">Virtuoso</a>.</p> <p>Interfaces are not good or bad, in and of themselves. Well-intentioned naïveté in their use is bad. As in nature, there are natural borders in the "technosphere"; declarative query languages, processor instruction sets, and network protocols are good examples. Behind a relatively narrow interface lies a world of complexity of which the unsuspecting have no idea. In biology, the cell membrane might be an analogy, but this is in all likelihood more permeable and diverse in function than the techno examples mentioned.</p> <p>With the experience of Vectorwise and later Virtuoso, it turns out that vectorization without compilation is good enough for <a href="http://www.tpc.org/tpch/" id="link-id0x10a6cc008">TPC-H</a>. Indeed, I see a few percent of gain at best from further breaking of interfaces and "biology-style" merging of operators and adding inter-stage communication and self-balancing. But TPC-H is not the end of all things, even though it is a sort of rite of passage: Jazz players will do their take on <i><a href="http://dbpedia.org/resource/On_Green_Dolphin_Street_(song)" id="link-id0xfce45c28">Green Dolphin Street</a></i> and <i><a href="http://dbpedia.org/resource/Summertime_(song)" id="link-id0x10e04daa8">Summertime</a></i>.</p> <p>Science is drawn towards a grand unification of all which is. Nature, on the other hand, discloses more and more diversity and special cases, the closer one looks. This may be true of physical things, but also of abstractions such as software systems or mathematics.</p> <p>So, let us look at the generalized DBMS, or the data analysis engine, as Peter put it. The use of DBMS technology is hampered by its interface, i.e., declarative query language. The well known counter-reactions to this are the <a href="http://dbpedia.org/resource/NoSQL" id="link-id0x2aab79e56bb8">NoSQL</a>, <a href="http://dbpedia.org/resource/MapReduce" id="link-id0xff54e4d8">MapReduce</a>, and <a href="http://dbpedia.org/resource/Graph_database" id="link-id0xfe5601b8">graph DB</a> memes, which expose lower level interfaces. But then the interface gets put in the whole wrong place, denying most of the things that make the analytics DBMS extremely good at what it does.</p> <p>We need better and smarter building blocks and interfaces at zero cost. We continue to need blocks of some sort, since algorithms would stop being understandable without any data/procedural abstraction. At run time, the blocks must overlap and interpenetrate: Scan plus hash plus reduction in one loop, for example. Inter-thread, inter-process status sharing for things like <i>top k</i> for faster convergence, for another. Vectorized execution of the <a href="http://dbpedia.org/resource/SIMD" id="link-id0x10e9d8c48">same algorithm on many data</a> for things like graph traversals. There are very good single blocks, like GPU graph algorithms, but interface and composability are ever the problem.</p> <p>So, we must unravel the package that encapsulates the wonders of the analytical DBMS. These consist of scan, hash/index lookup, partitioning, aggregation, expression evaluation, scheduling, message passing and related flow control for scale-out systems, just to mention a few. The complete list would be under 30 long, with blocks parameterized by data payload and specific computation.</p> <p>By putting these together in a few new ways, we will cover much more of the big data pipeline. <a href="http://dbpedia.org/resource/Just-in-time_compilation" id="link-id0x10f482d38">Just-in-time compilation</a> may well be the way to deliver these components in an application/environment tailored composition. Yes, keep talking about block diagrams, but never once believe that this represents how things work or ought to work. The algorithms are expressed as distinct things, but at the level of the physical manifestation, things are parallel and interleaved.</p> <p>The core skill for architecting the future of data analytics is correct discernment of abstraction and interface. What is generic enough to be broadly applicable yet concise enough to be usable? When should the computation move, and when should the data move? What are easy ways of talking about data location? How can protect the application developer be protected from various inevitable stupidities?</p> <p>No mistake about it, there are at present very few people with the background for formulating the blueprint for the generalized data pipeline. These will be mostly drawn from architects of DBMS. The prospective user is any present-day user of analytics DBMS, <a href="http://dbpedia.org/resource/Apache_Hadoop" id="link-id0x2aab79f6c878">Hadoop</a>, or the like. By and large, SQL has worked well within its area of applicability. If there had never been an anti-SQL rebel faction, SQL would not have been successful. Now that a broader workload definition calls for redefinition of interfaces, so as to use the best where it fits, there is a need for re-evaluation of the <a href="http://dbpedia.org/resource/Imperative_programming" id="link-id0x1117d4418">imperative</a> Vs. <a href="http://dbpedia.org/resource/Declarative_programming" id="link-id0x110b67598">declarative</a> question.</p> <p> <a href="http://dbpedia.org/resource/T._S._Eliot" id="link-id0x111756198">T. S. Eliot</a> once wrote that humankind cannot bear very much reality. It seems that we in reality can deconstruct the DBMS and redeploy the state of the art to serve novel purposes across a broader set of problems. This is a cross-over that slightly readjusts the mental frame of the DBMS expert but leaves the core precepts intact. In other words, this is a straightforward extension of core competence with no slide into the dilettantism of doing a little bit of everything.</p> <p>People like MapReduce and stand-alone graph programming frameworks, because these do one specific thing and are readily understood. By and large, these are orders of magnitude simpler than the DBMS. Even when the DBMS provides in-process Java or CLR, these are rarely used. The single-purpose framework is a much narrower core competence, and thus less exclusive, than the high art of the DBMS, plus it has a faster platform development cycle.</p> <p>In the short term, we will look at opening the SQL internal toolbox for graph analytics applications. I was discussing this idea with Thomas Neumann at Peter Boncz's party. He asked who would be the user. I answered that doing good parallel algorithms, even with powerful shorthands, was an expert task; so the people doing new types of analytics would be mostly on the system vendor side. However, modifying such for input selection and statistics gathering would be no harder than doing the same with ready-made SQL reports.</p> <p>There is significant possibility for generalization of the leading edge of database. How will this fare against single-model frameworks? We hope to shed some light on this in the final phase of <a href="http://www.ldbc.eu/" id="link-id0x112527518">LDBC</a> and beyond.</p>
2014-10-22T13:24:14.900227-04:00
Inaugural Lecture of Prof. Boncz at VU Amsterdam
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-10-22#1828
2014-10-22T17:24:07Z
<p>Last Friday, I attended the inaugural lecture of Professor <a href="http://www.cs.vu.nl/en/research/artificial-intelligence/kr/staff/boncz/index.asp" id="link-id0x10f0a64e8">Peter Boncz</a> at the <a href="http://dbpedia.org/page/VU_University_Amsterdam" id="link-id0x10e23e948">VU University Amsterdam</a>. As the reader is likely to know, Peter is one of the database luminaries of the 21st century, known among other things for architecting <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x10fb03ab8">MonetDB</a> and <a href="http://dbpedia.org/page/Vectorwise" id="link-id0x10eb1b1f8">Actian Vector (Vectorwise)</a> and publishing a stellar succession of core database papers. </p> <p>The lecture touched on the fact of the data economy and the possibilities of E-science. Peter proceeded to address issues of ethics of cyberspace and the fact of legal and regulatory practice trailing far behind the factual dynamics of cyberspace. In conclusion, Peter gave some pointers to his research agenda; for example, use of just-in-time compilation for fusing problem-specific logic with infrastructure software like databases for both performance and architecture adaptivity.</p> <p> </p> <p>There was later a party in Amsterdam with many of the local database people as well as some from further away, e.g., <a href="https://www.linkedin.com/pub/thomas-neumann/2/35/189" id="link-id0xfe423dd8">Thomas Neumann</a> of Munich, and <a href="https://www.linkedin.com/in/marcinzukowski" id="link-id0x103d43218">Marcin Zukowsky</a>, Vectorwise founder and initial CEO.</p> <p> </p> <p>I should have had the presence of mind to prepare a speech for Peter. <a href="http://homepages.cwi.nl/~manegold/" id="link-id0x10d769278">Stefan Manegold</a> of <a href="http://www.cwi.nl/" id="link-id0x100e502b8">CWI</a> did give a short address at the party, while presenting the gifts from Peter's CWI colleagues. To this I will add my belated part here, as follows:</p> <blockquote> <i><p>If I were to describe Prof. Boncz, our friend, co-worker, and mentor, in one word, this would be man of knowledge. If physicists define energy as that which can do work, then knowledge would be that which can do meaningful work. A schematic in itself does nothing. Knowledge is needed to bring this to life. Yet this is more than an outstanding specialist skill, as this implies discerning the right means in the right context and includes the will and ability to go through with this. As Peter now takes on the mantle of professor, the best students will, I am sure, not fail to recognize excellence and be accordingly inspired to strive for the sort of industry changing accomplishments we have come to associate with Peter's career so far. This is what our world needs. A big cheer for Prof. Boncz!</p> </i> </blockquote> <p>I did talk to many at the party, especially <a href="http://homepages.cwi.nl/~duc/" id="link-id0x1099e2b78">Pham Minh Duc</a>, who is doing schema-aware RDF in MonetDB, and many others among the excellent team at CWI. Stefan Manegold told me about <a href="http://www.rethinkbig-project.eu/" id="link-id0x2aabb478aab8">Rethink Big</a>, an FP7 for big data policy recommendations. I was meant to be an advisor and still hope to go to one of their meetings for some networking about policy. On the other hand, the EU agenda and priorities, as discussed with, for example, <a href="https://www.linkedin.com/in/stefanobertolo" id="link-id0x103da2e78">Stefano Bertolo</a>, are, as far as I am concerned, on the right track: The science of performance must meet with the real, or at least realistic, data. Peter did not fail to mention this same truth in his lecture: Spinoffs play a key part in research, and exposure to the world out there gives research both focus and credibility. As René Char put it in his poem <i>L'Allumette (The Matchstick</i>), "La tête seule à pouvoir de prendre feu au contact d'une réalité dure." ("The head alone has power to catch fire at the touch of hard reality.") Great deeds need great challenges, and there is nothing like reality to exceed man's imagination.</p> <p>For my part, I was advertising the imminent advances in the <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x10e99fb98">Virtuoso</a> RDF and graph functionality. Now that the SQL part, which is anyway the necessary foundation for all this, is really very competent, it is time to deploy these same things in slightly new ways. This will produce graph analytics and structure-aware RDF to match relational performance while keeping schema-last-ness. Anyway, the claim has been made; we will see how it is delivered during the final phase of <a href="http://www.ldbc.eu/" id="link-id0x10db84bc8">LDBC</a> and <a href="http://geoknow.eu/" id="link-id0x10586d568">Geoknow</a>.</p>
2014-10-22T13:24:07.576952-04:00
On Universality and Core Competence
http://www.openlinksw.com:443/weblog/oerling/?date=2014-10-22#1827
2014-10-22T17:23:58Z
<p>I will here develop some ideas on the platform of <a href="http://www.cs.vu.nl/en/research/artificial-intelligence/kr/staff/boncz/index.asp" id="link-id0x2aab794ab9f8">Peter Boncz</a>'s inaugural lecture mentioned in <a href="http://www.openlinksw.com/weblog/oerling/?id=1826" id="link-id0x2aab7921f328">the previous post</a>. This is a high-level look at where the leading edge of analytics will be, now that the column store is mainstream.</p> <p>Peter's description of his domain was roughly as follows, summarized from memory:</p> <blockquote> <i> <p>The new chair is for data analysis and engines for this purpose. The data analysis engine includes the analytical DBMS but is a broader category. For example, the diverse parts of the big data chain (including preprocessing, noise elimination, feature extraction, natural language extraction, graph analytics, and so forth) fall under this category, and most of these things are usually not done in a DBMS. For anything that is big, the main challenge remains one of performance and time to solution. These things are being done, and will increasingly be done, on a platform with heterogenous features, e.g., CPU/GPU clusters, possibly custom hardware like FPGAs, etc. This is driven by factors of cost and energy efficiency. Different processing stages will sometimes be distributed over a wide area, as for example in instrument networks and any network infrastructure, which is wide area by definition.</p> <p>The design space of database and all that is around it is huge, and any exhaustive exploration is impossible. Development times are long, and a platform might take ten years to be mature. This is ill compatible with academic funding cycles. However, we should not leave all the research in this to industry, as industry maximizes profit, not innovation or absolute performance. Architecting data systems has aspects of an art. Consider the parallel with architecture of buildings: There are considerations of function, compatibility with environment, cost, restrictions arising from the materials at hand, and so forth. How a specific design will work cannot be known without experiment. The experiments themselves must be designed to make sense. This is not an exact science with clear-cut procedures and exact metrics of success.</p> </i> </blockquote> <p>This is the gist of Peter's description of our art. Peter's successes, best exemplified by <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x10ff99dc8">MonetDB</a> and <a href="http://dbpedia.org/resource/Vectorwise" id="link-id0x11200e6d8">Vectorwise</a>, arise from focus over a special problem area and from developing and systematically applying specific insights to a specific problem. This process led to the emergence of the column store, which is now a mainstream thing. The DBMS that does not do columns is by now behind the times.</p> <p>Needless to say, I am a great believer in core competence. Not every core competence is exactly the same. But a core competence needs to be broad enough so that its integral mastery and consistent application can produce a unit of value valuable in itself. What and how broad this is varies a great deal. Typically such a unit of value is something that is behind a "natural interface." This defies exhaustive definition but the examples below may give a hint. Looking at value chains and all diverse things in them that have a price tag may be another guideline.</p> <p>There is a sort of <a href="http://dbpedia.org/resource/Hegelian_dialectic" id="link-id0x10dffc548">Hegelian dialectic</a> to technology trends: At the start, it was generally believed that a <a href="http://dbpedia.org/resource/Database" id="link-id0xff8f1508">DBMS</a> would be universal like the <a href="http://dbpedia.org/resource/Operating_system" id="link-id0x112383048">operating system</a> itself, with a few products with very similar functionality covering the whole field. The antithesis came with <a href="http://dbpedia.org/resource/Michael_Stonebraker" id="link-id0x1124d3718">Michael Stonebraker</a> declaring that one size no longer fit all. Since then the <a href="http://dbpedia.org/resource/Online_transaction_processing" id="link-id0x10ff77b18">transactional (OLTP)</a> and <a href="http://dbpedia.org/resource/Online_analytical_processing" id="link-id0x1113877a8">analytical (OLAP)</a> sides are clearly divided. The eventual synthesis may be in the air, with pioneering work like <a href="http://hyper-db.de/" id="link-id0x10f62b958">HyPer</a> led by <a href="https://www.linkedin.com/pub/thomas-neumann/2/35/189" id="link-id0x11094d628">Thomas Neumann</a> of <a href="http://dbpedia.org/resource/Technische_Universit%C3%A4t_M%C3%BCnchen" id="link-id0x110a34158">TU München</a>. Peter, following his <a href="http://www.cwi.nl/brief/2013/humboldt-research-award-peter-boncz" id="link-id0x112a93358">Humbolt prize</a>, has spent a couple of days a week in Thomas's group, and I have joined him there a few times. The key to eventually bridging the gap would be compilation and adaptivity. If the workload is compiled on demand, then the right data structures could always be at hand.</p> <p>This might be the start of a shift similar to the column store turning the DBMS on its side, so to say.</p> <p>In the mainstream of software engineering, objects, abstractions and interfaces are held to be a value almost in and of themselves. Our science, that of performance, stands in apparent opposition to at least any naive application of the paradigm of objects and interfaces. Interfaces have a cost, and boxes limit transparency into performance. So inlining and merging distinct (in principle) processing phases is necessary for performance. Vectoring is one take on this: An interface that is crossed just a few times is much less harmful than one crossed a billion times. Using compilation, or at least type-and-data-structure-specific variants of operators and switching their application based on run-time observed behaviors, is another aspect of this.</p> <p>Information systems thus take on more attributes of nature, i.e., more interconnectedness and adaptive behaviors.</p> <p>Something quite universal might emerge from the highly problem-specific technology of the column store. The big scan, selective hash join plus aggregation, has been explored in slightly different ways by all of HyPer, Vectorwise, and <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x10ffbeb88">Virtuoso</a>.</p> <p>Interfaces are not good or bad, in and of themselves. Well-intentioned naïveté in their use is bad. As in nature, there are natural borders in the "technosphere"; declarative query languages, processor instruction sets, and network protocols are good examples. Behind a relatively narrow interface lies a world of complexity of which the unsuspecting have no idea. In biology, the cell membrane might be an analogy, but this is in all likelihood more permeable and diverse in function than the techno examples mentioned.</p> <p>With the experience of Vectorwise and later Virtuoso, it turns out that vectorization without compilation is good enough for <a href="http://www.tpc.org/tpch/" id="link-id0x10a6cc008">TPC-H</a>. Indeed, I see a few percent of gain at best from further breaking of interfaces and "biology-style" merging of operators and adding inter-stage communication and self-balancing. But TPC-H is not the end of all things, even though it is a sort of rite of passage: Jazz players will do their take on <i><a href="http://dbpedia.org/resource/On_Green_Dolphin_Street_(song)" id="link-id0xfce45c28">Green Dolphin Street</a></i> and <i><a href="http://dbpedia.org/resource/Summertime_(song)" id="link-id0x10e04daa8">Summertime</a></i>.</p> <p>Science is drawn towards a grand unification of all which is. Nature, on the other hand, discloses more and more diversity and special cases, the closer one looks. This may be true of physical things, but also of abstractions such as software systems or mathematics.</p> <p>So, let us look at the generalized DBMS, or the data analysis engine, as Peter put it. The use of DBMS technology is hampered by its interface, i.e., declarative query language. The well known counter-reactions to this are the <a href="http://dbpedia.org/resource/NoSQL" id="link-id0x2aab79e56bb8">NoSQL</a>, <a href="http://dbpedia.org/resource/MapReduce" id="link-id0xff54e4d8">MapReduce</a>, and <a href="http://dbpedia.org/resource/Graph_database" id="link-id0xfe5601b8">graph DB</a> memes, which expose lower level interfaces. But then the interface gets put in the whole wrong place, denying most of the things that make the analytics DBMS extremely good at what it does.</p> <p>We need better and smarter building blocks and interfaces at zero cost. We continue to need blocks of some sort, since algorithms would stop being understandable without any data/procedural abstraction. At run time, the blocks must overlap and interpenetrate: Scan plus hash plus reduction in one loop, for example. Inter-thread, inter-process status sharing for things like <i>top k</i> for faster convergence, for another. Vectorized execution of the <a href="http://dbpedia.org/resource/SIMD" id="link-id0x10e9d8c48">same algorithm on many data</a> for things like graph traversals. There are very good single blocks, like GPU graph algorithms, but interface and composability are ever the problem.</p> <p>So, we must unravel the package that encapsulates the wonders of the analytical DBMS. These consist of scan, hash/index lookup, partitioning, aggregation, expression evaluation, scheduling, message passing and related flow control for scale-out systems, just to mention a few. The complete list would be under 30 long, with blocks parameterized by data payload and specific computation.</p> <p>By putting these together in a few new ways, we will cover much more of the big data pipeline. <a href="http://dbpedia.org/resource/Just-in-time_compilation" id="link-id0x10f482d38">Just-in-time compilation</a> may well be the way to deliver these components in an application/environment tailored composition. Yes, keep talking about block diagrams, but never once believe that this represents how things work or ought to work. The algorithms are expressed as distinct things, but at the level of the physical manifestation, things are parallel and interleaved.</p> <p>The core skill for architecting the future of data analytics is correct discernment of abstraction and interface. What is generic enough to be broadly applicable yet concise enough to be usable? When should the computation move, and when should the data move? What are easy ways of talking about data location? How can protect the application developer be protected from various inevitable stupidities?</p> <p>No mistake about it, there are at present very few people with the background for formulating the blueprint for the generalized data pipeline. These will be mostly drawn from architects of DBMS. The prospective user is any present-day user of analytics DBMS, <a href="http://dbpedia.org/resource/Apache_Hadoop" id="link-id0x2aab79f6c878">Hadoop</a>, or the like. By and large, SQL has worked well within its area of applicability. If there had never been an anti-SQL rebel faction, SQL would not have been successful. Now that a broader workload definition calls for redefinition of interfaces, so as to use the best where it fits, there is a need for re-evaluation of the <a href="http://dbpedia.org/resource/Imperative_programming" id="link-id0x1117d4418">imperative</a> Vs. <a href="http://dbpedia.org/resource/Declarative_programming" id="link-id0x110b67598">declarative</a> question.</p> <p> <a href="http://dbpedia.org/resource/T._S._Eliot" id="link-id0x111756198">T. S. Eliot</a> once wrote that humankind cannot bear very much reality. It seems that we in reality can deconstruct the DBMS and redeploy the state of the art to serve novel purposes across a broader set of problems. This is a cross-over that slightly readjusts the mental frame of the DBMS expert but leaves the core precepts intact. In other words, this is a straightforward extension of core competence with no slide into the dilettantism of doing a little bit of everything.</p> <p>People like MapReduce and stand-alone graph programming frameworks, because these do one specific thing and are readily understood. By and large, these are orders of magnitude simpler than the DBMS. Even when the DBMS provides in-process Java or CLR, these are rarely used. The single-purpose framework is a much narrower core competence, and thus less exclusive, than the high art of the DBMS, plus it has a faster platform development cycle.</p> <p>In the short term, we will look at opening the SQL internal toolbox for graph analytics applications. I was discussing this idea with Thomas Neumann at Peter Boncz's party. He asked who would be the user. I answered that doing good parallel algorithms, even with powerful shorthands, was an expert task; so the people doing new types of analytics would be mostly on the system vendor side. However, modifying such for input selection and statistics gathering would be no harder than doing the same with ready-made SQL reports.</p> <p>There is significant possibility for generalization of the leading edge of database. How will this fare against single-model frameworks? We hope to shed some light on this in the final phase of <a href="http://www.ldbc.eu/" id="link-id0x112527518">LDBC</a> and beyond.</p>
2014-10-22T13:23:58.362941-04:00
Inaugural Lecture of Prof. Boncz at VU Amsterdam
http://www.openlinksw.com:443/weblog/oerling/?date=2014-10-22#1826
2014-10-22T17:21:37Z
<p>Last Friday, I attended the inaugural lecture of Professor <a href="http://www.cs.vu.nl/en/research/artificial-intelligence/kr/staff/boncz/index.asp" id="link-id0x10f0a64e8">Peter Boncz</a> at the <a href="http://dbpedia.org/page/VU_University_Amsterdam" id="link-id0x10e23e948">VU University Amsterdam</a>. As the reader is likely to know, Peter is one of the database luminaries of the 21st century, known among other things for architecting <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x10fb03ab8">MonetDB</a> and <a href="http://dbpedia.org/page/Vectorwise" id="link-id0x10eb1b1f8">Actian Vector (Vectorwise)</a> and publishing a stellar succession of core database papers. </p> <p>The lecture touched on the fact of the data economy and the possibilities of E-science. Peter proceeded to address issues of ethics of cyberspace and the fact of legal and regulatory practice trailing far behind the factual dynamics of cyberspace. In conclusion, Peter gave some pointers to his research agenda; for example, use of just-in-time compilation for fusing problem-specific logic with infrastructure software like databases for both performance and architecture adaptivity.</p> <p> </p> <p>There was later a party in Amsterdam with many of the local database people as well as some from further away, e.g., <a href="https://www.linkedin.com/pub/thomas-neumann/2/35/189" id="link-id0xfe423dd8">Thomas Neumann</a> of Munich, and <a href="https://www.linkedin.com/in/marcinzukowski" id="link-id0x103d43218">Marcin Zukowsky</a>, Vectorwise founder and initial CEO.</p> <p> </p> <p>I should have had the presence of mind to prepare a speech for Peter. <a href="http://homepages.cwi.nl/~manegold/" id="link-id0x10d769278">Stefan Manegold</a> of <a href="http://www.cwi.nl/" id="link-id0x100e502b8">CWI</a> did give a short address at the party, while presenting the gifts from Peter's CWI colleagues. To this I will add my belated part here, as follows:</p> <blockquote> <i><p>If I were to describe Prof. Boncz, our friend, co-worker, and mentor, in one word, this would be man of knowledge. If physicists define energy as that which can do work, then knowledge would be that which can do meaningful work. A schematic in itself does nothing. Knowledge is needed to bring this to life. Yet this is more than an outstanding specialist skill, as this implies discerning the right means in the right context and includes the will and ability to go through with this. As Peter now takes on the mantle of professor, the best students will, I am sure, not fail to recognize excellence and be accordingly inspired to strive for the sort of industry changing accomplishments we have come to associate with Peter's career so far. This is what our world needs. A big cheer for Prof. Boncz!</p> </i> </blockquote> <p>I did talk to many at the party, especially <a href="http://homepages.cwi.nl/~duc/" id="link-id0x1099e2b78">Pham Minh Duc</a>, who is doing schema-aware RDF in MonetDB, and many others among the excellent team at CWI. Stefan Manegold told me about <a href="http://www.rethinkbig-project.eu/" id="link-id0x2aabb478aab8">Rethink Big</a>, an FP7 for big data policy recommendations. I was meant to be an advisor and still hope to go to one of their meetings for some networking about policy. On the other hand, the EU agenda and priorities, as discussed with, for example, <a href="https://www.linkedin.com/in/stefanobertolo" id="link-id0x103da2e78">Stefano Bertolo</a>, are, as far as I am concerned, on the right track: The science of performance must meet with the real, or at least realistic, data. Peter did not fail to mention this same truth in his lecture: Spinoffs play a key part in research, and exposure to the world out there gives research both focus and credibility. As René Char put it in his poem <i>L'Allumette (The Matchstick</i>), "La tête seule à pouvoir de prendre feu au contact d'une réalité dure." ("The head alone has power to catch fire at the touch of hard reality.") Great deeds need great challenges, and there is nothing like reality to exceed man's imagination.</p> <p>For my part, I was advertising the imminent advances in the <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x10e99fb98">Virtuoso</a> RDF and graph functionality. Now that the SQL part, which is anyway the necessary foundation for all this, is really very competent, it is time to deploy these same things in slightly new ways. This will produce graph analytics and structure-aware RDF to match relational performance while keeping schema-last-ness. Anyway, the claim has been made; we will see how it is delivered during the final phase of <a href="http://www.ldbc.eu/" id="link-id0x10db84bc8">LDBC</a> and <a href="http://geoknow.eu/" id="link-id0x10586d568">Geoknow</a>.</p>
2014-10-22T13:21:37.296395-04:00
In Hoc Signo Vinces (part 20 of n): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-10-06#1825
2014-10-06T17:53:24Z
<p>In the <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x2aab51a32ce8">introduction to scale out</a> piece, I promised to address the matter of data-to-memory ratio, and to talk about when scale-out makes sense. Here we will see that scale-out makes sense whenever data does not fit in memory on a single commodity server. The gains in processing power are immediate, even when going from one box to just two, with both systems having all in memory.</p> <p>As an initial take on the issue we run 100 GB and 1000 GB on the test system. 100 GB is trivially in memory, 1000 GB is not, as the memory is 384 GB total, of which 360 GB may be used for the processes.</p> <p>We run 2 workloads on the 100 GB database, having pre-loaded the data in memory:</p> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr style="text-align:center;"> <th>run</th> <th>power</th> <th>throughput</th> <th>composite</th> </tr> <tr> <th>1</th> <td>349,027.7</td> <td>420,503.1</td> <td>383,102.1</td> </tr> <tr> <th>2</th> <td>387,890.3</td> <td>433,066.6</td> <td>409,856.5</td> </tr> </table> <p>This is directly comparable to the 100 GB single-server results. Comparing the second runs, we see a 1.53x gain in power and a 1.8x gain in throughput from 2x the platform. This is fully on the level for a workload that is not trivially parallel, as we have seen in the previous articles. The difference between the first and second runs at 100 GB comes, for both single-server and cluster, from the latency of allocating transient query memory. For an official run, where the weakest link is the first power test, this would simply have to be pre-allocated.</p> <p>We run 2 workloads on the 1000 GB database, starting from cold.</p> <p>The result is:</p> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr style="text-align:center;"> <th>run</th> <th>power</th> <th>throughput</th> <th>composite</th> </tr> <tr> <th>1</th> <td>136,744.5</td> <td>147,374.6</td> <td>141,960.1</td> </tr> <tr> <th>2</th> <td>199,652.0</td> <td>125,161.1</td> <td>158,078.0</td> </tr> </table> <p>The 1000 GB result is not for competition with this platform; more memory would be needed. For actual applications, the numbers are still in the usable range, though.</p> <p>The 1000 GB setup uses 4 SSDs for storage, one per server process. The server processes are each bound to their own physical CPU.</p> <p>We look at the meters: 32M pages (8M per process) are in memory at each time. Over the 2 benchmark executions there are a total of 494M disk reads. The total CPU time is 165,674 seconds of CPU, of which about 10% are system, over 10,063 seconds of real-time. Cumulative disk-read wait-time is 130,177 s. This gives an average disk read throughput of 384 MB/s.</p> <p>This is easily sustained by 4 SSDs; in practice, the maximum throughput we see for reading is 1 GB/s (256 MB/s per SSD). Newer SSDs would do maybe twice that. Using rotating media would not be an option.</p> <p>Without the drop in CPU caused by waiting for SSD, we would have numbers very close to the 100 GB numbers. </p> <p>The interconnect traffic for the two runs was 1,077 GB with no message compression. The write block time was 448 seconds of thread-time. So we see that blocking on write hurts platform utilization when running under optimal conditions, but compared to going to secondary storage, it is not a large factor.</p> <p>The 1000 GB scale has a transient peak memory consumption of 42 GB. This consists of hash-join build sides and GROUP BYs. The greatest memory consumers are Q9 with 9 GB, Q13 with 11 GB, and Q16 with 7 GB. Having many of these at a time drives up the transient peak. The peak gets higher as the scale grows, also because a larger scale requires more concurrent query streams. At the 384 GB for 1000 GB ratio, we do not yet get into memory saving plans like hash joins in many passes or index use instead of hash. When the data size grows, replicated hash build sides will become less convenient, and communication will increase. Q9 and Q13 can be done by index with almost no transient memory, but these plans are easily 3x less efficient for CPU. These will probably help at 3000 GB and be necessary at least part of the time at 10,000 GB.</p> <p>The I/O volume in MB per index over the 2 executions is:</p> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr style="text-align:center;"> <th>index</th> <th>MB</th> </tr> <tr> <th><code>LINEITEM</code> </th> <td>1,987,483</td> </tr> <tr> <th><code>ORDERS</code> </th> <td>1,440,526</td> </tr> <tr> <th><code>PARTSUPP</code> </th> <td>199,335</td> </tr> <tr> <th><code>PART</code> </th> <td>161,717</td> </tr> <tr> <th><code>CUSTOMER</code> </th> <td>43,276</td> </tr> <tr> <th><code>O_CK</code> </th> <td>19,085</td> </tr> <tr> <th><code>SUPPLIER</code> </th> <td>13,393</td> </tr> </table> <p>Of this, maybe 600 GB could be saved by stream compressing <code>o_comment</code>. Otherwise this cannot be helped without adding memory. The <code>lineitem</code> reads are mostly for <code>l_extendedprice</code>, which is not compressible. If compressing <code>o_comment</code> made <code>l_extendedprice</code> always fit in memory, then there would be a radical drop in I/O. Also, as a matter of fact, the buffer management policy of least-recently-used works the very worst for big scans, specifically those of <code>l_extendedprice</code>: If the head is replaced when reading the tail, and the next read starts from the head, then the whole table/column is read all over again. Caching policies that specially recognized scans of this sort could further reduce I/O. Clustering <code>lineitems</code>/<code>orders</code> on <code>date</code>, as <a href="http://dbpedia.org/page/Vectorwise" id="link-id0x2aab50f9b008">Actian Vector</a> <a href="http://www.tpc.org/tpch/" id="link-id0x2aab51845d48">TPC-H</a> implementations do, also starts yielding a greater gain when not running from memory: One column (e.g., <code>l_shipdate</code>) may be scanned for the whole table but, if the matches are bunched together, then most of <code>l_extendedprice</code> will not be read at all. Still, if going for top ranks in the races, all will be from memory, or at least there will be SSDs with read throughput around 150 MB/s per core, so these tricks become relatively less important.</p> <p>In the 100 GB numerical quantities summaries, we see much the same picture as in the single-server. Queries get faster, but their relative times are not radically different. The throughput test (many queries at a time) times are more or less multiples of the power (single user) times. This picture breaks at 1000 GB where I/O first drops the performance to under half and introduces huge variation in execution times within a single query. The time entirely depends on which queries are running along with or right before the execution and on whether these have the same or different working sets. All the streams have the same queries with different parameters, but the query order in each stream is different.</p> <p>The numerical quantities follow for all the runs. Note that the first 1000 GB run is cold. A competition grade 1000 GB result can be made with double the memory, and the more CPU the better. We will try one at Amazon in a bit.</p> <p> *** </p> <p>The conclusion is that scale-out pays from the get-go. At present prices, a system with twice the power of a single node of the test system is cost effective. Scales of up to 500 GB are single commodity server, under $10K. Rather than going from a mid-to-large dual-socket box to a quad-socket box, one is likely to be better off having two cheaper dual-socket boxes. These are also readily available on clouds, whereas scale-up configurations are not. Onwards of 1 TB, a cluster is expected to clearly win. At 3 TB, a commodity cluster will clearly be the better deal for both price and absolute performance.</p> <h2>100 GB Run 1</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>October 3, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>100</td> </tr> <tr> <th style="text-align:right">Total Data Storage/Database Size</th> <td>0M</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>5</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td>349,027.7</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td>420,503.1</td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@100GB)</th> <td>383,102.1</td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td>94.273000 seconds</td> </tr> </table> <h3>Duration of stream execution</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>10/03/2014 15:05:07</td> <td>10/03/2014 15:05:40</td> <td>0:00:33</td> </tr> <tr> <th>Stream 1</th> <td>10/03/2014 15:05:42</td> <td>10/03/2014 15:07:15</td> <td>0:01:33</td> </tr> <tr> <th>Stream 2</th> <td>10/03/2014 15:05:42</td> <td>10/03/2014 15:07:15</td> <td>0:01:33</td> </tr> <tr> <th>Stream 3</th> <td>10/03/2014 15:05:42</td> <td>10/03/2014 15:07:16</td> <td>0:01:34</td> </tr> <tr> <th>Stream 4</th> <td>10/03/2014 15:05:42</td> <td>10/03/2014 15:07:14</td> <td>0:01:32</td> </tr> <tr> <th>Stream 5</th> <td>10/03/2014 15:05:42</td> <td>10/03/2014 15:07:15</td> <td>0:01:33</td> </tr> <tr> <th>Refresh 0</th> <td>10/03/2014 15:05:07</td> <td>10/03/2014 15:05:13</td> <td>0:00:06</td> </tr> <tr> <th> </th> <td>10/03/2014 15:05:41</td> <td>10/03/2014 15:05:42</td> <td>0:00:01</td> </tr> <tr> <th>Refresh 1</th> <td>10/03/2014 15:06:48</td> <td>10/03/2014 15:07:03</td> <td>0:00:15</td> </tr> <tr> <th>Refresh 2</th> <td>10/03/2014 15:05:42</td> <td>10/03/2014 15:06:06</td> <td>0:00:24</td> </tr> <tr> <th>Refresh 3</th> <td>10/03/2014 15:06:06</td> <td>10/03/2014 15:06:20</td> <td>0:00:14</td> </tr> <tr> <th>Refresh 4</th> <td>10/03/2014 15:06:20</td> <td>10/03/2014 15:06:35</td> <td>0:00:15</td> </tr> <tr> <th>Refresh 5</th> <td>10/03/2014 15:06:35</td> <td>10/03/2014 15:06:48</td> <td>0:00:13</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> </tr> <tr> <th>Stream 0</th> <td>2.045198</td> <td>0.337315</td> <td>1.129548</td> <td>0.327029</td> <td>1.230955</td> <td>0.473090</td> <td>0.979096</td> <td>0.852639</td> </tr> <tr> <th>Stream 1</th> <td>4.521951</td> <td>0.596538</td> <td>3.464342</td> <td>1.167101</td> <td>3.944699</td> <td>1.744325</td> <td>5.442328</td> <td>4.706185</td> </tr> <tr> <th>Stream 2</th> <td>4.678728</td> <td>0.837205</td> <td>3.594060</td> <td>1.911751</td> <td>3.942459</td> <td>0.947788</td> <td>3.821267</td> <td>4.686319</td> </tr> <tr> <th>Stream 3</th> <td>5.126384</td> <td>0.932394</td> <td>0.961762</td> <td>1.043759</td> <td>5.359990</td> <td>1.035597</td> <td>3.056079</td> <td>5.803445</td> </tr> <tr> <th>Stream 4</th> <td>4.497118</td> <td>0.381036</td> <td>4.665412</td> <td>1.224975</td> <td>5.316591</td> <td>1.666253</td> <td>2.297872</td> <td>6.425171</td> </tr> <tr> <th>Stream 5</th> <td>4.080968</td> <td>0.493741</td> <td>4.416305</td> <td>0.879202</td> <td>5.705877</td> <td>1.615987</td> <td>3.846881</td> <td>3.346686</td> </tr> <tr> <th>Min Qi</th> <td>4.080968</td> <td>0.381036</td> <td>0.961762</td> <td>0.879202</td> <td>3.942459</td> <td>0.947788</td> <td>2.297872</td> <td>3.346686</td> </tr> <tr> <th>Max Qi</th> <td>5.126384</td> <td>0.932394</td> <td>4.665412</td> <td>1.911751</td> <td>5.705877</td> <td>1.744325</td> <td>5.442328</td> <td>6.425171</td> </tr> <tr> <th>Avg Qi</th> <td>4.581030</td> <td>0.648183</td> <td>3.420376</td> <td>1.245358</td> <td>4.853923</td> <td>1.401990</td> <td>3.692885</td> <td>4.993561</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>3.575916</td> <td>2.786656</td> <td>1.579488</td> <td>0.611454</td> <td>3.132460</td> <td>0.685095</td> <td>0.955559</td> <td>1.060110</td> </tr> <tr> <th>Stream 1</th> <td>9.551437</td> <td>7.187181</td> <td>5.816455</td> <td>2.004946</td> <td>9.461347</td> <td>5.624020</td> <td>5.517677</td> <td>2.924265</td> </tr> <tr> <th>Stream 2</th> <td>9.637427</td> <td>6.641804</td> <td>6.359532</td> <td>2.412576</td> <td>8.819754</td> <td>3.335494</td> <td>4.549792</td> <td>3.163920</td> </tr> <tr> <th>Stream 3</th> <td>11.041451</td> <td>6.464479</td> <td>6.982671</td> <td>3.272975</td> <td>8.342983</td> <td>3.448635</td> <td>4.405911</td> <td>2.886393</td> </tr> <tr> <th>Stream 4</th> <td>8.860228</td> <td>6.754529</td> <td>7.065501</td> <td>3.225236</td> <td>8.789565</td> <td>3.419165</td> <td>4.240718</td> <td>2.399092</td> </tr> <tr> <th>Stream 5</th> <td>7.339672</td> <td>8.121027</td> <td>6.261988</td> <td>2.711946</td> <td>8.764934</td> <td>3.106366</td> <td>6.544712</td> <td>3.472092</td> </tr> <tr> <th>Min Qi</th> <td>7.339672</td> <td>6.464479</td> <td>5.816455</td> <td>2.004946</td> <td>8.342983</td> <td>3.106366</td> <td>4.240718</td> <td>2.399092</td> </tr> <tr> <th>Max Qi</th> <td>11.041451</td> <td>8.121027</td> <td>7.065501</td> <td>3.272975</td> <td>9.461347</td> <td>5.624020</td> <td>6.544712</td> <td>3.472092</td> </tr> <tr> <th>Avg Qi</th> <td>9.286043</td> <td>7.033804</td> <td>6.497229</td> <td>2.725536</td> <td>8.835717</td> <td>3.786736</td> <td>5.051762</td> <td>2.969152</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>1.433789</td> <td>0.972152</td> <td>0.780247</td> <td>1.287222</td> <td>1.360084</td> <td>0.254051</td> <td>6.201742</td> <td>1.219707</td> </tr> <tr> <th>Stream 1</th> <td>3.398354</td> <td>2.591249</td> <td>3.021207</td> <td>4.663204</td> <td>4.775704</td> <td>1.116547</td> <td>8.770115</td> <td>5.643550</td> </tr> <tr> <th>Stream 2</th> <td>6.811520</td> <td>3.411846</td> <td>2.634076</td> <td>4.296810</td> <td>4.669635</td> <td>2.282003</td> <td>18.039617</td> <td>6.060465</td> </tr> <tr> <th>Stream 3</th> <td>4.947110</td> <td>2.479268</td> <td>2.952951</td> <td>6.431644</td> <td>5.469152</td> <td>1.816467</td> <td>8.271266</td> <td>5.498956</td> </tr> <tr> <th>Stream 4</th> <td>5.240237</td> <td>2.062261</td> <td>2.734378</td> <td>6.055141</td> <td>2.997684</td> <td>2.519301</td> <td>7.889700</td> <td>6.944722</td> </tr> <tr> <th>Stream 5</th> <td>4.839670</td> <td>3.379315</td> <td>3.231582</td> <td>6.255944</td> <td>3.759509</td> <td>1.347830</td> <td>8.707303</td> <td>4.376033</td> </tr> <tr> <th>Min Qi</th> <td>3.398354</td> <td>2.062261</td> <td>2.634076</td> <td>4.296810</td> <td>2.997684</td> <td>1.116547</td> <td>7.889700</td> <td>4.376033</td> </tr> <tr> <th>Max Qi</th> <td>6.811520</td> <td>3.411846</td> <td>3.231582</td> <td>6.431644</td> <td>5.469152</td> <td>2.519301</td> <td>18.039617</td> <td>6.944722</td> </tr> <tr> <th>Avg Qi</th> <td>5.047378</td> <td>2.784788</td> <td>2.914839</td> <td>5.540549</td> <td>4.334337</td> <td>1.816430</td> <td>10.335600</td> <td>5.704745</td> </tr> </table> <h2>100 GB Run 2</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>October 3, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>100</td> </tr> <tr> <th style="text-align:right">Total Data Storage/Database Size</th> <td>0M</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>5</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td>387,890.3</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td>433,066.6</td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@100GB)</th> <td>409,856.5</td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td>91.541000 seconds</td> </tr> </table> <h3>Duration of stream execution</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>10/03/2014 15:07:19</td> <td>10/03/2014 15:07:47</td> <td>0:00:28</td> </tr> <tr> <th>Stream 1</th> <td>10/03/2014 15:07:48</td> <td>10/03/2014 15:09:19</td> <td>0:01:31</td> </tr> <tr> <th>Stream 2</th> <td>10/03/2014 15:07:48</td> <td>10/03/2014 15:09:16</td> <td>0:01:28</td> </tr> <tr> <th>Stream 3</th> <td>10/03/2014 15:07:48</td> <td>10/03/2014 15:09:17</td> <td>0:01:29</td> </tr> <tr> <th>Stream 4</th> <td>10/03/2014 15:07:48</td> <td>10/03/2014 15:09:16</td> <td>0:01:28</td> </tr> <tr> <th>Stream 5</th> <td>10/03/2014 15:07:48</td> <td>10/03/2014 15:09:20</td> <td>0:01:32</td> </tr> <tr> <th>Refresh 0</th> <td>10/03/2014 15:07:19</td> <td>10/03/2014 15:07:22</td> <td>0:00:03</td> </tr> <tr> <th> </th> <td>10/03/2014 15:07:47</td> <td>10/03/2014 15:07:48</td> <td>0:00:01</td> </tr> <tr> <th>Refresh 1</th> <td>10/03/2014 15:08:45</td> <td>10/03/2014 15:08:59</td> <td>0:00:14</td> </tr> <tr> <th>Refresh 2</th> <td>10/03/2014 15:07:49</td> <td>10/03/2014 15:08:02</td> <td>0:00:13</td> </tr> <tr> <th>Refresh 3</th> <td>10/03/2014 15:08:02</td> <td>10/03/2014 15:08:17</td> <td>0:00:15</td> </tr> <tr> <th>Refresh 4</th> <td>10/03/2014 15:08:17</td> <td>10/03/2014 15:08:29</td> <td>0:00:12</td> </tr> <tr> <th>Refresh 5</th> <td>10/03/2014 15:08:29</td> <td>10/03/2014 15:08:45</td> <td>0:00:16</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> </tr> <tr> <th>Stream 0</th> <td>2.081986</td> <td>0.208487</td> <td>0.902462</td> <td>0.313160</td> <td>1.312273</td> <td>0.493157</td> <td>0.926629</td> <td>0.786345</td> </tr> <tr> <th>Stream 1</th> <td>2.755427</td> <td>0.911578</td> <td>3.618085</td> <td>0.664407</td> <td>3.740112</td> <td>2.118189</td> <td>4.738754</td> <td>6.551446</td> </tr> <tr> <th>Stream 2</th> <td>4.189612</td> <td>0.957921</td> <td>5.267355</td> <td>2.152479</td> <td>6.068005</td> <td>1.263380</td> <td>4.251842</td> <td>3.620160</td> </tr> <tr> <th>Stream 3</th> <td>4.708834</td> <td>0.981651</td> <td>2.411839</td> <td>0.790955</td> <td>4.384516</td> <td>1.322670</td> <td>2.641571</td> <td>4.771831</td> </tr> <tr> <th>Stream 4</th> <td>3.739567</td> <td>1.185884</td> <td>2.863871</td> <td>1.517891</td> <td>5.946967</td> <td>1.179960</td> <td>3.840560</td> <td>4.926325</td> </tr> <tr> <th>Stream 5</th> <td>5.258746</td> <td>0.705228</td> <td>3.460904</td> <td>0.951328</td> <td>4.530620</td> <td>1.104500</td> <td>3.226494</td> <td>4.041142</td> </tr> <tr> <th>Min Qi</th> <td>2.755427</td> <td>0.705228</td> <td>2.411839</td> <td>0.664407</td> <td>3.740112</td> <td>1.104500</td> <td>2.641571</td> <td>3.620160</td> </tr> <tr> <th>Max Qi</th> <td>5.258746</td> <td>1.185884</td> <td>5.267355</td> <td>2.152479</td> <td>6.068005</td> <td>2.118189</td> <td>4.738754</td> <td>6.551446</td> </tr> <tr> <th>Avg Qi</th> <td>4.130437</td> <td>0.948452</td> <td>3.524411</td> <td>1.215412</td> <td>4.934044</td> <td>1.397740</td> <td>3.739844</td> <td>4.782181</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>3.226685</td> <td>1.878227</td> <td>1.802562</td> <td>0.676499</td> <td>3.145884</td> <td>0.653129</td> <td>0.963449</td> <td>0.990524</td> </tr> <tr> <th>Stream 1</th> <td>8.842030</td> <td>5.630466</td> <td>5.728147</td> <td>2.643227</td> <td>9.615551</td> <td>3.197855</td> <td>4.676538</td> <td>4.285251</td> </tr> <tr> <th>Stream 2</th> <td>9.508612</td> <td>5.288044</td> <td>4.319998</td> <td>1.492915</td> <td>9.431995</td> <td>3.206360</td> <td>3.859749</td> <td>3.201996</td> </tr> <tr> <th>Stream 3</th> <td>10.480224</td> <td>5.880274</td> <td>4.517320</td> <td>2.509405</td> <td>6.913159</td> <td>2.892479</td> <td>6.408602</td> <td>2.938061</td> </tr> <tr> <th>Stream 4</th> <td>8.824111</td> <td>5.752413</td> <td>5.997959</td> <td>2.581237</td> <td>8.954756</td> <td>3.351951</td> <td>2.420598</td> <td>4.148455</td> </tr> <tr> <th>Stream 5</th> <td>4.905553</td> <td>7.099111</td> <td>5.121041</td> <td>2.516020</td> <td>9.354924</td> <td>3.955638</td> <td>4.389209</td> <td>3.818902</td> </tr> <tr> <th>Min Qi</th> <td>4.905553</td> <td>5.288044</td> <td>4.319998</td> <td>1.492915</td> <td>6.913159</td> <td>2.892479</td> <td>2.420598</td> <td>2.938061</td> </tr> <tr> <th>Max Qi</th> <td>10.480224</td> <td>7.099111</td> <td>5.997959</td> <td>2.643227</td> <td>9.615551</td> <td>3.955638</td> <td>6.408602</td> <td>4.285251</td> </tr> <tr> <th>Avg Qi</th> <td>8.512106</td> <td>5.930062</td> <td>5.136893</td> <td>2.348561</td> <td>8.854077</td> <td>3.320857</td> <td>4.350939</td> <td>3.678533</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>1.405338</td> <td>0.868313</td> <td>0.806277</td> <td>1.123366</td> <td>1.314028</td> <td>0.233214</td> <td>2.590459</td> <td>1.230242</td> </tr> <tr> <th>Stream 1</th> <td>5.191045</td> <td>3.171244</td> <td>3.403836</td> <td>4.604523</td> <td>3.721133</td> <td>0.892096</td> <td>7.136841</td> <td>6.500452</td> </tr> <tr> <th>Stream 2</th> <td>6.282687</td> <td>2.845465</td> <td>3.024786</td> <td>4.086546</td> <td>3.530743</td> <td>0.619683</td> <td>9.263671</td> <td>4.826173</td> </tr> <tr> <th>Stream 3</th> <td>6.040787</td> <td>2.659766</td> <td>2.787273</td> <td>6.210077</td> <td>3.902190</td> <td>2.175417</td> <td>7.974860</td> <td>6.689780</td> </tr> <tr> <th>Stream 4</th> <td>4.978721</td> <td>2.542674</td> <td>3.518783</td> <td>4.385571</td> <td>3.906211</td> <td>0.918752</td> <td>6.303352</td> <td>5.139326</td> </tr> <tr> <th>Stream 5</th> <td>5.208600</td> <td>3.761975</td> <td>3.682886</td> <td>7.874493</td> <td>5.017600</td> <td>2.087150</td> <td>7.999074</td> <td>7.978154</td> </tr> <tr> <th>Min Qi</th> <td>4.978721</td> <td>2.542674</td> <td>2.787273</td> <td>4.086546</td> <td>3.530743</td> <td>0.619683</td> <td>6.303352</td> <td>4.826173</td> </tr> <tr> <th>Max Qi</th> <td>6.282687</td> <td>3.761975</td> <td>3.682886</td> <td>7.874493</td> <td>5.017600</td> <td>2.175417</td> <td>9.263671</td> <td>7.978154</td> </tr> <tr> <th>Avg Qi</th> <td>5.540368</td> <td>2.996225</td> <td>3.283513</td> <td>5.432242</td> <td>4.015575</td> <td>1.338620</td> <td>7.735560</td> <td>6.226777</td> </tr> </table> <h2>1000 GB Run 1</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>October 3, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>1000</td> </tr> <tr> <th style="text-align:right">Total Data Storage/Database Size</th> <td>26M</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>7</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td>136,744.5</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td>147,374.6</td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@1000GB)</th> <td>141,960.1</td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td>3,761.953000 seconds</td> </tr> </table> <h3>Duration of stream execution</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>10/03/2014 09:18:42</td> <td>10/03/2014 09:34:12</td> <td>0:15:30</td> </tr> <tr> <th>Stream 1</th> <td>10/03/2014 09:34:43</td> <td>10/03/2014 10:35:42</td> <td>1:00:59</td> </tr> <tr> <th>Stream 2</th> <td>10/03/2014 09:34:43</td> <td>10/03/2014 10:37:14</td> <td>1:02:31</td> </tr> <tr> <th>Stream 3</th> <td>10/03/2014 09:34:43</td> <td>10/03/2014 10:37:25</td> <td>1:02:42</td> </tr> <tr> <th>Stream 4</th> <td>10/03/2014 09:34:43</td> <td>10/03/2014 10:33:31</td> <td>0:58:48</td> </tr> <tr> <th>Stream 5</th> <td>10/03/2014 09:34:43</td> <td>10/03/2014 10:35:26</td> <td>1:00:43</td> </tr> <tr> <th>Stream 6</th> <td>10/03/2014 09:34:43</td> <td>10/03/2014 10:28:00</td> <td>0:53:17</td> </tr> <tr> <th>Stream 7</th> <td>10/03/2014 09:34:43</td> <td>10/03/2014 10:35:42</td> <td>1:00:59</td> </tr> <tr> <th>Refresh 0</th> <td>10/03/2014 09:18:42</td> <td>10/03/2014 09:19:27</td> <td>0:00:45</td> </tr> <tr> <th> </th> <td>10/03/2014 09:34:12</td> <td>10/03/2014 09:34:42</td> <td>0:00:30</td> </tr> <tr> <th>Refresh 1</th> <td>10/03/2014 09:43:03</td> <td>10/03/2014 09:43:38</td> <td>0:00:35</td> </tr> <tr> <th>Refresh 2</th> <td>10/03/2014 09:34:43</td> <td>10/03/2014 09:36:54</td> <td>0:02:11</td> </tr> <tr> <th>Refresh 3</th> <td>10/03/2014 09:36:53</td> <td>10/03/2014 09:38:39</td> <td>0:01:46</td> </tr> <tr> <th>Refresh 4</th> <td>10/03/2014 09:38:39</td> <td>10/03/2014 09:39:22</td> <td>0:00:43</td> </tr> <tr> <th>Refresh 5</th> <td>10/03/2014 09:39:23</td> <td>10/03/2014 09:41:09</td> <td>0:01:46</td> </tr> <tr> <th>Refresh 6</th> <td>10/03/2014 09:41:09</td> <td>10/03/2014 09:42:15</td> <td>0:01:06</td> </tr> <tr> <th>Refresh 7</th> <td>10/03/2014 09:42:15</td> <td>10/03/2014 09:43:02</td> <td>0:00:47</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> <th>Stream 0</th> <td>104.488583</td> <td>18.351559</td> <td>24.631282</td> <td>36.195531</td> <td>36.319915</td> <td>3.807790</td> <td>22.750889</td> <td>31.190630</td> </tr> <tr> <th>Stream 1</th> <td>209.323441</td> <td>26.205435</td> <td>59.637373</td> <td>245.808484</td> <td>60.699333</td> <td>22.369379</td> <td>289.435780</td> <td>335.733425</td> </tr> <tr> <th>Stream 2</th> <td>109.134446</td> <td>64.185831</td> <td>96.131735</td> <td>108.459418</td> <td>310.273986</td> <td>53.595127</td> <td>152.242755</td> <td>104.350098</td> </tr> <tr> <th>Stream 3</th> <td>73.321611</td> <td>215.535408</td> <td>69.543101</td> <td>12.423757</td> <td>64.445611</td> <td>38.254747</td> <td>122.952872</td> <td>98.713213</td> </tr> <tr> <th>Stream 4</th> <td>110.875875</td> <td>4.272757</td> <td>78.697314</td> <td>16.316807</td> <td>59.746855</td> <td>23.447211</td> <td>353.190412</td> <td>342.549908</td> </tr> <tr> <th>Stream 5</th> <td>41.972337</td> <td>5.978707</td> <td>60.784575</td> <td>34.219229</td> <td>42.372449</td> <td>344.590640</td> <td>146.186614</td> <td>274.972270</td> </tr> <tr> <th>Stream 6</th> <td>115.760155</td> <td>18.692078</td> <td>58.493147</td> <td>9.193234</td> <td>49.831932</td> <td>19.081395</td> <td>60.603109</td> <td>128.095501</td> </tr> <tr> <th>Stream 7</th> <td>58.601744</td> <td>118.126585</td> <td>297.327543</td> <td>298.578268</td> <td>714.284222</td> <td>108.475250</td> <td>91.868151</td> <td>55.881029</td> </tr> <tr> <th>Min Qi</th> <td>41.972337</td> <td>4.272757</td> <td>58.493147</td> <td>9.193234</td> <td>42.372449</td> <td>19.081395</td> <td>60.603109</td> <td>55.881029</td> </tr> <tr> <th>Max Qi</th> <td>209.323441</td> <td>215.535408</td> <td>297.327543</td> <td>298.578268</td> <td>714.284222</td> <td>344.590640</td> <td>353.190412</td> <td>342.549908</td> </tr> <tr> <th>Avg Qi</th> <td>102.712801</td> <td>64.713829</td> <td>102.944970</td> <td>103.571314</td> <td>185.950627</td> <td>87.116250</td> <td>173.782813</td> <td>191.470778</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>41.777880</td> <td>10.035063</td> <td>16.125611</td> <td>9.245638</td> <td>209.443782</td> <td>111.271310</td> <td>37.821595</td> <td>9.483838</td> </tr> <tr> <th>Stream 1</th> <td>244.243830</td> <td>63.473338</td> <td>207.741931</td> <td>33.696956</td> <td>561.057408</td> <td>141.026049</td> <td>126.818051</td> <td>54.774792</td> </tr> <tr> <th>Stream 2</th> <td>189.297446</td> <td>144.853756</td> <td>56.292537</td> <td>184.781273</td> <td>501.330052</td> <td>49.965102</td> <td>107.736393</td> <td>85.691079</td> </tr> <tr> <th>Stream 3</th> <td>231.060699</td> <td>355.394713</td> <td>43.483645</td> <td>11.806590</td> <td>555.445111</td> <td>36.722686</td> <td>251.241817</td> <td>9.057850</td> </tr> <tr> <th>Stream 4</th> <td>227.371508</td> <td>32.207115</td> <td>108.880658</td> <td>139.922550</td> <td>532.697956</td> <td>57.106583</td> <td>159.198489</td> <td>153.088913</td> </tr> <tr> <th>Stream 5</th> <td>416.113856</td> <td>108.689389</td> <td>62.847727</td> <td>702.712683</td> <td>622.906487</td> <td>58.198961</td> <td>89.707091</td> <td>85.614769</td> </tr> <tr> <th>Stream 6</th> <td>228.019243</td> <td>62.474213</td> <td>88.227994</td> <td>282.932978</td> <td>432.387869</td> <td>238.544027</td> <td>61.486269</td> <td>56.950548</td> </tr> <tr> <th>Stream 7</th> <td>230.564416</td> <td>69.197517</td> <td>130.708759</td> <td>120.531103</td> <td>551.112816</td> <td>57.438478</td> <td>82.256530</td> <td>63.796403</td> </tr> <tr> <th>Min Qi</th> <td>189.297446</td> <td>32.207115</td> <td>43.483645</td> <td>11.806590</td> <td>432.387869</td> <td>36.722686</td> <td>61.486269</td> <td>9.057850</td> </tr> <tr> <th>Max Qi</th> <td>416.113856</td> <td>355.394713</td> <td>207.741931</td> <td>702.712683</td> <td>622.906487</td> <td>238.544027</td> <td>251.241817</td> <td>153.088913</td> </tr> <tr> <th>Avg Qi</th> <td>252.381571</td> <td>119.470006</td> <td>99.740464</td> <td>210.912019</td> <td>536.705386</td> <td>91.285984</td> <td>125.492091</td> <td>72.710622</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>22.897349</td> <td>47.870269</td> <td>12.735580</td> <td>25.982194</td> <td>46.091766</td> <td>6.623306</td> <td>45.120559</td> <td>30.016788</td> </tr> <tr> <th>Stream 1</th> <td>123.444839</td> <td>22.212194</td> <td>647.523826</td> <td>97.431531</td> <td>81.592165</td> <td>4.573040</td> <td>21.068225</td> <td>14.486185</td> </tr> <tr> <th>Stream 2</th> <td>80.853865</td> <td>622.651044</td> <td>288.656211</td> <td>336.409076</td> <td>70.925079</td> <td>33.578052</td> <td>82.910543</td> <td>48.001583</td> </tr> <tr> <th>Stream 3</th> <td>392.340812</td> <td>84.967695</td> <td>57.181935</td> <td>473.720060</td> <td>497.262620</td> <td>66.966740</td> <td>54.778284</td> <td>50.940094</td> </tr> <tr> <th>Stream 4</th> <td>97.069440</td> <td>301.705125</td> <td>338.035788</td> <td>258.992426</td> <td>103.699408</td> <td>28.750257</td> <td>23.858757</td> <td>13.626079</td> </tr> <tr> <th>Stream 5</th> <td>69.882110</td> <td>34.277914</td> <td>146.031938</td> <td>179.656129</td> <td>104.788154</td> <td>10.836148</td> <td>54.319823</td> <td>52.077352</td> </tr> <tr> <th>Stream 6</th> <td>141.310431</td> <td>247.242904</td> <td>94.392791</td> <td>702.775460</td> <td>80.142930</td> <td>19.969889</td> <td>46.027410</td> <td>19.136271</td> </tr> <tr> <th>Stream 7</th> <td>89.018281</td> <td>51.105998</td> <td>281.234432</td> <td>79.046122</td> <td>84.341517</td> <td>26.221892</td> <td>33.169666</td> <td>13.309634</td> </tr> <tr> <th>Min Qi</th> <td>69.882110</td> <td>22.212194</td> <td>57.181935</td> <td>79.046122</td> <td>70.925079</td> <td>4.573040</td> <td>21.068225</td> <td>13.309634</td> </tr> <tr> <th>Max Qi</th> <td>392.340812</td> <td>622.651044</td> <td>647.523826</td> <td>702.775460</td> <td>497.262620</td> <td>66.966740</td> <td>82.910543</td> <td>52.077352</td> </tr> <tr> <th>Avg Qi</th> <td>141.988540</td> <td>194.880411</td> <td>264.722417</td> <td>304.004401</td> <td>146.107410</td> <td>27.270860</td> <td>45.161815</td> <td>30.225314</td> </tr> </table> <h2>1000 GB Run 2</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>October 3, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>1000</td> </tr> <tr> <th style="text-align:right">Total Data Storage/Database Size</th> <td>26M</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>7</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td>199,652.0</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td>125,161.1</td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@1000GB)</th> <td>158,078.0</td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td>4,429.608000 seconds</td> </tr> </table> <h3>Duration of stream execution</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>10/03/2014 10:37:29</td> <td>10/03/2014 10:52:26</td> <td>0:14:57</td> </tr> <tr> <th>Stream 1</th> <td>10/03/2014 10:52:35</td> <td>10/03/2014 12:05:19</td> <td>1:12:44</td> </tr> <tr> <th>Stream 2</th> <td>10/03/2014 10:52:35</td> <td>10/03/2014 12:06:25</td> <td>1:13:50</td> </tr> <tr> <th>Stream 3</th> <td>10/03/2014 10:52:35</td> <td>10/03/2014 12:03:08</td> <td>1:10:33</td> </tr> <tr> <th>Stream 4</th> <td>10/03/2014 10:52:35</td> <td>10/03/2014 12:05:20</td> <td>1:12:45</td> </tr> <tr> <th>Stream 5</th> <td>10/03/2014 10:52:35</td> <td>10/03/2014 11:57:40</td> <td>1:05:05</td> </tr> <tr> <th>Stream 6</th> <td>10/03/2014 10:52:35</td> <td>10/03/2014 12:05:28</td> <td>1:12:53</td> </tr> <tr> <th>Stream 7</th> <td>10/03/2014 10:52:35</td> <td>10/03/2014 12:05:25</td> <td>1:12:50</td> </tr> <tr> <th>Refresh 0</th> <td>10/03/2014 10:37:29</td> <td>10/03/2014 10:37:52</td> <td>0:00:23</td> </tr> <tr> <th> </th> <td>10/03/2014 10:52:25</td> <td>10/03/2014 10:52:34</td> <td>0:00:09</td> </tr> <tr> <th>Refresh 1</th> <td>10/03/2014 11:01:44</td> <td>10/03/2014 11:02:29</td> <td>0:00:45</td> </tr> <tr> <th>Refresh 2</th> <td>10/03/2014 10:52:35</td> <td>10/03/2014 10:54:50</td> <td>0:02:15</td> </tr> <tr> <th>Refresh 3</th> <td>10/03/2014 10:54:50</td> <td>10/03/2014 10:57:02</td> <td>0:02:12</td> </tr> <tr> <th>Refresh 4</th> <td>10/03/2014 10:57:05</td> <td>10/03/2014 10:58:47</td> <td>0:01:42</td> </tr> <tr> <th>Refresh 5</th> <td>10/03/2014 10:58:47</td> <td>10/03/2014 10:59:46</td> <td>0:00:59</td> </tr> <tr> <th>Refresh 6</th> <td>10/03/2014 10:59:45</td> <td>10/03/2014 11:00:38</td> <td>0:00:53</td> </tr> <tr> <th>Refresh 7</th> <td>10/03/2014 11:00:39</td> <td>10/03/2014 11:01:44</td> <td>0:01:05</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> </tr> <tr> <th>Stream 0</th> <td>34.105419</td> <td>1.439089</td> <td>9.802183</td> <td>2.033956</td> <td>10.525742</td> <td>3.356152</td> <td>23.953729</td> <td>36.199533</td> </tr> <tr> <th>Stream 1</th> <td>26.598252</td> <td>150.572833</td> <td>41.930330</td> <td>86.870320</td> <td>50.604856</td> <td>201.001372</td> <td>61.638366</td> <td>244.013359</td> </tr> <tr> <th>Stream 2</th> <td>50.129895</td> <td>102.219282</td> <td>12.380935</td> <td>102.319615</td> <td>62.577229</td> <td>43.454392</td> <td>891.076608</td> <td>407.640626</td> </tr> <tr> <th>Stream 3</th> <td>269.947278</td> <td>53.172724</td> <td>54.649973</td> <td>11.460062</td> <td>66.695722</td> <td>17.336698</td> <td>63.371232</td> <td>91.158050</td> </tr> <tr> <th>Stream 4</th> <td>41.149221</td> <td>22.520836</td> <td>28.707973</td> <td>509.984321</td> <td>68.916549</td> <td>17.525025</td> <td>702.191490</td> <td>666.450230</td> </tr> <tr> <th>Stream 5</th> <td>59.179045</td> <td>30.734442</td> <td>99.504351</td> <td>11.145990</td> <td>101.334340</td> <td>21.660836</td> <td>74.625589</td> <td>535.160207</td> </tr> <tr> <th>Stream 6</th> <td>225.105215</td> <td>55.567328</td> <td>46.749707</td> <td>554.474507</td> <td>215.657091</td> <td>54.362551</td> <td>72.960653</td> <td>442.194302</td> </tr> <tr> <th>Stream 7</th> <td>220.993226</td> <td>28.528230</td> <td>47.543365</td> <td>336.191006</td> <td>308.931194</td> <td>9.767397</td> <td>850.258452</td> <td>66.121298</td> </tr> <tr> <th>Min Qi</th> <td>26.598252</td> <td>22.520836</td> <td>12.380935</td> <td>11.145990</td> <td>50.604856</td> <td>9.767397</td> <td>61.638366</td> <td>66.121298</td> </tr> <tr> <th>Max Qi</th> <td>269.947278</td> <td>150.572833</td> <td>99.504351</td> <td>554.474507</td> <td>308.931194</td> <td>201.001372</td> <td>891.076608</td> <td>666.450230</td> </tr> <tr> <th>Avg Qi</th> <td>127.586019</td> <td>63.330811</td> <td>47.352376</td> <td>230.349403</td> <td>124.959569</td> <td>52.158324</td> <td>388.017484</td> <td>350.391153</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>50.439615</td> <td>9.287196</td> <td>15.892947</td> <td>7.112715</td> <td>250.527755</td> <td>131.478131</td> <td>54.458992</td> <td>10.525842</td> </tr> <tr> <th>Stream 1</th> <td>420.919329</td> <td>317.402771</td> <td>101.818338</td> <td>403.213385</td> <td>724.539887</td> <td>160.669174</td> <td>65.374584</td> <td>28.563034</td> </tr> <tr> <th>Stream 2</th> <td>464.378760</td> <td>210.938167</td> <td>23.395678</td> <td>545.086468</td> <td>736.005716</td> <td>54.680686</td> <td>398.880053</td> <td>34.018918</td> </tr> <tr> <th>Stream 3</th> <td>350.083270</td> <td>321.781561</td> <td>48.652019</td> <td>435.954962</td> <td>378.872739</td> <td>100.588804</td> <td>289.350342</td> <td>190.140640</td> </tr> <tr> <th>Stream 4</th> <td>306.265994</td> <td>249.621982</td> <td>79.280220</td> <td>221.255121</td> <td>348.932746</td> <td>49.555802</td> <td>100.062439</td> <td>61.368814</td> </tr> <tr> <th>Stream 5</th> <td>511.923087</td> <td>133.018420</td> <td>134.199065</td> <td>9.655693</td> <td>662.658830</td> <td>104.380635</td> <td>82.847242</td> <td>59.952271</td> </tr> <tr> <th>Stream 6</th> <td>578.362701</td> <td>61.221715</td> <td>145.613349</td> <td>47.957006</td> <td>621.993889</td> <td>256.150595</td> <td>77.124777</td> <td>91.163005</td> </tr> <tr> <th>Stream 7</th> <td>418.450091</td> <td>391.818564</td> <td>29.360218</td> <td>17.236628</td> <td>761.850888</td> <td>31.952329</td> <td>50.393082</td> <td>27.530882</td> </tr> <tr> <th>Min Qi</th> <td>306.265994</td> <td>61.221715</td> <td>23.395678</td> <td>9.655693</td> <td>348.932746</td> <td>31.952329</td> <td>50.393082</td> <td>27.530882</td> </tr> <tr> <th>Max Qi</th> <td>578.362701</td> <td>391.818564</td> <td>145.613349</td> <td>545.086468</td> <td>761.850888</td> <td>256.150595</td> <td>398.880053</td> <td>190.140640</td> </tr> <tr> <th>Avg Qi</th> <td>435.769033</td> <td>240.829026</td> <td>80.331270</td> <td>240.051323</td> <td>604.979242</td> <td>108.282575</td> <td>152.004646</td> <td>70.391081</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>22.444111</td> <td>37.978532</td> <td>13.347320</td> <td>26.553364</td> <td>115.511143</td> <td>7.670304</td> <td>22.771613</td> <td>8.761026</td> </tr> <tr> <th>Stream 1</th> <td>329.153807</td> <td>19.198590</td> <td>258.455295</td> <td>556.256015</td> <td>99.647793</td> <td>14.878746</td> <td>32.803289</td> <td>8.771923</td> </tr> <tr> <th>Stream 2</th> <td>76.940373</td> <td>74.916489</td> <td>75.246897</td> <td>16.035355</td> <td>14.403643</td> <td>32.348500</td> <td>91.981362</td> <td>41.426540</td> </tr> <tr> <th>Stream 3</th> <td>88.918404</td> <td>238.858707</td> <td>221.257060</td> <td>688.441713</td> <td>247.669761</td> <td>5.345632</td> <td>70.780594</td> <td>49.352955</td> </tr> <tr> <th>Stream 4</th> <td>497.105081</td> <td>167.874781</td> <td>67.668514</td> <td>76.820831</td> <td>78.585717</td> <td>3.655421</td> <td>73.165786</td> <td>29.401670</td> </tr> <tr> <th>Stream 5</th> <td>309.991618</td> <td>123.023557</td> <td>380.801141</td> <td>347.055909</td> <td>93.478502</td> <td>18.351491</td> <td>33.338814</td> <td>12.557542</td> </tr> <tr> <th>Stream 6</th> <td>57.200926</td> <td>154.489850</td> <td>386.007137</td> <td>103.558355</td> <td>32.676369</td> <td>92.863316</td> <td>35.576966</td> <td>14.061801</td> </tr> <tr> <th>Stream 7</th> <td>160.332088</td> <td>46.934177</td> <td>340.957970</td> <td>84.479720</td> <td>78.985110</td> <td>60.568796</td> <td>44.362737</td> <td>8.831746</td> </tr> <tr> <th>Min Qi</th> <td>57.200926</td> <td>19.198590</td> <td>67.668514</td> <td>16.035355</td> <td>14.403643</td> <td>3.655421</td> <td>32.803289</td> <td>8.771923</td> </tr> <tr> <th>Max Qi</th> <td>497.105081</td> <td>238.858707</td> <td>386.007137</td> <td>688.441713</td> <td>247.669761</td> <td>92.863316</td> <td>91.981362</td> <td>49.352955</td> </tr> <tr> <th>Avg Qi</th> <td>217.091757</td> <td>117.899450</td> <td>247.199145</td> <td>267.521128</td> <td>92.206699</td> <td>32.573129</td> <td>54.572793</td> <td>23.486311</td> </tr> </table> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aab52abc0b8"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aab53935178"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aab537ea348"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aab516a28b8"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aab514b53d8"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aaba8e12aa8"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aab537ea308"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aaba97e18d8"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aab52a6add8"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aaba9725b78"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aab51109e68"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aaba97e1898"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aab51a706a8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aaba97e18b8"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aaba9725b98"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x2aab52abc098"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aab5130e068"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aab52979f88"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aab53015408"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aac04bdd538"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:08:38.906994-04:00
In Hoc Signo Vinces (part 20 of n): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O
http://www.openlinksw.com:443/weblog/oerling/?date=2014-10-06#1824
2014-10-06T17:53:06Z
<p>In the <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x2aab51a32ce8">introduction to scale out</a> piece, I promised to address the matter of data-to-memory ratio, and to talk about when scale-out makes sense. Here we will see that scale-out makes sense whenever data does not fit in memory on a single commodity server. The gains in processing power are immediate, even when going from one box to just two, with both systems having all in memory.</p> <p>As an initial take on the issue we run 100 GB and 1000 GB on the test system. 100 GB is trivially in memory, 1000 GB is not, as the memory is 384 GB total, of which 360 GB may be used for the processes.</p> <p>We run 2 workloads on the 100 GB database, having pre-loaded the data in memory:</p> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr style="text-align:center;"> <th>run</th> <th>power</th> <th>throughput</th> <th>composite</th> </tr> <tr> <th>1</th> <td>349,027.7</td> <td>420,503.1</td> <td>383,102.1</td> </tr> <tr> <th>2</th> <td>387,890.3</td> <td>433,066.6</td> <td>409,856.5</td> </tr> </table> <p>This is directly comparable to the 100 GB single-server results. Comparing the second runs, we see a 1.53x gain in power and a 1.8x gain in throughput from 2x the platform. This is fully on the level for a workload that is not trivially parallel, as we have seen in the previous articles. The difference between the first and second runs at 100 GB comes, for both single-server and cluster, from the latency of allocating transient query memory. For an official run, where the weakest link is the first power test, this would simply have to be pre-allocated.</p> <p>We run 2 workloads on the 1000 GB database, starting from cold.</p> <p>The result is:</p> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr style="text-align:center;"> <th>run</th> <th>power</th> <th>throughput</th> <th>composite</th> </tr> <tr> <th>1</th> <td>136,744.5</td> <td>147,374.6</td> <td>141,960.1</td> </tr> <tr> <th>2</th> <td>199,652.0</td> <td>125,161.1</td> <td>158,078.0</td> </tr> </table> <p>The 1000 GB result is not for competition with this platform; more memory would be needed. For actual applications, the numbers are still in the usable range, though.</p> <p>The 1000 GB setup uses 4 SSDs for storage, one per server process. The server processes are each bound to their own physical CPU.</p> <p>We look at the meters: 32M pages (8M per process) are in memory at each time. Over the 2 benchmark executions there are a total of 494M disk reads. The total CPU time is 165,674 seconds of CPU, of which about 10% are system, over 10,063 seconds of real-time. Cumulative disk-read wait-time is 130,177 s. This gives an average disk read throughput of 384 MB/s.</p> <p>This is easily sustained by 4 SSDs; in practice, the maximum throughput we see for reading is 1 GB/s (256 MB/s per SSD). Newer SSDs would do maybe twice that. Using rotating media would not be an option.</p> <p>Without the drop in CPU caused by waiting for SSD, we would have numbers very close to the 100 GB numbers. </p> <p>The interconnect traffic for the two runs was 1,077 GB with no message compression. The write block time was 448 seconds of thread-time. So we see that blocking on write hurts platform utilization when running under optimal conditions, but compared to going to secondary storage, it is not a large factor.</p> <p>The 1000 GB scale has a transient peak memory consumption of 42 GB. This consists of hash-join build sides and GROUP BYs. The greatest memory consumers are Q9 with 9 GB, Q13 with 11 GB, and Q16 with 7 GB. Having many of these at a time drives up the transient peak. The peak gets higher as the scale grows, also because a larger scale requires more concurrent query streams. At the 384 GB for 1000 GB ratio, we do not yet get into memory saving plans like hash joins in many passes or index use instead of hash. When the data size grows, replicated hash build sides will become less convenient, and communication will increase. Q9 and Q13 can be done by index with almost no transient memory, but these plans are easily 3x less efficient for CPU. These will probably help at 3000 GB and be necessary at least part of the time at 10,000 GB.</p> <p>The I/O volume in MB per index over the 2 executions is:</p> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr style="text-align:center;"> <th>index</th> <th>MB</th> </tr> <tr> <th><code>LINEITEM</code> </th> <td>1,987,483</td> </tr> <tr> <th><code>ORDERS</code> </th> <td>1,440,526</td> </tr> <tr> <th><code>PARTSUPP</code> </th> <td>199,335</td> </tr> <tr> <th><code>PART</code> </th> <td>161,717</td> </tr> <tr> <th><code>CUSTOMER</code> </th> <td>43,276</td> </tr> <tr> <th><code>O_CK</code> </th> <td>19,085</td> </tr> <tr> <th><code>SUPPLIER</code> </th> <td>13,393</td> </tr> </table> <p>Of this, maybe 600 GB could be saved by stream compressing <code>o_comment</code>. Otherwise this cannot be helped without adding memory. The <code>lineitem</code> reads are mostly for <code>l_extendedprice</code>, which is not compressible. If compressing <code>o_comment</code> made <code>l_extendedprice</code> always fit in memory, then there would be a radical drop in I/O. Also, as a matter of fact, the buffer management policy of least-recently-used works the very worst for big scans, specifically those of <code>l_extendedprice</code>: If the head is replaced when reading the tail, and the next read starts from the head, then the whole table/column is read all over again. Caching policies that specially recognized scans of this sort could further reduce I/O. Clustering <code>lineitems</code>/<code>orders</code> on <code>date</code>, as <a href="http://dbpedia.org/page/Vectorwise" id="link-id0x2aab50f9b008">Actian Vector</a> <a href="http://www.tpc.org/tpch/" id="link-id0x2aab51845d48">TPC-H</a> implementations do, also starts yielding a greater gain when not running from memory: One column (e.g., <code>l_shipdate</code>) may be scanned for the whole table but, if the matches are bunched together, then most of <code>l_extendedprice</code> will not be read at all. Still, if going for top ranks in the races, all will be from memory, or at least there will be SSDs with read throughput around 150 MB/s per core, so these tricks become relatively less important.</p> <p>In the 100 GB numerical quantities summaries, we see much the same picture as in the single-server. Queries get faster, but their relative times are not radically different. The throughput test (many queries at a time) times are more or less multiples of the power (single user) times. This picture breaks at 1000 GB where I/O first drops the performance to under half and introduces huge variation in execution times within a single query. The time entirely depends on which queries are running along with or right before the execution and on whether these have the same or different working sets. All the streams have the same queries with different parameters, but the query order in each stream is different.</p> <p>The numerical quantities follow for all the runs. Note that the first 1000 GB run is cold. A competition grade 1000 GB result can be made with double the memory, and the more CPU the better. We will try one at Amazon in a bit.</p> <p> *** </p> <p>The conclusion is that scale-out pays from the get-go. At present prices, a system with twice the power of a single node of the test system is cost effective. Scales of up to 500 GB are single commodity server, under $10K. Rather than going from a mid-to-large dual-socket box to a quad-socket box, one is likely to be better off having two cheaper dual-socket boxes. These are also readily available on clouds, whereas scale-up configurations are not. Onwards of 1 TB, a cluster is expected to clearly win. At 3 TB, a commodity cluster will clearly be the better deal for both price and absolute performance.</p> <h2>100 GB Run 1</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>October 3, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>100</td> </tr> <tr> <th style="text-align:right">Total Data Storage/Database Size</th> <td>0M</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>5</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td>349,027.7</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td>420,503.1</td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@100GB)</th> <td>383,102.1</td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td>94.273000 seconds</td> </tr> </table> <h3>Duration of stream execution</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>10/03/2014 15:05:07</td> <td>10/03/2014 15:05:40</td> <td>0:00:33</td> </tr> <tr> <th>Stream 1</th> <td>10/03/2014 15:05:42</td> <td>10/03/2014 15:07:15</td> <td>0:01:33</td> </tr> <tr> <th>Stream 2</th> <td>10/03/2014 15:05:42</td> <td>10/03/2014 15:07:15</td> <td>0:01:33</td> </tr> <tr> <th>Stream 3</th> <td>10/03/2014 15:05:42</td> <td>10/03/2014 15:07:16</td> <td>0:01:34</td> </tr> <tr> <th>Stream 4</th> <td>10/03/2014 15:05:42</td> <td>10/03/2014 15:07:14</td> <td>0:01:32</td> </tr> <tr> <th>Stream 5</th> <td>10/03/2014 15:05:42</td> <td>10/03/2014 15:07:15</td> <td>0:01:33</td> </tr> <tr> <th>Refresh 0</th> <td>10/03/2014 15:05:07</td> <td>10/03/2014 15:05:13</td> <td>0:00:06</td> </tr> <tr> <th> </th> <td>10/03/2014 15:05:41</td> <td>10/03/2014 15:05:42</td> <td>0:00:01</td> </tr> <tr> <th>Refresh 1</th> <td>10/03/2014 15:06:48</td> <td>10/03/2014 15:07:03</td> <td>0:00:15</td> </tr> <tr> <th>Refresh 2</th> <td>10/03/2014 15:05:42</td> <td>10/03/2014 15:06:06</td> <td>0:00:24</td> </tr> <tr> <th>Refresh 3</th> <td>10/03/2014 15:06:06</td> <td>10/03/2014 15:06:20</td> <td>0:00:14</td> </tr> <tr> <th>Refresh 4</th> <td>10/03/2014 15:06:20</td> <td>10/03/2014 15:06:35</td> <td>0:00:15</td> </tr> <tr> <th>Refresh 5</th> <td>10/03/2014 15:06:35</td> <td>10/03/2014 15:06:48</td> <td>0:00:13</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> </tr> <tr> <th>Stream 0</th> <td>2.045198</td> <td>0.337315</td> <td>1.129548</td> <td>0.327029</td> <td>1.230955</td> <td>0.473090</td> <td>0.979096</td> <td>0.852639</td> </tr> <tr> <th>Stream 1</th> <td>4.521951</td> <td>0.596538</td> <td>3.464342</td> <td>1.167101</td> <td>3.944699</td> <td>1.744325</td> <td>5.442328</td> <td>4.706185</td> </tr> <tr> <th>Stream 2</th> <td>4.678728</td> <td>0.837205</td> <td>3.594060</td> <td>1.911751</td> <td>3.942459</td> <td>0.947788</td> <td>3.821267</td> <td>4.686319</td> </tr> <tr> <th>Stream 3</th> <td>5.126384</td> <td>0.932394</td> <td>0.961762</td> <td>1.043759</td> <td>5.359990</td> <td>1.035597</td> <td>3.056079</td> <td>5.803445</td> </tr> <tr> <th>Stream 4</th> <td>4.497118</td> <td>0.381036</td> <td>4.665412</td> <td>1.224975</td> <td>5.316591</td> <td>1.666253</td> <td>2.297872</td> <td>6.425171</td> </tr> <tr> <th>Stream 5</th> <td>4.080968</td> <td>0.493741</td> <td>4.416305</td> <td>0.879202</td> <td>5.705877</td> <td>1.615987</td> <td>3.846881</td> <td>3.346686</td> </tr> <tr> <th>Min Qi</th> <td>4.080968</td> <td>0.381036</td> <td>0.961762</td> <td>0.879202</td> <td>3.942459</td> <td>0.947788</td> <td>2.297872</td> <td>3.346686</td> </tr> <tr> <th>Max Qi</th> <td>5.126384</td> <td>0.932394</td> <td>4.665412</td> <td>1.911751</td> <td>5.705877</td> <td>1.744325</td> <td>5.442328</td> <td>6.425171</td> </tr> <tr> <th>Avg Qi</th> <td>4.581030</td> <td>0.648183</td> <td>3.420376</td> <td>1.245358</td> <td>4.853923</td> <td>1.401990</td> <td>3.692885</td> <td>4.993561</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>3.575916</td> <td>2.786656</td> <td>1.579488</td> <td>0.611454</td> <td>3.132460</td> <td>0.685095</td> <td>0.955559</td> <td>1.060110</td> </tr> <tr> <th>Stream 1</th> <td>9.551437</td> <td>7.187181</td> <td>5.816455</td> <td>2.004946</td> <td>9.461347</td> <td>5.624020</td> <td>5.517677</td> <td>2.924265</td> </tr> <tr> <th>Stream 2</th> <td>9.637427</td> <td>6.641804</td> <td>6.359532</td> <td>2.412576</td> <td>8.819754</td> <td>3.335494</td> <td>4.549792</td> <td>3.163920</td> </tr> <tr> <th>Stream 3</th> <td>11.041451</td> <td>6.464479</td> <td>6.982671</td> <td>3.272975</td> <td>8.342983</td> <td>3.448635</td> <td>4.405911</td> <td>2.886393</td> </tr> <tr> <th>Stream 4</th> <td>8.860228</td> <td>6.754529</td> <td>7.065501</td> <td>3.225236</td> <td>8.789565</td> <td>3.419165</td> <td>4.240718</td> <td>2.399092</td> </tr> <tr> <th>Stream 5</th> <td>7.339672</td> <td>8.121027</td> <td>6.261988</td> <td>2.711946</td> <td>8.764934</td> <td>3.106366</td> <td>6.544712</td> <td>3.472092</td> </tr> <tr> <th>Min Qi</th> <td>7.339672</td> <td>6.464479</td> <td>5.816455</td> <td>2.004946</td> <td>8.342983</td> <td>3.106366</td> <td>4.240718</td> <td>2.399092</td> </tr> <tr> <th>Max Qi</th> <td>11.041451</td> <td>8.121027</td> <td>7.065501</td> <td>3.272975</td> <td>9.461347</td> <td>5.624020</td> <td>6.544712</td> <td>3.472092</td> </tr> <tr> <th>Avg Qi</th> <td>9.286043</td> <td>7.033804</td> <td>6.497229</td> <td>2.725536</td> <td>8.835717</td> <td>3.786736</td> <td>5.051762</td> <td>2.969152</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>1.433789</td> <td>0.972152</td> <td>0.780247</td> <td>1.287222</td> <td>1.360084</td> <td>0.254051</td> <td>6.201742</td> <td>1.219707</td> </tr> <tr> <th>Stream 1</th> <td>3.398354</td> <td>2.591249</td> <td>3.021207</td> <td>4.663204</td> <td>4.775704</td> <td>1.116547</td> <td>8.770115</td> <td>5.643550</td> </tr> <tr> <th>Stream 2</th> <td>6.811520</td> <td>3.411846</td> <td>2.634076</td> <td>4.296810</td> <td>4.669635</td> <td>2.282003</td> <td>18.039617</td> <td>6.060465</td> </tr> <tr> <th>Stream 3</th> <td>4.947110</td> <td>2.479268</td> <td>2.952951</td> <td>6.431644</td> <td>5.469152</td> <td>1.816467</td> <td>8.271266</td> <td>5.498956</td> </tr> <tr> <th>Stream 4</th> <td>5.240237</td> <td>2.062261</td> <td>2.734378</td> <td>6.055141</td> <td>2.997684</td> <td>2.519301</td> <td>7.889700</td> <td>6.944722</td> </tr> <tr> <th>Stream 5</th> <td>4.839670</td> <td>3.379315</td> <td>3.231582</td> <td>6.255944</td> <td>3.759509</td> <td>1.347830</td> <td>8.707303</td> <td>4.376033</td> </tr> <tr> <th>Min Qi</th> <td>3.398354</td> <td>2.062261</td> <td>2.634076</td> <td>4.296810</td> <td>2.997684</td> <td>1.116547</td> <td>7.889700</td> <td>4.376033</td> </tr> <tr> <th>Max Qi</th> <td>6.811520</td> <td>3.411846</td> <td>3.231582</td> <td>6.431644</td> <td>5.469152</td> <td>2.519301</td> <td>18.039617</td> <td>6.944722</td> </tr> <tr> <th>Avg Qi</th> <td>5.047378</td> <td>2.784788</td> <td>2.914839</td> <td>5.540549</td> <td>4.334337</td> <td>1.816430</td> <td>10.335600</td> <td>5.704745</td> </tr> </table> <h2>100 GB Run 2</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>October 3, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>100</td> </tr> <tr> <th style="text-align:right">Total Data Storage/Database Size</th> <td>0M</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>5</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td>387,890.3</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td>433,066.6</td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@100GB)</th> <td>409,856.5</td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td>91.541000 seconds</td> </tr> </table> <h3>Duration of stream execution</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>10/03/2014 15:07:19</td> <td>10/03/2014 15:07:47</td> <td>0:00:28</td> </tr> <tr> <th>Stream 1</th> <td>10/03/2014 15:07:48</td> <td>10/03/2014 15:09:19</td> <td>0:01:31</td> </tr> <tr> <th>Stream 2</th> <td>10/03/2014 15:07:48</td> <td>10/03/2014 15:09:16</td> <td>0:01:28</td> </tr> <tr> <th>Stream 3</th> <td>10/03/2014 15:07:48</td> <td>10/03/2014 15:09:17</td> <td>0:01:29</td> </tr> <tr> <th>Stream 4</th> <td>10/03/2014 15:07:48</td> <td>10/03/2014 15:09:16</td> <td>0:01:28</td> </tr> <tr> <th>Stream 5</th> <td>10/03/2014 15:07:48</td> <td>10/03/2014 15:09:20</td> <td>0:01:32</td> </tr> <tr> <th>Refresh 0</th> <td>10/03/2014 15:07:19</td> <td>10/03/2014 15:07:22</td> <td>0:00:03</td> </tr> <tr> <th> </th> <td>10/03/2014 15:07:47</td> <td>10/03/2014 15:07:48</td> <td>0:00:01</td> </tr> <tr> <th>Refresh 1</th> <td>10/03/2014 15:08:45</td> <td>10/03/2014 15:08:59</td> <td>0:00:14</td> </tr> <tr> <th>Refresh 2</th> <td>10/03/2014 15:07:49</td> <td>10/03/2014 15:08:02</td> <td>0:00:13</td> </tr> <tr> <th>Refresh 3</th> <td>10/03/2014 15:08:02</td> <td>10/03/2014 15:08:17</td> <td>0:00:15</td> </tr> <tr> <th>Refresh 4</th> <td>10/03/2014 15:08:17</td> <td>10/03/2014 15:08:29</td> <td>0:00:12</td> </tr> <tr> <th>Refresh 5</th> <td>10/03/2014 15:08:29</td> <td>10/03/2014 15:08:45</td> <td>0:00:16</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> </tr> <tr> <th>Stream 0</th> <td>2.081986</td> <td>0.208487</td> <td>0.902462</td> <td>0.313160</td> <td>1.312273</td> <td>0.493157</td> <td>0.926629</td> <td>0.786345</td> </tr> <tr> <th>Stream 1</th> <td>2.755427</td> <td>0.911578</td> <td>3.618085</td> <td>0.664407</td> <td>3.740112</td> <td>2.118189</td> <td>4.738754</td> <td>6.551446</td> </tr> <tr> <th>Stream 2</th> <td>4.189612</td> <td>0.957921</td> <td>5.267355</td> <td>2.152479</td> <td>6.068005</td> <td>1.263380</td> <td>4.251842</td> <td>3.620160</td> </tr> <tr> <th>Stream 3</th> <td>4.708834</td> <td>0.981651</td> <td>2.411839</td> <td>0.790955</td> <td>4.384516</td> <td>1.322670</td> <td>2.641571</td> <td>4.771831</td> </tr> <tr> <th>Stream 4</th> <td>3.739567</td> <td>1.185884</td> <td>2.863871</td> <td>1.517891</td> <td>5.946967</td> <td>1.179960</td> <td>3.840560</td> <td>4.926325</td> </tr> <tr> <th>Stream 5</th> <td>5.258746</td> <td>0.705228</td> <td>3.460904</td> <td>0.951328</td> <td>4.530620</td> <td>1.104500</td> <td>3.226494</td> <td>4.041142</td> </tr> <tr> <th>Min Qi</th> <td>2.755427</td> <td>0.705228</td> <td>2.411839</td> <td>0.664407</td> <td>3.740112</td> <td>1.104500</td> <td>2.641571</td> <td>3.620160</td> </tr> <tr> <th>Max Qi</th> <td>5.258746</td> <td>1.185884</td> <td>5.267355</td> <td>2.152479</td> <td>6.068005</td> <td>2.118189</td> <td>4.738754</td> <td>6.551446</td> </tr> <tr> <th>Avg Qi</th> <td>4.130437</td> <td>0.948452</td> <td>3.524411</td> <td>1.215412</td> <td>4.934044</td> <td>1.397740</td> <td>3.739844</td> <td>4.782181</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>3.226685</td> <td>1.878227</td> <td>1.802562</td> <td>0.676499</td> <td>3.145884</td> <td>0.653129</td> <td>0.963449</td> <td>0.990524</td> </tr> <tr> <th>Stream 1</th> <td>8.842030</td> <td>5.630466</td> <td>5.728147</td> <td>2.643227</td> <td>9.615551</td> <td>3.197855</td> <td>4.676538</td> <td>4.285251</td> </tr> <tr> <th>Stream 2</th> <td>9.508612</td> <td>5.288044</td> <td>4.319998</td> <td>1.492915</td> <td>9.431995</td> <td>3.206360</td> <td>3.859749</td> <td>3.201996</td> </tr> <tr> <th>Stream 3</th> <td>10.480224</td> <td>5.880274</td> <td>4.517320</td> <td>2.509405</td> <td>6.913159</td> <td>2.892479</td> <td>6.408602</td> <td>2.938061</td> </tr> <tr> <th>Stream 4</th> <td>8.824111</td> <td>5.752413</td> <td>5.997959</td> <td>2.581237</td> <td>8.954756</td> <td>3.351951</td> <td>2.420598</td> <td>4.148455</td> </tr> <tr> <th>Stream 5</th> <td>4.905553</td> <td>7.099111</td> <td>5.121041</td> <td>2.516020</td> <td>9.354924</td> <td>3.955638</td> <td>4.389209</td> <td>3.818902</td> </tr> <tr> <th>Min Qi</th> <td>4.905553</td> <td>5.288044</td> <td>4.319998</td> <td>1.492915</td> <td>6.913159</td> <td>2.892479</td> <td>2.420598</td> <td>2.938061</td> </tr> <tr> <th>Max Qi</th> <td>10.480224</td> <td>7.099111</td> <td>5.997959</td> <td>2.643227</td> <td>9.615551</td> <td>3.955638</td> <td>6.408602</td> <td>4.285251</td> </tr> <tr> <th>Avg Qi</th> <td>8.512106</td> <td>5.930062</td> <td>5.136893</td> <td>2.348561</td> <td>8.854077</td> <td>3.320857</td> <td>4.350939</td> <td>3.678533</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>1.405338</td> <td>0.868313</td> <td>0.806277</td> <td>1.123366</td> <td>1.314028</td> <td>0.233214</td> <td>2.590459</td> <td>1.230242</td> </tr> <tr> <th>Stream 1</th> <td>5.191045</td> <td>3.171244</td> <td>3.403836</td> <td>4.604523</td> <td>3.721133</td> <td>0.892096</td> <td>7.136841</td> <td>6.500452</td> </tr> <tr> <th>Stream 2</th> <td>6.282687</td> <td>2.845465</td> <td>3.024786</td> <td>4.086546</td> <td>3.530743</td> <td>0.619683</td> <td>9.263671</td> <td>4.826173</td> </tr> <tr> <th>Stream 3</th> <td>6.040787</td> <td>2.659766</td> <td>2.787273</td> <td>6.210077</td> <td>3.902190</td> <td>2.175417</td> <td>7.974860</td> <td>6.689780</td> </tr> <tr> <th>Stream 4</th> <td>4.978721</td> <td>2.542674</td> <td>3.518783</td> <td>4.385571</td> <td>3.906211</td> <td>0.918752</td> <td>6.303352</td> <td>5.139326</td> </tr> <tr> <th>Stream 5</th> <td>5.208600</td> <td>3.761975</td> <td>3.682886</td> <td>7.874493</td> <td>5.017600</td> <td>2.087150</td> <td>7.999074</td> <td>7.978154</td> </tr> <tr> <th>Min Qi</th> <td>4.978721</td> <td>2.542674</td> <td>2.787273</td> <td>4.086546</td> <td>3.530743</td> <td>0.619683</td> <td>6.303352</td> <td>4.826173</td> </tr> <tr> <th>Max Qi</th> <td>6.282687</td> <td>3.761975</td> <td>3.682886</td> <td>7.874493</td> <td>5.017600</td> <td>2.175417</td> <td>9.263671</td> <td>7.978154</td> </tr> <tr> <th>Avg Qi</th> <td>5.540368</td> <td>2.996225</td> <td>3.283513</td> <td>5.432242</td> <td>4.015575</td> <td>1.338620</td> <td>7.735560</td> <td>6.226777</td> </tr> </table> <h2>1000 GB Run 1</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>October 3, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>1000</td> </tr> <tr> <th style="text-align:right">Total Data Storage/Database Size</th> <td>26M</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>7</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td>136,744.5</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td>147,374.6</td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@1000GB)</th> <td>141,960.1</td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td>3,761.953000 seconds</td> </tr> </table> <h3>Duration of stream execution</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>10/03/2014 09:18:42</td> <td>10/03/2014 09:34:12</td> <td>0:15:30</td> </tr> <tr> <th>Stream 1</th> <td>10/03/2014 09:34:43</td> <td>10/03/2014 10:35:42</td> <td>1:00:59</td> </tr> <tr> <th>Stream 2</th> <td>10/03/2014 09:34:43</td> <td>10/03/2014 10:37:14</td> <td>1:02:31</td> </tr> <tr> <th>Stream 3</th> <td>10/03/2014 09:34:43</td> <td>10/03/2014 10:37:25</td> <td>1:02:42</td> </tr> <tr> <th>Stream 4</th> <td>10/03/2014 09:34:43</td> <td>10/03/2014 10:33:31</td> <td>0:58:48</td> </tr> <tr> <th>Stream 5</th> <td>10/03/2014 09:34:43</td> <td>10/03/2014 10:35:26</td> <td>1:00:43</td> </tr> <tr> <th>Stream 6</th> <td>10/03/2014 09:34:43</td> <td>10/03/2014 10:28:00</td> <td>0:53:17</td> </tr> <tr> <th>Stream 7</th> <td>10/03/2014 09:34:43</td> <td>10/03/2014 10:35:42</td> <td>1:00:59</td> </tr> <tr> <th>Refresh 0</th> <td>10/03/2014 09:18:42</td> <td>10/03/2014 09:19:27</td> <td>0:00:45</td> </tr> <tr> <th> </th> <td>10/03/2014 09:34:12</td> <td>10/03/2014 09:34:42</td> <td>0:00:30</td> </tr> <tr> <th>Refresh 1</th> <td>10/03/2014 09:43:03</td> <td>10/03/2014 09:43:38</td> <td>0:00:35</td> </tr> <tr> <th>Refresh 2</th> <td>10/03/2014 09:34:43</td> <td>10/03/2014 09:36:54</td> <td>0:02:11</td> </tr> <tr> <th>Refresh 3</th> <td>10/03/2014 09:36:53</td> <td>10/03/2014 09:38:39</td> <td>0:01:46</td> </tr> <tr> <th>Refresh 4</th> <td>10/03/2014 09:38:39</td> <td>10/03/2014 09:39:22</td> <td>0:00:43</td> </tr> <tr> <th>Refresh 5</th> <td>10/03/2014 09:39:23</td> <td>10/03/2014 09:41:09</td> <td>0:01:46</td> </tr> <tr> <th>Refresh 6</th> <td>10/03/2014 09:41:09</td> <td>10/03/2014 09:42:15</td> <td>0:01:06</td> </tr> <tr> <th>Refresh 7</th> <td>10/03/2014 09:42:15</td> <td>10/03/2014 09:43:02</td> <td>0:00:47</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> <th>Stream 0</th> <td>104.488583</td> <td>18.351559</td> <td>24.631282</td> <td>36.195531</td> <td>36.319915</td> <td>3.807790</td> <td>22.750889</td> <td>31.190630</td> </tr> <tr> <th>Stream 1</th> <td>209.323441</td> <td>26.205435</td> <td>59.637373</td> <td>245.808484</td> <td>60.699333</td> <td>22.369379</td> <td>289.435780</td> <td>335.733425</td> </tr> <tr> <th>Stream 2</th> <td>109.134446</td> <td>64.185831</td> <td>96.131735</td> <td>108.459418</td> <td>310.273986</td> <td>53.595127</td> <td>152.242755</td> <td>104.350098</td> </tr> <tr> <th>Stream 3</th> <td>73.321611</td> <td>215.535408</td> <td>69.543101</td> <td>12.423757</td> <td>64.445611</td> <td>38.254747</td> <td>122.952872</td> <td>98.713213</td> </tr> <tr> <th>Stream 4</th> <td>110.875875</td> <td>4.272757</td> <td>78.697314</td> <td>16.316807</td> <td>59.746855</td> <td>23.447211</td> <td>353.190412</td> <td>342.549908</td> </tr> <tr> <th>Stream 5</th> <td>41.972337</td> <td>5.978707</td> <td>60.784575</td> <td>34.219229</td> <td>42.372449</td> <td>344.590640</td> <td>146.186614</td> <td>274.972270</td> </tr> <tr> <th>Stream 6</th> <td>115.760155</td> <td>18.692078</td> <td>58.493147</td> <td>9.193234</td> <td>49.831932</td> <td>19.081395</td> <td>60.603109</td> <td>128.095501</td> </tr> <tr> <th>Stream 7</th> <td>58.601744</td> <td>118.126585</td> <td>297.327543</td> <td>298.578268</td> <td>714.284222</td> <td>108.475250</td> <td>91.868151</td> <td>55.881029</td> </tr> <tr> <th>Min Qi</th> <td>41.972337</td> <td>4.272757</td> <td>58.493147</td> <td>9.193234</td> <td>42.372449</td> <td>19.081395</td> <td>60.603109</td> <td>55.881029</td> </tr> <tr> <th>Max Qi</th> <td>209.323441</td> <td>215.535408</td> <td>297.327543</td> <td>298.578268</td> <td>714.284222</td> <td>344.590640</td> <td>353.190412</td> <td>342.549908</td> </tr> <tr> <th>Avg Qi</th> <td>102.712801</td> <td>64.713829</td> <td>102.944970</td> <td>103.571314</td> <td>185.950627</td> <td>87.116250</td> <td>173.782813</td> <td>191.470778</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>41.777880</td> <td>10.035063</td> <td>16.125611</td> <td>9.245638</td> <td>209.443782</td> <td>111.271310</td> <td>37.821595</td> <td>9.483838</td> </tr> <tr> <th>Stream 1</th> <td>244.243830</td> <td>63.473338</td> <td>207.741931</td> <td>33.696956</td> <td>561.057408</td> <td>141.026049</td> <td>126.818051</td> <td>54.774792</td> </tr> <tr> <th>Stream 2</th> <td>189.297446</td> <td>144.853756</td> <td>56.292537</td> <td>184.781273</td> <td>501.330052</td> <td>49.965102</td> <td>107.736393</td> <td>85.691079</td> </tr> <tr> <th>Stream 3</th> <td>231.060699</td> <td>355.394713</td> <td>43.483645</td> <td>11.806590</td> <td>555.445111</td> <td>36.722686</td> <td>251.241817</td> <td>9.057850</td> </tr> <tr> <th>Stream 4</th> <td>227.371508</td> <td>32.207115</td> <td>108.880658</td> <td>139.922550</td> <td>532.697956</td> <td>57.106583</td> <td>159.198489</td> <td>153.088913</td> </tr> <tr> <th>Stream 5</th> <td>416.113856</td> <td>108.689389</td> <td>62.847727</td> <td>702.712683</td> <td>622.906487</td> <td>58.198961</td> <td>89.707091</td> <td>85.614769</td> </tr> <tr> <th>Stream 6</th> <td>228.019243</td> <td>62.474213</td> <td>88.227994</td> <td>282.932978</td> <td>432.387869</td> <td>238.544027</td> <td>61.486269</td> <td>56.950548</td> </tr> <tr> <th>Stream 7</th> <td>230.564416</td> <td>69.197517</td> <td>130.708759</td> <td>120.531103</td> <td>551.112816</td> <td>57.438478</td> <td>82.256530</td> <td>63.796403</td> </tr> <tr> <th>Min Qi</th> <td>189.297446</td> <td>32.207115</td> <td>43.483645</td> <td>11.806590</td> <td>432.387869</td> <td>36.722686</td> <td>61.486269</td> <td>9.057850</td> </tr> <tr> <th>Max Qi</th> <td>416.113856</td> <td>355.394713</td> <td>207.741931</td> <td>702.712683</td> <td>622.906487</td> <td>238.544027</td> <td>251.241817</td> <td>153.088913</td> </tr> <tr> <th>Avg Qi</th> <td>252.381571</td> <td>119.470006</td> <td>99.740464</td> <td>210.912019</td> <td>536.705386</td> <td>91.285984</td> <td>125.492091</td> <td>72.710622</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>22.897349</td> <td>47.870269</td> <td>12.735580</td> <td>25.982194</td> <td>46.091766</td> <td>6.623306</td> <td>45.120559</td> <td>30.016788</td> </tr> <tr> <th>Stream 1</th> <td>123.444839</td> <td>22.212194</td> <td>647.523826</td> <td>97.431531</td> <td>81.592165</td> <td>4.573040</td> <td>21.068225</td> <td>14.486185</td> </tr> <tr> <th>Stream 2</th> <td>80.853865</td> <td>622.651044</td> <td>288.656211</td> <td>336.409076</td> <td>70.925079</td> <td>33.578052</td> <td>82.910543</td> <td>48.001583</td> </tr> <tr> <th>Stream 3</th> <td>392.340812</td> <td>84.967695</td> <td>57.181935</td> <td>473.720060</td> <td>497.262620</td> <td>66.966740</td> <td>54.778284</td> <td>50.940094</td> </tr> <tr> <th>Stream 4</th> <td>97.069440</td> <td>301.705125</td> <td>338.035788</td> <td>258.992426</td> <td>103.699408</td> <td>28.750257</td> <td>23.858757</td> <td>13.626079</td> </tr> <tr> <th>Stream 5</th> <td>69.882110</td> <td>34.277914</td> <td>146.031938</td> <td>179.656129</td> <td>104.788154</td> <td>10.836148</td> <td>54.319823</td> <td>52.077352</td> </tr> <tr> <th>Stream 6</th> <td>141.310431</td> <td>247.242904</td> <td>94.392791</td> <td>702.775460</td> <td>80.142930</td> <td>19.969889</td> <td>46.027410</td> <td>19.136271</td> </tr> <tr> <th>Stream 7</th> <td>89.018281</td> <td>51.105998</td> <td>281.234432</td> <td>79.046122</td> <td>84.341517</td> <td>26.221892</td> <td>33.169666</td> <td>13.309634</td> </tr> <tr> <th>Min Qi</th> <td>69.882110</td> <td>22.212194</td> <td>57.181935</td> <td>79.046122</td> <td>70.925079</td> <td>4.573040</td> <td>21.068225</td> <td>13.309634</td> </tr> <tr> <th>Max Qi</th> <td>392.340812</td> <td>622.651044</td> <td>647.523826</td> <td>702.775460</td> <td>497.262620</td> <td>66.966740</td> <td>82.910543</td> <td>52.077352</td> </tr> <tr> <th>Avg Qi</th> <td>141.988540</td> <td>194.880411</td> <td>264.722417</td> <td>304.004401</td> <td>146.107410</td> <td>27.270860</td> <td>45.161815</td> <td>30.225314</td> </tr> </table> <h2>1000 GB Run 2</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>October 3, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>1000</td> </tr> <tr> <th style="text-align:right">Total Data Storage/Database Size</th> <td>26M</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>7</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td>199,652.0</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td>125,161.1</td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@1000GB)</th> <td>158,078.0</td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td>4,429.608000 seconds</td> </tr> </table> <h3>Duration of stream execution</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>10/03/2014 10:37:29</td> <td>10/03/2014 10:52:26</td> <td>0:14:57</td> </tr> <tr> <th>Stream 1</th> <td>10/03/2014 10:52:35</td> <td>10/03/2014 12:05:19</td> <td>1:12:44</td> </tr> <tr> <th>Stream 2</th> <td>10/03/2014 10:52:35</td> <td>10/03/2014 12:06:25</td> <td>1:13:50</td> </tr> <tr> <th>Stream 3</th> <td>10/03/2014 10:52:35</td> <td>10/03/2014 12:03:08</td> <td>1:10:33</td> </tr> <tr> <th>Stream 4</th> <td>10/03/2014 10:52:35</td> <td>10/03/2014 12:05:20</td> <td>1:12:45</td> </tr> <tr> <th>Stream 5</th> <td>10/03/2014 10:52:35</td> <td>10/03/2014 11:57:40</td> <td>1:05:05</td> </tr> <tr> <th>Stream 6</th> <td>10/03/2014 10:52:35</td> <td>10/03/2014 12:05:28</td> <td>1:12:53</td> </tr> <tr> <th>Stream 7</th> <td>10/03/2014 10:52:35</td> <td>10/03/2014 12:05:25</td> <td>1:12:50</td> </tr> <tr> <th>Refresh 0</th> <td>10/03/2014 10:37:29</td> <td>10/03/2014 10:37:52</td> <td>0:00:23</td> </tr> <tr> <th> </th> <td>10/03/2014 10:52:25</td> <td>10/03/2014 10:52:34</td> <td>0:00:09</td> </tr> <tr> <th>Refresh 1</th> <td>10/03/2014 11:01:44</td> <td>10/03/2014 11:02:29</td> <td>0:00:45</td> </tr> <tr> <th>Refresh 2</th> <td>10/03/2014 10:52:35</td> <td>10/03/2014 10:54:50</td> <td>0:02:15</td> </tr> <tr> <th>Refresh 3</th> <td>10/03/2014 10:54:50</td> <td>10/03/2014 10:57:02</td> <td>0:02:12</td> </tr> <tr> <th>Refresh 4</th> <td>10/03/2014 10:57:05</td> <td>10/03/2014 10:58:47</td> <td>0:01:42</td> </tr> <tr> <th>Refresh 5</th> <td>10/03/2014 10:58:47</td> <td>10/03/2014 10:59:46</td> <td>0:00:59</td> </tr> <tr> <th>Refresh 6</th> <td>10/03/2014 10:59:45</td> <td>10/03/2014 11:00:38</td> <td>0:00:53</td> </tr> <tr> <th>Refresh 7</th> <td>10/03/2014 11:00:39</td> <td>10/03/2014 11:01:44</td> <td>0:01:05</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> </tr> <tr> <th>Stream 0</th> <td>34.105419</td> <td>1.439089</td> <td>9.802183</td> <td>2.033956</td> <td>10.525742</td> <td>3.356152</td> <td>23.953729</td> <td>36.199533</td> </tr> <tr> <th>Stream 1</th> <td>26.598252</td> <td>150.572833</td> <td>41.930330</td> <td>86.870320</td> <td>50.604856</td> <td>201.001372</td> <td>61.638366</td> <td>244.013359</td> </tr> <tr> <th>Stream 2</th> <td>50.129895</td> <td>102.219282</td> <td>12.380935</td> <td>102.319615</td> <td>62.577229</td> <td>43.454392</td> <td>891.076608</td> <td>407.640626</td> </tr> <tr> <th>Stream 3</th> <td>269.947278</td> <td>53.172724</td> <td>54.649973</td> <td>11.460062</td> <td>66.695722</td> <td>17.336698</td> <td>63.371232</td> <td>91.158050</td> </tr> <tr> <th>Stream 4</th> <td>41.149221</td> <td>22.520836</td> <td>28.707973</td> <td>509.984321</td> <td>68.916549</td> <td>17.525025</td> <td>702.191490</td> <td>666.450230</td> </tr> <tr> <th>Stream 5</th> <td>59.179045</td> <td>30.734442</td> <td>99.504351</td> <td>11.145990</td> <td>101.334340</td> <td>21.660836</td> <td>74.625589</td> <td>535.160207</td> </tr> <tr> <th>Stream 6</th> <td>225.105215</td> <td>55.567328</td> <td>46.749707</td> <td>554.474507</td> <td>215.657091</td> <td>54.362551</td> <td>72.960653</td> <td>442.194302</td> </tr> <tr> <th>Stream 7</th> <td>220.993226</td> <td>28.528230</td> <td>47.543365</td> <td>336.191006</td> <td>308.931194</td> <td>9.767397</td> <td>850.258452</td> <td>66.121298</td> </tr> <tr> <th>Min Qi</th> <td>26.598252</td> <td>22.520836</td> <td>12.380935</td> <td>11.145990</td> <td>50.604856</td> <td>9.767397</td> <td>61.638366</td> <td>66.121298</td> </tr> <tr> <th>Max Qi</th> <td>269.947278</td> <td>150.572833</td> <td>99.504351</td> <td>554.474507</td> <td>308.931194</td> <td>201.001372</td> <td>891.076608</td> <td>666.450230</td> </tr> <tr> <th>Avg Qi</th> <td>127.586019</td> <td>63.330811</td> <td>47.352376</td> <td>230.349403</td> <td>124.959569</td> <td>52.158324</td> <td>388.017484</td> <td>350.391153</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>50.439615</td> <td>9.287196</td> <td>15.892947</td> <td>7.112715</td> <td>250.527755</td> <td>131.478131</td> <td>54.458992</td> <td>10.525842</td> </tr> <tr> <th>Stream 1</th> <td>420.919329</td> <td>317.402771</td> <td>101.818338</td> <td>403.213385</td> <td>724.539887</td> <td>160.669174</td> <td>65.374584</td> <td>28.563034</td> </tr> <tr> <th>Stream 2</th> <td>464.378760</td> <td>210.938167</td> <td>23.395678</td> <td>545.086468</td> <td>736.005716</td> <td>54.680686</td> <td>398.880053</td> <td>34.018918</td> </tr> <tr> <th>Stream 3</th> <td>350.083270</td> <td>321.781561</td> <td>48.652019</td> <td>435.954962</td> <td>378.872739</td> <td>100.588804</td> <td>289.350342</td> <td>190.140640</td> </tr> <tr> <th>Stream 4</th> <td>306.265994</td> <td>249.621982</td> <td>79.280220</td> <td>221.255121</td> <td>348.932746</td> <td>49.555802</td> <td>100.062439</td> <td>61.368814</td> </tr> <tr> <th>Stream 5</th> <td>511.923087</td> <td>133.018420</td> <td>134.199065</td> <td>9.655693</td> <td>662.658830</td> <td>104.380635</td> <td>82.847242</td> <td>59.952271</td> </tr> <tr> <th>Stream 6</th> <td>578.362701</td> <td>61.221715</td> <td>145.613349</td> <td>47.957006</td> <td>621.993889</td> <td>256.150595</td> <td>77.124777</td> <td>91.163005</td> </tr> <tr> <th>Stream 7</th> <td>418.450091</td> <td>391.818564</td> <td>29.360218</td> <td>17.236628</td> <td>761.850888</td> <td>31.952329</td> <td>50.393082</td> <td>27.530882</td> </tr> <tr> <th>Min Qi</th> <td>306.265994</td> <td>61.221715</td> <td>23.395678</td> <td>9.655693</td> <td>348.932746</td> <td>31.952329</td> <td>50.393082</td> <td>27.530882</td> </tr> <tr> <th>Max Qi</th> <td>578.362701</td> <td>391.818564</td> <td>145.613349</td> <td>545.086468</td> <td>761.850888</td> <td>256.150595</td> <td>398.880053</td> <td>190.140640</td> </tr> <tr> <th>Avg Qi</th> <td>435.769033</td> <td>240.829026</td> <td>80.331270</td> <td>240.051323</td> <td>604.979242</td> <td>108.282575</td> <td>152.004646</td> <td>70.391081</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>22.444111</td> <td>37.978532</td> <td>13.347320</td> <td>26.553364</td> <td>115.511143</td> <td>7.670304</td> <td>22.771613</td> <td>8.761026</td> </tr> <tr> <th>Stream 1</th> <td>329.153807</td> <td>19.198590</td> <td>258.455295</td> <td>556.256015</td> <td>99.647793</td> <td>14.878746</td> <td>32.803289</td> <td>8.771923</td> </tr> <tr> <th>Stream 2</th> <td>76.940373</td> <td>74.916489</td> <td>75.246897</td> <td>16.035355</td> <td>14.403643</td> <td>32.348500</td> <td>91.981362</td> <td>41.426540</td> </tr> <tr> <th>Stream 3</th> <td>88.918404</td> <td>238.858707</td> <td>221.257060</td> <td>688.441713</td> <td>247.669761</td> <td>5.345632</td> <td>70.780594</td> <td>49.352955</td> </tr> <tr> <th>Stream 4</th> <td>497.105081</td> <td>167.874781</td> <td>67.668514</td> <td>76.820831</td> <td>78.585717</td> <td>3.655421</td> <td>73.165786</td> <td>29.401670</td> </tr> <tr> <th>Stream 5</th> <td>309.991618</td> <td>123.023557</td> <td>380.801141</td> <td>347.055909</td> <td>93.478502</td> <td>18.351491</td> <td>33.338814</td> <td>12.557542</td> </tr> <tr> <th>Stream 6</th> <td>57.200926</td> <td>154.489850</td> <td>386.007137</td> <td>103.558355</td> <td>32.676369</td> <td>92.863316</td> <td>35.576966</td> <td>14.061801</td> </tr> <tr> <th>Stream 7</th> <td>160.332088</td> <td>46.934177</td> <td>340.957970</td> <td>84.479720</td> <td>78.985110</td> <td>60.568796</td> <td>44.362737</td> <td>8.831746</td> </tr> <tr> <th>Min Qi</th> <td>57.200926</td> <td>19.198590</td> <td>67.668514</td> <td>16.035355</td> <td>14.403643</td> <td>3.655421</td> <td>32.803289</td> <td>8.771923</td> </tr> <tr> <th>Max Qi</th> <td>497.105081</td> <td>238.858707</td> <td>386.007137</td> <td>688.441713</td> <td>247.669761</td> <td>92.863316</td> <td>91.981362</td> <td>49.352955</td> </tr> <tr> <th>Avg Qi</th> <td>217.091757</td> <td>117.899450</td> <td>247.199145</td> <td>267.521128</td> <td>92.206699</td> <td>32.573129</td> <td>54.572793</td> <td>23.486311</td> </tr> </table> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aab52abc0b8"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aab53935178"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aab537ea348"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aab516a28b8"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aab514b53d8"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aaba8e12aa8"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aab537ea308"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aaba97e18d8"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aab52a6add8"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aaba9725b78"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aab51109e68"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aaba97e1898"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aab51a706a8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aaba97e18b8"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aaba9725b98"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x2aab52abc098"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aab5130e068"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aab52979f88"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aab53015408"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aabd2975158"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:05:31.563713-04:00
In Hoc Signo Vinces (part 19 of n): Scalability, 1000G, and 3000G
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-09-30#1823
2014-09-30T20:33:49Z
<p>Scalability, specifically linear scalability, means that twice the data takes twice as long to process, or that double the gear processes the same data in half the time. This is only literally true for "embarrassingly parallel" workloads.</p> <p>There are parts of <a href="http://www.tpc.org/tpch/" id="link-id0x2aac693da068">TPC-H</a> which have an embarrassingly parallel nature, like <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac3757a2d8">Q1</a> and <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aac6ad4c0b8">Q7</a>. There are parts that are almost as easy, like Q14, <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac116f1f88">Q17</a>, <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aacaa6ec0b8">Q19</a>, and <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aaca8c66f18">Q21</a>, where there is a big scan and a selective hash join with a hash table small enough to replicate everywhere. The scan scales linearly; building the hash does not, since it is done at single-server speed (once in each process). Some queries like <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac116f2728">Q9</a> and <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aac6a56bd28">Q13</a> end up doing a big cross-partition join which runs into communication overheads.</p> <p>This is our first look at how performance behaves with bigger data and a larger platform. The results shown here are interesting but are not final. I bet I can do better; by how much is what we'll find out soon enough.</p> <p>We will here compare a 1000G setup on my desktop, and a 3000G setup at the <a href="http://www.cwi.nl/" id="link-id0x2aac6ad4c2b8">CWI</a>'s <a href="http://scilens.project.cwi.nl/" id="link-id0x2aac117f8cb8">Scilens cluster</a>. The former is 2 boxes of dual Xeon E5 2630, and the latter is 8 boxes of dual Xeon E5 2650v2. All things run from memory and both have QDR IB interconnect. Counting cores and clock, the CWI cluster is 6x larger.</p> <p>As a rough approximation, for the worst queries, 6x the gear runs 3x the data in the same amount of real time. The 1000G setup has near full platform utilization and the 3000G setup has about half platform utilization. In both cases, running two instances of the same query at the same time takes twice as long.</p> <p>We use <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aaca8c272f8">Q9</a> for this study. The plan makes a hash table of <code>part</code> with 1/14 of all <code>parts</code>, replicating to all processes. Then there is a hash table of <code>partsupp</code> with a key of <code>ps_partkey, ps_suppkey</code>, and a dependent of <code>ps_supplycost</code>. This is much larger than the <code>part</code> hash table and is therefore partitioned on <code>ps_partkey</code>. The build is for 1/14th of <code>partsupp</code>. Then there is a scan of <code>lineitem</code> filtered by the <code>part</code> hash table; then a cross-partition join to the <code>partsupp</code> hash table; then a cross partition join to <code>orders</code>, this time by index; then a hash join on a replicated hash table of <code>supplier</code>; then <code>nation</code>; then aggregation. The aggregation is done in each slice; then the slices are added up at the end.</p> <p>The plan could be made better by one fewer partition crossing. Now there is a crossing from <code>l_orderkey</code> to <code>l_partkey</code> and back to <code>o_orderkey</code>. This would not be so if the cost model knew that the <code>partsupp</code> always hits. The cost model thinks it hits 1/14 of the time, because it does not know that the selection on the build is exactly the same as on the probe.</p> <p>For the present purposes, the extra crossing just serves to make the matter of interest more visible.</p> <p>So, for the 1000G setup, we have 43.6 seconds (s) and </p> <blockquote> <pre><code>Cluster 4 nodes, 44 s. 459 m/s 119788 KB/s 3120% cpu 0% read 19% clw threads 1r 0w 0i buffers 17622126 68 d 0 w 0 pfs</code> </pre></blockquote> <p>For the 3000G setup, we have 49.9 s and </p> <blockquote> <pre><code>Cluster 16 nodes, 50 s. 49389 m/s 1801815 KB/s 7283% cpu 0% read 18% clw threads 1r 0w 0i buffers 135122893 15895255 d 0 w 17 pfs</code> </pre></blockquote> <p>The platform utilization on the small system is better, at 31/48 (running/total threads); the large one has 73/256.</p> <p>The large case is clearly network bound. If this were for CPU only, it should be done in half the time it takes the small system to do 1000G. </p> <p>We confirm this by looking at write wait: 3940 seconds of thread time blocked on write over 50s of real time. The figures on the small one are 3.9s of thread time blocked for 39s of real time. The data transfer on the large one is 93 GB. </p> <p>How to block less? One idea would be to write less. So we try compression; there is a Google snappy-based message compression option in Virtuoso. </p> <p>We now get 39.6 s and</p> <blockquote> <pre><code>Cluster 16 nodes, 40 s. 65161 m/s 1239922 KB/s 10201% cpu 0% read 21% clw threads 1r 0w 0i buffers 52828440 172 d 0 w 0 pfs</code> </pre></blockquote> <p>The write block time is 397 s of thread time over 39 s of real time, 10x better. The data transfer is 50.9 GB after compression. Snappy is somewhat effective for compression and very fast; in CPU profile, it is under 3% of Q9 on the small system. Gains on the small system are less, though, since blocking is not a big issue to start with.</p> <p>This is still not full platform. But if the data transfer is further cut in half by a better plan, the situation will be quite good. Now we have 102/256 threads running, meaning that there could be another 40-50% of throughput to be added. The last 128 threads are second threads of a core, so count for roughly 30% of a real core.</p> <p>The main cluster-specific operation is a send from one to many. This is now done by formulating the message to each recipient in a chain of string buffers; then, after all the messages are prepared, these are optionally compressed and sent to their recipient. This is needlessly simple: Compressing can proceed if ever there is a would-block situation on writing. If all the compression is done, then a blocked write should switch to another recipient, and only after all recipients have a would-block situation, then the thread can call-select with all descriptors and block on them collectively. There is a piece of code to this effect, but is not now being used. It has been seen to add no value in small cases, but could be useful here. </p> <p>The IB fabric has been seen to do 1.8 GB/s bidirectionally on multiple independent point-to-point TCP links. This is about half the nominal 4 GB/s (40 Gbit/s with 10/8 encoding). So the aggregate throughputs that we see here are nowhere near the nominal spec of the network. Lower level interfaces and the occasional busy wait on the reading end could be tried to some advantage. We have not tried 10GbE either; but if that works at nominal speed, then 10GbE should also be good enough. We will try this at Amazon in due time.</p> <p>In the meantime, there is a 3000G test made at the CWI cluster without message compression. The score is about 4x that of the single server at 300G using the same hardware. The run is with approximately half platform utilization. There are three runs of power plus throughput, the first run being cold.</p> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th>Run</th> <th>Power</th> <th>Throughput</th> <th>Composite </th> </tr> <tr> <th>Run 1</th> <td style="text-align:right">305,881.5</td> <td style="text-align:right">1,072,411.9</td> <td style="text-align:right">572,739.8</td> </tr> <tr> <th>Run 2</th> <td style="text-align:right">1,292,085.1</td> <td style="text-align:right">1,179,391.6</td> <td style="text-align:right">1,234,453.1</td> </tr> <tr> <th>Run 3</th> <td style="text-align:right">1,178,534.1</td> <td style="text-align:right">1,092,936.2</td> <td style="text-align:right">1,134,928.4</td> </tr> </table> <p>The numerical quantities summaries follow. One problem of the run is a high peak of query memory consumption leading to slowdown. Some parts should probably be done in multiple passes to keep the peak lower and not run into swapping. The details will have to be sorted out. This is a demonstration of capability; the perfected accomplishment is to follow.</p> <h2>3000G Run 1</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>September 29, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>3000</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>8</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td>305,881.5</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td>1,072,411.9</td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@100GB)</th> <td>572,739.8</td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td>1,772.554000 seconds</td> </tr> </table> <h3>Duration of stream execution</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>09/29/2014 12:54:52</td> <td>09/29/2014 13:31:17</td> <td>0:36:25</td> </tr> <tr> <th>Stream 1</th> <td>09/29/2014 13:31:24</td> <td>09/29/2014 13:59:24</td> <td>0:28:00</td> </tr> <tr> <th>Stream 2</th> <td>09/29/2014 13:31:24</td> <td>09/29/2014 13:58:59</td> <td>0:27:35</td> </tr> <tr> <th>Stream 3</th> <td>09/29/2014 13:31:24</td> <td>09/29/2014 13:58:29</td> <td>0:27:05</td> </tr> <tr> <th>Stream 4</th> <td>09/29/2014 13:31:24</td> <td>09/29/2014 13:58:52</td> <td>0:27:28</td> </tr> <tr> <th>Stream 5</th> <td>09/29/2014 13:31:24</td> <td>09/29/2014 14:00:06</td> <td>0:28:42</td> </tr> <tr> <th>Stream 6</th> <td>09/29/2014 13:31:24</td> <td>09/29/2014 13:58:18</td> <td>0:26:54</td> </tr> <tr> <th>Stream 7</th> <td>09/29/2014 13:31:24</td> <td>09/29/2014 13:59:25</td> <td>0:28:01</td> </tr> <tr> <th>Stream 8</th> <td>09/29/2014 13:31:24</td> <td>09/29/2014 13:58:50</td> <td>0:27:26</td> </tr> <tr> <th>Refresh 0</th> <td>09/29/2014 12:54:52</td> <td>09/29/2014 12:56:59</td> <td>0:02:07</td> </tr> <tr> <th> </th> <td>09/29/2014 13:31:17</td> <td>09/29/2014 13:31:23</td> <td>0:00:06</td> </tr> <tr> <th>Refresh 1</th> <td>09/29/2014 14:00:38</td> <td>09/29/2014 14:01:11</td> <td>0:00:33</td> </tr> <tr> <th>Refresh 2</th> <td>09/29/2014 13:31:25</td> <td>09/29/2014 13:36:57</td> <td>0:05:32</td> </tr> <tr> <th>Refresh 3</th> <td>09/29/2014 13:36:56</td> <td>09/29/2014 13:47:02</td> <td>0:10:06</td> </tr> <tr> <th>Refresh 4</th> <td>09/29/2014 13:47:03</td> <td>09/29/2014 13:51:40</td> <td>0:04:37</td> </tr> <tr> <th>Refresh 5</th> <td>09/29/2014 13:51:42</td> <td>09/29/2014 13:56:40</td> <td>0:04:58</td> </tr> <tr> <th>Refresh 6</th> <td>09/29/2014 13:56:40</td> <td>09/29/2014 13:59:25</td> <td>0:02:45</td> </tr> <tr> <th>Refresh 7</th> <td>09/29/2014 13:59:25</td> <td>09/29/2014 14:00:10</td> <td>0:00:45</td> </tr> <tr> <th>Refresh 8</th> <td>09/29/2014 14:00:11</td> <td>09/29/2014 14:00:37</td> <td>0:00:26</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> <th>Stream 0</th> <td>601.576975</td> <td>90.803782</td> <td>108.725110</td> <td>177.112667</td> <td>171.995572</td> <td>2.098138</td> <td>15.768311</td> <td>152.511444</td> </tr> <tr> <th>Stream 1</th> <td>13.310341</td> <td>32.722946</td> <td>125.551415</td> <td>1.912836</td> <td>46.041675</td> <td>13.294214</td> <td>85.345068</td> <td>165.424288</td> </tr> <tr> <th>Stream 2</th> <td>19.425885</td> <td>9.248670</td> <td>150.855556</td> <td>7.085737</td> <td>88.445566</td> <td>10.490432</td> <td>49.318554</td> <td>322.500839</td> </tr> <tr> <th>Stream 3</th> <td>30.534391</td> <td>14.273478</td> <td>100.987791</td> <td>59.341763</td> <td>46.442443</td> <td>9.613795</td> <td>64.186196</td> <td>146.324186</td> </tr> <tr> <th>Stream 4</th> <td>28.211213</td> <td>37.134522</td> <td>64.189335</td> <td>10.931513</td> <td>100.610673</td> <td>9.929866</td> <td>112.270530</td> <td>108.489951</td> </tr> <tr> <th>Stream 5</th> <td>29.226411</td> <td>18.132589</td> <td>95.245160</td> <td>63.100068</td> <td>115.663908</td> <td>6.151231</td> <td>46.251309</td> <td>127.742471</td> </tr> <tr> <th>Stream 6</th> <td>30.750930</td> <td>20.888658</td> <td>108.894177</td> <td>55.168565</td> <td>82.016828</td> <td>69.451493</td> <td>65.161517</td> <td>103.697733</td> </tr> <tr> <th>Stream 7</th> <td>13.462570</td> <td>18.033847</td> <td>32.065492</td> <td>78.910373</td> <td>202.998301</td> <td>10.688279</td> <td>47.167022</td> <td>139.601948</td> </tr> <tr> <th>Stream 8</th> <td>24.354314</td> <td>16.711503</td> <td>112.008551</td> <td>8.307098</td> <td>126.849630</td> <td>7.127605</td> <td>51.083118</td> <td>98.648077</td> </tr> <tr> <th>Min Qi</th> <td>13.310341</td> <td>9.248670</td> <td>32.065492</td> <td>1.912836</td> <td>46.041675</td> <td>6.151231</td> <td>46.251309</td> <td>98.648077</td> </tr> <tr> <th>Max Qi</th> <td>30.750930</td> <td>37.134522</td> <td>150.855556</td> <td>78.910373</td> <td>202.998301</td> <td>69.451493</td> <td>112.270530</td> <td>322.500839</td> </tr> <tr> <th>Avg Qi</th> <td>23.659507</td> <td>20.893277</td> <td>98.724685</td> <td>35.594744</td> <td>101.133628</td> <td>17.093364</td> <td>65.097914</td> <td>151.553687</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>92.991259</td> <td>5.175922</td> <td>42.238393</td> <td>29.239879</td> <td>367.805534</td> <td>3.604910</td> <td>15.557396</td> <td>11.650267</td> </tr> <tr> <th>Stream 1</th> <td>149.502128</td> <td>30.197806</td> <td>50.786184</td> <td>217.190836</td> <td>283.545905</td> <td>11.653171</td> <td>73.321150</td> <td>116.860455</td> </tr> <tr> <th>Stream 2</th> <td>245.783668</td> <td>22.278841</td> <td>50.578731</td> <td>36.301810</td> <td>181.405269</td> <td>32.236754</td> <td>57.631764</td> <td>61.540533</td> </tr> <tr> <th>Stream 3</th> <td>377.782738</td> <td>24.129319</td> <td>84.097657</td> <td>10.959661</td> <td>171.698669</td> <td>8.973519</td> <td>54.532180</td> <td>45.527142</td> </tr> <tr> <th>Stream 4</th> <td>341.148908</td> <td>74.358770</td> <td>85.782399</td> <td>43.116347</td> <td>151.146233</td> <td>22.870727</td> <td>74.439693</td> <td>51.871535</td> </tr> <tr> <th>Stream 5</th> <td>72.259919</td> <td>11.424035</td> <td>79.310504</td> <td>9.833135</td> <td>562.871920</td> <td>14.961209</td> <td>127.861874</td> <td>55.377721</td> </tr> <tr> <th>Stream 6</th> <td>373.301225</td> <td>41.379753</td> <td>81.983260</td> <td>9.373200</td> <td>95.039317</td> <td>19.071346</td> <td>76.159452</td> <td>48.324504</td> </tr> <tr> <th>Stream 7</th> <td>449.871952</td> <td>16.099152</td> <td>48.047940</td> <td>8.559784</td> <td>211.094730</td> <td>10.569071</td> <td>26.710228</td> <td>72.571454</td> </tr> <tr> <th>Stream 8</th> <td>395.771006</td> <td>33.537585</td> <td>54.850876</td> <td>141.526389</td> <td>153.763316</td> <td>12.997092</td> <td>127.961975</td> <td>57.100346</td> </tr> <tr> <th>Min Qi</th> <td>72.259919</td> <td>11.424035</td> <td>48.047940</td> <td>8.559784</td> <td>95.039317</td> <td>8.973519</td> <td>26.710228</td> <td>45.527142</td> </tr> <tr> <th>Max Qi</th> <td>449.871952</td> <td>74.358770</td> <td>85.782399</td> <td>217.190836</td> <td>562.871920</td> <td>32.236754</td> <td>127.961975</td> <td>116.860455</td> </tr> <tr> <th>Avg Qi</th> <td>300.677693</td> <td>31.675658</td> <td>66.929694</td> <td>59.607645</td> <td>226.320670</td> <td>16.666611</td> <td>77.327289</td> <td>63.646711</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>12.230334</td> <td>70.991261</td> <td>33.092797</td> <td>17.517230</td> <td>15.798438</td> <td>19.743562</td> <td>127.494687</td> <td>5.893471</td> </tr> <tr> <th>Stream 1</th> <td>27.550293</td> <td>14.970857</td> <td>16.442806</td> <td>111.138612</td> <td>68.214095</td> <td>7.884782</td> <td>27.109441</td> <td>6.087067</td> </tr> <tr> <th>Stream 2</th> <td>43.277918</td> <td>12.748690</td> <td>22.681844</td> <td>92.835566</td> <td>84.416610</td> <td>14.661934</td> <td>151.094498</td> <td>153.285076</td> </tr> <tr> <th>Stream 3</th> <td>129.696125</td> <td>13.435663</td> <td>14.674499</td> <td>129.179966</td> <td>39.176513</td> <td>6.286296</td> <td>181.596838</td> <td>416.052710</td> </tr> <tr> <th>Stream 4</th> <td>110.348816</td> <td>7.080225</td> <td>21.051910</td> <td>85.758973</td> <td>65.130356</td> <td>7.292999</td> <td>123.386514</td> <td>151.000786</td> </tr> <tr> <th>Stream 5</th> <td>43.365006</td> <td>9.847612</td> <td>32.881770</td> <td>94.752284</td> <td>67.788314</td> <td>9.035439</td> <td>72.539334</td> <td>223.967821</td> </tr> <tr> <th>Stream 6</th> <td>34.534280</td> <td>36.347298</td> <td>27.849276</td> <td>122.736244</td> <td>51.447492</td> <td>25.051058</td> <td>80.452175</td> <td>84.519426</td> </tr> <tr> <th>Stream 7</th> <td>48.021860</td> <td>30.594474</td> <td>22.522426</td> <td>99.245893</td> <td>73.076698</td> <td>7.260729</td> <td>38.585852</td> <td>5.697277</td> </tr> <tr> <th>Stream 8</th> <td>29.484201</td> <td>12.368769</td> <td>40.344043</td> <td>84.137820</td> <td>30.813313</td> <td>4.856991</td> <td>22.196547</td> <td>4.600057</td> </tr> <tr> <th>Min Qi</th> <td>27.550293</td> <td>7.080225</td> <td>14.674499</td> <td>84.137820</td> <td>30.813313</td> <td>4.856991</td> <td>22.196547</td> <td>4.600057</td> </tr> <tr> <th>Max Qi</th> <td>129.696125</td> <td>36.347298</td> <td>40.344043</td> <td>129.179966</td> <td>84.416610</td> <td>25.051058</td> <td>181.596838</td> <td>416.052710</td> </tr> <tr> <th>Avg Qi</th> <td>58.284812</td> <td>17.174198</td> <td>24.806072</td> <td>102.473170</td> <td>60.007924</td> <td>10.291279</td> <td>87.120150</td> <td>130.651277</td> </tr> </table> <h2>3000G Run 2</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>September 29, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>3000</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>8</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td> 1292085.1 </td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td> 1179391.6 </td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@100GB)</th> <td> 1234453.1 </td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td> 1611.779000 seconds</td> </tr> </table> <h3>Duration of stream execution</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>09/29/2014 14:01:15</td> <td>09/29/2014 14:06:48</td> <td>0:05:33</td> </tr> <tr> <th>Stream 1</th> <td>09/29/2014 14:06:53</td> <td>09/29/2014 14:30:22</td> <td>0:23:29</td> </tr> <tr> <th>Stream 2</th> <td>09/29/2014 14:06:53</td> <td>09/29/2014 14:32:30</td> <td>0:25:37</td> </tr> <tr> <th>Stream 3</th> <td>09/29/2014 14:06:53</td> <td>09/29/2014 14:31:23</td> <td>0:24:30</td> </tr> <tr> <th>Stream 4</th> <td>09/29/2014 14:06:53</td> <td>09/29/2014 14:31:34</td> <td>0:24:41</td> </tr> <tr> <th>Stream 5</th> <td>09/29/2014 14:06:53</td> <td>09/29/2014 14:32:53</td> <td>0:26:00</td> </tr> <tr> <th>Stream 6</th> <td>09/29/2014 14:06:53</td> <td>09/29/2014 14:29:51</td> <td>0:22:58</td> </tr> <tr> <th>Stream 7</th> <td>09/29/2014 14:06:53</td> <td>09/29/2014 14:31:34</td> <td>0:24:41</td> </tr> <tr> <th>Stream 8</th> <td>09/29/2014 14:06:53</td> <td>09/29/2014 14:30:35</td> <td>0:23:42</td> </tr> <tr> <th>Refresh 0</th> <td>09/29/2014 14:01:15</td> <td>09/29/2014 14:01:35</td> <td>0:00:20</td> </tr> <tr> <th> </th> <td>09/29/2014 14:06:49</td> <td>09/29/2014 14:06:53</td> <td>0:00:04</td> </tr> <tr> <th>Refresh 1</th> <td>09/29/2014 14:33:16</td> <td>09/29/2014 14:33:45</td> <td>0:00:29</td> </tr> <tr> <th>Refresh 2</th> <td>09/29/2014 14:06:55</td> <td>09/29/2014 14:12:28</td> <td>0:05:33</td> </tr> <tr> <th>Refresh 3</th> <td>09/29/2014 14:12:29</td> <td>09/29/2014 14:21:55</td> <td>0:09:26</td> </tr> <tr> <th>Refresh 4</th> <td>09/29/2014 14:21:55</td> <td>09/29/2014 14:27:40</td> <td>0:05:45</td> </tr> <tr> <th>Refresh 5</th> <td>09/29/2014 14:27:43</td> <td>09/29/2014 14:31:14</td> <td>0:03:31</td> </tr> <tr> <th>Refresh 6</th> <td>09/29/2014 14:31:14</td> <td>09/29/2014 14:31:51</td> <td>0:00:37</td> </tr> <tr> <th>Refresh 7</th> <td>09/29/2014 14:31:51</td> <td>09/29/2014 14:32:52</td> <td>0:01:01</td> </tr> <tr> <th>Refresh 8</th> <td>09/29/2014 14:32:52</td> <td>09/29/2014 14:33:16</td> <td>0:00:24</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> <th>Stream 0</th> <td>9.451169</td> <td>3.644118</td> <td>18.419151</td> <td>1.404395</td> <td>15.740525</td> <td>2.085038</td> <td>15.171847</td> <td>25.400834</td> </tr> <tr> <th>Stream 1</th> <td>19.558041</td> <td>6.607300</td> <td>85.774410</td> <td>4.503525</td> <td>81.448472</td> <td>11.976129</td> <td>92.140470</td> <td>145.743853</td> </tr> <tr> <th>Stream 2</th> <td>31.042019</td> <td>7.877299</td> <td>71.958033</td> <td>8.862111</td> <td>142.452144</td> <td>18.489193</td> <td>81.003310</td> <td>85.856529</td> </tr> <tr> <th>Stream 3</th> <td>38.833612</td> <td>12.440326</td> <td>86.063103</td> <td>7.165120</td> <td>84.707025</td> <td>16.931531</td> <td>100.442710</td> <td>122.411252</td> </tr> <tr> <th>Stream 4</th> <td>15.751913</td> <td>33.026762</td> <td>50.457193</td> <td>7.064220</td> <td>114.130257</td> <td>5.992556</td> <td>66.035959</td> <td>84.596973</td> </tr> <tr> <th>Stream 5</th> <td>18.462884</td> <td>28.047942</td> <td>110.690543</td> <td>16.566547</td> <td>104.403789</td> <td>5.303453</td> <td>72.552640</td> <td>402.383383</td> </tr> <tr> <th>Stream 6</th> <td>17.858339</td> <td>33.988800</td> <td>110.431091</td> <td>7.238431</td> <td>72.229953</td> <td>16.850955</td> <td>68.231546</td> <td>180.601000</td> </tr> <tr> <th>Stream 7</th> <td>23.055572</td> <td>17.044813</td> <td>96.105520</td> <td>8.941132</td> <td>171.130879</td> <td>8.423100</td> <td>70.634541</td> <td>147.261648</td> </tr> <tr> <th>Stream 8</th> <td>19.840798</td> <td>13.860740</td> <td>74.961175</td> <td>16.171566</td> <td>56.165875</td> <td>5.904921</td> <td>47.646217</td> <td>125.991819</td> </tr> <tr> <th>Min Qi</th> <td>15.751913</td> <td>6.607300</td> <td>50.457193</td> <td>4.503525</td> <td>56.165875</td> <td>5.303453</td> <td>47.646217</td> <td>84.596973</td> </tr> <tr> <th>Max Qi</th> <td>38.833612</td> <td>33.988800</td> <td>110.690543</td> <td>16.566547</td> <td>171.130879</td> <td>18.489193</td> <td>100.442710</td> <td>402.383383</td> </tr> <tr> <th>Avg Qi</th> <td>23.050397</td> <td>19.111748</td> <td>85.805134</td> <td>9.564082</td> <td>103.333549</td> <td>11.233980</td> <td>74.835924</td> <td>161.855807</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>54.766945</td> <td>5.551163</td> <td>29.216632</td> <td>3.035008</td> <td>52.816902</td> <td>3.346243</td> <td>15.767022</td> <td>10.066112</td> </tr> <tr> <th>Stream 1</th> <td>130.666380</td> <td>9.658277</td> <td>49.332720</td> <td>103.036705</td> <td>194.520370</td> <td>12.166344</td> <td>65.144599</td> <td>97.158571</td> </tr> <tr> <th>Stream 2</th> <td>254.754936</td> <td>22.605298</td> <td>38.102466</td> <td>21.121168</td> <td>300.467330</td> <td>12.262318</td> <td>108.203491</td> <td>50.696657</td> </tr> <tr> <th>Stream 3</th> <td>283.761567</td> <td>19.327164</td> <td>73.414574</td> <td>7.431651</td> <td>183.121904</td> <td>12.573854</td> <td>73.814766</td> <td>46.802493</td> </tr> <tr> <th>Stream 4</th> <td>290.341947</td> <td>57.452026</td> <td>58.354221</td> <td>13.066162</td> <td>189.263163</td> <td>18.998781</td> <td>121.269774</td> <td>54.831406</td> </tr> <tr> <th>Stream 5</th> <td>81.787025</td> <td>8.410538</td> <td>79.822552</td> <td>16.005077</td> <td>190.730342</td> <td>21.697136</td> <td>100.456487</td> <td>46.744884</td> </tr> <tr> <th>Stream 6</th> <td>202.558515</td> <td>39.360009</td> <td>74.519981</td> <td>15.960756</td> <td>137.321631</td> <td>26.583824</td> <td>57.537668</td> <td>60.758997</td> </tr> <tr> <th>Stream 7</th> <td>226.790801</td> <td>44.175536</td> <td>73.992368</td> <td>7.561897</td> <td>182.853851</td> <td>17.597471</td> <td>31.128055</td> <td>44.389893</td> </tr> <tr> <th>Stream 8</th> <td>275.423934</td> <td>21.980040</td> <td>60.538239</td> <td>39.736622</td> <td>173.574795</td> <td>58.786316</td> <td>95.124912</td> <td>25.564108</td> </tr> <tr> <th>Min Qi</th> <td>81.787025</td> <td>8.410538</td> <td>38.102466</td> <td>7.431651</td> <td>137.321631</td> <td>12.166344</td> <td>31.128055</td> <td>25.564108</td> </tr> <tr> <th>Max Qi</th> <td>290.341947</td> <td>57.452026</td> <td>79.822552</td> <td>103.036705</td> <td>300.467330</td> <td>58.786316</td> <td>121.269774</td> <td>97.158571</td> </tr> <tr> <th>Avg Qi</th> <td>218.260638</td> <td>27.871111</td> <td>63.509640</td> <td>27.990005</td> <td>193.981673</td> <td>22.583255</td> <td>81.584969</td> <td>53.368376</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>13.620157</td> <td>2.288504</td> <td>4.166807</td> <td>16.468447</td> <td>9.991810</td> <td>1.101775</td> <td>20.152227</td> <td>4.294680</td> </tr> <tr> <th>Stream 1</th> <td>44.026143</td> <td>31.720525</td> <td>25.684461</td> <td>134.254716</td> <td>30.797008</td> <td>9.568594</td> <td>24.328205</td> <td>4.319533</td> </tr> <tr> <th>Stream 2</th> <td>40.283148</td> <td>9.970277</td> <td>29.731019</td> <td>133.083785</td> <td>29.322194</td> <td>8.859556</td> <td>73.251098</td> <td>249.850045</td> </tr> <tr> <th>Stream 3</th> <td>44.288244</td> <td>18.914661</td> <td>38.162762</td> <td>144.458624</td> <td>22.556235</td> <td>6.184842</td> <td>117.267234</td> <td>445.700238</td> </tr> <tr> <th>Stream 4</th> <td>67.147744</td> <td>6.649451</td> <td>27.876825</td> <td>59.226248</td> <td>69.373248</td> <td>44.478703</td> <td>61.381724</td> <td>282.608075</td> </tr> <tr> <th>Stream 5</th> <td>36.403227</td> <td>12.226129</td> <td>21.997683</td> <td>95.912670</td> <td>44.219799</td> <td>21.117974</td> <td>106.473817</td> <td>97.896971</td> </tr> <tr> <th>Stream 6</th> <td>42.114038</td> <td>30.805969</td> <td>25.929027</td> <td>51.658733</td> <td>26.475662</td> <td>34.816500</td> <td>31.309953</td> <td>5.608395</td> </tr> <tr> <th>Stream 7</th> <td>48.601889</td> <td>18.708127</td> <td>18.893532</td> <td>132.558026</td> <td>50.476383</td> <td>12.309402</td> <td>22.661371</td> <td>37.610815</td> </tr> <tr> <th>Stream 8</th> <td>34.413417</td> <td>34.709883</td> <td>37.058335</td> <td>121.710608</td> <td>44.676485</td> <td>9.449332</td> <td>19.311945</td> <td>4.420232</td> </tr> <tr> <th>Min Qi</th> <td>34.413417</td> <td>6.649451</td> <td>18.893532</td> <td>51.658733</td> <td>22.556235</td> <td>6.184842</td> <td>19.311945</td> <td>4.319533</td> </tr> <tr> <th>Max Qi</th> <td>67.147744</td> <td>34.709883</td> <td>38.162762</td> <td>144.458624</td> <td>69.373248</td> <td>44.478703</td> <td>117.267234</td> <td>445.700238</td> </tr> <tr> <th>Avg Qi</th> <td>44.659731</td> <td>20.463128</td> <td>28.166705</td> <td>109.107926</td> <td>39.737127</td> <td>18.348113</td> <td>56.998168</td> <td>141.001788</td> </tr> </table> <h2>3000G Run 3</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>September 29, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>3000</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>8</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td> 1178534.1 </td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td> 1092936.2 </td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@100GB)</th> <td> 1134928.4 </td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td> 1739.269000 seconds</td> </tr> </table> <h3>Duration of stream execution</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>09/29/2014 14:33:48</td> <td>09/29/2014 14:40:59</td> <td>0:07:11</td> </tr> <tr> <th>Stream 1</th> <td>09/29/2014 14:41:04</td> <td>09/29/2014 15:10:02</td> <td>0:28:58</td> </tr> <tr> <th>Stream 2</th> <td>09/29/2014 14:41:04</td> <td>09/29/2014 15:09:07</td> <td>0:28:03</td> </tr> <tr> <th>Stream 3</th> <td>09/29/2014 14:41:04</td> <td>09/29/2014 15:09:17</td> <td>0:28:13</td> </tr> <tr> <th>Stream 4</th> <td>09/29/2014 14:41:04</td> <td>09/29/2014 15:09:55</td> <td>0:28:51</td> </tr> <tr> <th>Stream 5</th> <td>09/29/2014 14:41:04</td> <td>09/29/2014 15:09:39</td> <td>0:28:35</td> </tr> <tr> <th>Stream 6</th> <td>09/29/2014 14:41:04</td> <td>09/29/2014 15:09:46</td> <td>0:28:42</td> </tr> <tr> <th>Stream 7</th> <td>09/29/2014 14:41:04</td> <td>09/29/2014 15:09:58</td> <td>0:28:54</td> </tr> <tr> <th>Stream 8</th> <td>09/29/2014 14:41:04</td> <td>09/29/2014 15:08:58</td> <td>0:27:54</td> </tr> <tr> <th>Refresh 0</th> <td>09/29/2014 14:33:48</td> <td>09/29/2014 14:34:07</td> <td>0:00:19</td> </tr> <tr> <th> </th> <td>09/29/2014 14:40:59</td> <td>09/29/2014 14:41:04</td> <td>0:00:05</td> </tr> <tr> <th>Refresh 1</th> <td>09/29/2014 15:06:57</td> <td>09/29/2014 15:09:49</td> <td>0:02:52</td> </tr> <tr> <th>Refresh 2</th> <td>09/29/2014 14:41:05</td> <td>09/29/2014 14:47:39</td> <td>0:06:34</td> </tr> <tr> <th>Refresh 3</th> <td>09/29/2014 14:47:40</td> <td>09/29/2014 14:56:46</td> <td>0:09:06</td> </tr> <tr> <th>Refresh 4</th> <td>09/29/2014 14:56:49</td> <td>09/29/2014 15:03:19</td> <td>0:06:30</td> </tr> <tr> <th>Refresh 5</th> <td>09/29/2014 15:03:24</td> <td>09/29/2014 15:06:45</td> <td>0:03:21</td> </tr> <tr> <th>Refresh 6</th> <td>09/29/2014 15:06:46</td> <td>09/29/2014 15:06:49</td> <td>0:00:03</td> </tr> <tr> <th>Refresh 7</th> <td>09/29/2014 15:06:50</td> <td>09/29/2014 15:06:53</td> <td>0:00:03</td> </tr> <tr> <th>Refresh 8</th> <td>09/29/2014 15:06:53</td> <td>09/29/2014 15:10:04</td> <td>0:03:11</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> <th>Stream 0</th> <td>9.393632</td> <td>5.001910</td> <td>17.053567</td> <td>1.427500</td> <td>17.813839</td> <td>2.230451</td> <td>13.884490</td> <td>25.610995</td> </tr> <tr> <th>Stream 1</th> <td>12.971454</td> <td>9.383520</td> <td>94.257760</td> <td>1.603106</td> <td>127.940946</td> <td>20.791892</td> <td>78.869819</td> <td>138.521273</td> </tr> <tr> <th>Stream 2</th> <td>21.428177</td> <td>31.431513</td> <td>96.366083</td> <td>5.611843</td> <td>58.394596</td> <td>11.279502</td> <td>47.114473</td> <td>407.135077</td> </tr> <tr> <th>Stream 3</th> <td>23.377920</td> <td>37.474814</td> <td>83.640621</td> <td>9.152178</td> <td>71.186158</td> <td>11.001543</td> <td>46.763758</td> <td>110.015662</td> </tr> <tr> <th>Stream 4</th> <td>49.580860</td> <td>31.979940</td> <td>87.662950</td> <td>8.983661</td> <td>68.052295</td> <td>14.367631</td> <td>59.266063</td> <td>301.788652</td> </tr> <tr> <th>Stream 5</th> <td>13.483836</td> <td>20.203772</td> <td>391.980128</td> <td>12.505446</td> <td>77.966993</td> <td>10.487869</td> <td>52.989448</td> <td>226.837637</td> </tr> <tr> <th>Stream 6</th> <td>38.104903</td> <td>21.271630</td> <td>84.689348</td> <td>8.626460</td> <td>86.620802</td> <td>11.981171</td> <td>69.182098</td> <td>111.810485</td> </tr> <tr> <th>Stream 7</th> <td>20.243617</td> <td>12.298692</td> <td>99.547203</td> <td>6.020951</td> <td>151.584400</td> <td>17.528287</td> <td>62.037348</td> <td>101.023802</td> </tr> <tr> <th>Stream 8</th> <td>22.808294</td> <td>17.583072</td> <td>59.180595</td> <td>5.618565</td> <td>123.108771</td> <td>11.477376</td> <td>42.485363</td> <td>92.035709</td> </tr> <tr> <th>Min Qi</th> <td>12.971454</td> <td>9.383520</td> <td>59.180595</td> <td>1.603106</td> <td>58.394596</td> <td>10.487869</td> <td>42.485363</td> <td>92.035709</td> </tr> <tr> <th>Max Qi</th> <td>49.580860</td> <td>37.474814</td> <td>391.980128</td> <td>12.505446</td> <td>151.584400</td> <td>20.791892</td> <td>78.869819</td> <td>407.135077</td> </tr> <tr> <th>Avg Qi</th> <td>25.249883</td> <td>22.703369</td> <td>124.665586</td> <td>7.265276</td> <td>95.606870</td> <td>13.614409</td> <td>57.338546</td> <td>186.146037</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>146.487681</td> <td>6.798942</td> <td>29.834475</td> <td>3.177879</td> <td>55.067866</td> <td>4.503738</td> <td>17.215591</td> <td>9.333281</td> </tr> <tr> <th>Stream 1</th> <td>177.581204</td> <td>44.178095</td> <td>69.746005</td> <td>12.306166</td> <td>215.602727</td> <td>30.443709</td> <td>64.276384</td> <td>45.266949</td> </tr> <tr> <th>Stream 2</th> <td>211.311651</td> <td>27.403143</td> <td>61.412478</td> <td>12.173058</td> <td>216.879170</td> <td>18.272234</td> <td>96.753886</td> <td>35.587072</td> </tr> <tr> <th>Stream 3</th> <td>482.581456</td> <td>68.663026</td> <td>60.354163</td> <td>13.408513</td> <td>187.921639</td> <td>17.469237</td> <td>62.337222</td> <td>31.706120</td> </tr> <tr> <th>Stream 4</th> <td>178.297373</td> <td>23.711312</td> <td>67.129677</td> <td>15.216904</td> <td>328.149575</td> <td>20.258853</td> <td>78.891201</td> <td>84.852368</td> </tr> <tr> <th>Stream 5</th> <td>209.496498</td> <td>28.346366</td> <td>55.584081</td> <td>9.644075</td> <td>131.622351</td> <td>24.171156</td> <td>80.046801</td> <td>43.625932</td> </tr> <tr> <th>Stream 6</th> <td>521.691639</td> <td>24.126176</td> <td>72.964805</td> <td>15.311409</td> <td>146.152570</td> <td>34.748843</td> <td>71.957130</td> <td>58.470644</td> </tr> <tr> <th>Stream 7</th> <td>580.320149</td> <td>17.054563</td> <td>56.172396</td> <td>7.530832</td> <td>200.100326</td> <td>12.444021</td> <td>25.910599</td> <td>75.653693</td> </tr> <tr> <th>Stream 8</th> <td>472.231674</td> <td>15.064398</td> <td>89.875570</td> <td>42.394675</td> <td>166.589234</td> <td>12.831209</td> <td>81.697881</td> <td>73.821769</td> </tr> <tr> <th>Min Qi</th> <td>177.581204</td> <td>15.064398</td> <td>55.584081</td> <td>7.530832</td> <td>131.622351</td> <td>12.444021</td> <td>25.910599</td> <td>31.706120</td> </tr> <tr> <th>Max Qi</th> <td>580.320149</td> <td>68.663026</td> <td>89.875570</td> <td>42.394675</td> <td>328.149575</td> <td>34.748843</td> <td>96.753886</td> <td>84.852368</td> </tr> <tr> <th>Avg Qi</th> <td>354.188955</td> <td>31.068385</td> <td>66.654897</td> <td>15.998204</td> <td>199.127199</td> <td>21.329908</td> <td>70.233888</td> <td>56.123068</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>12.252670</td> <td>2.593733</td> <td>4.115862</td> <td>16.895672</td> <td>10.183350</td> <td>1.240096</td> <td>18.679685</td> <td>4.876067</td> </tr> <tr> <th>Stream 1</th> <td>356.740980</td> <td>21.197870</td> <td>30.422216</td> <td>81.779038</td> <td>65.468650</td> <td>3.947503</td> <td>63.933750</td> <td>107.563796</td> </tr> <tr> <th>Stream 2</th> <td>54.087768</td> <td>10.152604</td> <td>34.940701</td> <td>113.510640</td> <td>70.908809</td> <td>12.316233</td> <td>109.091578</td> <td>283.076004</td> </tr> <tr> <th>Stream 3</th> <td>52.807104</td> <td>18.525982</td> <td>13.740089</td> <td>212.364908</td> <td>16.413964</td> <td>17.998809</td> <td>58.653503</td> <td>483.718271</td> </tr> <tr> <th>Stream 4</th> <td>42.389062</td> <td>36.157809</td> <td>28.909260</td> <td>86.427025</td> <td>21.605419</td> <td>7.608729</td> <td>54.910853</td> <td>331.074114</td> </tr> <tr> <th>Stream 5</th> <td>48.214794</td> <td>15.778893</td> <td>20.681799</td> <td>130.560005</td> <td>43.846752</td> <td>33.905533</td> <td>54.536966</td> <td>139.563667</td> </tr> <tr> <th>Stream 6</th> <td>84.061840</td> <td>26.224851</td> <td>16.546432</td> <td>117.265210</td> <td>34.766856</td> <td>39.037423</td> <td>0.710642</td> <td>1.645351</td> </tr> <tr> <th>Stream 7</th> <td>63.034890</td> <td>15.966686</td> <td>31.666488</td> <td>112.689765</td> <td>28.661943</td> <td>12.828171</td> <td>1.274731</td> <td>1.780452</td> </tr> <tr> <th>Stream 8</th> <td>43.879104</td> <td>8.596666</td> <td>32.585746</td> <td>177.928730</td> <td>26.763334</td> <td>6.112333</td> <td>1.187693</td> <td>0.533668</td> </tr> <tr> <th>Min Qi</th> <td>42.389062</td> <td>8.596666</td> <td>13.740089</td> <td>81.779038</td> <td>16.413964</td> <td>3.947503</td> <td>0.710642</td> <td>0.533668</td> </tr> <tr> <th>Max Qi</th> <td>356.740980</td> <td>36.157809</td> <td>34.940701</td> <td>212.364908</td> <td>70.908809</td> <td>39.037423</td> <td>109.091578</td> <td>483.718271</td> </tr> <tr> <th>Avg Qi</th> <td>93.151943</td> <td>19.075170</td> <td>26.186591</td> <td>129.065665</td> <td>38.554466</td> <td>16.719342</td> <td>43.037465</td> <td>168.619415</td> </tr> </table> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac349bc698"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac347508c8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac347b2098"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac34f71318"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac346fa748"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac349de568"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aac34725b28"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aac36adde48"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aac34a7e368"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac348581b8"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac344340c8"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac34ee7058"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac34d388b8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x69cacf8"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aac00620518"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aabeaafeba8"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aabd202f958"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:09:07.453735-04:00
In Hoc Signo Vinces (part 19 of n): Scalability, 1000G, and 3000G
http://www.openlinksw.com:443/weblog/oerling/?date=2014-09-30#1822
2014-09-30T20:33:29Z
<p>Scalability, specifically linear scalability, means that twice the data takes twice as long to process, or that double the gear processes the same data in half the time. This is only literally true for "embarrassingly parallel" workloads.</p> <p>There are parts of <a href="http://www.tpc.org/tpch/" id="link-id0x2aac693da068">TPC-H</a> which have an embarrassingly parallel nature, like <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac3757a2d8">Q1</a> and <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aac6ad4c0b8">Q7</a>. There are parts that are almost as easy, like Q14, <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac116f1f88">Q17</a>, <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aacaa6ec0b8">Q19</a>, and <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aaca8c66f18">Q21</a>, where there is a big scan and a selective hash join with a hash table small enough to replicate everywhere. The scan scales linearly; building the hash does not, since it is done at single-server speed (once in each process). Some queries like <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac116f2728">Q9</a> and <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aac6a56bd28">Q13</a> end up doing a big cross-partition join which runs into communication overheads.</p> <p>This is our first look at how performance behaves with bigger data and a larger platform. The results shown here are interesting but are not final. I bet I can do better; by how much is what we'll find out soon enough.</p> <p>We will here compare a 1000G setup on my desktop, and a 3000G setup at the <a href="http://www.cwi.nl/" id="link-id0x2aac6ad4c2b8">CWI</a>'s <a href="http://scilens.project.cwi.nl/" id="link-id0x2aac117f8cb8">Scilens cluster</a>. The former is 2 boxes of dual Xeon E5 2630, and the latter is 8 boxes of dual Xeon E5 2650v2. All things run from memory and both have QDR IB interconnect. Counting cores and clock, the CWI cluster is 6x larger.</p> <p>As a rough approximation, for the worst queries, 6x the gear runs 3x the data in the same amount of real time. The 1000G setup has near full platform utilization and the 3000G setup has about half platform utilization. In both cases, running two instances of the same query at the same time takes twice as long.</p> <p>We use <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aaca8c272f8">Q9</a> for this study. The plan makes a hash table of <code>part</code> with 1/14 of all <code>parts</code>, replicating to all processes. Then there is a hash table of <code>partsupp</code> with a key of <code>ps_partkey, ps_suppkey</code>, and a dependent of <code>ps_supplycost</code>. This is much larger than the <code>part</code> hash table and is therefore partitioned on <code>ps_partkey</code>. The build is for 1/14th of <code>partsupp</code>. Then there is a scan of <code>lineitem</code> filtered by the <code>part</code> hash table; then a cross-partition join to the <code>partsupp</code> hash table; then a cross partition join to <code>orders</code>, this time by index; then a hash join on a replicated hash table of <code>supplier</code>; then <code>nation</code>; then aggregation. The aggregation is done in each slice; then the slices are added up at the end.</p> <p>The plan could be made better by one fewer partition crossing. Now there is a crossing from <code>l_orderkey</code> to <code>l_partkey</code> and back to <code>o_orderkey</code>. This would not be so if the cost model knew that the <code>partsupp</code> always hits. The cost model thinks it hits 1/14 of the time, because it does not know that the selection on the build is exactly the same as on the probe.</p> <p>For the present purposes, the extra crossing just serves to make the matter of interest more visible.</p> <p>So, for the 1000G setup, we have 43.6 seconds (s) and </p> <blockquote> <pre><code>Cluster 4 nodes, 44 s. 459 m/s 119788 KB/s 3120% cpu 0% read 19% clw threads 1r 0w 0i buffers 17622126 68 d 0 w 0 pfs</code> </pre></blockquote> <p>For the 3000G setup, we have 49.9 s and </p> <blockquote> <pre><code>Cluster 16 nodes, 50 s. 49389 m/s 1801815 KB/s 7283% cpu 0% read 18% clw threads 1r 0w 0i buffers 135122893 15895255 d 0 w 17 pfs</code> </pre></blockquote> <p>The platform utilization on the small system is better, at 31/48 (running/total threads); the large one has 73/256.</p> <p>The large case is clearly network bound. If this were for CPU only, it should be done in half the time it takes the small system to do 1000G. </p> <p>We confirm this by looking at write wait: 3940 seconds of thread time blocked on write over 50s of real time. The figures on the small one are 3.9s of thread time blocked for 39s of real time. The data transfer on the large one is 93 GB. </p> <p>How to block less? One idea would be to write less. So we try compression; there is a Google snappy-based message compression option in Virtuoso. </p> <p>We now get 39.6 s and</p> <blockquote> <pre><code>Cluster 16 nodes, 40 s. 65161 m/s 1239922 KB/s 10201% cpu 0% read 21% clw threads 1r 0w 0i buffers 52828440 172 d 0 w 0 pfs</code> </pre></blockquote> <p>The write block time is 397 s of thread time over 39 s of real time, 10x better. The data transfer is 50.9 GB after compression. Snappy is somewhat effective for compression and very fast; in CPU profile, it is under 3% of Q9 on the small system. Gains on the small system are less, though, since blocking is not a big issue to start with.</p> <p>This is still not full platform. But if the data transfer is further cut in half by a better plan, the situation will be quite good. Now we have 102/256 threads running, meaning that there could be another 40-50% of throughput to be added. The last 128 threads are second threads of a core, so count for roughly 30% of a real core.</p> <p>The main cluster-specific operation is a send from one to many. This is now done by formulating the message to each recipient in a chain of string buffers; then, after all the messages are prepared, these are optionally compressed and sent to their recipient. This is needlessly simple: Compressing can proceed if ever there is a would-block situation on writing. If all the compression is done, then a blocked write should switch to another recipient, and only after all recipients have a would-block situation, then the thread can call-select with all descriptors and block on them collectively. There is a piece of code to this effect, but is not now being used. It has been seen to add no value in small cases, but could be useful here. </p> <p>The IB fabric has been seen to do 1.8 GB/s bidirectionally on multiple independent point-to-point TCP links. This is about half the nominal 4 GB/s (40 Gbit/s with 10/8 encoding). So the aggregate throughputs that we see here are nowhere near the nominal spec of the network. Lower level interfaces and the occasional busy wait on the reading end could be tried to some advantage. We have not tried 10GbE either; but if that works at nominal speed, then 10GbE should also be good enough. We will try this at Amazon in due time.</p> <p>In the meantime, there is a 3000G test made at the CWI cluster without message compression. The score is about 4x that of the single server at 300G using the same hardware. The run is with approximately half platform utilization. There are three runs of power plus throughput, the first run being cold.</p> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th>Run</th> <th>Power</th> <th>Throughput</th> <th>Composite </th> </tr> <tr> <th>Run 1</th> <td style="text-align:right">305,881.5</td> <td style="text-align:right">1,072,411.9</td> <td style="text-align:right">572,739.8</td> </tr> <tr> <th>Run 2</th> <td style="text-align:right">1,292,085.1</td> <td style="text-align:right">1,179,391.6</td> <td style="text-align:right">1,234,453.1</td> </tr> <tr> <th>Run 3</th> <td style="text-align:right">1,178,534.1</td> <td style="text-align:right">1,092,936.2</td> <td style="text-align:right">1,134,928.4</td> </tr> </table> <p>The numerical quantities summaries follow. One problem of the run is a high peak of query memory consumption leading to slowdown. Some parts should probably be done in multiple passes to keep the peak lower and not run into swapping. The details will have to be sorted out. This is a demonstration of capability; the perfected accomplishment is to follow.</p> <h2>3000G Run 1</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>September 29, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>3000</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>8</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td>305,881.5</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td>1,072,411.9</td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@100GB)</th> <td>572,739.8</td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td>1,772.554000 seconds</td> </tr> </table> <h3>Duration of stream execution</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>09/29/2014 12:54:52</td> <td>09/29/2014 13:31:17</td> <td>0:36:25</td> </tr> <tr> <th>Stream 1</th> <td>09/29/2014 13:31:24</td> <td>09/29/2014 13:59:24</td> <td>0:28:00</td> </tr> <tr> <th>Stream 2</th> <td>09/29/2014 13:31:24</td> <td>09/29/2014 13:58:59</td> <td>0:27:35</td> </tr> <tr> <th>Stream 3</th> <td>09/29/2014 13:31:24</td> <td>09/29/2014 13:58:29</td> <td>0:27:05</td> </tr> <tr> <th>Stream 4</th> <td>09/29/2014 13:31:24</td> <td>09/29/2014 13:58:52</td> <td>0:27:28</td> </tr> <tr> <th>Stream 5</th> <td>09/29/2014 13:31:24</td> <td>09/29/2014 14:00:06</td> <td>0:28:42</td> </tr> <tr> <th>Stream 6</th> <td>09/29/2014 13:31:24</td> <td>09/29/2014 13:58:18</td> <td>0:26:54</td> </tr> <tr> <th>Stream 7</th> <td>09/29/2014 13:31:24</td> <td>09/29/2014 13:59:25</td> <td>0:28:01</td> </tr> <tr> <th>Stream 8</th> <td>09/29/2014 13:31:24</td> <td>09/29/2014 13:58:50</td> <td>0:27:26</td> </tr> <tr> <th>Refresh 0</th> <td>09/29/2014 12:54:52</td> <td>09/29/2014 12:56:59</td> <td>0:02:07</td> </tr> <tr> <th> </th> <td>09/29/2014 13:31:17</td> <td>09/29/2014 13:31:23</td> <td>0:00:06</td> </tr> <tr> <th>Refresh 1</th> <td>09/29/2014 14:00:38</td> <td>09/29/2014 14:01:11</td> <td>0:00:33</td> </tr> <tr> <th>Refresh 2</th> <td>09/29/2014 13:31:25</td> <td>09/29/2014 13:36:57</td> <td>0:05:32</td> </tr> <tr> <th>Refresh 3</th> <td>09/29/2014 13:36:56</td> <td>09/29/2014 13:47:02</td> <td>0:10:06</td> </tr> <tr> <th>Refresh 4</th> <td>09/29/2014 13:47:03</td> <td>09/29/2014 13:51:40</td> <td>0:04:37</td> </tr> <tr> <th>Refresh 5</th> <td>09/29/2014 13:51:42</td> <td>09/29/2014 13:56:40</td> <td>0:04:58</td> </tr> <tr> <th>Refresh 6</th> <td>09/29/2014 13:56:40</td> <td>09/29/2014 13:59:25</td> <td>0:02:45</td> </tr> <tr> <th>Refresh 7</th> <td>09/29/2014 13:59:25</td> <td>09/29/2014 14:00:10</td> <td>0:00:45</td> </tr> <tr> <th>Refresh 8</th> <td>09/29/2014 14:00:11</td> <td>09/29/2014 14:00:37</td> <td>0:00:26</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> <th>Stream 0</th> <td>601.576975</td> <td>90.803782</td> <td>108.725110</td> <td>177.112667</td> <td>171.995572</td> <td>2.098138</td> <td>15.768311</td> <td>152.511444</td> </tr> <tr> <th>Stream 1</th> <td>13.310341</td> <td>32.722946</td> <td>125.551415</td> <td>1.912836</td> <td>46.041675</td> <td>13.294214</td> <td>85.345068</td> <td>165.424288</td> </tr> <tr> <th>Stream 2</th> <td>19.425885</td> <td>9.248670</td> <td>150.855556</td> <td>7.085737</td> <td>88.445566</td> <td>10.490432</td> <td>49.318554</td> <td>322.500839</td> </tr> <tr> <th>Stream 3</th> <td>30.534391</td> <td>14.273478</td> <td>100.987791</td> <td>59.341763</td> <td>46.442443</td> <td>9.613795</td> <td>64.186196</td> <td>146.324186</td> </tr> <tr> <th>Stream 4</th> <td>28.211213</td> <td>37.134522</td> <td>64.189335</td> <td>10.931513</td> <td>100.610673</td> <td>9.929866</td> <td>112.270530</td> <td>108.489951</td> </tr> <tr> <th>Stream 5</th> <td>29.226411</td> <td>18.132589</td> <td>95.245160</td> <td>63.100068</td> <td>115.663908</td> <td>6.151231</td> <td>46.251309</td> <td>127.742471</td> </tr> <tr> <th>Stream 6</th> <td>30.750930</td> <td>20.888658</td> <td>108.894177</td> <td>55.168565</td> <td>82.016828</td> <td>69.451493</td> <td>65.161517</td> <td>103.697733</td> </tr> <tr> <th>Stream 7</th> <td>13.462570</td> <td>18.033847</td> <td>32.065492</td> <td>78.910373</td> <td>202.998301</td> <td>10.688279</td> <td>47.167022</td> <td>139.601948</td> </tr> <tr> <th>Stream 8</th> <td>24.354314</td> <td>16.711503</td> <td>112.008551</td> <td>8.307098</td> <td>126.849630</td> <td>7.127605</td> <td>51.083118</td> <td>98.648077</td> </tr> <tr> <th>Min Qi</th> <td>13.310341</td> <td>9.248670</td> <td>32.065492</td> <td>1.912836</td> <td>46.041675</td> <td>6.151231</td> <td>46.251309</td> <td>98.648077</td> </tr> <tr> <th>Max Qi</th> <td>30.750930</td> <td>37.134522</td> <td>150.855556</td> <td>78.910373</td> <td>202.998301</td> <td>69.451493</td> <td>112.270530</td> <td>322.500839</td> </tr> <tr> <th>Avg Qi</th> <td>23.659507</td> <td>20.893277</td> <td>98.724685</td> <td>35.594744</td> <td>101.133628</td> <td>17.093364</td> <td>65.097914</td> <td>151.553687</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>92.991259</td> <td>5.175922</td> <td>42.238393</td> <td>29.239879</td> <td>367.805534</td> <td>3.604910</td> <td>15.557396</td> <td>11.650267</td> </tr> <tr> <th>Stream 1</th> <td>149.502128</td> <td>30.197806</td> <td>50.786184</td> <td>217.190836</td> <td>283.545905</td> <td>11.653171</td> <td>73.321150</td> <td>116.860455</td> </tr> <tr> <th>Stream 2</th> <td>245.783668</td> <td>22.278841</td> <td>50.578731</td> <td>36.301810</td> <td>181.405269</td> <td>32.236754</td> <td>57.631764</td> <td>61.540533</td> </tr> <tr> <th>Stream 3</th> <td>377.782738</td> <td>24.129319</td> <td>84.097657</td> <td>10.959661</td> <td>171.698669</td> <td>8.973519</td> <td>54.532180</td> <td>45.527142</td> </tr> <tr> <th>Stream 4</th> <td>341.148908</td> <td>74.358770</td> <td>85.782399</td> <td>43.116347</td> <td>151.146233</td> <td>22.870727</td> <td>74.439693</td> <td>51.871535</td> </tr> <tr> <th>Stream 5</th> <td>72.259919</td> <td>11.424035</td> <td>79.310504</td> <td>9.833135</td> <td>562.871920</td> <td>14.961209</td> <td>127.861874</td> <td>55.377721</td> </tr> <tr> <th>Stream 6</th> <td>373.301225</td> <td>41.379753</td> <td>81.983260</td> <td>9.373200</td> <td>95.039317</td> <td>19.071346</td> <td>76.159452</td> <td>48.324504</td> </tr> <tr> <th>Stream 7</th> <td>449.871952</td> <td>16.099152</td> <td>48.047940</td> <td>8.559784</td> <td>211.094730</td> <td>10.569071</td> <td>26.710228</td> <td>72.571454</td> </tr> <tr> <th>Stream 8</th> <td>395.771006</td> <td>33.537585</td> <td>54.850876</td> <td>141.526389</td> <td>153.763316</td> <td>12.997092</td> <td>127.961975</td> <td>57.100346</td> </tr> <tr> <th>Min Qi</th> <td>72.259919</td> <td>11.424035</td> <td>48.047940</td> <td>8.559784</td> <td>95.039317</td> <td>8.973519</td> <td>26.710228</td> <td>45.527142</td> </tr> <tr> <th>Max Qi</th> <td>449.871952</td> <td>74.358770</td> <td>85.782399</td> <td>217.190836</td> <td>562.871920</td> <td>32.236754</td> <td>127.961975</td> <td>116.860455</td> </tr> <tr> <th>Avg Qi</th> <td>300.677693</td> <td>31.675658</td> <td>66.929694</td> <td>59.607645</td> <td>226.320670</td> <td>16.666611</td> <td>77.327289</td> <td>63.646711</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>12.230334</td> <td>70.991261</td> <td>33.092797</td> <td>17.517230</td> <td>15.798438</td> <td>19.743562</td> <td>127.494687</td> <td>5.893471</td> </tr> <tr> <th>Stream 1</th> <td>27.550293</td> <td>14.970857</td> <td>16.442806</td> <td>111.138612</td> <td>68.214095</td> <td>7.884782</td> <td>27.109441</td> <td>6.087067</td> </tr> <tr> <th>Stream 2</th> <td>43.277918</td> <td>12.748690</td> <td>22.681844</td> <td>92.835566</td> <td>84.416610</td> <td>14.661934</td> <td>151.094498</td> <td>153.285076</td> </tr> <tr> <th>Stream 3</th> <td>129.696125</td> <td>13.435663</td> <td>14.674499</td> <td>129.179966</td> <td>39.176513</td> <td>6.286296</td> <td>181.596838</td> <td>416.052710</td> </tr> <tr> <th>Stream 4</th> <td>110.348816</td> <td>7.080225</td> <td>21.051910</td> <td>85.758973</td> <td>65.130356</td> <td>7.292999</td> <td>123.386514</td> <td>151.000786</td> </tr> <tr> <th>Stream 5</th> <td>43.365006</td> <td>9.847612</td> <td>32.881770</td> <td>94.752284</td> <td>67.788314</td> <td>9.035439</td> <td>72.539334</td> <td>223.967821</td> </tr> <tr> <th>Stream 6</th> <td>34.534280</td> <td>36.347298</td> <td>27.849276</td> <td>122.736244</td> <td>51.447492</td> <td>25.051058</td> <td>80.452175</td> <td>84.519426</td> </tr> <tr> <th>Stream 7</th> <td>48.021860</td> <td>30.594474</td> <td>22.522426</td> <td>99.245893</td> <td>73.076698</td> <td>7.260729</td> <td>38.585852</td> <td>5.697277</td> </tr> <tr> <th>Stream 8</th> <td>29.484201</td> <td>12.368769</td> <td>40.344043</td> <td>84.137820</td> <td>30.813313</td> <td>4.856991</td> <td>22.196547</td> <td>4.600057</td> </tr> <tr> <th>Min Qi</th> <td>27.550293</td> <td>7.080225</td> <td>14.674499</td> <td>84.137820</td> <td>30.813313</td> <td>4.856991</td> <td>22.196547</td> <td>4.600057</td> </tr> <tr> <th>Max Qi</th> <td>129.696125</td> <td>36.347298</td> <td>40.344043</td> <td>129.179966</td> <td>84.416610</td> <td>25.051058</td> <td>181.596838</td> <td>416.052710</td> </tr> <tr> <th>Avg Qi</th> <td>58.284812</td> <td>17.174198</td> <td>24.806072</td> <td>102.473170</td> <td>60.007924</td> <td>10.291279</td> <td>87.120150</td> <td>130.651277</td> </tr> </table> <h2>3000G Run 2</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>September 29, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>3000</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>8</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td> 1292085.1 </td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td> 1179391.6 </td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@100GB)</th> <td> 1234453.1 </td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td> 1611.779000 seconds</td> </tr> </table> <h3>Duration of stream execution</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>09/29/2014 14:01:15</td> <td>09/29/2014 14:06:48</td> <td>0:05:33</td> </tr> <tr> <th>Stream 1</th> <td>09/29/2014 14:06:53</td> <td>09/29/2014 14:30:22</td> <td>0:23:29</td> </tr> <tr> <th>Stream 2</th> <td>09/29/2014 14:06:53</td> <td>09/29/2014 14:32:30</td> <td>0:25:37</td> </tr> <tr> <th>Stream 3</th> <td>09/29/2014 14:06:53</td> <td>09/29/2014 14:31:23</td> <td>0:24:30</td> </tr> <tr> <th>Stream 4</th> <td>09/29/2014 14:06:53</td> <td>09/29/2014 14:31:34</td> <td>0:24:41</td> </tr> <tr> <th>Stream 5</th> <td>09/29/2014 14:06:53</td> <td>09/29/2014 14:32:53</td> <td>0:26:00</td> </tr> <tr> <th>Stream 6</th> <td>09/29/2014 14:06:53</td> <td>09/29/2014 14:29:51</td> <td>0:22:58</td> </tr> <tr> <th>Stream 7</th> <td>09/29/2014 14:06:53</td> <td>09/29/2014 14:31:34</td> <td>0:24:41</td> </tr> <tr> <th>Stream 8</th> <td>09/29/2014 14:06:53</td> <td>09/29/2014 14:30:35</td> <td>0:23:42</td> </tr> <tr> <th>Refresh 0</th> <td>09/29/2014 14:01:15</td> <td>09/29/2014 14:01:35</td> <td>0:00:20</td> </tr> <tr> <th> </th> <td>09/29/2014 14:06:49</td> <td>09/29/2014 14:06:53</td> <td>0:00:04</td> </tr> <tr> <th>Refresh 1</th> <td>09/29/2014 14:33:16</td> <td>09/29/2014 14:33:45</td> <td>0:00:29</td> </tr> <tr> <th>Refresh 2</th> <td>09/29/2014 14:06:55</td> <td>09/29/2014 14:12:28</td> <td>0:05:33</td> </tr> <tr> <th>Refresh 3</th> <td>09/29/2014 14:12:29</td> <td>09/29/2014 14:21:55</td> <td>0:09:26</td> </tr> <tr> <th>Refresh 4</th> <td>09/29/2014 14:21:55</td> <td>09/29/2014 14:27:40</td> <td>0:05:45</td> </tr> <tr> <th>Refresh 5</th> <td>09/29/2014 14:27:43</td> <td>09/29/2014 14:31:14</td> <td>0:03:31</td> </tr> <tr> <th>Refresh 6</th> <td>09/29/2014 14:31:14</td> <td>09/29/2014 14:31:51</td> <td>0:00:37</td> </tr> <tr> <th>Refresh 7</th> <td>09/29/2014 14:31:51</td> <td>09/29/2014 14:32:52</td> <td>0:01:01</td> </tr> <tr> <th>Refresh 8</th> <td>09/29/2014 14:32:52</td> <td>09/29/2014 14:33:16</td> <td>0:00:24</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> <th>Stream 0</th> <td>9.451169</td> <td>3.644118</td> <td>18.419151</td> <td>1.404395</td> <td>15.740525</td> <td>2.085038</td> <td>15.171847</td> <td>25.400834</td> </tr> <tr> <th>Stream 1</th> <td>19.558041</td> <td>6.607300</td> <td>85.774410</td> <td>4.503525</td> <td>81.448472</td> <td>11.976129</td> <td>92.140470</td> <td>145.743853</td> </tr> <tr> <th>Stream 2</th> <td>31.042019</td> <td>7.877299</td> <td>71.958033</td> <td>8.862111</td> <td>142.452144</td> <td>18.489193</td> <td>81.003310</td> <td>85.856529</td> </tr> <tr> <th>Stream 3</th> <td>38.833612</td> <td>12.440326</td> <td>86.063103</td> <td>7.165120</td> <td>84.707025</td> <td>16.931531</td> <td>100.442710</td> <td>122.411252</td> </tr> <tr> <th>Stream 4</th> <td>15.751913</td> <td>33.026762</td> <td>50.457193</td> <td>7.064220</td> <td>114.130257</td> <td>5.992556</td> <td>66.035959</td> <td>84.596973</td> </tr> <tr> <th>Stream 5</th> <td>18.462884</td> <td>28.047942</td> <td>110.690543</td> <td>16.566547</td> <td>104.403789</td> <td>5.303453</td> <td>72.552640</td> <td>402.383383</td> </tr> <tr> <th>Stream 6</th> <td>17.858339</td> <td>33.988800</td> <td>110.431091</td> <td>7.238431</td> <td>72.229953</td> <td>16.850955</td> <td>68.231546</td> <td>180.601000</td> </tr> <tr> <th>Stream 7</th> <td>23.055572</td> <td>17.044813</td> <td>96.105520</td> <td>8.941132</td> <td>171.130879</td> <td>8.423100</td> <td>70.634541</td> <td>147.261648</td> </tr> <tr> <th>Stream 8</th> <td>19.840798</td> <td>13.860740</td> <td>74.961175</td> <td>16.171566</td> <td>56.165875</td> <td>5.904921</td> <td>47.646217</td> <td>125.991819</td> </tr> <tr> <th>Min Qi</th> <td>15.751913</td> <td>6.607300</td> <td>50.457193</td> <td>4.503525</td> <td>56.165875</td> <td>5.303453</td> <td>47.646217</td> <td>84.596973</td> </tr> <tr> <th>Max Qi</th> <td>38.833612</td> <td>33.988800</td> <td>110.690543</td> <td>16.566547</td> <td>171.130879</td> <td>18.489193</td> <td>100.442710</td> <td>402.383383</td> </tr> <tr> <th>Avg Qi</th> <td>23.050397</td> <td>19.111748</td> <td>85.805134</td> <td>9.564082</td> <td>103.333549</td> <td>11.233980</td> <td>74.835924</td> <td>161.855807</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>54.766945</td> <td>5.551163</td> <td>29.216632</td> <td>3.035008</td> <td>52.816902</td> <td>3.346243</td> <td>15.767022</td> <td>10.066112</td> </tr> <tr> <th>Stream 1</th> <td>130.666380</td> <td>9.658277</td> <td>49.332720</td> <td>103.036705</td> <td>194.520370</td> <td>12.166344</td> <td>65.144599</td> <td>97.158571</td> </tr> <tr> <th>Stream 2</th> <td>254.754936</td> <td>22.605298</td> <td>38.102466</td> <td>21.121168</td> <td>300.467330</td> <td>12.262318</td> <td>108.203491</td> <td>50.696657</td> </tr> <tr> <th>Stream 3</th> <td>283.761567</td> <td>19.327164</td> <td>73.414574</td> <td>7.431651</td> <td>183.121904</td> <td>12.573854</td> <td>73.814766</td> <td>46.802493</td> </tr> <tr> <th>Stream 4</th> <td>290.341947</td> <td>57.452026</td> <td>58.354221</td> <td>13.066162</td> <td>189.263163</td> <td>18.998781</td> <td>121.269774</td> <td>54.831406</td> </tr> <tr> <th>Stream 5</th> <td>81.787025</td> <td>8.410538</td> <td>79.822552</td> <td>16.005077</td> <td>190.730342</td> <td>21.697136</td> <td>100.456487</td> <td>46.744884</td> </tr> <tr> <th>Stream 6</th> <td>202.558515</td> <td>39.360009</td> <td>74.519981</td> <td>15.960756</td> <td>137.321631</td> <td>26.583824</td> <td>57.537668</td> <td>60.758997</td> </tr> <tr> <th>Stream 7</th> <td>226.790801</td> <td>44.175536</td> <td>73.992368</td> <td>7.561897</td> <td>182.853851</td> <td>17.597471</td> <td>31.128055</td> <td>44.389893</td> </tr> <tr> <th>Stream 8</th> <td>275.423934</td> <td>21.980040</td> <td>60.538239</td> <td>39.736622</td> <td>173.574795</td> <td>58.786316</td> <td>95.124912</td> <td>25.564108</td> </tr> <tr> <th>Min Qi</th> <td>81.787025</td> <td>8.410538</td> <td>38.102466</td> <td>7.431651</td> <td>137.321631</td> <td>12.166344</td> <td>31.128055</td> <td>25.564108</td> </tr> <tr> <th>Max Qi</th> <td>290.341947</td> <td>57.452026</td> <td>79.822552</td> <td>103.036705</td> <td>300.467330</td> <td>58.786316</td> <td>121.269774</td> <td>97.158571</td> </tr> <tr> <th>Avg Qi</th> <td>218.260638</td> <td>27.871111</td> <td>63.509640</td> <td>27.990005</td> <td>193.981673</td> <td>22.583255</td> <td>81.584969</td> <td>53.368376</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>13.620157</td> <td>2.288504</td> <td>4.166807</td> <td>16.468447</td> <td>9.991810</td> <td>1.101775</td> <td>20.152227</td> <td>4.294680</td> </tr> <tr> <th>Stream 1</th> <td>44.026143</td> <td>31.720525</td> <td>25.684461</td> <td>134.254716</td> <td>30.797008</td> <td>9.568594</td> <td>24.328205</td> <td>4.319533</td> </tr> <tr> <th>Stream 2</th> <td>40.283148</td> <td>9.970277</td> <td>29.731019</td> <td>133.083785</td> <td>29.322194</td> <td>8.859556</td> <td>73.251098</td> <td>249.850045</td> </tr> <tr> <th>Stream 3</th> <td>44.288244</td> <td>18.914661</td> <td>38.162762</td> <td>144.458624</td> <td>22.556235</td> <td>6.184842</td> <td>117.267234</td> <td>445.700238</td> </tr> <tr> <th>Stream 4</th> <td>67.147744</td> <td>6.649451</td> <td>27.876825</td> <td>59.226248</td> <td>69.373248</td> <td>44.478703</td> <td>61.381724</td> <td>282.608075</td> </tr> <tr> <th>Stream 5</th> <td>36.403227</td> <td>12.226129</td> <td>21.997683</td> <td>95.912670</td> <td>44.219799</td> <td>21.117974</td> <td>106.473817</td> <td>97.896971</td> </tr> <tr> <th>Stream 6</th> <td>42.114038</td> <td>30.805969</td> <td>25.929027</td> <td>51.658733</td> <td>26.475662</td> <td>34.816500</td> <td>31.309953</td> <td>5.608395</td> </tr> <tr> <th>Stream 7</th> <td>48.601889</td> <td>18.708127</td> <td>18.893532</td> <td>132.558026</td> <td>50.476383</td> <td>12.309402</td> <td>22.661371</td> <td>37.610815</td> </tr> <tr> <th>Stream 8</th> <td>34.413417</td> <td>34.709883</td> <td>37.058335</td> <td>121.710608</td> <td>44.676485</td> <td>9.449332</td> <td>19.311945</td> <td>4.420232</td> </tr> <tr> <th>Min Qi</th> <td>34.413417</td> <td>6.649451</td> <td>18.893532</td> <td>51.658733</td> <td>22.556235</td> <td>6.184842</td> <td>19.311945</td> <td>4.319533</td> </tr> <tr> <th>Max Qi</th> <td>67.147744</td> <td>34.709883</td> <td>38.162762</td> <td>144.458624</td> <td>69.373248</td> <td>44.478703</td> <td>117.267234</td> <td>445.700238</td> </tr> <tr> <th>Avg Qi</th> <td>44.659731</td> <td>20.463128</td> <td>28.166705</td> <td>109.107926</td> <td>39.737127</td> <td>18.348113</td> <td>56.998168</td> <td>141.001788</td> </tr> </table> <h2>3000G Run 3</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>September 29, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>3000</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>8</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td> 1178534.1 </td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td> 1092936.2 </td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@100GB)</th> <td> 1134928.4 </td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td> 1739.269000 seconds</td> </tr> </table> <h3>Duration of stream execution</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>09/29/2014 14:33:48</td> <td>09/29/2014 14:40:59</td> <td>0:07:11</td> </tr> <tr> <th>Stream 1</th> <td>09/29/2014 14:41:04</td> <td>09/29/2014 15:10:02</td> <td>0:28:58</td> </tr> <tr> <th>Stream 2</th> <td>09/29/2014 14:41:04</td> <td>09/29/2014 15:09:07</td> <td>0:28:03</td> </tr> <tr> <th>Stream 3</th> <td>09/29/2014 14:41:04</td> <td>09/29/2014 15:09:17</td> <td>0:28:13</td> </tr> <tr> <th>Stream 4</th> <td>09/29/2014 14:41:04</td> <td>09/29/2014 15:09:55</td> <td>0:28:51</td> </tr> <tr> <th>Stream 5</th> <td>09/29/2014 14:41:04</td> <td>09/29/2014 15:09:39</td> <td>0:28:35</td> </tr> <tr> <th>Stream 6</th> <td>09/29/2014 14:41:04</td> <td>09/29/2014 15:09:46</td> <td>0:28:42</td> </tr> <tr> <th>Stream 7</th> <td>09/29/2014 14:41:04</td> <td>09/29/2014 15:09:58</td> <td>0:28:54</td> </tr> <tr> <th>Stream 8</th> <td>09/29/2014 14:41:04</td> <td>09/29/2014 15:08:58</td> <td>0:27:54</td> </tr> <tr> <th>Refresh 0</th> <td>09/29/2014 14:33:48</td> <td>09/29/2014 14:34:07</td> <td>0:00:19</td> </tr> <tr> <th> </th> <td>09/29/2014 14:40:59</td> <td>09/29/2014 14:41:04</td> <td>0:00:05</td> </tr> <tr> <th>Refresh 1</th> <td>09/29/2014 15:06:57</td> <td>09/29/2014 15:09:49</td> <td>0:02:52</td> </tr> <tr> <th>Refresh 2</th> <td>09/29/2014 14:41:05</td> <td>09/29/2014 14:47:39</td> <td>0:06:34</td> </tr> <tr> <th>Refresh 3</th> <td>09/29/2014 14:47:40</td> <td>09/29/2014 14:56:46</td> <td>0:09:06</td> </tr> <tr> <th>Refresh 4</th> <td>09/29/2014 14:56:49</td> <td>09/29/2014 15:03:19</td> <td>0:06:30</td> </tr> <tr> <th>Refresh 5</th> <td>09/29/2014 15:03:24</td> <td>09/29/2014 15:06:45</td> <td>0:03:21</td> </tr> <tr> <th>Refresh 6</th> <td>09/29/2014 15:06:46</td> <td>09/29/2014 15:06:49</td> <td>0:00:03</td> </tr> <tr> <th>Refresh 7</th> <td>09/29/2014 15:06:50</td> <td>09/29/2014 15:06:53</td> <td>0:00:03</td> </tr> <tr> <th>Refresh 8</th> <td>09/29/2014 15:06:53</td> <td>09/29/2014 15:10:04</td> <td>0:03:11</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> <th>Stream 0</th> <td>9.393632</td> <td>5.001910</td> <td>17.053567</td> <td>1.427500</td> <td>17.813839</td> <td>2.230451</td> <td>13.884490</td> <td>25.610995</td> </tr> <tr> <th>Stream 1</th> <td>12.971454</td> <td>9.383520</td> <td>94.257760</td> <td>1.603106</td> <td>127.940946</td> <td>20.791892</td> <td>78.869819</td> <td>138.521273</td> </tr> <tr> <th>Stream 2</th> <td>21.428177</td> <td>31.431513</td> <td>96.366083</td> <td>5.611843</td> <td>58.394596</td> <td>11.279502</td> <td>47.114473</td> <td>407.135077</td> </tr> <tr> <th>Stream 3</th> <td>23.377920</td> <td>37.474814</td> <td>83.640621</td> <td>9.152178</td> <td>71.186158</td> <td>11.001543</td> <td>46.763758</td> <td>110.015662</td> </tr> <tr> <th>Stream 4</th> <td>49.580860</td> <td>31.979940</td> <td>87.662950</td> <td>8.983661</td> <td>68.052295</td> <td>14.367631</td> <td>59.266063</td> <td>301.788652</td> </tr> <tr> <th>Stream 5</th> <td>13.483836</td> <td>20.203772</td> <td>391.980128</td> <td>12.505446</td> <td>77.966993</td> <td>10.487869</td> <td>52.989448</td> <td>226.837637</td> </tr> <tr> <th>Stream 6</th> <td>38.104903</td> <td>21.271630</td> <td>84.689348</td> <td>8.626460</td> <td>86.620802</td> <td>11.981171</td> <td>69.182098</td> <td>111.810485</td> </tr> <tr> <th>Stream 7</th> <td>20.243617</td> <td>12.298692</td> <td>99.547203</td> <td>6.020951</td> <td>151.584400</td> <td>17.528287</td> <td>62.037348</td> <td>101.023802</td> </tr> <tr> <th>Stream 8</th> <td>22.808294</td> <td>17.583072</td> <td>59.180595</td> <td>5.618565</td> <td>123.108771</td> <td>11.477376</td> <td>42.485363</td> <td>92.035709</td> </tr> <tr> <th>Min Qi</th> <td>12.971454</td> <td>9.383520</td> <td>59.180595</td> <td>1.603106</td> <td>58.394596</td> <td>10.487869</td> <td>42.485363</td> <td>92.035709</td> </tr> <tr> <th>Max Qi</th> <td>49.580860</td> <td>37.474814</td> <td>391.980128</td> <td>12.505446</td> <td>151.584400</td> <td>20.791892</td> <td>78.869819</td> <td>407.135077</td> </tr> <tr> <th>Avg Qi</th> <td>25.249883</td> <td>22.703369</td> <td>124.665586</td> <td>7.265276</td> <td>95.606870</td> <td>13.614409</td> <td>57.338546</td> <td>186.146037</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>146.487681</td> <td>6.798942</td> <td>29.834475</td> <td>3.177879</td> <td>55.067866</td> <td>4.503738</td> <td>17.215591</td> <td>9.333281</td> </tr> <tr> <th>Stream 1</th> <td>177.581204</td> <td>44.178095</td> <td>69.746005</td> <td>12.306166</td> <td>215.602727</td> <td>30.443709</td> <td>64.276384</td> <td>45.266949</td> </tr> <tr> <th>Stream 2</th> <td>211.311651</td> <td>27.403143</td> <td>61.412478</td> <td>12.173058</td> <td>216.879170</td> <td>18.272234</td> <td>96.753886</td> <td>35.587072</td> </tr> <tr> <th>Stream 3</th> <td>482.581456</td> <td>68.663026</td> <td>60.354163</td> <td>13.408513</td> <td>187.921639</td> <td>17.469237</td> <td>62.337222</td> <td>31.706120</td> </tr> <tr> <th>Stream 4</th> <td>178.297373</td> <td>23.711312</td> <td>67.129677</td> <td>15.216904</td> <td>328.149575</td> <td>20.258853</td> <td>78.891201</td> <td>84.852368</td> </tr> <tr> <th>Stream 5</th> <td>209.496498</td> <td>28.346366</td> <td>55.584081</td> <td>9.644075</td> <td>131.622351</td> <td>24.171156</td> <td>80.046801</td> <td>43.625932</td> </tr> <tr> <th>Stream 6</th> <td>521.691639</td> <td>24.126176</td> <td>72.964805</td> <td>15.311409</td> <td>146.152570</td> <td>34.748843</td> <td>71.957130</td> <td>58.470644</td> </tr> <tr> <th>Stream 7</th> <td>580.320149</td> <td>17.054563</td> <td>56.172396</td> <td>7.530832</td> <td>200.100326</td> <td>12.444021</td> <td>25.910599</td> <td>75.653693</td> </tr> <tr> <th>Stream 8</th> <td>472.231674</td> <td>15.064398</td> <td>89.875570</td> <td>42.394675</td> <td>166.589234</td> <td>12.831209</td> <td>81.697881</td> <td>73.821769</td> </tr> <tr> <th>Min Qi</th> <td>177.581204</td> <td>15.064398</td> <td>55.584081</td> <td>7.530832</td> <td>131.622351</td> <td>12.444021</td> <td>25.910599</td> <td>31.706120</td> </tr> <tr> <th>Max Qi</th> <td>580.320149</td> <td>68.663026</td> <td>89.875570</td> <td>42.394675</td> <td>328.149575</td> <td>34.748843</td> <td>96.753886</td> <td>84.852368</td> </tr> <tr> <th>Avg Qi</th> <td>354.188955</td> <td>31.068385</td> <td>66.654897</td> <td>15.998204</td> <td>199.127199</td> <td>21.329908</td> <td>70.233888</td> <td>56.123068</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>12.252670</td> <td>2.593733</td> <td>4.115862</td> <td>16.895672</td> <td>10.183350</td> <td>1.240096</td> <td>18.679685</td> <td>4.876067</td> </tr> <tr> <th>Stream 1</th> <td>356.740980</td> <td>21.197870</td> <td>30.422216</td> <td>81.779038</td> <td>65.468650</td> <td>3.947503</td> <td>63.933750</td> <td>107.563796</td> </tr> <tr> <th>Stream 2</th> <td>54.087768</td> <td>10.152604</td> <td>34.940701</td> <td>113.510640</td> <td>70.908809</td> <td>12.316233</td> <td>109.091578</td> <td>283.076004</td> </tr> <tr> <th>Stream 3</th> <td>52.807104</td> <td>18.525982</td> <td>13.740089</td> <td>212.364908</td> <td>16.413964</td> <td>17.998809</td> <td>58.653503</td> <td>483.718271</td> </tr> <tr> <th>Stream 4</th> <td>42.389062</td> <td>36.157809</td> <td>28.909260</td> <td>86.427025</td> <td>21.605419</td> <td>7.608729</td> <td>54.910853</td> <td>331.074114</td> </tr> <tr> <th>Stream 5</th> <td>48.214794</td> <td>15.778893</td> <td>20.681799</td> <td>130.560005</td> <td>43.846752</td> <td>33.905533</td> <td>54.536966</td> <td>139.563667</td> </tr> <tr> <th>Stream 6</th> <td>84.061840</td> <td>26.224851</td> <td>16.546432</td> <td>117.265210</td> <td>34.766856</td> <td>39.037423</td> <td>0.710642</td> <td>1.645351</td> </tr> <tr> <th>Stream 7</th> <td>63.034890</td> <td>15.966686</td> <td>31.666488</td> <td>112.689765</td> <td>28.661943</td> <td>12.828171</td> <td>1.274731</td> <td>1.780452</td> </tr> <tr> <th>Stream 8</th> <td>43.879104</td> <td>8.596666</td> <td>32.585746</td> <td>177.928730</td> <td>26.763334</td> <td>6.112333</td> <td>1.187693</td> <td>0.533668</td> </tr> <tr> <th>Min Qi</th> <td>42.389062</td> <td>8.596666</td> <td>13.740089</td> <td>81.779038</td> <td>16.413964</td> <td>3.947503</td> <td>0.710642</td> <td>0.533668</td> </tr> <tr> <th>Max Qi</th> <td>356.740980</td> <td>36.157809</td> <td>34.940701</td> <td>212.364908</td> <td>70.908809</td> <td>39.037423</td> <td>109.091578</td> <td>483.718271</td> </tr> <tr> <th>Avg Qi</th> <td>93.151943</td> <td>19.075170</td> <td>26.186591</td> <td>129.065665</td> <td>38.554466</td> <td>16.719342</td> <td>43.037465</td> <td>168.619415</td> </tr> </table> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac349bc698"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac347508c8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac347b2098"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac34f71318"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac346fa748"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac349de568"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aac34725b28"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aac36adde48"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aac34a7e368"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac348581b8"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac344340c8"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac34ee7058"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac34d388b8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x69cacf8"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aac00620518"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aabeaafeba8"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab51a83638"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aac14161af8"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:05:50.674672-04:00
In Hoc Signo Vinces (part 18 of n): Cluster Dynamics
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-09-26#1821
2014-09-26T21:07:20Z
<p>This article is about how scale-out differs from single-server. This shows large effects of parameters whose very existence most would not anticipate, and some low level metrics for assessing these. The moral of the story is that this is the stuff which makes the difference between merely surviving scale-out and winning with it. The developer and DBA would not normally know about this; thus these things fall into the category of adaptive self-configuration expected from the DBMS. But since this series is about what makes performance, I will discuss the dynamics such as they are and how to play these.</p> <p>We take the prototypical cross partition join in Q13: Make a hash table of all <code>customers</code>, partitioned by <code>c_custkey</code>. This is independently done with full parallelism in each partition. Scan the <code>orders</code>, get the <code>customer</code> (in a different partition), and flag the <code>customers</code> that had at least one <code>order</code>. Then, to get the <code>customers</code> with no <code>orders</code>, return the <code>customers</code> that were not flagged in the previous pass.</p> <p>The single-server time in part 12 was 7.8 and 6.0 with a single user. We consider the better of the times. The difference is due to allocating memory on the first go; on the second go the memory is already in reserve.</p> <p>With default settings, we get 4595 ms (microseconds), with per node resource utilization at:</p> <blockquote> <pre><code> Cluster 4 nodes, 4 s. 112405 m/s 742602 KB/s 2749% cpu 0% read 4% clw threads 1r 0w 0i buffers 8577766 287874 d 0 w 0 pfs cl 1: 27867 m/s 185654 KB/s 733% cpu 0% read 4% clw threads 1r 0w 0i buffers 2144242 71757 d 0 w 0 pfs cl 2: 28149 m/s 185372 KB/s 672% cpu 0% read 0% clw threads 0r 0w 0i buffers 2144640 71903 d 0 w 0 pfs cl 3: 28220 m/s 185621 KB/s 675% cpu 0% read 0% clw threads 0r 0w 0i buffers 2144454 71962 d 0 w 0 pfs cl 4: 28150 m/s 185837 KB/s 667% cpu 0% read 0% clw threads 0r 0w 0i buffers 2144430 72252 d 0 w 0 pfs </code> </pre></blockquote> <p>The top line is the summary; the lines below are per-process. The <code>m/s</code> is messages-per-second; <code>KB/s</code> is interconnect traffic per second; <code>clw %</code> is idle time spent waiting for a reply from another process. The cluster is set up with 4 processes across 2 machines, each with 2 NUMA nodes. Each process has affinity to the NUMA node, so local memory only. The time is reasonable in light of the overall CPU of 2700%. The maximum would be 4800% with all threads of all cores busy all the time.</p> <p>The catch here is that we do not have a steady half-platform utilization all the time, but full platform peaks followed by synchronization barriers with very low utilization. So, we set the batch size differently:</p> <blockquote> <pre><code> cl_exec ('__dbf_set (''cl_dfg_batch_bytes'', 50000000)'); </code> </pre></blockquote> <p>This means that we set, on each process, the <code>cl_dfg_batch_bytes</code> to 50M from a default of 10M. The effect is that each scan of <code>orders</code>, one thread per slice, 48 slices total, will produce 50MB worth of <code>o_custkeys</code> to be sent to the other partition for getting the <code>customer</code>. After each 50M, the thread stops and will produce the next batch when all are done and a global continue message is sent by the coordinator.</p> <p>The time is now 3173 ms with:</p> <blockquote> <pre><code> Cluster 4 nodes, 3 s. 158220 m/s 1054944 KB/s 3676% cpu 0% read 1% clw threads 1r 0w 0i buffers 8577766 287874 d 0 w 0 pfs cl 1: 39594 m/s 263962 KB/s 947% cpu 0% read 1% clw threads 1r 0w 0i buffers 2144242 71757 d 0 w 0 pfs cl 2: 39531 m/s 263476 KB/s 894% cpu 0% read 0% clw threads 0r 0w 0i buffers 2144640 71903 d 0 w 0 pfs cl 3: 39523 m/s 263684 KB/s 933% cpu 0% read 0% clw threads 0r 0w 0i buffers 2144454 71962 d 0 w 0 pfs cl 4: 39535 m/s 263586 KB/s 900% cpu 0% read 0% clw threads 0r 0w 0i buffers 2144430 72252 d 0 w 0 pfs </code> </pre></blockquote> <p>The platform utilization is better as we see. The throughput is nearly double that of the single-server, which is pretty good for a communication-heavy query.</p> <p>This was done with a vector size of 10K. In other words, each partition gets 10K <code>o_custkeys</code> and splits these 48 ways to go to every recipient. 1/4 are in the same process, 1/4 in a different process on the same machine, and 2/4 on a different machine. The recipient gets messages with an average of 208 <code>o_custkey</code> values, puts them back together in batches of 10K, and passes these to the hash join with <code>customer</code>.</p> <p>We try different vector sizes, such as 100K: </p> <blockquote> <pre><code> cl_exec ('__dbf_set (''dc_batch_sz'', 100000)'); </code> </pre></blockquote> <p>There are two metrics of interest here: The write block time, and the scheduling overhead. The write block time is microseconds, which increases whenever a thread must wait before it can write to a connection. The scheduling overhead is cumulative clocks spent by threads while waiting for a critical section that deals with dispatching messages to consumer threads. Long messages make blocking; short messages make frequent scheduling decisions.</p> <blockquote> <pre><code> SELECT cl_sys_stat ('local_cll_clk', clr=>1), cl_sys_stat ('write_block_usec', clr=>1) ; </code> </pre></blockquote> <p> <code>cl_sys_stat</code> gets the counters from all processes and returns the sum. <code>clr=>1</code> means that the counter is cleared after read.</p> <p>We do Q13 with vector sizes of 10, 100, and 1000K.</p> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th>Vector size</th> <th>msec</th> <th>mtx</th> <th>wblock</th> </tr> <tr> <th>10K</th> <td style="text-align:right">3297</td> <td style="text-align:right">10,829,910,329</td> <td style="text-align:right">0</td> </tr> <tr> <th>100K</th> <td style="text-align:right">3150</td> <td style="text-align:right">1,663,238,367</td> <td style="text-align:right">59,132</td> </tr> <tr> <th>1000K</th> <td style="text-align:right">3876</td> <td style="text-align:right">414,631,129</td> <td style="text-align:right">4,578,003</td> </tr> </table> <p>So, 100K seems to strike the best balance between scheduling and blocking on write.</p> <p>The times are measured after several samples with each setting. The times stabilize after a few runs, as the appropriate size memory blocks are in reserve. Calling <code>mmap</code> to allocate these on the first run with each size has a very high penalty, e.g., 60s for the first run with 1M vector size. We note that blocking on write is really bad even though 1/3 of the time there is no network and 2/3 of the time there is a fast network (QDR IB) with no other load. Further, the affinities are set so that the thread responsible for incoming messages is always on core. Result variability on consecutive runs is under 5%, which is similar to single-server behavior.</p> <p>It would seem that a mutex, as bad as it is, is still better than a distributed cause for going off core (blocking on write). The latency for continuing a thread thus blocked is of course higher than the latency for continuing one that is waiting for a mutex.</p> <p>We note that a cluster with more machines can take a longer vector size because a vector spreads out to more recipients. The key seems to be to set the message size so that blocking on write is not common. This is a possible adaptive execution feature. We have seen no particular benefit from SDP (Sockets Direct Protocol) and its zero copy. This is a TCP replacement that comes with the InfiniBand drivers.</p> <p>We will next look at replication/partitioning tradeoffs for hash joins. Then we can look at full runs.</p> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac349bc698"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac347508c8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac347b2098"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac34f71318"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac346fa748"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac349de568"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aac34725b28"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aac36adde48"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aac34a7e368"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac348581b8"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac344340c8"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac34ee7058"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac34d388b8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x69cacf8"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aac00620518"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> In Hoc Signo Vinces (part 18): Cluster Dynamics<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aabd70b7238"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aabd0b58938"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:09:25.032600-04:00
In Hoc Signo Vinces (part 17 of n): 100G and 300G Runs on Dual Xeon E5 2650v2
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-09-26#1820
2014-09-26T21:07:14Z
<p>This is an update presenting sample results on a newer platform for a single-server configuration. This is to verify that performance scales with the addition of cores and clock speed. Further, we note that the jump from 100G to 300G changes very little about the score. 3x larger takes approximately 3x longer, as long as things are in memory.</p> <p>The platform is one node of the <a href="http://www.cwi.nl/" id="link-id0x2aac007ba808">CWI</a> <a href="http://scilens.project.cwi.nl/" id="link-id0x2aac010f21a8">cluster</a> which was also used for the <a href="http://www.openlinksw.com/weblog/oerling/?id=1804" id="link-id0x2aac007ba828">500Gt RDF experiments reported on this blog</a>. The specification is dual Xeon E5 2650v2 (8 core, 16 thread, 2.6 GHz) with 256 GB RAM. The disk setup is a RAID-0 of three 2 TB rotating disks.</p> <p>For the 100G, we go from 240 to 395, which is about 1.64x. The new platform has 16 vs 12 cores and a clock of 2.6 as opposed to 2.3. This makes a multiplier of 1.5. The rest of the acceleration is probably attributable to faster memory clock. Anyway, the point of more speed from larger platform is made.</p> <p>The top level scores per run are as follows; the numerical quantities summaries are appended.</p> <h3>100G</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th>Run</th> <th>Power</th> <th>Throughput</th> <th>Composite</th> </tr> <tr> <th>Run 1</th> <td style="text-align:right">391,000.1</td> <td style="text-align:right">401,029.4</td> <td style="text-align:right">395,983.0</td> </tr> <tr> <th>Run 2</th> <td style="text-align:right">388,746.2</td> <td style="text-align:right">404,189.3</td> <td style="text-align:right">396,392.6</td> </tr> </table> <h3>300G</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th>Run</th> <th>Power</th> <th>Throughput</th> <th>Composite </th> </tr> <tr> <th>Run 1</th> <td style="text-align:right">61,988.7</td> <td style="text-align:right">384,883.7</td> <td style="text-align:right">154,461.6</td> </tr> <tr> <th>Run 2</th> <td style="text-align:right">423,431.8</td> <td style="text-align:right">387,248.6</td> <td style="text-align:right">404,936.3</td> </tr> <tr> <th>Run 3</th> <td style="text-align:right">417,672.0</td> <td style="text-align:right">389,719.5</td> <td style="text-align:right">403,453.7</td> </tr> </table> <p>The interested may reproduce the results using the <a href="https://github.com/v7fasttrack/virtuoso-opensource/tree/feature/analytics" id="link-id0x2aabe9517e78">feature/analytics branch</a> of the <a href="https://github.com/v7fasttrack" id="link-id0x2aabeb525948">v7fasttrack git repository</a> on <a href="https://github.com/" id="link-id0x2aac007a2198">GitHub</a> as described in <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac00e379f8">Part 13</a>.</p> <p>For the 300G runs, we note a much longer load time; see below, as this is seriously IO bound.</p> <p>The first power test at 300G is a non-starter, even though this comes right after bulk load. Still, the data is not in working set and getting it from disk is simply an automatic disqualification, unless maybe one had 300 separate disks. This happens in TPC benchmarks, but not very often in the field. Looking at the first power run, the first queries take the longest, but by the time the power run starts, the working set is there. By an artifact of the metric (use of geometric mean for the power test), long queries are penalized less there than in the throughput run.</p> <p>So, we run 3 executions instead of the prescribed 2, to have 2 executions from warm state.</p> <p>To do 300G well in 256 GB of RAM, one needs either to use several SSDs, or to increase compression and keep all in memory, so no secondary storage at all. In order to keep all in memory, one could have stream-compression on string columns. Stream-compressing strings (e.g., <code>o_comment</code>, <code>l_comment</code>) does not pay if one is already in memory, but if stream-compressing strings eliminates going to secondary storage, then the win is sure.</p> <p>As before, all caveats apply; the results are unaudited and for information only. Therefore we do not use the official metric name.</p> <h2>100G Run 1</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>September 15, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>100</td> </tr> <tr> <th style="text-align:right">Start of Database Load</th> <td>09/15/2014 07:04:08</td> </tr> <tr> <th style="text-align:right">End of Database Load</th> <td>09/15/2014 07:15:58</td> </tr> <tr> <th style="text-align:right">Database Load Time</th> <td>0:11:50</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>5</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td>391,000.1</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td>401,029.4</td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@100GB)</th> <td>395,983.0</td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td>98.846000 seconds</td> </tr> </table> <h3>Duration of stream execution</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>09/15/2014 13:13:01</td> <td>09/15/2014 13:13:28</td> <td style="text-align:right">0:00:27</td> </tr> <tr> <th>Stream 1</th> <td>09/15/2014 13:13:29</td> <td>09/15/2014 13:15:06</td> <td style="text-align:right">0:01:37</td> </tr> <tr> <th>Stream 2</th> <td>09/15/2014 13:13:29</td> <td>09/15/2014 13:15:07</td> <td style="text-align:right">0:01:38</td> </tr> <tr> <th>Stream 3</th> <td>09/15/2014 13:13:29</td> <td>09/15/2014 13:15:07</td> <td style="text-align:right">0:01:38</td> </tr> <tr> <th>Stream 4</th> <td>09/15/2014 13:13:29</td> <td>09/15/2014 13:15:04</td> <td style="text-align:right">0:01:35</td> </tr> <tr> <th>Stream 5</th> <td>09/15/2014 13:13:29</td> <td>09/15/2014 13:15:08</td> <td style="text-align:right">0:01:39</td> </tr> <tr> <th>Refresh 0</th> <td>09/15/2014 13:13:01</td> <td>09/15/2014 13:13:03</td> <td style="text-align:right">0:00:02</td> </tr> <tr> <th> </th> <td>09/15/2014 13:13:28</td> <td>09/15/2014 13:13:29</td> <td style="text-align:right">0:00:01</td> </tr> <tr> <th>Refresh 1</th> <td>09/15/2014 13:14:10</td> <td>09/15/2014 13:14:16</td> <td style="text-align:right">0:00:06</td> </tr> <tr> <th>Refresh 2</th> <td>09/15/2014 13:13:29</td> <td>09/15/2014 13:13:42</td> <td style="text-align:right">0:00:13</td> </tr> <tr> <th>Refresh 3</th> <td>09/15/2014 13:13:42</td> <td>09/15/2014 13:13:53</td> <td style="text-align:right">0:00:11</td> </tr> <tr> <th>Refresh 4</th> <td>09/15/2014 13:13:53</td> <td>09/15/2014 13:14:02</td> <td style="text-align:right">0:00:09</td> </tr> <tr> <th>Refresh 5</th> <td>09/15/2014 13:14:02</td> <td>09/15/2014 13:14:10</td> <td style="text-align:right">0:00:08</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> <th>Stream 0</th> <td>1.442477</td> <td>0.304513</td> <td>0.720263</td> <td>0.351285</td> <td>0.979414</td> <td>0.479455</td> <td>0.865992</td> <td>0.875236</td> </tr> <tr> <th>Stream 1</th> <td>3.938133</td> <td>0.920533</td> <td>3.738724</td> <td>2.769707</td> <td>3.209728</td> <td>1.339146</td> <td>2.759384</td> <td>3.626868</td> </tr> <tr> <th>Stream 2</th> <td>4.104738</td> <td>0.952245</td> <td>4.719658</td> <td>0.865586</td> <td>2.139267</td> <td>0.850909</td> <td>2.044402</td> <td>2.600373</td> </tr> <tr> <th>Stream 3</th> <td>3.692119</td> <td>1.024876</td> <td>3.430172</td> <td>1.579846</td> <td>4.097845</td> <td>1.859468</td> <td>2.312921</td> <td>6.238070</td> </tr> <tr> <th>Stream 4</th> <td>5.419537</td> <td>0.531571</td> <td>2.116176</td> <td>1.256836</td> <td>4.787617</td> <td>2.117995</td> <td>3.517466</td> <td>3.982180</td> </tr> <tr> <th>Stream 5</th> <td>5.167029</td> <td>0.746720</td> <td>3.157557</td> <td>1.255182</td> <td>3.004802</td> <td>2.131963</td> <td>3.648316</td> <td>2.835751</td> </tr> <tr> <th>Min Qi</th> <td>3.692119</td> <td>0.531571</td> <td>2.116176</td> <td>0.865586</td> <td>2.139267</td> <td>0.850909</td> <td>2.044402</td> <td>2.600373</td> </tr> <tr> <th>Max Qi</th> <td>5.419537</td> <td>1.024876</td> <td>4.719658</td> <td>2.769707</td> <td>4.787617</td> <td>2.131963</td> <td>3.648316</td> <td>6.238070</td> </tr> <tr> <th>Avg Qi</th> <td>4.464311</td> <td>0.835189</td> <td>3.432457</td> <td>1.545431</td> <td>3.447852</td> <td>1.659896</td> <td>2.856498</td> <td>3.856648</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>2.606044</td> <td>1.117063</td> <td>1.847930</td> <td>0.618534</td> <td>4.327600</td> <td>1.110908</td> <td>0.995289</td> <td>0.975910</td> </tr> <tr> <th>Stream 1</th> <td>7.463593</td> <td>4.686463</td> <td>4.549733</td> <td>4.168129</td> <td>15.759178</td> <td>5.247666</td> <td>4.495030</td> <td>4.075198</td> </tr> <tr> <th>Stream 2</th> <td>9.398552</td> <td>5.170904</td> <td>3.934405</td> <td>1.880683</td> <td>19.968787</td> <td>3.767992</td> <td>6.965337</td> <td>3.849845</td> </tr> <tr> <th>Stream 3</th> <td>7.581069</td> <td>4.109905</td> <td>4.301159</td> <td>2.123634</td> <td>17.683200</td> <td>5.383603</td> <td>4.376887</td> <td>2.854777</td> </tr> <tr> <th>Stream 4</th> <td>9.927887</td> <td>6.913209</td> <td>3.351489</td> <td>2.802724</td> <td>16.985827</td> <td>3.925148</td> <td>4.691474</td> <td>4.080586</td> </tr> <tr> <th>Stream 5</th> <td>7.035080</td> <td>3.921425</td> <td>6.844778</td> <td>2.899238</td> <td>14.839509</td> <td>4.986742</td> <td>6.629664</td> <td>4.089547</td> </tr> <tr> <th>Min Qi</th> <td>7.035080</td> <td>3.921425</td> <td>3.351489</td> <td>1.880683</td> <td>14.839509</td> <td>3.767992</td> <td>4.376887</td> <td>2.854777</td> </tr> <tr> <th>Max Qi</th> <td>9.927887</td> <td>6.913209</td> <td>6.844778</td> <td>4.168129</td> <td>19.968787</td> <td>5.383603</td> <td>6.965337</td> <td>4.089547</td> </tr> <tr> <th>Avg Qi</th> <td>8.281236</td> <td>4.960381</td> <td>4.596313</td> <td>2.774882</td> <td>17.047300</td> <td>4.662230</td> <td>5.431678</td> <td>3.789991</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>1.215956</td> <td>0.745257</td> <td>0.699801</td> <td>1.281834</td> <td>1.291110</td> <td>0.518425</td> <td>1.827192</td> <td>1.014431</td> </tr> <tr> <th>Stream 1</th> <td>5.779854</td> <td>2.383264</td> <td>2.396793</td> <td>6.130511</td> <td>5.002700</td> <td>1.968425</td> <td>4.172437</td> <td>2.427047</td> </tr> <tr> <th>Stream 2</th> <td>7.828176</td> <td>1.833416</td> <td>3.175649</td> <td>4.785709</td> <td>5.385834</td> <td>1.403290</td> <td>6.383005</td> <td>6.366525</td> </tr> <tr> <th>Stream 3</th> <td>5.880139</td> <td>1.797383</td> <td>3.258024</td> <td>5.601364</td> <td>6.373216</td> <td>1.977848</td> <td>5.235542</td> <td>6.385010</td> </tr> <tr> <th>Stream 4</th> <td>3.989621</td> <td>1.252891</td> <td>2.478303</td> <td>4.678629</td> <td>3.212176</td> <td>2.740586</td> <td>5.037995</td> <td>3.911379</td> </tr> <tr> <th>Stream 5</th> <td>5.030440</td> <td>2.010988</td> <td>4.188428</td> <td>6.221990</td> <td>5.418788</td> <td>2.187718</td> <td>3.589915</td> <td>3.517380</td> </tr> <tr> <th>Min Qi</th> <td>3.989621</td> <td>1.252891</td> <td>2.396793</td> <td>4.678629</td> <td>3.212176</td> <td>1.403290</td> <td>3.589915</td> <td>2.427047</td> </tr> <tr> <th>Max Qi</th> <td>7.828176</td> <td>2.383264</td> <td>4.188428</td> <td>6.221990</td> <td>6.373216</td> <td>2.740586</td> <td>6.383005</td> <td>6.385010</td> </tr> <tr> <th>Avg Qi</th> <td>5.701646</td> <td>1.855588</td> <td>3.099439</td> <td>5.483641</td> <td>5.078543</td> <td>2.055573</td> <td>4.883779</td> <td>4.521468</td> </tr> </table> <h2>100G Run 2</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>September 15, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>100</td> </tr> <tr> <th style="text-align:right">Total Data Storage/Database Size</th> <td>87,312M</td> </tr> <tr> <th style="text-align:right">Start of Database Load</th> <td>09/15/2014 07:04:08</td> </tr> <tr> <th style="text-align:right">End of Database Load</th> <td>09/15/2014 07:15:58</td> </tr> <tr> <th style="text-align:right">Database Load Time</th> <td>0:11:50</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>5</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td>388,746.2</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td>404,189.3</td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@100GB)</th> <td>396,392.6</td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td>98.074000 seconds</td> </tr> </table> <h3>Duration of stream execution:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>09/15/2014 13:15:11</td> <td>09/15/2014 13:15:38</td> <td>0:00:27</td> </tr> <tr> <th>Stream 1</th> <td>09/15/2014 13:15:39</td> <td>09/15/2014 13:17:13</td> <td>0:01:34</td> </tr> <tr> <th>Stream 2</th> <td>09/15/2014 13:15:39</td> <td>09/15/2014 13:17:16</td> <td>0:01:37</td> </tr> <tr> <th>Stream 3</th> <td>09/15/2014 13:15:39</td> <td>09/15/2014 13:17:15</td> <td>0:01:36</td> </tr> <tr> <th>Stream 4</th> <td>09/15/2014 13:15:39</td> <td>09/15/2014 13:17:17</td> <td>0:01:38</td> </tr> <tr> <th>Stream 5</th> <td>09/15/2014 13:15:39</td> <td>09/15/2014 13:17:15</td> <td>0:01:36</td> </tr> <tr> <th>Refresh 0</th> <td>09/15/2014 13:15:11</td> <td>09/15/2014 13:15:12</td> <td>0:00:01</td> </tr> <tr> <th> </th> <td>09/15/2014 13:15:38</td> <td>09/15/2014 13:15:39</td> <td>0:00:01</td> </tr> <tr> <th>Refresh 1</th> <td>09/15/2014 13:16:13</td> <td>09/15/2014 13:16:20</td> <td>0:00:07</td> </tr> <tr> <th>Refresh 2</th> <td>09/15/2014 13:15:39</td> <td>09/15/2014 13:15:47</td> <td>0:00:08</td> </tr> <tr> <th>Refresh 3</th> <td>09/15/2014 13:15:47</td> <td>09/15/2014 13:15:56</td> <td>0:00:09</td> </tr> <tr> <th>Refresh 4</th> <td>09/15/2014 13:15:56</td> <td>09/15/2014 13:16:03</td> <td>0:00:07</td> </tr> <tr> <th>Refresh 5</th> <td>09/15/2014 13:16:03</td> <td>09/15/2014 13:16:12</td> <td>0:00:09</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> <th>Stream 0</th> <td>1.467681</td> <td>0.277665</td> <td>0.766102</td> <td>0.365185</td> <td>0.941206</td> <td>0.549381</td> <td>0.938998</td> <td>0.803514</td> </tr> <tr> <th>Stream 1</th> <td>3.883169</td> <td>1.488521</td> <td>3.366920</td> <td>1.627478</td> <td>3.632321</td> <td>2.065565</td> <td>2.911138</td> <td>2.444544</td> </tr> <tr> <th>Stream 2</th> <td>3.294589</td> <td>1.138066</td> <td>3.260775</td> <td>1.899615</td> <td>5.367725</td> <td>1.820374</td> <td>3.655119</td> <td>2.186642</td> </tr> <tr> <th>Stream 3</th> <td>3.797641</td> <td>0.995877</td> <td>3.239690</td> <td>2.483035</td> <td>2.737690</td> <td>1.505998</td> <td>4.058083</td> <td>4.268644</td> </tr> <tr> <th>Stream 4</th> <td>4.099187</td> <td>0.402685</td> <td>4.704959</td> <td>1.469825</td> <td>5.367910</td> <td>2.783018</td> <td>2.706164</td> <td>2.551061</td> </tr> <tr> <th>Stream 5</th> <td>3.651273</td> <td>1.598314</td> <td>2.051899</td> <td>1.283754</td> <td>4.711897</td> <td>1.519763</td> <td>2.851300</td> <td>2.484093</td> </tr> <tr> <th>Min Qi</th> <td>3.294589</td> <td>0.402685</td> <td>2.051899</td> <td>1.283754</td> <td>2.737690</td> <td>1.505998</td> <td>2.706164</td> <td>2.186642</td> </tr> <tr> <th>Max Qi</th> <td>4.099187</td> <td>1.598314</td> <td>4.704959</td> <td>2.483035</td> <td>5.367910</td> <td>2.783018</td> <td>4.058083</td> <td>4.268644</td> </tr> <tr> <th>Avg Qi</th> <td>3.745172</td> <td>1.124693</td> <td>3.324849</td> <td>1.752741</td> <td>4.363509</td> <td>1.938944</td> <td>3.236361</td> <td>2.786997</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>2.734812</td> <td>1.115539</td> <td>1.679910</td> <td>0.633239</td> <td>4.391739</td> <td>1.130082</td> <td>1.137284</td> <td>0.919646</td> </tr> <tr> <th>Stream 1</th> <td>9.271071</td> <td>5.664855</td> <td>3.377869</td> <td>2.148228</td> <td>16.046021</td> <td>2.935643</td> <td>4.897009</td> <td>2.891040</td> </tr> <tr> <th>Stream 2</th> <td>10.272523</td> <td>4.578427</td> <td>4.086788</td> <td>2.312762</td> <td>16.295728</td> <td>2.714776</td> <td>6.393897</td> <td>2.414951</td> </tr> <tr> <th>Stream 3</th> <td>7.095213</td> <td>4.544636</td> <td>4.073433</td> <td>2.710320</td> <td>18.789088</td> <td>3.903873</td> <td>5.471600</td> <td>2.994184</td> </tr> <tr> <th>Stream 4</th> <td>7.567924</td> <td>3.691088</td> <td>3.951049</td> <td>2.207944</td> <td>18.189014</td> <td>4.985841</td> <td>6.568935</td> <td>3.965322</td> </tr> <tr> <th>Stream 5</th> <td>8.173577</td> <td>4.959777</td> <td>4.736593</td> <td>3.507469</td> <td>17.106990</td> <td>5.405699</td> <td>7.357104</td> <td>3.125788</td> </tr> <tr> <th>Min Qi</th> <td>7.095213</td> <td>3.691088</td> <td>3.377869</td> <td>2.148228</td> <td>16.046021</td> <td>2.714776</td> <td>4.897009</td> <td>2.414951</td> </tr> <tr> <th>Max Qi</th> <td>10.272523</td> <td>5.664855</td> <td>4.736593</td> <td>3.507469</td> <td>18.789088</td> <td>5.405699</td> <td>7.357104</td> <td>3.965322</td> </tr> <tr> <th>Avg Qi</th> <td>8.476062</td> <td>4.687757</td> <td>4.045146</td> <td>2.577345</td> <td>17.285368</td> <td>3.989166</td> <td>6.137709</td> <td>3.078257</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>1.206347</td> <td>0.792013</td> <td>0.699476</td> <td>1.349182</td> <td>1.505387</td> <td>0.543947</td> <td>1.549135</td> <td>0.824344</td> </tr> <tr> <th>Stream 1</th> <td>5.135036</td> <td>1.873195</td> <td>4.978155</td> <td>5.988226</td> <td>4.705365</td> <td>1.211049</td> <td>4.175947</td> <td>3.579242</td> </tr> <tr> <th>Stream 2</th> <td>7.656125</td> <td>2.229819</td> <td>2.805272</td> <td>6.629781</td> <td>4.138014</td> <td>1.423334</td> <td>5.165700</td> <td>3.197300</td> </tr> <tr> <th>Stream 3</th> <td>6.385983</td> <td>2.086301</td> <td>3.450305</td> <td>3.292353</td> <td>5.503905</td> <td>2.302992</td> <td>4.860041</td> <td>3.865383</td> </tr> <tr> <th>Stream 4</th> <td>6.514967</td> <td>2.876895</td> <td>3.481100</td> <td>1.629007</td> <td>5.715903</td> <td>2.121692</td> <td>3.681208</td> <td>3.347289</td> </tr> <tr> <th>Stream 5</th> <td>4.100205</td> <td>2.400816</td> <td>2.142291</td> <td>4.710677</td> <td>5.765320</td> <td>1.616445</td> <td>6.095817</td> <td>3.007436</td> </tr> <tr> <th>Min Qi</th> <td>4.100205</td> <td>1.873195</td> <td>2.142291</td> <td>1.629007</td> <td>4.138014</td> <td>1.211049</td> <td>3.681208</td> <td>3.007436</td> </tr> <tr> <th>Max Qi</th> <td>7.656125</td> <td>2.876895</td> <td>4.978155</td> <td>6.629781</td> <td>5.765320</td> <td>2.302992</td> <td>6.095817</td> <td>3.865383</td> </tr> <tr> <th>Avg Qi</th> <td>5.958463</td> <td>2.293405</td> <td>3.371425</td> <td>4.450009</td> <td>5.165701</td> <td>1.735102</td> <td>4.795743</td> <td>3.399330</td> </tr> </table> <h2>300G Run 1</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>September 25, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>300</td> </tr> <tr> <th style="text-align:right">Start of Database Load</th> <td>09/25/2014 16:38:20</td> </tr> <tr> <th style="text-align:right">End of Database Load</th> <td>09/25/2014 18:32:06</td> </tr> <tr> <th style="text-align:right">Database Load Time</th> <td>1:53:46</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>6</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td>61,988.7</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td>384,883.7</td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@300GB)</th> <td>154,461.6</td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td>370.498000 seconds</td> </tr> </table> <h3>Duration of stream execution:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>09/25/2014 19:00:29</td> <td>09/25/2014 19:22:25</td> <td>0:21:56</td> </tr> <tr> <th>Stream 1</th> <td>09/25/2014 19:22:27</td> <td>09/25/2014 19:28:23</td> <td>0:05:56</td> </tr> <tr> <th>Stream 2</th> <td>09/25/2014 19:22:27</td> <td>09/25/2014 19:28:23</td> <td>0:05:56</td> </tr> <tr> <th>Stream 3</th> <td>09/25/2014 19:22:27</td> <td>09/25/2014 19:28:26</td> <td>0:05:59</td> </tr> <tr> <th>Stream 4</th> <td>09/25/2014 19:22:27</td> <td>09/25/2014 19:28:13</td> <td>0:05:46</td> </tr> <tr> <th>Stream 5</th> <td>09/25/2014 19:22:27</td> <td>09/25/2014 19:28:38</td> <td>0:06:11</td> </tr> <tr> <th>Stream 6</th> <td>09/25/2014 19:22:27</td> <td>09/25/2014 19:28:38</td> <td>0:06:11</td> </tr> <tr> <th>Refresh 0</th> <td>09/25/2014 19:00:29</td> <td>09/25/2014 19:03:56</td> <td>0:03:27</td> </tr> <tr> <th> </th> <td>09/25/2014 19:22:25</td> <td>09/25/2014 19:22:27</td> <td>0:00:02</td> </tr> <tr> <th>Refresh 1</th> <td>09/25/2014 19:25:22</td> <td>09/25/2014 19:25:58</td> <td>0:00:36</td> </tr> <tr> <th>Refresh 2</th> <td>09/25/2014 19:22:27</td> <td>09/25/2014 19:23:11</td> <td>0:00:44</td> </tr> <tr> <th>Refresh 3</th> <td>09/25/2014 19:23:10</td> <td>09/25/2014 19:23:40</td> <td>0:00:30</td> </tr> <tr> <th>Refresh 4</th> <td>09/25/2014 19:23:40</td> <td>09/25/2014 19:24:21</td> <td>0:00:41</td> </tr> <tr> <th>Refresh 5</th> <td>09/25/2014 19:24:21</td> <td>09/25/2014 19:24:58</td> <td>0:00:37</td> </tr> <tr> <th>Refresh 6</th> <td>09/25/2014 19:24:59</td> <td>09/25/2014 19:25:22</td> <td>0:00:23</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> <th>Stream 0</th> <td>183.735463</td> <td>95.826361</td> <td>79.826802</td> <td>87.603164</td> <td>47.099641</td> <td>1.301704</td> <td>2.606488</td> <td>52.667426</td> </tr> <tr> <th>Stream 1</th> <td>9.400003</td> <td>1.983777</td> <td>15.839250</td> <td>3.001843</td> <td>15.593335</td> <td>6.067716</td> <td>8.870516</td> <td>11.679706</td> </tr> <tr> <th>Stream 2</th> <td>12.634711</td> <td>3.472203</td> <td>13.683075</td> <td>8.057952</td> <td>16.500741</td> <td>5.403771</td> <td>11.181661</td> <td>12.393932</td> </tr> <tr> <th>Stream 3</th> <td>10.807287</td> <td>3.793587</td> <td>15.844244</td> <td>3.214977</td> <td>15.960600</td> <td>7.099744</td> <td>10.424530</td> <td>21.001623</td> </tr> <tr> <th>Stream 4</th> <td>11.900829</td> <td>3.741707</td> <td>14.219904</td> <td>5.616907</td> <td>16.487144</td> <td>14.229782</td> <td>11.100193</td> <td>8.769539</td> </tr> <tr> <th>Stream 5</th> <td>13.933423</td> <td>2.916529</td> <td>19.453452</td> <td>5.258843</td> <td>16.706269</td> <td>7.948711</td> <td>8.982104</td> <td>17.566729</td> </tr> <tr> <th>Stream 6</th> <td>17.084445</td> <td>0.738683</td> <td>11.503079</td> <td>8.324812</td> <td>23.483917</td> <td>20.101834</td> <td>9.207737</td> <td>10.311292</td> </tr> <tr> <th>Min Qi</th> <td>9.400003</td> <td>0.738683</td> <td>11.503079</td> <td>3.001843</td> <td>15.593335</td> <td>5.403771</td> <td>8.870516</td> <td>8.769539</td> </tr> <tr> <th>Max Qi</th> <td>17.084445</td> <td>3.793587</td> <td>19.453452</td> <td>8.324812</td> <td>23.483917</td> <td>20.101834</td> <td>11.181661</td> <td>21.001623</td> </tr> <tr> <th>Avg Qi</th> <td>12.626783</td> <td>2.774414</td> <td>15.090501</td> <td>5.579222</td> <td>17.455334</td> <td>10.141926</td> <td>9.961123</td> <td>13.620470</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>41.997798</td> <td>2.727870</td> <td>21.651730</td> <td>25.704209</td> <td>293.103984</td> <td>3.171437</td> <td>2.886688</td> <td>5.298823</td> </tr> <tr> <th>Stream 1</th> <td>29.662265</td> <td>22.788618</td> <td>12.979253</td> <td>7.121358</td> <td>62.774323</td> <td>22.132581</td> <td>22.616793</td> <td>21.625334</td> </tr> <tr> <th>Stream 2</th> <td>28.041750</td> <td>22.481172</td> <td>19.262140</td> <td>5.790272</td> <td>58.105179</td> <td>16.809177</td> <td>32.813330</td> <td>12.692499</td> </tr> <tr> <th>Stream 3</th> <td>32.534297</td> <td>15.460256</td> <td>12.038047</td> <td>7.012926</td> <td>59.413740</td> <td>18.540284</td> <td>25.968635</td> <td>16.716208</td> </tr> <tr> <th>Stream 4</th> <td>28.759993</td> <td>15.123651</td> <td>21.734471</td> <td>6.920480</td> <td>63.119744</td> <td>12.848884</td> <td>21.372432</td> <td>11.662102</td> </tr> <tr> <th>Stream 5</th> <td>18.315308</td> <td>21.781800</td> <td>26.141212</td> <td>8.230858</td> <td>60.985590</td> <td>22.369824</td> <td>27.098660</td> <td>25.283066</td> </tr> <tr> <th>Stream 6</th> <td>31.455961</td> <td>27.078707</td> <td>12.954580</td> <td>11.081669</td> <td>72.483462</td> <td>12.376376</td> <td>22.129120</td> <td>11.439147</td> </tr> <tr> <th>Min Qi</th> <td>18.315308</td> <td>15.123651</td> <td>12.038047</td> <td>5.790272</td> <td>58.105179</td> <td>12.376376</td> <td>21.372432</td> <td>11.439147</td> </tr> <tr> <th>Max Qi</th> <td>32.534297</td> <td>27.078707</td> <td>26.141212</td> <td>11.081669</td> <td>72.483462</td> <td>22.369824</td> <td>32.813330</td> <td>25.283066</td> </tr> <tr> <th>Avg Qi</th> <td>28.128262</td> <td>20.785701</td> <td>17.518284</td> <td>7.692927</td> <td>62.813673</td> <td>17.512854</td> <td>25.333162</td> <td>16.569726</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>7.793403</td> <td>81.545934</td> <td>41.648484</td> <td>4.638731</td> <td>25.003179</td> <td>0.536267</td> <td>206.980380</td> <td>2.501589</td> </tr> <tr> <th>Stream 1</th> <td>27.058060</td> <td>3.894254</td> <td>8.664394</td> <td>25.315007</td> <td>11.921265</td> <td>3.561859</td> <td>22.936601</td> <td>13.235777</td> </tr> <tr> <th>Stream 2</th> <td>25.718500</td> <td>6.140657</td> <td>8.856586</td> <td>14.761290</td> <td>11.870351</td> <td>7.728217</td> <td>13.882613</td> <td>29.328859</td> </tr> <tr> <th>Stream 3</th> <td>15.896774</td> <td>8.631035</td> <td>15.742406</td> <td>20.621604</td> <td>13.370582</td> <td>5.536313</td> <td>14.677463</td> <td>14.772753</td> </tr> <tr> <th>Stream 4</th> <td>22.458327</td> <td>5.319241</td> <td>11.973431</td> <td>22.344017</td> <td>11.534642</td> <td>2.402683</td> <td>24.214115</td> <td>16.236299</td> </tr> <tr> <th>Stream 5</th> <td>13.407745</td> <td>5.413278</td> <td>8.800650</td> <td>18.055743</td> <td>17.528827</td> <td>4.173171</td> <td>15.927165</td> <td>21.636801</td> </tr> <tr> <th>Stream 6</th> <td>8.069721</td> <td>5.531066</td> <td>13.233927</td> <td>21.321389</td> <td>7.622026</td> <td>12.064182</td> <td>11.457848</td> <td>12.342336</td> </tr> <tr> <th>Min Qi</th> <td>8.069721</td> <td>3.894254</td> <td>8.664394</td> <td>14.761290</td> <td>7.622026</td> <td>2.402683</td> <td>11.457848</td> <td>12.342336</td> </tr> <tr> <th>Max Qi</th> <td>27.058060</td> <td>8.631035</td> <td>15.742406</td> <td>25.315007</td> <td>17.528827</td> <td>12.064182</td> <td>24.214115</td> <td>29.328859</td> </tr> <tr> <th>Avg Qi</th> <td>18.768188</td> <td>5.821588</td> <td>11.211899</td> <td>20.403175</td> <td>12.307949</td> <td>5.911071</td> <td>17.182634</td> <td>17.925471</td> </tr> </table> <h2>300G run 2</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>September 25, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>300</td> </tr> <tr> <th style="text-align:right">Start of Database Load</th> <td>09/25/2014 16:38:20</td> </tr> <tr> <th style="text-align:right">End of Database Load</th> <td>09/25/2014 18:32:06</td> </tr> <tr> <th style="text-align:right">Database Load Time</th> <td>1:53:46</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>6</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td>423,431.8</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td>387,248.6</td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@300GB)</th> <td>404,936.3</td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td>368.236000 seconds</td> </tr> </table> <h3>Duration of stream execution:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>09/25/2014 19:28:42</td> <td>09/25/2014 19:29:58</td> <td>0:01:16</td> </tr> <tr> <th>Stream 1</th> <td>09/25/2014 19:30:00</td> <td>09/25/2014 19:36:04</td> <td>0:06:04</td> </tr> <tr> <th>Stream 2</th> <td>09/25/2014 19:30:00</td> <td>09/25/2014 19:36:00</td> <td>0:06:00</td> </tr> <tr> <th>Stream 3</th> <td>09/25/2014 19:30:00</td> <td>09/25/2014 19:36:06</td> <td>0:06:06</td> </tr> <tr> <th>Stream 4</th> <td>09/25/2014 19:30:00</td> <td>09/25/2014 19:36:07</td> <td>0:06:07</td> </tr> <tr> <th>Stream 5</th> <td>09/25/2014 19:30:00</td> <td>09/25/2014 19:35:53</td> <td>0:05:53</td> </tr> <tr> <th>Stream 6</th> <td>09/25/2014 19:30:00</td> <td>09/25/2014 19:36:08</td> <td>0:06:08</td> </tr> <tr> <th>Refresh 0</th> <td>09/25/2014 19:28:41</td> <td>09/25/2014 19:28:46</td> <td>0:00:05</td> </tr> <tr> <th> </th> <td>09/25/2014 19:29:58</td> <td>09/25/2014 19:30:00</td> <td>0:00:02</td> </tr> <tr> <th>Refresh 1</th> <td>09/25/2014 19:32:23</td> <td>09/25/2014 19:32:55</td> <td>0:00:32</td> </tr> <tr> <th>Refresh 2</th> <td>09/25/2014 19:30:00</td> <td>09/25/2014 19:30:31</td> <td>0:00:31</td> </tr> <tr> <th>Refresh 3</th> <td>09/25/2014 19:30:31</td> <td>09/25/2014 19:31:00</td> <td>0:00:29</td> </tr> <tr> <th>Refresh 4</th> <td>09/25/2014 19:31:01</td> <td>09/25/2014 19:31:23</td> <td>0:00:22</td> </tr> <tr> <th>Refresh 5</th> <td>09/25/2014 19:31:23</td> <td>09/25/2014 19:31:54</td> <td>0:00:31</td> </tr> <tr> <th>Refresh 6</th> <td>09/25/2014 19:31:55</td> <td>09/25/2014 19:32:23</td> <td>0:00:28</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> <th>Stream 0</th> <td>4.197427</td> <td>1.011516</td> <td>2.535959</td> <td>0.858781</td> <td>2.857279</td> <td>1.293530</td> <td>2.682266</td> <td>2.260502</td> </tr> <tr> <th>Stream 1</th> <td>15.467757</td> <td>3.517499</td> <td>13.820864</td> <td>4.157259</td> <td>13.141556</td> <td>10.902710</td> <td>16.899687</td> <td>8.986535</td> </tr> <tr> <th>Stream 2</th> <td>15.639991</td> <td>6.026485</td> <td>13.521624</td> <td>3.918031</td> <td>17.336458</td> <td>1.975310</td> <td>9.718194</td> <td>15.165247</td> </tr> <tr> <th>Stream 3</th> <td>14.891929</td> <td>4.481383</td> <td>15.322621</td> <td>5.272911</td> <td>15.266543</td> <td>6.771253</td> <td>13.430646</td> <td>20.171084</td> </tr> <tr> <th>Stream 4</th> <td>14.560526</td> <td>2.464157</td> <td>11.567112</td> <td>5.526629</td> <td>20.531540</td> <td>5.225971</td> <td>16.288606</td> <td>17.209475</td> </tr> <tr> <th>Stream 5</th> <td>10.390577</td> <td>3.549165</td> <td>9.598328</td> <td>8.783847</td> <td>17.351211</td> <td>6.308214</td> <td>12.606512</td> <td>13.035716</td> </tr> <tr> <th>Stream 6</th> <td>16.275922</td> <td>4.086475</td> <td>14.109963</td> <td>4.385887</td> <td>10.174709</td> <td>6.703266</td> <td>8.936217</td> <td>16.798526</td> </tr> <tr> <th>Min Qi</th> <td>10.390577</td> <td>2.464157</td> <td>9.598328</td> <td>3.918031</td> <td>10.174709</td> <td>1.975310</td> <td>8.936217</td> <td>8.986535</td> </tr> <tr> <th>Max Qi</th> <td>16.275922</td> <td>6.026485</td> <td>15.322621</td> <td>8.783847</td> <td>20.531540</td> <td>10.902710</td> <td>16.899687</td> <td>20.171084</td> </tr> <tr> <th>Avg Qi</th> <td>14.537784</td> <td>4.020861</td> <td>12.990085</td> <td>5.340761</td> <td>15.633670</td> <td>6.314454</td> <td>12.979977</td> <td>15.227764</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>8.300092</td> <td>2.598145</td> <td>5.168418</td> <td>1.619399</td> <td>11.958836</td> <td>3.191672</td> <td>3.097822</td> <td>2.497410</td> </tr> <tr> <th>Stream 1</th> <td>26.412829</td> <td>17.354745</td> <td>12.942454</td> <td>8.169447</td> <td>58.600101</td> <td>15.227942</td> <td>32.985324</td> <td>13.914978</td> </tr> <tr> <th>Stream 2</th> <td>34.523245</td> <td>17.635531</td> <td>15.193748</td> <td>8.435375</td> <td>62.442800</td> <td>16.276300</td> <td>26.533303</td> <td>12.414575</td> </tr> <tr> <th>Stream 3</th> <td>25.334301</td> <td>18.595422</td> <td>11.663933</td> <td>10.029387</td> <td>63.664992</td> <td>20.378320</td> <td>24.760768</td> <td>15.710589</td> </tr> <tr> <th>Stream 4</th> <td>36.971957</td> <td>15.645673</td> <td>14.672851</td> <td>13.196301</td> <td>58.214728</td> <td>17.375053</td> <td>26.581101</td> <td>11.624989</td> </tr> <tr> <th>Stream 5</th> <td>30.891797</td> <td>12.993365</td> <td>14.089049</td> <td>10.515091</td> <td>65.232712</td> <td>20.807026</td> <td>26.920526</td> <td>11.362095</td> </tr> <tr> <th>Stream 6</th> <td>38.143281</td> <td>21.106772</td> <td>15.152299</td> <td>18.845766</td> <td>66.240343</td> <td>12.295624</td> <td>22.510610</td> <td>18.081103</td> </tr> <tr> <th>Min Qi</th> <td>25.334301</td> <td>12.993365</td> <td>11.663933</td> <td>8.169447</td> <td>58.214728</td> <td>12.295624</td> <td>22.510610</td> <td>11.362095</td> </tr> <tr> <th>Max Qi</th> <td>38.143281</td> <td>21.106772</td> <td>15.193748</td> <td>18.845766</td> <td>66.240343</td> <td>20.807026</td> <td>32.985324</td> <td>18.081103</td> </tr> <tr> <th>Avg Qi</th> <td>32.046235</td> <td>17.221918</td> <td>13.952389</td> <td>11.531894</td> <td>62.399279</td> <td>17.060044</td> <td>26.715272</td> <td>13.851388</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>4.016212</td> <td>1.603004</td> <td>1.836489</td> <td>3.542383</td> <td>3.901876</td> <td>0.515102</td> <td>4.759612</td> <td>2.358873</td> </tr> <tr> <th>Stream 1</th> <td>22.162387</td> <td>10.067834</td> <td>15.772705</td> <td>22.091355</td> <td>12.974776</td> <td>8.354196</td> <td>19.342171</td> <td>12.771250</td> </tr> <tr> <th>Stream 2</th> <td>25.647926</td> <td>4.263008</td> <td>11.590737</td> <td>19.179326</td> <td>17.899770</td> <td>4.137031</td> <td>15.720245</td> <td>14.719776</td> </tr> <tr> <th>Stream 3</th> <td>14.511279</td> <td>7.484608</td> <td>20.735250</td> <td>13.041037</td> <td>17.139046</td> <td>6.014141</td> <td>16.234122</td> <td>13.454647</td> </tr> <tr> <th>Stream 4</th> <td>19.297494</td> <td>10.110707</td> <td>10.907458</td> <td>19.649066</td> <td>15.206251</td> <td>3.423503</td> <td>11.268082</td> <td>11.852223</td> </tr> <tr> <th>Stream 5</th> <td>17.445165</td> <td>5.582309</td> <td>15.266324</td> <td>19.788382</td> <td>14.245770</td> <td>2.810949</td> <td>16.601461</td> <td>14.019717</td> </tr> <tr> <th>Stream 6</th> <td>25.115339</td> <td>6.896503</td> <td>11.661563</td> <td>21.900028</td> <td>5.520025</td> <td>3.093050</td> <td>15.436258</td> <td>13.353446</td> </tr> <tr> <th>Min Qi</th> <td>14.511279</td> <td>4.263008</td> <td>10.907458</td> <td>13.041037</td> <td>5.520025</td> <td>2.810949</td> <td>11.268082</td> <td>11.852223</td> </tr> <tr> <th>Max Qi</th> <td>25.647926</td> <td>10.110707</td> <td>20.735250</td> <td>22.091355</td> <td>17.899770</td> <td>8.354196</td> <td>19.342171</td> <td>14.719776</td> </tr> <tr> <th>Avg Qi</th> <td>20.696598</td> <td>7.400828</td> <td>14.322339</td> <td>19.274866</td> <td>13.830940</td> <td>4.638812</td> <td>15.767057</td> <td>13.361843</td> </tr> </table> <h2>300G run 3:</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>September 25, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>300</td> </tr> <tr> <th style="text-align:right">Total Data Storage/Database Size</th> <td>258,888M</td> </tr> <tr> <th style="text-align:right">Start of Database Load</th> <td>09/25/2014 16:38:20</td> </tr> <tr> <th style="text-align:right">End of Database Load</th> <td>09/25/2014 18:32:06</td> </tr> <tr> <th style="text-align:right">Database Load Time</th> <td>1:53:46</td> </tr> <tr> <th style="text-align:right">Query Streams for <br />Throughput Test</th> <td>6</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td>417,672.0</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td>389,719.5</td> </tr> <tr> <th style="text-align:right">Virt-H Composite <br />Query-per-Hour Metric <br />(Qph@300GB)</th> <td>403,453.7</td> </tr> <tr> <th style="text-align:right">Measurement Interval in <br />Throughput Test (Ts)</th> <td>365.902000 seconds</td> </tr> </table> <h3>Duration of stream execution:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>09/25/2014 19:36:11</td> <td>09/25/2014 19:37:29</td> <td>0:01:18</td> </tr> <tr> <th>Stream 1</th> <td>09/25/2014 19:37:32</td> <td>09/25/2014 19:43:13</td> <td>0:05:41</td> </tr> <tr> <th>Stream 2</th> <td>09/25/2014 19:37:32</td> <td>09/25/2014 19:43:31</td> <td>0:05:59</td> </tr> <tr> <th>Stream 3</th> <td>09/25/2014 19:37:32</td> <td>09/25/2014 19:43:37</td> <td>0:06:05</td> </tr> <tr> <th>Stream 4</th> <td>09/25/2014 19:37:32</td> <td>09/25/2014 19:43:33</td> <td>0:06:01</td> </tr> <tr> <th>Stream 5</th> <td>09/25/2014 19:37:32</td> <td>09/25/2014 19:43:32</td> <td>0:06:00</td> </tr> <tr> <th>Stream 6</th> <td>09/25/2014 19:37:32</td> <td>09/25/2014 19:43:37</td> <td>0:06:05</td> </tr> <tr> <th>Refresh 0</th> <td>09/25/2014 19:36:12</td> <td>09/25/2014 19:36:16</td> <td>0:00:04</td> </tr> <tr> <th> </th> <td>09/25/2014 19:37:29</td> <td>09/25/2014 19:37:31</td> <td>0:00:02</td> </tr> <tr> <th>Refresh 1</th> <td>09/25/2014 19:40:02</td> <td>09/25/2014 19:40:33</td> <td>0:00:31</td> </tr> <tr> <th>Refresh 2</th> <td>09/25/2014 19:37:31</td> <td>09/25/2014 19:38:01</td> <td>0:00:30</td> </tr> <tr> <th>Refresh 3</th> <td>09/25/2014 19:38:01</td> <td>09/25/2014 19:38:30</td> <td>0:00:29</td> </tr> <tr> <th>Refresh 4</th> <td>09/25/2014 19:38:30</td> <td>09/25/2014 19:38:58</td> <td>0:00:28</td> </tr> <tr> <th>Refresh 5</th> <td>09/25/2014 19:38:58</td> <td>09/25/2014 19:39:27</td> <td>0:00:29</td> </tr> <tr> <th>Refresh 6</th> <td>09/25/2014 19:39:27</td> <td>09/25/2014 19:40:01</td> <td>0:00:34</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> <th>Stream 0</th> <td>4.305006</td> <td>1.083442</td> <td>2.502758</td> <td>0.845763</td> <td>2.840824</td> <td>1.346166</td> <td>2.659511</td> <td>2.233550</td> </tr> <tr> <th>Stream 1</th> <td>11.513360</td> <td>3.732513</td> <td>14.530428</td> <td>3.819517</td> <td>14.821291</td> <td>7.561547</td> <td>10.435082</td> <td>8.984230</td> </tr> <tr> <th>Stream 2</th> <td>13.486433</td> <td>3.373689</td> <td>9.620363</td> <td>3.914320</td> <td>16.857542</td> <td>5.837487</td> <td>10.695443</td> <td>17.901191</td> </tr> <tr> <th>Stream 3</th> <td>11.015942</td> <td>1.780220</td> <td>4.830412</td> <td>9.073543</td> <td>15.587709</td> <td>9.661989</td> <td>12.374931</td> <td>15.262485</td> </tr> <tr> <th>Stream 4</th> <td>13.600461</td> <td>0.820899</td> <td>12.254226</td> <td>7.799415</td> <td>19.860761</td> <td>13.145017</td> <td>14.404345</td> <td>11.807583</td> </tr> <tr> <th>Stream 5</th> <td>13.358000</td> <td>3.885118</td> <td>11.099935</td> <td>4.845043</td> <td>18.286721</td> <td>6.424272</td> <td>9.735255</td> <td>15.041608</td> </tr> <tr> <th>Stream 6</th> <td>13.588873</td> <td>3.789631</td> <td>13.503399</td> <td>5.130389</td> <td>13.104065</td> <td>3.517076</td> <td>14.929079</td> <td>19.831639</td> </tr> <tr> <th>Min Qi</th> <td>11.015942</td> <td>0.820899</td> <td>4.830412</td> <td>3.819517</td> <td>13.104065</td> <td>3.517076</td> <td>9.735255</td> <td>8.984230</td> </tr> <tr> <th>Max Qi</th> <td>13.600461</td> <td>3.885118</td> <td>14.530428</td> <td>9.073543</td> <td>19.860761</td> <td>13.145017</td> <td>14.929079</td> <td>19.831639</td> </tr> <tr> <th>Avg Qi</th> <td>12.760511</td> <td>2.897012</td> <td>10.973127</td> <td>5.763705</td> <td>16.419681</td> <td>7.691231</td> <td>12.095689</td> <td>14.804789</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>8.553183</td> <td>3.215484</td> <td>4.652364</td> <td>1.620089</td> <td>11.936052</td> <td>2.916132</td> <td>3.219969</td> <td>2.374276</td> </tr> <tr> <th>Stream 1</th> <td>29.441108</td> <td>20.348266</td> <td>9.994556</td> <td>14.965432</td> <td>60.537168</td> <td>13.302875</td> <td>30.159402</td> <td>10.277570</td> </tr> <tr> <th>Stream 2</th> <td>41.799347</td> <td>18.197400</td> <td>16.773638</td> <td>6.510347</td> <td>67.461446</td> <td>20.362328</td> <td>0.109929</td> <td>9.908769</td> </tr> <tr> <th>Stream 3</th> <td>24.306937</td> <td>20.555376</td> <td>17.140758</td> <td>16.715188</td> <td>61.724168</td> <td>22.469230</td> <td>27.967206</td> <td>13.434167</td> </tr> <tr> <th>Stream 4</th> <td>34.820796</td> <td>11.795664</td> <td>18.015120</td> <td>7.176057</td> <td>63.134711</td> <td>11.427374</td> <td>23.959842</td> <td>16.759246</td> </tr> <tr> <th>Stream 5</th> <td>23.139366</td> <td>12.655317</td> <td>13.152401</td> <td>7.258740</td> <td>64.273225</td> <td>22.854106</td> <td>28.803059</td> <td>12.832364</td> </tr> <tr> <th>Stream 6</th> <td>27.955059</td> <td>24.633526</td> <td>11.046285</td> <td>5.995041</td> <td>74.965966</td> <td>15.636579</td> <td>22.803890</td> <td>13.221303</td> </tr> <tr> <th>Min Qi</th> <td>23.139366</td> <td>11.795664</td> <td>9.994556</td> <td>5.995041</td> <td>60.537168</td> <td>11.427374</td> <td>0.109929</td> <td>9.908769</td> </tr> <tr> <th>Max Qi</th> <td>41.799347</td> <td>24.633526</td> <td>18.015120</td> <td>16.715188</td> <td>74.965966</td> <td>22.854106</td> <td>30.159402</td> <td>16.759246</td> </tr> <tr> <th>Avg Qi</th> <td>30.243769</td> <td>18.030925</td> <td>14.353793</td> <td>9.770134</td> <td>65.349447</td> <td>17.675415</td> <td>22.300555</td> <td>12.738903</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>4.298092</td> <td>1.702071</td> <td>1.894548</td> <td>4.118591</td> <td>3.922889</td> <td>0.491145</td> <td>4.519734</td> <td>2.347913</td> </tr> <tr> <th>Stream 1</th> <td>16.432222</td> <td>6.908918</td> <td>17.749058</td> <td>18.756674</td> <td>11.148628</td> <td>5.464975</td> <td>18.300673</td> <td>12.972871</td> </tr> <tr> <th>Stream 2</th> <td>20.588544</td> <td>4.387662</td> <td>14.527229</td> <td>23.844364</td> <td>15.500462</td> <td>15.543458</td> <td>13.666574</td> <td>15.240662</td> </tr> <tr> <th>Stream 3</th> <td>14.008049</td> <td>6.222633</td> <td>12.833421</td> <td>22.811602</td> <td>16.013232</td> <td>9.449069</td> <td>16.486111</td> <td>12.974515</td> </tr> <tr> <th>Stream 4</th> <td>16.964699</td> <td>8.106044</td> <td>11.207675</td> <td>22.483826</td> <td>17.354675</td> <td>4.641183</td> <td>14.583941</td> <td>13.679087</td> </tr> <tr> <th>Stream 5</th> <td>25.243144</td> <td>7.359437</td> <td>16.986615</td> <td>19.855391</td> <td>17.183725</td> <td>5.750937</td> <td>14.759597</td> <td>13.052316</td> </tr> <tr> <th>Stream 6</th> <td>12.986721</td> <td>10.160993</td> <td>17.496662</td> <td>19.267026</td> <td>17.300224</td> <td>4.955930</td> <td>19.267721</td> <td>15.421241</td> </tr> <tr> <th>Min Qi</th> <td>12.986721</td> <td>4.387662</td> <td>11.207675</td> <td>18.756674</td> <td>11.148628</td> <td>4.641183</td> <td>13.666574</td> <td>12.972871</td> </tr> <tr> <th>Max Qi</th> <td>25.243144</td> <td>10.160993</td> <td>17.749058</td> <td>23.844364</td> <td>17.354675</td> <td>15.543458</td> <td>19.267721</td> <td>15.421241</td> </tr> <tr> <th>Avg Qi</th> <td>17.703896</td> <td>7.190948</td> <td>15.133443</td> <td>21.169814</td> <td>15.750158</td> <td>7.634259</td> <td>16.177436</td> <td>13.890115</td> </tr> </table> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac349bc698"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac347508c8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac347b2098"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac34f71318"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac346fa748"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac349de568"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aac34725b28"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aac36adde48"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aac34a7e368"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac348581b8"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac344340c8"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac34ee7058"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac34d388b8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x69cacf8"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aabeaafeba8"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x1d5023a8"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aabd131ab08"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:09:30.150737-04:00
In Hoc Signo Vinces (part 18 of n): Cluster Dynamics
http://www.openlinksw.com:443/weblog/oerling/?date=2014-09-26#1819
2014-09-26T21:02:58Z
<p>This article is about how scale-out differs from single-server. This shows large effects of parameters whose very existence most would not anticipate, and some low level metrics for assessing these. The moral of the story is that this is the stuff which makes the difference between merely surviving scale-out and winning with it. The developer and DBA would not normally know about this; thus these things fall into the category of adaptive self-configuration expected from the DBMS. But since this series is about what makes performance, I will discuss the dynamics such as they are and how to play these.</p> <p>We take the prototypical cross partition join in Q13: Make a hash table of all <code>customers</code>, partitioned by <code>c_custkey</code>. This is independently done with full parallelism in each partition. Scan the <code>orders</code>, get the <code>customer</code> (in a different partition), and flag the <code>customers</code> that had at least one <code>order</code>. Then, to get the <code>customers</code> with no <code>orders</code>, return the <code>customers</code> that were not flagged in the previous pass.</p> <p>The single-server time in part 12 was 7.8 and 6.0 with a single user. We consider the better of the times. The difference is due to allocating memory on the first go; on the second go the memory is already in reserve.</p> <p>With default settings, we get 4595 ms (microseconds), with per node resource utilization at:</p> <blockquote> <pre><code> Cluster 4 nodes, 4 s. 112405 m/s 742602 KB/s 2749% cpu 0% read 4% clw threads 1r 0w 0i buffers 8577766 287874 d 0 w 0 pfs cl 1: 27867 m/s 185654 KB/s 733% cpu 0% read 4% clw threads 1r 0w 0i buffers 2144242 71757 d 0 w 0 pfs cl 2: 28149 m/s 185372 KB/s 672% cpu 0% read 0% clw threads 0r 0w 0i buffers 2144640 71903 d 0 w 0 pfs cl 3: 28220 m/s 185621 KB/s 675% cpu 0% read 0% clw threads 0r 0w 0i buffers 2144454 71962 d 0 w 0 pfs cl 4: 28150 m/s 185837 KB/s 667% cpu 0% read 0% clw threads 0r 0w 0i buffers 2144430 72252 d 0 w 0 pfs </code> </pre></blockquote> <p>The top line is the summary; the lines below are per-process. The <code>m/s</code> is messages-per-second; <code>KB/s</code> is interconnect traffic per second; <code>clw %</code> is idle time spent waiting for a reply from another process. The cluster is set up with 4 processes across 2 machines, each with 2 NUMA nodes. Each process has affinity to the NUMA node, so local memory only. The time is reasonable in light of the overall CPU of 2700%. The maximum would be 4800% with all threads of all cores busy all the time.</p> <p>The catch here is that we do not have a steady half-platform utilization all the time, but full platform peaks followed by synchronization barriers with very low utilization. So, we set the batch size differently:</p> <blockquote> <pre><code> cl_exec ('__dbf_set (''cl_dfg_batch_bytes'', 50000000)'); </code> </pre></blockquote> <p>This means that we set, on each process, the <code>cl_dfg_batch_bytes</code> to 50M from a default of 10M. The effect is that each scan of <code>orders</code>, one thread per slice, 48 slices total, will produce 50MB worth of <code>o_custkeys</code> to be sent to the other partition for getting the <code>customer</code>. After each 50M, the thread stops and will produce the next batch when all are done and a global continue message is sent by the coordinator.</p> <p>The time is now 3173 ms with:</p> <blockquote> <pre><code> Cluster 4 nodes, 3 s. 158220 m/s 1054944 KB/s 3676% cpu 0% read 1% clw threads 1r 0w 0i buffers 8577766 287874 d 0 w 0 pfs cl 1: 39594 m/s 263962 KB/s 947% cpu 0% read 1% clw threads 1r 0w 0i buffers 2144242 71757 d 0 w 0 pfs cl 2: 39531 m/s 263476 KB/s 894% cpu 0% read 0% clw threads 0r 0w 0i buffers 2144640 71903 d 0 w 0 pfs cl 3: 39523 m/s 263684 KB/s 933% cpu 0% read 0% clw threads 0r 0w 0i buffers 2144454 71962 d 0 w 0 pfs cl 4: 39535 m/s 263586 KB/s 900% cpu 0% read 0% clw threads 0r 0w 0i buffers 2144430 72252 d 0 w 0 pfs </code> </pre></blockquote> <p>The platform utilization is better as we see. The throughput is nearly double that of the single-server, which is pretty good for a communication-heavy query.</p> <p>This was done with a vector size of 10K. In other words, each partition gets 10K <code>o_custkeys</code> and splits these 48 ways to go to every recipient. 1/4 are in the same process, 1/4 in a different process on the same machine, and 2/4 on a different machine. The recipient gets messages with an average of 208 <code>o_custkey</code> values, puts them back together in batches of 10K, and passes these to the hash join with <code>customer</code>.</p> <p>We try different vector sizes, such as 100K: </p> <blockquote> <pre><code> cl_exec ('__dbf_set (''dc_batch_sz'', 100000)'); </code> </pre></blockquote> <p>There are two metrics of interest here: The write block time, and the scheduling overhead. The write block time is microseconds, which increases whenever a thread must wait before it can write to a connection. The scheduling overhead is cumulative clocks spent by threads while waiting for a critical section that deals with dispatching messages to consumer threads. Long messages make blocking; short messages make frequent scheduling decisions.</p> <blockquote> <pre><code> SELECT cl_sys_stat ('local_cll_clk', clr=>1), cl_sys_stat ('write_block_usec', clr=>1) ; </code> </pre></blockquote> <p> <code>cl_sys_stat</code> gets the counters from all processes and returns the sum. <code>clr=>1</code> means that the counter is cleared after read.</p> <p>We do Q13 with vector sizes of 10, 100, and 1000K.</p> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th>Vector size</th> <th>msec</th> <th>mtx</th> <th>wblock</th> </tr> <tr> <th>10K</th> <td style="text-align:right">3297</td> <td style="text-align:right">10,829,910,329</td> <td style="text-align:right">0</td> </tr> <tr> <th>100K</th> <td style="text-align:right">3150</td> <td style="text-align:right">1,663,238,367</td> <td style="text-align:right">59,132</td> </tr> <tr> <th>1000K</th> <td style="text-align:right">3876</td> <td style="text-align:right">414,631,129</td> <td style="text-align:right">4,578,003</td> </tr> </table> <p>So, 100K seems to strike the best balance between scheduling and blocking on write.</p> <p>The times are measured after several samples with each setting. The times stabilize after a few runs, as the appropriate size memory blocks are in reserve. Calling <code>mmap</code> to allocate these on the first run with each size has a very high penalty, e.g., 60s for the first run with 1M vector size. We note that blocking on write is really bad even though 1/3 of the time there is no network and 2/3 of the time there is a fast network (QDR IB) with no other load. Further, the affinities are set so that the thread responsible for incoming messages is always on core. Result variability on consecutive runs is under 5%, which is similar to single-server behavior.</p> <p>It would seem that a mutex, as bad as it is, is still better than a distributed cause for going off core (blocking on write). The latency for continuing a thread thus blocked is of course higher than the latency for continuing one that is waiting for a mutex.</p> <p>We note that a cluster with more machines can take a longer vector size because a vector spreads out to more recipients. The key seems to be to set the message size so that blocking on write is not common. This is a possible adaptive execution feature. We have seen no particular benefit from SDP (Sockets Direct Protocol) and its zero copy. This is a TCP replacement that comes with the InfiniBand drivers.</p> <p>We will next look at replication/partitioning tradeoffs for hash joins. Then we can look at full runs.</p> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac349bc698"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac347508c8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac347b2098"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac34f71318"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac346fa748"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac349de568"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aac34725b28"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aac36adde48"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aac34a7e368"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac348581b8"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac344340c8"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac34ee7058"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac34d388b8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x69cacf8"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aac00620518"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> In Hoc Signo Vinces (part 18): Cluster Dynamics<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aaca8027e88"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab5182ee48"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aabf845ad68"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:06:09.183299-04:00
In Hoc Signo Vinces (part 17 of n): 100G and 300G Runs on Dual Xeon E5 2650v2
http://www.openlinksw.com:443/weblog/oerling/?date=2014-09-26#1818
2014-09-26T21:02:54Z
<p>This is an update presenting sample results on a newer platform for a single-server configuration. This is to verify that performance scales with the addition of cores and clock speed. Further, we note that the jump from 100G to 300G changes very little about the score. 3x larger takes approximately 3x longer, as long as things are in memory.</p> <p>The platform is one node of the <a href="http://www.cwi.nl/" id="link-id0x2aac002989b8">CWI</a> <a href="http://scilens.project.cwi.nl/" id="link-id0x2aabe8ca0848">cluster</a> which was also used for the <a href="http://www.openlinksw.com/weblog/oerling/?id=1804" id="link-id0x2aac001ada18">500Gt RDF experiments reported on this blog</a>. The specification is dual Xeon E5 2650v2 (8 core, 16 thread, 2.6 GHz) with 256 GB RAM. The disk setup is a RAID-0 of three 2 TB rotating disks.</p> <p>For the 100G, we go from 240 to 395, which is about 1.64x. The new platform has 16 vs 12 cores and a clock of 2.6 as opposed to 2.3. This makes a multiplier of 1.5. The rest of the acceleration is probably attributable to faster memory clock. Anyway, the point of more speed from larger platform is made.</p> <p>The top level scores per run are as follows; the numerical quantities summaries are appended.</p> <h3>100G</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th>Run</th> <th>Power</th> <th>Throughput</th> <th>Composite</th> </tr> <tr> <th>Run 1</th> <td style="text-align:right">391,000.1</td> <td style="text-align:right">401,029.4</td> <td style="text-align:right">395,983.0</td> </tr> <tr> <th>Run 2</th> <td style="text-align:right">388,746.2</td> <td style="text-align:right">404,189.3</td> <td style="text-align:right">396,392.6</td> </tr> </table> <h3>300G</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th>Run</th> <th>Power</th> <th>Throughput</th> <th>Composite </th> </tr> <tr> <th>Run 1</th> <td style="text-align:right">61,988.7</td> <td style="text-align:right">384,883.7</td> <td style="text-align:right">154,461.6</td> </tr> <tr> <th>Run 2</th> <td style="text-align:right">423,431.8</td> <td style="text-align:right">387,248.6</td> <td style="text-align:right">404,936.3</td> </tr> <tr> <th>Run 3</th> <td style="text-align:right">417,672.0</td> <td style="text-align:right">389,719.5</td> <td style="text-align:right">403,453.7</td> </tr> </table> <p>The interested may reproduce the results using the <a href="https://github.com/v7fasttrack/virtuoso-opensource/tree/feature/analytics" id="link-id0x2aabebc12d88">feature/analytics branch</a> of the <a href="https://github.com/v7fasttrack" id="link-id0x2aabe9d1ec68">v7fasttrack git repository</a> on <a href="https://github.com/" id="link-id0x2aabe83bbb28">GitHub</a> as described in <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aabe9a230c8">Part 13</a>.</p> <p>For the 300G runs, we note a much longer load time; see below, as this is seriously IO bound.</p> <p>The first power test at 300G is a non-starter, even though this comes right after bulk load. Still, the data is not in working set and getting it from disk is simply an automatic disqualification, unless maybe one had 300 separate disks. This happens in TPC benchmarks, but not very often in the field. Looking at the first power run, the first queries take the longest, but by the time the power run starts, the working set is there. By an artifact of the metric (use of geometric mean for the power test), long queries are penalized less there than in the throughput run.</p> <p>So, we run 3 executions instead of the prescribed 2, to have 2 executions from warm state.</p> <p>To do 300G well in 256 GB of RAM, one needs either to use several SSDs, or to increase compression and keep all in memory, so no secondary storage at all. In order to keep all in memory, one could have stream-compression on string columns. Stream-compressing strings (e.g., <code>o_comment</code>, <code>l_comment</code>) does not pay if one is already in memory, but if stream-compressing strings eliminates going to secondary storage, then the win is sure.</p> <p>As before, all caveats apply; the results are unaudited and for information only. Therefore we do not use the official metric name.</p> <h2>100G Run 1</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>September 15, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>100</td> </tr> <tr> <th style="text-align:right">Start of Database Load</th> <td>09/15/2014 07:04:08</td> </tr> <tr> <th style="text-align:right">End of Database Load</th> <td>09/15/2014 07:15:58</td> </tr> <tr> <th style="text-align:right">Database Load Time</th> <td>0:11:50</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>5</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td>391,000.1</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td>401,029.4</td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@100GB)</th> <td>395,983.0</td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td>98.846000 seconds</td> </tr> </table> <h3>Duration of stream execution</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>09/15/2014 13:13:01</td> <td>09/15/2014 13:13:28</td> <td style="text-align:right">0:00:27</td> </tr> <tr> <th>Stream 1</th> <td>09/15/2014 13:13:29</td> <td>09/15/2014 13:15:06</td> <td style="text-align:right">0:01:37</td> </tr> <tr> <th>Stream 2</th> <td>09/15/2014 13:13:29</td> <td>09/15/2014 13:15:07</td> <td style="text-align:right">0:01:38</td> </tr> <tr> <th>Stream 3</th> <td>09/15/2014 13:13:29</td> <td>09/15/2014 13:15:07</td> <td style="text-align:right">0:01:38</td> </tr> <tr> <th>Stream 4</th> <td>09/15/2014 13:13:29</td> <td>09/15/2014 13:15:04</td> <td style="text-align:right">0:01:35</td> </tr> <tr> <th>Stream 5</th> <td>09/15/2014 13:13:29</td> <td>09/15/2014 13:15:08</td> <td style="text-align:right">0:01:39</td> </tr> <tr> <th>Refresh 0</th> <td>09/15/2014 13:13:01</td> <td>09/15/2014 13:13:03</td> <td style="text-align:right">0:00:02</td> </tr> <tr> <th> </th> <td>09/15/2014 13:13:28</td> <td>09/15/2014 13:13:29</td> <td style="text-align:right">0:00:01</td> </tr> <tr> <th>Refresh 1</th> <td>09/15/2014 13:14:10</td> <td>09/15/2014 13:14:16</td> <td style="text-align:right">0:00:06</td> </tr> <tr> <th>Refresh 2</th> <td>09/15/2014 13:13:29</td> <td>09/15/2014 13:13:42</td> <td style="text-align:right">0:00:13</td> </tr> <tr> <th>Refresh 3</th> <td>09/15/2014 13:13:42</td> <td>09/15/2014 13:13:53</td> <td style="text-align:right">0:00:11</td> </tr> <tr> <th>Refresh 4</th> <td>09/15/2014 13:13:53</td> <td>09/15/2014 13:14:02</td> <td style="text-align:right">0:00:09</td> </tr> <tr> <th>Refresh 5</th> <td>09/15/2014 13:14:02</td> <td>09/15/2014 13:14:10</td> <td style="text-align:right">0:00:08</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> <th>Stream 0</th> <td>1.442477</td> <td>0.304513</td> <td>0.720263</td> <td>0.351285</td> <td>0.979414</td> <td>0.479455</td> <td>0.865992</td> <td>0.875236</td> </tr> <tr> <th>Stream 1</th> <td>3.938133</td> <td>0.920533</td> <td>3.738724</td> <td>2.769707</td> <td>3.209728</td> <td>1.339146</td> <td>2.759384</td> <td>3.626868</td> </tr> <tr> <th>Stream 2</th> <td>4.104738</td> <td>0.952245</td> <td>4.719658</td> <td>0.865586</td> <td>2.139267</td> <td>0.850909</td> <td>2.044402</td> <td>2.600373</td> </tr> <tr> <th>Stream 3</th> <td>3.692119</td> <td>1.024876</td> <td>3.430172</td> <td>1.579846</td> <td>4.097845</td> <td>1.859468</td> <td>2.312921</td> <td>6.238070</td> </tr> <tr> <th>Stream 4</th> <td>5.419537</td> <td>0.531571</td> <td>2.116176</td> <td>1.256836</td> <td>4.787617</td> <td>2.117995</td> <td>3.517466</td> <td>3.982180</td> </tr> <tr> <th>Stream 5</th> <td>5.167029</td> <td>0.746720</td> <td>3.157557</td> <td>1.255182</td> <td>3.004802</td> <td>2.131963</td> <td>3.648316</td> <td>2.835751</td> </tr> <tr> <th>Min Qi</th> <td>3.692119</td> <td>0.531571</td> <td>2.116176</td> <td>0.865586</td> <td>2.139267</td> <td>0.850909</td> <td>2.044402</td> <td>2.600373</td> </tr> <tr> <th>Max Qi</th> <td>5.419537</td> <td>1.024876</td> <td>4.719658</td> <td>2.769707</td> <td>4.787617</td> <td>2.131963</td> <td>3.648316</td> <td>6.238070</td> </tr> <tr> <th>Avg Qi</th> <td>4.464311</td> <td>0.835189</td> <td>3.432457</td> <td>1.545431</td> <td>3.447852</td> <td>1.659896</td> <td>2.856498</td> <td>3.856648</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>2.606044</td> <td>1.117063</td> <td>1.847930</td> <td>0.618534</td> <td>4.327600</td> <td>1.110908</td> <td>0.995289</td> <td>0.975910</td> </tr> <tr> <th>Stream 1</th> <td>7.463593</td> <td>4.686463</td> <td>4.549733</td> <td>4.168129</td> <td>15.759178</td> <td>5.247666</td> <td>4.495030</td> <td>4.075198</td> </tr> <tr> <th>Stream 2</th> <td>9.398552</td> <td>5.170904</td> <td>3.934405</td> <td>1.880683</td> <td>19.968787</td> <td>3.767992</td> <td>6.965337</td> <td>3.849845</td> </tr> <tr> <th>Stream 3</th> <td>7.581069</td> <td>4.109905</td> <td>4.301159</td> <td>2.123634</td> <td>17.683200</td> <td>5.383603</td> <td>4.376887</td> <td>2.854777</td> </tr> <tr> <th>Stream 4</th> <td>9.927887</td> <td>6.913209</td> <td>3.351489</td> <td>2.802724</td> <td>16.985827</td> <td>3.925148</td> <td>4.691474</td> <td>4.080586</td> </tr> <tr> <th>Stream 5</th> <td>7.035080</td> <td>3.921425</td> <td>6.844778</td> <td>2.899238</td> <td>14.839509</td> <td>4.986742</td> <td>6.629664</td> <td>4.089547</td> </tr> <tr> <th>Min Qi</th> <td>7.035080</td> <td>3.921425</td> <td>3.351489</td> <td>1.880683</td> <td>14.839509</td> <td>3.767992</td> <td>4.376887</td> <td>2.854777</td> </tr> <tr> <th>Max Qi</th> <td>9.927887</td> <td>6.913209</td> <td>6.844778</td> <td>4.168129</td> <td>19.968787</td> <td>5.383603</td> <td>6.965337</td> <td>4.089547</td> </tr> <tr> <th>Avg Qi</th> <td>8.281236</td> <td>4.960381</td> <td>4.596313</td> <td>2.774882</td> <td>17.047300</td> <td>4.662230</td> <td>5.431678</td> <td>3.789991</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>1.215956</td> <td>0.745257</td> <td>0.699801</td> <td>1.281834</td> <td>1.291110</td> <td>0.518425</td> <td>1.827192</td> <td>1.014431</td> </tr> <tr> <th>Stream 1</th> <td>5.779854</td> <td>2.383264</td> <td>2.396793</td> <td>6.130511</td> <td>5.002700</td> <td>1.968425</td> <td>4.172437</td> <td>2.427047</td> </tr> <tr> <th>Stream 2</th> <td>7.828176</td> <td>1.833416</td> <td>3.175649</td> <td>4.785709</td> <td>5.385834</td> <td>1.403290</td> <td>6.383005</td> <td>6.366525</td> </tr> <tr> <th>Stream 3</th> <td>5.880139</td> <td>1.797383</td> <td>3.258024</td> <td>5.601364</td> <td>6.373216</td> <td>1.977848</td> <td>5.235542</td> <td>6.385010</td> </tr> <tr> <th>Stream 4</th> <td>3.989621</td> <td>1.252891</td> <td>2.478303</td> <td>4.678629</td> <td>3.212176</td> <td>2.740586</td> <td>5.037995</td> <td>3.911379</td> </tr> <tr> <th>Stream 5</th> <td>5.030440</td> <td>2.010988</td> <td>4.188428</td> <td>6.221990</td> <td>5.418788</td> <td>2.187718</td> <td>3.589915</td> <td>3.517380</td> </tr> <tr> <th>Min Qi</th> <td>3.989621</td> <td>1.252891</td> <td>2.396793</td> <td>4.678629</td> <td>3.212176</td> <td>1.403290</td> <td>3.589915</td> <td>2.427047</td> </tr> <tr> <th>Max Qi</th> <td>7.828176</td> <td>2.383264</td> <td>4.188428</td> <td>6.221990</td> <td>6.373216</td> <td>2.740586</td> <td>6.383005</td> <td>6.385010</td> </tr> <tr> <th>Avg Qi</th> <td>5.701646</td> <td>1.855588</td> <td>3.099439</td> <td>5.483641</td> <td>5.078543</td> <td>2.055573</td> <td>4.883779</td> <td>4.521468</td> </tr> </table> <h2>100G Run 2</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>September 15, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>100</td> </tr> <tr> <th style="text-align:right">Total Data Storage/Database Size</th> <td>87,312M</td> </tr> <tr> <th style="text-align:right">Start of Database Load</th> <td>09/15/2014 07:04:08</td> </tr> <tr> <th style="text-align:right">End of Database Load</th> <td>09/15/2014 07:15:58</td> </tr> <tr> <th style="text-align:right">Database Load Time</th> <td>0:11:50</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>5</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td>388,746.2</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td>404,189.3</td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@100GB)</th> <td>396,392.6</td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td>98.074000 seconds</td> </tr> </table> <h3>Duration of stream execution:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>09/15/2014 13:15:11</td> <td>09/15/2014 13:15:38</td> <td>0:00:27</td> </tr> <tr> <th>Stream 1</th> <td>09/15/2014 13:15:39</td> <td>09/15/2014 13:17:13</td> <td>0:01:34</td> </tr> <tr> <th>Stream 2</th> <td>09/15/2014 13:15:39</td> <td>09/15/2014 13:17:16</td> <td>0:01:37</td> </tr> <tr> <th>Stream 3</th> <td>09/15/2014 13:15:39</td> <td>09/15/2014 13:17:15</td> <td>0:01:36</td> </tr> <tr> <th>Stream 4</th> <td>09/15/2014 13:15:39</td> <td>09/15/2014 13:17:17</td> <td>0:01:38</td> </tr> <tr> <th>Stream 5</th> <td>09/15/2014 13:15:39</td> <td>09/15/2014 13:17:15</td> <td>0:01:36</td> </tr> <tr> <th>Refresh 0</th> <td>09/15/2014 13:15:11</td> <td>09/15/2014 13:15:12</td> <td>0:00:01</td> </tr> <tr> <th> </th> <td>09/15/2014 13:15:38</td> <td>09/15/2014 13:15:39</td> <td>0:00:01</td> </tr> <tr> <th>Refresh 1</th> <td>09/15/2014 13:16:13</td> <td>09/15/2014 13:16:20</td> <td>0:00:07</td> </tr> <tr> <th>Refresh 2</th> <td>09/15/2014 13:15:39</td> <td>09/15/2014 13:15:47</td> <td>0:00:08</td> </tr> <tr> <th>Refresh 3</th> <td>09/15/2014 13:15:47</td> <td>09/15/2014 13:15:56</td> <td>0:00:09</td> </tr> <tr> <th>Refresh 4</th> <td>09/15/2014 13:15:56</td> <td>09/15/2014 13:16:03</td> <td>0:00:07</td> </tr> <tr> <th>Refresh 5</th> <td>09/15/2014 13:16:03</td> <td>09/15/2014 13:16:12</td> <td>0:00:09</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> <th>Stream 0</th> <td>1.467681</td> <td>0.277665</td> <td>0.766102</td> <td>0.365185</td> <td>0.941206</td> <td>0.549381</td> <td>0.938998</td> <td>0.803514</td> </tr> <tr> <th>Stream 1</th> <td>3.883169</td> <td>1.488521</td> <td>3.366920</td> <td>1.627478</td> <td>3.632321</td> <td>2.065565</td> <td>2.911138</td> <td>2.444544</td> </tr> <tr> <th>Stream 2</th> <td>3.294589</td> <td>1.138066</td> <td>3.260775</td> <td>1.899615</td> <td>5.367725</td> <td>1.820374</td> <td>3.655119</td> <td>2.186642</td> </tr> <tr> <th>Stream 3</th> <td>3.797641</td> <td>0.995877</td> <td>3.239690</td> <td>2.483035</td> <td>2.737690</td> <td>1.505998</td> <td>4.058083</td> <td>4.268644</td> </tr> <tr> <th>Stream 4</th> <td>4.099187</td> <td>0.402685</td> <td>4.704959</td> <td>1.469825</td> <td>5.367910</td> <td>2.783018</td> <td>2.706164</td> <td>2.551061</td> </tr> <tr> <th>Stream 5</th> <td>3.651273</td> <td>1.598314</td> <td>2.051899</td> <td>1.283754</td> <td>4.711897</td> <td>1.519763</td> <td>2.851300</td> <td>2.484093</td> </tr> <tr> <th>Min Qi</th> <td>3.294589</td> <td>0.402685</td> <td>2.051899</td> <td>1.283754</td> <td>2.737690</td> <td>1.505998</td> <td>2.706164</td> <td>2.186642</td> </tr> <tr> <th>Max Qi</th> <td>4.099187</td> <td>1.598314</td> <td>4.704959</td> <td>2.483035</td> <td>5.367910</td> <td>2.783018</td> <td>4.058083</td> <td>4.268644</td> </tr> <tr> <th>Avg Qi</th> <td>3.745172</td> <td>1.124693</td> <td>3.324849</td> <td>1.752741</td> <td>4.363509</td> <td>1.938944</td> <td>3.236361</td> <td>2.786997</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>2.734812</td> <td>1.115539</td> <td>1.679910</td> <td>0.633239</td> <td>4.391739</td> <td>1.130082</td> <td>1.137284</td> <td>0.919646</td> </tr> <tr> <th>Stream 1</th> <td>9.271071</td> <td>5.664855</td> <td>3.377869</td> <td>2.148228</td> <td>16.046021</td> <td>2.935643</td> <td>4.897009</td> <td>2.891040</td> </tr> <tr> <th>Stream 2</th> <td>10.272523</td> <td>4.578427</td> <td>4.086788</td> <td>2.312762</td> <td>16.295728</td> <td>2.714776</td> <td>6.393897</td> <td>2.414951</td> </tr> <tr> <th>Stream 3</th> <td>7.095213</td> <td>4.544636</td> <td>4.073433</td> <td>2.710320</td> <td>18.789088</td> <td>3.903873</td> <td>5.471600</td> <td>2.994184</td> </tr> <tr> <th>Stream 4</th> <td>7.567924</td> <td>3.691088</td> <td>3.951049</td> <td>2.207944</td> <td>18.189014</td> <td>4.985841</td> <td>6.568935</td> <td>3.965322</td> </tr> <tr> <th>Stream 5</th> <td>8.173577</td> <td>4.959777</td> <td>4.736593</td> <td>3.507469</td> <td>17.106990</td> <td>5.405699</td> <td>7.357104</td> <td>3.125788</td> </tr> <tr> <th>Min Qi</th> <td>7.095213</td> <td>3.691088</td> <td>3.377869</td> <td>2.148228</td> <td>16.046021</td> <td>2.714776</td> <td>4.897009</td> <td>2.414951</td> </tr> <tr> <th>Max Qi</th> <td>10.272523</td> <td>5.664855</td> <td>4.736593</td> <td>3.507469</td> <td>18.789088</td> <td>5.405699</td> <td>7.357104</td> <td>3.965322</td> </tr> <tr> <th>Avg Qi</th> <td>8.476062</td> <td>4.687757</td> <td>4.045146</td> <td>2.577345</td> <td>17.285368</td> <td>3.989166</td> <td>6.137709</td> <td>3.078257</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>1.206347</td> <td>0.792013</td> <td>0.699476</td> <td>1.349182</td> <td>1.505387</td> <td>0.543947</td> <td>1.549135</td> <td>0.824344</td> </tr> <tr> <th>Stream 1</th> <td>5.135036</td> <td>1.873195</td> <td>4.978155</td> <td>5.988226</td> <td>4.705365</td> <td>1.211049</td> <td>4.175947</td> <td>3.579242</td> </tr> <tr> <th>Stream 2</th> <td>7.656125</td> <td>2.229819</td> <td>2.805272</td> <td>6.629781</td> <td>4.138014</td> <td>1.423334</td> <td>5.165700</td> <td>3.197300</td> </tr> <tr> <th>Stream 3</th> <td>6.385983</td> <td>2.086301</td> <td>3.450305</td> <td>3.292353</td> <td>5.503905</td> <td>2.302992</td> <td>4.860041</td> <td>3.865383</td> </tr> <tr> <th>Stream 4</th> <td>6.514967</td> <td>2.876895</td> <td>3.481100</td> <td>1.629007</td> <td>5.715903</td> <td>2.121692</td> <td>3.681208</td> <td>3.347289</td> </tr> <tr> <th>Stream 5</th> <td>4.100205</td> <td>2.400816</td> <td>2.142291</td> <td>4.710677</td> <td>5.765320</td> <td>1.616445</td> <td>6.095817</td> <td>3.007436</td> </tr> <tr> <th>Min Qi</th> <td>4.100205</td> <td>1.873195</td> <td>2.142291</td> <td>1.629007</td> <td>4.138014</td> <td>1.211049</td> <td>3.681208</td> <td>3.007436</td> </tr> <tr> <th>Max Qi</th> <td>7.656125</td> <td>2.876895</td> <td>4.978155</td> <td>6.629781</td> <td>5.765320</td> <td>2.302992</td> <td>6.095817</td> <td>3.865383</td> </tr> <tr> <th>Avg Qi</th> <td>5.958463</td> <td>2.293405</td> <td>3.371425</td> <td>4.450009</td> <td>5.165701</td> <td>1.735102</td> <td>4.795743</td> <td>3.399330</td> </tr> </table> <h2>300G Run 1</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>September 25, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>300</td> </tr> <tr> <th style="text-align:right">Start of Database Load</th> <td>09/25/2014 16:38:20</td> </tr> <tr> <th style="text-align:right">End of Database Load</th> <td>09/25/2014 18:32:06</td> </tr> <tr> <th style="text-align:right">Database Load Time</th> <td>1:53:46</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>6</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td>61,988.7</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td>384,883.7</td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@300GB)</th> <td>154,461.6</td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td>370.498000 seconds</td> </tr> </table> <h3>Duration of stream execution:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>09/25/2014 19:00:29</td> <td>09/25/2014 19:22:25</td> <td>0:21:56</td> </tr> <tr> <th>Stream 1</th> <td>09/25/2014 19:22:27</td> <td>09/25/2014 19:28:23</td> <td>0:05:56</td> </tr> <tr> <th>Stream 2</th> <td>09/25/2014 19:22:27</td> <td>09/25/2014 19:28:23</td> <td>0:05:56</td> </tr> <tr> <th>Stream 3</th> <td>09/25/2014 19:22:27</td> <td>09/25/2014 19:28:26</td> <td>0:05:59</td> </tr> <tr> <th>Stream 4</th> <td>09/25/2014 19:22:27</td> <td>09/25/2014 19:28:13</td> <td>0:05:46</td> </tr> <tr> <th>Stream 5</th> <td>09/25/2014 19:22:27</td> <td>09/25/2014 19:28:38</td> <td>0:06:11</td> </tr> <tr> <th>Stream 6</th> <td>09/25/2014 19:22:27</td> <td>09/25/2014 19:28:38</td> <td>0:06:11</td> </tr> <tr> <th>Refresh 0</th> <td>09/25/2014 19:00:29</td> <td>09/25/2014 19:03:56</td> <td>0:03:27</td> </tr> <tr> <th> </th> <td>09/25/2014 19:22:25</td> <td>09/25/2014 19:22:27</td> <td>0:00:02</td> </tr> <tr> <th>Refresh 1</th> <td>09/25/2014 19:25:22</td> <td>09/25/2014 19:25:58</td> <td>0:00:36</td> </tr> <tr> <th>Refresh 2</th> <td>09/25/2014 19:22:27</td> <td>09/25/2014 19:23:11</td> <td>0:00:44</td> </tr> <tr> <th>Refresh 3</th> <td>09/25/2014 19:23:10</td> <td>09/25/2014 19:23:40</td> <td>0:00:30</td> </tr> <tr> <th>Refresh 4</th> <td>09/25/2014 19:23:40</td> <td>09/25/2014 19:24:21</td> <td>0:00:41</td> </tr> <tr> <th>Refresh 5</th> <td>09/25/2014 19:24:21</td> <td>09/25/2014 19:24:58</td> <td>0:00:37</td> </tr> <tr> <th>Refresh 6</th> <td>09/25/2014 19:24:59</td> <td>09/25/2014 19:25:22</td> <td>0:00:23</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> <th>Stream 0</th> <td>183.735463</td> <td>95.826361</td> <td>79.826802</td> <td>87.603164</td> <td>47.099641</td> <td>1.301704</td> <td>2.606488</td> <td>52.667426</td> </tr> <tr> <th>Stream 1</th> <td>9.400003</td> <td>1.983777</td> <td>15.839250</td> <td>3.001843</td> <td>15.593335</td> <td>6.067716</td> <td>8.870516</td> <td>11.679706</td> </tr> <tr> <th>Stream 2</th> <td>12.634711</td> <td>3.472203</td> <td>13.683075</td> <td>8.057952</td> <td>16.500741</td> <td>5.403771</td> <td>11.181661</td> <td>12.393932</td> </tr> <tr> <th>Stream 3</th> <td>10.807287</td> <td>3.793587</td> <td>15.844244</td> <td>3.214977</td> <td>15.960600</td> <td>7.099744</td> <td>10.424530</td> <td>21.001623</td> </tr> <tr> <th>Stream 4</th> <td>11.900829</td> <td>3.741707</td> <td>14.219904</td> <td>5.616907</td> <td>16.487144</td> <td>14.229782</td> <td>11.100193</td> <td>8.769539</td> </tr> <tr> <th>Stream 5</th> <td>13.933423</td> <td>2.916529</td> <td>19.453452</td> <td>5.258843</td> <td>16.706269</td> <td>7.948711</td> <td>8.982104</td> <td>17.566729</td> </tr> <tr> <th>Stream 6</th> <td>17.084445</td> <td>0.738683</td> <td>11.503079</td> <td>8.324812</td> <td>23.483917</td> <td>20.101834</td> <td>9.207737</td> <td>10.311292</td> </tr> <tr> <th>Min Qi</th> <td>9.400003</td> <td>0.738683</td> <td>11.503079</td> <td>3.001843</td> <td>15.593335</td> <td>5.403771</td> <td>8.870516</td> <td>8.769539</td> </tr> <tr> <th>Max Qi</th> <td>17.084445</td> <td>3.793587</td> <td>19.453452</td> <td>8.324812</td> <td>23.483917</td> <td>20.101834</td> <td>11.181661</td> <td>21.001623</td> </tr> <tr> <th>Avg Qi</th> <td>12.626783</td> <td>2.774414</td> <td>15.090501</td> <td>5.579222</td> <td>17.455334</td> <td>10.141926</td> <td>9.961123</td> <td>13.620470</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>41.997798</td> <td>2.727870</td> <td>21.651730</td> <td>25.704209</td> <td>293.103984</td> <td>3.171437</td> <td>2.886688</td> <td>5.298823</td> </tr> <tr> <th>Stream 1</th> <td>29.662265</td> <td>22.788618</td> <td>12.979253</td> <td>7.121358</td> <td>62.774323</td> <td>22.132581</td> <td>22.616793</td> <td>21.625334</td> </tr> <tr> <th>Stream 2</th> <td>28.041750</td> <td>22.481172</td> <td>19.262140</td> <td>5.790272</td> <td>58.105179</td> <td>16.809177</td> <td>32.813330</td> <td>12.692499</td> </tr> <tr> <th>Stream 3</th> <td>32.534297</td> <td>15.460256</td> <td>12.038047</td> <td>7.012926</td> <td>59.413740</td> <td>18.540284</td> <td>25.968635</td> <td>16.716208</td> </tr> <tr> <th>Stream 4</th> <td>28.759993</td> <td>15.123651</td> <td>21.734471</td> <td>6.920480</td> <td>63.119744</td> <td>12.848884</td> <td>21.372432</td> <td>11.662102</td> </tr> <tr> <th>Stream 5</th> <td>18.315308</td> <td>21.781800</td> <td>26.141212</td> <td>8.230858</td> <td>60.985590</td> <td>22.369824</td> <td>27.098660</td> <td>25.283066</td> </tr> <tr> <th>Stream 6</th> <td>31.455961</td> <td>27.078707</td> <td>12.954580</td> <td>11.081669</td> <td>72.483462</td> <td>12.376376</td> <td>22.129120</td> <td>11.439147</td> </tr> <tr> <th>Min Qi</th> <td>18.315308</td> <td>15.123651</td> <td>12.038047</td> <td>5.790272</td> <td>58.105179</td> <td>12.376376</td> <td>21.372432</td> <td>11.439147</td> </tr> <tr> <th>Max Qi</th> <td>32.534297</td> <td>27.078707</td> <td>26.141212</td> <td>11.081669</td> <td>72.483462</td> <td>22.369824</td> <td>32.813330</td> <td>25.283066</td> </tr> <tr> <th>Avg Qi</th> <td>28.128262</td> <td>20.785701</td> <td>17.518284</td> <td>7.692927</td> <td>62.813673</td> <td>17.512854</td> <td>25.333162</td> <td>16.569726</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>7.793403</td> <td>81.545934</td> <td>41.648484</td> <td>4.638731</td> <td>25.003179</td> <td>0.536267</td> <td>206.980380</td> <td>2.501589</td> </tr> <tr> <th>Stream 1</th> <td>27.058060</td> <td>3.894254</td> <td>8.664394</td> <td>25.315007</td> <td>11.921265</td> <td>3.561859</td> <td>22.936601</td> <td>13.235777</td> </tr> <tr> <th>Stream 2</th> <td>25.718500</td> <td>6.140657</td> <td>8.856586</td> <td>14.761290</td> <td>11.870351</td> <td>7.728217</td> <td>13.882613</td> <td>29.328859</td> </tr> <tr> <th>Stream 3</th> <td>15.896774</td> <td>8.631035</td> <td>15.742406</td> <td>20.621604</td> <td>13.370582</td> <td>5.536313</td> <td>14.677463</td> <td>14.772753</td> </tr> <tr> <th>Stream 4</th> <td>22.458327</td> <td>5.319241</td> <td>11.973431</td> <td>22.344017</td> <td>11.534642</td> <td>2.402683</td> <td>24.214115</td> <td>16.236299</td> </tr> <tr> <th>Stream 5</th> <td>13.407745</td> <td>5.413278</td> <td>8.800650</td> <td>18.055743</td> <td>17.528827</td> <td>4.173171</td> <td>15.927165</td> <td>21.636801</td> </tr> <tr> <th>Stream 6</th> <td>8.069721</td> <td>5.531066</td> <td>13.233927</td> <td>21.321389</td> <td>7.622026</td> <td>12.064182</td> <td>11.457848</td> <td>12.342336</td> </tr> <tr> <th>Min Qi</th> <td>8.069721</td> <td>3.894254</td> <td>8.664394</td> <td>14.761290</td> <td>7.622026</td> <td>2.402683</td> <td>11.457848</td> <td>12.342336</td> </tr> <tr> <th>Max Qi</th> <td>27.058060</td> <td>8.631035</td> <td>15.742406</td> <td>25.315007</td> <td>17.528827</td> <td>12.064182</td> <td>24.214115</td> <td>29.328859</td> </tr> <tr> <th>Avg Qi</th> <td>18.768188</td> <td>5.821588</td> <td>11.211899</td> <td>20.403175</td> <td>12.307949</td> <td>5.911071</td> <td>17.182634</td> <td>17.925471</td> </tr> </table> <h2>300G run 2</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>September 25, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>300</td> </tr> <tr> <th style="text-align:right">Start of Database Load</th> <td>09/25/2014 16:38:20</td> </tr> <tr> <th style="text-align:right">End of Database Load</th> <td>09/25/2014 18:32:06</td> </tr> <tr> <th style="text-align:right">Database Load Time</th> <td>1:53:46</td> </tr> <tr> <th style="text-align:right">Query Streams for<br /> Throughput Test</th> <td>6</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td>423,431.8</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td>387,248.6</td> </tr> <tr> <th style="text-align:right">Virt-H Composite<br /> Query-per-Hour Metric<br /> (Qph@300GB)</th> <td>404,936.3</td> </tr> <tr> <th style="text-align:right">Measurement Interval in<br /> Throughput Test (Ts)</th> <td>368.236000 seconds</td> </tr> </table> <h3>Duration of stream execution:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>09/25/2014 19:28:42</td> <td>09/25/2014 19:29:58</td> <td>0:01:16</td> </tr> <tr> <th>Stream 1</th> <td>09/25/2014 19:30:00</td> <td>09/25/2014 19:36:04</td> <td>0:06:04</td> </tr> <tr> <th>Stream 2</th> <td>09/25/2014 19:30:00</td> <td>09/25/2014 19:36:00</td> <td>0:06:00</td> </tr> <tr> <th>Stream 3</th> <td>09/25/2014 19:30:00</td> <td>09/25/2014 19:36:06</td> <td>0:06:06</td> </tr> <tr> <th>Stream 4</th> <td>09/25/2014 19:30:00</td> <td>09/25/2014 19:36:07</td> <td>0:06:07</td> </tr> <tr> <th>Stream 5</th> <td>09/25/2014 19:30:00</td> <td>09/25/2014 19:35:53</td> <td>0:05:53</td> </tr> <tr> <th>Stream 6</th> <td>09/25/2014 19:30:00</td> <td>09/25/2014 19:36:08</td> <td>0:06:08</td> </tr> <tr> <th>Refresh 0</th> <td>09/25/2014 19:28:41</td> <td>09/25/2014 19:28:46</td> <td>0:00:05</td> </tr> <tr> <th> </th> <td>09/25/2014 19:29:58</td> <td>09/25/2014 19:30:00</td> <td>0:00:02</td> </tr> <tr> <th>Refresh 1</th> <td>09/25/2014 19:32:23</td> <td>09/25/2014 19:32:55</td> <td>0:00:32</td> </tr> <tr> <th>Refresh 2</th> <td>09/25/2014 19:30:00</td> <td>09/25/2014 19:30:31</td> <td>0:00:31</td> </tr> <tr> <th>Refresh 3</th> <td>09/25/2014 19:30:31</td> <td>09/25/2014 19:31:00</td> <td>0:00:29</td> </tr> <tr> <th>Refresh 4</th> <td>09/25/2014 19:31:01</td> <td>09/25/2014 19:31:23</td> <td>0:00:22</td> </tr> <tr> <th>Refresh 5</th> <td>09/25/2014 19:31:23</td> <td>09/25/2014 19:31:54</td> <td>0:00:31</td> </tr> <tr> <th>Refresh 6</th> <td>09/25/2014 19:31:55</td> <td>09/25/2014 19:32:23</td> <td>0:00:28</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> <th>Stream 0</th> <td>4.197427</td> <td>1.011516</td> <td>2.535959</td> <td>0.858781</td> <td>2.857279</td> <td>1.293530</td> <td>2.682266</td> <td>2.260502</td> </tr> <tr> <th>Stream 1</th> <td>15.467757</td> <td>3.517499</td> <td>13.820864</td> <td>4.157259</td> <td>13.141556</td> <td>10.902710</td> <td>16.899687</td> <td>8.986535</td> </tr> <tr> <th>Stream 2</th> <td>15.639991</td> <td>6.026485</td> <td>13.521624</td> <td>3.918031</td> <td>17.336458</td> <td>1.975310</td> <td>9.718194</td> <td>15.165247</td> </tr> <tr> <th>Stream 3</th> <td>14.891929</td> <td>4.481383</td> <td>15.322621</td> <td>5.272911</td> <td>15.266543</td> <td>6.771253</td> <td>13.430646</td> <td>20.171084</td> </tr> <tr> <th>Stream 4</th> <td>14.560526</td> <td>2.464157</td> <td>11.567112</td> <td>5.526629</td> <td>20.531540</td> <td>5.225971</td> <td>16.288606</td> <td>17.209475</td> </tr> <tr> <th>Stream 5</th> <td>10.390577</td> <td>3.549165</td> <td>9.598328</td> <td>8.783847</td> <td>17.351211</td> <td>6.308214</td> <td>12.606512</td> <td>13.035716</td> </tr> <tr> <th>Stream 6</th> <td>16.275922</td> <td>4.086475</td> <td>14.109963</td> <td>4.385887</td> <td>10.174709</td> <td>6.703266</td> <td>8.936217</td> <td>16.798526</td> </tr> <tr> <th>Min Qi</th> <td>10.390577</td> <td>2.464157</td> <td>9.598328</td> <td>3.918031</td> <td>10.174709</td> <td>1.975310</td> <td>8.936217</td> <td>8.986535</td> </tr> <tr> <th>Max Qi</th> <td>16.275922</td> <td>6.026485</td> <td>15.322621</td> <td>8.783847</td> <td>20.531540</td> <td>10.902710</td> <td>16.899687</td> <td>20.171084</td> </tr> <tr> <th>Avg Qi</th> <td>14.537784</td> <td>4.020861</td> <td>12.990085</td> <td>5.340761</td> <td>15.633670</td> <td>6.314454</td> <td>12.979977</td> <td>15.227764</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>8.300092</td> <td>2.598145</td> <td>5.168418</td> <td>1.619399</td> <td>11.958836</td> <td>3.191672</td> <td>3.097822</td> <td>2.497410</td> </tr> <tr> <th>Stream 1</th> <td>26.412829</td> <td>17.354745</td> <td>12.942454</td> <td>8.169447</td> <td>58.600101</td> <td>15.227942</td> <td>32.985324</td> <td>13.914978</td> </tr> <tr> <th>Stream 2</th> <td>34.523245</td> <td>17.635531</td> <td>15.193748</td> <td>8.435375</td> <td>62.442800</td> <td>16.276300</td> <td>26.533303</td> <td>12.414575</td> </tr> <tr> <th>Stream 3</th> <td>25.334301</td> <td>18.595422</td> <td>11.663933</td> <td>10.029387</td> <td>63.664992</td> <td>20.378320</td> <td>24.760768</td> <td>15.710589</td> </tr> <tr> <th>Stream 4</th> <td>36.971957</td> <td>15.645673</td> <td>14.672851</td> <td>13.196301</td> <td>58.214728</td> <td>17.375053</td> <td>26.581101</td> <td>11.624989</td> </tr> <tr> <th>Stream 5</th> <td>30.891797</td> <td>12.993365</td> <td>14.089049</td> <td>10.515091</td> <td>65.232712</td> <td>20.807026</td> <td>26.920526</td> <td>11.362095</td> </tr> <tr> <th>Stream 6</th> <td>38.143281</td> <td>21.106772</td> <td>15.152299</td> <td>18.845766</td> <td>66.240343</td> <td>12.295624</td> <td>22.510610</td> <td>18.081103</td> </tr> <tr> <th>Min Qi</th> <td>25.334301</td> <td>12.993365</td> <td>11.663933</td> <td>8.169447</td> <td>58.214728</td> <td>12.295624</td> <td>22.510610</td> <td>11.362095</td> </tr> <tr> <th>Max Qi</th> <td>38.143281</td> <td>21.106772</td> <td>15.193748</td> <td>18.845766</td> <td>66.240343</td> <td>20.807026</td> <td>32.985324</td> <td>18.081103</td> </tr> <tr> <th>Avg Qi</th> <td>32.046235</td> <td>17.221918</td> <td>13.952389</td> <td>11.531894</td> <td>62.399279</td> <td>17.060044</td> <td>26.715272</td> <td>13.851388</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>4.016212</td> <td>1.603004</td> <td>1.836489</td> <td>3.542383</td> <td>3.901876</td> <td>0.515102</td> <td>4.759612</td> <td>2.358873</td> </tr> <tr> <th>Stream 1</th> <td>22.162387</td> <td>10.067834</td> <td>15.772705</td> <td>22.091355</td> <td>12.974776</td> <td>8.354196</td> <td>19.342171</td> <td>12.771250</td> </tr> <tr> <th>Stream 2</th> <td>25.647926</td> <td>4.263008</td> <td>11.590737</td> <td>19.179326</td> <td>17.899770</td> <td>4.137031</td> <td>15.720245</td> <td>14.719776</td> </tr> <tr> <th>Stream 3</th> <td>14.511279</td> <td>7.484608</td> <td>20.735250</td> <td>13.041037</td> <td>17.139046</td> <td>6.014141</td> <td>16.234122</td> <td>13.454647</td> </tr> <tr> <th>Stream 4</th> <td>19.297494</td> <td>10.110707</td> <td>10.907458</td> <td>19.649066</td> <td>15.206251</td> <td>3.423503</td> <td>11.268082</td> <td>11.852223</td> </tr> <tr> <th>Stream 5</th> <td>17.445165</td> <td>5.582309</td> <td>15.266324</td> <td>19.788382</td> <td>14.245770</td> <td>2.810949</td> <td>16.601461</td> <td>14.019717</td> </tr> <tr> <th>Stream 6</th> <td>25.115339</td> <td>6.896503</td> <td>11.661563</td> <td>21.900028</td> <td>5.520025</td> <td>3.093050</td> <td>15.436258</td> <td>13.353446</td> </tr> <tr> <th>Min Qi</th> <td>14.511279</td> <td>4.263008</td> <td>10.907458</td> <td>13.041037</td> <td>5.520025</td> <td>2.810949</td> <td>11.268082</td> <td>11.852223</td> </tr> <tr> <th>Max Qi</th> <td>25.647926</td> <td>10.110707</td> <td>20.735250</td> <td>22.091355</td> <td>17.899770</td> <td>8.354196</td> <td>19.342171</td> <td>14.719776</td> </tr> <tr> <th>Avg Qi</th> <td>20.696598</td> <td>7.400828</td> <td>14.322339</td> <td>19.274866</td> <td>13.830940</td> <td>4.638812</td> <td>15.767057</td> <td>13.361843</td> </tr> </table> <h2>300G run 3:</h2> <h3>Virt-H Executive Summary</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;"> <tr> <th style="text-align:right">Report Date</th> <td>September 25, 2014</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td>300</td> </tr> <tr> <th style="text-align:right">Total Data Storage/Database Size</th> <td>258,888M</td> </tr> <tr> <th style="text-align:right">Start of Database Load</th> <td>09/25/2014 16:38:20</td> </tr> <tr> <th style="text-align:right">End of Database Load</th> <td>09/25/2014 18:32:06</td> </tr> <tr> <th style="text-align:right">Database Load Time</th> <td>1:53:46</td> </tr> <tr> <th style="text-align:right">Query Streams for <br />Throughput Test</th> <td>6</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td>417,672.0</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td>389,719.5</td> </tr> <tr> <th style="text-align:right">Virt-H Composite <br />Query-per-Hour Metric <br />(Qph@300GB)</th> <td>403,453.7</td> </tr> <tr> <th style="text-align:right">Measurement Interval in <br />Throughput Test (Ts)</th> <td>365.902000 seconds</td> </tr> </table> <h3>Duration of stream execution:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>09/25/2014 19:36:11</td> <td>09/25/2014 19:37:29</td> <td>0:01:18</td> </tr> <tr> <th>Stream 1</th> <td>09/25/2014 19:37:32</td> <td>09/25/2014 19:43:13</td> <td>0:05:41</td> </tr> <tr> <th>Stream 2</th> <td>09/25/2014 19:37:32</td> <td>09/25/2014 19:43:31</td> <td>0:05:59</td> </tr> <tr> <th>Stream 3</th> <td>09/25/2014 19:37:32</td> <td>09/25/2014 19:43:37</td> <td>0:06:05</td> </tr> <tr> <th>Stream 4</th> <td>09/25/2014 19:37:32</td> <td>09/25/2014 19:43:33</td> <td>0:06:01</td> </tr> <tr> <th>Stream 5</th> <td>09/25/2014 19:37:32</td> <td>09/25/2014 19:43:32</td> <td>0:06:00</td> </tr> <tr> <th>Stream 6</th> <td>09/25/2014 19:37:32</td> <td>09/25/2014 19:43:37</td> <td>0:06:05</td> </tr> <tr> <th>Refresh 0</th> <td>09/25/2014 19:36:12</td> <td>09/25/2014 19:36:16</td> <td>0:00:04</td> </tr> <tr> <th> </th> <td>09/25/2014 19:37:29</td> <td>09/25/2014 19:37:31</td> <td>0:00:02</td> </tr> <tr> <th>Refresh 1</th> <td>09/25/2014 19:40:02</td> <td>09/25/2014 19:40:33</td> <td>0:00:31</td> </tr> <tr> <th>Refresh 2</th> <td>09/25/2014 19:37:31</td> <td>09/25/2014 19:38:01</td> <td>0:00:30</td> </tr> <tr> <th>Refresh 3</th> <td>09/25/2014 19:38:01</td> <td>09/25/2014 19:38:30</td> <td>0:00:29</td> </tr> <tr> <th>Refresh 4</th> <td>09/25/2014 19:38:30</td> <td>09/25/2014 19:38:58</td> <td>0:00:28</td> </tr> <tr> <th>Refresh 5</th> <td>09/25/2014 19:38:58</td> <td>09/25/2014 19:39:27</td> <td>0:00:29</td> </tr> <tr> <th>Refresh 6</th> <td>09/25/2014 19:39:27</td> <td>09/25/2014 19:40:01</td> <td>0:00:34</td> </tr> </table> <h3>Numerical Quantities Summary Timing Intervals in Seconds:</h3> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> <th>Stream 0</th> <td>4.305006</td> <td>1.083442</td> <td>2.502758</td> <td>0.845763</td> <td>2.840824</td> <td>1.346166</td> <td>2.659511</td> <td>2.233550</td> </tr> <tr> <th>Stream 1</th> <td>11.513360</td> <td>3.732513</td> <td>14.530428</td> <td>3.819517</td> <td>14.821291</td> <td>7.561547</td> <td>10.435082</td> <td>8.984230</td> </tr> <tr> <th>Stream 2</th> <td>13.486433</td> <td>3.373689</td> <td>9.620363</td> <td>3.914320</td> <td>16.857542</td> <td>5.837487</td> <td>10.695443</td> <td>17.901191</td> </tr> <tr> <th>Stream 3</th> <td>11.015942</td> <td>1.780220</td> <td>4.830412</td> <td>9.073543</td> <td>15.587709</td> <td>9.661989</td> <td>12.374931</td> <td>15.262485</td> </tr> <tr> <th>Stream 4</th> <td>13.600461</td> <td>0.820899</td> <td>12.254226</td> <td>7.799415</td> <td>19.860761</td> <td>13.145017</td> <td>14.404345</td> <td>11.807583</td> </tr> <tr> <th>Stream 5</th> <td>13.358000</td> <td>3.885118</td> <td>11.099935</td> <td>4.845043</td> <td>18.286721</td> <td>6.424272</td> <td>9.735255</td> <td>15.041608</td> </tr> <tr> <th>Stream 6</th> <td>13.588873</td> <td>3.789631</td> <td>13.503399</td> <td>5.130389</td> <td>13.104065</td> <td>3.517076</td> <td>14.929079</td> <td>19.831639</td> </tr> <tr> <th>Min Qi</th> <td>11.015942</td> <td>0.820899</td> <td>4.830412</td> <td>3.819517</td> <td>13.104065</td> <td>3.517076</td> <td>9.735255</td> <td>8.984230</td> </tr> <tr> <th>Max Qi</th> <td>13.600461</td> <td>3.885118</td> <td>14.530428</td> <td>9.073543</td> <td>19.860761</td> <td>13.145017</td> <td>14.929079</td> <td>19.831639</td> </tr> <tr> <th>Avg Qi</th> <td>12.760511</td> <td>2.897012</td> <td>10.973127</td> <td>5.763705</td> <td>16.419681</td> <td>7.691231</td> <td>12.095689</td> <td>14.804789</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right;"> <tr> <th>Query</th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> <th>Stream 0</th> <td>8.553183</td> <td>3.215484</td> <td>4.652364</td> <td>1.620089</td> <td>11.936052</td> <td>2.916132</td> <td>3.219969</td> <td>2.374276</td> </tr> <tr> <th>Stream 1</th> <td>29.441108</td> <td>20.348266</td> <td>9.994556</td> <td>14.965432</td> <td>60.537168</td> <td>13.302875</td> <td>30.159402</td> <td>10.277570</td> </tr> <tr> <th>Stream 2</th> <td>41.799347</td> <td>18.197400</td> <td>16.773638</td> <td>6.510347</td> <td>67.461446</td> <td>20.362328</td> <td>0.109929</td> <td>9.908769</td> </tr> <tr> <th>Stream 3</th> <td>24.306937</td> <td>20.555376</td> <td>17.140758</td> <td>16.715188</td> <td>61.724168</td> <td>22.469230</td> <td>27.967206</td> <td>13.434167</td> </tr> <tr> <th>Stream 4</th> <td>34.820796</td> <td>11.795664</td> <td>18.015120</td> <td>7.176057</td> <td>63.134711</td> <td>11.427374</td> <td>23.959842</td> <td>16.759246</td> </tr> <tr> <th>Stream 5</th> <td>23.139366</td> <td>12.655317</td> <td>13.152401</td> <td>7.258740</td> <td>64.273225</td> <td>22.854106</td> <td>28.803059</td> <td>12.832364</td> </tr> <tr> <th>Stream 6</th> <td>27.955059</td> <td>24.633526</td> <td>11.046285</td> <td>5.995041</td> <td>74.965966</td> <td>15.636579</td> <td>22.803890</td> <td>13.221303</td> </tr> <tr> <th>Min Qi</th> <td>23.139366</td> <td>11.795664</td> <td>9.994556</td> <td>5.995041</td> <td>60.537168</td> <td>11.427374</td> <td>0.109929</td> <td>9.908769</td> </tr> <tr> <th>Max Qi</th> <td>41.799347</td> <td>24.633526</td> <td>18.015120</td> <td>16.715188</td> <td>74.965966</td> <td>22.854106</td> <td>30.159402</td> <td>16.759246</td> </tr> <tr> <th>Avg Qi</th> <td>30.243769</td> <td>18.030925</td> <td>14.353793</td> <td>9.770134</td> <td>65.349447</td> <td>17.675415</td> <td>22.300555</td> <td>12.738903</td> </tr> </table> <table style="padding:10px;border-spacing:10px;margin-left:auto;margin-right:auto;text-align:right"> <tr> <th>Query</th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> <th>Stream 0</th> <td>4.298092</td> <td>1.702071</td> <td>1.894548</td> <td>4.118591</td> <td>3.922889</td> <td>0.491145</td> <td>4.519734</td> <td>2.347913</td> </tr> <tr> <th>Stream 1</th> <td>16.432222</td> <td>6.908918</td> <td>17.749058</td> <td>18.756674</td> <td>11.148628</td> <td>5.464975</td> <td>18.300673</td> <td>12.972871</td> </tr> <tr> <th>Stream 2</th> <td>20.588544</td> <td>4.387662</td> <td>14.527229</td> <td>23.844364</td> <td>15.500462</td> <td>15.543458</td> <td>13.666574</td> <td>15.240662</td> </tr> <tr> <th>Stream 3</th> <td>14.008049</td> <td>6.222633</td> <td>12.833421</td> <td>22.811602</td> <td>16.013232</td> <td>9.449069</td> <td>16.486111</td> <td>12.974515</td> </tr> <tr> <th>Stream 4</th> <td>16.964699</td> <td>8.106044</td> <td>11.207675</td> <td>22.483826</td> <td>17.354675</td> <td>4.641183</td> <td>14.583941</td> <td>13.679087</td> </tr> <tr> <th>Stream 5</th> <td>25.243144</td> <td>7.359437</td> <td>16.986615</td> <td>19.855391</td> <td>17.183725</td> <td>5.750937</td> <td>14.759597</td> <td>13.052316</td> </tr> <tr> <th>Stream 6</th> <td>12.986721</td> <td>10.160993</td> <td>17.496662</td> <td>19.267026</td> <td>17.300224</td> <td>4.955930</td> <td>19.267721</td> <td>15.421241</td> </tr> <tr> <th>Min Qi</th> <td>12.986721</td> <td>4.387662</td> <td>11.207675</td> <td>18.756674</td> <td>11.148628</td> <td>4.641183</td> <td>13.666574</td> <td>12.972871</td> </tr> <tr> <th>Max Qi</th> <td>25.243144</td> <td>10.160993</td> <td>17.749058</td> <td>23.844364</td> <td>17.354675</td> <td>15.543458</td> <td>19.267721</td> <td>15.421241</td> </tr> <tr> <th>Avg Qi</th> <td>17.703896</td> <td>7.190948</td> <td>15.133443</td> <td>21.169814</td> <td>15.750158</td> <td>7.634259</td> <td>16.177436</td> <td>13.890115</td> </tr> </table> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac349bc698"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac347508c8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac347b2098"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac34f71318"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac346fa748"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac349de568"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aac34725b28"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aac36adde48"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aac34a7e368"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac348581b8"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac344340c8"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac34ee7058"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac34d388b8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x69cacf8"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aabeaafeba8"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aac68efcef8"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab74c90428"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aac0c39ad38"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:06:15.589027-04:00
In Hoc Signo Vinces (part 16 of n): Introduction to Scale-Out
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-09-24#1817
2014-09-24T17:05:34Z
<p>So far, we have analyzed <a href="http://www.tpc.org/tpch/" id="link-id0x1a41d4f8">TPC-H</a> in a single-server, memory-only setting. We will now move to larger data and cluster implementations. In principle, TPC-H parallelizes well, so we should expect near-linear scalability; i.e., twice the gear runs twice as fast, or close enough.</p> <p>In practice, things are not quite so simple. Larger data, particularly a different data-to-memory ratio, and the fact of having no shared memory, all play a role. There is also a network, so partitioned operations, which also existed in the single-server case, now have to send messages across machines, not across threads. For data loading and refreshes, there is generally no shared file system, so data distribution and parallelism have to be considered.</p> <p>As an initial pass, we look at 100G and 1000G scales on the same test system as before. This is two machines, each with dual Xeon E5-2630, 192 GB RAM, 2 x 512 GB SSD, and QDR InfiniBand. We will also try other platforms, but if nothing else is said, this is the test system.</p> <p>As of this writing, there is a working implementation, but it is not guaranteed to be optimal as yet. We will adjust it as we go through the workload. One outcome of the experiment will be a precise determination of the data-volume-to-RAM ratio that still gives good performance.</p> <p> <i>A priori, </i>we know of the following things that complicate life with clusters:</p> <ul> <li> <p> <b>Distributed memory</b> — The working set must be in memory for a run to have a competitive score. A cluster can have a lot of memory, and the data is such that it partitions very evenly, so this appears at first not a problem. The difficulty comes with query memory: If each machine has 1/16th of the total RAM and a hash table would be 1/64th of the working set, on a single-server it is no problem just building the hash table. On a scale-out system, the hash table would be 1/4 of the working set if replicated on each node, which will not fit, especially if there are many such hash tables at the same time. Two main approaches exist: The hash table can be partitioned, but this will force the probe to go cross-partition, which takes time. The other possibility is to build the hash table many times, each time with a fraction of the data, and to run the probe side many times. Since hash tables often have Bloom filters, it is sometimes possible to replicate the Bloom filter and partition the hash table. One has also heard of hash tables that go to secondary storage, but should this happen, the race is already lost; so, we do not go there.</p> <p> We must evaluate different combinations of these techniques and have a cost model that accurately predicts the performance of each variant. Adding to realism is always safe but halfway difficult to do.</p> </li> <li> <p> <b>NUMA</b> — Most servers are <a href="http://dbpedia.org/resource/Non-uniform_memory_access" id="link-id0x2aabbef390d8">NUMA (non-uniform memory architecture)</a>, where each CPU socket has its own local memory. For single-server cases, we use all the memory for the process. Some implementations have special logic for memory affinity between threads. With scale-out there is the choice of having a server process per-NUMA-node or per-physical-machine. If per-NUMA-node, we are guaranteed only local memory accesses. This is a tradeoff to be evaluated.</p> </li> <li> <p> <b>Network and Scheduling</b> — Execution on a cluster is always <a href="http://docs.openlinksw.com/virtuoso/vexqrparl.html" id="link-id0x2aabbef26648">vectored</a>, for the simple reason that sending single-tuple messages is unfeasible in terms of performance. With an otherwise vectored architecture, the message batching required on a cluster comes naturally. However, the larger the cluster, the more partitions there are, which rapidly gets into shorter messages. Increasing the vector size is possible and messages become longer, but indefinite increase in vector size has drawbacks for cache locality and takes memory. To run well, each thread must stay on core. There are two ways of being taken off core ahead of time: Blocking for a mutex, and blocking for network. Lots of short messages run into scheduling overhead, since the recipient must decide what to do with each, which is not really possible without some sort of critical section. This is more efficient if messages are longer, as the decision time does not depend on message length. Longer messages are however liable to block on write at the sender side. So one pays in either case. This is another tradeoff to be balanced.</p> </li> <li> <p> <b>Flow control</b> — A query is a pipeline of producers and consumers. Sometimes the consumer is in a different partition. The producer must not get indefinitely ahead of the consumer because this would run out of memory, but it must stay sufficiently ahead so as not to stop the consumer. In practice, there are synchronization barriers to check even progress. These will decrease platform utilization, because two threads never finish at exactly the same time. The price of not having these is having no cap on transient memory consumption.</p> </li> <li> <p> <b>Un-homogenous performance</b> — Identical machines do not always perform identically. This is seen especially with disk, where wear on SSDs can affect write speed, and where uncontrollable hazards of data placement will get uneven read speeds on rotating media. Purely memory-bound performance is quite close, though. Un-anticipatable and uncontrollable hazards of scheduling cause different times of arrival of network messages, which introduces variation in run time on consecutive runs. Single-servers have some such variation from threading, but the effects are larger with a network.</p> </li> </ul> <p>The logical side of query optimization stays the same. Pushing down predicates is always good, and all the logical tricks with moving conditions between subqueries stay the same.</p> <p>Schema design stays much the same, but there is the extra question of partitioning keys. In this implementation, there are only indices on identifiers, not on dates, for example. So, for a primary key to foreign key join, if there is an index on the foreign key, the index should be partitioned the same way as the primary key. So, joining from <code>orders</code> to <code>lineitem</code> on <code>orderkey</code> will be co-located. Joining from <code>customer</code> to <code>orders</code> by index will be colocated for the <code>c_custkey = o_custkey</code> part (assuming an index on <code>o_custkey</code>) and cross-partition for getting the <code>customer</code> row on <code>c_custkey</code>, supposing that the query needs some property of the <code>customer</code> other than <code>c_custkey</code> or <code>c_orderkey</code>.</p> <p>A secondary question is the partition granularity. For good compression, nearby values should be consecutive, so here we leave the low 12 bits out of the partitioning. This has effect on bulk load and refreshes, for example, so that a batch of 10,000 <code>lineitems</code>, ordered on <code>l_orderkey</code> will go to only 2 or 3 distinct destinations, thus getting longer messages and longer insert batches, which is more efficient.</p> <p>This is a quick overview of the wisdom so far. In subsequent installments, we will take a quantitative look at the tradeoffs and consider actual queries. As a conclusion, we will show a full run on a couple of different platforms, and likely provide Amazon machine images for the interested to see for themselves. <a href="http://virtuoso.openlinksw.com/features-comparison-matrix/" id="link-id0x2aabdd36fd68">Virtuoso Cluster is not open source</a>, but the cloud will provide easy access.</p> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac349bc698"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac347508c8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac347b2098"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac34f71318"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac346fa748"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac349de568"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aac34725b28"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aac36adde48"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aac34a7e368"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac348581b8"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac344340c8"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac34ee7058"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac34d388b8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> In Hoc Signo Vinces (part 16): Introduction to Scale-Out<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x18bd44b8"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x854ee68"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x19773848"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aac0421e0d8"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:09:35.168159-04:00
In Hoc Signo Vinces (part 16 of n): Introduction to Scale-Out
http://www.openlinksw.com:443/weblog/oerling/?date=2014-09-24#1816
2014-09-24T17:05:24Z
<p>So far, we have analyzed <a href="http://www.tpc.org/tpch/" id="link-id0x1a41d4f8">TPC-H</a> in a single-server, memory-only setting. We will now move to larger data and cluster implementations. In principle, TPC-H parallelizes well, so we should expect near-linear scalability; i.e., twice the gear runs twice as fast, or close enough.</p> <p>In practice, things are not quite so simple. Larger data, particularly a different data-to-memory ratio, and the fact of having no shared memory, all play a role. There is also a network, so partitioned operations, which also existed in the single-server case, now have to send messages across machines, not across threads. For data loading and refreshes, there is generally no shared file system, so data distribution and parallelism have to be considered.</p> <p>As an initial pass, we look at 100G and 1000G scales on the same test system as before. This is two machines, each with dual Xeon E5-2630, 192 GB RAM, 2 x 512 GB SSD, and QDR InfiniBand. We will also try other platforms, but if nothing else is said, this is the test system.</p> <p>As of this writing, there is a working implementation, but it is not guaranteed to be optimal as yet. We will adjust it as we go through the workload. One outcome of the experiment will be a precise determination of the data-volume-to-RAM ratio that still gives good performance.</p> <p> <i>A priori, </i>we know of the following things that complicate life with clusters:</p> <ul> <li> <p> <b>Distributed memory</b> — The working set must be in memory for a run to have a competitive score. A cluster can have a lot of memory, and the data is such that it partitions very evenly, so this appears at first not a problem. The difficulty comes with query memory: If each machine has 1/16th of the total RAM and a hash table would be 1/64th of the working set, on a single-server it is no problem just building the hash table. On a scale-out system, the hash table would be 1/4 of the working set if replicated on each node, which will not fit, especially if there are many such hash tables at the same time. Two main approaches exist: The hash table can be partitioned, but this will force the probe to go cross-partition, which takes time. The other possibility is to build the hash table many times, each time with a fraction of the data, and to run the probe side many times. Since hash tables often have Bloom filters, it is sometimes possible to replicate the Bloom filter and partition the hash table. One has also heard of hash tables that go to secondary storage, but should this happen, the race is already lost; so, we do not go there.</p> <p> We must evaluate different combinations of these techniques and have a cost model that accurately predicts the performance of each variant. Adding to realism is always safe but halfway difficult to do.</p> </li> <li> <p> <b>NUMA</b> — Most servers are <a href="http://dbpedia.org/resource/Non-uniform_memory_access" id="link-id0x2aabbef390d8">NUMA (non-uniform memory architecture)</a>, where each CPU socket has its own local memory. For single-server cases, we use all the memory for the process. Some implementations have special logic for memory affinity between threads. With scale-out there is the choice of having a server process per-NUMA-node or per-physical-machine. If per-NUMA-node, we are guaranteed only local memory accesses. This is a tradeoff to be evaluated.</p> </li> <li> <p> <b>Network and Scheduling</b> — Execution on a cluster is always <a href="http://docs.openlinksw.com/virtuoso/vexqrparl.html" id="link-id0x2aabbef26648">vectored</a>, for the simple reason that sending single-tuple messages is unfeasible in terms of performance. With an otherwise vectored architecture, the message batching required on a cluster comes naturally. However, the larger the cluster, the more partitions there are, which rapidly gets into shorter messages. Increasing the vector size is possible and messages become longer, but indefinite increase in vector size has drawbacks for cache locality and takes memory. To run well, each thread must stay on core. There are two ways of being taken off core ahead of time: Blocking for a mutex, and blocking for network. Lots of short messages run into scheduling overhead, since the recipient must decide what to do with each, which is not really possible without some sort of critical section. This is more efficient if messages are longer, as the decision time does not depend on message length. Longer messages are however liable to block on write at the sender side. So one pays in either case. This is another tradeoff to be balanced.</p> </li> <li> <p> <b>Flow control</b> — A query is a pipeline of producers and consumers. Sometimes the consumer is in a different partition. The producer must not get indefinitely ahead of the consumer because this would run out of memory, but it must stay sufficiently ahead so as not to stop the consumer. In practice, there are synchronization barriers to check even progress. These will decrease platform utilization, because two threads never finish at exactly the same time. The price of not having these is having no cap on transient memory consumption.</p> </li> <li> <p> <b>Un-homogenous performance</b> — Identical machines do not always perform identically. This is seen especially with disk, where wear on SSDs can affect write speed, and where uncontrollable hazards of data placement will get uneven read speeds on rotating media. Purely memory-bound performance is quite close, though. Un-anticipatable and uncontrollable hazards of scheduling cause different times of arrival of network messages, which introduces variation in run time on consecutive runs. Single-servers have some such variation from threading, but the effects are larger with a network.</p> </li> </ul> <p>The logical side of query optimization stays the same. Pushing down predicates is always good, and all the logical tricks with moving conditions between subqueries stay the same.</p> <p>Schema design stays much the same, but there is the extra question of partitioning keys. In this implementation, there are only indices on identifiers, not on dates, for example. So, for a primary key to foreign key join, if there is an index on the foreign key, the index should be partitioned the same way as the primary key. So, joining from <code>orders</code> to <code>lineitem</code> on <code>orderkey</code> will be co-located. Joining from <code>customer</code> to <code>orders</code> by index will be colocated for the <code>c_custkey = o_custkey</code> part (assuming an index on <code>o_custkey</code>) and cross-partition for getting the <code>customer</code> row on <code>c_custkey</code>, supposing that the query needs some property of the <code>customer</code> other than <code>c_custkey</code> or <code>c_orderkey</code>.</p> <p>A secondary question is the partition granularity. For good compression, nearby values should be consecutive, so here we leave the low 12 bits out of the partitioning. This has effect on bulk load and refreshes, for example, so that a batch of 10,000 <code>lineitems</code>, ordered on <code>l_orderkey</code> will go to only 2 or 3 distinct destinations, thus getting longer messages and longer insert batches, which is more efficient.</p> <p>This is a quick overview of the wisdom so far. In subsequent installments, we will take a quantitative look at the tradeoffs and consider actual queries. As a conclusion, we will show a full run on a couple of different platforms, and likely provide Amazon machine images for the interested to see for themselves. <a href="http://virtuoso.openlinksw.com/features-comparison-matrix/" id="link-id0x2aabdd36fd68">Virtuoso Cluster is not open source</a>, but the cloud will provide easy access.</p> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac349bc698"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac347508c8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac347b2098"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac34f71318"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac346fa748"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac349de568"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aac34725b28"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aac36adde48"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aac34a7e368"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac348581b8"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac344340c8"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac34ee7058"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac34d388b8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> In Hoc Signo Vinces (part 16): Introduction to Scale-Out<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aabe9e5aaf8"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aabe8468c28"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aaca9151718"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab74235cc8"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aac100938b8"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:06:22.354982-04:00
SEMANTiCS 2014 (part 3 of 3): Conversations
http://www.openlinksw.com:443/weblog/oerling/?date=2014-09-08#1815
2014-09-08T20:11:10Z
<p>I was asked for an oracular statement about the future of <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x2aab7fe44ae8">relational database (RDBMS)</a> at the conference. The answer, without doubt or hesitation, is that this is forever. But this does not mean that the RDBMS world would be immutable, quite the opposite.</p> <p>The specializations converge. The RDBMS becomes more adaptable and less schema-first. Of course the RDBMS also take new data models beside the relational. <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x2aabe158ff18">RDF</a> and other property graph models, for instance.</p> <p>The schema-last-ness is now well in evidence. For example, <a href="http://dbpedia.org/resource/PostgreSQL" id="link-id0x2aab7fe865f8">PostgreSQL</a> has an <i><code>hstore</code></i> column type which is a list of key-value pairs. <a href="http://dbpedia.org/resource/Vertica" id="link-id0x2aabe021af48">Vertica</a> has a feature called <i>flex tables</i> where a column can be added on a row-by-row basis.</p> <p>Specialized indexing for text and geometries is a well established practice. However, dedicated IR systems, often <a href="http://dbpedia.org/resource/Lucene" id="link-id0x2aabe008f4e8">Lucene</a> derivatives, can offer more transparency in the IR domain for things like vector-space-models and hit-scoring. There is specialized faceted search support which is quite good. I do not know of an RDBMS that would do the exact same trick as Lucene for facets, but, of course, in the forever expanding scope of RDB, this is added easily enough.</p> <p> <a href="http://dbpedia.org/resource/JSON" id="link-id0x2aabfde79cb8">JSON</a> is all the rage in the web developer world. <a href="https://www.linkedin.com/in/philarcher" id="link-id0x2aab7fc19b88">Phil Archer</a> even said in his keynote, as a parody of the web developer: " I will never touch that crap of RDF or the semantic web; this is a pipe dream of reality ignoring academics and I will not have it. I will only use <a href="http://dbpedia.org/resource/JSON-LD" id="link-id0x2aabe229b9c8">JSON-LD</a>."</p> <p> <a href="http://dbpedia.org/resource/XML" id="link-id0x2aabe16aa168">XML</a> and JSON are much the same thing. While most databases have had XML support for over a decade, there is a crop of specialized JSON systems like <a href="http://dbpedia.org/resource/MongoDB" id="link-id0x2aabe0694fe8">MongoDB</a>. PostgreSQL also has a JSON datatype. Unsurprisingly, <a href="http://dbpedia.org/resource/MarkLogic" id="link-id0x2aabe0176298">MarkLogic</a> too has JSON, as this is pretty much the same thing as their core competence of XML.</p> <p> <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aab7deff0c8">Virtuoso</a>, too, naturally has a JSON parser, and mapping this to the native XML data type is a non-issue. This should probably be done.</p> <p> <a href="http://www.linkedin.com/in/stefanobertolo" id="link-id0x2aabe0d67928">Stefano Bertolo</a> of the <a href="http://dbpedia.org/resource/European_Commission" id="link-id0x2aabe0ee7358">EC</a>, also <a href="http://lod2.eu" id="link-id0x2aabe1c462c8">LOD2</a> project officer, used the word Cambrian explosion when talking about the proliferation of new database approaches in recent years.</p> <p> <a href="http://dbpedia.org/resource/Apache_Hadoop" id="link-id0x2aabe3b55fc8">Hadoop</a> is a big factor in some environments. <a href="http://www.actian.com/" id="link-id0x2aab7f623e98">Actian</a> <a href="http://dbpedia.org/resource/Vectorwise" id="link-id0x2aac10774118">Vector (née VectorWise)</a>, for example, can use this as its file system. <a href="http://dbpedia.org/resource/Hadoop_Distributed_File_System" id="link-id0x2aabe215c028">HDFS</a> is singularly cumbersome for this but still not impossible and riding the Hadoop bandwagon makes this adaptation likely worthwhile.</p> <p>Graphs are popular in database research. We have a good deal of exposure to this via <a href="http://ldbcouncil.org/" id="link-id0x2aabe3a3bef8">LDBC</a>. Going back to an API for database access, as is often done in <a href="http://dbpedia.org/resource/Graph_database" id="link-id0x2aabfd289ec8">graph database</a>, can have its point, especially as a reaction to the opaque and sometimes hard to predict query optimization of declarative languages. This just keeps getting more complex, so a counter-reaction is understandable. APIs are good if crossed infrequently and bad otherwise. So, graph database APIs will develop vectoring, is my prediction and even recommendation in LDBC deliverables.</p> <p>So, there are diverse responses to the same evolutionary pressures. These are of initial necessity one-off special-purpose systems, since the time to solution is manageable. Doing these things inside an RDBMS usually takes longer. The geek also likes to start from scratch. Well, not always, as there have been some cases of grafting some entirely non-<a href="http://dbpedia.org/resource/MySQL" id="link-id0x2aabe26dfc68">MySQL</a>-like functionality, e.g. <a href="http://dbpedia.org/resource/Infobright" id="link-id0x2aabe3fb9468">Infobright</a> and <a href="http://dbpedia.org/resource/Kickfire" id="link-id0x2aab7ec81888">Kickfire</a>, onto MySQL.</p> <p>From the Virtuoso angle, adding new data and control structures has been done many times. There is no reason why this cannot continue. The next instances will consist of some graph processing (BSP, or Bulk Synchronous Processing) in the query languages. Another recent example is an interface for pluggable specialized content indices. One can make chemical structure indices, use alternate full text indices, etc., with this.</p> <p>Most of this diversification has to do with physical design. The common logical side is a demand for more flexibility in schema and sometimes in scaling, e.g., various forms of elasticity in growing scale-out clusters, especially with the big web players.</p> <p>The diversification is a fact, but the results tend to migrate into the RDBMS given enough time.</p> <p>On the other hand, when a new species like the RDF store emerges, with products that do this and no other thing and are numerous enough to form a market, the RDBMS functionality seeps in. <a href="http://bigdata.com/bigdata" id="link-id0x2aabe18f25c8">Bigdata</a> has a sort of multicolumn table feature, if I am not mistaken. We just heard about the wish for strict schema, views, and triggers. By all means.</p> <p>From the Virtuoso angle, with structure awareness, the difference of <a href="http://www.w3.org/SQL/" id="link-id0x2aabfc9469d8">SQL</a> and RDF gradually fades, and any advance can be exploited to equal effect on either side.</p> <p>Right now, I would say we have convergence when all the experimental streams feel many of the same necessities.</p> <p>Of course you cannot have a semantic tech conference without the matter of the public <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x2aabfdc751f8">SPARQL</a> end point coming up. The answer is very simple: If you have operational need for SPARQL accessible data, you must have your own infrastructure. No public end points. Public end points are for lookups and discovery; sort of a dataset demo. If operational data is in all other instances the responsibility of the one running the operation, why should it be otherwise here? Outsourcing is of course possible, either for platform (<a href="http://dbpedia.org/resource/Cloud_computing" id="link-id0x2aabe17fbc48">cloud</a>) or software (<a href="http://dbpedia.org/resource/Software_as_a_service" id="link-id0x2aabe09ec158">SaaS</a>). To outsource something with a service level, the service level must be specifiable. A service level cannot be specified in terms of throughput with arbitrary queries but in terms of well defined transactions; hence the services world runs via APIs, as in the case of <a href="http://dbpedia.org/resource/OpenPHACTS" id="link-id0x2aabfd873108">Open PHACTS</a>. For arbitrary queries (i.e., analytics on demand), with the huge variation in performance dependent on query plans and configuration of schema, the best is to try these things with platform on demand in a cloud. Like this, there can be a clear understanding of performance, which cannot be had with an entirely uncontrolled concurrent utilization. For systems in constant operation, having one's own equipment is cheaper, but still might be impossible to procure due to governance.</p> <p>Having clarified this, the incentives for operators also become clearer. A public end point is a free evaluation; a SaaS deal or product sale is the commercial offering.</p> <p>Anyway, common datasets like <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x2aabe0fe7a68">DBpedia</a> are available preconfigured on <a href="http://dbpedia.org/resource/Amazon_Web_Services" id="link-id0x2aabe1203a78">AWS</a> with a <a href="http://virtuoso.openlinksw.com/" id="link-id0x2aabe223a398">Virtuoso</a> server. For larger data, there is a point to making ready-to-run cluster configurations available for evaluation, now that AWS has suitable equipment (e.g., dual E5 2670 with 240 GB RAM and SSD for USD 2.8 an hour). According to Amazon, up to five of these are available at a time without special request. We will try this during the fall and make the images available.</p> <h3>SEMANTiCS 2014 Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1813" id="link-id0x2aabe0a77d78">SEMANTiCS 2014 (part 1): Keynote</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1814" id="link-id0x2aabe08954a8">SEMANTiCS 2014 (part 2): RDF Data Shapes</a> </li> <li> SEMANTiCS 2014 (part 3): Conversations<i> (this post)</i> </li> </ul>
2014-09-08T16:11:10.206057-04:00
SEMANTiCS 2014 (part 2 of 3): RDF Data Shapes
http://www.openlinksw.com:443/weblog/oerling/?date=2014-09-08#1814
2014-09-08T19:22:41Z
<p>The first keynote of <a href="http://www.semantics.cc/" id="link-id0x2aab7f183ee8">Semantics 2014</a> was by <a href="https://www.linkedin.com/in/philarcher" id="link-id0x2aabe103dba8">Phil Archer</a> of the <a href="http://dbpedia.org/resource/World_Wide_Web_Consortium" id="link-id0x2aabe0eb90d8">W3C</a>, entitled "<a href="http://www.semantics.cc/programme/talks/archer-10-years-of-archivement/" id="link-id0x2aabe01e42d8">10 Years of Achievement</a>." After my talk, in the questions, Phil brought up the matter of the upcoming W3C work group charter on <a href="http://www.w3.org/2014/data-shapes/charter" id="link-id0x2aab7f3ae098">RDF Data Shapes</a>. We had discussed this already at the reception the night before and I will here give some ideas about this.</p> <p>After the talk, my answer was that naturally the existence of something that expressed the same sort of thing as <a href="http://dbpedia.org/resource/Data_definition_language" id="link-id0x2aab7f24cac8">SQL DDL</a>, with W3C backing, can only be a good thing and will give the structure awareness work by OpenLink in Virtuoso and probably others a more official seal of approval. Quite importantly, this will be a facilitator of interoperability and will raise this from a product specific optimization trick to a respectable, generally-approved piece of functionality.</p> <p>This is the general gist of the matter and can hardly be otherwise. But underneath is a whole world of details, which we discussed at the reception.</p> <p>Phil noted that there was controversy around whether a lightweight <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x2aabe3a08a68">OWL</a>-style representation or <a href="http://www.w3.org/Submission/spin-overview/" id="link-id0x2aabe0a74268">SPIN</a> should function as the basis for data shapes.</p> <p>Phil stated in the keynote that the W3C considered the RDF series of standards as good and complete, but would still have working groups for filling in gaps as these came up. This is what I had understood from my previous talks with him at the <a href="http://www.w3.org/2014/03/lgd/" id="link-id0x2aabe1cf4988">Linking Geospatial Data</a> workshop in London earlier this year.</p> <p>So, against this backdrop, as well as what I had discussed with <a href="https://www.linkedin.com/in/ralphhodgson" id="link-id0x2aabe08d3d68">Ralph Hodgson</a> of <a href="https://www.linkedin.com/company/topquadrant/" id="link-id0x2aab7ed73958">Top Quadrant</a> at a previous <a href="http://www.ldbc.eu:8090/display/TUC/Fourth+TUC+Meeting,+April+2014" id="link-id0x2aab7f44daf8">LDBC TUC meeting</a> in Amsterdam, SPIN seems to me a good fit.</p> <p>Now, it turns out that we are talking about two different use cases. Phil said that the RDF Data Shapes use case was about making explicit what applications required of data. For example, all products should have a unit price, and this should have one value that is a number.</p> <p>The SPIN proposition on the other hand, as Ralph himself put it in the <a href="http://ldbcouncil.org/" id="link-id0x2aab7fd3d158">LDBC</a> meeting, is providing to the linked data space functionality that roughly corresponds to <a href="http://dbpedia.org/resource/SQL" id="link-id0x2aabe06304d8">SQL</a> views. Well, this is one major point, but SPIN involves more than this.</p> <p>So, is it DDL or views? These are quite different. I proposed to Phil that there was in fact little point in fighting over this; best to just have two profiles.</p> <p>To be quite exact, even SQL DDL equivalence is tricky, since enforcing this requires a <a href="http://dbpedia.org/resource/Database" id="link-id0x2aabe0820508">DBMS</a>; consider, for instance, foreign key and check constraints. At the reception, Phil stressed that SPIN was certainly good but since it could not be conceived without a <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x2aab7ded80f8">SPARQL</a> implementation, it was too heavy to use as a filter for an application that, for example, just processed a stream of triples.</p> <p>The point, as I see it, is that there is a wish to have data shape enforcement, at least to a level, in a form that can apply to a stream without random access capability or general purpose query language. This can make sense for some big data style applications, like an <a href="http://dbpedia.org/resource/Extract,_transform,_load" id="link-id0x2aab7ffe64e8">ETL</a>-stage pre-cooking of data before the application. Applications mostly run against a DBMS, but in some cases, this could be a specialized map-reduce or graph analytics job also, so no low cost random access.</p> <p>My own take is that views are quite necessary, especially for complex query; this is why Virtuoso has the SPARQL macro extension. This will do, by query expansion, a large part of what general purpose inference will do, except for complex recursive cases. Simple recursive cases come down to transitivity and still fit the profile. SPIN is a more generic thing, but has a large intersection with SPARQL macro functionality.</p> <p>My other take is that structure awareness needs a way of talking about structure. This is a use case that is clearly distinct from views.</p> <p>A favorite example of mine is the business rule that a good customer is one that has ordered more than 5 times in the last year, for a total of more than so much, and has no returns or complaints. This can be stated as a macro or SPIN rule with some aggregates and existences. This cannot be stated in any of the OWL profiles. When presented with this, Phil said that this was not the use case. Fair enough. I would not want to describe what amounts to SQL DDL in these terms either.</p> <p>A related topic that has come up in other conversations is the equivalent of the trigger. One use case of this is enforcement of business rules and complex access rights for updates. So, we see that the whole <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x2aabe07f7ac8">RDBMS</a> repertoire is getting recreated.</p> <p>Now, talking from the viewpoint of the structure-aware RDF store, or the triple-stream application for that matter, I will outline some of what data shapes should do. The triggers and views matter is left out, here.</p> <p>The commonality of bulk-load, ETL, and stream processing, is that they should not rely on arbitrary database access. This would slow them down. Still, they must check the following sorts of things:</p> <ul> <li> Data types </li> <li> Presence of some required attributes </li> <li> Cardinality — e.g., a person has no more than one date of birth </li> <li> Ranges — e.g., a product's price is a positive number; gender is male/female; etc. </li> <li> Limited referential integrity — e.g., a product has one product type, and this is a subject of the RDF type product type. </li> <li> Limited intra-subject checks — e.g.. delivery date is greater-than-or-equal-to ship date. </li> </ul> <p>All these checks depend on previous triples about the subject; for example, these checks may be conditional on the subject having a certain RDF type. In a data model with a join per attribute, some joining cannot be excluded. Checking conditions that can be resolved one triple at a time is probably not enough, at least not for the structure-aware RDF store case.</p> <p>But, to avoid arbitrary joins which would require a DBMS, we have to introduce a processing window. The triples in the window must be cross-checkable within the window. With RDF set semantics, some reference data may be replicated among processing windows (e.g., files) with no ill effect.</p> <p>A version of foreign key declarations is useful. To fit within a processing window, complete enforcement may not be possible but the declaration should still be possible, a little like in SQL where one can turn off checking.</p> <p>In SQL, it is conventional to name columns by prefixing them with an abbreviation of the table name. All the TPC schemas are like that, for example. Generally in coding, it is good to prefix names with data type or subsystem abbreviation. In RDF, this is not the practice. For reuse of vocabularies, where a property may occur in anything, the namespace or other prefix denotes where the property comes from, not where it occurs.</p> <p>So, in TPC-H, <code>l_partkey</code> and <code>ps_partkey</code> are both foreign keys that refer to part, plus that <code>l_partkey</code> is also a part of a composite foreign key to <code>partsupp</code>. By RDF practices, these would be called <code>rdfh:hasPart</code>. So, depending on which subject type we have, <code>rdfh:hasPart</code> is 30:1 or 4:1. (distinct subjects:distinct objects) Due to this usage, the property's features are not dependent only on the property, but on the property plus the subject/object where it occurs.</p> <p>In the relational model, when there is a parent and a child item (one to many), the child item usually has a composite key prefixed with the parent's key, with a distinguishing column appended, e.g., <code>l_orderkey, l_linenumber</code>. In RDF, this is <code>rdfh:hasOrder</code> as a property of the <code>lineitem</code> subject. In SQL, there is no single part <code>lineitem</code> subject at all, but in RDF, one must be made since everything must be referenceable with a single value. This does not have to matter very much, as long as it is possible to declare that <code>lineitems</code> will be primarily accessed via their order. It is either this or a scan of all <code>lineitems</code>. Sometimes a group of <code>lineitems</code> are accessed by the composite foreign key of <code>l_partkey, l_suppkey</code>. There could be a composite index on these. Furthermore, for each <code>l_partkey, l_suppkey </code> in <code>lineitem</code> there exists a <code>partsupp</code>. In an RDF translation, the <code>rdfh:hasPart</code> and <code>rdfh:hasSupplier</code>, when they occur in a <code>lineitem</code> subject, specify exactly one subject of type <code>partsupp</code>. When they occur in a <code>partsupp</code> subject, they are unique as a pair. Again, because names are not explicit as to where they occur and what role they play, the referential properties do not depend only on the name, but on the name plus included data shape. Declaring and checking all this is conventional in the mainstream and actually useful for query optimization also.</p> <p>Take the other example of a social network where the <code>foaf:knows</code> edge is qualified by a date when this edge was created. This may be by reification, or more usually by an "entitized" relationship where the <code>foaf:knows</code> is made into a subject with the persons who know each other and the date of acquaintance as properties. In a SQL schema, this is a key <code>person1, person2 -> date</code>. In RDF, there are two join steps to go from <code>person1</code> to <code>person2</code>; in SQL, 1. This is eliminated by saying that the <code>foaf:knows</code> entity is usually referenced by the person1 Object or person2 Object, not the Subject identifier of the <code>foaf:knows</code>.</p> <p>This allows making the physical storage by <code>O, S, G -> O2, O3, …</code>. A secondary index with <code>S, G, O</code> still allows access by the mandatory subject identifier. In SQL, a structure like this is called a clustered table. In other words, the row is arranged contiguous with a key that is not necessarily the primary key.</p> <p>So, identifying a clustering key in RDF can be important.</p> <p>Identifying whether there are value-based accesses on a given <code>Object</code> without making the <code>Object</code> a clustering key is also important. This is equivalent to creating a secondary index in SQL. In the tradition of homogenous access by anything, such indexing may be on by default, except if the property is explicitly declared of low cardinality. For example, an index on gender makes no sense. The same is most often true of <code>rdfs:type</code>. Some properties may have many distinct values (e.g., price), but are still not good for indexing, as this makes for the extreme difference in load time between SQL and the all-indexing RDF.</p> <p>Identifying whether a column will be frequently updated is another useful thing. This will turn off indexing and use an easy-to-update physical representation. Plus, properties which are frequently updated are best put physically together. This may, for example, guide the choice between row-wise and column-wise representation. A customer's account balance and orders year-to-date would be an example of such properties.</p> <p>Some short string valued properties may be frequently returned or used as sorting keys. This requires accessing the literal via an ID in the dictionary table. Non-string literals, numbers, dates, etc., are always inlined (at least in most implementations), but strings are a special question. <a href="http://bigdata.com/bigdata" id="link-id0x2aabe20460e8">Bigdata</a> and early versions of Virtuoso would inline short ones; later versions of Virtuoso would not. So specifying, per property/class combination, a length limit for an inlined string is very high gain and trivial to do. The <a href="http://wifo5-03.informatik.uni-mannheim.de/bizer/berlinsparqlbenchmark/" id="link-id0x2aabe1f9b678">BSBM</a> explore score at large scales can get a factor of 2 gain just from inlining one label. BSBM is out of its league here, but this is still really true and yields benefits across the board. The simpler the application, the greater the win.</p> <p>If there are foreign keys, then data should be loaded with the referenced entities first. This makes dimensional clustering possible at load time. If the foreign key is frequently used for accessing the referencing item (for example, if customers are often accessed by country), then loading customers so that customers of the same country end up next to each other can result in great gains. The same applies to a time dimension, which in SQL is often done as a dimension table, but rarely so in linked data. Anyhow, if date is a frequent selection criterion, physically putting items in certain date ranges together can give great gains.</p> <p>The trick here is not necessarily to index on date, but rather to use zone maps (aka min/max index). If nearby values are together, then just storing a min-max value for thousands of consecutive column values is very compact and fast to check, provided that the rows have nearby values. <a href="http://www.actian.com" id="link-id0x2aabe0c098b8">Actian</a> Vector's (<a href="http://dbpedia.org/resource/Vectorwise" id="link-id0x2aab7fc02998">VectorWise</a>) prowess in TPC-H is in part from smart use of date order in this style.</p> <p>To recap, the data shapes desiderata from the viewpoint of guiding physical storage is as follows:</p> <p>(I will use "data shape" to mean "characteristic set," or "set of Subjects subject to the same set of constraints." A Subject belonging to a data shape may be determined either by its <code>rdfs:type</code> or by the fact of it having, within the processing window, all or some of a set of properties.)</p> <ul> <li> All normal range, domain, cardinality, optionality, etc. — Specifically, declaring something as single valued (as with SQL's <code>UNIQUE constraint) and mandatory (as with SQL's <code>NOT NULL</code> constraint) is good. </code> </li> <li> Primary access path — The Properties whose Objects are dominant access criteria is important </li> <li> No-index — Declare that no index will be made on the Object of a Property within a data shape. </li> <li> Inlined string — String values of up to so many characters in this data shape are inlined </li> <li> Clustering key — The Subject identifiers will be picked to be correlated with the Object of this Property in this data shape. This can be qualified by a number of buckets (e.g., if dates are from 2000 to 2020, then this interval may be 100 buckets), with an exception bucket for out of range values. </li> <li> No full text index — A string value will not need to be full text indexed in this Property even if full text indexing is generally on. </li> <li> Full text index desired — This means that if the value of the property is a string, then the row must be locatable via this string. The string may or may not be inlined, but an index will exist on the literal ID of the string, e.g., POSG. </li> <li> Co-location — This is akin to clustering but specifies, for a high cardinality Object, that the Subject identifier should be picked to fall in the same partition as the Object. The Object is typically a parent of the Subject being loaded; for example, the containing assembly of a sub-assembly. Traversing the assembly created in this way will be local on a scale-out system. This can also apply to geometries or text values: If primary access is by text or geo index, then the metadata represented as triples should be in the same partition as the entry in the full text/geo index. </li> <li> Update group — A set of properties that will often change together. Implies no index and some form of co-location, plus update-friendly physical representation. Many update groups may exist, in which case they may or may not be collocated. </li> <li> Composite foreign/primary key — A data shape can have a multicolumn foreign key, e.g., <code>l_partkey, l_suppkey</code> in <code>lineitem</code> with the matching primary key of <code>ps_partkey, ps_suppkey</code> in <code>partsupp</code>. This can be used for checking and for query optimization: Looking at <code>l_partkey</code> and <code>l_suppkey</code> as independent properties, the guess would be that there hardly ever exists a <code>partsupp</code>, whereas one does always exist. The XML standards stack also has a notion of a composite key for random access on multiple attributes. </li> </ul> <p>These things have the semantic of "hint for physical storage" and may all be ignored without effect on semantics, at least if the data is constraint-compliant to start with.</p> <p>These things will have some degree of reference implementation through the evolution of Virtuoso structure awareness, though not necessarily immediately. These are, to the semanticist, surely dirty low-level disgraceful un-abstractions, some of the very abominations the early semanticists abhorred or were blissfully ignorant of when they first raised their revolutionary standard.</p> <p>Still, these are well-established principles of the broader science of database. SQL does not standardize some of these, nor does it have much need to, as the use of these features is system-specific. The support varies widely and the performance impacts are diverse. However, since RDF excels as a reference model and as a data interchange format, giving these indications as hints to back-end systems cannot hurt, and can make a difference of night and day in load and query time.</p> <p>As Phil Archer said, the idea of RDF Data Shapes is for an application to say that "it will barf if it gets data that is not like this." An extension is for the data to say what the intended usage pattern is so that the system may optimize for this.</p> <p>All these things may be learned from static analysis and workload traces. The danger of this is over-fitting a particular profile. This enters a gray area in benchmarking. For big data, if RDF is to be used as the logical model and the race is about highest absolute performance, never mind what the physical model ends up being, all this and more is necessary. And if one is stretching the envelope for scale, the race is always about highest absolute performance. For this reason, these things will figure at the leading edge with or without standardization. I would say that the build-up of experience in the RDBMS world is sufficient for these things to be included as hints in a profile of data shapes. The compliance cost will be nil if these are ignored, so for the W3C, these will not make the implementation effort for compliance with an eventual data shapes recommendation prohibitive.</p> <p>The use case is primarily the data warehouse to go. If many departments or organizations publish data for eventual use by their peers, users within the organization may compose different combinations of extractions for different purposes. Exhaustive indexing of everything by default makes the process slow and needlessly expensive, as we have seen. Much of such exploration is bounded by load time. Federated approaches for analytics are just not good, even though they may work for infrequent lookups. If datasets are a commodity to be plugged in and out, the load and query investment must be minimized without the user/DBA having to run workload analysis and manual schema optimization. Therefore, bundling guidelines such as these with data shapes in a dataset manifest can do no harm and can in cases provide 10-50x gains in load speeds and 2-4x in space consumption, not to mention unbounded gains in query time, as good and bad plans easily differ by 10-100x, especially in analytics.</p> <p>So, here is the pitch: </p> <ul> <li> Dramatic gains in ad hoc user experience </li> <li> Minimal effort by data publishers, as much of the physical guidelines can be made from workload trace and dataset; the point is that the ad hoc user does not have to do this. </li> <li> Great optimization potential for system vendors; low cost for initial compliance </li> <li> Better understanding of the science of performance by the semantic community </li> </ul> <p>To be continued...</p> <h3>SEMANTiCS 2014 Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1813" id="link-id0x2aabe1f87ad8">SEMANTiCS 2014 (part 1): Keynote</a> </li> <li> SEMANTiCS 2014 (part 2): RDF Data Shapes<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1815" id="link-id0x1b13a308">SEMANTiCS 2014 (part 3): Conversation</a> </li> </ul>
2014-10-29T13:31:37.019897-04:00
SEMANTiCS 2014 (part 1 of 3): Keynote
http://www.openlinksw.com:443/weblog/oerling/?date=2014-09-08#1813
2014-09-08T17:17:15Z
<p>I was invited to give a keynote at <a href="http://www.semantics.cc" id="link-id0x2aab7f6caaf8">SEMANTiCS 2014</a> in <a href="http://dbpedia.org/resource/Leipzig" id="link-id0x2aab7f8e74e8">Leipzig</a>, <a href="http://dbpedia.org/resource/Germany" id="link-id0x2aab7f3a8128">Germany</a> last Thursday. I will here recap some of the main points, and comment on some of the ensuing controversy. The talk was initially titled <i><a href="http://www.slideshare.net/kidehen/ogl-semantic-leipzigsemantics2014" id="link-id0x2aab7f3a81c8">Virtuoso, the Prometheus of RDF</a></i>. Well, mythical Prometheus did perform a service but ended up paying for it. Still, the mythical reference is sometimes used when talking of major breakthroughs and big-gain ambitions. In the <a href="http://www.slideshare.net/kidehen/ogl-semantic-leipzigsemantics2014/2" id="link-id0x2aab7f352808">first slide</a>, I changed it to <a href="http://dbpedia.org/resource/Linked_data" id="link-id0x2aab7f2b8158">Linked Data</a> at Dawn, which is less product specific and more a reflection on the state of the linked data enterprise at large.</p> <p>The first part of the talk was under the heading of the promise and the practice. The promise we know well and find no fault with: Schema-last-ness, persistent unique identifiers, self-describing data, some but not too much inference. The applications usually involve some form of integration and often have a mix of strictly structured content with semi-structured or textual content.</p> <p>These values are by now uncontroversial and embraced by many; however, most instances of this embracing do not occur in the context of RDF as such. For example, the big online systems on the web: all have some schema-last (key-value) functionality. Applications involving long-term data retention have diverse means of having persistent IDs and self description, from <i>UUIDs</i> to having the table name in a column so that one can tell where a CSV dump came from.</p> <p>The practice involves competing with diverse alternative technologies: SQL, key-value, information retrieval (often <a href="http://dbpedia.org/resource/Lucene" id="link-id0x2aab7f3529e8">Lucene</a>-derived). In some instances, graph databases occur as alternatives: Young semanticist, do or die.</p> <p>In this race, linked data is often the prettiest and most flexible, but gets a hit on different aspects of performance and scalability. This is a database gig, and database is a performance game; make no mistake.</p> <p>After these preliminaries we come to the <i>"<a href="http://www.slideshare.net/kidehen/ogl-semantic-leipzigsemantics2014/9" id="link-id0x2aab7ef90508">RDF tax</a>,"</i> or the more or less intrinsic overheads of describing all as triples. The word "triple" is used by habit. In fact, we nearly always talk about quads, i.e., subject-predicate-object-graph (<code>SPOG</code>). The next slide is provocatively titled the <i><a href="http://www.slideshare.net/kidehen/ogl-semantic-leipzigsemantics2014/10" id="link-id0x2aab7f63ff58">Bane of the Triple</a>,</i> and is about why having all as triples is, on the surface, much like relational, except it makes life hard, where tables make it at least manageable, if still not altogether trivial.</p> <p>The very first statement on <a href="http://www.slideshare.net/kidehen/ogl-semantic-leipzigsemantics2014/9" id="link-id0x2aab7f0e8098">the tax slide</a> reads "90% of bad performance comes from non-optimal query plans." If one does triples in the customary way (i.e., a table of quads plus dictionary tables to map URIs and literal strings to internal IDs), one incurs certain fixed costs.</p> <p>These costs are deemed acceptable by users who deploy linked data. If these costs were not acceptable, the proof of concept would have already disqualified linked data.</p> <p>The support cases that come my way are nearly always about things taking too much time. Much less frequently, are these about something unambiguously not working. Database has well defined semantics, so whether something works or not is clear cut.</p> <p>So, support cases are overwhelmingly about query optimization. The problems fall in two categories:</p> <ul> <li> The plan is good in the end, but it takes much longer to make the plan than to execute it. </li> <li> The plan either does the wrong things or does things in the wrong order, but produces a correct result. </li> </ul> <p>Getting no plan at all or getting a clearly wrong result is much less frequent.</p> <p>If the RDF overheads incurred with a good query plan were show stoppers, the show would have already stopped.</p> <p>So, let's look at this in more detail; then we will talk about the fixed overheads.</p> <p>The join selectivity of triple patterns is correlated. Some properties occur together all the time; some occur rarely; some not at all. Some property values can be correlated, i.e., order number and order date. Capturing these by sampling in a multicolumn table is easy; capturing this in triples would require doing the join in the cost model, which is not done since it would further extend compilation times. When everything is a join, selectivity estimation errors build up fast. When everything is a join, the space of possible graph query plans explodes as opposed to tables; thus, while the full plan space can be covered with 7 tables, it cannot be covered with 18 triple patterns. This is not factorial (number of permutations). For different join types (index/hash) and the different compositions of the hash build side, this is much worse, in some nameless outer space fringe of non-polynomiality.</p> <p> <a href="http://www.tpc.org/tpch/" id="link-id0x2aab7f352ba8">TPC-H</a> can be run with success because the cost model hits the right plan every time. The primary reason for this is the fact that the schema and queries unambiguously suggest the structure, even without foreign key declarations. The other reason is that with a handful of tables, all plans can be reviewed, and the cost model reliably tells how many rows will result from each sequence of operations.</p> <p>Try this with triples; you will know what I mean.</p> <p>Now, some people have suggested purely rule-based models of SPARQL query compilation. These are arguably faster to run and more predictable. But the thing that must be done, yet will not be done with these, is the right trade-off between index and hash. This is the crux of the matter, and without this, one can forget about anything but lookups. The choice depends on reliable estimation of cardinality (number of rows, number of distinct keys) on either side of the join. Quantity, not pattern matching.</p> <p>Well, many linked data applications are lookups. The graph database API world is sometimes attractive because it gives manual control. Map reduce in the analytical space is sometimes attractive for the same reason.</p> <p>On the other hand, query languages also give manual control, but then this depends on system specific hints and cheats. People are often black and white: Either all declarative or all imperative. We stand for declarative, but still allow physical control of plan, like most DBMS.</p> <p>To round off, I will give a concrete example:</p> <blockquote> <code><pre> { ?thing rdfs:label ?lbl . ?thing dc:title ?title . ?lbl bif:contains "gizmo" . ?title bif:contains "widget" . ?thing a xx:Document . ?thing dc:date ?dt . FILTER ( ?dt > "2014-01-01"^^xsd:date ) }</pre> </code> </blockquote> <p>There are two full text conditions, one date, and one class, all on the same subject. How do you do this? Most selective text first, then get the data and check, then check the second full text given the literal and the condition, then check the class? Wrong. If widgets and gizmos are both frequent and most documents new, this is very bad because using a text index to check for a specific ID having a specific string is not easily vectorable. So, the right plan is: Take the more selective text expression, then check the date and class for the results, put the <code><i>?things</i></code> in a hash table. Then do the less selective text condition, and drop the ones that are not in the hash table. Easily 10x better. Simple? In the end yes, but you do not know this unless you know the quantities.</p> <p>This gives the general flavor of the problem. Doing this with TPC-H in RDF is way harder, but you catch my drift.</p> <p>Each individual instance is do-able. Having closer and closer alignment between reality and prediction will improve the situation indefinitely, but since the space is as good as infinite there cannot be a guarantee of optimality except for toy cases.</p> <p>The <a href="http://dbpedia.org/resource/Gordian_Knot" id="link-id0x2aab7f4df478">Gordian Knot</a> shall not be defeated with pincers but by the sword.</p> <p>We will come to this in a bit.</p> <p>Now, let us talk of the fixed overheads. The embarrassments are in the query optimization domain; the daily grind, relative cost, and provisioning are in this one.</p> <p>The overheads come from:</p> <ul> <li> Indexing everything </li> <li> Having literals and URI strings via dictionary </li> <li> Having a join for every attribute </li> </ul> <p>These all fall under the category of having little to no physical design room.</p> <p>In the indexing everything department, we load 100 GB TPC-H in 15 minutes in SQL with ordering only on primary keys and almost no other indexing. The equivalent with triples is around 12 hours. This data can be found on this blog (<a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aab7f352c08">TPC-H series</a> and <i><a href="http://www.openlinksw.com/weblog/oerling/?id=1769" id="link-id0x2aab7f3a81e8">Meeting the Challenges of Linked Data in the Enterprise</a></i>). This is on the order of confusing a screwdriver with a hammer. If the nail is not too big, the wood not too hard, and you hit it just right — the nail might still go in. The RDF bulk load is close to the fastest possible given the general constraints of what it does. The same logic is used for the record-breaking 15 minutes of TPC-H bulk load, so the code is good. But indexing everything is just silly.</p> <p>The second, namely the dictionary of URIs and literals, is a dual edge. I talked to <a href="http://www.linkedin.com/pub/bryan-thompson/30/390/54b" id="link-id0x2aab7f0e8058">Bryan Thompson</a> of <a href="http://www.systap.com/about" id="link-id0x2aab7f6cacd8">SYSTAP</a> (<a href="http://bigdata.com/bigdata" id="link-id0x2aab7f3a3df8">Bigdata</a> RDF store) in <a href="http://dbpedia.org/page/Washington,_D.C." id="link-id0x2aab7f290278">D.C.</a> at the <a href="http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?reload=true&punumber=6226952" id="link-id0x2aab7fa3e688">ICDE</a> there. He said that they do short strings inline and long ones via dictionary. I said we used to do the same but stopped in the interest of better compression. What is best depends on workload and working-set-to-memory ratio. But if you must make the choice once and for all, or at least as a database-wide global setting, you are between a rock and a hard place. Physical vs. logical design, again.</p> <p>The other aspect of this is the applications that do regexps on URI strings or literals. Doing this is like driving a Formula 1 race in reverse gear. Use a text index. Always. This is why most implementations have one even though SPARQL itself makes no provisions for this. If you really need regexps, and on supposedly opaque URIs at that, tokenize them and put them in a text index as a text literal. Or if an inverted-file-word index is really not what you need, use a trigram one. So far, nobody has wanted one hard enough for us to offer this, even though this is easy enough. But special indices for special data types (e.g., chemical structure) are sometimes wanted, and we have a generic solution for all this, to be introduced shortly on this blog. Again, physical design.</p> <p>I deliberately name the self-join-per-attribute point last, even though this is often the first and only intrinsic overhead that is named. True, if the physical model is triples, each attribute is a join against the triple table. Vectored execution and right use of hash-join help, though. The Star Schema Benchmark SQL to SPARQL gap is only 2.5x, as <a href="http://www.openlinksw.com/weblogs/oerling/?id=1732" id="link-id0x2aab7eebb0b8">documented last year on this blog</a>. This makes SPARQL win by 100+x against <a href="http://dbpedia.org/resource/MySQL" id="link-id0x2aab7fa3e6e8">MySQL</a> and lose by only 0.8x against column store pioneer <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x2aab7f82e4c8">MonetDB</a>. Let it be said that this is so far the best case and that the gap is wider in pretty much all other cases. This gap is well and truly due to the self-join matter, even after the self-joins are done vectored, local, ordered; in one word, <i>right.</i> The literal and URI translation matter plays no role here. The needless indexing hurts at load but has no effect at query time, since none of the bloat participates in the running. Again, physical design.</p> <p>Triples are done right, so?</p> <p>In the summer of 2013, after the Star Schema results, it became clear that maybe further gains could be had and query optimization made smoother and more predictable, but that these would be paths of certain progress but with diminishing returns per effort. No, not the pincers; give me the sword. So, between fall 2013 and spring 2014, aside from doing diverse maintenance, I did the <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aab7f32ca38">TPC-H series</a>. This is the proficiency run for big league databases; the America's Cup, not a regatta on the semantic lake.</p> <p>Even if the audience is principally Linked Data, the baseline must be that of the senior science of SQL.</p> <p>It stands to reason and has been demonstrated by extensive experimentation at <a href="http://www.cwi.nl" id="link-id0x2aab7f589578">CWI</a> that RDF data, by and large, has structure. This structure will carry linked data through the last mile to being a real runner against the alternative technologies (SQL, IR, key value) mentioned earlier.</p> <p>The operative principles have been mentioned earlier and are set forth on the slides. In forthcoming articles I will display some results.</p> <p>One important proposal for structure awareness was by <a href="https://www.linkedin.com/pub/thomas-neumann/2/35/189" id="link-id0x2aab7f589318">Thomas Neumann</a> in an <a href="https://domino.mpi-inf.mpg.de/intranet/ag5/ag5publ.nsf/0/AD3DBAFA6FB90DD2C1257593002FF3DF/$file/rdf3x.pdf" id="link-id0x2aab7ec91688">RDF3X paper introducing characteristic sets</a>. There, the application was creation of more predictable cost estimates. Neumann correctly saw this as possibly the greatest barrier to predictable RDF performance. <a href="http://homepages.cwi.nl/~boncz/" id="link-id0x2aab7f32cb58">Peter Boncz</a> and I discussed the use of this for physical optimization once when driving back to <a href="http://dbpedia.org/resource/Amsterdam" id="link-id0x2aab7f589378">Amsterdam</a> from a <a href="http://lod2.eu/" id="link-id0x2aab7eebaff8">LOD2</a> review in <a href="http://dbpedia.org/resource/Luxembourg" id="link-id0x2aab7f589478">Luxembourg</a>. <a href="http://homepages.cwi.nl/~duc/" id="link-id0x2aab7f589498">Pham Minh Duc</a> of CWI did much of the schema discovery research, documented in the now published LOD2 book (<i><a href="http://link.springer.com/book/10.1007%2F978-3-319-09846-3" id="link-id0x2aab7f589538">Linked Open Data -- Creating Knowledge Out of Interlinked Data</a></i>). The initial Virtuoso implementation had to wait for the TPC-H and general squeezing of the quads model to be near complete. It will likely turn out that the greatest gain of all with structure awareness will be bringing optimization predictability to SQL levels. This will open the whole bag of tricks known to data warehousing to safe deployment for linked data. Of course, much of this has to do with exploiting physical layout; hence it also needs the physical model to be adapted. Many of these techniques have high negative impact if used in the wrong place; hence the cost model must guess right. But they work in SQL and, as per Thomas Neumann's initial vision, there is no reason why these would not do so in a schema-less model if adapted in a smart enough manner.</p> <p>All this gives rise to some sociological or psychological observations. <a href="http://jens-lehmann.org/" id="link-id0x2aab7f5892f8">Jens Lehmann</a> asked me why now, why not earlier; after all, over the years many people have suggested property tables and other structured representations. This is now because there is no further breakthroughs within an undifferentiated physical model.</p> <p>For completeness, we must here mention other approaches to alternative, if still undifferentiated, physical models. A number of research papers mention memory-only, pointer-based (i.e., no index, no hash-join) implementations of triples or quads. Some of these are on graph processing frameworks, some stand-alone. <a href="http://www.yarcdata.com" id="link-id0x2aab7f589258">Yarc Data</a> is a commercial implementation that falls in this category. These may have higher top speeds than column stores, even after all vectoring and related optimizations. However the space utilization is perforce larger than with optimum column compression and this plus the requirement of 100% in memory makes these more expensive to scale. The linked data proposition is usually about integration, and this implies initially large data even if not all ends up being used.</p> <p>The graph analytics, pointer-based item will be specially good for a per-application extraction, as suggested by <a href="http://dbpedia.org/resource/Oracle_Corporation" id="link-id0x2aab7f5893f8">Oracle</a> in their paper at <a href="http://event.cwi.nl/grades2013/" id="link-id0x2aab7f589518">GRADES 13</a>. No doubt this will come under discussion at <a href="http://ldbcouncil.org/" id="link-id0x2aab7f589558">LDBC</a>, where Oracle Labs is now a participant.</p> <p>But back to physical model. What we have in mind is relational column store — multicolumn-ordered column-wise compressed tables — a bit like <a href="http://www.vertica.com" id="link-id0x2aab7f589598">Vertica</a> and <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aab7f589918">Virtuoso</a> in SQL mode for the regular parts and quads for the rest. What is big is regular, since a big thing perforce comes from something that happens a lot, like click streams, commercial transactions, instrument readings. For the 8-lane-motorway of regular data, you get the F1 racer with the hardcore best in column store tech. When the autobahn ends and turns into the mountain trail, the engine morphs into a dirt bike.</p> <p>This is complex enough, and until all the easy gains have been extracted from quads, there is little incentive. Plus this has the prerequisite of quads done right, plus the need for top of the line relational capability for not falling on your face once the speedway begins.</p> <p> <a href="http://world.marklogic.com/locations/amsterdam/speakers/stephen-buxton/" id="link-id0x2aab7f589838">Steve Buxton</a> of <a href="http://dbpedia.org/resource/MarkLogic" id="link-id0x2aab7f5891b8">MarkLogic</a> gave a talk right before mine. Coming from a document-centric world, it stands to reason that MarkLogic would have a whole continuum of different mixes between SPARQL and document oriented queries. Steve correctly observed that some users found this great; others found this a near blasphemy, an unholy heterodoxy of confusing distinct principles.</p> <p>This is our experience as well, since usage of <a href="http://dbpedia.org/resource/XML" id="link-id0x2aab7f36c368">XML</a> fragments in SPARQL with <a href="http://dbpedia.org/resource/XPath" id="link-id0x2aab7f5896d8">XPath</a> and such things in Virtuoso is possible but very seldom practiced. This is not the same as MarkLogic, though, as MarkLogic is about triples-in-documents, and the Virtuoso take is more like documents-in-triples. Not to mention that use of SQL and stored procedures in Virtuoso is rare among the SPARQL users.</p> <p>The whole thing about the absence of physical design in RDF is a related, but broader instance of such purism.</p> <p>In my talk, I had a slide titled <i><a href="http://www.slideshare.net/kidehen/ogl-semantic-leipzigsemantics2014/17" id="link-id0x2aab7f4baba8">The Cycle of Adventure</a>,</i> generally philosophizing on the dynamics of innovation. All progress begins with an irritation with the status quo; to mention a few examples: the No-SQL rebellion; the rejection of parallel SQL database in favor of key-value and map-reduce; the admission that central schema authority at web scale is impossible; the anti-ACID stance when having wide-area geographies to deal with. The stage of radicalism tends to discard the baby with the bathwater. But when the purists have their own enclave, free of the noxious corruption of the rejected world, they find that life is hard and defects of human character persist, even when all subscribe to the same religion. Of course, here we may have further splinter groups. After this, the dogma adapts to reality: the truly valuable insights of the original rebellion gain in appreciation, and the extremism becomes more moderate. Finally there is integration with mainstream, which becomes enriched by new content.</p> <p>By the time the term Linked Data came to broad use, the RDF enterprise had its break-away colonies that started to shed some of the initial zeal. By now, we have the last phase of reconciliation in its early stages.</p> <p>This process is in principle complete when linked data is no longer a radical bet, but a technology to be routinely applied to data when the nature of the data fits the profile. The structure awareness and other technology discussed here will mostly eliminate the differential in deployment cost.</p> <p>The spreading perception of an expertise gap in this domain will even-out the cost in terms of personnel. The flexibility gains that were the initial drive for the movement will be more widely enjoyed when these factors fuel broader adoption.</p> <p>To help this along, we have <a href="http://ldbcouncil.org" id="link-id0x2aab7f4baac8">LDBC, the Linked Data Benchmark Council</a>, with the agenda of creating industry consensus on measuring progress across the linked data and graph DB frontiers. I duly invited MarkLogic to join.</p> <p>There were many other interesting conversations at the conference, I will later comment on these.</p> <p>To be continued...</p> <h3>SEMANTiCS 2014 Series</h3> <ul> <li> SEMANTiCS 2014 (part 1): Keynote<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1814" id="link-id0x2aab7f4ba9e8">SEMANTiCS 2014 (part 2): RDF Data Shapes</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1815" id="link-id0x1baff828">SEMANTiCS 2014 (part 3): Conversation</a> </li> </ul>
2014-09-08T16:12:46.770-04:00
SEMANTiCS 2014 (part 3 of 3): Conversations
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-09-07#1812
2014-09-07T17:54:16Z
<p>I was asked for an oracular statement about the future of <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x2aabe22b2ef8">relational database (RDBMS)</a> at the conference. The answer, without doubt or hesitation, is that this is forever. But this does not mean that the RDBMS world would be immutable, quite the opposite.</p> <p>The specializations converge. The RDBMS becomes more adaptable and less schema-first. Of course the RDBMS also take new data models beside the relational. <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x2aabfe2a02e8">RDF</a> and other property graph models, for instance.</p> <p>The schema-last-ness is now well in evidence. For example, <a href="http://dbpedia.org/resource/PostgreSQL" id="link-id0x2aabe06ebcf8">PostgreSQL</a> has an <i><code>hstore</code></i> column type which is a list of key-value pairs. <a href="http://dbpedia.org/resource/Vertica" id="link-id0x2aabf4750a98">Vertica</a> has a feature called <i>flex tables</i> where a column can be added on a row-by-row basis.</p> <p>Specialized indexing for text and geometries is a well established practice. However, dedicated IR systems, often <a href="http://dbpedia.org/resource/Lucene" id="link-id0x2aabe1e34f18">Lucene</a> derivatives, can offer more transparency in the IR domain for things like vector-space-models and hit-scoring. There is specialized faceted search support which is quite good. I do not know of an RDBMS that would do the exact same trick as Lucene for facets, but, of course, in the forever expanding scope of RDB, this is added easily enough.</p> <p> <a href="http://dbpedia.org/resource/JSON" id="link-id0x2aabe0021578">JSON</a> is all the rage in the web developer world. <a href="https://www.linkedin.com/in/philarcher" id="link-id0x2aabfdfacf88">Phil Archer</a> even said in his keynote, as a parody of the web developer: " I will never touch that crap of RDF or the semantic web; this is a pipe dream of reality ignoring academics and I will not have it. I will only use <a href="http://dbpedia.org/resource/JSON-LD" id="link-id0x2aabfc9baea8">JSON-LD</a>."</p> <p> <a href="http://dbpedia.org/resource/XML" id="link-id0x2aabe38e34a8">XML</a> and JSON are much the same thing. While most databases have had XML support for over a decade, there is a crop of specialized JSON systems like <a href="http://dbpedia.org/resource/MongoDB" id="link-id0x2aabe17f25f8">MongoDB</a>. PostgreSQL also has a JSON datatype. Unsurprisingly, <a href="http://dbpedia.org/resource/MarkLogic" id="link-id0x2aabe06204e8">MarkLogic</a> too has JSON, as this is pretty much the same thing as their core competence of XML.</p> <p> <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aabe098c068">Virtuoso</a>, too, naturally has a JSON parser, and mapping this to the native XML data type is a non-issue. This should probably be done.</p> <p> <a href="http://www.linkedin.com/in/stefanobertolo" id="link-id0x2aabfdbcfa48">Stefano Bertolo</a> of the <a href="http://dbpedia.org/resource/European_Commission" id="link-id0x2aabe1f79dd8">EC</a>, also <a href="http://lod2.eu" id="link-id0x2aabe220d658">LOD2</a> project officer, used the word Cambrian explosion when talking about the proliferation of new database approaches in recent years.</p> <p> <a href="http://dbpedia.org/resource/Apache_Hadoop" id="link-id0x2aabfd687d78">Hadoop</a> is a big factor in some environments. <a href="http://www.actian.com/" id="link-id0x2aabe1c5d218">Actian</a> <a href="http://dbpedia.org/resource/Vectorwise" id="link-id0x2aab7f733468">Vector (née VectorWise)</a>, for example, can use this as its file system. <a href="http://dbpedia.org/resource/Hadoop_Distributed_File_System" id="link-id0x2aabe1ced978">HDFS</a> is singularly cumbersome for this but still not impossible and riding the Hadoop bandwagon makes this adaptation likely worthwhile.</p> <p>Graphs are popular in database research. We have a good deal of exposure to this via <a href="http://ldbcouncil.org/" id="link-id0x2aab7f55c6d8">LDBC</a>. Going back to an API for database access, as is often done in <a href="http://dbpedia.org/resource/Graph_database" id="link-id0x2aabe17383b8">graph database</a>, can have its point, especially as a reaction to the opaque and sometimes hard to predict query optimization of declarative languages. This just keeps getting more complex, so a counter-reaction is understandable. APIs are good if crossed infrequently and bad otherwise. So, graph database APIs will develop vectoring, is my prediction and even recommendation in LDBC deliverables.</p> <p>So, there are diverse responses to the same evolutionary pressures. These are of initial necessity one-off special-purpose systems, since the time to solution is manageable. Doing these things inside an RDBMS usually takes longer. The geek also likes to start from scratch. Well, not always, as there have been some cases of grafting some entirely non-<a href="http://dbpedia.org/resource/MySQL" id="link-id0x2aabfe29b628">MySQL</a>-like functionality, e.g. <a href="http://dbpedia.org/resource/Infobright" id="link-id0x2aab7f9e0338">Infobright</a> and <a href="http://dbpedia.org/resource/Kickfire" id="link-id0x2aabe314aa88">Kickfire</a>, onto MySQL.</p> <p>From the Virtuoso angle, adding new data and control structures has been done many times. There is no reason why this cannot continue. The next instances will consist of some graph processing (BSP, or Bulk Synchronous Processing) in the query languages. Another recent example is an interface for pluggable specialized content indices. One can make chemical structure indices, use alternate full text indices, etc., with this.</p> <p>Most of this diversification has to do with physical design. The common logical side is a demand for more flexibility in schema and sometimes in scaling, e.g., various forms of elasticity in growing scale-out clusters, especially with the big web players.</p> <p>The diversification is a fact, but the results tend to migrate into the RDBMS given enough time.</p> <p>On the other hand, when a new species like the RDF store emerges, with products that do this and no other thing and are numerous enough to form a market, the RDBMS functionality seeps in. <a href="http://bigdata.com/bigdata" id="link-id0x2aabfd94e978">Bigdata</a> has a sort of multicolumn table feature, if I am not mistaken. We just heard about the wish for strict schema, views, and triggers. By all means.</p> <p>From the Virtuoso angle, with structure awareness, the difference of <a href="http://www.w3.org/SQL/" id="link-id0x2aabe33bba48">SQL</a> and RDF gradually fades, and any advance can be exploited to equal effect on either side.</p> <p>Right now, I would say we have convergence when all the experimental streams feel many of the same necessities.</p> <p>Of course you cannot have a semantic tech conference without the matter of the public <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x2aabe0fcfce8">SPARQL</a> end point coming up. The answer is very simple: If you have operational need for SPARQL accessible data, you must have your own infrastructure. No public end points. Public end points are for lookups and discovery; sort of a dataset demo. If operational data is in all other instances the responsibility of the one running the operation, why should it be otherwise here? Outsourcing is of course possible, either for platform (<a href="http://dbpedia.org/resource/Cloud_computing" id="link-id0x2aabe1f6ec38">cloud</a>) or software (<a href="http://dbpedia.org/resource/Software_as_a_service" id="link-id0x2aabe1d1c388">SaaS</a>). To outsource something with a service level, the service level must be specifiable. A service level cannot be specified in terms of throughput with arbitrary queries but in terms of well defined transactions; hence the services world runs via APIs, as in the case of <a href="http://dbpedia.org/resource/OpenPHACTS" id="link-id0x2aabf45b03e8">Open PHACTS</a>. For arbitrary queries (i.e., analytics on demand), with the huge variation in performance dependent on query plans and configuration of schema, the best is to try these things with platform on demand in a cloud. Like this, there can be a clear understanding of performance, which cannot be had with an entirely uncontrolled concurrent utilization. For systems in constant operation, having one's own equipment is cheaper, but still might be impossible to procure due to governance.</p> <p>Having clarified this, the incentives for operators also become clearer. A public end point is a free evaluation; a SaaS deal or product sale is the commercial offering.</p> <p>Anyway, common datasets like <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x2aab7fa73dc8">DBpedia</a> are available preconfigured on <a href="http://dbpedia.org/resource/Amazon_Web_Services" id="link-id0x2aabe1b38bf8">AWS</a> with a <a href="http://virtuoso.openlinksw.com/" id="link-id0x2aabe17f2598">Virtuoso</a> server. For larger data, there is a point to making ready-to-run cluster configurations available for evaluation, now that AWS has suitable equipment (e.g., dual E5 2670 with 240 GB RAM and SSD for USD 2.8 an hour). According to Amazon, up to five of these are available at a time without special request. We will try this during the fall and make the images available.</p> <h3>SEMANTiCS 2014 Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1813" id="link-id0x2aabe156c7d8">SEMANTiCS 2014 (part 1): Keynote</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1814" id="link-id0x2aabe1f88178">SEMANTiCS 2014 (part 2): RDF Data Shapes</a> </li> <li> SEMANTiCS 2014 (part 3): Conversations<i> (this post)</i> </li> </ul>
2014-09-08T16:12:34.860931-04:00
SEMANTiCS 2014 (part 2 of 3): RDF Data Shapes
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-09-07#1811
2014-09-07T17:13:13Z
<p>The first keynote of <a href="http://www.semantics.cc/" id="link-id0x2aab7f183ee8">Semantics 2014</a> was by <a href="https://www.linkedin.com/in/philarcher" id="link-id0x2aabe103dba8">Phil Archer</a> of the <a href="http://dbpedia.org/resource/World_Wide_Web_Consortium" id="link-id0x2aabe0eb90d8">W3C</a>, entitled "<a href="http://www.semantics.cc/programme/talks/archer-10-years-of-archivement/" id="link-id0x2aabe01e42d8">10 Years of Achievement</a>." After my talk, in the questions, Phil brought up the matter of the upcoming W3C work group charter on <a href="http://www.w3.org/2014/data-shapes/charter" id="link-id0x2aab7f3ae098">RDF Data Shapes</a>. We had discussed this already at the reception the night before and I will here give some ideas about this.</p> <p>After the talk, my answer was that naturally the existence of something that expressed the same sort of thing as <a href="http://dbpedia.org/resource/Data_definition_language" id="link-id0x2aab7f24cac8">SQL DDL</a>, with W3C backing, can only be a good thing and will give the structure awareness work by OpenLink in Virtuoso and probably others a more official seal of approval. Quite importantly, this will be a facilitator of interoperability and will raise this from a product specific optimization trick to a respectable, generally-approved piece of functionality.</p> <p>This is the general gist of the matter and can hardly be otherwise. But underneath is a whole world of details, which we discussed at the reception.</p> <p>Phil noted that there was controversy around whether a lightweight <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x2aabe3a08a68">OWL</a>-style representation or <a href="http://www.w3.org/Submission/spin-overview/" id="link-id0x2aabe0a74268">SPIN</a> should function as the basis for data shapes.</p> <p>Phil stated in the keynote that the W3C considered the RDF series of standards as good and complete, but would still have working groups for filling in gaps as these came up. This is what I had understood from my previous talks with him at the <a href="http://www.w3.org/2014/03/lgd/" id="link-id0x2aabe1cf4988">Linking Geospatial Data</a> workshop in London earlier this year.</p> <p>So, against this backdrop, as well as what I had discussed with <a href="https://www.linkedin.com/in/ralphhodgson" id="link-id0x2aabe08d3d68">Ralph Hodgson</a> of <a href="https://www.linkedin.com/company/topquadrant/" id="link-id0x2aab7ed73958">Top Quadrant</a> at a previous <a href="http://www.ldbc.eu:8090/display/TUC/Fourth+TUC+Meeting,+April+2014" id="link-id0x2aab7f44daf8">LDBC TUC meeting</a> in Amsterdam, SPIN seems to me a good fit.</p> <p>Now, it turns out that we are talking about two different use cases. Phil said that the RDF Data Shapes use case was about making explicit what applications required of data. For example, all products should have a unit price, and this should have one value that is a number.</p> <p>The SPIN proposition on the other hand, as Ralph himself put it in the <a href="http://ldbcouncil.org/" id="link-id0x2aab7fd3d158">LDBC</a> meeting, is providing to the linked data space functionality that roughly corresponds to <a href="http://dbpedia.org/resource/SQL" id="link-id0x2aabe06304d8">SQL</a> views. Well, this is one major point, but SPIN involves more than this.</p> <p>So, is it DDL or views? These are quite different. I proposed to Phil that there was in fact little point in fighting over this; best to just have two profiles.</p> <p>To be quite exact, even SQL DDL equivalence is tricky, since enforcing this requires a <a href="http://dbpedia.org/resource/Database" id="link-id0x2aabe0820508">DBMS</a>; consider, for instance, foreign key and check constraints. At the reception, Phil stressed that SPIN was certainly good but since it could not be conceived without a <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x2aab7ded80f8">SPARQL</a> implementation, it was too heavy to use as a filter for an application that, for example, just processed a stream of triples.</p> <p>The point, as I see it, is that there is a wish to have data shape enforcement, at least to a level, in a form that can apply to a stream without random access capability or general purpose query language. This can make sense for some big data style applications, like an <a href="http://dbpedia.org/resource/Extract,_transform,_load" id="link-id0x2aab7ffe64e8">ETL</a>-stage pre-cooking of data before the application. Applications mostly run against a DBMS, but in some cases, this could be a specialized map-reduce or graph analytics job also, so no low cost random access.</p> <p>My own take is that views are quite necessary, especially for complex query; this is why Virtuoso has the SPARQL macro extension. This will do, by query expansion, a large part of what general purpose inference will do, except for complex recursive cases. Simple recursive cases come down to transitivity and still fit the profile. SPIN is a more generic thing, but has a large intersection with SPARQL macro functionality.</p> <p>My other take is that structure awareness needs a way of talking about structure. This is a use case that is clearly distinct from views.</p> <p>A favorite example of mine is the business rule that a good customer is one that has ordered more than 5 times in the last year, for a total of more than so much, and has no returns or complaints. This can be stated as a macro or SPIN rule with some aggregates and existences. This cannot be stated in any of the OWL profiles. When presented with this, Phil said that this was not the use case. Fair enough. I would not want to describe what amounts to SQL DDL in these terms either.</p> <p>A related topic that has come up in other conversations is the equivalent of the trigger. One use case of this is enforcement of business rules and complex access rights for updates. So, we see that the whole <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x2aabe07f7ac8">RDBMS</a> repertoire is getting recreated.</p> <p>Now, talking from the viewpoint of the structure-aware RDF store, or the triple-stream application for that matter, I will outline some of what data shapes should do. The triggers and views matter is left out, here.</p> <p>The commonality of bulk-load, ETL, and stream processing, is that they should not rely on arbitrary database access. This would slow them down. Still, they must check the following sorts of things:</p> <ul> <li> Data types </li> <li> Presence of some required attributes </li> <li> Cardinality — e.g., a person has no more than one date of birth </li> <li> Ranges — e.g., a product's price is a positive number; gender is male/female; etc. </li> <li> Limited referential integrity — e.g., a product has one product type, and this is a subject of the RDF type product type. </li> <li> Limited intra-subject checks — e.g.. delivery date is greater-than-or-equal-to ship date. </li> </ul> <p>All these checks depend on previous triples about the subject; for example, these checks may be conditional on the subject having a certain RDF type. In a data model with a join per attribute, some joining cannot be excluded. Checking conditions that can be resolved one triple at a time is probably not enough, at least not for the structure-aware RDF store case.</p> <p>But, to avoid arbitrary joins which would require a DBMS, we have to introduce a processing window. The triples in the window must be cross-checkable within the window. With RDF set semantics, some reference data may be replicated among processing windows (e.g., files) with no ill effect.</p> <p>A version of foreign key declarations is useful. To fit within a processing window, complete enforcement may not be possible but the declaration should still be possible, a little like in SQL where one can turn off checking.</p> <p>In SQL, it is conventional to name columns by prefixing them with an abbreviation of the table name. All the TPC schemas are like that, for example. Generally in coding, it is good to prefix names with data type or subsystem abbreviation. In RDF, this is not the practice. For reuse of vocabularies, where a property may occur in anything, the namespace or other prefix denotes where the property comes from, not where it occurs.</p> <p>So, in TPC-H, <code>l_partkey</code> and <code>ps_partkey</code> are both foreign keys that refer to part, plus that <code>l_partkey</code> is also a part of a composite foreign key to <code>partsupp</code>. By RDF practices, these would be called <code>rdfh:hasPart</code>. So, depending on which subject type we have, <code>rdfh:hasPart</code> is 30:1 or 4:1. (distinct subjects:distinct objects) Due to this usage, the property's features are not dependent only on the property, but on the property plus the subject/object where it occurs.</p> <p>In the relational model, when there is a parent and a child item (one to many), the child item usually has a composite key prefixed with the parent's key, with a distinguishing column appended, e.g., <code>l_orderkey, l_linenumber</code>. In RDF, this is <code>rdfh:hasOrder</code> as a property of the <code>lineitem</code> subject. In SQL, there is no single part <code>lineitem</code> subject at all, but in RDF, one must be made since everything must be referenceable with a single value. This does not have to matter very much, as long as it is possible to declare that <code>lineitems</code> will be primarily accessed via their order. It is either this or a scan of all <code>lineitems</code>. Sometimes a group of <code>lineitems</code> are accessed by the composite foreign key of <code>l_partkey, l_suppkey</code>. There could be a composite index on these. Furthermore, for each <code>l_partkey, l_suppkey </code> in <code>lineitem</code> there exists a <code>partsupp</code>. In an RDF translation, the <code>rdfh:hasPart</code> and <code>rdfh:hasSupplier</code>, when they occur in a <code>lineitem</code> subject, specify exactly one subject of type <code>partsupp</code>. When they occur in a <code>partsupp</code> subject, they are unique as a pair. Again, because names are not explicit as to where they occur and what role they play, the referential properties do not depend only on the name, but on the name plus included data shape. Declaring and checking all this is conventional in the mainstream and actually useful for query optimization also.</p> <p>Take the other example of a social network where the <code>foaf:knows</code> edge is qualified by a date when this edge was created. This may be by reification, or more usually by an "entitized" relationship where the <code>foaf:knows</code> is made into a subject with the persons who know each other and the date of acquaintance as properties. In a SQL schema, this is a key <code>person1, person2 -> date</code>. In RDF, there are two join steps to go from <code>person1</code> to <code>person2</code>; in SQL, 1. This is eliminated by saying that the <code>foaf:knows</code> entity is usually referenced by the person1 Object or person2 Object, not the Subject identifier of the <code>foaf:knows</code>.</p> <p>This allows making the physical storage by <code>O, S, G -> O2, O3, …</code>. A secondary index with <code>S, G, O</code> still allows access by the mandatory subject identifier. In SQL, a structure like this is called a clustered table. In other words, the row is arranged contiguous with a key that is not necessarily the primary key.</p> <p>So, identifying a clustering key in RDF can be important.</p> <p>Identifying whether there are value-based accesses on a given <code>Object</code> without making the <code>Object</code> a clustering key is also important. This is equivalent to creating a secondary index in SQL. In the tradition of homogenous access by anything, such indexing may be on by default, except if the property is explicitly declared of low cardinality. For example, an index on gender makes no sense. The same is most often true of <code>rdfs:type</code>. Some properties may have many distinct values (e.g., price), but are still not good for indexing, as this makes for the extreme difference in load time between SQL and the all-indexing RDF.</p> <p>Identifying whether a column will be frequently updated is another useful thing. This will turn off indexing and use an easy-to-update physical representation. Plus, properties which are frequently updated are best put physically together. This may, for example, guide the choice between row-wise and column-wise representation. A customer's account balance and orders year-to-date would be an example of such properties.</p> <p>Some short string valued properties may be frequently returned or used as sorting keys. This requires accessing the literal via an ID in the dictionary table. Non-string literals, numbers, dates, etc., are always inlined (at least in most implementations), but strings are a special question. <a href="http://bigdata.com/bigdata" id="link-id0x2aabfca038e8">Bigdata</a> and early versions of Virtuoso would inline short ones; later versions of Virtuoso would not. So specifying, per property/class combination, a length limit for an inlined string is very high gain and trivial to do. The <a href="http://wifo5-03.informatik.uni-mannheim.de/bizer/berlinsparqlbenchmark/" id="link-id0x2aabe1f9b678">BSBM</a> explore score at large scales can get a factor of 2 gain just from inlining one label. BSBM is out of its league here, but this is still really true and yields benefits across the board. The simpler the application, the greater the win.</p> <p>If there are foreign keys, then data should be loaded with the referenced entities first. This makes dimensional clustering possible at load time. If the foreign key is frequently used for accessing the referencing item (for example, if customers are often accessed by country), then loading customers so that customers of the same country end up next to each other can result in great gains. The same applies to a time dimension, which in SQL is often done as a dimension table, but rarely so in linked data. Anyhow, if date is a frequent selection criterion, physically putting items in certain date ranges together can give great gains.</p> <p>The trick here is not necessarily to index on date, but rather to use zone maps (aka min/max index). If nearby values are together, then just storing a min-max value for thousands of consecutive column values is very compact and fast to check, provided that the rows have nearby values. <a href="http://www.actian.com" id="link-id0x2aabe0c098b8">Actian</a> Vector's (<a href="http://dbpedia.org/resource/Vectorwise" id="link-id0x2aab7fc02998">VectorWise</a>) prowess in TPC-H is in part from smart use of date order in this style.</p> <p>To recap, the data shapes desiderata from the viewpoint of guiding physical storage is as follows:</p> <p>(I will use "data shape" to mean "characteristic set," or "set of Subjects subject to the same set of constraints." A Subject belonging to a data shape may be determined either by its <code>rdfs:type</code> or by the fact of it having, within the processing window, all or some of a set of properties.)</p> <ul> <li> All normal range, domain, cardinality, optionality, etc. — Specifically, declaring something as single valued (as with SQL's <code>UNIQUE constraint) and mandatory (as with SQL's <code>NOT NULL</code> constraint) is good. </code> </li> <li> Primary access path — The Properties whose Objects are dominant access criteria is important </li> <li> No-index — Declare that no index will be made on the Object of a Property within a data shape. </li> <li> Inlined string — String values of up to so many characters in this data shape are inlined </li> <li> Clustering key — The Subject identifiers will be picked to be correlated with the Object of this Property in this data shape. This can be qualified by a number of buckets (e.g., if dates are from 2000 to 2020, then this interval may be 100 buckets), with an exception bucket for out of range values. </li> <li> No full text index — A string value will not need to be full text indexed in this Property even if full text indexing is generally on. </li> <li> Full text index desired — This means that if the value of the property is a string, then the row must be locatable via this string. The string may or may not be inlined, but an index will exist on the literal ID of the string, e.g., POSG. </li> <li> Co-location — This is akin to clustering but specifies, for a high cardinality Object, that the Subject identifier should be picked to fall in the same partition as the Object. The Object is typically a parent of the Subject being loaded; for example, the containing assembly of a sub-assembly. Traversing the assembly created in this way will be local on a scale-out system. This can also apply to geometries or text values: If primary access is by text or geo index, then the metadata represented as triples should be in the same partition as the entry in the full text/geo index. </li> <li> Update group — A set of properties that will often change together. Implies no index and some form of co-location, plus update-friendly physical representation. Many update groups may exist, in which case they may or may not be collocated. </li> <li> Composite foreign/primary key — A data shape can have a multicolumn foreign key, e.g., <code>l_partkey, l_suppkey</code> in <code>lineitem</code> with the matching primary key of <code>ps_partkey, ps_suppkey</code> in <code>partsupp</code>. This can be used for checking and for query optimization: Looking at <code>l_partkey</code> and <code>l_suppkey</code> as independent properties, the guess would be that there hardly ever exists a <code>partsupp</code>, whereas one does always exist. The XML standards stack also has a notion of a composite key for random access on multiple attributes. </li> </ul> <p>These things have the semantic of "hint for physical storage" and may all be ignored without effect on semantics, at least if the data is constraint-compliant to start with.</p> <p>These things will have some degree of reference implementation through the evolution of Virtuoso structure awareness, though not necessarily immediately. These are, to the semanticist, surely dirty low-level disgraceful un-abstractions, some of the very abominations the early semanticists abhorred or were blissfully ignorant of when they first raised their revolutionary standard.</p> <p>Still, these are well-established principles of the broader science of database. SQL does not standardize some of these, nor does it have much need to, as the use of these features is system-specific. The support varies widely and the performance impacts are diverse. However, since RDF excels as a reference model and as a data interchange format, giving these indications as hints to back-end systems cannot hurt, and can make a difference of night and day in load and query time.</p> <p>As Phil Archer said, the idea of RDF Data Shapes is for an application to say that "it will barf if it gets data that is not like this." An extension is for the data to say what the intended usage pattern is so that the system may optimize for this.</p> <p>All these things may be learned from static analysis and workload traces. The danger of this is over-fitting a particular profile. This enters a gray area in benchmarking. For big data, if RDF is to be used as the logical model and the race is about highest absolute performance, never mind what the physical model ends up being, all this and more is necessary. And if one is stretching the envelope for scale, the race is always about highest absolute performance. For this reason, these things will figure at the leading edge with or without standardization. I would say that the build-up of experience in the RDBMS world is sufficient for these things to be included as hints in a profile of data shapes. The compliance cost will be nil if these are ignored, so for the W3C, these will not make the implementation effort for compliance with an eventual data shapes recommendation prohibitive.</p> <p>The use case is primarily the data warehouse to go. If many departments or organizations publish data for eventual use by their peers, users within the organization may compose different combinations of extractions for different purposes. Exhaustive indexing of everything by default makes the process slow and needlessly expensive, as we have seen. Much of such exploration is bounded by load time. Federated approaches for analytics are just not good, even though they may work for infrequent lookups. If datasets are a commodity to be plugged in and out, the load and query investment must be minimized without the user/DBA having to run workload analysis and manual schema optimization. Therefore, bundling guidelines such as these with data shapes in a dataset manifest can do no harm and can in cases provide 10-50x gains in load speeds and 2-4x in space consumption, not to mention unbounded gains in query time, as good and bad plans easily differ by 10-100x, especially in analytics.</p> <p>So, here is the pitch: </p> <ul> <li> Dramatic gains in ad hoc user experience </li> <li> Minimal effort by data publishers, as much of the physical guidelines can be made from workload trace and dataset; the point is that the ad hoc user does not have to do this. </li> <li> Great optimization potential for system vendors; low cost for initial compliance </li> <li> Better understanding of the science of performance by the semantic community </li> </ul> <p>To be continued...</p> <h3>SEMANTiCS 2014 Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1813" id="link-id0x2aabe1f87ad8">SEMANTiCS 2014 (part 1): Keynote</a> </li> <li> SEMANTiCS 2014 (part 2): RDF Data Shapes<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1815" id="link-id0x2aabe12fb048">SEMANTiCS 2014 (part 3): Conversation</a> </li> </ul>
2014-10-29T13:31:54.399768-04:00
SEMANTiCS 2014 (part 1 of 3): Keynote
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-09-06#1810
2014-09-07T00:44:19Z
<p>I was invited to give a keynote at <a href="http://www.semantics.cc" id="link-id0x2aab7f0f1c28">SEMANTiCS 2014</a> in <a href="http://dbpedia.org/resource/Leipzig" id="link-id0x2aab7ef64778">Leipzig</a>, <a href="http://dbpedia.org/resource/Germany" id="link-id0x2aab7f488268">Germany</a> last Thursday. I will here recap some of the main points, and comment on some of the ensuing controversy. The talk was initially titled <i><a href="http://www.slideshare.net/kidehen/ogl-semantic-leipzigsemantics2014" id="link-id0x2aab7f3af178">Virtuoso, the Prometheus of RDF</a></i>. Well, mythical Prometheus did perform a service but ended up paying for it. Still, the mythical reference is sometimes used when talking of major breakthroughs and big-gain ambitions. In the <a href="http://www.slideshare.net/kidehen/ogl-semantic-leipzigsemantics2014/2" id="link-id0x2aab7ef4a308">first slide</a>, I changed it to <a href="http://dbpedia.org/resource/Linked_data" id="link-id0x2aab7f510bb8">Linked Data</a> at Dawn, which is less product specific and more a reflection on the state of the linked data enterprise at large.</p> <p>The first part of the talk was under the heading of the promise and the practice. The promise we know well and find no fault with: Schema-last-ness, persistent unique identifiers, self-describing data, some but not too much inference. The applications usually involve some form of integration and often have a mix of strictly structured content with semi-structured or textual content.</p> <p>These values are by now uncontroversial and embraced by many; however, most instances of this embracing do not occur in the context of RDF as such. For example, the big online systems on the web: all have some schema-last (key-value) functionality. Applications involving long-term data retention have diverse means of having persistent IDs and self description, from <i>UUIDs</i> to having the table name in a column so that one can tell where a CSV dump came from.</p> <p>The practice involves competing with diverse alternative technologies: SQL, key-value, information retrieval (often <a href="http://dbpedia.org/resource/Lucene" id="link-id0x2aab7f503868">Lucene</a>-derived). In some instances, graph databases occur as alternatives: Young semanticist, do or die.</p> <p>In this race, linked data is often the prettiest and most flexible, but gets a hit on different aspects of performance and scalability. This is a database gig, and database is a performance game; make no mistake.</p> <p>After these preliminaries we come to the <i>"<a href="http://www.slideshare.net/kidehen/ogl-semantic-leipzigsemantics2014/9" id="link-id0x2aab7f59e228">RDF tax</a>,"</i> or the more or less intrinsic overheads of describing all as triples. The word "triple" is used by habit. In fact, we nearly always talk about quads, i.e., subject-predicate-object-graph (<code>SPOG</code>). The next slide is provocatively titled the <i><a href="http://www.slideshare.net/kidehen/ogl-semantic-leipzigsemantics2014/10" id="link-id0x2aab7f456428">Bane of the Triple</a>,</i> and is about why having all as triples is, on the surface, much like relational, except it makes life hard, where tables make it at least manageable, if still not altogether trivial.</p> <p>The very first statement on <a href="http://www.slideshare.net/kidehen/ogl-semantic-leipzigsemantics2014/9" id="link-id0x2aab7f07f6e8">the tax slide</a> reads "90% of bad performance comes from non-optimal query plans." If one does triples in the customary way (i.e., a table of quads plus dictionary tables to map URIs and literal strings to internal IDs), one incurs certain fixed costs.</p> <p>These costs are deemed acceptable by users who deploy linked data. If these costs were not acceptable, the proof of concept would have already disqualified linked data.</p> <p>The support cases that come my way are nearly always about things taking too much time. Much less frequently, are these about something unambiguously not working. Database has well defined semantics, so whether something works or not is clear cut.</p> <p>So, support cases are overwhelmingly about query optimization. The problems fall in two categories:</p> <ul> <li> The plan is good in the end, but it takes much longer to make the plan than to execute it. </li> <li> The plan either does the wrong things or does things in the wrong order, but produces a correct result. </li> </ul> <p>Getting no plan at all or getting a clearly wrong result is much less frequent.</p> <p>If the RDF overheads incurred with a good query plan were show stoppers, the show would have already stopped.</p> <p>So, let's look at this in more detail; then we will talk about the fixed overheads.</p> <p>The join selectivity of triple patterns is correlated. Some properties occur together all the time; some occur rarely; some not at all. Some property values can be correlated, i.e., order number and order date. Capturing these by sampling in a multicolumn table is easy; capturing this in triples would require doing the join in the cost model, which is not done since it would further extend compilation times. When everything is a join, selectivity estimation errors build up fast. When everything is a join, the space of possible graph query plans explodes as opposed to tables; thus, while the full plan space can be covered with 7 tables, it cannot be covered with 18 triple patterns. This is not factorial (number of permutations). For different join types (index/hash) and the different compositions of the hash build side, this is much worse, in some nameless outer space fringe of non-polynomiality.</p> <p> <a href="http://www.tpc.org/tpch/" id="link-id0x2aab7ec91108">TPC-H</a> can be run with success because the cost model hits the right plan every time. The primary reason for this is the fact that the schema and queries unambiguously suggest the structure, even without foreign key declarations. The other reason is that with a handful of tables, all plans can be reviewed, and the cost model reliably tells how many rows will result from each sequence of operations.</p> <p>Try this with triples; you will know what I mean.</p> <p>Now, some people have suggested purely rule-based models of SPARQL query compilation. These are arguably faster to run and more predictable. But the thing that must be done, yet will not be done with these, is the right trade-off between index and hash. This is the crux of the matter, and without this, one can forget about anything but lookups. The choice depends on reliable estimation of cardinality (number of rows, number of distinct keys) on either side of the join. Quantity, not pattern matching.</p> <p>Well, many linked data applications are lookups. The graph database API world is sometimes attractive because it gives manual control. Map reduce in the analytical space is sometimes attractive for the same reason.</p> <p>On the other hand, query languages also give manual control, but then this depends on system specific hints and cheats. People are often black and white: Either all declarative or all imperative. We stand for declarative, but still allow physical control of plan, like most DBMS.</p> <p>To round off, I will give a concrete example:</p> <blockquote> <code><pre> { ?thing rdfs:label ?lbl . ?thing dc:title ?title . ?lbl bif:contains "gizmo" . ?title bif:contains "widget" . ?thing a xx:Document . ?thing dc:date ?dt . FILTER ( ?dt > "2014-01-01"^^xsd:date ) }</pre> </code> </blockquote> <p>There are two full text conditions, one date, and one class, all on the same subject. How do you do this? Most selective text first, then get the data and check, then check the second full text given the literal and the condition, then check the class? Wrong. If widgets and gizmos are both frequent and most documents new, this is very bad because using a text index to check for a specific ID having a specific string is not easily vectorable. So, the right plan is: Take the more selective text expression, then check the date and class for the results, put the <code><i>?things</i></code> in a hash table. Then do the less selective text condition, and drop the ones that are not in the hash table. Easily 10x better. Simple? In the end yes, but you do not know this unless you know the quantities.</p> <p>This gives the general flavor of the problem. Doing this with TPC-H in RDF is way harder, but you catch my drift.</p> <p>Each individual instance is do-able. Having closer and closer alignment between reality and prediction will improve the situation indefinitely, but since the space is as good as infinite there cannot be a guarantee of optimality except for toy cases.</p> <p>The <a href="http://dbpedia.org/resource/Gordian_Knot" id="link-id0x2aab7f03acb8">Gordian Knot</a> shall not be defeated with pincers but by the sword.</p> <p>We will come to this in a bit.</p> <p>Now, let us talk of the fixed overheads. The embarrassments are in the query optimization domain; the daily grind, relative cost, and provisioning are in this one.</p> <p>The overheads come from:</p> <ul> <li> Indexing everything </li> <li> Having literals and URI strings via dictionary </li> <li> Having a join for every attribute </li> </ul> <p>These all fall under the category of having little to no physical design room.</p> <p>In the indexing everything department, we load 100 GB TPC-H in 15 minutes in SQL with ordering only on primary keys and almost no other indexing. The equivalent with triples is around 12 hours. This data can be found on this blog (<a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aab7ef220e8">TPC-H series</a> and <i><a href="http://www.openlinksw.com/weblog/oerling/?id=1769" id="link-id0x2aab7f6881a8">Meeting the Challenges of Linked Data in the Enterprise</a></i>). This is on the order of confusing a screwdriver with a hammer. If the nail is not too big, the wood not too hard, and you hit it just right — the nail might still go in. The RDF bulk load is close to the fastest possible given the general constraints of what it does. The same logic is used for the record-breaking 15 minutes of TPC-H bulk load, so the code is good. But indexing everything is just silly.</p> <p>The second, namely the dictionary of URIs and literals, is a dual edge. I talked to <a href="http://www.linkedin.com/pub/bryan-thompson/30/390/54b" id="link-id0x2aab7ec1cab8">Bryan Thompson</a> of <a href="http://www.systap.com/about" id="link-id0x2aab7ef42808">SYSTAP</a> (<a href="http://bigdata.com/bigdata" id="link-id0x2aab7ef42248">Bigdata</a> RDF store) in <a href="http://dbpedia.org/page/Washington,_D.C." id="link-id0x2aab7fa84ce8">D.C.</a> at the <a href="http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?reload=true&punumber=6226952" id="link-id0x2aab7f6cd528">ICDE</a> there. He said that they do short strings inline and long ones via dictionary. I said we used to do the same but stopped in the interest of better compression. What is best depends on workload and working-set-to-memory ratio. But if you must make the choice once and for all, or at least as a database-wide global setting, you are between a rock and a hard place. Physical vs. logical design, again.</p> <p>The other aspect of this is the applications that do regexps on URI strings or literals. Doing this is like driving a Formula 1 race in reverse gear. Use a text index. Always. This is why most implementations have one even though SPARQL itself makes no provisions for this. If you really need regexps, and on supposedly opaque URIs at that, tokenize them and put them in a text index as a text literal. Or if an inverted-file-word index is really not what you need, use a trigram one. So far, nobody has wanted one hard enough for us to offer this, even though this is easy enough. But special indices for special data types (e.g., chemical structure) are sometimes wanted, and we have a generic solution for all this, to be introduced shortly on this blog. Again, physical design.</p> <p>I deliberately name the self-join-per-attribute point last, even though this is often the first and only intrinsic overhead that is named. True, if the physical model is triples, each attribute is a join against the triple table. Vectored execution and right use of hash-join help, though. The Star Schema Benchmark SQL to SPARQL gap is only 2.5x, as <a href="http://www.openlinksw.com/weblogs/oerling/?id=1732" id="link-id0x2aab7ec7cc68">documented last year on this blog</a>. This makes SPARQL win by 100+x against <a href="http://dbpedia.org/resource/MySQL" id="link-id0x2aab7f3a6148">MySQL</a> and lose by only 0.8x against column store pioneer <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x2aab7f342508">MonetDB</a>. Let it be said that this is so far the best case and that the gap is wider in pretty much all other cases. This gap is well and truly due to the self-join matter, even after the self-joins are done vectored, local, ordered; in one word, <i>right.</i> The literal and URI translation matter plays no role here. The needless indexing hurts at load but has no effect at query time, since none of the bloat participates in the running. Again, physical design.</p> <p>Triples are done right, so?</p> <p>In the summer of 2013, after the Star Schema results, it became clear that maybe further gains could be had and query optimization made smoother and more predictable, but that these would be paths of certain progress but with diminishing returns per effort. No, not the pincers; give me the sword. So, between fall 2013 and spring 2014, aside from doing diverse maintenance, I did the <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aab7f6caad8">TPC-H series</a>. This is the proficiency run for big league databases; the America's Cup, not a regatta on the semantic lake.</p> <p>Even if the audience is principally Linked Data, the baseline must be that of the senior science of SQL.</p> <p>It stands to reason and has been demonstrated by extensive experimentation at <a href="http://www.cwi.nl" id="link-id0x2aab7eed20b8">CWI</a> that RDF data, by and large, has structure. This structure will carry linked data through the last mile to being a real runner against the alternative technologies (SQL, IR, key value) mentioned earlier.</p> <p>The operative principles have been mentioned earlier and are set forth on the slides. In forthcoming articles I will display some results.</p> <p>One important proposal for structure awareness was by <a href="https://www.linkedin.com/pub/thomas-neumann/2/35/189" id="link-id0x2aab7f6ee928">Thomas Neumann</a> in an <a href="https://domino.mpi-inf.mpg.de/intranet/ag5/ag5publ.nsf/0/AD3DBAFA6FB90DD2C1257593002FF3DF/$file/rdf3x.pdf" id="link-id0x2aab7f3a2d38">RDF3X paper introducing characteristic sets</a>. There, the application was creation of more predictable cost estimates. Neumann correctly saw this as possibly the greatest barrier to predictable RDF performance. <a href="http://homepages.cwi.nl/~boncz/" id="link-id0x2aab7f8b9218">Peter Boncz</a> and I discussed the use of this for physical optimization once when driving back to <a href="http://dbpedia.org/resource/Amsterdam" id="link-id0x2aab7f9e9ab8">Amsterdam</a> from a <a href="http://lod2.eu/" id="link-id0x2aab7f59dfa8">LOD2</a> review in <a href="http://dbpedia.org/resource/Luxembourg" id="link-id0x2aab7f06bd98">Luxembourg</a>. <a href="http://homepages.cwi.nl/~duc/" id="link-id0x2aab7ec98078">Pham Minh Duc</a> of CWI did much of the schema discovery research, documented in the now published LOD2 book (<i><a href="http://link.springer.com/book/10.1007%2F978-3-319-09846-3" id="link-id0x2aab7f293f78">Linked Open Data -- Creating Knowledge Out of Interlinked Data</a></i>). The initial Virtuoso implementation had to wait for the TPC-H and general squeezing of the quads model to be near complete. It will likely turn out that the greatest gain of all with structure awareness will be bringing optimization predictability to SQL levels. This will open the whole bag of tricks known to data warehousing to safe deployment for linked data. Of course, much of this has to do with exploiting physical layout; hence it also needs the physical model to be adapted. Many of these techniques have high negative impact if used in the wrong place; hence the cost model must guess right. But they work in SQL and, as per Thomas Neumann's initial vision, there is no reason why these would not do so in a schema-less model if adapted in a smart enough manner.</p> <p>All this gives rise to some sociological or psychological observations. <a href="http://jens-lehmann.org/" id="link-id0x2aab7f1e1af8">Jens Lehmann</a> asked me why now, why not earlier; after all, over the years many people have suggested property tables and other structured representations. This is now because there is no further breakthroughs within an undifferentiated physical model.</p> <p>For completeness, we must here mention other approaches to alternative, if still undifferentiated, physical models. A number of research papers mention memory-only, pointer-based (i.e., no index, no hash-join) implementations of triples or quads. Some of these are on graph processing frameworks, some stand-alone. <a href="http://www.yarcdata.com" id="link-id0x2aab7f455638">Yarc Data</a> is a commercial implementation that falls in this category. These may have higher top speeds than column stores, even after all vectoring and related optimizations. However the space utilization is perforce larger than with optimum column compression and this plus the requirement of 100% in memory makes these more expensive to scale. The linked data proposition is usually about integration, and this implies initially large data even if not all ends up being used.</p> <p>The graph analytics, pointer-based item will be specially good for a per-application extraction, as suggested by <a href="http://dbpedia.org/resource/Oracle_Corporation" id="link-id0x2aab7eed1fd8">Oracle</a> in their paper at <a href="http://event.cwi.nl/grades2013/" id="link-id0x2aab7f1c3e98">GRADES 13</a>. No doubt this will come under discussion at <a href="http://ldbcouncil.org/" id="link-id0x2aab7f11ebd8">LDBC</a>, where Oracle Labs is now a participant.</p> <p>But back to physical model. What we have in mind is relational column store — multicolumn-ordered column-wise compressed tables — a bit like <a href="http://www.vertica.com" id="link-id0x2aab7f0f19a8">Vertica</a> and <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aab7f3138d8">Virtuoso</a> in SQL mode for the regular parts and quads for the rest. What is big is regular, since a big thing perforce comes from something that happens a lot, like click streams, commercial transactions, instrument readings. For the 8-lane-motorway of regular data, you get the F1 racer with the hardcore best in column store tech. When the autobahn ends and turns into the mountain trail, the engine morphs into a dirt bike.</p> <p>This is complex enough, and until all the easy gains have been extracted from quads, there is little incentive. Plus this has the prerequisite of quads done right, plus the need for top of the line relational capability for not falling on your face once the speedway begins.</p> <p> <a href="http://world.marklogic.com/locations/amsterdam/speakers/stephen-buxton/" id="link-id0x2aab7f688468">Steve Buxton</a> of <a href="http://dbpedia.org/resource/MarkLogic" id="link-id0x2aab7f1b7b18">MarkLogic</a> gave a talk right before mine. Coming from a document-centric world, it stands to reason that MarkLogic would have a whole continuum of different mixes between SPARQL and document oriented queries. Steve correctly observed that some users found this great; others found this a near blasphemy, an unholy heterodoxy of confusing distinct principles.</p> <p>This is our experience as well, since usage of <a href="http://dbpedia.org/resource/XML" id="link-id0x2aab7f06c298">XML</a> fragments in SPARQL with <a href="http://dbpedia.org/resource/XPath" id="link-id0x2aab7ec1d7e8">XPath</a> and such things in Virtuoso is possible but very seldom practiced. This is not the same as MarkLogic, though, as MarkLogic is about triples-in-documents, and the Virtuoso take is more like documents-in-triples. Not to mention that use of SQL and stored procedures in Virtuoso is rare among the SPARQL users.</p> <p>The whole thing about the absence of physical design in RDF is a related, but broader instance of such purism.</p> <p>In my talk, I had a slide titled <i><a href="http://www.slideshare.net/kidehen/ogl-semantic-leipzigsemantics2014/17" id="link-id0x2aab7e0bec18">The Cycle of Adventure</a>,</i> generally philosophizing on the dynamics of innovation. All progress begins with an irritation with the status quo; to mention a few examples: the No-SQL rebellion; the rejection of parallel SQL database in favor of key-value and map-reduce; the admission that central schema authority at web scale is impossible; the anti-ACID stance when having wide-area geographies to deal with. The stage of radicalism tends to discard the baby with the bathwater. But when the purists have their own enclave, free of the noxious corruption of the rejected world, they find that life is hard and defects of human character persist, even when all subscribe to the same religion. Of course, here we may have further splinter groups. After this, the dogma adapts to reality: the truly valuable insights of the original rebellion gain in appreciation, and the extremism becomes more moderate. Finally there is integration with mainstream, which becomes enriched by new content.</p> <p>By the time the term Linked Data came to broad use, the RDF enterprise had its break-away colonies that started to shed some of the initial zeal. By now, we have the last phase of reconciliation in its early stages.</p> <p>This process is in principle complete when linked data is no longer a radical bet, but a technology to be routinely applied to data when the nature of the data fits the profile. The structure awareness and other technology discussed here will mostly eliminate the differential in deployment cost.</p> <p>The spreading perception of an expertise gap in this domain will even-out the cost in terms of personnel. The flexibility gains that were the initial drive for the movement will be more widely enjoyed when these factors fuel broader adoption.</p> <p>To help this along, we have <a href="http://ldbcouncil.org" id="link-id0x2aab7efeb4d8">LDBC, the Linked Data Benchmark Council</a>, with the agenda of creating industry consensus on measuring progress across the linked data and graph DB frontiers. I duly invited MarkLogic to join.</p> <p>There were many other interesting conversations at the conference, I will later comment on these.</p> <p>To be continued...</p> <h3>SEMANTiCS 2014 Series</h3> <ul> <li> SEMANTiCS 2014 (part 1): Keynote<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1814" id="link-id0x2aab7f6fdcc8">SEMANTiCS 2014 (part 2): RDF Data Shapes</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1815" id="link-id0x2aab7f7b4e88">SEMANTiCS 2014 (part 3): Conversation</a> </li> </ul>
2014-09-08T16:12:23.246649-04:00
LOD2 Finale (part 3 of n): The 500 Giga-triple Runs
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-08-29#1809
2014-08-29T16:50:05Z
<p>In the evening of day 8, we have kernel settings in the cluster changed to allow more <code>mmaps</code>. At this point, we notice that the dataset is missing the implied <code>types</code> of <code>products</code>; i.e., the most specific <code>type</code> is given but its superclasses are not directly associated with the <code>product</code>. We have always run this with this unique inference materialized, which is also how the data generator makes the data, with the right switch. But the switch was not used. So a further 10 Gt (Giga-triples) are added, by running a SQL script to make the superclasses explicit.</p> <p>At this point, we run BSBM <i>explore</i> for the first time. To what degree does the 37.5 Gt predict the 500 Gt behavior? First, there is an overflow that causes a query plan cost to come out negative if the default graph is specified. This is a bona fide software bug you don't get unless a sample is quite large. Also, we note that starting the databases takes a few minutes due to disk. Further, the first query takes a long time to compile, again because of sampling the database for overall statistics.</p> <p>The statistics are therefore gathered by running a few queries, and then saved. Subsequent runs will reload the stats from the file system, saving some minutes of start time. There is a function for this, <code><a href="http://docs.openlinksw.com/virtuoso/fn_stat_import.html" id="link-id0x2aabc6382718">stat_import</a></code> and <code><a href="http://docs.openlinksw.com/virtuoso/fn_stat_export.html" id="link-id0x2aabf4834688">stat_export</a></code>. These are used for a similar purpose by some users.</p> <p>On day 10, Wednesday August 20, we have some results of BSBM <i>explore</i>.</p> <p>Then, we get into BSBM <i>updates</i>. The BSBM generator makes an update dataset, but it cannot be made large enough. The BSBM test driver suite is by now hated and feared in equal measure. Is it bad in and of itself? Depends. It was certainly not made for large data. Anyway, no fix will be attempted this time. Instead, a couple of SQL procedures are made to drive a random update workload. These can run long enough to get a steady state with warm cache, which is what any OLTP measurement needs.</p> <p>On day 12, some updates are measured, with a one hour ramp-up to steady-state, but these are not quite the right mix, since these are <code>products</code> only and the mix needs to contain <code>offers</code> and <code>reviews</code> also. The first steady-state rate was 109 Kt/s, a full 50x less than the bulk load, but then this was very badly bound by latency. So, the updates are adjusted to have more variety. The final measurement was on day 17. Now the steady-state rate is 2563 Kt/s, which is better but still quite bound by network. By adding diversity to the dataset, we get slammed by a sharp rise in warm-up time (now 2 hours to be at 230 Kt/s), at which point we launch the <i>explore</i> mix to be timed during <i>update</i>. Time is short and we do not want to find out exactly how long it takes to get the plateau in insert rate. As it happens, the <i>explore</i> mix is hardly slowed down by the <i>updates</i>, but the <i>updates</i> get hit worse, so that the rate goes to about 1/3 of what it was, then comes back up when the <i>explore</i> is finished. Finally, half an hour after this, there is a steady state of 263 Kt/s <i>update</i> rate.</p> <p>Of course, the main object of the festivities is still the <i>business intelligence</i> (BI) mix. This is our (specifically, Orri's) own invention from years back, subsequently formulated in SPARQL by <a href="http://www.fu-berlin.de/" id="link-id0x2aabf519c648">FU Berlin</a> (<a href="http://www.wiwiss.fu-berlin.de/en/fachbereich/bwl/pwo/bizer/team/SchultzAndreas.html" id="link-id0x2aabc6711de8">Andreas Schultz</a>). Well, it is already something to do big joins with 150 Gt, all on index and vectored random access, as was done in January 2013, the last time results were published on the <a href="http://www.cwi.nl/" id="link-id0x2aabc64b5438">CWI</a> <a href="http://scilens.project.cwi.nl/" id="link-id0x2aabc610c9d8">cluster</a>. You may remember that there was <a href="http://www.openlinksw.com/weblog/oerling/?id=1767" id="link-id0x2aabc63f7cc8">an aborted attempt in January 2014</a>. So now, with the LOD2 end date under two weeks away, we will take the BI racer out for a spin with 500 Gt. This is now a very different proposition from Jan 2013, as we have by now done the whole <a href="http://www.tpc.org/tpch/" id="link-id0x2aabf4ecc6f8">TPC-H</a> work <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aabc6447a18">documented on this blog</a>. This serves to show, <i>inter alia,</i> that we can run with the best in the much bigger and harder mainstream database sports. The full benefits of this will be realized for the semantic data public still this year, so this is more than personal vanity.</p> <p>So we will see. The BI mix is not exactly TPC-H, but what is good for one is good for the other. Checking that the plans are good on the 37 Gt scale model is done around day 12. On day 13, we try this on the larger cluster. You never know — pushing the envelope, even when you know what you are doing and have written the whole thing, is still a dive in the fog. Claiming otherwise would be a lie lacking credibility. The iceberg which first emerges is overflow and partition skew. Well, there can be a lot of messages if all messages go via the same path. So we make the data structure different and retry and now die from out of memory. On the scale model, this looks like a little imbalance you don't bother to notice; at 13x scale, this kills. So, as is the case with most database problems, the query plan is bad. Instead of using a <code>PSOG</code> index, it uses a <code>POSG</code> index, and there is a constant for <code>O</code>. Partitioning is on either <code>S</code> or <code>O</code>, whichever is first. Not hard to fix, but still needs a cost-model adjustment to penalize low-cardinality partition columns. This is something you don't get with TPC-H, where there are hardly any indices. Once this is fixed there are other problems, such as Q5, which we ended up leaving out. The scale model is good; the large one does not produce a plan, because some search-space corner is visited that is not visited in the scale model, due to different ratios of things in the cost model. Could be a couple of days to track; this is complex stuff. So we dropped it. It is not a big part of the metric, and its omission is immaterial to the broader claim of handling 500 Gt in all safety and comfort. The moral is: never get stuck; only do what is predictable, insofar as anything in this shadowy frontier is such.</p> <p>So, on days 15 and 16, the BI mix that is reported was run. The multiuser score was negatively impacted by memory skew, so some swapping on one of the nodes, but the run finished in about 2 hours anyway. The peak of transient memory consumption is another thing that you cannot forecast with exact precision. There is no model for that; the query streams are in random order, and you just have to try. And it is a few hours per iteration, so you don't want to be stuck doing that either. A rerun would get a higher multiuser BI score; maybe one will be made but not before all the rest is wrapped up.</p> <p>Now we are talking 2 hours, versus 9 hours with the 150 Gt set back in January 2013. So 3.3x the data, 4.5x less time, 1.5x the gear. This comes out at one order of magnitude. With a better score from better memory balance and some other fixes, a 15x improvement for BSBM BI is in the cards.</p> <p>The final <i>explore</i> runs were made on day 18, while writing the report to be published at the LOD2 deliverables repository. The report contains in depth discussion on the query plans and diverse database tricks and their effectiveness.</p> <p>The overall moral of this trip into these uncharted spaces is this: Expect things to break. You have to be the designer and author of the system to take it past its limits. You will cut it or you won't, and nobody can do anything about it, not with the best intentions, nor even with the best expertise, which both were present. This is true of the last minute daredevil stuff like this; if you have a year full time instead of the last 20 days of a project, all is quite different, and these things are more leisurely. This might then become a committee affair, though, which has different problems. In the end, the <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aabf48373a8">Virtuoso DBMS</a> has never thrown anything at us we could not handle. The uncertainty in trips of this sort is with the hardware platform, of which we had to <a href="http://www.openlinksw.com/weblog/oerling/?id=1805" id="link-id0x2aabc753df48">replace 2 units to get on the way</a>, and with how fast you can locate and fix a software problem. So you pick the quickest ones and leave the uncertain aside. There is another category of rare events like network failures that in theory cannot happen. Yet they do. So, to program a cluster, you have to have some recovery things for these. We saw <a href="http://www.openlinksw.com/weblog/oerling/?id=1805" id="link-id0x2aab709be078">a couple of these along the way</a>. Duplication of these can take days, and whether this correlates with specific links or is a bona fide software thing is time consuming to prove, and getting into this is a sure way to lose the race. These seem to be load peaks outside of steady-state; steady-state is in fact very steady once it is there. Except at the start, network glitches were not a big factor in these experiments. The bulk of these went away after replacing a machine. After this we twice witnessed something that cannot exist but knew better than to get stuck with that. Neither incident happened again. This is days of running at a cross sectional 1 GB/s of traffic. These are the truly unpredictable, and, in a crash course like this, can sink the whole gig no matter how good you are.</p> <p>Thanks are due to CWI and especially <a href="http://homepages.cwi.nl/~boncz/" id="link-id0x2aab728cbd38">Peter Boncz</a> for providing the race track as well as advice and support.</p> <p>In the next installments of this series, we will look at how schema and characteristic sets will deliver the promise of RDF without its cost. All the experiments so far were done with a quads table, as always before. So we could say that the present level is close to the limit of the achievable within this physical model. The future lies beyond the misconception of triples/quads as primary physical model.</p> <p>To be continued...</p> <h3>LOD2 Finale Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1804" id="link-id0x2aabf5507888"> LOD2 Finale (part 1): RDF Before The Dawn</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1805" id="link-id0x2aabf5175be8"> LOD2 Finale (part 2): The 500 Giga-triples</a> </li> <li> LOD2 Finale (part 3): The 500 Giga-triple Runs<i> (this post)</i> </li> </ul>
2014-08-29T12:50:05.038492-04:00
LOD2 Finale (part 3 of n): The 500 Giga-triple Runs
http://www.openlinksw.com:443/weblog/oerling/?date=2014-08-29#1808
2014-08-29T16:49:53Z
<p>In the evening of day 8, we have kernel settings in the cluster changed to allow more <code>mmaps</code>. At this point, we notice that the dataset is missing the implied <code>types</code> of <code>products</code>; i.e., the most specific <code>type</code> is given but its superclasses are not directly associated with the <code>product</code>. We have always run this with this unique inference materialized, which is also how the data generator makes the data, with the right switch. But the switch was not used. So a further 10 Gt (Giga-triples) are added, by running a SQL script to make the superclasses explicit.</p> <p>At this point, we run BSBM <i>explore</i> for the first time. To what degree does the 37.5 Gt predict the 500 Gt behavior? First, there is an overflow that causes a query plan cost to come out negative if the default graph is specified. This is a bona fide software bug you don't get unless a sample is quite large. Also, we note that starting the databases takes a few minutes due to disk. Further, the first query takes a long time to compile, again because of sampling the database for overall statistics.</p> <p>The statistics are therefore gathered by running a few queries, and then saved. Subsequent runs will reload the stats from the file system, saving some minutes of start time. There is a function for this, <code><a href="http://docs.openlinksw.com/virtuoso/fn_stat_import.html" id="link-id0x2aabc6382718">stat_import</a></code> and <code><a href="http://docs.openlinksw.com/virtuoso/fn_stat_export.html" id="link-id0x2aabf4834688">stat_export</a></code>. These are used for a similar purpose by some users.</p> <p>On day 10, Wednesday August 20, we have some results of BSBM <i>explore</i>.</p> <p>Then, we get into BSBM <i>updates</i>. The BSBM generator makes an update dataset, but it cannot be made large enough. The BSBM test driver suite is by now hated and feared in equal measure. Is it bad in and of itself? Depends. It was certainly not made for large data. Anyway, no fix will be attempted this time. Instead, a couple of SQL procedures are made to drive a random update workload. These can run long enough to get a steady state with warm cache, which is what any OLTP measurement needs.</p> <p>On day 12, some updates are measured, with a one hour ramp-up to steady-state, but these are not quite the right mix, since these are <code>products</code> only and the mix needs to contain <code>offers</code> and <code>reviews</code> also. The first steady-state rate was 109 Kt/s, a full 50x less than the bulk load, but then this was very badly bound by latency. So, the updates are adjusted to have more variety. The final measurement was on day 17. Now the steady-state rate is 2563 Kt/s, which is better but still quite bound by network. By adding diversity to the dataset, we get slammed by a sharp rise in warm-up time (now 2 hours to be at 230 Kt/s), at which point we launch the <i>explore</i> mix to be timed during <i>update</i>. Time is short and we do not want to find out exactly how long it takes to get the plateau in insert rate. As it happens, the <i>explore</i> mix is hardly slowed down by the <i>updates</i>, but the <i>updates</i> get hit worse, so that the rate goes to about 1/3 of what it was, then comes back up when the <i>explore</i> is finished. Finally, half an hour after this, there is a steady state of 263 Kt/s <i>update</i> rate.</p> <p>Of course, the main object of the festivities is still the <i>business intelligence</i> (BI) mix. This is our (specifically, Orri's) own invention from years back, subsequently formulated in SPARQL by <a href="http://www.fu-berlin.de/" id="link-id0x2aabf519c648">FU Berlin</a> (<a href="http://www.wiwiss.fu-berlin.de/en/fachbereich/bwl/pwo/bizer/team/SchultzAndreas.html" id="link-id0x2aabc6711de8">Andreas Schultz</a>). Well, it is already something to do big joins with 150 Gt, all on index and vectored random access, as was done in January 2013, the last time results were published on the <a href="http://www.cwi.nl/" id="link-id0x2aabc64b5438">CWI</a> <a href="http://scilens.project.cwi.nl/" id="link-id0x2aabc610c9d8">cluster</a>. You may remember that there was <a href="http://www.openlinksw.com/weblog/oerling/?id=1767" id="link-id0x2aabc63f7cc8">an aborted attempt in January 2014</a>. So now, with the LOD2 end date under two weeks away, we will take the BI racer out for a spin with 500 Gt. This is now a very different proposition from Jan 2013, as we have by now done the whole <a href="http://www.tpc.org/tpch/" id="link-id0x2aabf4ecc6f8">TPC-H</a> work <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aabc6447a18">documented on this blog</a>. This serves to show, <i>inter alia,</i> that we can run with the best in the much bigger and harder mainstream database sports. The full benefits of this will be realized for the semantic data public still this year, so this is more than personal vanity.</p> <p>So we will see. The BI mix is not exactly TPC-H, but what is good for one is good for the other. Checking that the plans are good on the 37 Gt scale model is done around day 12. On day 13, we try this on the larger cluster. You never know — pushing the envelope, even when you know what you are doing and have written the whole thing, is still a dive in the fog. Claiming otherwise would be a lie lacking credibility. The iceberg which first emerges is overflow and partition skew. Well, there can be a lot of messages if all messages go via the same path. So we make the data structure different and retry and now die from out of memory. On the scale model, this looks like a little imbalance you don't bother to notice; at 13x scale, this kills. So, as is the case with most database problems, the query plan is bad. Instead of using a <code>PSOG</code> index, it uses a <code>POSG</code> index, and there is a constant for <code>O</code>. Partitioning is on either <code>S</code> or <code>O</code>, whichever is first. Not hard to fix, but still needs a cost-model adjustment to penalize low-cardinality partition columns. This is something you don't get with TPC-H, where there are hardly any indices. Once this is fixed there are other problems, such as Q5, which we ended up leaving out. The scale model is good; the large one does not produce a plan, because some search-space corner is visited that is not visited in the scale model, due to different ratios of things in the cost model. Could be a couple of days to track; this is complex stuff. So we dropped it. It is not a big part of the metric, and its omission is immaterial to the broader claim of handling 500 Gt in all safety and comfort. The moral is: never get stuck; only do what is predictable, insofar as anything in this shadowy frontier is such.</p> <p>So, on days 15 and 16, the BI mix that is reported was run. The multiuser score was negatively impacted by memory skew, so some swapping on one of the nodes, but the run finished in about 2 hours anyway. The peak of transient memory consumption is another thing that you cannot forecast with exact precision. There is no model for that; the query streams are in random order, and you just have to try. And it is a few hours per iteration, so you don't want to be stuck doing that either. A rerun would get a higher multiuser BI score; maybe one will be made but not before all the rest is wrapped up.</p> <p>Now we are talking 2 hours, versus 9 hours with the 150 Gt set back in January 2013. So 3.3x the data, 4.5x less time, 1.5x the gear. This comes out at one order of magnitude. With a better score from better memory balance and some other fixes, a 15x improvement for BSBM BI is in the cards.</p> <p>The final <i>explore</i> runs were made on day 18, while writing the report to be published at the LOD2 deliverables repository. The report contains in depth discussion on the query plans and diverse database tricks and their effectiveness.</p> <p>The overall moral of this trip into these uncharted spaces is this: Expect things to break. You have to be the designer and author of the system to take it past its limits. You will cut it or you won't, and nobody can do anything about it, not with the best intentions, nor even with the best expertise, which both were present. This is true of the last minute daredevil stuff like this; if you have a year full time instead of the last 20 days of a project, all is quite different, and these things are more leisurely. This might then become a committee affair, though, which has different problems. In the end, the <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aabf48373a8">Virtuoso DBMS</a> has never thrown anything at us we could not handle. The uncertainty in trips of this sort is with the hardware platform, of which we had to <a href="http://www.openlinksw.com/weblog/oerling/?id=1805" id="link-id0x2aabc753df48">replace 2 units to get on the way</a>, and with how fast you can locate and fix a software problem. So you pick the quickest ones and leave the uncertain aside. There is another category of rare events like network failures that in theory cannot happen. Yet they do. So, to program a cluster, you have to have some recovery things for these. We saw <a href="http://www.openlinksw.com/weblog/oerling/?id=1805" id="link-id0x2aab709be078">a couple of these along the way</a>. Duplication of these can take days, and whether this correlates with specific links or is a bona fide software thing is time consuming to prove, and getting into this is a sure way to lose the race. These seem to be load peaks outside of steady-state; steady-state is in fact very steady once it is there. Except at the start, network glitches were not a big factor in these experiments. The bulk of these went away after replacing a machine. After this we twice witnessed something that cannot exist but knew better than to get stuck with that. Neither incident happened again. This is days of running at a cross sectional 1 GB/s of traffic. These are the truly unpredictable, and, in a crash course like this, can sink the whole gig no matter how good you are.</p> <p>Thanks are due to CWI and especially <a href="http://homepages.cwi.nl/~boncz/" id="link-id0x2aab728cbd38">Peter Boncz</a> for providing the race track as well as advice and support.</p> <p>In the next installments of this series, we will look at how schema and characteristic sets will deliver the promise of RDF without its cost. All the experiments so far were done with a quads table, as always before. So we could say that the present level is close to the limit of the achievable within this physical model. The future lies beyond the misconception of triples/quads as primary physical model.</p> <p>To be continued...</p> <h3>LOD2 Finale Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1804" id="link-id0x2aabf5507888"> LOD2 Finale (part 1): RDF Before The Dawn</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1805" id="link-id0x2aabf5175be8"> LOD2 Finale (part 2): The 500 Giga-triples</a> </li> <li> LOD2 Finale (part 3): The 500 Giga-triple Runs<i> (this post)</i> </li> </ul>
2014-08-29T12:49:53.491880-04:00
LOD2 Finale (part 2 of n): The 500 Giga-triples
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-08-18#1807
2014-08-18T20:55:57Z
<p>No epic is complete without a descent into hell. Enter the <i>historia calamitatum</i> of the 500 Giga-triples (Gt) at <a href="http://www.cwi.nl/" id="link-id0x15b8cbb8">CWI</a>'s <a href="http://scilens.project.cwi.nl/" id="link-id0x15ead658">Scilens</a> cluster.</p> <p>Now, <a href="http://www.openlinksw.com/weblog/oerling/?id=1767" id="link-id0x15a53918">from last time</a>, we know to generate the data without 10 GB of namespace prefixes per file and with many short files. So we have 1.5 TB of gzipped data in 40,000 files, spread over 12 machines. The data generator has again been modified. Now the generation was about 4 days. Also from last time, we know to treat small integers specially when they occur as partition keys: 1 and 2 are very common values and skew becomes severe if they all go to the same partition; hence consecutive small <code>INTs</code> each go to a different partition, but for larger ones the low 8 bits are ignored, which is good for compression: Consecutive values must fall in consecutive places, but not for small <code>INTs</code>. Another uniquely brain-dead feature of the BSBM generator has also been rectified: When generating multiple files, the program would put things in files in a round-robin manner, instead of putting consecutive numbers in consecutive places, which is how every other data generator or exporter does it. This impacts bulk load locality and as you, dear reader, ought to know by now, performance comes from (1) locality and (2) parallelism.</p> <p>The machines are similar to last time: each a dual E5 2650 v2 with 256 GB RAM and QDR InfiniBand (IB). No SSD this time, but a slightly higher clock than last time; anyway, a different set of machines.</p> <p>The first experiment is with triples, so no characteristic sets, no schema.</p> <p>So, first day (Monday), we notice that one cannot allocate more than 9 GB of memory. Then we figure out that it cannot be done with <code>malloc</code>, whether in small or large pieces, but it can with <code>mmap</code>. Ain't seen that before. One day shot. Then, towards the end of day 2, load begins. But it does not run for more than 15 minutes before a network error causes the whole thing to abort. All subsequent tries die within 15 minutes. Then, in the morning of day 3, we switch from IB to Gigabit Ethernet (GigE). For loading this is all the same; the maximal aggregate throughput is 800 MB/s, which is around 40% of the nominal bidirectional capacity of 12 GigE's. So, it works better, for 30 minutes, and one can even stop the load and do a checkpoint. But after resuming, one box just dies; does not even respond to ping. We change this to another. After this, still running on GigE, there are no more network errors. So, at the end of day 3, maybe 10% of the data are in. But now it takes 2h21min to make a checkpoint, i.e., make the loaded data durable on disk. One of the boxes manages to write 2 MB/s to a RAID-0 of three 2 TB drives. Bad disk, seen such before. The data can however be read back once the write is finally done.</p> <p>Well, this is a non-starter. So, by mid-day of day 4, another machine has been replaced. Now writing to disk is possible within expected delays.</p> <p>In the afternoon of day 4, the load rate is about 4.3 Mega-triples (Mt) per second, all going in RAM.</p> <p>In the evening of day 4, adding more files to load in parallel increases the load rate to between 4.9 and 5.2 Mt/s. This is about as fast as this will go, since the load is not exactly even. This comes from the RDF stupidity of keeping an index on everything, so even object values where an index is useless get indexed, leading to some load peaks. For example, there is an index on <code>POSG</code> for triples were the predicate is <code>rdf:type</code> and the object is a common type. Use of characteristic sets will stop this nonsense.</p> <p>But let us not get ahead of the facts: At 9:10 PM of day 4, the whole cluster goes unreachable. No, this is not a software crash or swapping; this also affects boxes on which nothing of the experiment was running. A whole night of running is shot. </p> <p>A previous scale model experiment of loading 37.5 Gt in 192 GB of RAM, paging to a pair of 2 TB disks, has been done a week before. This finishes in time, keeping a load rate of above 400 Kt/s on a 12-core box.</p> <p>At 10AM on day 5 (Friday), the cluster is rebooted; a whole night's run missed. The cluster starts and takes about 30 minutes to get to its former 5 Mt/s load rate. We now try switching the network back to InfiniBand. The whole ethernet network seemed to have crashed at 9PM on day 4. This is of course unexplained but the experiment had been driving the ethernet at about half its cross-sectional throughput, so maybe a switch crashed. We will never know. We will now try IB rather than risk this happening again, especially since if it did repeat, the whole weekend would be shot, as we would have to wait for the admin to reboot the lot on Monday (day 8).</p> <p>So, at noon on day 5, the cluster is restarted with IB. The cruising speed is now 6.2 Mt/s, thanks to the faster network. The cross sectional throughput is about 960 MB/s, up from 720 MB/s, which accounts for the difference. CPU load is correspondingly up. This is still not full platform since there is load unbalance as noted above.</p> <p>At 9PM on day 5, the rate is around 5.7 Mt/s with the peak node at 1500% CPU out of a possible 1600%. The next one is under 800%, which is just to show what it means to index everything. In specific, the node that has the highest CPU is the one in whose partition the <code>bsbm:offer</code> class falls, so that there is a local peak since one of every 9 or so triples says that something is an <code>offer</code>. The stupidity of the triple store is to index garbage like this to begin with. The reason why the performance is still good is that a <code>POSG</code> index where <code>P</code> and <code>O</code> are fixed and the <code>S</code> is densely ascending is very good, with everything but the <code>S</code> represented as run lengths and the <code>S</code> as bitmaps. Still, no representation at all is better for performance than even the most efficient representation.</p> <p>The journey consists of 3 different parts. At 10PM, the 3rd and last part is started. The triples have more literals, but the load is more even. The cruising speed is 4.3 Mt/s down from 6.2, but the data has a different shape, including more literals.</p> <p>The last stretch of the data is about reviews. This stretch of the data has less skew. So we increase parallelism, running 8 x 24 files at a time. The load rate goes above 6.3 Mt/s.</p> <p>At 6:45 in the morning of day 6, the data is all loaded. The count of triples is 490.0 billion. If the load were done in a single stretch without stops and reconfiguration, it would likely go in under 24h. The average rate for a 4 hour sample between midnight and 4AM of day 6 is 6.8 MT/s. The resulting database files add up to 10.9 TB, with about 20% of the volume in unallocated pages.</p> <p>At this time, noon of day 6, we find that some cross-partition joins need more distinct pieces of memory than the default kernel settings allow per process. A large number of partitions makes a large number of sometimes long messages which makes many <code>mmaps</code>. So we will wait until morning of day 8 (Monday) for the administrator to set these. In the meantime, we analyze the behavior of the workload on the 37 Gt scale model cluster on my desktop.</p> <p>To be continued...</p> <h3>LOD2 Finale Series</h3> <ul> <li> <a id="link-id0x2aabe4b591b8">Indexing everything</a> </li> <li> Having literals and URI strings via dictionary </li> <li> Having a join for every attribute </li> </ul>
2014-09-06T20:48:58.008009-04:00
LOD2 Finale (part 1 of n): RDF Before The Dawn
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-08-18#1806
2014-08-18T20:55:52Z
<p>The <a href="http://lod2.eu/Welcome.html" id="link-id0x2aabceb4bed8">LOD2 FP7</a> ends at the end of August, 2014. This post begins a series that will crown the project with a grand finale, another decisive step towards the project’s chief goal of giving RDF and linked data performance parity with SQL systems.</p> <p>In a nutshell, LOD2 went like this:</p> <ol> <li> <p>Triples were done right, taking the best of the column store world and adapting it to <a href="http://dbpedia.org/page/Resource_Description_Framework" id="link-id0x2aabf08d88b8">RDF</a>. This is now in widespread use.</p> </li> <li> <p> <a href="http://dbpedia.org/page/SQL" id="link-id0x2aabf08cc318">SQL</a> was done right, as I have described in detail in the <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aabf0880b28">TPC-H series</a>. This is generally available as open source in <a href="https://github.com/v7fasttrack" id="link-id0x2aabf086a3e8">v7fasttrack</a>. SQL is the senior science and a runner-up like sem-tech will not carry the day without mastering this.</p> </li> <li> <p>RDF is now breaking free of the triple store. RDF is a very general, minimalistic way of talking about things. It is not a prescription on how to do database. Confusing these two things has given rise to RDF’s relative cost against alternatives. To cap off LOD2, we will have the flexibility of triples with the speed of the best SQL.</p> </li> </ol> <p>In this post we will look at accomplishments so far and outline what is to follow during August. We will also look at what in fact constitutes the RDF overhead, why this is presently so, and why this does not have to stay thus.</p> <p>This series will be of special interest to anybody concerned with RDF efficiency and scalability.</p> <p>At the beginning of LOD2, I wrote <a href="http://www.openlinksw.com/www.openlinksw.com/weblog/oerling/?id=1627" id="link-id0x2aabcfb37208">a blog post discussing the RDF technology and its planned revolution in terms of the legend of Perseus</a>. The classics give us exemplars and archetypes, but actual histories seldom follow them one-to-one; rather, events may have a fractal nature where subplots reproduce the overall scheme of the containing story.</p> <p>So it is also with LOD2: The Promethean pattern of fetching the fire (state of the art of the column store) from the gods (the DB world) and bringing it to fuel the campfires of the primitive semantic tribes is one phase, but it is not the totality. This is successfully concluded, and Virtuoso 7 is widely used at present. Space efficiency gains are about 3x over the previous, with performance gains anywhere from 3 to 100x. As pointed out in the Star Schema Benchmark series (<a href="http://www.openlinksw.com/weblogs/oerling/?id=1732" id="link-id0x2aabf08c5b08">part 1</a> and <a href="http://www.openlinksw.com/weblogs/oerling/?id=1733" id="link-id0x2aabf086a528">part 2</a>), in the good case one can run circles in SPARQL around anything but the best SQL analytics databases.</p> <p>In the larger scheme of things, this is just preparation. In the classical pattern, there is the call or the crisis: Presently this is that having done triples about as right as they can be done, the mediocre in SQL can be vanquished, but the best cannot. Then there is the actual preparation: Perseus talking to Athena and receiving the shield of polished brass and the winged sandals. In the present case, this is my second pilgrimage to Mount Database, consisting of the <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aabf081f358">TPC-H series</a>. Now, the incense has been burned and libations offered at each of the 22 stations. This is not reading papers, but personally making one of the best-ever implementations of this foundational workload. This establishes Virtuoso as one of the top-of-the-line SQL analytics engines. The RDF public, which is anyway the principal Virtuoso constituency today, may ask what this does for them.</p> <p>Well, without this step, the LOD2 goal of performance parity with SQL would be both meaningless and unattainable. The goal of parity is worth something only if you compare the RDF contestant to the very best SQL. And the comparison cannot possibly be successful unless it incorporates the very same hard core of down-to-the-metal competence the SQL world has been pursuing now for over forty years.</p> <p>It is now time to cut the Gorgon’s head. The knowledge and prerequisite conditions exist.</p> <p>The epic story is mostly about principles. If it is about personal combat, the persons stand for values and principles rather than for individuals. Here the enemy is actually an illusion, an error of perception, that has kept RDF in chains all this time. Yes, RDF is defined as a data model with triples in named graphs, i.e., quads. If nothing else is said, an RDF Store is a thing that can take arbitrary triples and retrieve them with SPARQL. The naïve implementation is to store things as rows in a quad table, indexed in any number of ways. There have been other approaches suggested, such as property tables or materialized views of some joins, but these tend to flush the baby with the bathwater: If RDF is used in the first place, it is used for its schema-less-ness and for having global identifiers. In some cases, there is also some inference, but the matter of schema-less-ness and identifiers predominates.</p> <p>We need to go beyond a triple table and a dictionary of URI names while maintaining the present semantics and flexibility. Nobody said that physical structure needs to follow this. Everybody just implements things this way because this is the minimum that will in any case be required. Combining this with a SQL database for some other part of the data/workload hits basically insoluble problems of impedance mismatch between the SQL and SPARQL type systems, maybe using multiple servers for different parts of a query, etc. But if you own one of the hottest SQL racers in DB city and can make it do anything you want, most of these problems fall away.</p> <p>The idea is simple: Put the <i>de facto</i> rectangular part of RDF data into tables; do not naively index everything in places where an index gives no benefit; keep the irregular or sparse part of the data as quads. Optimize queries according to the table-like structure, as that is where the volume is and where getting the best plan is a make or break matter, as we saw in the TPC-H series. Then, execute in a way where the details of the physical plan track the data; i.e., sometimes the operator is on a table, sometimes on triples, for the long tail of exceptions.</p> <p>In the next articles we will look at how this works and what the gains are.</p> <p>These experiments will for the first time showcase the <i>adaptive schema</i> features of the Virtuoso RDF store. Some of these features will be commercial only, but the interested will be able to reproduce the single server experiments themselves using the v7fasttrack open source preview. This will be updated around the second week of September to give a preview of this with BSBM and possibly some other datasets, e.g., Uniprot. Performance gains for regular datasets will be very large.</p> <p>To be continued...</p> <h3>LOD2 Finale Series</h3> <ul> <li> LOD2 Finale (part 1): RDF Before The Dawn<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1805" id="link-id0x2aabf08c6048"> LOD2 Finale (part 2): The 500 Giga-triples</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1808" id="link-id0x2aabc37bf8e8"> LOD2 Finale (part 3): The 500 Giga-triple Runs</a> </li> </ul>
2014-08-29T12:50:47.730911-04:00
LOD2 Finale (part 2 of n): The 500 Giga-triples
http://www.openlinksw.com:443/weblog/oerling/?date=2014-08-18#1805
2014-08-18T20:54:38Z
<p>No epic is complete without a descent into hell. Enter the <i>historia calamitatum</i> of the 500 Giga-triples (Gt) at <a href="http://www.cwi.nl/" id="link-id0x15b8cbb8">CWI</a>'s <a href="http://scilens.project.cwi.nl/" id="link-id0x15ead658">Scilens</a> cluster.</p> <p>Now, <a href="http://www.openlinksw.com/weblog/oerling/?id=1767" id="link-id0x15a53918">from last time</a>, we know to generate the data without 10 GB of namespace prefixes per file and with many short files. So we have 1.5 TB of gzipped data in 40,000 files, spread over 12 machines. The data generator has again been modified. Now the generation was about 4 days. Also from last time, we know to treat small integers specially when they occur as partition keys: 1 and 2 are very common values and skew becomes severe if they all go to the same partition; hence consecutive small <code>INTs</code> each go to a different partition, but for larger ones the low 8 bits are ignored, which is good for compression: Consecutive values must fall in consecutive places, but not for small <code>INTs</code>. Another uniquely brain-dead feature of the BSBM generator has also been rectified: When generating multiple files, the program would put things in files in a round-robin manner, instead of putting consecutive numbers in consecutive places, which is how every other data generator or exporter does it. This impacts bulk load locality and as you, dear reader, ought to know by now, performance comes from (1) locality and (2) parallelism.</p> <p>The machines are similar to last time: each a dual E5 2650 v2 with 256 GB RAM and QDR InfiniBand (IB). No SSD this time, but a slightly higher clock than last time; anyway, a different set of machines.</p> <p>The first experiment is with triples, so no characteristic sets, no schema.</p> <p>So, first day (Monday), we notice that one cannot allocate more than 9 GB of memory. Then we figure out that it cannot be done with <code>malloc</code>, whether in small or large pieces, but it can with <code>mmap</code>. Ain't seen that before. One day shot. Then, towards the end of day 2, load begins. But it does not run for more than 15 minutes before a network error causes the whole thing to abort. All subsequent tries die within 15 minutes. Then, in the morning of day 3, we switch from IB to Gigabit Ethernet (GigE). For loading this is all the same; the maximal aggregate throughput is 800 MB/s, which is around 40% of the nominal bidirectional capacity of 12 GigE's. So, it works better, for 30 minutes, and one can even stop the load and do a checkpoint. But after resuming, one box just dies; does not even respond to ping. We change this to another. After this, still running on GigE, there are no more network errors. So, at the end of day 3, maybe 10% of the data are in. But now it takes 2h21min to make a checkpoint, i.e., make the loaded data durable on disk. One of the boxes manages to write 2 MB/s to a RAID-0 of three 2 TB drives. Bad disk, seen such before. The data can however be read back once the write is finally done.</p> <p>Well, this is a non-starter. So, by mid-day of day 4, another machine has been replaced. Now writing to disk is possible within expected delays.</p> <p>In the afternoon of day 4, the load rate is about 4.3 Mega-triples (Mt) per second, all going in RAM.</p> <p>In the evening of day 4, adding more files to load in parallel increases the load rate to between 4.9 and 5.2 Mt/s. This is about as fast as this will go, since the load is not exactly even. This comes from the RDF stupidity of keeping an index on everything, so even object values where an index is useless get indexed, leading to some load peaks. For example, there is an index on <code>POSG</code> for triples were the predicate is <code>rdf:type</code> and the object is a common type. Use of characteristic sets will stop this nonsense.</p> <p>But let us not get ahead of the facts: At 9:10 PM of day 4, the whole cluster goes unreachable. No, this is not a software crash or swapping; this also affects boxes on which nothing of the experiment was running. A whole night of running is shot. </p> <p>A previous scale model experiment of loading 37.5 Gt in 192 GB of RAM, paging to a pair of 2 TB disks, has been done a week before. This finishes in time, keeping a load rate of above 400 Kt/s on a 12-core box.</p> <p>At 10AM on day 5 (Friday), the cluster is rebooted; a whole night's run missed. The cluster starts and takes about 30 minutes to get to its former 5 Mt/s load rate. We now try switching the network back to InfiniBand. The whole ethernet network seemed to have crashed at 9PM on day 4. This is of course unexplained but the experiment had been driving the ethernet at about half its cross-sectional throughput, so maybe a switch crashed. We will never know. We will now try IB rather than risk this happening again, especially since if it did repeat, the whole weekend would be shot, as we would have to wait for the admin to reboot the lot on Monday (day 8).</p> <p>So, at noon on day 5, the cluster is restarted with IB. The cruising speed is now 6.2 Mt/s, thanks to the faster network. The cross sectional throughput is about 960 MB/s, up from 720 MB/s, which accounts for the difference. CPU load is correspondingly up. This is still not full platform since there is load unbalance as noted above.</p> <p>At 9PM on day 5, the rate is around 5.7 Mt/s with the peak node at 1500% CPU out of a possible 1600%. The next one is under 800%, which is just to show what it means to index everything. In specific, the node that has the highest CPU is the one in whose partition the <code>bsbm:offer</code> class falls, so that there is a local peak since one of every 9 or so triples says that something is an <code>offer</code>. The stupidity of the triple store is to index garbage like this to begin with. The reason why the performance is still good is that a <code>POSG</code> index where <code>P</code> and <code>O</code> are fixed and the <code>S</code> is densely ascending is very good, with everything but the <code>S</code> represented as run lengths and the <code>S</code> as bitmaps. Still, no representation at all is better for performance than even the most efficient representation.</p> <p>The journey consists of 3 different parts. At 10PM, the 3rd and last part is started. The triples have more literals, but the load is more even. The cruising speed is 4.3 Mt/s down from 6.2, but the data has a different shape, including more literals.</p> <p>The last stretch of the data is about reviews. This stretch of the data has less skew. So we increase parallelism, running 8 x 24 files at a time. The load rate goes above 6.3 Mt/s.</p> <p>At 6:45 in the morning of day 6, the data is all loaded. The count of triples is 490.0 billion. If the load were done in a single stretch without stops and reconfiguration, it would likely go in under 24h. The average rate for a 4 hour sample between midnight and 4AM of day 6 is 6.8 MT/s. The resulting database files add up to 10.9 TB, with about 20% of the volume in unallocated pages.</p> <p>At this time, noon of day 6, we find that some cross-partition joins need more distinct pieces of memory than the default kernel settings allow per process. A large number of partitions makes a large number of sometimes long messages which makes many <code>mmaps</code>. So we will wait until morning of day 8 (Monday) for the administrator to set these. In the meantime, we analyze the behavior of the workload on the 37 Gt scale model cluster on my desktop.</p> <p>To be continued...</p> <h3>LOD2 Finale Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1804" id="link-id0x1623e958"> LOD2 Finale (part 1): RDF Before The Dawn</a> </li> <li> LOD2 Finale (part 2): The 500 Giga-triples<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1808" id="link-id0x2aabec4df498"> LOD2 Finale (part 3): The 500 Giga-triple Runs</a> </li> </ul>
2014-08-29T12:50:27.602886-04:00
LOD2 Finale (part 1 of n): RDF Before The Dawn
http://www.openlinksw.com:443/weblog/oerling/?date=2014-08-18#1804
2014-08-18T20:54:12Z
<p>The <a href="http://lod2.eu/Welcome.html" id="link-id0x2aabceb4bed8">LOD2 FP7</a> ends at the end of August, 2014. This post begins a series that will crown the project with a grand finale, another decisive step towards the project’s chief goal of giving RDF and linked data performance parity with SQL systems.</p> <p>In a nutshell, LOD2 went like this:</p> <ol> <li> <p>Triples were done right, taking the best of the column store world and adapting it to <a href="http://dbpedia.org/page/Resource_Description_Framework" id="link-id0x2aabf08d88b8">RDF</a>. This is now in widespread use.</p> </li> <li> <p> <a href="http://dbpedia.org/page/SQL" id="link-id0x2aabf08cc318">SQL</a> was done right, as I have described in detail in the <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aabf0880b28">TPC-H series</a>. This is generally available as open source in <a href="https://github.com/v7fasttrack" id="link-id0x2aabf086a3e8">v7fasttrack</a>. SQL is the senior science and a runner-up like sem-tech will not carry the day without mastering this.</p> </li> <li> <p>RDF is now breaking free of the triple store. RDF is a very general, minimalistic way of talking about things. It is not a prescription on how to do database. Confusing these two things has given rise to RDF’s relative cost against alternatives. To cap off LOD2, we will have the flexibility of triples with the speed of the best SQL.</p> </li> </ol> <p>In this post we will look at accomplishments so far and outline what is to follow during August. We will also look at what in fact constitutes the RDF overhead, why this is presently so, and why this does not have to stay thus.</p> <p>This series will be of special interest to anybody concerned with RDF efficiency and scalability.</p> <p>At the beginning of LOD2, I wrote <a href="http://www.openlinksw.com/www.openlinksw.com/weblog/oerling/?id=1627" id="link-id0x2aabcfb37208">a blog post discussing the RDF technology and its planned revolution in terms of the legend of Perseus</a>. The classics give us exemplars and archetypes, but actual histories seldom follow them one-to-one; rather, events may have a fractal nature where subplots reproduce the overall scheme of the containing story.</p> <p>So it is also with LOD2: The Promethean pattern of fetching the fire (state of the art of the column store) from the gods (the DB world) and bringing it to fuel the campfires of the primitive semantic tribes is one phase, but it is not the totality. This is successfully concluded, and Virtuoso 7 is widely used at present. Space efficiency gains are about 3x over the previous, with performance gains anywhere from 3 to 100x. As pointed out in the Star Schema Benchmark series (<a href="http://www.openlinksw.com/weblogs/oerling/?id=1732" id="link-id0x2aabf08c5b08">part 1</a> and <a href="http://www.openlinksw.com/weblogs/oerling/?id=1733" id="link-id0x2aabf086a528">part 2</a>), in the good case one can run circles in SPARQL around anything but the best SQL analytics databases.</p> <p>In the larger scheme of things, this is just preparation. In the classical pattern, there is the call or the crisis: Presently this is that having done triples about as right as they can be done, the mediocre in SQL can be vanquished, but the best cannot. Then there is the actual preparation: Perseus talking to Athena and receiving the shield of polished brass and the winged sandals. In the present case, this is my second pilgrimage to Mount Database, consisting of the <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aabf081f358">TPC-H series</a>. Now, the incense has been burned and libations offered at each of the 22 stations. This is not reading papers, but personally making one of the best-ever implementations of this foundational workload. This establishes Virtuoso as one of the top-of-the-line SQL analytics engines. The RDF public, which is anyway the principal Virtuoso constituency today, may ask what this does for them.</p> <p>Well, without this step, the LOD2 goal of performance parity with SQL would be both meaningless and unattainable. The goal of parity is worth something only if you compare the RDF contestant to the very best SQL. And the comparison cannot possibly be successful unless it incorporates the very same hard core of down-to-the-metal competence the SQL world has been pursuing now for over forty years.</p> <p>It is now time to cut the Gorgon’s head. The knowledge and prerequisite conditions exist.</p> <p>The epic story is mostly about principles. If it is about personal combat, the persons stand for values and principles rather than for individuals. Here the enemy is actually an illusion, an error of perception, that has kept RDF in chains all this time. Yes, RDF is defined as a data model with triples in named graphs, i.e., quads. If nothing else is said, an RDF Store is a thing that can take arbitrary triples and retrieve them with SPARQL. The naïve implementation is to store things as rows in a quad table, indexed in any number of ways. There have been other approaches suggested, such as property tables or materialized views of some joins, but these tend to flush the baby with the bathwater: If RDF is used in the first place, it is used for its schema-less-ness and for having global identifiers. In some cases, there is also some inference, but the matter of schema-less-ness and identifiers predominates.</p> <p>We need to go beyond a triple table and a dictionary of URI names while maintaining the present semantics and flexibility. Nobody said that physical structure needs to follow this. Everybody just implements things this way because this is the minimum that will in any case be required. Combining this with a SQL database for some other part of the data/workload hits basically insoluble problems of impedance mismatch between the SQL and SPARQL type systems, maybe using multiple servers for different parts of a query, etc. But if you own one of the hottest SQL racers in DB city and can make it do anything you want, most of these problems fall away.</p> <p>The idea is simple: Put the <i>de facto</i> rectangular part of RDF data into tables; do not naively index everything in places where an index gives no benefit; keep the irregular or sparse part of the data as quads. Optimize queries according to the table-like structure, as that is where the volume is and where getting the best plan is a make or break matter, as we saw in the TPC-H series. Then, execute in a way where the details of the physical plan track the data; i.e., sometimes the operator is on a table, sometimes on triples, for the long tail of exceptions.</p> <p>In the next articles we will look at how this works and what the gains are.</p> <p>These experiments will for the first time showcase the <i>adaptive schema</i> features of the Virtuoso RDF store. Some of these features will be commercial only, but the interested will be able to reproduce the single server experiments themselves using the v7fasttrack open source preview. This will be updated around the second week of September to give a preview of this with BSBM and possibly some other datasets, e.g., Uniprot. Performance gains for regular datasets will be very large.</p> <p>To be continued...</p> <h3>LOD2 Finale Series</h3> <ul> <li> LOD2 Finale (part 1): RDF Before The Dawn<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1805" id="link-id0x2aabf08c6048"> LOD2 Finale (part 2): The 500 Giga-triples</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1808" id="link-id0x2aabec790038"> LOD2 Finale (part 3): The 500 Giga-triple Runs</a> </li> </ul>
2014-08-29T12:50:31.754399-04:00
In Hoc Signo Vinces (part 15 of n): TPC-H and the Science of Hash
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-05-28#1803
2014-05-28T21:12:18Z
<p>This piece is dedicated to <a href="http://homepages.cwi.nl/~boncz/" id="link-id0x2aac213639d8">Peter Boncz</a>, architect of <a href="http://dbpedia.org/page/Vectorwise" id="link-id0x2aac22b5bc88">Actian Vector</a> and <a href="http://dbpedia.org/page/MonetDB" id="link-id0x2aac22b5bc28">MonetDB</a>.</p> <p>Query optimization is hard. It is a set of mutually interacting tricks and special cases. Execution is also hard, but there the tricks do not interact quite as much or as unpredictably. So, if there is a few percent of score to be had from optimization of either execution or query, I will take execution first. It is less likely to break things and will probably benefit a larger set of use cases.</p> <p>As we see from the profile in <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac2287c4f8">the previous article</a>, hash join is the main piece of execution in <a href="http://www.tpc.org/tpch/" id="link-id0x2aac2287c4b8">TPC-H</a>. So between the <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac2287c498">article on late projection</a> and the <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac2287c408">first result preview</a>, I changed the hash table used in <code><a href="http://dbpedia.org/page/Hash_join" id="link-id0x2aac21e835e8">HASH JOIN</a></code> and <code>GROUP BY</code> from cuckoo to linear.</p> <p>Let's see how the hash tables work: <a href="http://dbpedia.org/page/Cuckoo_hashing" id="link-id0x2aac20599e98">Cuckoo hash</a> is a scheme where an entry can be in one of two possible places in the table. If a new entry is inserted and either of the possible places is unoccupied, it goes there. If both are occupied, it could be that one contains an entry whose other possible location is free -- and then that entry may be relocated. Thus an insert may push the previous occupant of the place somewhere else, which in turn may push another, and so on. It may happen that an insert is still not possible, in which case the entry to insert goes into an exceptions list.</p> <p>To look up an entry, you get a hash number, and use different fields of it to pick the two places. Look in one, then the other, then the exceptions. If there is no match and the table is reasonably close to capacity, you will have looked in at least 3 widely-separated places to determine the absence of a match. In practice, the hash table consists on a prime number of distinct arrays of a fixed size (partitions), and each partition has its own exception list. A modulo of the hash number picks the array, then two further modulos of different parts of the number pick the places in the array.</p> <p>In most cases in TPC-H, the hash joins are selective; i.e., most items on the probe side find no match in the hash table.</p> <p>So, quite often you have 3 cache misses to show that there is no hit. This is, at least in theory, quite bad.</p> <p>There are <a href="http://dbpedia.org/page/Bloom_filter" id="link-id0x2aac22b2ffb8">Bloom filters</a> before the hash table. The Bloom filter will prune most of the probes that would miss. A Bloom filter is an array of bits. Given a hash number, the Bloom filter will very efficiently tell you whether the entry is sure <i>not</i> to be in the hash table. If the Bloom filter says it <i>can</i> be in the hash table, you must look.</p> <p>In the Virtuoso case, for each entry in the hash table, the Bloom filter has 8 bits. The Bloom check uses a field of the hash number to pick a 64-bit word from the Bloom filter. Then different fields of the hash number are used to set up-to 4 bits in a 64-bit bit-mask. When building the hash table, the masks are OR-ed into the Bloom filter. When probing, before looking in the hash table, the system checks to see if the bits corresponding to the hash number are all on in the appropriate word. If they are not, the hash lookup is sure to miss.</p> <p>Most expositions of Bloom filters talk about setting two bits for every value. With two bits set, we found 8 bits-per-value to work best. More bits makes a larger filter and misses the cache more; fewer bits makes too many collisions, and the Bloom filter produces too many false positives. A significant finding is that with 8 bits-per-value, setting 4 bits instead of 2 causes the filter to be twice as selective. The simple trick of setting 4 bits cuts the number of hash lookups for items that passed the Bloom filter to half in many selective hash joins. Examples are the many joins of <code>lineitem</code> and <code>part</code> or <code>supplier</code> where there is a condition on the smaller table.</p> <p>Still, even with Bloom filters, a cuckoo hash will make too many cache misses.</p> <p>So, enter <a href="http://dbpedia.org/page/Linear_hashing" id="link-id0x2aac2219ccc8">linear hash</a>. The idea is simple: The hash number picks a place in an array. Either the entry being sought is in the vicinity, or it is not in the hash table. If the vicinity is full of other entries, the entry can still be in an exception list.</p> <p>With this and cuckoo alike, there are 3 different variants of hash table:</p> <ol> <li> <p>A set of single unique integers</p> </li> <li> <p>A single-integer key with 0 or more dependent values, possibly with a next link if the key is not unique</p> </li> <li> <p>A key of <i>n</i> arbitrary values, 0 or more dependent values, optional next link if the key is not unique</p> </li> </ol> <p>In the first case, the hash table is an array of values; in the two other cases, it is an array of pointers. But since a pointer is 64 bits, of which the high 16 are not in the address space of x86_64, these high bits can be used to keep a part of the hash number. It will be necessary to dereference the pointer only if the high bits match the hash number. This means that nearly all lookups that do not find a match are handled with a single cache miss.</p> <p>Each cache miss brings in a cache line of 8 words. The lookup starts at a point given by the hash number and wraps around at the end of the cache line. Only in the case that all 8 words are occupied but do not match does one need to look at the exceptions. There is one exception list for each partition of the hash table, like in the cuckoo scheme.</p> <p>A hash lookup is always done on a vector of keys; the loop that takes most of the time is in fact the Bloom filter check. It goes as follows:</p> <blockquote> <code><pre> #define CHB_VAR(n) \ uint64 h##n, w##n, mask##n; #define CHB_INIT(n, i) \ MHASH_STEP_1 (h##n, i); \ w##n = bf[BF_WORD (h##n, sz)]; \ mask##n = BF_MASK (h##n); #define CHB_CK(n) \ { matches[mfill] = inx + n; \ mfill += (w##n & mask##n) == mask##n; } for (inx = inx; inx < last; inx ++) { CHB_VAR (0); CHB_INIT (0, REF ((ce_first + sizeof (ELT_T) * inx))); CHB_CK (0); } </pre> </code> </blockquote> <p>This is the perfect loop for out-of-order execution. Now, I have tried every variation you can imagine, and this does not get better. The loop calculates a hash number, fetches the corresponding word from the Bloom filter, calculates a mask, stores the index of the key in a results array, and increments the results counter if all the bits were set. There is no control dependency anywhere, just a data dependency between successive iterations; i.e., to know where the result must go, you must know if the previous was a hit.</p> <p>You can unroll this loop very easily, so, for example, take 4 keys, do the numbers, fetch the words, and then check them one after the other. One would think this would have more misses in flight at any one time, which it does. But it does not run any faster.</p> <p>Maybe the loop is too long. Circumstantial evidence suggests that short loops are better for instruction prefetching. So, one can also make a loop that gets any number of words of the Bloom filter and puts them in one local array and the hash numbers in another array. A subsequent loop then reads the hash number, calculates the mask, and checks if there is a hit. In this way one can generate as many misses as one wants and check them as late as one wants. It so happens that doing 8 misses and then checking them is better than either 4 or 16. But 8 is still marginally worse than the loop first mentioned.</p> <p>One can also vary the test. Instead of adding a truth value to the result counter, one can have</p> <blockquote> <code><pre>if ((word & mask) == mask) result[fill++] = inx;</pre> </code> </blockquote> <p>There is no clear difference between predication (incrementing the fill by truth value) and a conditional jump. The theory of out-of-order execution would predict predication to be better, but the difference is lost in measurement noise. This is true on both Intel Nehalem (Xeon 55xx) and Sandy Bridge (E5 26xx), but could be different on other architectures.</p> <p>The multicore scalability of the test will give some information about platform utilization.</p> <p>This is the ultimately simplified selective hash join:</p> <blockquote> <code><pre> SELECT COUNT (*) FROM lineitem, part WHERE l_partkey = p_partkey AND p_size < 15 ; </pre> </code> </blockquote> <p>This is the second simplest hash join but misses the cache much more; since this now has a key and a dependent part in the hash table, there is an extra pointer to follow, and the hash entry is two words plus the pointer to these in the hash table array.</p> <blockquote> <code><pre> SELECT SUM (p_retailprice) FROM lineitem, part WHERE l_partkey = p_partkey AND p_size < 15 ; </pre> </code> </blockquote> <p>By adjusting the number of parts selected, we can vary the Bloom filter selectivity and the size of the hash table. Below, we show the times for the two queries with single-thread and 24-thread execution, with different percentages of the <code>part</code> table on the build side of the hash join. All runs are against warm 100G TPC-H on the same test system as in the rest of the TPC-H series (dual Xeon E5-2630).</p> <p>This table compares the performance of the linear and cuckoo implementations on the above queries (count vs. sum) on either 24 threads or 1 thread. Four data points are given for different sizes of hash table, given as percentage of the <code>part</code> table (having 400K - 20M entries) in the hash table. The rightmost column, which represents the case where the entire <code>part</code> table is on the build side does not have a Bloom filter; the other cases do. The Bloom bits are 8/4 for linear and 8/2 for cuckoo. The times are all in milliseconds, and the thousands separator is a comma.</p> <table style="padding:10px;border-spacing:10px;"> <tr> <th style="text-align:center">Hash type</th> <th style="text-align:center">Query type</th> <th style="text-align:center">Threads</th> <th style="text-align:center">2% <br /> (ms)</th> <th style="text-align:center">10% <br /> (ms)</th> <th style="text-align:center">30% <br /> (ms)</th> <th style="text-align:center">100% <br /> (ms)</th> </tr> <tr> </tr> <tr> <th style="text-align:center">Linear</th> <th style="text-align:center"><code>COUNT</code></th> <th style="text-align:center">24</th> <td style="text-align:right">1,204</td> <td style="text-align:right">1,683</td> <td style="text-align:right">3,100</td> <td style="text-align:right">6,214</td> </tr> <tr> <th style="text-align:center">Linear</th> <th style="text-align:center"><code>SUM</code></th> <th style="text-align:center">24</th> <td style="text-align:right">1,261</td> <td style="text-align:right">2,447</td> <td style="text-align:right">5,059</td> <td style="text-align:right">13,086</td> </tr> <tr> <th style="text-align:center">Linear</th> <th style="text-align:center"><code>COUNT</code></th> <th style="text-align:center">1</th> <td style="text-align:right">15,286</td> <td style="text-align:right">22,451</td> <td style="text-align:right">38,863</td> <td style="text-align:right">66,722</td> </tr> <tr> <th style="text-align:center">Linear</th> <th style="text-align:center"><code>SUM</code></th> <th style="text-align:center">1</th> <td style="text-align:right">17,575</td> <td style="text-align:right">33,664</td> <td style="text-align:right">81,927</td> <td style="text-align:right">179,013</td> </tr> <tr> <th style="text-align:center">Cuckoo</th> <th style="text-align:center"><code>COUNT</code></th> <th style="text-align:center">24</th> <td style="text-align:right">1,849</td> <td style="text-align:right">2,840</td> <td style="text-align:right">4,105</td> <td style="text-align:right">6,203</td> </tr> <tr> <th style="text-align:center">Cuckoo</th> <th style="text-align:center"><code>SUM</code></th> <th style="text-align:center">24</th> <td style="text-align:right">2,833</td> <td style="text-align:right">4,903</td> <td style="text-align:right">9,446</td> <td style="text-align:right">19,652</td> </tr> <tr> <th style="text-align:center">Cuckoo</th> <th style="text-align:center"><code>COUNT</code></th> <th style="text-align:center">1</th> <td style="text-align:right">25,146</td> <td style="text-align:right">39,064</td> <td style="text-align:right">57,383</td> <td style="text-align:right">85,105</td> </tr> <tr> <th style="text-align:center">Cuckoo</th> <th style="text-align:center"><code>SUM</code></th> <th style="text-align:center">1</th> <td style="text-align:right">33,647</td> <td style="text-align:right">67,089</td> <td style="text-align:right">121,989</td> <td style="text-align:right">240,941</td> </tr> </table> <p>We clearly see cache effects on the two first lines, where the <code>SUM</code> and <code>COUNT</code> run in almost the same time on a small hash table but have a 2x difference on the larger hash table. The instruction path length is not very different for <code>SUM</code> and <code>COUNT</code>, but the memory footprint has a 3x difference. </p> <p>We note that the SMP scalability of linear is slightly better, contrasting the ratio of 24-thread <code>SUM</code> to single-thread <code>SUM</code>. Both numbers are over 12x, indicating net benefit from core multithreading. (The test system has 12 physical cores.) The linear hash systematically outperforms cuckoo, understandably, since it makes a smaller net number of cache misses. The overall effect on the TPC-H score is noticeable, at around 15-20K units of composite score at 100G.</p> <p>In conclusion, the Virtuoso hash join implementation is certainly on the level, with only small gains to be expected from further vectoring and prefetching. These results may be reproduced using the <a href="https://github.com/v7fasttrack" id="link-id0x2aac22b45068">v7fasttrack</a> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSIndex" id="link-id0x2aac22b45028">Virtuoso Open Source</a> releases from <a href="https://github.com/" id="link-id0x2aac22b44fc8">GitHub</a>; <a href="https://github.com/v7fasttrack/virtuoso-opensource/tree/develop/7" id="link-id0x2aac20ead1d8">develop/7</a> for cuckoo and <a href="https://github.com/v7fasttrack/virtuoso-opensource/tree/feature/analytics" id="link-id0x2aac20ead1b8">feature/analytics</a> for linear hash.</p> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac349bc698"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac347508c8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac347b2098"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac34f71318"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac346fa748"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac349de568"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aac34725b28"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aac36adde48"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aac34a7e368"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac348581b8"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac344340c8"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac34ee7058"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac34d388b8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x69cacf8"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aabdcc22658"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x18d1d108"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aabe8f39e58"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aab64065478"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:09:43.081730-04:00
In Hoc Signo Vinces (part 15 of n): TPC-H and the Science of Hash
http://www.openlinksw.com:443/weblog/oerling/?date=2014-05-28#1802
2014-05-28T21:11:44Z
<p>This piece is dedicated to <a href="http://homepages.cwi.nl/~boncz/" id="link-id0x2aac213639d8">Peter Boncz</a>, architect of <a href="http://dbpedia.org/page/Vectorwise" id="link-id0x2aac22b5bc88">Actian Vector</a> and <a href="http://dbpedia.org/page/MonetDB" id="link-id0x2aac22b5bc28">MonetDB</a>.</p> <p>Query optimization is hard. It is a set of mutually interacting tricks and special cases. Execution is also hard, but there the tricks do not interact quite as much or as unpredictably. So, if there is a few percent of score to be had from optimization of either execution or query, I will take execution first. It is less likely to break things and will probably benefit a larger set of use cases.</p> <p>As we see from the profile in <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac2287c4f8">the previous article</a>, hash join is the main piece of execution in <a href="http://www.tpc.org/tpch/" id="link-id0x2aac2287c4b8">TPC-H</a>. So between the <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac2287c498">article on late projection</a> and the <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac2287c408">first result preview</a>, I changed the hash table used in <code><a href="http://dbpedia.org/page/Hash_join" id="link-id0x2aac21e835e8">HASH JOIN</a></code> and <code>GROUP BY</code> from cuckoo to linear.</p> <p>Let's see how the hash tables work: <a href="http://dbpedia.org/page/Cuckoo_hashing" id="link-id0x2aac20599e98">Cuckoo hash</a> is a scheme where an entry can be in one of two possible places in the table. If a new entry is inserted and either of the possible places is unoccupied, it goes there. If both are occupied, it could be that one contains an entry whose other possible location is free -- and then that entry may be relocated. Thus an insert may push the previous occupant of the place somewhere else, which in turn may push another, and so on. It may happen that an insert is still not possible, in which case the entry to insert goes into an exceptions list.</p> <p>To look up an entry, you get a hash number, and use different fields of it to pick the two places. Look in one, then the other, then the exceptions. If there is no match and the table is reasonably close to capacity, you will have looked in at least 3 widely-separated places to determine the absence of a match. In practice, the hash table consists on a prime number of distinct arrays of a fixed size (partitions), and each partition has its own exception list. A modulo of the hash number picks the array, then two further modulos of different parts of the number pick the places in the array.</p> <p>In most cases in TPC-H, the hash joins are selective; i.e., most items on the probe side find no match in the hash table.</p> <p>So, quite often you have 3 cache misses to show that there is no hit. This is, at least in theory, quite bad.</p> <p>There are <a href="http://dbpedia.org/page/Bloom_filter" id="link-id0x2aac22b2ffb8">Bloom filters</a> before the hash table. The Bloom filter will prune most of the probes that would miss. A Bloom filter is an array of bits. Given a hash number, the Bloom filter will very efficiently tell you whether the entry is sure <i>not</i> to be in the hash table. If the Bloom filter says it <i>can</i> be in the hash table, you must look.</p> <p>In the Virtuoso case, for each entry in the hash table, the Bloom filter has 8 bits. The Bloom check uses a field of the hash number to pick a 64-bit word from the Bloom filter. Then different fields of the hash number are used to set up-to 4 bits in a 64-bit bit-mask. When building the hash table, the masks are OR-ed into the Bloom filter. When probing, before looking in the hash table, the system checks to see if the bits corresponding to the hash number are all on in the appropriate word. If they are not, the hash lookup is sure to miss.</p> <p>Most expositions of Bloom filters talk about setting two bits for every value. With two bits set, we found 8 bits-per-value to work best. More bits makes a larger filter and misses the cache more; fewer bits makes too many collisions, and the Bloom filter produces too many false positives. A significant finding is that with 8 bits-per-value, setting 4 bits instead of 2 causes the filter to be twice as selective. The simple trick of setting 4 bits cuts the number of hash lookups for items that passed the Bloom filter to half in many selective hash joins. Examples are the many joins of <code>lineitem</code> and <code>part</code> or <code>supplier</code> where there is a condition on the smaller table.</p> <p>Still, even with Bloom filters, a cuckoo hash will make too many cache misses.</p> <p>So, enter <a href="http://dbpedia.org/page/Linear_hashing" id="link-id0x2aac2219ccc8">linear hash</a>. The idea is simple: The hash number picks a place in an array. Either the entry being sought is in the vicinity, or it is not in the hash table. If the vicinity is full of other entries, the entry can still be in an exception list.</p> <p>With this and cuckoo alike, there are 3 different variants of hash table:</p> <ol> <li> <p>A set of single unique integers</p> </li> <li> <p>A single-integer key with 0 or more dependent values, possibly with a next link if the key is not unique</p> </li> <li> <p>A key of <i>n</i> arbitrary values, 0 or more dependent values, optional next link if the key is not unique</p> </li> </ol> <p>In the first case, the hash table is an array of values; in the two other cases, it is an array of pointers. But since a pointer is 64 bits, of which the high 16 are not in the address space of x86_64, these high bits can be used to keep a part of the hash number. It will be necessary to dereference the pointer only if the high bits match the hash number. This means that nearly all lookups that do not find a match are handled with a single cache miss.</p> <p>Each cache miss brings in a cache line of 8 words. The lookup starts at a point given by the hash number and wraps around at the end of the cache line. Only in the case that all 8 words are occupied but do not match does one need to look at the exceptions. There is one exception list for each partition of the hash table, like in the cuckoo scheme.</p> <p>A hash lookup is always done on a vector of keys; the loop that takes most of the time is in fact the Bloom filter check. It goes as follows:</p> <blockquote> <code><pre> #define CHB_VAR(n) \ uint64 h##n, w##n, mask##n; #define CHB_INIT(n, i) \ MHASH_STEP_1 (h##n, i); \ w##n = bf[BF_WORD (h##n, sz)]; \ mask##n = BF_MASK (h##n); #define CHB_CK(n) \ { matches[mfill] = inx + n; \ mfill += (w##n & mask##n) == mask##n; } for (inx = inx; inx < last; inx ++) { CHB_VAR (0); CHB_INIT (0, REF ((ce_first + sizeof (ELT_T) * inx))); CHB_CK (0); } </pre> </code> </blockquote> <p>This is the perfect loop for out-of-order execution. Now, I have tried every variation you can imagine, and this does not get better. The loop calculates a hash number, fetches the corresponding word from the Bloom filter, calculates a mask, stores the index of the key in a results array, and increments the results counter if all the bits were set. There is no control dependency anywhere, just a data dependency between successive iterations; i.e., to know where the result must go, you must know if the previous was a hit.</p> <p>You can unroll this loop very easily, so, for example, take 4 keys, do the numbers, fetch the words, and then check them one after the other. One would think this would have more misses in flight at any one time, which it does. But it does not run any faster.</p> <p>Maybe the loop is too long. Circumstantial evidence suggests that short loops are better for instruction prefetching. So, one can also make a loop that gets any number of words of the Bloom filter and puts them in one local array and the hash numbers in another array. A subsequent loop then reads the hash number, calculates the mask, and checks if there is a hit. In this way one can generate as many misses as one wants and check them as late as one wants. It so happens that doing 8 misses and then checking them is better than either 4 or 16. But 8 is still marginally worse than the loop first mentioned.</p> <p>One can also vary the test. Instead of adding a truth value to the result counter, one can have</p> <blockquote> <code><pre>if ((word & mask) == mask) result[fill++] = inx;</pre> </code> </blockquote> <p>There is no clear difference between predication (incrementing the fill by truth value) and a conditional jump. The theory of out-of-order execution would predict predication to be better, but the difference is lost in measurement noise. This is true on both Intel Nehalem (Xeon 55xx) and Sandy Bridge (E5 26xx), but could be different on other architectures.</p> <p>The multicore scalability of the test will give some information about platform utilization.</p> <p>This is the ultimately simplified selective hash join:</p> <blockquote> <code><pre> SELECT COUNT (*) FROM lineitem, part WHERE l_partkey = p_partkey AND p_size < 15 ; </pre> </code> </blockquote> <p>This is the second simplest hash join but misses the cache much more; since this now has a key and a dependent part in the hash table, there is an extra pointer to follow, and the hash entry is two words plus the pointer to these in the hash table array.</p> <blockquote> <code><pre> SELECT SUM (p_retailprice) FROM lineitem, part WHERE l_partkey = p_partkey AND p_size < 15 ; </pre> </code> </blockquote> <p>By adjusting the number of parts selected, we can vary the Bloom filter selectivity and the size of the hash table. Below, we show the times for the two queries with single-thread and 24-thread execution, with different percentages of the <code>part</code> table on the build side of the hash join. All runs are against warm 100G TPC-H on the same test system as in the rest of the TPC-H series (dual Xeon E5-2630).</p> <p>This table compares the performance of the linear and cuckoo implementations on the above queries (count vs. sum) on either 24 threads or 1 thread. Four data points are given for different sizes of hash table, given as percentage of the <code>part</code> table (having 400K - 20M entries) in the hash table. The rightmost column, which represents the case where the entire <code>part</code> table is on the build side does not have a Bloom filter; the other cases do. The Bloom bits are 8/4 for linear and 8/2 for cuckoo. The times are all in milliseconds, and the thousands separator is a comma.</p> <table style="padding:10px;border-spacing:10px;"> <tr> <th style="text-align:center">Hash type</th> <th style="text-align:center">Query type</th> <th style="text-align:center">Threads</th> <th style="text-align:center">2% <br /> (ms)</th> <th style="text-align:center">10% <br /> (ms)</th> <th style="text-align:center">30% <br /> (ms)</th> <th style="text-align:center">100% <br /> (ms)</th> </tr> <tr> </tr> <tr> <th style="text-align:center">Linear</th> <th style="text-align:center"><code>COUNT</code></th> <th style="text-align:center">24</th> <td style="text-align:right">1,204</td> <td style="text-align:right">1,683</td> <td style="text-align:right">3,100</td> <td style="text-align:right">6,214</td> </tr> <tr> <th style="text-align:center">Linear</th> <th style="text-align:center"><code>SUM</code></th> <th style="text-align:center">24</th> <td style="text-align:right">1,261</td> <td style="text-align:right">2,447</td> <td style="text-align:right">5,059</td> <td style="text-align:right">13,086</td> </tr> <tr> <th style="text-align:center">Linear</th> <th style="text-align:center"><code>COUNT</code></th> <th style="text-align:center">1</th> <td style="text-align:right">15,286</td> <td style="text-align:right">22,451</td> <td style="text-align:right">38,863</td> <td style="text-align:right">66,722</td> </tr> <tr> <th style="text-align:center">Linear</th> <th style="text-align:center"><code>SUM</code></th> <th style="text-align:center">1</th> <td style="text-align:right">17,575</td> <td style="text-align:right">33,664</td> <td style="text-align:right">81,927</td> <td style="text-align:right">179,013</td> </tr> <tr> <th style="text-align:center">Cuckoo</th> <th style="text-align:center"><code>COUNT</code></th> <th style="text-align:center">24</th> <td style="text-align:right">1,849</td> <td style="text-align:right">2,840</td> <td style="text-align:right">4,105</td> <td style="text-align:right">6,203</td> </tr> <tr> <th style="text-align:center">Cuckoo</th> <th style="text-align:center"><code>SUM</code></th> <th style="text-align:center">24</th> <td style="text-align:right">2,833</td> <td style="text-align:right">4,903</td> <td style="text-align:right">9,446</td> <td style="text-align:right">19,652</td> </tr> <tr> <th style="text-align:center">Cuckoo</th> <th style="text-align:center"><code>COUNT</code></th> <th style="text-align:center">1</th> <td style="text-align:right">25,146</td> <td style="text-align:right">39,064</td> <td style="text-align:right">57,383</td> <td style="text-align:right">85,105</td> </tr> <tr> <th style="text-align:center">Cuckoo</th> <th style="text-align:center"><code>SUM</code></th> <th style="text-align:center">1</th> <td style="text-align:right">33,647</td> <td style="text-align:right">67,089</td> <td style="text-align:right">121,989</td> <td style="text-align:right">240,941</td> </tr> </table> <p>We clearly see cache effects on the two first lines, where the <code>SUM</code> and <code>COUNT</code> run in almost the same time on a small hash table but have a 2x difference on the larger hash table. The instruction path length is not very different for <code>SUM</code> and <code>COUNT</code>, but the memory footprint has a 3x difference. </p> <p>We note that the SMP scalability of linear is slightly better, contrasting the ratio of 24-thread <code>SUM</code> to single-thread <code>SUM</code>. Both numbers are over 12x, indicating net benefit from core multithreading. (The test system has 12 physical cores.) The linear hash systematically outperforms cuckoo, understandably, since it makes a smaller net number of cache misses. The overall effect on the TPC-H score is noticeable, at around 15-20K units of composite score at 100G.</p> <p>In conclusion, the Virtuoso hash join implementation is certainly on the level, with only small gains to be expected from further vectoring and prefetching. These results may be reproduced using the <a href="https://github.com/v7fasttrack" id="link-id0x2aac22b45068">v7fasttrack</a> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSIndex" id="link-id0x2aac22b45028">Virtuoso Open Source</a> releases from <a href="https://github.com/" id="link-id0x2aac22b44fc8">GitHub</a>; <a href="https://github.com/v7fasttrack/virtuoso-opensource/tree/develop/7" id="link-id0x2aac20ead1d8">develop/7</a> for cuckoo and <a href="https://github.com/v7fasttrack/virtuoso-opensource/tree/feature/analytics" id="link-id0x2aac20ead1b8">feature/analytics</a> for linear hash.</p> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac349bc698"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac347508c8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac347b2098"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac34f71318"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac346fa748"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac349de568"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aac34725b28"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aac36adde48"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aac34a7e368"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac348581b8"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac344340c8"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac34ee7058"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac34d388b8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x8a7ec38"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aabe9bdeb28"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aabe9801c28"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aabf6e3f3e8"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aac10002308"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:06:34.680582-04:00
In Hoc Signo Vinces (part 14 of n): Virtuoso TPC-H Implementation Analysis
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-05-01#1801
2014-05-01T17:07:57Z
<p>In <a href="http://www.openlinksw.com/weblogs/oerling/?id=1798" id="link-id0x2aac10209c18">the previous article</a> we saw an unofficial result of running the full workload. Here we will look more closely at the performance profile.</p> <p>In this article we look at what the server actually does. The <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20140501TPCHImplementationAnalysis" id="link-id0x2aac10912a48">execution profiles for all the queries</a> are available for download. To experiment with parallelism, you may <a href="https://github.com/v7fasttrack/" id="link-id0x2aabbfc25fc8">download the software</a> and run it locally. An Amazon image may be provided later.</p> <h2>Execution Profile</h2> <p>Below is the top of the <code>oprofile</code> output for a run of the 22 queries with qualification parameters against the 100G database. The operation in <a href="http://www.tpc.org/tpch/" id="link-id0x2aabbc954738">TPC-H</a> terms is given under each heading.</p> <blockquote> <pre><code>CPU: Intel Sandy Bridge microarchitecture, speed 2299.98 MHz (estimated) Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000 samples % symbol name 1406009 9.5117 ce_vec_int_range_hash</code> </pre><blockquote> <i>-- Bloom filter before selective hash join where selective hash join is best or only condition on a scan, e.g., <code>lineitem</code> scan where <code>l_partkey</code> checked against a Bloom filter </i> </blockquote> <pre><code>730935 4.9448 ce_vec_int_sets_hash</code> </pre><blockquote> <i>-- Bloom filter check where another condition is applied first, e.g., <code>lineitem</code> scan with condition on <code>l_shipdate</code>, then Bloom filter check on <code>l_partkey</code></i> </blockquote> <pre><code>617091 4.1746 hash_source_chash_input_1i_n</code> </pre><blockquote> <i>-- Q13 right outer hash join with probe from <code>orders</code>, build from <code>customer</code>, <code>NOT EXISTS</code> test in Q16 </i> </blockquote> <pre><code>586938 3.9706 cs_decode</code> </pre><blockquote> <i>-- Generic reading of a column, all queries</i> </blockquote> <pre><code>536273 3.6279 ce_intd_range_ltgt</code> </pre><blockquote> <i>-- Date range comparison, most queries, e.g., Q1, 3, 4, 5, 6, 7, 8, 20</i> </blockquote> <pre><code>479898 3.2465 cha_cmp_1i</code> </pre><blockquote> <i>-- Q13 <code>GROUP BY</code> on <code>c_custkey</code>, Q15 <code>GROUP BY</code> on <code>S_suppkey</code>. Indicates missing the cache on high cardinality <code>GROUP BY</code> single <code>INT</code> grouping key</i> </blockquote> <pre><code>473721 3.2047 cha_inline_1i_n_int</code> </pre><blockquote> <i>-- Selective hash join after prefiltering with Bloom filter. Check only that key in hash table, no dependent part. For example Q8, Q9, Q17, Q20, with <code>lineitem</code> filtered by <code>part</code> </i> </blockquote> <pre><code>463723 3.1371 ce_dict_generic_range_filter</code> </pre><blockquote> <i>-- Range condition on low cardinality column (dictionary encoded), e.g., <code>l_quantity</code>, <code>l_discount</code> </i> </blockquote> <pre><code>425149 2.8761 cha_inline_1i_int</code> </pre><blockquote> <i>-- Hash join check that a single <code>INT</code> key with a dependent part is in a hash table, fetch the dependent part for hits</i> </blockquote> <pre><code>365040 2.4695 setp_chash_run</code> </pre><blockquote> <i>-- <code>GROUP BY</code>, e.g., <code>GROUP BY</code> on <code>c_custkey</code> in Q13</i> </blockquote> <pre><code>359645 2.4330 clrg_partition_dc</code> </pre><blockquote> <i>-- Partitioning a vector of values, occurs in all high cardinality <code>GROUP BYs</code>, e.g., Q13, Q15</i> </blockquote> <pre><code>349473 2.3642 gb_aggregate</code> </pre><blockquote> <i>-- Updating aggregates after the grouping keys are resolved, e.g., Q1</i> </blockquote> <pre><code>331926 2.2455 ce_dict_any_sets_decode</code> </pre><blockquote> <i>-- Fetching non-contiguous dictionary encoded strings, e.g., <code>l_returnflag</code>, <code>l_linestatus</code> in Q1 </i> </blockquote> <pre><code>316731 2.1427 cha_insert_1i</code> </pre><blockquote> <i>-- Building a hash join hash table from a 1 <code>INT</code> key to a dependent part, e.g., Q14 from <code>l_partkey</code> to <code>lineitems</code> in a time window with a given <code>l_partkey</code></i> </blockquote> <pre><code>286865 1.9406 ce_search_rld</code> </pre><blockquote> <i>-- Index lookup for run length delta compressed keys, e.g., <code>l_orderkey</code>, <code>ps_partkey</code> </i> </blockquote> <pre><code>231390 1.5654 cha_insert</code> </pre><blockquote> <i>-- Build of a hash join hash table for multipart keys, e.g., hash join with <code>partsupp</code> (Q9) or <code>lineitem</code> (Q17, Q20) on build side </i> </blockquote> <pre><code>224070 1.5158 ce_dict_int64_sets_decode</code> </pre><blockquote> <i>-- Fetching a non-contiguous set of double column values from dictionary encoding, e.g., <code>l_discount</code>, <code>l_quantity</code> </i> </blockquote> <pre><code>218506 1.4782 ce_intd_any_sets_decode</code> </pre><blockquote> <i>-- Fetching a non-contiguous set of date column values, e.g., <code>l_shipdate</code> in Q9</i> </blockquote> <pre><code>200686 1.3576 page_wait_access</code> </pre><blockquote> <i>-- Translating page numbers into buffers in buffer pool for column access</i> </blockquote> <pre><code>198854 1.3452 itc_col_seg</code> </pre><blockquote> <i>-- Generic part of table scan/index access</i> </blockquote> <pre><code>197645 1.3371 cha_insert_1i_n</code> </pre><blockquote> <i>-- Hash join build for hash tables with a 1 <code>INT</code> key and no dependent, e.g., <code>lineitem</code> to <code>part</code> join in Q9, Q17, Q20</i> </blockquote> <pre><code>195300 1.3212 cha_bloom_unroll_a</code> </pre><blockquote> <i>-- Bloom filter for hash based in predicate </i> </blockquote> <pre><code>192309 1.3010 hash_source_chash_input</code> </pre><blockquote> <i>-- Hash join probe with multi-part key, e.g., Q9 against <code>partsupp</code></i> </blockquote> <pre><code>191313 1.2942 strstr_sse42</code> </pre><blockquote> <i>-- SSE 4.2 substring match, e.g. <code>NOT LIKE</code> condition in Q13</i> </blockquote> <pre><code>186325 1.2605 itc_fetch_col_vec</code> </pre><blockquote> <i>-- Translating page numbers into buffers in buffer pool for column access</i> </blockquote> <pre><code>159198 1.0770 ce_vec_int64_sets_decode</code> </pre> <blockquote> <i>-- Fetching non-contiguous 64-bit values from array represented column, e.g., <code>l_extendedprice</code> </i> </blockquote> </blockquote> <p>So what does TPC-H do? It is all selective hash joins. Then it is table scans with a condition on a <code>DATE</code> column. The <code>DATE</code> column part is because this implementation does not order the big tables (<code>lineitem</code>, <code>orders</code>) on their <code>DATE</code> columns. Third, TPC-H does big <code>GROUP BYs</code>, with Q13 representing most of this; all other <code>GROUP BYs</code> have at least 10x fewer groups. Then it just extracts column values, most often after selecting the rows on some condition; quite often the condition is a foreign key column of the row finding a hit in a hash table. This last is called <i>invisible hash join.</i> </p> <p>Then there are some index lookups, but there the join is usually a merge pattern, like reading <code>orders</code> in order of <code>o_orderkey</code> and then getting <code>l_orderkey</code> matches from <code>lineitem</code> (e.g., Q21). Then there are quite often partitioning operations, i.e., many threads produce tuples and these tuples go to a consumer thread selected based on a partitioning column on the tuple. This is also called exchange. This is in all the high cardinality <code>GROUP BYs</code> and in the <code>RIGHT OUTER JOIN</code> of Q13.</p> <p>Whether the implementation is RAM-only or paging from disk makes next to no difference. <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aabbef46ec8">Virtuoso</a> and <a href="http://dbpedia.org/resource/Vectorwise" id="link-id0x2aac103dc588">Actian Vector</a> both have a buffer pool, with Virtuoso using substantially smaller pages (8K vs. 256K or 512K). The page-number-to-buffer translation and the latching that goes with it is under 3% (<code>itc_fetch_col_vec</code>, <code>page_wait_access</code>). Of course, actually accessing secondary storage will kill the score, but checking that something is in memory is safe as long as this is always a hit.</p> <p>So, once the query plans are right, the problem resolves into a few different loops. The bulk of the effort in making a TPC-H implementation is in query optimization so that the right loops run in the right order. I will further explain what the loops should contain in the next article.</p> <p>Many of the functions seen in the profile are instantiations of a template for a specific data type. There are also different templates for variants of a data structure, like a hash table with different keys.</p> <p>Compilation is a way of generating exactly the right loop for any set of data types. We do not find much need for this here, though. The type-specific operations that are in fact needed are anticipatable and can be predefined based on templates.</p> <h2>Future Gains </h2> <p>There are possible gains in the following domains:</p> <ul> <li> <p>Query optimization - Q15, Q17 do some work in duplicate, so reuse will roughly cut the time in half. In addition to reuse, there is a possibility to convert a scalar subquery into a derived table with a <code>GROUP BY</code> (decorrelation). Decorrelation also applies in Q20. Reuse will give some 10-12K more score; decorrelation is probably below measurement noise.</p> </li> <li> <p>Group join - Merging the <code>GROUP BY</code> and the hash probe in Q13 will save 2% of time on throughput, maybe 5K more score.</p> </li> <li> <p>Better parallelism in power test - A good power test takes 42s, and 5 times the work takes 165s, so the platform utilization of the power test is roughly 4/5. This could be slightly better. </p> </li> <li> <p> <a href="http://dbpedia.org/resource/Non-uniform_memory_access" id="link-id0x2aabbf0dc408">NUMA</a> - Up to 20% performance gains have been seen on scale-out configurations from binding a process to one CPU socket. This gain will be readily seen in a scale-out setting when running one process per physical CPU. Gains in the NUMA department are rather fickle and fragile and are worth less in real life than in benchmarks. So I would say that work in single-server NUMA optimization is not immediately needed since scale-out configurations will get these gains as soon as one sets affinities for processes. But whether binding processes to CPUs makes sense depends on how even the workload is. TPC-H is even, but reality is less so.</p> </li> </ul> <p>In principle, a composite score of a good run could go from 250K to 280K by diverse incremental improvements. There is some noise in run times, so two consecutive runs can deviate by a few percent, which is already significant when talking of small improvements. </p> <h2>Larger Scales</h2> <p>For a 100 GB run with 5 throughput streams, the peak query memory consumption is 3.5 GB. This includes all the <code>GROUP BY</code> and hash join tables that are made. Q13 is the most expensive in terms of space and allocates a peak of 1 GB for a single execution. This is trivial in comparison with the working set. But at 100x greater scale (10,000 GB or 10 TB) this becomes about 630 GB, as there will be a minimum of 9 concurrent streams instead of 5.</p> <p>Now 10 TB is clearly a scale-out size, so the 630 GB transient memory is not on a single machine. </p> <p>Still going to large scale-out will change the nature of some queries and introduce significant data transfer. We will know soon enough. </p> <h2>Problems of the Metric </h2> <p>A small-scale TPC-H, e.g., 100 GB, starts to have features of a lookup workload. This means that there is high variability between consecutive executions, and that the pre-execution state of the system does have large effect on a measurement.</p> <p>The rules say that power test follows bulk load with maybe some checking of correctness of load in between. The bulk load is basically unregulated and usually will include statistics gathering.</p> <p>The first power test shows significant variation, anything from 220 K to 240 K, while the second power test is steadily around 255 K. Since the reported score is the lower of the two, the biggest return to the implementor is in making sure the first power test is good.</p> <p>The second throughput test is usually 5-10 K higher than the first; the throughput test is less sensitive. The difference does not come from I/O, but from system time for memory allocation. The memory and the same quantities and block sizes is reused by the second run.</p> <p>A good power run is 42s from 100% warm RAM. A power run that gets the data from the OS disk buffers is 70s or so. A power run that gets data from SSD is worse, maybe 120s. </p> <p>To cite an example, increasing the buffer pool size from 64 GB to 72 GB gets the first post-load power test from 120-150K to 230-240K while having no appreciable effect on subsequent tests. The effect is exacerbated by the fact that the power score is based on a geometric mean of run times. Very short queries (e.g., Q2) vary between consecutive in-memory executions from 120ms to 220ms. A similar variation occurs in Q13 which is on either side of 6s. Due to geometric mean, the same variability has very different impact depending on which query it hits. A Q2 that reads data from out of process can take 2s instead of the expected under 200ms. This kills the score even if a delay of 1.8s as such did not. So increasing the buffer pool in the example just serves to make sure the small <code>supplier</code> table is in memory. Fetching it from the OS is simply not an option in the first Q2 even if it were an option in a longer running query. Remember, the the lower of the two scores is reported, and the first power test will be bad unless it is somehow primed by some trick like bulk load order.</p> <p>When differences between implementations are small, variation between consecutive runs becomes important. This is why OLTP benchmarks are required to run for a relatively long time and only measure the steady state portion. This would also be appropriate for small TPC-H runs.</p> <h2>Conclusions</h2> <p>Query performance reduces to a few loops when all the conditions are right. Getting the conditions to be right depends on query optimization and the right architecture choices. These results would be unthinkable without vectored execution and a compressed column store design. These in and of themselves guarantee nothing unless all plans are right. A previous Virtuoso 7 takes over 10x longer because of bad plans and missing some key execution techniques like the <code>RIGHT OUTER JOIN</code> in Q13 and partitioned <code>GROUP BY</code>.</p> <p>Last summer, we did runs of the much simpler Star Schema Benchmark. The <a href="http://www.openlinksw.com/weblog/oerling/?id=1733" id="link-id0x2aabbceaa7b8">results were very good</a>, because the engine had the much smaller set of tricks and the right plans. Repeating these tests now would show some gains from a still better hash join but nothing dramatic.</p> <p>In the next article we will look at the finer points of hash join. After this we move to larger scales and clusters.</p> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac349bc698"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac347508c8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac347b2098"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac34f71318"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac346fa748"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac349de568"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aac34725b28"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aac36adde48"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aac34a7e368"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac348581b8"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac344340c8"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac34ee7058"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac34d388b8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x1a521418"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aabded72328"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x14a32e98"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aac3589ec88"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aab521c9cb8"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:09:47.821517-04:00
In Hoc Signo Vinces (part 14 of n): Virtuoso TPC-H Implementation Analysis
http://www.openlinksw.com:443/weblog/oerling/?date=2014-05-01#1800
2014-05-01T17:03:38Z
<p>In <a href="http://www.openlinksw.com/weblogs/oerling/?id=1798" id="link-id0x2aac10209c18">the previous article</a> we saw an unofficial result of running the full workload. Here we will look more closely at the performance profile.</p> <p>In this article we look at what the server actually does. The <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20140501TPCHImplementationAnalysis" id="link-id0x2aac10912a48">execution profiles for all the queries</a> are available for download. To experiment with parallelism, you may <a href="https://github.com/v7fasttrack/" id="link-id0x2aabbfc25fc8">download the software</a> and run it locally. An Amazon image may be provided later.</p> <h2>Execution Profile</h2> <p>Below is the top of the <code>oprofile</code> output for a run of the 22 queries with qualification parameters against the 100G database. The operation in <a href="http://www.tpc.org/tpch/" id="link-id0x2aabbc954738">TPC-H</a> terms is given under each heading.</p> <blockquote> <pre><code>CPU: Intel Sandy Bridge microarchitecture, speed 2299.98 MHz (estimated) Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000 samples % symbol name 1406009 9.5117 ce_vec_int_range_hash</code> </pre><blockquote> <i>-- Bloom filter before selective hash join where selective hash join is best or only condition on a scan, e.g., <code>lineitem</code> scan where <code>l_partkey</code> checked against a Bloom filter </i> </blockquote> <pre><code>730935 4.9448 ce_vec_int_sets_hash</code> </pre><blockquote> <i>-- Bloom filter check where another condition is applied first, e.g., <code>lineitem</code> scan with condition on <code>l_shipdate</code>, then Bloom filter check on <code>l_partkey</code></i> </blockquote> <pre><code>617091 4.1746 hash_source_chash_input_1i_n</code> </pre><blockquote> <i>-- Q13 right outer hash join with probe from <code>orders</code>, build from <code>customer</code>, <code>NOT EXISTS</code> test in Q16 </i> </blockquote> <pre><code>586938 3.9706 cs_decode</code> </pre><blockquote> <i>-- Generic reading of a column, all queries</i> </blockquote> <pre><code>536273 3.6279 ce_intd_range_ltgt</code> </pre><blockquote> <i>-- Date range comparison, most queries, e.g., Q1, 3, 4, 5, 6, 7, 8, 20</i> </blockquote> <pre><code>479898 3.2465 cha_cmp_1i</code> </pre><blockquote> <i>-- Q13 <code>GROUP BY</code> on <code>c_custkey</code>, Q15 <code>GROUP BY</code> on <code>S_suppkey</code>. Indicates missing the cache on high cardinality <code>GROUP BY</code> single <code>INT</code> grouping key</i> </blockquote> <pre><code>473721 3.2047 cha_inline_1i_n_int</code> </pre><blockquote> <i>-- Selective hash join after prefiltering with Bloom filter. Check only that key in hash table, no dependent part. For example Q8, Q9, Q17, Q20, with <code>lineitem</code> filtered by <code>part</code> </i> </blockquote> <pre><code>463723 3.1371 ce_dict_generic_range_filter</code> </pre><blockquote> <i>-- Range condition on low cardinality column (dictionary encoded), e.g., <code>l_quantity</code>, <code>l_discount</code> </i> </blockquote> <pre><code>425149 2.8761 cha_inline_1i_int</code> </pre><blockquote> <i>-- Hash join check that a single <code>INT</code> key with a dependent part is in a hash table, fetch the dependent part for hits</i> </blockquote> <pre><code>365040 2.4695 setp_chash_run</code> </pre><blockquote> <i>-- <code>GROUP BY</code>, e.g., <code>GROUP BY</code> on <code>c_custkey</code> in Q13</i> </blockquote> <pre><code>359645 2.4330 clrg_partition_dc</code> </pre><blockquote> <i>-- Partitioning a vector of values, occurs in all high cardinality <code>GROUP BYs</code>, e.g., Q13, Q15</i> </blockquote> <pre><code>349473 2.3642 gb_aggregate</code> </pre><blockquote> <i>-- Updating aggregates after the grouping keys are resolved, e.g., Q1</i> </blockquote> <pre><code>331926 2.2455 ce_dict_any_sets_decode</code> </pre><blockquote> <i>-- Fetching non-contiguous dictionary encoded strings, e.g., <code>l_returnflag</code>, <code>l_linestatus</code> in Q1 </i> </blockquote> <pre><code>316731 2.1427 cha_insert_1i</code> </pre><blockquote> <i>-- Building a hash join hash table from a 1 <code>INT</code> key to a dependent part, e.g., Q14 from <code>l_partkey</code> to <code>lineitems</code> in a time window with a given <code>l_partkey</code></i> </blockquote> <pre><code>286865 1.9406 ce_search_rld</code> </pre><blockquote> <i>-- Index lookup for run length delta compressed keys, e.g., <code>l_orderkey</code>, <code>ps_partkey</code> </i> </blockquote> <pre><code>231390 1.5654 cha_insert</code> </pre><blockquote> <i>-- Build of a hash join hash table for multipart keys, e.g., hash join with <code>partsupp</code> (Q9) or <code>lineitem</code> (Q17, Q20) on build side </i> </blockquote> <pre><code>224070 1.5158 ce_dict_int64_sets_decode</code> </pre><blockquote> <i>-- Fetching a non-contiguous set of double column values from dictionary encoding, e.g., <code>l_discount</code>, <code>l_quantity</code> </i> </blockquote> <pre><code>218506 1.4782 ce_intd_any_sets_decode</code> </pre><blockquote> <i>-- Fetching a non-contiguous set of date column values, e.g., <code>l_shipdate</code> in Q9</i> </blockquote> <pre><code>200686 1.3576 page_wait_access</code> </pre><blockquote> <i>-- Translating page numbers into buffers in buffer pool for column access</i> </blockquote> <pre><code>198854 1.3452 itc_col_seg</code> </pre><blockquote> <i>-- Generic part of table scan/index access</i> </blockquote> <pre><code>197645 1.3371 cha_insert_1i_n</code> </pre><blockquote> <i>-- Hash join build for hash tables with a 1 <code>INT</code> key and no dependent, e.g., <code>lineitem</code> to <code>part</code> join in Q9, Q17, Q20</i> </blockquote> <pre><code>195300 1.3212 cha_bloom_unroll_a</code> </pre><blockquote> <i>-- Bloom filter for hash based in predicate </i> </blockquote> <pre><code>192309 1.3010 hash_source_chash_input</code> </pre><blockquote> <i>-- Hash join probe with multi-part key, e.g., Q9 against <code>partsupp</code></i> </blockquote> <pre><code>191313 1.2942 strstr_sse42</code> </pre><blockquote> <i>-- SSE 4.2 substring match, e.g. <code>NOT LIKE</code> condition in Q13</i> </blockquote> <pre><code>186325 1.2605 itc_fetch_col_vec</code> </pre><blockquote> <i>-- Translating page numbers into buffers in buffer pool for column access</i> </blockquote> <pre><code>159198 1.0770 ce_vec_int64_sets_decode</code> </pre> <blockquote> <i>-- Fetching non-contiguous 64-bit values from array represented column, e.g., <code>l_extendedprice</code> </i> </blockquote> </blockquote> <p>So what does TPC-H do? It is all selective hash joins. Then it is table scans with a condition on a <code>DATE</code> column. The <code>DATE</code> column part is because this implementation does not order the big tables (<code>lineitem</code>, <code>orders</code>) on their <code>DATE</code> columns. Third, TPC-H does big <code>GROUP BYs</code>, with Q13 representing most of this; all other <code>GROUP BYs</code> have at least 10x fewer groups. Then it just extracts column values, most often after selecting the rows on some condition; quite often the condition is a foreign key column of the row finding a hit in a hash table. This last is called <i>invisible hash join.</i> </p> <p>Then there are some index lookups, but there the join is usually a merge pattern, like reading <code>orders</code> in order of <code>o_orderkey</code> and then getting <code>l_orderkey</code> matches from <code>lineitem</code> (e.g., Q21). Then there are quite often partitioning operations, i.e., many threads produce tuples and these tuples go to a consumer thread selected based on a partitioning column on the tuple. This is also called exchange. This is in all the high cardinality <code>GROUP BYs</code> and in the <code>RIGHT OUTER JOIN</code> of Q13.</p> <p>Whether the implementation is RAM-only or paging from disk makes next to no difference. <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aabbef46ec8">Virtuoso</a> and <a href="http://dbpedia.org/resource/Vectorwise" id="link-id0x2aac103dc588">Actian Vector</a> both have a buffer pool, with Virtuoso using substantially smaller pages (8K vs. 256K or 512K). The page-number-to-buffer translation and the latching that goes with it is under 3% (<code>itc_fetch_col_vec</code>, <code>page_wait_access</code>). Of course, actually accessing secondary storage will kill the score, but checking that something is in memory is safe as long as this is always a hit.</p> <p>So, once the query plans are right, the problem resolves into a few different loops. The bulk of the effort in making a TPC-H implementation is in query optimization so that the right loops run in the right order. I will further explain what the loops should contain in the next article.</p> <p>Many of the functions seen in the profile are instantiations of a template for a specific data type. There are also different templates for variants of a data structure, like a hash table with different keys.</p> <p>Compilation is a way of generating exactly the right loop for any set of data types. We do not find much need for this here, though. The type-specific operations that are in fact needed are anticipatable and can be predefined based on templates.</p> <h2>Future Gains </h2> <p>There are possible gains in the following domains:</p> <ul> <li> <p>Query optimization - Q15, Q17 do some work in duplicate, so reuse will roughly cut the time in half. In addition to reuse, there is a possibility to convert a scalar subquery into a derived table with a <code>GROUP BY</code> (decorrelation). Decorrelation also applies in Q20. Reuse will give some 10-12K more score; decorrelation is probably below measurement noise.</p> </li> <li> <p>Group join - Merging the <code>GROUP BY</code> and the hash probe in Q13 will save 2% of time on throughput, maybe 5K more score.</p> </li> <li> <p>Better parallelism in power test - A good power test takes 42s, and 5 times the work takes 165s, so the platform utilization of the power test is roughly 4/5. This could be slightly better. </p> </li> <li> <p> <a href="http://dbpedia.org/resource/Non-uniform_memory_access" id="link-id0x2aabbf0dc408">NUMA</a> - Up to 20% performance gains have been seen on scale-out configurations from binding a process to one CPU socket. This gain will be readily seen in a scale-out setting when running one process per physical CPU. Gains in the NUMA department are rather fickle and fragile and are worth less in real life than in benchmarks. So I would say that work in single-server NUMA optimization is not immediately needed since scale-out configurations will get these gains as soon as one sets affinities for processes. But whether binding processes to CPUs makes sense depends on how even the workload is. TPC-H is even, but reality is less so.</p> </li> </ul> <p>In principle, a composite score of a good run could go from 250K to 280K by diverse incremental improvements. There is some noise in run times, so two consecutive runs can deviate by a few percent, which is already significant when talking of small improvements. </p> <h2>Larger Scales</h2> <p>For a 100 GB run with 5 throughput streams, the peak query memory consumption is 3.5 GB. This includes all the <code>GROUP BY</code> and hash join tables that are made. Q13 is the most expensive in terms of space and allocates a peak of 1 GB for a single execution. This is trivial in comparison with the working set. But at 100x greater scale (10,000 GB or 10 TB) this becomes about 630 GB, as there will be a minimum of 9 concurrent streams instead of 5.</p> <p>Now 10 TB is clearly a scale-out size, so the 630 GB transient memory is not on a single machine. </p> <p>Still going to large scale-out will change the nature of some queries and introduce significant data transfer. We will know soon enough. </p> <h2>Problems of the Metric </h2> <p>A small-scale TPC-H, e.g., 100 GB, starts to have features of a lookup workload. This means that there is high variability between consecutive executions, and that the pre-execution state of the system does have large effect on a measurement.</p> <p>The rules say that power test follows bulk load with maybe some checking of correctness of load in between. The bulk load is basically unregulated and usually will include statistics gathering.</p> <p>The first power test shows significant variation, anything from 220 K to 240 K, while the second power test is steadily around 255 K. Since the reported score is the lower of the two, the biggest return to the implementor is in making sure the first power test is good.</p> <p>The second throughput test is usually 5-10 K higher than the first; the throughput test is less sensitive. The difference does not come from I/O, but from system time for memory allocation. The memory and the same quantities and block sizes is reused by the second run.</p> <p>A good power run is 42s from 100% warm RAM. A power run that gets the data from the OS disk buffers is 70s or so. A power run that gets data from SSD is worse, maybe 120s. </p> <p>To cite an example, increasing the buffer pool size from 64 GB to 72 GB gets the first post-load power test from 120-150K to 230-240K while having no appreciable effect on subsequent tests. The effect is exacerbated by the fact that the power score is based on a geometric mean of run times. Very short queries (e.g., Q2) vary between consecutive in-memory executions from 120ms to 220ms. A similar variation occurs in Q13 which is on either side of 6s. Due to geometric mean, the same variability has very different impact depending on which query it hits. A Q2 that reads data from out of process can take 2s instead of the expected under 200ms. This kills the score even if a delay of 1.8s as such did not. So increasing the buffer pool in the example just serves to make sure the small <code>supplier</code> table is in memory. Fetching it from the OS is simply not an option in the first Q2 even if it were an option in a longer running query. Remember, the the lower of the two scores is reported, and the first power test will be bad unless it is somehow primed by some trick like bulk load order.</p> <p>When differences between implementations are small, variation between consecutive runs becomes important. This is why OLTP benchmarks are required to run for a relatively long time and only measure the steady state portion. This would also be appropriate for small TPC-H runs.</p> <h2>Conclusions</h2> <p>Query performance reduces to a few loops when all the conditions are right. Getting the conditions to be right depends on query optimization and the right architecture choices. These results would be unthinkable without vectored execution and a compressed column store design. These in and of themselves guarantee nothing unless all plans are right. A previous Virtuoso 7 takes over 10x longer because of bad plans and missing some key execution techniques like the <code>RIGHT OUTER JOIN</code> in Q13 and partitioned <code>GROUP BY</code>.</p> <p>Last summer, we did runs of the much simpler Star Schema Benchmark. The <a href="http://www.openlinksw.com/weblog/oerling/?id=1733" id="link-id0x2aabbceaa7b8">results were very good</a>, because the engine had the much smaller set of tricks and the right plans. Repeating these tests now would show some gains from a still better hash join but nothing dramatic.</p> <p>In the next article we will look at the finer points of hash join. After this we move to larger scales and clusters.</p> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac349bc698"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac347508c8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac347b2098"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac34f71318"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac346fa748"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac349de568"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aac34725b28"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aac36adde48"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aac34a7e368"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac348581b8"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac344340c8"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac34ee7058"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac34d388b8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0xfa3c378"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aabeb9f3b48"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aac00a05758"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aab79ee5de8"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aabd2285048"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:06:41.706498-04:00
In Hoc Signo Vinces (part 13 of n): Virtuoso TPC-H Kit Now on V7 Fast Track
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-04-28#1799
2014-04-28T16:10:05Z
<p>This article shows how you can reproduce the <a href="http://www.tpc.org/tpch/" id="link-id0x2aac307f0348">TPC-H</a> experiments from the <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac30c9eed8">previous posts</a> in this series.</p> <p>All the code is in the <a href="https://github.com/v7fasttrack/virtuoso-opensource/tree/feature/analytics" id="link-id0x2aac5ab2e128">feature/analytics branch</a> of the <a href="https://github.com/v7fasttrack" id="link-id0x2aac79d948a8">v7fasttrack git repository</a> on <a href="https://github.com/" id="link-id0x2aac5ae175e8">GitHub</a>.</p> <h2>Prerequisites</h2> <p>Start by checking out and compiling Virtuoso Open Source (VOS).</p> <blockquote> <code><pre> git clone https://github.com/v7fasttrack/virtuoso-opensource cd virtuoso-opensource git checkout feature/analytics ./autogen.sh export CFLAGS="-msse4.2 -DSSE42" ./configure make -j 24 make install </pre> </code> </blockquote> <p>The system should be an x86_64, <a href="http://dbpedia.org/resource/Core_i7" id="link-id0x2aac6ccb0048">Intel Core i7</a> or later, with <a href="http://dbpedia.org/resource/SSE4" id="link-id0x2aac6d3ec688">SSE 4.2</a> support. (Running without SSE 4.2 is possible, but for better score you need to define it before doing configure.) The gcc may be any that supports SSE 4.2.</p> <p>To have a good result, the system should have at least 96 GB of RAM and SSD.</p> <p>To get a good load time, both the database files and the CSV files made by <code>dbgen</code> should be on SSD.</p> <h2>Running TPC-H</h2> <h3>Set up</h3> <p>Copy the <code>binsrc/tests/tpc-h</code> directory from the check-out to an SSD, if it is not already on one. Set <code>$HOME</code> to point to the root directory of the check-out. Rename the <code>virtuoso.ini-100G</code> in the <code>tpc-h</code> directory to <code>virtuoso.ini</code>. Edit the database file paths in the <code>virtuoso.ini</code>. Where it says --</p> <blockquote> <code><pre> Segment1 = 1024, /1s1/dbs/tpch100cp-1.db = q1, /1s2/dbs/tpch100cp-2.db = q2 </pre> </code> </blockquote> <p>-- change the file names, and add or remove files as appropriate, each file with a different <code>= q<i>n</i></code>, until you have one file per independent device. If this is a RAID, one file per distinct device in the RAID usually brings improvement. Edit the <code>TransactionFile</code> entry, and replace <code>/1s2/dbs/</code> with a suitable path.</p> <p>Edit <code>ThreadsPerQuery</code> to be the number of threads on the machine. For i7, this is double the number of cores; your environment may vary. <code>AsyncQueueMaxThreads</code> should be set to double <code>ThreadsPerQuery</code>.</p> <p>The memory settings (<code>NumberOfBuffers</code> and <code>MaxDirtyBuffers</code>) are OK for 100G scale. For larger scale, make the memory settings correspondingly larger, not to exceed 75% of system memory. Count 8.5 KB per buffer. If you have less memory, you can decrease these. If so, the first power test will be hit the worst, so the scores will not be as good.</p> <p>The default BIOS settings are usually OK. Disabling prefetch of adjacent cache line does not help, and turning off core threads does not help either.</p> <p>For 100G, the data files for loading will take 100 GB, and the database files will take 88 GB divided among however many files. Be sure there is enough space before you start.</p> <h3>Generate the data</h3> <p>In the <code>tpc-h</code> directory (copy of <code>binsrc/tests/tpc-h</code>) --</p> <blockquote> <code><pre> ./gen.sh 100 5 2 </pre> </code> </blockquote> <p>The first parameter is the scale factor; the second is the number of streams to use in the throughput test; the last is the number of consecutive test runs. The minimum number of streams is 5 for 100G; each successive scale adds one more. A larger number of streams is allowed, but will not make a better result in this case. A test always consists of 2 runs; you could specify more, but the extra tests will not influence the score.</p> <p>Making the data files takes the longest time. You may run <code>dbgen</code> multithreaded to make the dataset in parallel, but then the load scripts will have to be changed to match.</p> <h3>Run the load</h3> <p>Start Virtuoso.</p> <blockquote> <code><pre> ./load.sh 100 </pre> </code> </blockquote> <p>Looking at <code>iostat</code>, you will see a read rate of about 140 MB/s from the source files.</p> <h3>Run the test</h3> <blockquote> <code><pre> ./run.sh 100 5 2 </pre> </code> </blockquote> <p>The parameters have the same meaning as in <code>gen.sh</code>, and the same values must be specified.</p> <h2>Outputs</h2> <p>The run produces two main files, <code>report1.txt</code> and <code>report2.txt</code>. These are the numerical quantity summaries for the first and second run. Additionally, there are output files for each query stream. The <code>suppfiles.sh</code> script can be used to collect the supporting files archive for a TPC-H full disclosure report.</p> <p>The database is left running. To reuse the loaded data for another experiment, kill the Virtuoso process, delete the transaction log, and restart. This will have the data in the post-load state. To get warm cache, use the <code>warm.sql</code> script in the <code>tpc-h</code> directory.</p> <p>On the test system used in this series, 12 core E5 at 2.3GHz, we expect 240K for the first run and 250K for the second. With a top-of-the-line E5, we expect around 400K. For an 8-core 2.26GHz Nehalem, we expect 150K.</p> <p>If you get scores that are significantly different, something is broken; <a href="https://lists.sourceforge.net/lists/listinfo/virtuoso-devel" id="link-id0x2aac309efa88">we would like to know about this</a>.</p> <p>If you have this audited according to the TPC rules, you will be allowed to call this a TPC-H result. Without such audit, the result should be labeled Virt-H.</p> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac6eb05828"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac6d1e0968"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac6e5afef8"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac3d8ccbc8"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac79669d78"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac79c4aee8"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aac7914f948"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aac30e1a9c8"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aac79f4af68"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac78c36c48"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aab7a1ef0f8"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac6ff591d8"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac34cfd578"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x19166398"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aabcb5a6c08"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x187e86a8"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aabf7c73ca8"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aab64084868"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:09:52.503337-04:00
In Hoc Signo Vinces (part 13 of n): Virtuoso TPC-H Kit Now on V7 Fast Track
http://www.openlinksw.com:443/weblog/oerling/?date=2014-04-28#1798
2014-04-28T16:09:56Z
<p>This article shows how you can reproduce the <a href="http://www.tpc.org/tpch/" id="link-id0x2aac307f0348">TPC-H</a> experiments from the <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac30c9eed8">previous posts</a> in this series.</p> <p>All the code is in the <a href="https://github.com/v7fasttrack/virtuoso-opensource/tree/feature/analytics" id="link-id0x2aac5ab2e128">feature/analytics branch</a> of the <a href="https://github.com/v7fasttrack" id="link-id0x2aac79d948a8">v7fasttrack git repository</a> on <a href="https://github.com/" id="link-id0x2aac5ae175e8">GitHub</a>.</p> <h2>Prerequisites</h2> <p>Start by checking out and compiling Virtuoso Open Source (VOS).</p> <blockquote> <code><pre> git clone https://github.com/v7fasttrack/virtuoso-opensource cd virtuoso-opensource git checkout feature/analytics ./autogen.sh export CFLAGS="-msse4.2 -DSSE42" ./configure make -j 24 make install </pre> </code> </blockquote> <p>The system should be an x86_64, <a href="http://dbpedia.org/resource/Core_i7" id="link-id0x2aac6ccb0048">Intel Core i7</a> or later, with <a href="http://dbpedia.org/resource/SSE4" id="link-id0x2aac790d77f8">SSE 4.2</a> support. (Running without SSE 4.2 is possible, but for better score you need to define it before doing configure.) The gcc may be any that supports SSE 4.2.</p> <p>To have a good result, the system should have at least 96 GB of RAM and SSD.</p> <p>To get a good load time, both the database files and the CSV files made by <code>dbgen</code> should be on SSD.</p> <h2>Running TPC-H</h2> <h3>Set up</h3> <p>Copy the <code>binsrc/tests/tpc-h</code> directory from the check-out to an SSD, if it is not already on one. Set <code>$HOME</code> to point to the root directory of the check-out. Rename the <code>virtuoso.ini-100G</code> in the <code>tpc-h</code> directory to <code>virtuoso.ini</code>. Edit the database file paths in the <code>virtuoso.ini</code>. Where it says --</p> <blockquote> <code><pre> Segment1 = 1024, /1s1/dbs/tpch100cp-1.db = q1, /1s2/dbs/tpch100cp-2.db = q2 </pre> </code> </blockquote> <p>-- change the file names, and add or remove files as appropriate, each file with a different <code>= q<i>n</i></code>, until you have one file per independent device. If this is a RAID, one file per distinct device in the RAID usually brings improvement. Edit the <code>TransactionFile</code> entry, and replace <code>/1s2/dbs/</code> with a suitable path.</p> <p>Edit <code>ThreadsPerQuery</code> to be the number of threads on the machine. For i7, this is double the number of cores; your environment may vary. <code>AsyncQueueMaxThreads</code> should be set to double <code>ThreadsPerQuery</code>.</p> <p>The memory settings (<code>NumberOfBuffers</code> and <code>MaxDirtyBuffers</code>) are OK for 100G scale. For larger scale, make the memory settings correspondingly larger, not to exceed 75% of system memory. Count 8.5 KB per buffer. If you have less memory, you can decrease these. If so, the first power test will be hit the worst, so the scores will not be as good.</p> <p>The default BIOS settings are usually OK. Disabling prefetch of adjacent cache line does not help, and turning off core threads does not help either.</p> <p>For 100G, the data files for loading will take 100 GB, and the database files will take 88 GB divided among however many files. Be sure there is enough space before you start.</p> <h3>Generate the data</h3> <p>In the <code>tpc-h</code> directory (copy of <code>binsrc/tests/tpc-h</code>) --</p> <blockquote> <code><pre> ./gen.sh 100 5 2 </pre> </code> </blockquote> <p>The first parameter is the scale factor; the second is the number of streams to use in the throughput test; the last is the number of consecutive test runs. The minimum number of streams is 5 for 100G; each successive scale adds one more. A larger number of streams is allowed, but will not make a better result in this case. A test always consists of 2 runs; you could specify more, but the extra tests will not influence the score.</p> <p>Making the data files takes the longest time. You may run <code>dbgen</code> multithreaded to make the dataset in parallel, but then the load scripts will have to be changed to match.</p> <h3>Run the load</h3> <p>Start Virtuoso.</p> <blockquote> <code><pre> ./load.sh 100 </pre> </code> </blockquote> <p>Looking at <code>iostat</code>, you will see a read rate of about 140 MB/s from the source files.</p> <h3>Run the test</h3> <blockquote> <code><pre> ./run.sh 100 5 2 </pre> </code> </blockquote> <p>The parameters have the same meaning as in <code>gen.sh</code>, and the same values must be specified.</p> <h2>Outputs</h2> <p>The run produces two main files, <code>report1.txt</code> and <code>report2.txt</code>. These are the numerical quantity summaries for the first and second run. Additionally, there are output files for each query stream. The <code>suppfiles.sh</code> script can be used to collect the supporting files archive for a TPC-H full disclosure report.</p> <p>The database is left running. To reuse the loaded data for another experiment, kill the Virtuoso process, delete the transaction log, and restart. This will have the data in the post-load state. To get warm cache, use the <code>warm.sql</code> script in the <code>tpc-h</code> directory.</p> <p>On the test system used in this series, 12 core E5 at 2.3GHz, we expect 240K for the first run and 250K for the second. With a top-of-the-line E5, we expect around 400K. For an 8-core 2.26GHz Nehalem, we expect 150K.</p> <p>If you get scores that are significantly different, something is broken; <a href="https://lists.sourceforge.net/lists/listinfo/virtuoso-devel" id="link-id0x2aac309efa88">we would like to know about this</a>.</p> <p>If you have this audited according to the TPC rules, you will be allowed to call this a TPC-H result. Without such audit, the result should be labeled Virt-H.</p> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac6eb05828"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac6d1e0968"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac6e5afef8"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac3d8ccbc8"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac79669d78"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac79c4aee8"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aac7914f948"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aac30e1a9c8"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aac79f4af68"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac78c36c48"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aab7a1ef0f8"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac6ff591d8"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x19d784a8"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aabea8bd858"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x14a92388"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aabd6a12f48"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aab51599bb8"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:06:49.199708-04:00
In Hoc Signo Vinces (part 12 of n): TPC-H: Result Preview
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-04-22#1797
2014-04-22T15:33:27Z
<p>In this article, we look at the 100 GB single-server results for the whole workload. We will call this <i>Virt-H</i> instead of <i><a href="http://www.tpc.org/tpch/" id="link-id0x2aabdeab62f8">TPC-H</a></i> in order to comply with the TPC rules: Use of the TPC-H label requires an audit.</p> <p>The test consists of a bulk load followed by two runs. Each run consists of a single user power test and a multi-user throughput test. The number of users in the throughput test is up to the test sponsor but must be at least 5 for the 100 GB scale. The reported score is the lower of the two scores.</p> <h2>Result Summary </h2> <table style="padding:10px;border-spacing:10px;"> <tr> <th style="text-align:right">Scale Factor</th> <td style="text-align:center">100 GB</td> </tr> <tr> <th style="text-align:right">dbgen</th> <td style="text-align:center">version 2.15</td> </tr> <tr> <th style="text-align:right">Lload time</th> <td style="text-align:center">0:15:02</td> </tr> <tr> <th style="text-align:right">Composite qph</th> <td style="text-align:center">241,482.3</td> </tr> <tr> <th style="text-align:right">System Availability Date</th> <td style="text-align:center">2014-04-22</td> </tr> </table> <p>The price/performance is left open. The hardware costs about 5000 euros and the software is open source so the cost per performance would be a minimum of 0.02 euros per qph at 100G. This is not compliant with the TPC pricing rules though. These require 3 year maintenance contracts for all parts.</p> <p>The software configuration did not use RAID. Otherwise the software would be auditable to the best of my knowledge. The hardware would have to be the same from Dell, HP, or other large brand to satisfy the TPC pricing rule.</p> <h2>Executive Summaries of Each Run</h2> <h3>Run 1</h3> <table style="padding:10px;border-spacing:10px;"> <tr> <th style="text-align:right">Report Date</th> <td style="text-align:center">2014-04-21</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td style="text-align:center">100</td> </tr> <tr> <th style="text-align:right">Total Data Storage/Database Size</th> <td style="text-align:center">1 TB / 87,496 MB</td> </tr> <tr> <th style="text-align:right">Start of Database Load</th> <td style="text-align:center">2014-04-21 21:02:43</td> </tr> <tr> <th style="text-align:right">End of Database Load</th> <td style="text-align:center">2014-04-21 21:17:45</td> </tr> <tr> <th style="text-align:right">Database Load Time</th> <td style="text-align:center">0:15:02</td> </tr> <tr> <th style="text-align:right">Query Streams for Throughput Test</th> <td style="text-align:center">5</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td style="text-align:center">239,785.1</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td style="text-align:center">243,191.4</td> </tr> <tr> <th style="text-align:right">Virt-H Composite Query-per-Hour Metric (Qph@100GB)</th> <td style="text-align:center">241,482.3</td> </tr> <tr> <th style="text-align:right">Measurement Interval in Throughput Test (Ts)</th> <td style="text-align:center">162.935000 seconds</td> </tr> </table> <h4>Duration of stream execution</h4> <table style="padding:10px;border-spacing:10px;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>2014-04-21 21:17:46</td> <td>2014-04-21 21:18:33</td> <td style="text-align:right">0:00:47</td> </tr> <tr> <th>Stream 1</th> <td>2014-04-21 21:18:33</td> <td>2014-04-21 21:21:13</td> <td style="text-align:right">0:02:40</td> </tr> <tr> <th>Stream 2</th> <td>2014-04-21 21:18:33</td> <td>2014-04-21 21:21:13</td> <td style="text-align:right">0:02:40</td> </tr> <tr> <th>Stream 3</th> <td>2014-04-21 21:18:33</td> <td>2014-04-21 21:21:06</td> <td style="text-align:right">0:02:33</td> </tr> <tr> <th>Stream 4</th> <td>2014-04-21 21:18:33</td> <td>2014-04-21 21:21:10</td> <td style="text-align:right">0:02:37</td> </tr> <tr> <th>Stream 5</th> <td>2014-04-21 21:18:33</td> <td>2014-04-21 21:21:16</td> <td style="text-align:right">0:02:43</td> </tr> <tr> <th>Refresh 0</th> <td>2014-04-21 21:17:46</td> <td>2014-04-21 21:17:49</td> <td style="text-align:right">0:00:03</td> </tr> <tr> <th> </th> <td>2014-04-21 21:17:50</td> <td>2014-04-21 21:17:51</td> <td style="text-align:right">0:00:01</td> </tr> <tr> <th>Refresh 1</th> <td>2014-04-21 21:19:25</td> <td>2014-04-21 21:19:38</td> <td style="text-align:right">0:00:13</td> </tr> <tr> <th>Refresh 2</th> <td>2014-04-21 21:18:33</td> <td>2014-04-21 21:18:48</td> <td style="text-align:right">0:00:15</td> </tr> <tr> <th>Refresh 3</th> <td>2014-04-21 21:18:49</td> <td>2014-04-21 21:19:01</td> <td style="text-align:right">0:00:12</td> </tr> <tr> <th>Refresh 4</th> <td>2014-04-21 21:19:01</td> <td>2014-04-21 21:19:13</td> <td style="text-align:right">0:00:12</td> </tr> <tr> <th>Refresh 5</th> <td>2014-04-21 21:19:13</td> <td>2014-04-21 21:19:25</td> <td style="text-align:right">0:00:12</td> </tr> </table> <h4>Numerical Quantities Summary -- Timing Intervals in Seconds</h4> <table style="padding:10px;border-spacing:10px;"> <tr> <th> </th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> </tr> <tr> <th>Stream 0</th> <td style="text-align:right">2.311882</td> <td style="text-align:right">0.383459</td> <td style="text-align:right">1.143286</td> <td style="text-align:right">0.439926</td> <td style="text-align:right">1.594027</td> <td style="text-align:right">0.736482</td> <td style="text-align:right">1.440826</td> <td style="text-align:right">1.198925</td> </tr> <tr> <th>Stream 1</th> <td style="text-align:right">5.192341</td> <td style="text-align:right">0.952574</td> <td style="text-align:right">6.184940</td> <td style="text-align:right">1.194804</td> <td style="text-align:right">6.998207</td> <td style="text-align:right">5.122059</td> <td style="text-align:right">5.962717</td> <td style="text-align:right">6.773401</td> </tr> <tr> <th>Stream 2</th> <td style="text-align:right">7.354001</td> <td style="text-align:right">1.191604</td> <td style="text-align:right">4.238262</td> <td style="text-align:right">1.770639</td> <td style="text-align:right">5.782669</td> <td style="text-align:right">1.357578</td> <td style="text-align:right">4.034697</td> <td style="text-align:right">6.354747</td> </tr> <tr> <th>Stream 3</th> <td style="text-align:right">6.489788</td> <td style="text-align:right">1.585291</td> <td style="text-align:right">4.645022</td> <td style="text-align:right">3.358926</td> <td style="text-align:right">7.904636</td> <td style="text-align:right">3.220767</td> <td style="text-align:right">5.694622</td> <td style="text-align:right">7.431067</td> </tr> <tr> <th>Stream 4</th> <td style="text-align:right">5.609555</td> <td style="text-align:right">1.066582</td> <td style="text-align:right">6.740518</td> <td style="text-align:right">2.503038</td> <td style="text-align:right">9.439980</td> <td style="text-align:right">3.424101</td> <td style="text-align:right">4.404849</td> <td style="text-align:right">4.256317</td> </tr> <tr> <th>Stream 5</th> <td style="text-align:right">10.346825</td> <td style="text-align:right">1.787459</td> <td style="text-align:right">4.391000</td> <td style="text-align:right">3.151059</td> <td style="text-align:right">4.974037</td> <td style="text-align:right">2.932079</td> <td style="text-align:right">6.191782</td> <td style="text-align:right">3.619255</td> </tr> <tr> <th>Min Qi</th> <td style="text-align:right">5.192341</td> <td style="text-align:right">0.952574</td> <td style="text-align:right">4.238262</td> <td style="text-align:right">1.194804</td> <td style="text-align:right">4.974037</td> <td style="text-align:right">1.357578</td> <td style="text-align:right">4.034697</td> <td style="text-align:right">3.619255</td> </tr> <tr> <th>Max Qi</th> <td style="text-align:right">10.346825</td> <td style="text-align:right">1.787459</td> <td style="text-align:right">6.740518</td> <td style="text-align:right">3.358926</td> <td style="text-align:right">9.439980</td> <td style="text-align:right">5.122059</td> <td style="text-align:right">6.191782</td> <td style="text-align:right">7.431067</td> </tr> <tr> <th>Avg Qi</th> <td style="text-align:right">6.998502</td> <td style="text-align:right">1.316702</td> <td style="text-align:right">5.239948</td> <td style="text-align:right">2.395693</td> <td style="text-align:right">7.019906</td> <td style="text-align:right">3.211317</td> <td style="text-align:right">5.257733</td> <td style="text-align:right">5.686957</td> </tr> <tr> <th> </th> </tr> <tr> <th> </th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> </tr> <tr> <th>Stream 0</th> <td style="text-align:right">4.476940</td> <td style="text-align:right">2.004782</td> <td style="text-align:right">2.070967</td> <td style="text-align:right">1.015134</td> <td style="text-align:right">7.995799</td> <td style="text-align:right">2.142581</td> <td style="text-align:right">1.989357</td> <td style="text-align:right">1.581758</td> </tr> <tr> <th>Stream 1</th> <td style="text-align:right">11.351299</td> <td style="text-align:right">6.657059</td> <td style="text-align:right">7.719765</td> <td style="text-align:right">5.157236</td> <td style="text-align:right">25.156379</td> <td style="text-align:right">8.566067</td> <td style="text-align:right">7.028898</td> <td style="text-align:right">8.146883</td> </tr> <tr> <th>Stream 2</th> <td style="text-align:right">13.954105</td> <td style="text-align:right">8.341359</td> <td style="text-align:right">10.265949</td> <td style="text-align:right">3.289724</td> <td style="text-align:right">25.249435</td> <td style="text-align:right">6.370577</td> <td style="text-align:right">11.262650</td> <td style="text-align:right">7.684574</td> </tr> <tr> <th>Stream 3</th> <td style="text-align:right">13.597277</td> <td style="text-align:right">5.783821</td> <td style="text-align:right">5.944240</td> <td style="text-align:right">5.214661</td> <td style="text-align:right">24.253991</td> <td style="text-align:right">8.742896</td> <td style="text-align:right">7.701709</td> <td style="text-align:right">5.801641</td> </tr> <tr> <th>Stream 4</th> <td style="text-align:right">15.612070</td> <td style="text-align:right">6.126494</td> <td style="text-align:right">4.533748</td> <td style="text-align:right">5.733828</td> <td style="text-align:right">23.021583</td> <td style="text-align:right">6.423207</td> <td style="text-align:right">8.358223</td> <td style="text-align:right">6.866477</td> </tr> <tr> <th>Stream 5</th> <td style="text-align:right">8.421209</td> <td style="text-align:right">9.040726</td> <td style="text-align:right">7.799425</td> <td style="text-align:right">3.908758</td> <td style="text-align:right">23.342975</td> <td style="text-align:right">9.934672</td> <td style="text-align:right">11.455598</td> <td style="text-align:right">8.258504</td> </tr> <tr> <th>Min Qi</th> <td style="text-align:right">8.421209</td> <td style="text-align:right">5.783821</td> <td style="text-align:right">4.533748</td> <td style="text-align:right">3.289724</td> <td style="text-align:right">23.021583</td> <td style="text-align:right">6.370577</td> <td style="text-align:right">7.028898</td> <td style="text-align:right">5.801641</td> </tr> <tr> <th>Max Qi</th> <td style="text-align:right">15.612070</td> <td style="text-align:right">9.040726</td> <td style="text-align:right">10.265949</td> <td style="text-align:right">5.733828</td> <td style="text-align:right">25.249435</td> <td style="text-align:right">9.934672</td> <td style="text-align:right">11.455598</td> <td style="text-align:right">8.258504</td> </tr> <tr> <th>Avg Qi</th> <td style="text-align:right">12.587192</td> <td style="text-align:right">7.189892</td> <td style="text-align:right">7.252625</td> <td style="text-align:right">4.660841</td> <td style="text-align:right">24.204873</td> <td style="text-align:right">8.007484</td> <td style="text-align:right">9.161416</td> <td style="text-align:right">7.351616</td> </tr> <tr> <th> </th> </tr> <tr> <th> </th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> </tr> <tr> <th>Stream 0</th> <td style="text-align:right">2.258070</td> <td style="text-align:right">0.981896</td> <td style="text-align:right">1.161602</td> <td style="text-align:right">1.933124</td> <td style="text-align:right">2.203497</td> <td style="text-align:right">1.042949</td> <td style="text-align:right">3.349407</td> <td style="text-align:right">1.296630</td> </tr> <tr> <th>Stream 1</th> <td style="text-align:right">8.213340</td> <td style="text-align:right">4.070175</td> <td style="text-align:right">5.662723</td> <td style="text-align:right">12.260503</td> <td style="text-align:right">7.792825</td> <td style="text-align:right">3.323136</td> <td style="text-align:right">9.296430</td> <td style="text-align:right">3.939927</td> </tr> <tr> <th>Stream 2</th> <td style="text-align:right">16.754827</td> <td style="text-align:right">3.895688</td> <td style="text-align:right">4.413773</td> <td style="text-align:right">7.529466</td> <td style="text-align:right">6.288539</td> <td style="text-align:right">2.717479</td> <td style="text-align:right">11.222082</td> <td style="text-align:right">4.135510</td> </tr> <tr> <th>Stream 3</th> <td style="text-align:right">8.486809</td> <td style="text-align:right">2.615640</td> <td style="text-align:right">7.426936</td> <td style="text-align:right">7.274289</td> <td style="text-align:right">6.706145</td> <td style="text-align:right">3.402654</td> <td style="text-align:right">8.278881</td> <td style="text-align:right">4.260483</td> </tr> <tr> <th>Stream 4</th> <td style="text-align:right">12.604905</td> <td style="text-align:right">7.735042</td> <td style="text-align:right">5.627039</td> <td style="text-align:right">6.343302</td> <td style="text-align:right">7.242370</td> <td style="text-align:right">3.492640</td> <td style="text-align:right">6.503095</td> <td style="text-align:right">3.698821</td> </tr> <tr> <th>Stream 5</th> <td style="text-align:right">8.221733</td> <td style="text-align:right">2.670036</td> <td style="text-align:right">5.866626</td> <td style="text-align:right">13.108081</td> <td style="text-align:right">9.428098</td> <td style="text-align:right">4.282014</td> <td style="text-align:right">8.213320</td> <td style="text-align:right">4.088321</td> </tr> <tr> <th>Min Qi</th> <td style="text-align:right">8.213340</td> <td style="text-align:right">2.615640</td> <td style="text-align:right">4.413773</td> <td style="text-align:right">6.343302</td> <td style="text-align:right">6.288539</td> <td style="text-align:right">2.717479</td> <td style="text-align:right">6.503095</td> <td style="text-align:right">3.698821</td> </tr> <tr> <th>Max Qi</th> <td style="text-align:right">16.754827</td> <td style="text-align:right">7.735042</td> <td style="text-align:right">7.426936</td> <td style="text-align:right">13.108081</td> <td style="text-align:right">9.428098</td> <td style="text-align:right">4.282014</td> <td style="text-align:right">11.222082</td> <td style="text-align:right">4.260483</td> </tr> <tr> <th>Avg Qi</th> <td style="text-align:right">10.856323</td> <td style="text-align:right">4.197316</td> <td style="text-align:right">5.799419</td> <td style="text-align:right">9.303128</td> <td style="text-align:right">7.491595</td> <td style="text-align:right">3.443585</td> <td style="text-align:right">8.702762</td> <td style="text-align:right">4.024612</td> </tr> </table> <h3>Run 2</h3> <table style="padding:10px;border-spacing:10px;"> <tr> <th style="text-align:right">Report Date</th> <td style="text-align:center">2014-04-21</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td style="text-align:center">100</td> </tr> <tr> <th style="text-align:right">Total Data Storage/Database Size</th> <td style="text-align:center">1 TB / 87,496 MB</td> </tr> <tr> <th style="text-align:right">Start of Database Load</th> <td style="text-align:center">2014-04-21 21:02:43</td> </tr> <tr> <th style="text-align:right">End of Database Load</th> <td style="text-align:center">2014-04-21 21:17:45</td> </tr> <tr> <th style="text-align:right">Database Load Time</th> <td style="text-align:center">0:15:02</td> </tr> <tr> <th style="text-align:right">Query Streams for Throughput Test</th> <td style="text-align:center">5</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td style="text-align:center">257,944.7</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td style="text-align:center">240,998.0</td> </tr> <tr> <th style="text-align:right">Virt-H Composite Query-per-Hour Metric (Qph@100GB)</th> <td style="text-align:center">249,327.4</td> </tr> <tr> <th style="text-align:right">Measurement Interval in Throughput Test (Ts)</th> <td style="text-align:center">164.417000 seconds</td> </tr> </table> <h4>Duration of stream execution</h4> <table style="padding:10px;border-spacing:10px;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>2014-04-21 21:21:20</td> <td>2014-04-21 21:22:01</td> <td style="text-align:right">0:00:41</td> </tr> <tr> <th>Stream 1</th> <td>2014-04-21 21:22:02</td> <td>2014-04-21 21:24:41</td> <td style="text-align:right">0:02:39</td> </tr> <tr> <th>Stream 2</th> <td>2014-04-21 21:22:02</td> <td>2014-04-21 21:24:41</td> <td style="text-align:right">0:02:39</td> </tr> <tr> <th>Stream 3</th> <td>2014-04-21 21:22:02</td> <td>2014-04-21 21:24:41</td> <td style="text-align:right">0:02:39</td> </tr> <tr> <th>Stream 4</th> <td>2014-04-21 21:22:02</td> <td>2014-04-21 21:24:44</td> <td style="text-align:right">0:02:42</td> </tr> <tr> <th>Stream 5</th> <td>2014-04-21 21:22:02</td> <td>2014-04-21 21:24:46</td> <td style="text-align:right">0:02:44</td> </tr> <tr> <th>Refresh 0</th> <td>2014-04-21 21:21:20</td> <td>2014-04-21 21:21:22</td> <td style="text-align:right">0:00:02</td> </tr> <tr> <th>&$160;</th> <td>2014-04-21 21:21:22</td> <td>2014-04-21 21:21:23</td> <td style="text-align:right">0:00:01</td> </tr> <tr> <th>Refresh 1</th> <td>2014-04-21 21:22:49</td> <td>2014-04-21 21:23:04</td> <td style="text-align:right">0:00:15</td> </tr> <tr> <th>Refresh 2</th> <td>2014-04-21 21:22:01</td> <td>2014-04-21 21:22:14</td> <td style="text-align:right">0:00:13</td> </tr> <tr> <th>Refresh 3</th> <td>2014-04-21 21:22:14</td> <td>2014-04-21 21:22:27</td> <td style="text-align:right">0:00:13</td> </tr> <tr> <th>Refresh 4</th> <td>2014-04-21 21:22:26</td> <td>2014-04-21 21:22:39</td> <td style="text-align:right">0:00:13</td> </tr> <tr> <th>Refresh 5</th> <td>2014-04-21 21:22:39</td> <td>2014-04-21 21:22:49</td> <td style="text-align:right">0:00:10</td> </tr> </table> <h4>Numerical Quantities Summary -- Timing Intervals in Seconds</h4> <table style="padding:10px;border-spacing:10px;"> <tr> <th> </th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> </tr> <tr> <th>Stream 0</th> <td style="text-align:right">2.437262</td> <td style="text-align:right">0.227516</td> <td style="text-align:right">1.172620</td> <td style="text-align:right">0.541201</td> <td style="text-align:right">1.542084</td> <td style="text-align:right">0.743255</td> <td style="text-align:right">1.459368</td> <td style="text-align:right">1.183166</td> </tr> <tr> <th>Stream 1</th> <td style="text-align:right">5.205225</td> <td style="text-align:right">0.499833</td> <td style="text-align:right">4.854558</td> <td style="text-align:right">4.818087</td> <td style="text-align:right">5.920773</td> <td style="text-align:right">3.347414</td> <td style="text-align:right">5.446411</td> <td style="text-align:right">3.723247</td> </tr> <tr> <th>Stream 2</th> <td style="text-align:right">5.833803</td> <td style="text-align:right">0.659051</td> <td style="text-align:right">6.023266</td> <td style="text-align:right">3.123523</td> <td style="text-align:right">4.358200</td> <td style="text-align:right">3.371315</td> <td style="text-align:right">6.772453</td> <td style="text-align:right">4.978415</td> </tr> <tr> <th>Stream 3</th> <td style="text-align:right">6.308935</td> <td style="text-align:right">0.662744</td> <td style="text-align:right">7.573807</td> <td style="text-align:right">5.000859</td> <td style="text-align:right">5.282467</td> <td style="text-align:right">4.391930</td> <td style="text-align:right">5.280472</td> <td style="text-align:right">7.852718</td> </tr> <tr> <th>Stream 4</th> <td style="text-align:right">5.791856</td> <td style="text-align:right">0.421592</td> <td style="text-align:right">5.953592</td> <td style="text-align:right">4.688037</td> <td style="text-align:right">9.949038</td> <td style="text-align:right">3.098282</td> <td style="text-align:right">4.153124</td> <td style="text-align:right">4.824209</td> </tr> <tr> <th>Stream 5</th> <td style="text-align:right">13.537098</td> <td style="text-align:right">1.760386</td> <td style="text-align:right">3.308982</td> <td style="text-align:right">2.299178</td> <td style="text-align:right">4.882695</td> <td style="text-align:right">2.652497</td> <td style="text-align:right">5.383128</td> <td style="text-align:right">10.178447</td> </tr> <tr> <th>Min Qi</th> <td style="text-align:right">5.205225</td> <td style="text-align:right">0.421592</td> <td style="text-align:right">3.308982</td> <td style="text-align:right">2.299178</td> <td style="text-align:right">4.358200</td> <td style="text-align:right">2.652497</td> <td style="text-align:right">4.153124</td> <td style="text-align:right">3.723247</td> </tr> <tr> <th>Max Qi</th> <td style="text-align:right">13.537098</td> <td style="text-align:right">1.760386</td> <td style="text-align:right">7.573807</td> <td style="text-align:right">5.000859</td> <td style="text-align:right">9.949038</td> <td style="text-align:right">4.391930</td> <td style="text-align:right">6.772453</td> <td style="text-align:right">10.178447</td> </tr> <tr> <th>Avg Qi</th> <td style="text-align:right">7.335383</td> <td style="text-align:right">0.800721</td> <td style="text-align:right">5.542841</td> <td style="text-align:right">3.985937</td> <td style="text-align:right">6.078635</td> <td style="text-align:right">3.372288</td> <td style="text-align:right">5.407118</td> <td style="text-align:right">6.311407</td> </tr> <tr> <th> </th> </tr> <tr> <th> </th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> </tr> <tr> <th>Stream 0</th> <td style="text-align:right">4.441940</td> <td style="text-align:right">1.948770</td> <td style="text-align:right">2.154384</td> <td style="text-align:right">1.148494</td> <td style="text-align:right">6.014453</td> <td style="text-align:right">1.647725</td> <td style="text-align:right">1.437587</td> <td style="text-align:right">1.585284</td> </tr> <tr> <th>Stream 1</th> <td style="text-align:right">14.127674</td> <td style="text-align:right">7.824844</td> <td style="text-align:right">7.100679</td> <td style="text-align:right">3.586457</td> <td style="text-align:right">28.216115</td> <td style="text-align:right">7.587547</td> <td style="text-align:right">9.859152</td> <td style="text-align:right">5.829869</td> </tr> <tr> <th>Stream 2</th> <td style="text-align:right">16.102880</td> <td style="text-align:right">7.676986</td> <td style="text-align:right">5.887327</td> <td style="text-align:right">2.796729</td> <td style="text-align:right">24.847035</td> <td style="text-align:right">7.146757</td> <td style="text-align:right">11.408922</td> <td style="text-align:right">7.641239</td> </tr> <tr> <th>Stream 3</th> <td style="text-align:right">15.678701</td> <td style="text-align:right">5.786427</td> <td style="text-align:right">9.221883</td> <td style="text-align:right">2.692321</td> <td style="text-align:right">28.434916</td> <td style="text-align:right">6.657457</td> <td style="text-align:right">8.219745</td> <td style="text-align:right">7.706585</td> </tr> <tr> <th>Stream 4</th> <td style="text-align:right">11.985421</td> <td style="text-align:right">10.182807</td> <td style="text-align:right">5.667618</td> <td style="text-align:right">6.875264</td> <td style="text-align:right">27.547492</td> <td style="text-align:right">7.438075</td> <td style="text-align:right">9.065924</td> <td style="text-align:right">8.895070</td> </tr> <tr> <th>Stream 5</th> <td style="text-align:right">6.913707</td> <td style="text-align:right">7.662703</td> <td style="text-align:right">8.657333</td> <td style="text-align:right">3.282895</td> <td style="text-align:right">24.126612</td> <td style="text-align:right">10.963691</td> <td style="text-align:right">12.138564</td> <td style="text-align:right">7.962654</td> </tr> <tr> <th>Min Qi</th> <td style="text-align:right">6.913707</td> <td style="text-align:right">5.786427</td> <td style="text-align:right">5.667618</td> <td style="text-align:right">2.692321</td> <td style="text-align:right">24.126612</td> <td style="text-align:right">6.657457</td> <td style="text-align:right">8.219745</td> <td style="text-align:right">5.829869</td> </tr> <tr> <th>Max Qi</th> <td style="text-align:right">16.102880</td> <td style="text-align:right">10.182807</td> <td style="text-align:right">9.221883</td> <td style="text-align:right">6.875264</td> <td style="text-align:right">28.434916</td> <td style="text-align:right">10.963691</td> <td style="text-align:right">12.138564</td> <td style="text-align:right">8.895070</td> </tr> <tr> <th>Avg Qi</th> <td style="text-align:right">12.961677</td> <td style="text-align:right">7.826753</td> <td style="text-align:right">7.306968</td> <td style="text-align:right">3.846733</td> <td style="text-align:right">26.634434</td> <td style="text-align:right">7.958705</td> <td style="text-align:right">10.138461</td> <td style="text-align:right">7.607083</td> </tr> <tr> <th> </th> </tr> <tr> <th> </th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> </tr> <tr> <th>Stream 0</th> <td style="text-align:right">2.275267</td> <td style="text-align:right">1.139390</td> <td style="text-align:right">1.165591</td> <td style="text-align:right">2.073658</td> <td style="text-align:right">2.261869</td> <td style="text-align:right">0.703055</td> <td style="text-align:right">2.327755</td> <td style="text-align:right">1.146501</td> </tr> <tr> <th>Stream 1</th> <td style="text-align:right">13.720792</td> <td style="text-align:right">4.428528</td> <td style="text-align:right">3.651645</td> <td style="text-align:right">9.841610</td> <td style="text-align:right">6.710473</td> <td style="text-align:right">2.595879</td> <td style="text-align:right">9.783844</td> <td style="text-align:right">3.800103</td> </tr> <tr> <th>Stream 2</th> <td style="text-align:right">12.532257</td> <td style="text-align:right">2.312755</td> <td style="text-align:right">6.182661</td> <td style="text-align:right">8.666967</td> <td style="text-align:right">9.383983</td> <td style="text-align:right">1.414853</td> <td style="text-align:right">7.570509</td> <td style="text-align:right">4.539598</td> </tr> <tr> <th>Stream 3</th> <td style="text-align:right">7.578779</td> <td style="text-align:right">3.342352</td> <td style="text-align:right">8.155356</td> <td style="text-align:right">4.925493</td> <td style="text-align:right">6.590047</td> <td style="text-align:right">2.612912</td> <td style="text-align:right">8.497542</td> <td style="text-align:right">4.638512</td> </tr> <tr> <th>Stream 4</th> <td style="text-align:right">10.967178</td> <td style="text-align:right">2.173935</td> <td style="text-align:right">6.382803</td> <td style="text-align:right">5.082562</td> <td style="text-align:right">8.744671</td> <td style="text-align:right">3.074768</td> <td style="text-align:right">7.577794</td> <td style="text-align:right">4.435140</td> </tr> <tr> <th>Stream 5</th> <td style="text-align:right">9.438581</td> <td style="text-align:right">2.551124</td> <td style="text-align:right">8.375607</td> <td style="text-align:right">8.339441</td> <td style="text-align:right">8.201650</td> <td style="text-align:right">1.982935</td> <td style="text-align:right">7.334306</td> <td style="text-align:right">3.404017</td> </tr> <tr> <th>Min Qi</th> <td style="text-align:right">7.578779</td> <td style="text-align:right">2.173935</td> <td style="text-align:right">3.651645</td> <td style="text-align:right">4.925493</td> <td style="text-align:right">6.590047</td> <td style="text-align:right">1.414853</td> <td style="text-align:right">7.334306</td> <td style="text-align:right">3.404017</td> </tr> <tr> <th>Max Qi</th> <td style="text-align:right">13.720792</td> <td style="text-align:right">4.428528</td> <td style="text-align:right">8.375607</td> <td style="text-align:right">9.841610</td> <td style="text-align:right">9.383983</td> <td style="text-align:right">3.074768</td> <td style="text-align:right">9.783844</td> <td style="text-align:right">4.638512</td> </tr> <tr> <th>Avg Qi</th> <td style="text-align:right">10.847517</td> <td style="text-align:right">2.961739</td> <td style="text-align:right">6.549614</td> <td style="text-align:right">7.371215</td> <td style="text-align:right">7.926165</td> <td style="text-align:right">2.336269</td> <td style="text-align:right">8.152799</td> <td style="text-align:right">4.163474</td> </tr> </table> <h2>Details of System Under Test (SUT)</h2> <table style="padding:10px;border-spacing:10px;"> <tr> <th style="text-align:left"><h3>Hardware</h3> </th> </tr> <tr> </tr> <tr> <th style="text-align:right">Chassis</th> <td style="text-align:center">Supermicro 2U</td> </tr> <tr> <th style="text-align:right">Motherboard</th> <td style="text-align:center">Supermicro X9DR3-LN4F+</td> </tr> <tr> <th style="text-align:right">CPU</th> <td style="text-align:center">2 x Intel Xeon E5-2630 @ 2.3 GHz <br />(6 cores, 12 threads each; <br />total 12 cores, 24 threads)</td> </tr> <tr> <th style="text-align:right">RAM</th> <td style="text-align:center">192 GB DDR3 (24 x 8 GB, 1066MHz)</td> </tr> <tr> <th style="text-align:right">Storage</th> <td style="text-align:center">2 x Crucial 512 GB SSD</td> </tr> <tr> <th style="text-align:left"> </th> </tr> <tr> </tr> <tr> <th style="text-align:left"><h3>Software</h3> </th> </tr> <tr> </tr> <tr> <th style="text-align:right">DBMS</th> <td style="text-align:center"><a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/" id="link-id0x2aabde830908">Virtuoso Open Source</a> 7.11.3209 <br /> (feature/analytics on <a href="https://github.com/v7fasttrack/" id="link-id0x2aabf8012028">v7fasttrack on GitHub</a>)</td> </tr> <tr> <th style="text-align:right">OS</th> <td style="text-align:center">CentOS 6.2</td> </tr> </table> <h2>Conclusions</h2> <p>This experiment places <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aabfaa9a6b8">Virtuoso</a> in the ballpark with Actian Vector (formerly branded <a href="http://dbpedia.org/resource/Vectorwise" id="link-id0x2aabf9649118">Vectorwise</a>), which has dominated the TPC-H score board in recent years. The published Vector results are on more cores and/or faster clock; one would have to run on the exact same platform to make precise comparisons. </p> <p>Virtuoso ups the ante by providing this level of performance in open source. For a comparison with <a href="http://dbpedia.org/resource/EXASOL" id="link-id0x2aabfa689d98">EXASolution</a> and Actian Matrix (formerly <a href="http://dbpedia.org/resource/ParAccel" id="link-id0x2aabdee3a0e8">ParAccel</a>), we will have to go to the Virtuoso scale-out configuration, to follow shortly.</p> <p>The next articles will provide a detailed analysis of performance and instructions for reproducing the results. The run outputs and scripts are <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20140422TPCHResultsPreview" id="link-id0x2aabfb0e67f8">available for download</a>.</p> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x142d4f68"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0xf3282f8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x20ffa438"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x25c97448"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x1e70db48"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x14592b38"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x144af558"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x17d64988"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x172c9668"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x1cbe1308"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aabdf302818"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> In Hoc Signo Vinces (part 12): TPC-H: Result Preview<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac65be5358"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac312f46d8"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x710ece8"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aab74622528"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aabc933d038"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aab725ea578"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aac042d2928"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:09:58.952030-04:00
In Hoc Signo Vinces (part 12 of n): TPC-H: Result Preview
http://www.openlinksw.com:443/weblog/oerling/?date=2014-04-22#1796
2014-04-22T15:30:53Z
<p>In this article, we look at the 100 GB single-server results for the whole workload. We will call this <i>Virt-H</i> instead of <i><a href="http://www.tpc.org/tpch/" id="link-id0x2aabdeab62f8">TPC-H</a></i> in order to comply with the TPC rules: Use of the TPC-H label requires an audit.</p> <p>The test consists of a bulk load followed by two runs. Each run consists of a single user power test and a multi-user throughput test. The number of users in the throughput test is up to the test sponsor but must be at least 5 for the 100 GB scale. The reported score is the lower of the two scores.</p> <h2>Result Summary </h2> <table style="padding:10px;border-spacing:10px;"> <tr> <th style="text-align:right">Scale Factor</th> <td style="text-align:center">100 GB</td> </tr> <tr> <th style="text-align:right">dbgen</th> <td style="text-align:center">version 2.15</td> </tr> <tr> <th style="text-align:right">Lload time</th> <td style="text-align:center">0:15:02</td> </tr> <tr> <th style="text-align:right">Composite qph</th> <td style="text-align:center">241,482.3</td> </tr> <tr> <th style="text-align:right">System Availability Date</th> <td style="text-align:center">2014-04-22</td> </tr> </table> <p>The price/performance is left open. The hardware costs about 5000 euros and the software is open source so the cost per performance would be a minimum of 0.02 euros per qph at 100G. This is not compliant with the TPC pricing rules though. These require 3 year maintenance contracts for all parts.</p> <p>The software configuration did not use RAID. Otherwise the software would be auditable to the best of my knowledge. The hardware would have to be the same from Dell, HP, or other large brand to satisfy the TPC pricing rule.</p> <h2>Executive Summaries of Each Run</h2> <h3>Run 1</h3> <table style="padding:10px;border-spacing:10px;"> <tr> <th style="text-align:right">Report Date</th> <td style="text-align:center">2014-04-21</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td style="text-align:center">100</td> </tr> <tr> <th style="text-align:right">Total Data Storage/Database Size</th> <td style="text-align:center">1 TB / 87,496 MB</td> </tr> <tr> <th style="text-align:right">Start of Database Load</th> <td style="text-align:center">2014-04-21 21:02:43</td> </tr> <tr> <th style="text-align:right">End of Database Load</th> <td style="text-align:center">2014-04-21 21:17:45</td> </tr> <tr> <th style="text-align:right">Database Load Time</th> <td style="text-align:center">0:15:02</td> </tr> <tr> <th style="text-align:right">Query Streams for Throughput Test</th> <td style="text-align:center">5</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td style="text-align:center">239,785.1</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td style="text-align:center">243,191.4</td> </tr> <tr> <th style="text-align:right">Virt-H Composite Query-per-Hour Metric (Qph@100GB)</th> <td style="text-align:center">241,482.3</td> </tr> <tr> <th style="text-align:right">Measurement Interval in Throughput Test (Ts)</th> <td style="text-align:center">162.935000 seconds</td> </tr> </table> <h4>Duration of stream execution</h4> <table style="padding:10px;border-spacing:10px;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>2014-04-21 21:17:46</td> <td>2014-04-21 21:18:33</td> <td style="text-align:right">0:00:47</td> </tr> <tr> <th>Stream 1</th> <td>2014-04-21 21:18:33</td> <td>2014-04-21 21:21:13</td> <td style="text-align:right">0:02:40</td> </tr> <tr> <th>Stream 2</th> <td>2014-04-21 21:18:33</td> <td>2014-04-21 21:21:13</td> <td style="text-align:right">0:02:40</td> </tr> <tr> <th>Stream 3</th> <td>2014-04-21 21:18:33</td> <td>2014-04-21 21:21:06</td> <td style="text-align:right">0:02:33</td> </tr> <tr> <th>Stream 4</th> <td>2014-04-21 21:18:33</td> <td>2014-04-21 21:21:10</td> <td style="text-align:right">0:02:37</td> </tr> <tr> <th>Stream 5</th> <td>2014-04-21 21:18:33</td> <td>2014-04-21 21:21:16</td> <td style="text-align:right">0:02:43</td> </tr> <tr> <th>Refresh 0</th> <td>2014-04-21 21:17:46</td> <td>2014-04-21 21:17:49</td> <td style="text-align:right">0:00:03</td> </tr> <tr> <th> </th> <td>2014-04-21 21:17:50</td> <td>2014-04-21 21:17:51</td> <td style="text-align:right">0:00:01</td> </tr> <tr> <th>Refresh 1</th> <td>2014-04-21 21:19:25</td> <td>2014-04-21 21:19:38</td> <td style="text-align:right">0:00:13</td> </tr> <tr> <th>Refresh 2</th> <td>2014-04-21 21:18:33</td> <td>2014-04-21 21:18:48</td> <td style="text-align:right">0:00:15</td> </tr> <tr> <th>Refresh 3</th> <td>2014-04-21 21:18:49</td> <td>2014-04-21 21:19:01</td> <td style="text-align:right">0:00:12</td> </tr> <tr> <th>Refresh 4</th> <td>2014-04-21 21:19:01</td> <td>2014-04-21 21:19:13</td> <td style="text-align:right">0:00:12</td> </tr> <tr> <th>Refresh 5</th> <td>2014-04-21 21:19:13</td> <td>2014-04-21 21:19:25</td> <td style="text-align:right">0:00:12</td> </tr> </table> <h4>Numerical Quantities Summary -- Timing Intervals in Seconds</h4> <table style="padding:10px;border-spacing:10px;"> <tr> <th> </th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> </tr> <tr> <th>Stream 0</th> <td style="text-align:right">2.311882</td> <td style="text-align:right">0.383459</td> <td style="text-align:right">1.143286</td> <td style="text-align:right">0.439926</td> <td style="text-align:right">1.594027</td> <td style="text-align:right">0.736482</td> <td style="text-align:right">1.440826</td> <td style="text-align:right">1.198925</td> </tr> <tr> <th>Stream 1</th> <td style="text-align:right">5.192341</td> <td style="text-align:right">0.952574</td> <td style="text-align:right">6.184940</td> <td style="text-align:right">1.194804</td> <td style="text-align:right">6.998207</td> <td style="text-align:right">5.122059</td> <td style="text-align:right">5.962717</td> <td style="text-align:right">6.773401</td> </tr> <tr> <th>Stream 2</th> <td style="text-align:right">7.354001</td> <td style="text-align:right">1.191604</td> <td style="text-align:right">4.238262</td> <td style="text-align:right">1.770639</td> <td style="text-align:right">5.782669</td> <td style="text-align:right">1.357578</td> <td style="text-align:right">4.034697</td> <td style="text-align:right">6.354747</td> </tr> <tr> <th>Stream 3</th> <td style="text-align:right">6.489788</td> <td style="text-align:right">1.585291</td> <td style="text-align:right">4.645022</td> <td style="text-align:right">3.358926</td> <td style="text-align:right">7.904636</td> <td style="text-align:right">3.220767</td> <td style="text-align:right">5.694622</td> <td style="text-align:right">7.431067</td> </tr> <tr> <th>Stream 4</th> <td style="text-align:right">5.609555</td> <td style="text-align:right">1.066582</td> <td style="text-align:right">6.740518</td> <td style="text-align:right">2.503038</td> <td style="text-align:right">9.439980</td> <td style="text-align:right">3.424101</td> <td style="text-align:right">4.404849</td> <td style="text-align:right">4.256317</td> </tr> <tr> <th>Stream 5</th> <td style="text-align:right">10.346825</td> <td style="text-align:right">1.787459</td> <td style="text-align:right">4.391000</td> <td style="text-align:right">3.151059</td> <td style="text-align:right">4.974037</td> <td style="text-align:right">2.932079</td> <td style="text-align:right">6.191782</td> <td style="text-align:right">3.619255</td> </tr> <tr> <th>Min Qi</th> <td style="text-align:right">5.192341</td> <td style="text-align:right">0.952574</td> <td style="text-align:right">4.238262</td> <td style="text-align:right">1.194804</td> <td style="text-align:right">4.974037</td> <td style="text-align:right">1.357578</td> <td style="text-align:right">4.034697</td> <td style="text-align:right">3.619255</td> </tr> <tr> <th>Max Qi</th> <td style="text-align:right">10.346825</td> <td style="text-align:right">1.787459</td> <td style="text-align:right">6.740518</td> <td style="text-align:right">3.358926</td> <td style="text-align:right">9.439980</td> <td style="text-align:right">5.122059</td> <td style="text-align:right">6.191782</td> <td style="text-align:right">7.431067</td> </tr> <tr> <th>Avg Qi</th> <td style="text-align:right">6.998502</td> <td style="text-align:right">1.316702</td> <td style="text-align:right">5.239948</td> <td style="text-align:right">2.395693</td> <td style="text-align:right">7.019906</td> <td style="text-align:right">3.211317</td> <td style="text-align:right">5.257733</td> <td style="text-align:right">5.686957</td> </tr> <tr> <th> </th> </tr> <tr> <th> </th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> </tr> <tr> <th>Stream 0</th> <td style="text-align:right">4.476940</td> <td style="text-align:right">2.004782</td> <td style="text-align:right">2.070967</td> <td style="text-align:right">1.015134</td> <td style="text-align:right">7.995799</td> <td style="text-align:right">2.142581</td> <td style="text-align:right">1.989357</td> <td style="text-align:right">1.581758</td> </tr> <tr> <th>Stream 1</th> <td style="text-align:right">11.351299</td> <td style="text-align:right">6.657059</td> <td style="text-align:right">7.719765</td> <td style="text-align:right">5.157236</td> <td style="text-align:right">25.156379</td> <td style="text-align:right">8.566067</td> <td style="text-align:right">7.028898</td> <td style="text-align:right">8.146883</td> </tr> <tr> <th>Stream 2</th> <td style="text-align:right">13.954105</td> <td style="text-align:right">8.341359</td> <td style="text-align:right">10.265949</td> <td style="text-align:right">3.289724</td> <td style="text-align:right">25.249435</td> <td style="text-align:right">6.370577</td> <td style="text-align:right">11.262650</td> <td style="text-align:right">7.684574</td> </tr> <tr> <th>Stream 3</th> <td style="text-align:right">13.597277</td> <td style="text-align:right">5.783821</td> <td style="text-align:right">5.944240</td> <td style="text-align:right">5.214661</td> <td style="text-align:right">24.253991</td> <td style="text-align:right">8.742896</td> <td style="text-align:right">7.701709</td> <td style="text-align:right">5.801641</td> </tr> <tr> <th>Stream 4</th> <td style="text-align:right">15.612070</td> <td style="text-align:right">6.126494</td> <td style="text-align:right">4.533748</td> <td style="text-align:right">5.733828</td> <td style="text-align:right">23.021583</td> <td style="text-align:right">6.423207</td> <td style="text-align:right">8.358223</td> <td style="text-align:right">6.866477</td> </tr> <tr> <th>Stream 5</th> <td style="text-align:right">8.421209</td> <td style="text-align:right">9.040726</td> <td style="text-align:right">7.799425</td> <td style="text-align:right">3.908758</td> <td style="text-align:right">23.342975</td> <td style="text-align:right">9.934672</td> <td style="text-align:right">11.455598</td> <td style="text-align:right">8.258504</td> </tr> <tr> <th>Min Qi</th> <td style="text-align:right">8.421209</td> <td style="text-align:right">5.783821</td> <td style="text-align:right">4.533748</td> <td style="text-align:right">3.289724</td> <td style="text-align:right">23.021583</td> <td style="text-align:right">6.370577</td> <td style="text-align:right">7.028898</td> <td style="text-align:right">5.801641</td> </tr> <tr> <th>Max Qi</th> <td style="text-align:right">15.612070</td> <td style="text-align:right">9.040726</td> <td style="text-align:right">10.265949</td> <td style="text-align:right">5.733828</td> <td style="text-align:right">25.249435</td> <td style="text-align:right">9.934672</td> <td style="text-align:right">11.455598</td> <td style="text-align:right">8.258504</td> </tr> <tr> <th>Avg Qi</th> <td style="text-align:right">12.587192</td> <td style="text-align:right">7.189892</td> <td style="text-align:right">7.252625</td> <td style="text-align:right">4.660841</td> <td style="text-align:right">24.204873</td> <td style="text-align:right">8.007484</td> <td style="text-align:right">9.161416</td> <td style="text-align:right">7.351616</td> </tr> <tr> <th> </th> </tr> <tr> <th> </th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> </tr> <tr> <th>Stream 0</th> <td style="text-align:right">2.258070</td> <td style="text-align:right">0.981896</td> <td style="text-align:right">1.161602</td> <td style="text-align:right">1.933124</td> <td style="text-align:right">2.203497</td> <td style="text-align:right">1.042949</td> <td style="text-align:right">3.349407</td> <td style="text-align:right">1.296630</td> </tr> <tr> <th>Stream 1</th> <td style="text-align:right">8.213340</td> <td style="text-align:right">4.070175</td> <td style="text-align:right">5.662723</td> <td style="text-align:right">12.260503</td> <td style="text-align:right">7.792825</td> <td style="text-align:right">3.323136</td> <td style="text-align:right">9.296430</td> <td style="text-align:right">3.939927</td> </tr> <tr> <th>Stream 2</th> <td style="text-align:right">16.754827</td> <td style="text-align:right">3.895688</td> <td style="text-align:right">4.413773</td> <td style="text-align:right">7.529466</td> <td style="text-align:right">6.288539</td> <td style="text-align:right">2.717479</td> <td style="text-align:right">11.222082</td> <td style="text-align:right">4.135510</td> </tr> <tr> <th>Stream 3</th> <td style="text-align:right">8.486809</td> <td style="text-align:right">2.615640</td> <td style="text-align:right">7.426936</td> <td style="text-align:right">7.274289</td> <td style="text-align:right">6.706145</td> <td style="text-align:right">3.402654</td> <td style="text-align:right">8.278881</td> <td style="text-align:right">4.260483</td> </tr> <tr> <th>Stream 4</th> <td style="text-align:right">12.604905</td> <td style="text-align:right">7.735042</td> <td style="text-align:right">5.627039</td> <td style="text-align:right">6.343302</td> <td style="text-align:right">7.242370</td> <td style="text-align:right">3.492640</td> <td style="text-align:right">6.503095</td> <td style="text-align:right">3.698821</td> </tr> <tr> <th>Stream 5</th> <td style="text-align:right">8.221733</td> <td style="text-align:right">2.670036</td> <td style="text-align:right">5.866626</td> <td style="text-align:right">13.108081</td> <td style="text-align:right">9.428098</td> <td style="text-align:right">4.282014</td> <td style="text-align:right">8.213320</td> <td style="text-align:right">4.088321</td> </tr> <tr> <th>Min Qi</th> <td style="text-align:right">8.213340</td> <td style="text-align:right">2.615640</td> <td style="text-align:right">4.413773</td> <td style="text-align:right">6.343302</td> <td style="text-align:right">6.288539</td> <td style="text-align:right">2.717479</td> <td style="text-align:right">6.503095</td> <td style="text-align:right">3.698821</td> </tr> <tr> <th>Max Qi</th> <td style="text-align:right">16.754827</td> <td style="text-align:right">7.735042</td> <td style="text-align:right">7.426936</td> <td style="text-align:right">13.108081</td> <td style="text-align:right">9.428098</td> <td style="text-align:right">4.282014</td> <td style="text-align:right">11.222082</td> <td style="text-align:right">4.260483</td> </tr> <tr> <th>Avg Qi</th> <td style="text-align:right">10.856323</td> <td style="text-align:right">4.197316</td> <td style="text-align:right">5.799419</td> <td style="text-align:right">9.303128</td> <td style="text-align:right">7.491595</td> <td style="text-align:right">3.443585</td> <td style="text-align:right">8.702762</td> <td style="text-align:right">4.024612</td> </tr> </table> <h3>Run 2</h3> <table style="padding:10px;border-spacing:10px;"> <tr> <th style="text-align:right">Report Date</th> <td style="text-align:center">2014-04-21</td> </tr> <tr> <th style="text-align:right">Database Scale Factor</th> <td style="text-align:center">100</td> </tr> <tr> <th style="text-align:right">Total Data Storage/Database Size</th> <td style="text-align:center">1 TB / 87,496 MB</td> </tr> <tr> <th style="text-align:right">Start of Database Load</th> <td style="text-align:center">2014-04-21 21:02:43</td> </tr> <tr> <th style="text-align:right">End of Database Load</th> <td style="text-align:center">2014-04-21 21:17:45</td> </tr> <tr> <th style="text-align:right">Database Load Time</th> <td style="text-align:center">0:15:02</td> </tr> <tr> <th style="text-align:right">Query Streams for Throughput Test</th> <td style="text-align:center">5</td> </tr> <tr> <th style="text-align:right">Virt-H Power</th> <td style="text-align:center">257,944.7</td> </tr> <tr> <th style="text-align:right">Virt-H Throughput</th> <td style="text-align:center">240,998.0</td> </tr> <tr> <th style="text-align:right">Virt-H Composite Query-per-Hour Metric (Qph@100GB)</th> <td style="text-align:center">249,327.4</td> </tr> <tr> <th style="text-align:right">Measurement Interval in Throughput Test (Ts)</th> <td style="text-align:center">164.417000 seconds</td> </tr> </table> <h4>Duration of stream execution</h4> <table style="padding:10px;border-spacing:10px;"> <tr> <th> </th> <th>Start Date/Time</th> <th>End Date/Time</th> <th>Duration</th> </tr> <tr> <th>Stream 0</th> <td>2014-04-21 21:21:20</td> <td>2014-04-21 21:22:01</td> <td style="text-align:right">0:00:41</td> </tr> <tr> <th>Stream 1</th> <td>2014-04-21 21:22:02</td> <td>2014-04-21 21:24:41</td> <td style="text-align:right">0:02:39</td> </tr> <tr> <th>Stream 2</th> <td>2014-04-21 21:22:02</td> <td>2014-04-21 21:24:41</td> <td style="text-align:right">0:02:39</td> </tr> <tr> <th>Stream 3</th> <td>2014-04-21 21:22:02</td> <td>2014-04-21 21:24:41</td> <td style="text-align:right">0:02:39</td> </tr> <tr> <th>Stream 4</th> <td>2014-04-21 21:22:02</td> <td>2014-04-21 21:24:44</td> <td style="text-align:right">0:02:42</td> </tr> <tr> <th>Stream 5</th> <td>2014-04-21 21:22:02</td> <td>2014-04-21 21:24:46</td> <td style="text-align:right">0:02:44</td> </tr> <tr> <th>Refresh 0</th> <td>2014-04-21 21:21:20</td> <td>2014-04-21 21:21:22</td> <td style="text-align:right">0:00:02</td> </tr> <tr> <th>&$160;</th> <td>2014-04-21 21:21:22</td> <td>2014-04-21 21:21:23</td> <td style="text-align:right">0:00:01</td> </tr> <tr> <th>Refresh 1</th> <td>2014-04-21 21:22:49</td> <td>2014-04-21 21:23:04</td> <td style="text-align:right">0:00:15</td> </tr> <tr> <th>Refresh 2</th> <td>2014-04-21 21:22:01</td> <td>2014-04-21 21:22:14</td> <td style="text-align:right">0:00:13</td> </tr> <tr> <th>Refresh 3</th> <td>2014-04-21 21:22:14</td> <td>2014-04-21 21:22:27</td> <td style="text-align:right">0:00:13</td> </tr> <tr> <th>Refresh 4</th> <td>2014-04-21 21:22:26</td> <td>2014-04-21 21:22:39</td> <td style="text-align:right">0:00:13</td> </tr> <tr> <th>Refresh 5</th> <td>2014-04-21 21:22:39</td> <td>2014-04-21 21:22:49</td> <td style="text-align:right">0:00:10</td> </tr> </table> <h4>Numerical Quantities Summary -- Timing Intervals in Seconds</h4> <table style="padding:10px;border-spacing:10px;"> <tr> <th> </th> <th>Q1</th> <th>Q2</th> <th>Q3</th> <th>Q4</th> <th>Q5</th> <th>Q6</th> <th>Q7</th> <th>Q8</th> </tr> <tr> </tr> <tr> <th>Stream 0</th> <td style="text-align:right">2.437262</td> <td style="text-align:right">0.227516</td> <td style="text-align:right">1.172620</td> <td style="text-align:right">0.541201</td> <td style="text-align:right">1.542084</td> <td style="text-align:right">0.743255</td> <td style="text-align:right">1.459368</td> <td style="text-align:right">1.183166</td> </tr> <tr> <th>Stream 1</th> <td style="text-align:right">5.205225</td> <td style="text-align:right">0.499833</td> <td style="text-align:right">4.854558</td> <td style="text-align:right">4.818087</td> <td style="text-align:right">5.920773</td> <td style="text-align:right">3.347414</td> <td style="text-align:right">5.446411</td> <td style="text-align:right">3.723247</td> </tr> <tr> <th>Stream 2</th> <td style="text-align:right">5.833803</td> <td style="text-align:right">0.659051</td> <td style="text-align:right">6.023266</td> <td style="text-align:right">3.123523</td> <td style="text-align:right">4.358200</td> <td style="text-align:right">3.371315</td> <td style="text-align:right">6.772453</td> <td style="text-align:right">4.978415</td> </tr> <tr> <th>Stream 3</th> <td style="text-align:right">6.308935</td> <td style="text-align:right">0.662744</td> <td style="text-align:right">7.573807</td> <td style="text-align:right">5.000859</td> <td style="text-align:right">5.282467</td> <td style="text-align:right">4.391930</td> <td style="text-align:right">5.280472</td> <td style="text-align:right">7.852718</td> </tr> <tr> <th>Stream 4</th> <td style="text-align:right">5.791856</td> <td style="text-align:right">0.421592</td> <td style="text-align:right">5.953592</td> <td style="text-align:right">4.688037</td> <td style="text-align:right">9.949038</td> <td style="text-align:right">3.098282</td> <td style="text-align:right">4.153124</td> <td style="text-align:right">4.824209</td> </tr> <tr> <th>Stream 5</th> <td style="text-align:right">13.537098</td> <td style="text-align:right">1.760386</td> <td style="text-align:right">3.308982</td> <td style="text-align:right">2.299178</td> <td style="text-align:right">4.882695</td> <td style="text-align:right">2.652497</td> <td style="text-align:right">5.383128</td> <td style="text-align:right">10.178447</td> </tr> <tr> <th>Min Qi</th> <td style="text-align:right">5.205225</td> <td style="text-align:right">0.421592</td> <td style="text-align:right">3.308982</td> <td style="text-align:right">2.299178</td> <td style="text-align:right">4.358200</td> <td style="text-align:right">2.652497</td> <td style="text-align:right">4.153124</td> <td style="text-align:right">3.723247</td> </tr> <tr> <th>Max Qi</th> <td style="text-align:right">13.537098</td> <td style="text-align:right">1.760386</td> <td style="text-align:right">7.573807</td> <td style="text-align:right">5.000859</td> <td style="text-align:right">9.949038</td> <td style="text-align:right">4.391930</td> <td style="text-align:right">6.772453</td> <td style="text-align:right">10.178447</td> </tr> <tr> <th>Avg Qi</th> <td style="text-align:right">7.335383</td> <td style="text-align:right">0.800721</td> <td style="text-align:right">5.542841</td> <td style="text-align:right">3.985937</td> <td style="text-align:right">6.078635</td> <td style="text-align:right">3.372288</td> <td style="text-align:right">5.407118</td> <td style="text-align:right">6.311407</td> </tr> <tr> <th> </th> </tr> <tr> <th> </th> <th>Q9</th> <th>Q10</th> <th>Q11</th> <th>Q12</th> <th>Q13</th> <th>Q14</th> <th>Q15</th> <th>Q16</th> </tr> <tr> </tr> <tr> <th>Stream 0</th> <td style="text-align:right">4.441940</td> <td style="text-align:right">1.948770</td> <td style="text-align:right">2.154384</td> <td style="text-align:right">1.148494</td> <td style="text-align:right">6.014453</td> <td style="text-align:right">1.647725</td> <td style="text-align:right">1.437587</td> <td style="text-align:right">1.585284</td> </tr> <tr> <th>Stream 1</th> <td style="text-align:right">14.127674</td> <td style="text-align:right">7.824844</td> <td style="text-align:right">7.100679</td> <td style="text-align:right">3.586457</td> <td style="text-align:right">28.216115</td> <td style="text-align:right">7.587547</td> <td style="text-align:right">9.859152</td> <td style="text-align:right">5.829869</td> </tr> <tr> <th>Stream 2</th> <td style="text-align:right">16.102880</td> <td style="text-align:right">7.676986</td> <td style="text-align:right">5.887327</td> <td style="text-align:right">2.796729</td> <td style="text-align:right">24.847035</td> <td style="text-align:right">7.146757</td> <td style="text-align:right">11.408922</td> <td style="text-align:right">7.641239</td> </tr> <tr> <th>Stream 3</th> <td style="text-align:right">15.678701</td> <td style="text-align:right">5.786427</td> <td style="text-align:right">9.221883</td> <td style="text-align:right">2.692321</td> <td style="text-align:right">28.434916</td> <td style="text-align:right">6.657457</td> <td style="text-align:right">8.219745</td> <td style="text-align:right">7.706585</td> </tr> <tr> <th>Stream 4</th> <td style="text-align:right">11.985421</td> <td style="text-align:right">10.182807</td> <td style="text-align:right">5.667618</td> <td style="text-align:right">6.875264</td> <td style="text-align:right">27.547492</td> <td style="text-align:right">7.438075</td> <td style="text-align:right">9.065924</td> <td style="text-align:right">8.895070</td> </tr> <tr> <th>Stream 5</th> <td style="text-align:right">6.913707</td> <td style="text-align:right">7.662703</td> <td style="text-align:right">8.657333</td> <td style="text-align:right">3.282895</td> <td style="text-align:right">24.126612</td> <td style="text-align:right">10.963691</td> <td style="text-align:right">12.138564</td> <td style="text-align:right">7.962654</td> </tr> <tr> <th>Min Qi</th> <td style="text-align:right">6.913707</td> <td style="text-align:right">5.786427</td> <td style="text-align:right">5.667618</td> <td style="text-align:right">2.692321</td> <td style="text-align:right">24.126612</td> <td style="text-align:right">6.657457</td> <td style="text-align:right">8.219745</td> <td style="text-align:right">5.829869</td> </tr> <tr> <th>Max Qi</th> <td style="text-align:right">16.102880</td> <td style="text-align:right">10.182807</td> <td style="text-align:right">9.221883</td> <td style="text-align:right">6.875264</td> <td style="text-align:right">28.434916</td> <td style="text-align:right">10.963691</td> <td style="text-align:right">12.138564</td> <td style="text-align:right">8.895070</td> </tr> <tr> <th>Avg Qi</th> <td style="text-align:right">12.961677</td> <td style="text-align:right">7.826753</td> <td style="text-align:right">7.306968</td> <td style="text-align:right">3.846733</td> <td style="text-align:right">26.634434</td> <td style="text-align:right">7.958705</td> <td style="text-align:right">10.138461</td> <td style="text-align:right">7.607083</td> </tr> <tr> <th> </th> </tr> <tr> <th> </th> <th>Q17</th> <th>Q18</th> <th>Q19</th> <th>Q20</th> <th>Q21</th> <th>Q22</th> <th>RF1</th> <th>RF2</th> </tr> <tr> </tr> <tr> <th>Stream 0</th> <td style="text-align:right">2.275267</td> <td style="text-align:right">1.139390</td> <td style="text-align:right">1.165591</td> <td style="text-align:right">2.073658</td> <td style="text-align:right">2.261869</td> <td style="text-align:right">0.703055</td> <td style="text-align:right">2.327755</td> <td style="text-align:right">1.146501</td> </tr> <tr> <th>Stream 1</th> <td style="text-align:right">13.720792</td> <td style="text-align:right">4.428528</td> <td style="text-align:right">3.651645</td> <td style="text-align:right">9.841610</td> <td style="text-align:right">6.710473</td> <td style="text-align:right">2.595879</td> <td style="text-align:right">9.783844</td> <td style="text-align:right">3.800103</td> </tr> <tr> <th>Stream 2</th> <td style="text-align:right">12.532257</td> <td style="text-align:right">2.312755</td> <td style="text-align:right">6.182661</td> <td style="text-align:right">8.666967</td> <td style="text-align:right">9.383983</td> <td style="text-align:right">1.414853</td> <td style="text-align:right">7.570509</td> <td style="text-align:right">4.539598</td> </tr> <tr> <th>Stream 3</th> <td style="text-align:right">7.578779</td> <td style="text-align:right">3.342352</td> <td style="text-align:right">8.155356</td> <td style="text-align:right">4.925493</td> <td style="text-align:right">6.590047</td> <td style="text-align:right">2.612912</td> <td style="text-align:right">8.497542</td> <td style="text-align:right">4.638512</td> </tr> <tr> <th>Stream 4</th> <td style="text-align:right">10.967178</td> <td style="text-align:right">2.173935</td> <td style="text-align:right">6.382803</td> <td style="text-align:right">5.082562</td> <td style="text-align:right">8.744671</td> <td style="text-align:right">3.074768</td> <td style="text-align:right">7.577794</td> <td style="text-align:right">4.435140</td> </tr> <tr> <th>Stream 5</th> <td style="text-align:right">9.438581</td> <td style="text-align:right">2.551124</td> <td style="text-align:right">8.375607</td> <td style="text-align:right">8.339441</td> <td style="text-align:right">8.201650</td> <td style="text-align:right">1.982935</td> <td style="text-align:right">7.334306</td> <td style="text-align:right">3.404017</td> </tr> <tr> <th>Min Qi</th> <td style="text-align:right">7.578779</td> <td style="text-align:right">2.173935</td> <td style="text-align:right">3.651645</td> <td style="text-align:right">4.925493</td> <td style="text-align:right">6.590047</td> <td style="text-align:right">1.414853</td> <td style="text-align:right">7.334306</td> <td style="text-align:right">3.404017</td> </tr> <tr> <th>Max Qi</th> <td style="text-align:right">13.720792</td> <td style="text-align:right">4.428528</td> <td style="text-align:right">8.375607</td> <td style="text-align:right">9.841610</td> <td style="text-align:right">9.383983</td> <td style="text-align:right">3.074768</td> <td style="text-align:right">9.783844</td> <td style="text-align:right">4.638512</td> </tr> <tr> <th>Avg Qi</th> <td style="text-align:right">10.847517</td> <td style="text-align:right">2.961739</td> <td style="text-align:right">6.549614</td> <td style="text-align:right">7.371215</td> <td style="text-align:right">7.926165</td> <td style="text-align:right">2.336269</td> <td style="text-align:right">8.152799</td> <td style="text-align:right">4.163474</td> </tr> </table> <h2>Details of System Under Test (SUT)</h2> <table style="padding:10px;border-spacing:10px;"> <tr> <th style="text-align:left"><h3>Hardware</h3> </th> </tr> <tr> </tr> <tr> <th style="text-align:right">Chassis</th> <td style="text-align:center">Supermicro 2U</td> </tr> <tr> <th style="text-align:right">Motherboard</th> <td style="text-align:center">Supermicro X9DR3-LN4F+</td> </tr> <tr> <th style="text-align:right">CPU</th> <td style="text-align:center">2 x Intel Xeon E5-2630 @ 2.3 GHz <br />(6 cores, 12 threads each; <br />total 12 cores, 24 threads)</td> </tr> <tr> <th style="text-align:right">RAM</th> <td style="text-align:center">192 GB DDR3 (24 x 8 GB, 1066MHz)</td> </tr> <tr> <th style="text-align:right">Storage</th> <td style="text-align:center">2 x Crucial 512 GB SSD</td> </tr> <tr> <th style="text-align:left"> </th> </tr> <tr> </tr> <tr> <th style="text-align:left"><h3>Software</h3> </th> </tr> <tr> </tr> <tr> <th style="text-align:right">DBMS</th> <td style="text-align:center"><a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/" id="link-id0x2aabde830908">Virtuoso Open Source</a> 7.11.3209 <br /> (feature/analytics on <a href="https://github.com/v7fasttrack/" id="link-id0x2aabf8012028">v7fasttrack on GitHub</a>)</td> </tr> <tr> <th style="text-align:right">OS</th> <td style="text-align:center">CentOS 6.2</td> </tr> </table> <h2>Conclusions</h2> <p>This experiment places <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aabfaa9a6b8">Virtuoso</a> in the ballpark with Actian Vector (formerly branded <a href="http://dbpedia.org/resource/Vectorwise" id="link-id0x2aabf9649118">Vectorwise</a>), which has dominated the TPC-H score board in recent years. The published Vector results are on more cores and/or faster clock; one would have to run on the exact same platform to make precise comparisons. </p> <p>Virtuoso ups the ante by providing this level of performance in open source. For a comparison with <a href="http://dbpedia.org/resource/EXASOL" id="link-id0x2aabfa689d98">EXASolution</a> and Actian Matrix (formerly <a href="http://dbpedia.org/resource/ParAccel" id="link-id0x2aabdee3a0e8">ParAccel</a>), we will have to go to the Virtuoso scale-out configuration, to follow shortly.</p> <p>The next articles will provide a detailed analysis of performance and instructions for reproducing the results. The run outputs and scripts are <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFiles20140422TPCHResultsPreview" id="link-id0x2aabfb0e67f8">available for download</a>.</p> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x142d4f68"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0xf3282f8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x20ffa438"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x25c97448"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x1e70db48"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x14592b38"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x144af558"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x17d64988"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x172c9668"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x1cbe1308"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aabdf302818"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> In Hoc Signo Vinces (part 12): TPC-H: Result Preview<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac7533f0c8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0xba10998"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x19da8318"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0xe9fec78"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0xc87ada8"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aabd6e8dee8"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aac04bd3b48"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:06:58.564111-04:00
In Hoc Signo Vinces (part 11 of n): TPC-H Q2, Q10 - Late Projection
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-04-07#1795
2014-04-07T16:30:54Z
<p>Analytics is generally about making something small out of something large. This reduction is obtained by a <code>TOP k</code> operator (i.e., show only the 10 best by some metric) and/or by grouping and aggregation (i.e., for a set of items, show some attributes of these items and a sum, count, or other aggregate of dependent items for each).</p> <p>In this installment we will look at <i>late projection,</i> also sometimes known as <i>late materialization.</i> If many attributes are returned and there is a cutoff of some sort, then the query does not need to be concerned about attributes on which there are no conditions, except for fetching them at the last moment, only for the entities which in fact will be returned to the user.</p> <p>We look at <a href="http://www.tpc.org/tpch/" id="link-id0x2aac3c57d488">TPC-H</a> Q2 and Q10.</p> <h3>Q2:</h3> <blockquote> <code><pre> SELECT TOP 100 s_acctbal, s_name, n_name, p_partkey, p_mfgr, s_address, s_phone, s_comment FROM part, supplier, partsupp, nation, region WHERE p_partkey = ps_partkey AND s_suppkey = ps_suppkey AND p_size = 15 AND p_type LIKE '%BRASS' AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'EUROPE' AND ps_supplycost = ( SELECT MIN(ps_supplycost) FROM partsupp, supplier, nation, region WHERE p_partkey = ps_partkey AND s_suppkey = ps_suppkey AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'EUROPE' ) ORDER BY s_acctbal DESC, n_name, s_name, p_partkey </pre> </code> </blockquote> <p>The intent is to return information about <code>parts</code> and <code>suppliers</code>, such that the <code>part</code> is available from a <code>supplier</code> in Europe, and the <code>supplier</code> has the lowest <code>price</code> for the <code>part</code> among all European <code>suppliers</code>.</p> <h3>Q10:</h3> <blockquote> <code><pre> SELECT TOP 20 c_custkey, c_name, SUM(l_extendedprice * (1 - l_discount)) AS revenue, c_acctbal, n_name, c_address, c_phone, c_comment FROM customer, orders, lineitem, nation WHERE c_custkey = o_custkey AND l_orderkey = o_orderkey AND o_orderdate >= CAST ('1993-10-01' AS DATE) AND o_orderdate < DATEADD ('month', 3, CAST ('1993-10-01' AS DATE)) AND l_returnflag = 'R' AND c_nationkey = n_nationkey GROUP BY c_custkey, c_name, c_acctbal, c_phone, n_name, c_address, c_comment ORDER BY revenue DESC </pre> </code> </blockquote> <p>The intent is to list the <code>customers</code> who cause the greatest loss of revenue in a given quarter by returning <code>items</code> ordered in said quarter.</p> <p>We notice that both queries return many columns on which there are no conditions, and that both have a cap on returned rows. The difference is that in Q2 the major <code>ORDER BY</code> is on a grouping column, and in Q10 it is on the aggregate of the <code>GROUP BY</code>. Thus the <code>TOP k</code> trick discussed in <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac1e584578">the previous article</a> does apply to Q2 but not to Q10.</p> <p>The profile for Q2 follows:</p> <blockquote> <code><pre> { time 6.1e-05% fanout 1 input 1 rows time 1.1% fanout 1 input 1 rows { hash filler Subquery 27 { time 0.0012% fanout 1 input 1 rows REGION 1 rows(t10.R_REGIONKEY) R_NAME = <c EUROPE> time 0.00045% fanout 5 input 1 rows NATION 5 rows(t9.N_NATIONKEY) N_REGIONKEY = t10.R_REGIONKEY time 1.6% fanout 40107 input 5 rows SUPPLIER 4.2e+04 rows(t8.S_SUPPKEY) S_NATIONKEY = t9.N_NATIONKEY After code: 0: t8.S_SUPPKEY := := artm t8.S_SUPPKEY 4: BReturn 0 time 0.1% fanout 0 input 200535 rows Sort hf 49 (t8.S_SUPPKEY) } } time 0.0004% fanout 1 input 1 rows { fork time 21% fanout 79591 input 1 rows PART 8e+04 rows(.P_PARTKEY) P_TYPE LIKE <c %BRASS> LIKE <c > , P_SIZE = 15 time 44% fanout 0.591889 input 79591 rows Precode: 0: { time 0.083% fanout 1 input 79591 rows time 0.13% fanout 1 input 79591 rows { fork time 24% fanout 0.801912 input 79591 rows PARTSUPP 3.5 rows(.PS_SUPPKEY, .PS_SUPPLYCOST) inlined PS_PARTKEY = k_.P_PARTKEY hash partition+bloom by 62 (tmp)hash join merged always card 0.2 -> () time 1.3% fanout 0 input 63825 rows Hash source 49 merged into ts not partitionable 0.2 rows(.PS_SUPPKEY) -> () After code: 0: min min.PS_SUPPLYCOSTset no set_ctr 5: BReturn 0 } After code: 0: aggregate := := artm min 4: BReturn 0 time 0.19% fanout 0 input 79591 rows Subquery Select(aggregate) } 8: BReturn 0 PARTSUPP 5e-08 rows(.PS_SUPPKEY) inlined PS_PARTKEY = k_.P_PARTKEY PS_SUPPLYCOST = k_scalar time 5.9% fanout 0.247023 input 47109 rows SUPPLIER unq 0.9 rows (.S_ACCTBAL, .S_NATIONKEY, .S_NAME, .S_SUPPKEY) inlined S_SUPPKEY = .PS_SUPPKEY top k on S_ACCTBAL time 0.077% fanout 1 input 11637 rows NATION unq 1 rows (.N_REGIONKEY, .N_NAME) inlined N_NATIONKEY = .S_NATIONKEY time 0.051% fanout 1 input 11637 rows REGION unq 0.2 rows () inlined R_REGIONKEY = .N_REGIONKEY R_NAME = <c EUROPE> time 0.42% fanout 0 input 11637 rows Sort (.S_ACCTBAL, .N_NAME, .S_NAME, .P_PARTKEY) -> (.S_SUPPKEY) } time 0.0016% fanout 100 input 1 rows top order by read (.S_SUPPKEY, .P_PARTKEY, .N_NAME, .S_NAME, .S_ACCTBAL) time 0.02% fanout 1 input 100 rows PART unq 0.95 rows (.P_MFGR) inlined P_PARTKEY = .P_PARTKEY time 0.054% fanout 1 input 100 rows SUPPLIER unq 1 rows (.S_PHONE, .S_ADDRESS, .S_COMMENT) inlined S_SUPPKEY = k_.S_SUPPKEY time 6.7e-05% fanout 0 input 100 rows Select (.S_ACCTBAL, .S_NAME, .N_NAME, .P_PARTKEY, .P_MFGR, .S_ADDRESS, .S_PHONE, .S_COMMENT) } 128 msec 1007% cpu, 196992 rnd 2.53367e+07 seq 50.4135% same seg 45.3574% same pg </pre> </code> </blockquote> <p>The query starts with a scan looking for the qualifying <code>parts</code>. It then looks for the best <code>price</code> for each <code>part</code> from a European <code>supplier</code>. All the European <code>suppliers</code> have been previously put in a hash table by the hash filler subquery at the start of the plan. Thus, to find the minimum <code>price</code>, the query takes the <code>partsupp</code> for the <code>part</code> by index, and then eliminates all non-European <code>suppliers</code> by a selective hash join. After this, there is a second index lookup on <code>partsupp</code> where we look for the <code>part</code> and the <code>price</code> equal to the minimum <code>price</code> found earlier. These operations could in principle be merged, as the minimum <code>price</code> <code>partsupp</code> has already been seen. The gain would not be very large, though.</p> <p>Here we note that the cost model guesses that very few rows will survive the check of <code>ps_supplycost =</code> minimum <code>cost</code>. It does not know that the minimum is not just any value, but one of the values that do occur in the <code>ps_supplycost</code> column for the part. Because of this, the remainder of the plan is carried out by index, which is just as well. The point is that if very few rows of input are expected, it is not worthwhile to make a hash table for a hash join. The hash table made for the European <code>suppliers</code> could be reused here, maybe with some small gain. It would however need more columns, which might make it not worthwhile. We note that the major order with the <code>TOP k</code> is on the <code>supplier</code> <code>s_acctbal</code>, hence as soon as there are 100 <code>suppliers</code> found, one can add a restriction on the <code>s_acctbal</code> for subsequent ones.</p> <p>At the end of the plan, after the <code>TOP k ORDER BY</code> and the reading of the results, we have a separate index-based lookup for getting only the columns that are returned. We note that this is done on 100 rows whereas the previous operations are done on tens-of-thousands of rows. The <code>TOP k</code> restriction produces some benefit, but it is relatively late in the plan, and not many operations follow it.</p> <p>The plan is easily good enough, with only small space for improvement. Q2 is one of the fastest queries of the set. </p> <p>Let us now consider the execution of Q10:</p> <blockquote> <code><pre> { time 1.1e-06% fanout 1 input 1 rows time 4.4e-05% fanout 1 input 1 rows { hash filler time 1.6e-05% fanout 25 input 1 rows NATION 25 rows(.N_NATIONKEY, .N_NAME) time 6.7e-06% fanout 0 input 25 rows Sort hf 35 (.N_NATIONKEY) -> (.N_NAME) } time 1.5e-06% fanout 1 input 1 rows { fork time 2.4e-06% fanout 1 input 1 rows { fork time 13% fanout 5.73038e+06 input 1 rows ORDERS 5.1e+06 rows(.O_ORDERKEY, .O_CUSTKEY) O_ORDERDATE >= <c 1993-10-01> < <c 1994-01-01> time 4.8% fanout 2.00042 input 5.73038e+06 rows LINEITEM 1.1 rows(.L_EXTENDEDPRICE, .L_DISCOUNT) inlined L_ORDERKEY = .O_ORDERKEY L_RETURNFLAG = <c R> time 25% fanout 1 input 1.14632e+07 rows Precode: 0: temp := artm 1 - .L_DISCOUNT 4: temp := artm .L_EXTENDEDPRICE * temp 8: BReturn 0 CUSTOMER unq 1 rows (.C_NATIONKEY, .C_CUSTKEY) inlined C_CUSTKEY = k_.O_CUSTKEY hash partition+bloom by 39 (tmp)hash join merged always card 1 -> (.N_NAME) time 0.0023% fanout 1 input 1.14632e+07 rows Hash source 35 merged into ts 1 rows(.C_NATIONKEY) -> (.N_NAME) time 2.3% fanout 1 input 1.14632e+07 rows Stage 2 time 3.6% fanout 0 input 1.14632e+07 rows Sort (q_.C_CUSTKEY, .N_NAME) -> (temp) } time 0.6% fanout 3.88422e+06 input 1 rows group by read node (.C_CUSTKEY, .N_NAME, revenue)in each partition slice time 0.57% fanout 0 input 3.88422e+06 rows Sort (revenue) -> (.N_NAME, .C_CUSTKEY) } time 6.9e-06% fanout 20 input 1 rows top order by read (.N_NAME, revenue, .C_CUSTKEY) time 0.00036% fanout 1 input 20 rows CUSTOMER unq 1 rows (.C_PHONE, .C_NAME, .C_ACCTBAL, .C_ADDRESS, .C_COMMENT) inlined C_CUSTKEY = .C_CUSTKEY time 1.1e-06% fanout 0 input 20 rows Select (.C_CUSTKEY, .C_NAME, revenue, .C_ACCTBAL, .N_NAME, .C_ADDRESS, .C_PHONE, .C_COMMENT) } 2153 msec 2457% cpu, 1.71845e+07 rnd 1.67177e+08 seq 76.3221% same seg 21.1204% same pg </pre> </code> </blockquote> <p>The plan is by index, except for the lookup of <code>nation name</code> for the <code>customer</code>. The most selective condition is on <code>order date</code>, followed by the <code>returnflag</code> on <code>lineitem</code>. Getting the <code>customer</code> by index turns out to be better than by hash, even though almost all <code>customers</code> are hit. See the input cardinality above the first <code>customer</code> entry in the plan -- over 10M. The key point here is that only the <code>c_custkey</code> and <code>c_nationkey</code> get fetched, which saves a lot of time. In fact the <code>c_custkey</code> is needless since this is anyway equal to the <code>o_custkey</code>, but this makes little difference.</p> <p>One could argue that <code>customer</code> should be between <code>lineitem</code> and <code>orders</code> in <code>join</code> order. Doing this would lose the <code>ORDER BY</code> on <code>orders</code> and <code>lineitem</code>, but would prevent some <code>customer</code> rows from being hit twice for a single <code>order</code>. The difference would not be large, though. For a scale-out setting, one definitely wants to have <code>orders</code> and <code>lineitem</code> without <code>customer</code> in between if the former are partitioned on the same key.</p> <p>The <code>c_nationkey</code> is next translated into a <code>n_name</code> by hash, and there is a partitioned <code>GROUP BY</code> on <code>c_custkey</code>. The <code>GROUP BY</code> is partitioned because there are many different <code>c_custkey</code> values (155M for 100G scale). </p> <p>The most important trick is fetching all the many dependent columns of <code>c_custkey</code> only after the <code>TOP k ORDER BY</code>. The last access to <code>customer</code> in the plan does this and is only executed on 20 rows.</p> <p>Without the <code>TOP k</code> trick, the plan is identical, except that the dependent columns are fetched for nearly all <code>customers</code>. If this is done, the run time is 16s, which is bad enough to sink the whole score.</p> <p>There is another approach to the challenge of this query: If foreign keys are declared and enforced, the system will know that every <code>order</code> has an actually existing <code>customer</code> and that every <code>customer</code> has a <code>country</code>. If so, the whole <code>GROUP BY</code> and <code>TOP k</code> can be done without any reference to <code>customer</code>, which is a notch better still, at least for this query. In this implementation, we do not declare foreign keys, thus the database must check that the <code>customer</code> and its <code>country</code> in fact exist before doing the <code>GROUP BY</code>. This makes the late projection trick mandatory, but does save the expense of checking foreign keys on updates. In both cases, the optimizer must recognize that the columns to be fetched at the end (late projected) are functionally dependent on a grouping key (<code>c_custkey</code>).</p> <p>The late projection trick is generally useful, since almost all applications aside from bulk data export have some sort of limit on result set size. A column store especially benefits from this, since some columns of a row can be read without even coming near to other ones. A row store can also benefit from this in the form of decreased intermediate result size. This is especially good when returning long columns, such as text fields or blobs, on which there are most often no search conditions. If there are conditions of such, then these will most often be implemented via a special text index and not a scan.</p> <p align="center">* * * * *</p> <p>In the next installment we will have a look at the overall 100G single server performance. After this we will recap the tricks so far. Then it will be time to look at implications of scale out for performance and run at larger scales. After the relational ground has been covered, we can look at implications of schema-lastness, i.e., triples for this type of workload.</p> <p>So, while the most salient tricks have been at least briefly mentioned, we are far from having exhausted this most foundational of database topics.</p> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x142d4f68"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0xf3282f8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x20ffa438"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x25c97448"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x1e70db48"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x14592b38"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x144af558"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x17d64988"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x172c9668"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x1cbe1308"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x15391cb8"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac7a7d96d8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac30610368"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x16bd8ba8"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aabde5ee298"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x170500c8"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x1ae9f688"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aac04120018"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:10:19.696678-04:00
In Hoc Signo Vinces (part 11 of n): TPC-H Q2, Q10 - Late Projection
http://www.openlinksw.com:443/weblog/oerling/?date=2014-04-07#1793
2014-04-07T16:28:26Z
<p>Analytics is generally about making something small out of something large. This reduction is obtained by a <code>TOP k</code> operator (i.e., show only the 10 best by some metric) and/or by grouping and aggregation (i.e., for a set of items, show some attributes of these items and a sum, count, or other aggregate of dependent items for each).</p> <p>In this installment we will look at <i>late projection,</i> also sometimes known as <i>late materialization.</i> If many attributes are returned and there is a cutoff of some sort, then the query does not need to be concerned about attributes on which there are no conditions, except for fetching them at the last moment, only for the entities which in fact will be returned to the user.</p> <p>We look at <a href="http://www.tpc.org/tpch/" id="link-id0x2aac3c57d488">TPC-H</a> Q2 and Q10.</p> <h3>Q2:</h3> <blockquote> <code><pre> SELECT TOP 100 s_acctbal, s_name, n_name, p_partkey, p_mfgr, s_address, s_phone, s_comment FROM part, supplier, partsupp, nation, region WHERE p_partkey = ps_partkey AND s_suppkey = ps_suppkey AND p_size = 15 AND p_type LIKE '%BRASS' AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'EUROPE' AND ps_supplycost = ( SELECT MIN(ps_supplycost) FROM partsupp, supplier, nation, region WHERE p_partkey = ps_partkey AND s_suppkey = ps_suppkey AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'EUROPE' ) ORDER BY s_acctbal DESC, n_name, s_name, p_partkey </pre> </code> </blockquote> <p>The intent is to return information about <code>parts</code> and <code>suppliers</code>, such that the <code>part</code> is available from a <code>supplier</code> in Europe, and the <code>supplier</code> has the lowest <code>price</code> for the <code>part</code> among all European <code>suppliers</code>.</p> <h3>Q10:</h3> <blockquote> <code><pre> SELECT TOP 20 c_custkey, c_name, SUM(l_extendedprice * (1 - l_discount)) AS revenue, c_acctbal, n_name, c_address, c_phone, c_comment FROM customer, orders, lineitem, nation WHERE c_custkey = o_custkey AND l_orderkey = o_orderkey AND o_orderdate >= CAST ('1993-10-01' AS DATE) AND o_orderdate < DATEADD ('month', 3, CAST ('1993-10-01' AS DATE)) AND l_returnflag = 'R' AND c_nationkey = n_nationkey GROUP BY c_custkey, c_name, c_acctbal, c_phone, n_name, c_address, c_comment ORDER BY revenue DESC </pre> </code> </blockquote> <p>The intent is to list the <code>customers</code> who cause the greatest loss of revenue in a given quarter by returning <code>items</code> ordered in said quarter.</p> <p>We notice that both queries return many columns on which there are no conditions, and that both have a cap on returned rows. The difference is that in Q2 the major <code>ORDER BY</code> is on a grouping column, and in Q10 it is on the aggregate of the <code>GROUP BY</code>. Thus the <code>TOP k</code> trick discussed in <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac1e584578">the previous article</a> does apply to Q2 but not to Q10.</p> <p>The profile for Q2 follows:</p> <blockquote> <code><pre> { time 6.1e-05% fanout 1 input 1 rows time 1.1% fanout 1 input 1 rows { hash filler Subquery 27 { time 0.0012% fanout 1 input 1 rows REGION 1 rows(t10.R_REGIONKEY) R_NAME = <c EUROPE> time 0.00045% fanout 5 input 1 rows NATION 5 rows(t9.N_NATIONKEY) N_REGIONKEY = t10.R_REGIONKEY time 1.6% fanout 40107 input 5 rows SUPPLIER 4.2e+04 rows(t8.S_SUPPKEY) S_NATIONKEY = t9.N_NATIONKEY After code: 0: t8.S_SUPPKEY := := artm t8.S_SUPPKEY 4: BReturn 0 time 0.1% fanout 0 input 200535 rows Sort hf 49 (t8.S_SUPPKEY) } } time 0.0004% fanout 1 input 1 rows { fork time 21% fanout 79591 input 1 rows PART 8e+04 rows(.P_PARTKEY) P_TYPE LIKE <c %BRASS> LIKE <c > , P_SIZE = 15 time 44% fanout 0.591889 input 79591 rows Precode: 0: { time 0.083% fanout 1 input 79591 rows time 0.13% fanout 1 input 79591 rows { fork time 24% fanout 0.801912 input 79591 rows PARTSUPP 3.5 rows(.PS_SUPPKEY, .PS_SUPPLYCOST) inlined PS_PARTKEY = k_.P_PARTKEY hash partition+bloom by 62 (tmp)hash join merged always card 0.2 -> () time 1.3% fanout 0 input 63825 rows Hash source 49 merged into ts not partitionable 0.2 rows(.PS_SUPPKEY) -> () After code: 0: min min.PS_SUPPLYCOSTset no set_ctr 5: BReturn 0 } After code: 0: aggregate := := artm min 4: BReturn 0 time 0.19% fanout 0 input 79591 rows Subquery Select(aggregate) } 8: BReturn 0 PARTSUPP 5e-08 rows(.PS_SUPPKEY) inlined PS_PARTKEY = k_.P_PARTKEY PS_SUPPLYCOST = k_scalar time 5.9% fanout 0.247023 input 47109 rows SUPPLIER unq 0.9 rows (.S_ACCTBAL, .S_NATIONKEY, .S_NAME, .S_SUPPKEY) inlined S_SUPPKEY = .PS_SUPPKEY top k on S_ACCTBAL time 0.077% fanout 1 input 11637 rows NATION unq 1 rows (.N_REGIONKEY, .N_NAME) inlined N_NATIONKEY = .S_NATIONKEY time 0.051% fanout 1 input 11637 rows REGION unq 0.2 rows () inlined R_REGIONKEY = .N_REGIONKEY R_NAME = <c EUROPE> time 0.42% fanout 0 input 11637 rows Sort (.S_ACCTBAL, .N_NAME, .S_NAME, .P_PARTKEY) -> (.S_SUPPKEY) } time 0.0016% fanout 100 input 1 rows top order by read (.S_SUPPKEY, .P_PARTKEY, .N_NAME, .S_NAME, .S_ACCTBAL) time 0.02% fanout 1 input 100 rows PART unq 0.95 rows (.P_MFGR) inlined P_PARTKEY = .P_PARTKEY time 0.054% fanout 1 input 100 rows SUPPLIER unq 1 rows (.S_PHONE, .S_ADDRESS, .S_COMMENT) inlined S_SUPPKEY = k_.S_SUPPKEY time 6.7e-05% fanout 0 input 100 rows Select (.S_ACCTBAL, .S_NAME, .N_NAME, .P_PARTKEY, .P_MFGR, .S_ADDRESS, .S_PHONE, .S_COMMENT) } 128 msec 1007% cpu, 196992 rnd 2.53367e+07 seq 50.4135% same seg 45.3574% same pg </pre> </code> </blockquote> <p>The query starts with a scan looking for the qualifying <code>parts</code>. It then looks for the best <code>price</code> for each <code>part</code> from a European <code>supplier</code>. All the European <code>suppliers</code> have been previously put in a hash table by the hash filler subquery at the start of the plan. Thus, to find the minimum <code>price</code>, the query takes the <code>partsupp</code> for the <code>part</code> by index, and then eliminates all non-European <code>suppliers</code> by a selective hash join. After this, there is a second index lookup on <code>partsupp</code> where we look for the <code>part</code> and the <code>price</code> equal to the minimum <code>price</code> found earlier. These operations could in principle be merged, as the minimum <code>price</code> <code>partsupp</code> has already been seen. The gain would not be very large, though.</p> <p>Here we note that the cost model guesses that very few rows will survive the check of <code>ps_supplycost =</code> minimum <code>cost</code>. It does not know that the minimum is not just any value, but one of the values that do occur in the <code>ps_supplycost</code> column for the part. Because of this, the remainder of the plan is carried out by index, which is just as well. The point is that if very few rows of input are expected, it is not worthwhile to make a hash table for a hash join. The hash table made for the European <code>suppliers</code> could be reused here, maybe with some small gain. It would however need more columns, which might make it not worthwhile. We note that the major order with the <code>TOP k</code> is on the <code>supplier</code> <code>s_acctbal</code>, hence as soon as there are 100 <code>suppliers</code> found, one can add a restriction on the <code>s_acctbal</code> for subsequent ones.</p> <p>At the end of the plan, after the <code>TOP k ORDER BY</code> and the reading of the results, we have a separate index-based lookup for getting only the columns that are returned. We note that this is done on 100 rows whereas the previous operations are done on tens-of-thousands of rows. The <code>TOP k</code> restriction produces some benefit, but it is relatively late in the plan, and not many operations follow it.</p> <p>The plan is easily good enough, with only small space for improvement. Q2 is one of the fastest queries of the set. </p> <p>Let us now consider the execution of Q10:</p> <blockquote> <code><pre> { time 1.1e-06% fanout 1 input 1 rows time 4.4e-05% fanout 1 input 1 rows { hash filler time 1.6e-05% fanout 25 input 1 rows NATION 25 rows(.N_NATIONKEY, .N_NAME) time 6.7e-06% fanout 0 input 25 rows Sort hf 35 (.N_NATIONKEY) -> (.N_NAME) } time 1.5e-06% fanout 1 input 1 rows { fork time 2.4e-06% fanout 1 input 1 rows { fork time 13% fanout 5.73038e+06 input 1 rows ORDERS 5.1e+06 rows(.O_ORDERKEY, .O_CUSTKEY) O_ORDERDATE >= <c 1993-10-01> < <c 1994-01-01> time 4.8% fanout 2.00042 input 5.73038e+06 rows LINEITEM 1.1 rows(.L_EXTENDEDPRICE, .L_DISCOUNT) inlined L_ORDERKEY = .O_ORDERKEY L_RETURNFLAG = <c R> time 25% fanout 1 input 1.14632e+07 rows Precode: 0: temp := artm 1 - .L_DISCOUNT 4: temp := artm .L_EXTENDEDPRICE * temp 8: BReturn 0 CUSTOMER unq 1 rows (.C_NATIONKEY, .C_CUSTKEY) inlined C_CUSTKEY = k_.O_CUSTKEY hash partition+bloom by 39 (tmp)hash join merged always card 1 -> (.N_NAME) time 0.0023% fanout 1 input 1.14632e+07 rows Hash source 35 merged into ts 1 rows(.C_NATIONKEY) -> (.N_NAME) time 2.3% fanout 1 input 1.14632e+07 rows Stage 2 time 3.6% fanout 0 input 1.14632e+07 rows Sort (q_.C_CUSTKEY, .N_NAME) -> (temp) } time 0.6% fanout 3.88422e+06 input 1 rows group by read node (.C_CUSTKEY, .N_NAME, revenue)in each partition slice time 0.57% fanout 0 input 3.88422e+06 rows Sort (revenue) -> (.N_NAME, .C_CUSTKEY) } time 6.9e-06% fanout 20 input 1 rows top order by read (.N_NAME, revenue, .C_CUSTKEY) time 0.00036% fanout 1 input 20 rows CUSTOMER unq 1 rows (.C_PHONE, .C_NAME, .C_ACCTBAL, .C_ADDRESS, .C_COMMENT) inlined C_CUSTKEY = .C_CUSTKEY time 1.1e-06% fanout 0 input 20 rows Select (.C_CUSTKEY, .C_NAME, revenue, .C_ACCTBAL, .N_NAME, .C_ADDRESS, .C_PHONE, .C_COMMENT) } 2153 msec 2457% cpu, 1.71845e+07 rnd 1.67177e+08 seq 76.3221% same seg 21.1204% same pg </pre> </code> </blockquote> <p>The plan is by index, except for the lookup of <code>nation name</code> for the <code>customer</code>. The most selective condition is on <code>order date</code>, followed by the <code>returnflag</code> on <code>lineitem</code>. Getting the <code>customer</code> by index turns out to be better than by hash, even though almost all <code>customers</code> are hit. See the input cardinality above the first <code>customer</code> entry in the plan -- over 10M. The key point here is that only the <code>c_custkey</code> and <code>c_nationkey</code> get fetched, which saves a lot of time. In fact the <code>c_custkey</code> is needless since this is anyway equal to the <code>o_custkey</code>, but this makes little difference.</p> <p>One could argue that <code>customer</code> should be between <code>lineitem</code> and <code>orders</code> in <code>join</code> order. Doing this would lose the <code>ORDER BY</code> on <code>orders</code> and <code>lineitem</code>, but would prevent some <code>customer</code> rows from being hit twice for a single <code>order</code>. The difference would not be large, though. For a scale-out setting, one definitely wants to have <code>orders</code> and <code>lineitem</code> without <code>customer</code> in between if the former are partitioned on the same key.</p> <p>The <code>c_nationkey</code> is next translated into a <code>n_name</code> by hash, and there is a partitioned <code>GROUP BY</code> on <code>c_custkey</code>. The <code>GROUP BY</code> is partitioned because there are many different <code>c_custkey</code> values (155M for 100G scale). </p> <p>The most important trick is fetching all the many dependent columns of <code>c_custkey</code> only after the <code>TOP k ORDER BY</code>. The last access to <code>customer</code> in the plan does this and is only executed on 20 rows.</p> <p>Without the <code>TOP k</code> trick, the plan is identical, except that the dependent columns are fetched for nearly all <code>customers</code>. If this is done, the run time is 16s, which is bad enough to sink the whole score.</p> <p>There is another approach to the challenge of this query: If foreign keys are declared and enforced, the system will know that every <code>order</code> has an actually existing <code>customer</code> and that every <code>customer</code> has a <code>country</code>. If so, the whole <code>GROUP BY</code> and <code>TOP k</code> can be done without any reference to <code>customer</code>, which is a notch better still, at least for this query. In this implementation, we do not declare foreign keys, thus the database must check that the <code>customer</code> and its <code>country</code> in fact exist before doing the <code>GROUP BY</code>. This makes the late projection trick mandatory, but does save the expense of checking foreign keys on updates. In both cases, the optimizer must recognize that the columns to be fetched at the end (late projected) are functionally dependent on a grouping key (<code>c_custkey</code>).</p> <p>The late projection trick is generally useful, since almost all applications aside from bulk data export have some sort of limit on result set size. A column store especially benefits from this, since some columns of a row can be read without even coming near to other ones. A row store can also benefit from this in the form of decreased intermediate result size. This is especially good when returning long columns, such as text fields or blobs, on which there are most often no search conditions. If there are conditions of such, then these will most often be implemented via a special text index and not a scan.</p> <p align="center">* * * * *</p> <p>In the next installment we will have a look at the overall 100G single server performance. After this we will recap the tricks so far. Then it will be time to look at implications of scale out for performance and run at larger scales. After the relational ground has been covered, we can look at implications of schema-lastness, i.e., triples for this type of workload.</p> <p>So, while the most salient tricks have been at least briefly mentioned, we are far from having exhausted this most foundational of database topics.</p> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x142d4f68"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0xf3282f8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x20ffa438"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x25c97448"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x1e70db48"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x14592b38"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x144af558"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x17d64988"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x172c9668"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x1cbe1308"> In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games </a> </li> <li> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac2adc6e88"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac67c7d218"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x11055068"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x1927ecf8"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aabe8549578"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x94fc248"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aac83875dd8"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aab6ef1afd8"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:07:07.568642-04:00
OpenPHACTS in Vienna
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-03-31#1792
2014-03-31T15:49:54Z
<p> <a href="http://www.linkedin.com/pub/hugh-williams/0/1a0/559" id="link-id0x1d043058">Hugh Williams</a> and I (<a href="http://www.linkedin.com/in/orrierling" id="link-id0x1ce71de8">Orri Erling</a>) went to the <a href="http://www.OpenPHACTS.org/" id="link-id0xcc549f8">Open PHACTS</a> <a href="http://www.OpenPHACTS.org/news-and-events/news-archive/2014/344-7th-steering-committee-meeting-held" id="link-id0x19e23b68">Steering Committee meeting</a> in Vienna last week. I am a great fan of Open PHACTS; the meetings are fun, with a great team spirit, and there is always something new to learn.</p> <p> <a href="http://www.few.vu.nl/~pgroth/" id="link-id0x2257d818">Paul Groth</a> gave a talk about the stellar success of the the initial term of Open PHACTS.</p> <ul> <li>Three releases of platform and data</li> <li>18 applications</li> <li> <a href="http://www.openphactsfoundation.org/" id="link-id0x1e33a478">Open PHACTS Foundation</a> for sustainable exploitation and further development of the platform</li> <li>Superb culture of collaboration <ul> <li>great team spirit</li> <li>great output from distributed organization</li> <li>lots of face-to-face time</li> <li>example to every other big collaborative project</li> </ul> </li> </ul> <p>"The reincarnation of <a href="http://dbpedia.org/page/Steve_Jobs" id="link-id0x1e8009c8">Steve Jobs</a>," commented someone from the audience. "Except I am a nice guy," retorted Paul. </p> <p>Commented one attendee, "The semantic web…., I just was in Boston at a semantic web meeting – so nerdy, something to make you walk out of the room… so it is a definite victory for Open PHACTS and why not also semantic web, that something based on these principles actually works."</p> <p>It is a win anyhow, so I did not say anything at the meeting. So I will say something here, where I have more space as the message bears repeating.</p> <p>We share part of the perception, so we hardly ever say "semantic web." The word is "linked data," and it means flexible schema and global identifiers. <i>Flexible schema</i> means that everything does not have to be modeled upfront. <i>Global identifiers</i> means that data, when transferred out of its silo of origin, remains interpretable and self-describing, so you can mix it with other data without things getting confused. <i>"Desiloization"</i> is a wonderful new word for describing this.</p> <p>This ties right into <a href="http://datafairport.org/" id="link-id0x20a450f8">FAIRport</a> and FAIR data: <a href="http://dbpedia.org/page/Findability" id="link-id0x1f32bb68">Findable</a>, <a href="http://dbpedia.org/page/Accessibility" id="link-id0x1eb7ca28">Accessible</a>, <a href="http://dbpedia.org/page/Interoperability" id="link-id0x1ea03c68">Interoperable</a>, <a href="http://dbpedia.org/page/Reusability" id="link-id0x20c78888">Reusable</a>. <a href="http://dbpedia.org/page/Barend_Mons" id="link-id0xb2ca558">Barend Mons</a> talked a lot about this: <i>open</i> just means downloadable; <i>fair</i> means something you can do science with. Barend’s take is that <a href="http://dbpedia.org/page/Resource_Description_Framework" id="link-id0x20767568">RDF</a> with a <a href="http://dbpedia.org/page/Uniform_resource_identifier" id="link-id0x19586538">URI</a> for everything is the super wire format for exchanging data. When you process it, you will diversely cook it, so an RDF store is one destination but not the only possibility. It has been said before: there is a range of choices between storing triples verbatim, and making application specific extractions, including ones with a schema, whether graph DB or relational.</p> <p> <a href="http://nanopub.org/" id="link-id0x156cfef8">Nanopublications</a> are also moving ahead. <a href="http://ch.linkedin.com/pub/christine-chichester/3/9a0/85b" id="link-id0x15928e48">Christine Chichester</a> told me about pending publications involving Open PHACTS nanopublictions about post-translation modification of proteins and their expression in different tissues. So there are nanopublications out there and they can be joined, just as intended. Victory of e-science and data integration.</p> <p>The Open PHACTS project is now officially extended for another two-year term, bringing the total duration to five years. The Open PHACTS Foundation exists as a legal entity and has its first members. This is meant to be a non-profit industry association for sharing of pre-competitive data and services around these between players in the pharma space, in industry as well as academia. There are press releases to follow in due time.</p> <p>I am looking forward to more Open PHACTS. From the <a href="http://www.openlinksw.com/" id="link-id0x1270cb58">OpenLink</a> and <a href="http://virtuoso.openlinksw.com/" id="link-id0x204c2dd8">Virtuoso</a> side, there are directly relevant developments that will enter production in the next few months, including query caching discussed <a href="http://www.openlinksw.com/weblog/oerling/?id=1771" id="link-id0x1de0fcb8">earlier on this blog</a>, as well as running on the TPC-H tuned analytics branch for overall better query optimization. Adaptive schema is something of evident value to Open PHACTS, as much of the integrated data comes from relational sources, so is regular enough. Therefore taking advantage of this for storage cannot hurt. We will see this still within the scope of the project extension.</p> <p>Otherwise, more cooperation in formulating the queries for the business questions will also help.</p> <p>All in all, Open PHACTS is the celebrated beauty queen of all the <a href="http://dbpedia.org/page/Innovative_Medicines_Initiative" id="link-id0x20ae59d8">Innovative Medicine Initiative</a>, it would seem. Superbly connected, unparalleled logo cloud, actually working and useful data integration, delivering on time on all in fact very complex business questions.</p>
2014-03-31T11:49:54.000001-04:00
OpenPHACTS in Vienna
http://www.openlinksw.com:443/weblog/oerling/?date=2014-03-31#1791
2014-03-31T15:49:44Z
<p> <a href="http://www.linkedin.com/pub/hugh-williams/0/1a0/559" id="link-id0x1d043058">Hugh Williams</a> and I (<a href="http://www.linkedin.com/in/orrierling" id="link-id0x1ce71de8">Orri Erling</a>) went to the <a href="http://www.OpenPHACTS.org/" id="link-id0xcc549f8">Open PHACTS</a> <a href="http://www.OpenPHACTS.org/news-and-events/news-archive/2014/344-7th-steering-committee-meeting-held" id="link-id0x19e23b68">Steering Committee meeting</a> in Vienna last week. I am a great fan of Open PHACTS; the meetings are fun, with a great team spirit, and there is always something new to learn.</p> <p> <a href="http://www.few.vu.nl/~pgroth/" id="link-id0x2257d818">Paul Groth</a> gave a talk about the stellar success of the the initial term of Open PHACTS.</p> <ul> <li>Three releases of platform and data</li> <li>18 applications</li> <li> <a href="http://www.openphactsfoundation.org/" id="link-id0x1e33a478">Open PHACTS Foundation</a> for sustainable exploitation and further development of the platform</li> <li>Superb culture of collaboration <ul> <li>great team spirit</li> <li>great output from distributed organization</li> <li>lots of face-to-face time</li> <li>example to every other big collaborative project</li> </ul> </li> </ul> <p>"The reincarnation of <a href="http://dbpedia.org/page/Steve_Jobs" id="link-id0x1e8009c8">Steve Jobs</a>," commented someone from the audience. "Except I am a nice guy," retorted Paul. </p> <p>Commented one attendee, "The semantic web…., I just was in Boston at a semantic web meeting – so nerdy, something to make you walk out of the room… so it is a definite victory for Open PHACTS and why not also semantic web, that something based on these principles actually works."</p> <p>It is a win anyhow, so I did not say anything at the meeting. So I will say something here, where I have more space as the message bears repeating.</p> <p>We share part of the perception, so we hardly ever say "semantic web." The word is "linked data," and it means flexible schema and global identifiers. <i>Flexible schema</i> means that everything does not have to be modeled upfront. <i>Global identifiers</i> means that data, when transferred out of its silo of origin, remains interpretable and self-describing, so you can mix it with other data without things getting confused. <i>"Desiloization"</i> is a wonderful new word for describing this.</p> <p>This ties right into <a href="http://datafairport.org/" id="link-id0x20a450f8">FAIRport</a> and FAIR data: <a href="http://dbpedia.org/page/Findability" id="link-id0x1f32bb68">Findable</a>, <a href="http://dbpedia.org/page/Accessibility" id="link-id0x1eb7ca28">Accessible</a>, <a href="http://dbpedia.org/page/Interoperability" id="link-id0x1ea03c68">Interoperable</a>, <a href="http://dbpedia.org/page/Reusability" id="link-id0x20c78888">Reusable</a>. <a href="http://dbpedia.org/page/Barend_Mons" id="link-id0xb2ca558">Barend Mons</a> talked a lot about this: <i>open</i> just means downloadable; <i>fair</i> means something you can do science with. Barend’s take is that <a href="http://dbpedia.org/page/Resource_Description_Framework" id="link-id0x20767568">RDF</a> with a <a href="http://dbpedia.org/page/Uniform_resource_identifier" id="link-id0x19586538">URI</a> for everything is the super wire format for exchanging data. When you process it, you will diversely cook it, so an RDF store is one destination but not the only possibility. It has been said before: there is a range of choices between storing triples verbatim, and making application specific extractions, including ones with a schema, whether graph DB or relational.</p> <p> <a href="http://nanopub.org/" id="link-id0x156cfef8">Nanopublications</a> are also moving ahead. <a href="http://ch.linkedin.com/pub/christine-chichester/3/9a0/85b" id="link-id0x15928e48">Christine Chichester</a> told me about pending publications involving Open PHACTS nanopublictions about post-translation modification of proteins and their expression in different tissues. So there are nanopublications out there and they can be joined, just as intended. Victory of e-science and data integration.</p> <p>The Open PHACTS project is now officially extended for another two-year term, bringing the total duration to five years. The Open PHACTS Foundation exists as a legal entity and has its first members. This is meant to be a non-profit industry association for sharing of pre-competitive data and services around these between players in the pharma space, in industry as well as academia. There are press releases to follow in due time.</p> <p>I am looking forward to more Open PHACTS. From the <a href="http://www.openlinksw.com/" id="link-id0x1270cb58">OpenLink</a> and <a href="http://virtuoso.openlinksw.com/" id="link-id0x204c2dd8">Virtuoso</a> side, there are directly relevant developments that will enter production in the next few months, including query caching discussed <a href="http://www.openlinksw.com/weblog/oerling/?id=1771" id="link-id0x1de0fcb8">earlier on this blog</a>, as well as running on the TPC-H tuned analytics branch for overall better query optimization. Adaptive schema is something of evident value to Open PHACTS, as much of the integrated data comes from relational sources, so is regular enough. Therefore taking advantage of this for storage cannot hurt. We will see this still within the scope of the project extension.</p> <p>Otherwise, more cooperation in formulating the queries for the business questions will also help.</p> <p>All in all, Open PHACTS is the celebrated beauty queen of all the <a href="http://dbpedia.org/page/Innovative_Medicines_Initiative" id="link-id0x20ae59d8">Innovative Medicine Initiative</a>, it would seem. Superbly connected, unparalleled logo cloud, actually working and useful data integration, delivering on time on all in fact very complex business questions.</p>
2014-03-31T11:49:44-04:00
In Hoc Signo Vinces (part 10 of n): TPC-H Q9, Q17, Q20 - Predicate Games
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-03-20#1790
2014-03-20T20:03:34Z
<p> <a href="http://www.tpc.org/tpch/" id="link-id0x24011b28">TPC-H</a> is a hash join game. The rules do allow indices, but maintaining these takes time, and indices will quickly result in non-local access patterns. Indices also take space. Besides, somebody must know what indices to create, which is not obvious. Thus, it is best if a <a href="http://dbpedia.org/resource/Business_intelligence" id="link-id0xf2d8b08">BI</a> data warehouse works without.</p> <p>Once you go to <a href="http://dbpedia.org/resource/Hash_join" id="link-id0x135b9dc8">hash join</a>, one side of the join will be materialized, which takes space, which <i>ipso facto</i> is bad. So, the predicate games are about moving conditions so that the hash table made for the hash join will be as small as possible. Only items that may in fact be retrieved should be put in the hash table. If you know that the query deals with shipments of green <code>parts</code>, putting <code>lineitems</code> of <code>parts</code> that are not green in a hash table makes no sense since only green ones are being looked for.</p> <p>So, let's consider Q9. The query is:</p> <blockquote> <code><pre> SELECT nation, o_year, SUM(amount) AS sum_profit FROM ( SELECT n_name AS nation, EXTRACT ( YEAR FROM o_orderdate ) AS o_year, l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity AS amount FROM part, supplier, lineitem, partsupp, orders, nation WHERE s_suppkey = l_suppkey AND ps_suppkey = l_suppkey AND ps_partkey = l_partkey AND p_partkey = l_partkey AND o_orderkey = l_orderkey AND s_nationkey = n_nationkey AND p_name like '%green%' ) AS profit GROUP BY nation, o_year ORDER BY nation, o_year DESC ; </pre> </code> </blockquote> <p>The intent is to calculate profit from the sale of a type of <code>part</code>, broken down by <code>year</code> and supplier <code>nation</code>. All <code>orders</code>, <code>lineitems</code>, <code>partsupps</code>, and <code>suppliers</code> involving the <code>parts</code> of interest are visited. This is one of the longest running of the queries. The query is restricted by <code>part</code> only, and the condition selects 1/17 of all <code>parts</code>.</p> <p>The execution plan is below. First the plan builds hash tables of all <code>nations</code> and <code>suppliers</code>. We expect to do frequent lookups, thus making a hash is faster than using the index. <code>Partsupp</code> is the 3rd largest table in the database. This has a primary key of <code>ps_partkey, ps_suppkey</code>, referenced by the compound foreign key <code>l_partkey, l_suppkey</code> in <code>lineitem</code>. This could be accessed by index, but we expect to hit each <code>partsupp</code> row multiple times, hence hash is better. We further note that only <code>partsupp</code> rows where the <code>part</code> satisfies the condition will contribute to the result. Thus we import the join with <code>part</code> into the hash build. The <code>ps_partkey</code> is not directly joined to <code>p_partkey</code>, but rather the system must understand that this follows from <code>l_partkey = ps_partkey</code> and <code>l_partkey = p_partkey</code>. In this way, the hash table is 1/17th of the size it would otherwise be, which is a crucial gain.</p> <p>Looking further into the plan, we note a scan of <code>lineitem</code> followed by a hash join with <code>part</code>. Restricting the build of the <code>partsupp</code> hash would have the same effect, hence <code>part</code> is here used twice while it occurs only once in the query. This is deliberate, since the selective hash join with <code>part</code> restricts <code>lineitem</code> faster than the more complex hash join with a 2 part key <code>(l_partkey, l_suppkey)</code>. Both joins perform the identical restriction, but doing the <code>part</code> first is faster since this becomes a single-key, invisible hash join, merged into the <code>lineitem</code> scan, done before even accessing the <code>l_suppkey</code> and other columns.</p> <blockquote> <code><pre> { time 3.9e-06% fanout 1 input 1 rows time 4.7e-05% fanout 1 input 1 rows { hash filler time 3.6e-05% fanout 25 input 1 rows NATION 25 rows(.N_NATIONKEY, nation) time 8.8e-06% fanout 0 input 25 rows Sort hf 35 (.N_NATIONKEY) -> (nation) } time 0.16% fanout 1 input 1 rows { hash filler time 0.011% fanout 1e+06 input 1 rows SUPPLIER 1e+06 rows(.S_SUPPKEY, .S_NATIONKEY) time 0.03% fanout 0 input 1e+06 rows Sort hf 49 (.S_SUPPKEY) -> (.S_NATIONKEY) } time 0.57% fanout 1 input 1 rows { hash filler Subquery 58 { time 1.6% fanout 1.17076e+06 input 1 rows PART 1.2e+06 rows(t1.P_PARTKEY) P_NAME LIKE <c %green%> LIKE <c > time 1.1% fanout 4 input 1.17076e+06 rows PARTSUPP 3.9 rows(t4.PS_SUPPKEY, t4.PS_PARTKEY, t4.PS_SUPPLYCOST) inlined PS_PARTKEY = t1.P_PARTKEY After code: 0: t4.PS_SUPPKEY := := artm t4.PS_SUPPKEY 4: t4.PS_PARTKEY := := artm t4.PS_PARTKEY 8: t1.P_PARTKEY := := artm t1.P_PARTKEY 12: t4.PS_SUPPLYCOST := := artm t4.PS_SUPPLYCOST 16: BReturn 0 time 0.33% fanout 0 input 4.68305e+06 rows Sort hf 82 (t4.PS_SUPPKEY, t4.PS_PARTKEY) -> (t1.P_PARTKEY, t4.PS_SUPPLYCOST) } } time 0.18% fanout 1 input 1 rows { hash filler time 1.6% fanout 1.17076e+06 input 1 rows PART 1.2e+06 rows(.P_PARTKEY) P_NAME LIKE <c %green%> LIKE <c > time 0.017% fanout 0 input 1.17076e+06 rows Sort hf 101 (.P_PARTKEY) } time 5.1e-06% fanout 1 input 1 rows { fork time 4.1e-06% fanout 1 input 1 rows { fork time 59% fanout 3.51125e+07 input 1 rows LINEITEM 6e+08 rows(.L_PARTKEY, .L_ORDERKEY, .L_SUPPKEY, .L_EXTENDEDPRICE, .L_DISCOUNT, .L_QUANTITY) hash partition+bloom by 108 (tmp)hash join merged always card 0.058 -> () hash partition+bloom by 56 (tmp)hash join merged always card 1 -> (.S_NATIONKEY) time 0.18% fanout 1 input 3.51125e+07 rows Precode: 0: temp := artm 1 - .L_DISCOUNT 4: temp := artm .L_EXTENDEDPRICE * temp 8: BReturn 0 Hash source 101 merged into ts 0.058 rows(.L_PARTKEY) -> () time 17% fanout 1 input 3.51125e+07 rows Hash source 82 0.057 rows(.L_SUPPKEY, .L_PARTKEY) -> ( <none> , .PS_SUPPLYCOST) time 6.2% fanout 1 input 3.51125e+07 rows Precode: 0: temp := artm .PS_SUPPLYCOST * .L_QUANTITY 4: temp := artm temp - temp 8: BReturn 0 ORDERS unq 1 rows (.O_ORDERDATE) inlined O_ORDERKEY = k_.L_ORDERKEY time 0.0055% fanout 1 input 3.51125e+07 rows Hash source 49 merged into ts 1 rows(k_.L_SUPPKEY) -> (.S_NATIONKEY) time 3.5% fanout 1 input 3.51125e+07 rows Hash source 35 1 rows(k_.S_NATIONKEY) -> (nation) time 8.8% fanout 0 input 3.51125e+07 rows Precode: 0: o_year := Call year (.O_ORDERDATE) 5: BReturn 0 Sort (nation, o_year) -> (temp) } time 4.7e-05% fanout 175 input 1 rows group by read node (nation, o_year, sum_profit) time 0.00028% fanout 0 input 175 rows Sort (nation, o_year) -> (sum_profit) } time 2.2e-05% fanout 175 input 1 rows Key from temp (nation, o_year, sum_profit) time 1.6e-06% fanout 0 input 175 rows Select (nation, o_year, sum_profit) } 6114 msec 1855% cpu, 3.62624e+07 rnd 6.44384e+08 seq 99.6068% same seg 0.357328% same pg </pre> </code> </blockquote> <p>6.1s is a good score for this query. When executing the same in 5 parallel invocations, the fastest ends in 13.7s and the slowest in 27.6s. For five concurrent executions, the peak transient memory utilization is 4.7 GB for the hash tables, which is very reasonable.</p> <p align="center">* * * * *</p> <p>Let us next consider Q17.</p> <blockquote> <code><pre> SELECT SUM(l_extendedprice) / 7.0 AS avg_yearly FROM lineitem, part WHERE p_partkey = l_partkey AND p_brand = 'Brand#23' AND p_container = 'MED BOX' AND l_quantity < ( SELECT 2e-1 * AVG(l_quantity) FROM lineitem WHERE l_partkey = p_partkey ) </pre> </code> </blockquote> <p>Deceptively simple? This calculates the total value of small <code>orders</code> (below 1/5 of average quantity for the <code>part</code>) for all <code>parts</code> of a given <code>brand</code> with a specific <code>container</code>. </p> <p>If there is an index on <code>l_partkey</code>, the plan is easy enough: Take the <code>parts</code>, look up the average <code>quantity</code> for each, then recheck <code>lineitem</code> and add up the small <code>lineitems</code>. This takes about 1s. But we do not want indices for this workload.</p> <p>If we made a hash from <code>l_partkey</code> to <code>l_quantity</code> for all <code>lineitems</code>, we could run out of space, and this would take so long the race would be automatically lost on this point alone. The trick is to import the restriction on <code>l_partkey</code> into the hash build. This gives us a plan that does a scan of <code>lineitem</code> twice, doing a very selective hash join (few <code>parts</code>). There is a lookup for the <code>average</code> for each <code>lineitem</code> with the <code>part</code>. The <code>average</code> is calculated potentially several times.</p> <p>The below plan is workable but better is possible: We notice that the very selective join need be done just once; it is cheaper to remember the result than to do it twice, and the result is not large. The other trick is that the correlated subquery can be rewritten as </p> <blockquote> <code><pre> SELECT ... FROM lineitem, part, ( SELECT l_partkey, 0.2 * AVG (l_quantity) AS qty FROM lineitem, part ... ) f WHERE l_partkey = f.l_partkey ... </pre> </code> </blockquote> <p>In this form, one can put the entire derived table <code>f</code> on the build side of a hash join. In this way, the <code>average</code> is never done more than once per <code>part</code>. </p> <blockquote> <code><pre> { time 7.9e-06% fanout 1 input 1 rows time 0.0031% fanout 1 input 1 rows { hash filler time 0.27% fanout 20031 input 1 rows PART 2e+04 rows(.P_PARTKEY) P_BRAND = <c Brand#23> , P_CONTAINER = <c MED BOX> time 0.00047% fanout 0 input 20031 rows Sort hf 34 (.P_PARTKEY) } time 0.1% fanout 1 input 1 rows { hash filler Subquery 40 { time 46% fanout 600982 input 1 rows LINEITEM 6e+08 rows(t4.L_PARTKEY, t4.L_QUANTITY) hash partition+bloom by 38 (tmp)hash join merged always card 0.001 -> () time 0.0042% fanout 1 input 600982 rows Hash source 34 merged into ts not partitionable 0.001 rows(t4.L_PARTKEY) -> () After code: 0: t4.L_PARTKEY := := artm t4.L_PARTKEY 4: t4.L_QUANTITY := := artm t4.L_QUANTITY 8: BReturn 0 time 0.059% fanout 0 input 600982 rows Sort hf 62 (t4.L_PARTKEY) -> (t4.L_QUANTITY) } } time 6.8e-05% fanout 1 input 1 rows { fork time 46% fanout 600982 input 1 rows LINEITEM 6e+08 rows(.L_PARTKEY, .L_QUANTITY, .L_EXTENDEDPRICE) hash partition+bloom by 38 (tmp)hash join merged always card 0.00052 -> () time 0.00021% fanout 1 input 600982 rows Hash source 34 merged into ts 0.00052 rows(.L_PARTKEY) -> () Precode: 0: .P_PARTKEY := := artm .L_PARTKEY 4: BReturn 0 END Node After test: 0: { time 0.038% fanout 1 input 600982 rows time 0.17% fanout 1 input 600982 rows { fork time 6.8% fanout 0 input 600982 rows Hash source 62 not partitionable 0.03 rows(k_.P_PARTKEY) -> (.L_QUANTITY) After code: 0: sum sum.L_QUANTITYset no set_ctr 5: sum count 1 set no set_ctr 10: BReturn 0 } After code: 0: temp := artm sum / count 4: temp := artm 0.2 * temp 8: aggregate := := artm temp 12: BReturn 0 time 0.042% fanout 0 input 600982 rows Subquery Select(aggregate) } 8: if (.L_QUANTITY < scalar) then 12 else 13 unkn 13 12: BReturn 1 13: BReturn 0 After code: 0: sum sum.L_EXTENDEDPRICE 5: BReturn 0 } After code: 0: avg_yearly := artm sum / 7 4: BReturn 0 time 4.6e-06% fanout 0 input 1 rows Select (avg_yearly) } 2695 msec 1996% cpu, 3 rnd 1.18242e+09 seq 0% same seg 0% same pg </pre> </code> </blockquote> <p>2.7s is tolerable, but if this drags down the overall score by too much, we know that a 2+x improvement is readily available. Playing the rest of the tricks would result in the hash plan almost catching up with the 1s execution time of the index-based plan. </p> <p align="center">* * * * *</p> <p>Q20 is not very long-running, but it is maybe the hardest to optimize of the lot. But as usual, failure to recognize its most salient traps will automatically lose the race, so pay attention.</p> <blockquote> <code><pre> SELECT TOP 100 s_name, s_address FROM supplier, nation WHERE s_suppkey IN ( SELECT ps_suppkey FROM partsupp WHERE ps_partkey IN ( SELECT p_partkey FROM part WHERE p_name LIKE 'forest%' ) AND ps_availqty > ( SELECT 0.5 * SUM(l_quantity) FROM lineitem WHERE l_partkey = ps_partkey AND l_suppkey = ps_suppkey AND l_shipdate >= CAST ('1994-01-01' AS DATE) AND l_shipdate < DATEADD ('year', 1, CAST ('1994-01-01' AS DATE)) ) ) AND s_nationkey = n_nationkey AND n_name = 'CANADA' ORDER BY s_name </pre> </code> </blockquote> <p>This identifies <code>suppliers</code> that have <code>parts</code> in stock in excess of half a year's shipments of said <code>part</code>.</p> <p>The use of <code>IN</code> to denote a <code>join</code> is the first catch. The second is joining to <code>lineitem</code> by hash without building an overly large hash table. We know that <code>IN</code> becomes <code>EXISTS</code> which in turn can become a <code>join</code> as follows:</p> <blockquote> <code><pre> SELECT l_suppkey FROM lineitem WHERE l_partkey IN ( SELECT p_partkey FROM part WHERE p_name LIKE 'forest%' ) ; </pre> </code> </blockquote> <p>-- is --</p> <blockquote> <code><pre> SELECT l_suppkey FROM lineitem WHERE EXISTS ( SELECT p_partkey FROM part WHERE p_partkey = l_partkey AND p_name LIKE 'forest%') ; </pre> </code> </blockquote> <p>-- is --</p> <blockquote> <code><pre> SELECT l_suppkey FROM lineitem, ( SELECT DISTINCT p_partkey FROM part WHERE p_name LIKE 'forest%') f WHERE l_partkey = f.p_partkey ; </pre> </code> </blockquote> <p>But since <code>p_partkey</code> is unique, the <code>DISTINCT</code> drops off and we have.</p> <blockquote> <code><pre> SELECT l_suppkey FROM lineitem, part WHERE p_name LIKE 'forest% AND l_partkey = f.p_partkey ; </pre> </code> </blockquote> <p>You see, the innermost <code>IN</code> with the <code>ps_partkey</code> goes through all these changes, and just becomes a <code>join</code>. The outermost <code>IN</code> stays as a distinct derived table, since <code>ps_suppkey</code> is not unique, and the meaning of <code>IN</code> is not to return a given <code>supplier</code> more than once.</p> <p>The derived table is flattened and the <code>DISTINCT</code> is done partitioned; hence the stage node in front of the distinct. A <code>DISTINCT</code> can be multithreaded, if each thread gets a specific subset of all the keys. The stage node is an exchange of tuples between several threads. Each thread then does a <code>TOP k</code> sort. The <code>TOP k</code> trick <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x1ecc67c8">we saw in Q18</a> is used, but does not contribute much here. </p> <blockquote> <code><pre> { time 8.2e-06% fanout 1 input 1 rows time 0.00017% fanout 1 input 1 rows { hash filler time 6.1e-05% fanout 1 input 1 rows NATION 1 rows(.N_NATIONKEY) N_NAME = <c CANADA> time 1.2e-05% fanout 0 input 1 rows Sort hf 34 (.N_NATIONKEY) } time 0.073% fanout 1 input 1 rows { hash filler time 4.1% fanout 240672 input 1 rows PART 2.4e+05 rows(t74.P_PARTKEY) P_NAME LIKE <c forest%> LIKE <c > time 0.011% fanout 0 input 240672 rows Sort hf 47 (t74.P_PARTKEY) } time 0.69% fanout 1 input 1 rows { hash filler Subquery 56 { time 42% fanout 1.09657e+06 input 1 rows LINEITEM 9.1e+07 rows(t76.L_PARTKEY, t76.L_SUPPKEY, t76.L_QUANTITY) L_SHIPDATE >= <c 1994-01-01> < <c 1995-01-01> hash partition+bloom by 54 (tmp)hash join merged always card 0.012 -> () time 0.022% fanout 1 input 1.09657e+06 rows Hash source 47 merged into ts not partitionable 0.012 rows(t76.L_PARTKEY) -> () After code: 0: t76.L_PARTKEY := := artm t76.L_PARTKEY 4: t76.L_SUPPKEY := := artm t76.L_SUPPKEY 8: t76.L_QUANTITY := := artm t76.L_QUANTITY 12: BReturn 0 time 0.22% fanout 0 input 1.09657e+06 rows Sort hf 80 (t76.L_PARTKEY, t76.L_SUPPKEY) -> (t76.L_QUANTITY) } } time 2.1e-05% fanout 1 input 1 rows time 3.2e-05% fanout 1 input 1 rows { fork time 5.3% fanout 240672 input 1 rows PART 2.4e+05 rows(t6.P_PARTKEY) P_NAME LIKE <c forest%> LIKE <c > time 1.9% fanout 4 input 240672 rows PARTSUPP 1.2 rows(t4.PS_AVAILQTY, t4.PS_PARTKEY, t4.PS_SUPPKEY) inlined PS_PARTKEY = t6.P_PARTKEY time 16% fanout 0.680447 input 962688 rows END Node After test: 0: { time 0.08% fanout 1 input 962688 rows time 9.4% fanout 1 input 962688 rows { fork time 3.6% fanout 0 input 962688 rows Hash source 80 0.013 rows(k_t4.PS_PARTKEY, k_t4.PS_SUPPKEY) -> (t8.L_QUANTITY) After code: 0: sum sumt8.L_QUANTITYset no set_ctr 5: BReturn 0 } After code: 0: temp := artm 0.5 * sum 4: aggregate := := artm temp 8: BReturn 0 time 0.85% fanout 0 input 962688 rows Subquery Select(aggregate) } 8: if (t4.PS_AVAILQTY > scalar) then 12 else 13 unkn 13 12: BReturn 1 13: BReturn 0 time 1% fanout 1 input 655058 rows Stage 2 time 0.071% fanout 1 input 655058 rows Distinct (q_t4.PS_SUPPKEY) After code: 0: PS_SUPPKEY := := artm t4.PS_SUPPKEY 4: BReturn 0 time 0.016% fanout 1 input 655058 rows Subquery Select(PS_SUPPKEY) time 3.2% fanout 0.0112845 input 655058 rows SUPPLIER unq 0.075 rows (.S_NAME, .S_NATIONKEY, .S_ADDRESS) inlined S_SUPPKEY = PS_SUPPKEY hash partition+bloom by 38 (tmp)hash join merged always card 0.04 -> () top k on S_NAME time 0.0012% fanout 1 input 7392 rows Hash source 34 merged into ts 0.04 rows(.S_NATIONKEY) -> () time 0.074% fanout 0 input 7392 rows Sort (.S_NAME) -> (.S_ADDRESS) } time 0.00013% fanout 100 input 1 rows top order by read (.S_NAME, .S_ADDRESS) time 5e-06% fanout 0 input 100 rows Select (.S_NAME, .S_ADDRESS) } 1777 msec 1355% cpu, 894483 rnd 6.39422e+08 seq 79.1214% same seg 19.3093% same pg </pre> </code> </blockquote> <p>1.8s is sufficient, and in the ballpark with <a href="http://dbpedia.org/resource/Vectorwise" id="link-id0x2329ad38">VectorWise</a>. Some further gain is possible, as the <code>lineitem</code> hash table can also be restricted by <code>supplier</code>; after all, only 1/25 of all <code>suppliers</code> are in the end considered. Further simplifications are possible. Another 20% of time could be saved. The tricks are however quite complex and specific, and there are easier gains to be had -- for example, in reusing intermediates in Q17 and Q15. </p> <p>The next installment will discuss late projection and some miscellaneous tricks not mentioned so far. After this, we are ready to take an initial look at the performance of the system as a whole.</p> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x142d4f68"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0xf3282f8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x20ffa438"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x25c97448"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x1e70db48"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x14592b38"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x144af558"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x17d64988"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x172c9668"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> In Hoc Signo Vinces (part 10): TPC-H: TPC-H Q9, Q17, Q20 - Predicate Games<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac1d054778"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac2b837ad8"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac0b666eb8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac342ee378"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x53991f8"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aabe97e35b8"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x14e2b818"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x1b0ecf18"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aac051f7f68"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:10:25.712710-04:00
In Hoc Signo Vinces (part 10 of n): TPC-H Q9, Q17, Q20 - Predicate Games
http://www.openlinksw.com:443/weblog/oerling/?date=2014-03-20#1789
2014-03-20T20:01:54Z
<p> <a href="http://www.tpc.org/tpch/" id="link-id0x24011b28">TPC-H</a> is a hash join game. The rules do allow indices, but maintaining these takes time, and indices will quickly result in non-local access patterns. Indices also take space. Besides, somebody must know what indices to create, which is not obvious. Thus, it is best if a <a href="http://dbpedia.org/resource/Business_intelligence" id="link-id0xf2d8b08">BI</a> data warehouse works without.</p> <p>Once you go to <a href="http://dbpedia.org/resource/Hash_join" id="link-id0x135b9dc8">hash join</a>, one side of the join will be materialized, which takes space, which <i>ipso facto</i> is bad. So, the predicate games are about moving conditions so that the hash table made for the hash join will be as small as possible. Only items that may in fact be retrieved should be put in the hash table. If you know that the query deals with shipments of green <code>parts</code>, putting <code>lineitems</code> of <code>parts</code> that are not green in a hash table makes no sense since only green ones are being looked for.</p> <p>So, let's consider Q9. The query is:</p> <blockquote> <code><pre> SELECT nation, o_year, SUM(amount) AS sum_profit FROM ( SELECT n_name AS nation, EXTRACT ( YEAR FROM o_orderdate ) AS o_year, l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity AS amount FROM part, supplier, lineitem, partsupp, orders, nation WHERE s_suppkey = l_suppkey AND ps_suppkey = l_suppkey AND ps_partkey = l_partkey AND p_partkey = l_partkey AND o_orderkey = l_orderkey AND s_nationkey = n_nationkey AND p_name like '%green%' ) AS profit GROUP BY nation, o_year ORDER BY nation, o_year DESC ; </pre> </code> </blockquote> <p>The intent is to calculate profit from the sale of a type of <code>part</code>, broken down by <code>year</code> and supplier <code>nation</code>. All <code>orders</code>, <code>lineitems</code>, <code>partsupps</code>, and <code>suppliers</code> involving the <code>parts</code> of interest are visited. This is one of the longest running of the queries. The query is restricted by <code>part</code> only, and the condition selects 1/17 of all <code>parts</code>.</p> <p>The execution plan is below. First the plan builds hash tables of all <code>nations</code> and <code>suppliers</code>. We expect to do frequent lookups, thus making a hash is faster than using the index. <code>Partsupp</code> is the 3rd largest table in the database. This has a primary key of <code>ps_partkey, ps_suppkey</code>, referenced by the compound foreign key <code>l_partkey, l_suppkey</code> in <code>lineitem</code>. This could be accessed by index, but we expect to hit each <code>partsupp</code> row multiple times, hence hash is better. We further note that only <code>partsupp</code> rows where the <code>part</code> satisfies the condition will contribute to the result. Thus we import the join with <code>part</code> into the hash build. The <code>ps_partkey</code> is not directly joined to <code>p_partkey</code>, but rather the system must understand that this follows from <code>l_partkey = ps_partkey</code> and <code>l_partkey = p_partkey</code>. In this way, the hash table is 1/17th of the size it would otherwise be, which is a crucial gain.</p> <p>Looking further into the plan, we note a scan of <code>lineitem</code> followed by a hash join with <code>part</code>. Restricting the build of the <code>partsupp</code> hash would have the same effect, hence <code>part</code> is here used twice while it occurs only once in the query. This is deliberate, since the selective hash join with <code>part</code> restricts <code>lineitem</code> faster than the more complex hash join with a 2 part key <code>(l_partkey, l_suppkey)</code>. Both joins perform the identical restriction, but doing the <code>part</code> first is faster since this becomes a single-key, invisible hash join, merged into the <code>lineitem</code> scan, done before even accessing the <code>l_suppkey</code> and other columns.</p> <blockquote> <code><pre> { time 3.9e-06% fanout 1 input 1 rows time 4.7e-05% fanout 1 input 1 rows { hash filler time 3.6e-05% fanout 25 input 1 rows NATION 25 rows(.N_NATIONKEY, nation) time 8.8e-06% fanout 0 input 25 rows Sort hf 35 (.N_NATIONKEY) -> (nation) } time 0.16% fanout 1 input 1 rows { hash filler time 0.011% fanout 1e+06 input 1 rows SUPPLIER 1e+06 rows(.S_SUPPKEY, .S_NATIONKEY) time 0.03% fanout 0 input 1e+06 rows Sort hf 49 (.S_SUPPKEY) -> (.S_NATIONKEY) } time 0.57% fanout 1 input 1 rows { hash filler Subquery 58 { time 1.6% fanout 1.17076e+06 input 1 rows PART 1.2e+06 rows(t1.P_PARTKEY) P_NAME LIKE <c %green%> LIKE <c > time 1.1% fanout 4 input 1.17076e+06 rows PARTSUPP 3.9 rows(t4.PS_SUPPKEY, t4.PS_PARTKEY, t4.PS_SUPPLYCOST) inlined PS_PARTKEY = t1.P_PARTKEY After code: 0: t4.PS_SUPPKEY := := artm t4.PS_SUPPKEY 4: t4.PS_PARTKEY := := artm t4.PS_PARTKEY 8: t1.P_PARTKEY := := artm t1.P_PARTKEY 12: t4.PS_SUPPLYCOST := := artm t4.PS_SUPPLYCOST 16: BReturn 0 time 0.33% fanout 0 input 4.68305e+06 rows Sort hf 82 (t4.PS_SUPPKEY, t4.PS_PARTKEY) -> (t1.P_PARTKEY, t4.PS_SUPPLYCOST) } } time 0.18% fanout 1 input 1 rows { hash filler time 1.6% fanout 1.17076e+06 input 1 rows PART 1.2e+06 rows(.P_PARTKEY) P_NAME LIKE <c %green%> LIKE <c > time 0.017% fanout 0 input 1.17076e+06 rows Sort hf 101 (.P_PARTKEY) } time 5.1e-06% fanout 1 input 1 rows { fork time 4.1e-06% fanout 1 input 1 rows { fork time 59% fanout 3.51125e+07 input 1 rows LINEITEM 6e+08 rows(.L_PARTKEY, .L_ORDERKEY, .L_SUPPKEY, .L_EXTENDEDPRICE, .L_DISCOUNT, .L_QUANTITY) hash partition+bloom by 108 (tmp)hash join merged always card 0.058 -> () hash partition+bloom by 56 (tmp)hash join merged always card 1 -> (.S_NATIONKEY) time 0.18% fanout 1 input 3.51125e+07 rows Precode: 0: temp := artm 1 - .L_DISCOUNT 4: temp := artm .L_EXTENDEDPRICE * temp 8: BReturn 0 Hash source 101 merged into ts 0.058 rows(.L_PARTKEY) -> () time 17% fanout 1 input 3.51125e+07 rows Hash source 82 0.057 rows(.L_SUPPKEY, .L_PARTKEY) -> ( <none> , .PS_SUPPLYCOST) time 6.2% fanout 1 input 3.51125e+07 rows Precode: 0: temp := artm .PS_SUPPLYCOST * .L_QUANTITY 4: temp := artm temp - temp 8: BReturn 0 ORDERS unq 1 rows (.O_ORDERDATE) inlined O_ORDERKEY = k_.L_ORDERKEY time 0.0055% fanout 1 input 3.51125e+07 rows Hash source 49 merged into ts 1 rows(k_.L_SUPPKEY) -> (.S_NATIONKEY) time 3.5% fanout 1 input 3.51125e+07 rows Hash source 35 1 rows(k_.S_NATIONKEY) -> (nation) time 8.8% fanout 0 input 3.51125e+07 rows Precode: 0: o_year := Call year (.O_ORDERDATE) 5: BReturn 0 Sort (nation, o_year) -> (temp) } time 4.7e-05% fanout 175 input 1 rows group by read node (nation, o_year, sum_profit) time 0.00028% fanout 0 input 175 rows Sort (nation, o_year) -> (sum_profit) } time 2.2e-05% fanout 175 input 1 rows Key from temp (nation, o_year, sum_profit) time 1.6e-06% fanout 0 input 175 rows Select (nation, o_year, sum_profit) } 6114 msec 1855% cpu, 3.62624e+07 rnd 6.44384e+08 seq 99.6068% same seg 0.357328% same pg </pre> </code> </blockquote> <p>6.1s is a good score for this query. When executing the same in 5 parallel invocations, the fastest ends in 13.7s and the slowest in 27.6s. For five concurrent executions, the peak transient memory utilization is 4.7 GB for the hash tables, which is very reasonable.</p> <p align="center">* * * * *</p> <p>Let us next consider Q17.</p> <blockquote> <code><pre> SELECT SUM(l_extendedprice) / 7.0 AS avg_yearly FROM lineitem, part WHERE p_partkey = l_partkey AND p_brand = 'Brand#23' AND p_container = 'MED BOX' AND l_quantity < ( SELECT 2e-1 * AVG(l_quantity) FROM lineitem WHERE l_partkey = p_partkey ) </pre> </code> </blockquote> <p>Deceptively simple? This calculates the total value of small <code>orders</code> (below 1/5 of average quantity for the <code>part</code>) for all <code>parts</code> of a given <code>brand</code> with a specific <code>container</code>. </p> <p>If there is an index on <code>l_partkey</code>, the plan is easy enough: Take the <code>parts</code>, look up the average <code>quantity</code> for each, then recheck <code>lineitem</code> and add up the small <code>lineitems</code>. This takes about 1s. But we do not want indices for this workload.</p> <p>If we made a hash from <code>l_partkey</code> to <code>l_quantity</code> for all <code>lineitems</code>, we could run out of space, and this would take so long the race would be automatically lost on this point alone. The trick is to import the restriction on <code>l_partkey</code> into the hash build. This gives us a plan that does a scan of <code>lineitem</code> twice, doing a very selective hash join (few <code>parts</code>). There is a lookup for the <code>average</code> for each <code>lineitem</code> with the <code>part</code>. The <code>average</code> is calculated potentially several times.</p> <p>The below plan is workable but better is possible: We notice that the very selective join need be done just once; it is cheaper to remember the result than to do it twice, and the result is not large. The other trick is that the correlated subquery can be rewritten as </p> <blockquote> <code><pre> SELECT ... FROM lineitem, part, ( SELECT l_partkey, 0.2 * AVG (l_quantity) AS qty FROM lineitem, part ... ) f WHERE l_partkey = f.l_partkey ... </pre> </code> </blockquote> <p>In this form, one can put the entire derived table <code>f</code> on the build side of a hash join. In this way, the <code>average</code> is never done more than once per <code>part</code>. </p> <blockquote> <code><pre> { time 7.9e-06% fanout 1 input 1 rows time 0.0031% fanout 1 input 1 rows { hash filler time 0.27% fanout 20031 input 1 rows PART 2e+04 rows(.P_PARTKEY) P_BRAND = <c Brand#23> , P_CONTAINER = <c MED BOX> time 0.00047% fanout 0 input 20031 rows Sort hf 34 (.P_PARTKEY) } time 0.1% fanout 1 input 1 rows { hash filler Subquery 40 { time 46% fanout 600982 input 1 rows LINEITEM 6e+08 rows(t4.L_PARTKEY, t4.L_QUANTITY) hash partition+bloom by 38 (tmp)hash join merged always card 0.001 -> () time 0.0042% fanout 1 input 600982 rows Hash source 34 merged into ts not partitionable 0.001 rows(t4.L_PARTKEY) -> () After code: 0: t4.L_PARTKEY := := artm t4.L_PARTKEY 4: t4.L_QUANTITY := := artm t4.L_QUANTITY 8: BReturn 0 time 0.059% fanout 0 input 600982 rows Sort hf 62 (t4.L_PARTKEY) -> (t4.L_QUANTITY) } } time 6.8e-05% fanout 1 input 1 rows { fork time 46% fanout 600982 input 1 rows LINEITEM 6e+08 rows(.L_PARTKEY, .L_QUANTITY, .L_EXTENDEDPRICE) hash partition+bloom by 38 (tmp)hash join merged always card 0.00052 -> () time 0.00021% fanout 1 input 600982 rows Hash source 34 merged into ts 0.00052 rows(.L_PARTKEY) -> () Precode: 0: .P_PARTKEY := := artm .L_PARTKEY 4: BReturn 0 END Node After test: 0: { time 0.038% fanout 1 input 600982 rows time 0.17% fanout 1 input 600982 rows { fork time 6.8% fanout 0 input 600982 rows Hash source 62 not partitionable 0.03 rows(k_.P_PARTKEY) -> (.L_QUANTITY) After code: 0: sum sum.L_QUANTITYset no set_ctr 5: sum count 1 set no set_ctr 10: BReturn 0 } After code: 0: temp := artm sum / count 4: temp := artm 0.2 * temp 8: aggregate := := artm temp 12: BReturn 0 time 0.042% fanout 0 input 600982 rows Subquery Select(aggregate) } 8: if (.L_QUANTITY < scalar) then 12 else 13 unkn 13 12: BReturn 1 13: BReturn 0 After code: 0: sum sum.L_EXTENDEDPRICE 5: BReturn 0 } After code: 0: avg_yearly := artm sum / 7 4: BReturn 0 time 4.6e-06% fanout 0 input 1 rows Select (avg_yearly) } 2695 msec 1996% cpu, 3 rnd 1.18242e+09 seq 0% same seg 0% same pg </pre> </code> </blockquote> <p>2.7s is tolerable, but if this drags down the overall score by too much, we know that a 2+x improvement is readily available. Playing the rest of the tricks would result in the hash plan almost catching up with the 1s execution time of the index-based plan. </p> <p align="center">* * * * *</p> <p>Q20 is not very long-running, but it is maybe the hardest to optimize of the lot. But as usual, failure to recognize its most salient traps will automatically lose the race, so pay attention.</p> <blockquote> <code><pre> SELECT TOP 100 s_name, s_address FROM supplier, nation WHERE s_suppkey IN ( SELECT ps_suppkey FROM partsupp WHERE ps_partkey IN ( SELECT p_partkey FROM part WHERE p_name LIKE 'forest%' ) AND ps_availqty > ( SELECT 0.5 * SUM(l_quantity) FROM lineitem WHERE l_partkey = ps_partkey AND l_suppkey = ps_suppkey AND l_shipdate >= CAST ('1994-01-01' AS DATE) AND l_shipdate < DATEADD ('year', 1, CAST ('1994-01-01' AS DATE)) ) ) AND s_nationkey = n_nationkey AND n_name = 'CANADA' ORDER BY s_name </pre> </code> </blockquote> <p>This identifies <code>suppliers</code> that have <code>parts</code> in stock in excess of half a year's shipments of said <code>part</code>.</p> <p>The use of <code>IN</code> to denote a <code>join</code> is the first catch. The second is joining to <code>lineitem</code> by hash without building an overly large hash table. We know that <code>IN</code> becomes <code>EXISTS</code> which in turn can become a <code>join</code> as follows:</p> <blockquote> <code><pre> SELECT l_suppkey FROM lineitem WHERE l_partkey IN ( SELECT p_partkey FROM part WHERE p_name LIKE 'forest%' ) ; </pre> </code> </blockquote> <p>-- is --</p> <blockquote> <code><pre> SELECT l_suppkey FROM lineitem WHERE EXISTS ( SELECT p_partkey FROM part WHERE p_partkey = l_partkey AND p_name LIKE 'forest%') ; </pre> </code> </blockquote> <p>-- is --</p> <blockquote> <code><pre> SELECT l_suppkey FROM lineitem, ( SELECT DISTINCT p_partkey FROM part WHERE p_name LIKE 'forest%') f WHERE l_partkey = f.p_partkey ; </pre> </code> </blockquote> <p>But since <code>p_partkey</code> is unique, the <code>DISTINCT</code> drops off and we have.</p> <blockquote> <code><pre> SELECT l_suppkey FROM lineitem, part WHERE p_name LIKE 'forest% AND l_partkey = f.p_partkey ; </pre> </code> </blockquote> <p>You see, the innermost <code>IN</code> with the <code>ps_partkey</code> goes through all these changes, and just becomes a <code>join</code>. The outermost <code>IN</code> stays as a distinct derived table, since <code>ps_suppkey</code> is not unique, and the meaning of <code>IN</code> is not to return a given <code>supplier</code> more than once.</p> <p>The derived table is flattened and the <code>DISTINCT</code> is done partitioned; hence the stage node in front of the distinct. A <code>DISTINCT</code> can be multithreaded, if each thread gets a specific subset of all the keys. The stage node is an exchange of tuples between several threads. Each thread then does a <code>TOP k</code> sort. The <code>TOP k</code> trick <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x1ecc67c8">we saw in Q18</a> is used, but does not contribute much here. </p> <blockquote> <code><pre> { time 8.2e-06% fanout 1 input 1 rows time 0.00017% fanout 1 input 1 rows { hash filler time 6.1e-05% fanout 1 input 1 rows NATION 1 rows(.N_NATIONKEY) N_NAME = <c CANADA> time 1.2e-05% fanout 0 input 1 rows Sort hf 34 (.N_NATIONKEY) } time 0.073% fanout 1 input 1 rows { hash filler time 4.1% fanout 240672 input 1 rows PART 2.4e+05 rows(t74.P_PARTKEY) P_NAME LIKE <c forest%> LIKE <c > time 0.011% fanout 0 input 240672 rows Sort hf 47 (t74.P_PARTKEY) } time 0.69% fanout 1 input 1 rows { hash filler Subquery 56 { time 42% fanout 1.09657e+06 input 1 rows LINEITEM 9.1e+07 rows(t76.L_PARTKEY, t76.L_SUPPKEY, t76.L_QUANTITY) L_SHIPDATE >= <c 1994-01-01> < <c 1995-01-01> hash partition+bloom by 54 (tmp)hash join merged always card 0.012 -> () time 0.022% fanout 1 input 1.09657e+06 rows Hash source 47 merged into ts not partitionable 0.012 rows(t76.L_PARTKEY) -> () After code: 0: t76.L_PARTKEY := := artm t76.L_PARTKEY 4: t76.L_SUPPKEY := := artm t76.L_SUPPKEY 8: t76.L_QUANTITY := := artm t76.L_QUANTITY 12: BReturn 0 time 0.22% fanout 0 input 1.09657e+06 rows Sort hf 80 (t76.L_PARTKEY, t76.L_SUPPKEY) -> (t76.L_QUANTITY) } } time 2.1e-05% fanout 1 input 1 rows time 3.2e-05% fanout 1 input 1 rows { fork time 5.3% fanout 240672 input 1 rows PART 2.4e+05 rows(t6.P_PARTKEY) P_NAME LIKE <c forest%> LIKE <c > time 1.9% fanout 4 input 240672 rows PARTSUPP 1.2 rows(t4.PS_AVAILQTY, t4.PS_PARTKEY, t4.PS_SUPPKEY) inlined PS_PARTKEY = t6.P_PARTKEY time 16% fanout 0.680447 input 962688 rows END Node After test: 0: { time 0.08% fanout 1 input 962688 rows time 9.4% fanout 1 input 962688 rows { fork time 3.6% fanout 0 input 962688 rows Hash source 80 0.013 rows(k_t4.PS_PARTKEY, k_t4.PS_SUPPKEY) -> (t8.L_QUANTITY) After code: 0: sum sumt8.L_QUANTITYset no set_ctr 5: BReturn 0 } After code: 0: temp := artm 0.5 * sum 4: aggregate := := artm temp 8: BReturn 0 time 0.85% fanout 0 input 962688 rows Subquery Select(aggregate) } 8: if (t4.PS_AVAILQTY > scalar) then 12 else 13 unkn 13 12: BReturn 1 13: BReturn 0 time 1% fanout 1 input 655058 rows Stage 2 time 0.071% fanout 1 input 655058 rows Distinct (q_t4.PS_SUPPKEY) After code: 0: PS_SUPPKEY := := artm t4.PS_SUPPKEY 4: BReturn 0 time 0.016% fanout 1 input 655058 rows Subquery Select(PS_SUPPKEY) time 3.2% fanout 0.0112845 input 655058 rows SUPPLIER unq 0.075 rows (.S_NAME, .S_NATIONKEY, .S_ADDRESS) inlined S_SUPPKEY = PS_SUPPKEY hash partition+bloom by 38 (tmp)hash join merged always card 0.04 -> () top k on S_NAME time 0.0012% fanout 1 input 7392 rows Hash source 34 merged into ts 0.04 rows(.S_NATIONKEY) -> () time 0.074% fanout 0 input 7392 rows Sort (.S_NAME) -> (.S_ADDRESS) } time 0.00013% fanout 100 input 1 rows top order by read (.S_NAME, .S_ADDRESS) time 5e-06% fanout 0 input 100 rows Select (.S_NAME, .S_ADDRESS) } 1777 msec 1355% cpu, 894483 rnd 6.39422e+08 seq 79.1214% same seg 19.3093% same pg </pre> </code> </blockquote> <p>1.8s is sufficient, and in the ballpark with <a href="http://dbpedia.org/resource/Vectorwise" id="link-id0x2329ad38">VectorWise</a>. Some further gain is possible, as the <code>lineitem</code> hash table can also be restricted by <code>supplier</code>; after all, only 1/25 of all <code>suppliers</code> are in the end considered. Further simplifications are possible. Another 20% of time could be saved. The tricks are however quite complex and specific, and there are easier gains to be had -- for example, in reusing intermediates in Q17 and Q15. </p> <p>The next installment will discuss late projection and some miscellaneous tricks not mentioned so far. After this, we are ready to take an initial look at the performance of the system as a whole.</p> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x142d4f68"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0xf3282f8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x20ffa438"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x25c97448"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x1e70db48"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x14592b38"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x144af558"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x17d64988"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x172c9668"> In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K</a> </li> <li> In Hoc Signo Vinces (part 10): TPC-H: TPC-H Q9, Q17, Q20 - Predicate Games<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aab6beee348"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aabf9e9bf78"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac748db938"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0xb56b548"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x18a88838"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aabe842fd18"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aabebf52f68"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aabf677a8d8"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aab6df52238"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:07:14.784424-04:00
Linked Geospatial Data 2014 Workshop, Part 4: GeoKnow, London, Brussels, The Message
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-03-18#1788
2014-03-18T14:45:49Z
<p>Last Friday (2014-03-14) I gave a talk about <a href="http://geoknow.eu" id="link-id0x2aabc3ce6228">GeoKnow</a> at the <a href="http://www.copernicus.eu/pages-secondaires/events/event-detail/?tx_julleevents_pi1%5BshowUid%5D=533&cHash=6b72b992422ee703f83e7c6de4c7e697" id="link-id0x2aabc3ce6288">EC Copernicus Big Data workshop</a>. This was a trial run for more streamlined messaging. I have, aside the practice of geekcraft, occupied myself with questions of communication these last weeks.</p> <p>The clear take-home from London and Brussels alike is that these events have full days and 4 or more talks an hour. It is not quite TV commercial spots yet but it is going in this direction.</p> <p>If you say something complex, little will get across unless the audience already knows what you will be saying.</p> <p>I had a set of slides from <a href="http://jens-lehmann.org/" id="link-id0x2aabc3ce6368">Jens Lehmann</a>, the GeoKnow project coordinator, for whom I was standing in. Now these are a fine rendition of the description of work. What is wrong with partners, work packages, objectives, etc? Nothing, except everybody has them.</p> <p>I recall the old story about the journalist and the Zen master: The Zen master repeatedly advises the reporter to cut the story in half. We get the same from PR professionals, "If it is short, they have at least thought about what should go in there," said one recently, talking of pitches and messages. The other advice was to use pictures. And to have a personal dimension to it.</p> <p>Enter "Ms. Globe" and "Mr. Cube". <a href="http://nl.linkedin.com/pub/frans-knibbe/5a/93/409" id="link-id0x2aabc3ce6488">Frans Knibbe</a> of <a href="http://www.geodan.nl/" id="link-id0x2aabc3ce6568">Geodan</a> gave the <a href="http://www.w3.org/2014/03/lgd/" id="link-id0x2aabc3ce65c8">Linked Geospatial Data 2014</a> workshop's most memorable talk entitled <a href="http://www.w3.org/2014/03/lgd/Frans" id="link-id0x2aabc3ce6628">"Linked Data and Geoinformatics - a love story" (pdf)</a> about the excitement and the pitfalls of the burgeoning courtship of Ms. Globe (geoinformatics) and Mr. Cube (semantic technology). They get to talking, later Ms. Globe thinks to herself... "Desiloisazation, explicit semantics, integrated metadata..." Mr. Cube, young upstart now approaching a more experienced and sophisticated lady, dreams of finally making an entry into adult society, "critical mass, global scope, relevant applications..." There is a vibration in the air.</p> <p>So, with <a href="http://nl.linkedin.com/pub/frans-knibbe/5a/93/409" id="link-id0x2aabc3ce66a8">Frans Knibbe</a>'s gracious permission <a href="http://www.slideshare.net/OpenLinkSoftware/geo-knowcopernicusbigdataworkshopbrusselsmarch2014-32446207" id="link-id0x2aabc3ce6648">I borrowed the storyline and some of the pictures</a>. </p> <div style="margin: 0 auto; text-align: center"> <iframe align="middle" src="http://www.slideshare.net/slideshow/embed_code/32446207" width="427" height="356" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px 1px 0; margin-bottom:5px; max-width: 100%;" allowfullscreen="allowfullscreen"> </iframe> </div>We ought to make a series of cartoons about the couple. There will be twists and turns in the story to come. <p>Mr. Cube is not Ms. Globe's first lover, though; there is also rich and worldly Mr. Table. How will Mr. Cube prove himself? The eternal question... Well, not by moping around, not by wise-cracking about semantics, no. By boldly setting out upon a journey to fetch the Golden Fleece from beyond the crashing rocks. "Column store, vectored execution, scale out, data clustering, adaptive schema..." he affirms, with growing confidence.</p> <p>This is where the story stands, right now. <a href="http://virtuoso.openlinksw.com" id="link-id0x2aabc3ce6768">Virtuoso</a> run circles around <a href="http://postgis.net/" id="link-id0x2aabc3ce6808">PostGIS</a> doing aggregations and lookups on geometries in a map-scrolling scenario (GeoKnow's <a href="https://github.com/GeoKnow/GeoBenchLab" id="link-id0x2aabc3ce6868">GeoBenchLab</a>). Virtuoso <a href="http://en.wikipedia.org/wiki/SPARQL" id="link-id0x2aabdd806b48">SPARQL</a> outperforms PostGIS <a href="http://en.wikipedia.org/wiki/SQL" id="link-id0x2aabdd806ba8">SQL</a> against planet-scale <a href="http://www.openstreetmap.org/" id="link-id0x2aabdd806c08">OpenStreetMap</a>; Virtuoso SQL goes 5-10x faster still.</p> <p>Mr Cube is fast on the draw, but still some corners can be smoothed out.</p> <p>Later in GeoKnow, there will be still more speed but also near parity between SQL and SPARQL via taking advantage of data regularity in guiding physical storage. If it is big, it is bound to have repeating structure.</p> <p>The love story grows more real by the day. To be consummated still within GeoKnow.</p> <p>Talking of databases has the great advantage that this has been a performance game from the start. There are few people who need convincing about the desirability of performance, as this also makes for lower cost and more flexibility on the application side.</p> <p>But this is not all there is to it.</p> <p>In Brussels, the public was about E-science (Earth observation). In science, it is understood that qualitative aspects can be even more crucial. I told the story about an E-science-oriented workshop I attended in America years ago. The practitioners, from high energy physics to life sciences to climate, had invariably come across the need for self-description of data and for schema-last. This was essentially never provided by <a href="http://en.wikipedia.org/wiki/Resource_Description_Framework" id="link-id0x2aabdd806fe8">RDF</a>, except for some life science cases. Rather, we had one-off schemes, ranging from key-value pairs to putting the table name in a column of the same table to preserve the origin across data export.</p> <p>Explicit semantics and integrated metadata are important, Ms. Globe knows, but she cannot sacrifice operational capacity for this. So it is more than a <a href="http://kingsley.idehen.net/describe/?url=http%3A%2F%2Fkingsley.idehen.net%2FDAV%2Fhome%2Fkidehen%2FPublic%2FLinked%2520Data%2520Documents%2FGlossaryOfTerms.ttl%23DBMS" id="link-id0x2aabdd8070e8">DBMS</a> or even data model choice -- there must be a solid tool chain for data integration and visualization. GeoKnow provides many tools in this space.</p> <p>Some of these, such as the <a href="http://svn.aksw.org/papers/2011/WWW_LIMES/public.pdf" id="link-id0x2aabdd807148">LIMES entity matching framework (pdf)</a> are probably close to the best there is. For other parts, the SQL-based products with hundreds of person years invested in user interaction are simply unbeatable.</p> <p>In these cases, the world can continue to talk SQL. If the regular part of the data is in fact tables already, so much the better. You connect to Virtuoso via SQL, just like to PostGIS or <a href="http://www.oracle.com/technetwork/database/options/spatialandgraph/overview/index.html" id="link-id0x2aabdd807168">Oracle Spatial</a>, and talk <a href="http://en.wikipedia.org/wiki/Simple_Features" id="link-id0x2aabdd807268">SQL MM</a>. The triples, in the sense of flexible annotation and integrated metadata, stay there; you just do not see them if you do not want them.</p> <p> There are possibilities all right. In the coming months I will showcase some of the progress, starting with a detailed look at the OpenStreetMap experiments we have made in GeoKnow.</p> <h3>Linked Geospatial Data 2014 Workshop posts:</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1781" id="link-id0x2aabdd807428">Linked Geospatial Data 2014 Workshop, Part 1: Web Services or SPARQL Modeling?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1782" id="link-id0x2aabdd8074e8">Linked Geospatial Data 2014 Workshop, Part 2: Is SPARQL Slow?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1783" id="link-id0x2aabdd8075a8">Linked Geospatial Data 2014 Workshop, Part 3: The Stellar Reach of OKFN</a> </li> <li> Linked Geospatial Data 2014 Workshop, Part 4: GeoKnow, London, Brussels, The Message<i> (this post)</i> </li> </ul>
2014-03-18T10:52:29-04:00
Linked Geospatial Data 2014 Workshop, Part 3: The Stellar Reach of OKFN
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-03-18#1787
2014-03-18T14:45:43Z
<p>The <a href="http://okfn.org/" id="link-id0x2aabc38976f8">Open Knowledge Foundation</a> (OKFN) held a <a href="http://www.eventbrite.co.uk/e/london-open-data-meetup-march-2014-tickets-10574052275" id="link-id0x2aabdd08bb18">London Open Data Meetup</a> in the evening of the first day of the <a href="http://www.w3.org/2014/03/lgd/" id="link-id0x127f1648">Linked Geospatial Data 2014</a> workshop. The event was, as they themselves put it, at the amazing open concept office of OKFN at the <a href="http://blog.okfn.org/2011/06/23/okfns-london-office-opens-at-the-centre-for-creative-collaboration/" id="link-id0x2aabc2ffb288">Center for Creative Collaboration</a> in Central London. What could sound cooler? True, OKFN threw a good party, with ever engaging and charismatic founder <a href="http://okfn.org/about/team/#rufus-pollock-8212-founder" id="link-id0x2aabc2bbdd68">Rufus Pollock</a> presiding. <a href="http://philarcher.org/" id="link-id0x2aabc37081d8">Phil Archer</a> noted, only half in jest, that OKFN was so influential, visible, had the ear of government and public alike, etc., that it put <a href="http://www.w3.org/" id="link-id0x2aabc3cabb58">W3C</a> to shame.</p> <p>Now, OKFN is a party in the <a href="http://lod2.ec" id="link-id0x2aabdd5e9ab8">LOD2</a> FP7 project, so I have over the years met people from there on and off. In LOD2, OKFN is praised to the skies for its visibility and influence and outreach and sometimes, in passing, critiqued for not publishing enough <a href="http://dbpedia.org/page/Resource_Description_Framework" id="link-id0x2aabdd60e5b8">RDF</a>, let alone five star linked data.</p> <p>As it happens, <a href="http://dbpedia.org/page/Comma-separated_values" id="link-id0x128d93d8">CSV</a> rules, and even the W3C will, it appears, undertake to standardize a CSV-to-RDF mapping. As far as I am concerned, as long as there is no alignment of identifiers or vocabulary, whether a thing is CSV or exactly equivalent RDF, there is no great difference, except that CSV is smaller and loads into Excel.</p> <p>For OKFN, which has a mission of opening data, insisting on any particular format would just hinder the cause.</p> <p>What do we learn from this? OKFN is praised not only for government relations but also for developer friendliness. Lobbying for open data is something I can understand, but how do you do developer relations? This is not like talking to customers, where the customer wants to do something and it is usually possible to give some kind of advice or recommendation on how they can use our technology for the purpose.</p> <p>Are <a href="http://dbpedia.org/page/JSON" id="link-id0x2aabc1987188">JSON</a> and <a href="https://www.mongodb.org/" id="link-id0x2aabc10ba4e8">Mongo DB</a> the key? A well renowned database guy once said that to be with the times, JSON is your data model, <a href="http://hadoop.apache.org/" id="link-id0x2aabdd60e488">Hadoop</a> your file system, Mongo DB your database, and <a href="http://en.wikipedia.org/wiki/JavaScript" id="link-id0x2aabdcc71938">JavaScript</a> your language, and failing this, you are an old fart, a legacy suit, well, some uncool fossil.</p> <p>The key is not limited to JSON. More generally, it is zero time to some result and no learning curve. Some people will sacrifice almost anything for this, such as the possibility of doing arbitrary joins. People will even write code, even lots of it, if it only happens to be in their framework of choice.</p> <p>Phil again deplored the early fiasco of RDF messaging. "Triples are not so difficult. It is not true that RDF has a very steep learning curve." I would have to agree. The earlier gaffes of the <a href="http://dbpedia.org/resource/RDF/XML" id="link-id0x2aabc316f4f8">RDF/XML</a> syntax and the infamous <a href="http://dbpedia.org/resource/Semantic_Web_Stack" id="link-id0x2aabc1a85bb8">semantic web layer cake</a> <a href="https://en.wikipedia.org/wiki/File:Semantic-web-stack.png" id="link-id0x2aabdcdbb248">diagram</a> now lie buried and unlamented; let them be.</p> <p>Generating user experience from data or schema is an old mirage that has never really worked out. The imagined gain from eliminating application writing has however continued to fascinate IT minds and attempts in this direction have never really ceased. The lesson of history seems to be that coding is not to be eliminated, but that it should have fast turnaround time and immediately visible results.</p> <p>And since this is the age of data, databases should follow this lead. Schema-last is a good point, maybe adding JSON alongside <a href="http://en.wikipedia.org/wiki/XML" id="link-id0x1b8e638">XML</a> as an object type in RDF might not be so bad. There are already XML functions, so why not the analog for JSON? Just don't mention XML to the JSON folks...</p> <p>How does this relate to OKFN? Well, in the first instance this is the cultural impression I received from the meetup, but in a broader sense these factors are critical to realizing the full potential of OKFN's successes so far. OKFN is a data opening advocacy group; it is not a domain-specific think tank or special interest group. The data owners and their consultants will do analytics and even data integration if they see enough benefit in this, all in the established ways. However, the widespread opening of data does create possibilities that did not exist before. Actual benefits depend in great part on constant lowering of access barriers, and on a commitment by publishers to keep the data up to date, so that developers can build more than just a one-off mashup.</p> <p>True, there are government users of open data, since there is a productivity gain in already having the neighboring department's data opened to a point; one does no longer have to go through red tape to gain access to it.</p> <p>For an application ecosystem to keep growing on the base of tens of thousands of very heterogeneous datasets coming into the open, continuing to lower barriers is key. This is a very different task from making faster and faster databases or of optimizing a particular business process, and it demands different thinking.</p> <h3>Linked Geospatial Data 2014 Workshop posts:</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1781" id="link-id0x2aabdd4ac8c8">Linked Geospatial Data 2014 Workshop, Part 1: Web Services or SPARQL Modeling?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1782" id="link-id0x19a93298">Linked Geospatial Data 2014 Workshop, Part 2: Is SPARQL Slow?</a> </li> <li> Linked Geospatial Data 2014 Workshop, Part 3: The Stellar Reach of OKFN<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1784" id="link-id0x2aabc353fe38">Linked Geospatial Data 2014 Workshop, Part 4: GeoKnow, London, Brussels, The Message</a> </li> </ul>
2014-03-18T10:45:43-04:00
Linked Geospatial Data 2014 Workshop, Part 2: Is SPARQL Slow?
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-03-18#1786
2014-03-18T14:45:38Z
<p>I had a conversation with <a href="http://uk.linkedin.com/in/andyseaborne" id="link-id0x2aabc2be7ef8">Andy Seaborne</a> of <a href="http://www.epimorphics.com/" id="link-id0x2aabdcdec268">Epimorphics</a>, initial founder of the <a href="http://jena.apache.org/" id="link-id0x12a38db8">Jena RDF Framework</a> tool chain and editor of many <a href="https://www.w3.org/" id="link-id0x2aabdd5cdcd8">W3C</a> recommendations, among which the two <a href="http://dbpedia.org/page/SPARQL" id="link-id0x2aabc3cabb58">SPARQL</a>s. We exchanged some news; I told Andy about our progress in cutting the <a href="http://dbpedia.org/page/Resource_Description_Framework" id="link-id0x2aabdc894518">RDF</a>-to-<a href="http://dbpedia.org/page/SQL" id="link-id0x2aabc33b0138">SQL</a> performance penalty and doing more and better SQL tricks. Andy asked me if there were use cases doing analytics over RDF, not in the business intelligence sense, but in the sense of machine learning or discovery of structure. There is, in effect, such work, notably in data set summarization and description. A part of this has to do with learning the schema, like one would if wanting to put triples into tables when appropriate. <a href="http://www.cwi.nl/" id="link-id0x2aabc0ef1518">CWI</a> in <a href="http://lod2.ec" id="link-id0x2aabdc620178">LOD2</a> has worked in this direction, as has <a href="http://www.deri.ie/" id="link-id0x2aabc2df7e88">DERI</a> (<a href="http://www.deri.ie/users/giovanni-tummarello" id="link-id0x2aabc8076d58">Giovanni Tummarello</a>'s team), in the context of giving hints to SPARQL query writers. I would also mention <a href="http://dws.informatik.uni-mannheim.de/en/people/professors/prof-dr-christian-bizer/" id="link-id0x2aabc2a41468">Chris Bizer</a> <i>et al.,</i> at University of Mannheim, with their data integration work, which is all about similarity detection in a schema-less world, e.g., the 150M HTML tables in the Common Crawl, briefly mentioned in the <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x2aabc2c90ee8">previous blog</a>. <a href="http://jens-lehmann.org/" id="link-id0x993aea8">Jens Lehmann</a> from University of Leipzig has also done work in learning a schema from the data, this time in OWL.</p> <p>Andy was later on a panel where <a href="http://philarcher.org/" id="link-id0x2aabc10baba8">Phil Archer</a> asked him whether SPARQL was slow by nature or whether this was a matter of bad implementations. Andy answered approximately as follows: "If you allow for arbitrary <i>ad hoc</i> structure, you will always pay something for this. However, if you tell the engine what your data is like, it is no different from executing SQL." This is essentially the gist of our conversation. Most likely we will make this happen via adaptive schema for the regular part and exceptions as quads.</p> <p>Later I talked with Phil about the "SPARQL is slow" meme. The fact is that <a href="http://virtuoso.openlinksw.com" id="link-id0x2aabdd227d98">Virtuoso</a> SPARQL will outperform or match <a href="http://postgis.net/" id="link-id0x2aabdd41f238">PostGIS</a> SQL for Geospatial lookups against the <a href="http://www.openstreetmap.org/" id="link-id0x2aabdc7fbae8">OpenStreetMap</a> dataset. Virtuoso SQL will win by a factor of 5 to 10. Still, the SPARQL is slow meme is not entirely without a basis in fact. I would say that the really blatant cases that give SPARQL a bad name are query optimization problems. With 50 triple patterns in a query there are 50-factorial ways of getting a bad plan. This is where the catastrophic failures of 100+ times worse than SQL come from. The regular penalty of doing triples vs tables is somewhere between 2.5 (<a href="http://people.aifb.kit.edu/bka/ssb-benchmark/" id="link-id0x13912118">Star Schema Benchmark</a>) and 10 (lookups with many literals), quite acceptable for many applications. Some really bad cases can occur with regular expressions on URI strings or literals, but then, if this is the core of the application, it should use a different data model or an <a href="http://dbpedia.org/resource/N-gram" id="link-id0x2aabca30d8d8"><i>n</i>-gram</a> index.</p> <p>The solutions, including more dependable query plan choice, will flow from adaptive schema which essentially reduces RDF back into relational, however without forcing schema first and with accommodation for exceptions in the data.</p> <p>Phil noted here that there already exist many (so far, proprietary) ways of describing the shape of a graph. He said there would be a W3C activity for converging these. If so, a vocabulary that can express relationships, the types of related entities, their cardinalities, etc., comes close to a SQL schema and its statistics. Such a thing can be the output of data analysis, or the input to a query optimizer or storage engine, for using a schema where one in fact exists. Like this, there is no reason why things would be less predictable than with SQL. The idea of a re-convergence of data models is definitely in the air; this is in no sense limited to us.</p> <h3>Linked Geospatial Data 2014 Workshop posts:</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1781" id="link-id0x2aabc13c08e8">Linked Geospatial Data 2014 Workshop, Part 1: Web Services or SPARQL Modeling?</a> </li> <li> Linked Geospatial Data 2014 Workshop, Part 2: Is SPARQL Slow?<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1783" id="link-id0x2aabc2bd21e8">Linked Geospatial Data 2014 Workshop, Part 3: The Stellar Reach of OKFN</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1784" id="link-id0x2aabdd6840e8">Linked Geospatial Data 2014 Workshop, Part 4: GeoKnow, London, Brussels, The Message</a> </li> </ul>
2014-03-18T10:45:38.000004-04:00
Linked Geospatial Data 2014 Workshop, Part 1: Web Services or SPARQL Modeling?
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-03-18#1785
2014-03-18T14:45:32Z
<p>The <a href="http://www.w3.org/" id="link-id0x2aabdd526b38">W3C</a> (World Wide Web Consortium) and <a href="http://www.opengeospatial.org/" id="link-id0x2aabdd526af8">OGC</a> (Open Geospatial Consortium) organized the <a href="http://www.w3.org/2014/03/lgd/" id="link-id0x2aab69bcf828">Linked Geospatial Data 2014</a> workshop in London this week. The <a href="http://geoknow.eu" id="link-id0x2aab69bcf7e8">GeoKnow project</a> was represented by <a href="http://aksw.org/ClausStadler.html" id="link-id0x2aab69bcf7a8">Claus Stadler</a> of <a href="http://www.zv.uni-leipzig.de/" id="link-id0x2aabc3d44088">Universität Leipzig</a>, and Hugh Williams and myself (Orri Erling) from <a href="http://www.openlinksw.com" id="link-id0x2aabc3d44068">OpenLink Software</a>. The <a href="http://okfn.org/" id="link-id0x2aabc36fe418">Open Knowledge Foundation</a> (OKFN) also held an <a href="http://www.eventbrite.co.uk/e/london-open-data-meetup-march-2014-tickets-10574052275" id="link-id0x2aabc36fe3f8">Open Data Meetup</a> in the evening of the first day of the workshop.</p> <p>Reporting on each talk and the many highly diverse topics addressed is beyond the scope of this article; for this you can go to the program and the slides that will be online. Instead, I will talk about questions that to me seemed to be in the air, and about some conversations I had with the relevant people.</p> <p>The trend in events like this is towards shorter and shorter talks and more and more interaction. In this workshop, talks were given in series of three talks with all questions at the end, with all the presenters on stage. This is not a bad idea since we get a panel-like effect where many presenters can address the same question. If the subject matter allows, a panel is my preferred format.</p> <h3>Web services or SPARQL? Is GeoSPARQL good? Is it about Linked Data or about ontologies?</h3> <p>Geospatial data tends to be exposed via web services, e.g., WFS (<a href="http://dbpedia.org/page/Web_Feature_Service" id="link-id0x2aabdc7781e8">Web Feature Service</a>). This allows item retrieval on a lookup basis and some predefined filtering, transformation, and content negotiation. Capabilities vary; OGC now has WFS 2.0, and there are open source implementations that do a fair job of providing the functionality.</p> <p>Of course, a real query language is much more expressive, but a service API is more scalable, as people say. What they mean is that an API is more predictable. For pretty much any complex data task, a query language is near-infinitely more efficient than going back-and-forth, often on a wide area network, via an API. So, as <a href="http://www.aifb.kit.edu/web/Andreas_Harth/en" id="link-id0x2aabdc778188">Andreas Harth</a> put it: for data publishers, make an API; an open <a href="http://dbpedia.org/page/SPARQL" id="link-id0x2aabdd45cbd8">SPARQL</a> endpoint is too "brave," [Andreas' word, with the meaning of foolhardy]. When you analyze, he continued, then you load it into a endpoint, but you use your own. Any <a href="http://dbpedia.org/page/Quality_of_service" id="link-id0x2aabdd45cbb8">quality of service</a> terms must be formulated with respect to a fixed workload, this is not meaningful with <i>ad hoc</i> queries in an expressive language. Things like anytime semantics (return whatever is found within a time limit) are only good for a first interactive look, not for applications.</p> <p>Should the application go to the data or the reverse? Some data is big and moving it is not self-evident. A culture of datasets being hosted on a cloud may be forming. Of course some linked data like DBpedia has for a long time been available as <a href="http://kidehen.blogspot.co.uk/2014/03/preloaded-preconfigured-dbpedia.html" id="link-id0x2aabdc6fbd48">Amazon images</a>. Recently, <a href="http://www.sindicetech.com/" id="link-id0x2aabdc6fbce8">SindiceTech</a> has made a <a href="http://www.sindicetech.com/2/post/2014/02/sindicetech-freebase-distribution-launched-on-google-developers-live.html" id="link-id0x2aabdc6fbca8">similar packaging</a> of <a href="http://dbpedia.org/resource/Freebase" id="link-id0x2aabdd1b5c58">Freebase</a>. The data of interest here is larger and its target audience is more specific, on the e-science side.</p> <p>How should geometries be modeled? I have met the <a href="http://dbpedia.org/page/GeoSPARQL" id="link-id0x2aabdd1b5bf8">GeoSPARQL</a> and the <a href="http://dbpedia.org/page/Simple_Features" id="link-id0x2aabdc6fcaf8">SQL MM</a> on which it is based with a sense of relief, as these are reasonable things that can be efficiently implemented. There are proposals where points have URIs, and linestrings are ordered sets of points, and collections are actual trees with <a href="http://dbpedia.org/page/Resource_Description_Framework" id="link-id0x2aabdd661c28">RDF</a> subjects as nodes. As a standard, such a thing is beyond horrible, as it hits all the RDF penalties and overheads full force, and promises easily 10x worse space consumption and 100x worse run times compared to the sweetly reasonable GeoSPARQL. One presenter said that cases of actually hanging attributes off points of complex geometries had been heard of but were, in his words, anecdotal. He posed a question to the audience about use cases where points in fact needed separately addressable identities. Several cases did emerge, involving, for example, different measurement certainties for different points on on a trajectory trace obtained by radar. Applications that need data of this sort will perforce be very domain specific. <a href="http://www.openstreetmap.org/" id="link-id0x2aabdd581008">OpenStreetMap</a> (OSM) itself is a bit like this, but there the points that have individual identity also have predominantly non-geometry attributes and stand for actually-distinct entities. OSM being a practical project, these are then again collapsed into linestrings for cases where this is more efficient. The OGC data types themselves have up to 4 dimensions, of which the 4th could be used as an identifier of a point in the event this really were needed. If so, this would likely be empty for most points and would compress away if the data representation were done right.</p> <p>For data publishing, Andreas proposed to give OGC geometries URIs, i.e., the borders of a country can be more or less precisely modeled, and the large polygon may have different versions and provenances. This is reasonable enough, as long as the geometries are big. For applications, one will then collapse the <code>1:n</code> between entity and its geometry into a <code>1:1</code>. In the end, when you make an application, even an RDF one, you do not just throw all the data in a bucket and write queries against that. Some alignment and transformation is generally involved.</p> <h3>Linked Geospatial Data 2014 Workshop posts:</h3> <ul> <li> Linked Geospatial Data 2014 Workshop, Part 1: Web Services or SPARQL Modeling?<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1782" id="link-id0x2aabc02fa778">Linked Geospatial Data 2014 Workshop, Part 2: Is SPARQL Slow?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1783" id="link-id0x2aabdd11e238">Linked Geospatial Data 2014 Workshop, Part 3: The Stellar Reach of OKFN</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1784" id="link-id0x2aabdd59c448">Linked Geospatial Data 2014 Workshop, Part 4: GeoKnow, London, Brussels, The Message</a> </li> </ul>
2014-03-18T10:45:32.000021-04:00
Linked Geospatial Data 2014 Workshop, Part 4: GeoKnow, London, Brussels, The Message
http://www.openlinksw.com:443/weblog/oerling/?date=2014-03-18#1784
2014-03-18T14:20:31Z
<p>Last Friday (2014-03-14) I gave a talk about <a href="http://geoknow.eu" id="link-id0x2aabc3ce6228">GeoKnow</a> at the <a href="http://www.copernicus.eu/pages-secondaires/events/event-detail/?tx_julleevents_pi1%5BshowUid%5D=533&cHash=6b72b992422ee703f83e7c6de4c7e697" id="link-id0x2aabc3ce6288">EC Copernicus Big Data workshop</a>. This was a trial run for more streamlined messaging. I have, aside the practice of geekcraft, occupied myself with questions of communication these last weeks.</p> <p>The clear take-home from London and Brussels alike is that these events have full days and 4 or more talks an hour. It is not quite TV commercial spots yet but it is going in this direction.</p> <p>If you say something complex, little will get across unless the audience already knows what you will be saying.</p> <p>I had a set of slides from <a href="http://jens-lehmann.org/" id="link-id0x2aabc3ce6368">Jens Lehmann</a>, the GeoKnow project coordinator, for whom I was standing in. Now these are a fine rendition of the description of work. What is wrong with partners, work packages, objectives, etc? Nothing, except everybody has them.</p> <p>I recall the old story about the journalist and the Zen master: The Zen master repeatedly advises the reporter to cut the story in half. We get the same from PR professionals, "If it is short, they have at least thought about what should go in there," said one recently, talking of pitches and messages. The other advice was to use pictures. And to have a personal dimension to it.</p> <p>Enter "Ms. Globe" and "Mr. Cube". <a href="http://nl.linkedin.com/pub/frans-knibbe/5a/93/409" id="link-id0x2aabc3ce6488">Frans Knibbe</a> of <a href="http://www.geodan.nl/" id="link-id0x2aabc3ce6568">Geodan</a> gave the <a href="http://www.w3.org/2014/03/lgd/" id="link-id0x2aabc3ce65c8">Linked Geospatial Data 2014</a> workshop's most memorable talk entitled <a href="http://www.w3.org/2014/03/lgd/Frans" id="link-id0x2aabc3ce6628">"Linked Data and Geoinformatics - a love story" (pdf)</a> about the excitement and the pitfalls of the burgeoning courtship of Ms. Globe (geoinformatics) and Mr. Cube (semantic technology). They get to talking, later Ms. Globe thinks to herself... "Desiloisazation, explicit semantics, integrated metadata..." Mr. Cube, young upstart now approaching a more experienced and sophisticated lady, dreams of finally making an entry into adult society, "critical mass, global scope, relevant applications..." There is a vibration in the air.</p> <p>So, with <a href="http://nl.linkedin.com/pub/frans-knibbe/5a/93/409" id="link-id0x2aabc3ce66a8">Frans Knibbe</a>'s gracious permission <a href="http://www.slideshare.net/OpenLinkSoftware/geo-knowcopernicusbigdataworkshopbrusselsmarch2014-32446207" id="link-id0x2aabc3ce6648">I borrowed the storyline and some of the pictures</a>. </p> <div style="margin: 0 auto; text-align: center"> <iframe align="middle" src="http://www.slideshare.net/slideshow/embed_code/32446207" width="427" height="356" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px 1px 0; margin-bottom:5px; max-width: 100%;" allowfullscreen="allowfullscreen"> </iframe> </div>We ought to make a series of cartoons about the couple. There will be twists and turns in the story to come. <p>Mr. Cube is not Ms. Globe's first lover, though; there is also rich and worldly Mr. Table. How will Mr. Cube prove himself? The eternal question... Well, not by moping around, not by wise-cracking about semantics, no. By boldly setting out upon a journey to fetch the Golden Fleece from beyond the crashing rocks. "Column store, vectored execution, scale out, data clustering, adaptive schema..." he affirms, with growing confidence.</p> <p>This is where the story stands, right now. <a href="http://virtuoso.openlinksw.com" id="link-id0x2aabc3ce6768">Virtuoso</a> run circles around <a href="http://postgis.net/" id="link-id0x2aabc3ce6808">PostGIS</a> doing aggregations and lookups on geometries in a map-scrolling scenario (GeoKnow's <a href="https://github.com/GeoKnow/GeoBenchLab" id="link-id0x2aabc3ce6868">GeoBenchLab</a>). Virtuoso <a href="http://en.wikipedia.org/wiki/SPARQL" id="link-id0x2aabdd806b48">SPARQL</a> outperforms PostGIS <a href="http://en.wikipedia.org/wiki/SQL" id="link-id0x2aabdd806ba8">SQL</a> against planet-scale <a href="http://www.openstreetmap.org/" id="link-id0x2aabdd806c08">OpenStreetMap</a>; Virtuoso SQL goes 5-10x faster still.</p> <p>Mr Cube is fast on the draw, but still some corners can be smoothed out.</p> <p>Later in GeoKnow, there will be still more speed but also near parity between SQL and SPARQL via taking advantage of data regularity in guiding physical storage. If it is big, it is bound to have repeating structure.</p> <p>The love story grows more real by the day. To be consummated still within GeoKnow.</p> <p>Talking of databases has the great advantage that this has been a performance game from the start. There are few people who need convincing about the desirability of performance, as this also makes for lower cost and more flexibility on the application side.</p> <p>But this is not all there is to it.</p> <p>In Brussels, the public was about E-science (Earth observation). In science, it is understood that qualitative aspects can be even more crucial. I told the story about an E-science-oriented workshop I attended in America years ago. The practitioners, from high energy physics to life sciences to climate, had invariably come across the need for self-description of data and for schema-last. This was essentially never provided by <a href="http://en.wikipedia.org/wiki/Resource_Description_Framework" id="link-id0x2aabdd806fe8">RDF</a>, except for some life science cases. Rather, we had one-off schemes, ranging from key-value pairs to putting the table name in a column of the same table to preserve the origin across data export.</p> <p>Explicit semantics and integrated metadata are important, Ms. Globe knows, but she cannot sacrifice operational capacity for this. So it is more than a <a href="http://kingsley.idehen.net/describe/?url=http%3A%2F%2Fkingsley.idehen.net%2FDAV%2Fhome%2Fkidehen%2FPublic%2FLinked%2520Data%2520Documents%2FGlossaryOfTerms.ttl%23DBMS" id="link-id0x2aabdd8070e8">DBMS</a> or even data model choice -- there must be a solid tool chain for data integration and visualization. GeoKnow provides many tools in this space.</p> <p>Some of these, such as the <a href="http://svn.aksw.org/papers/2011/WWW_LIMES/public.pdf" id="link-id0x2aabdd807148">LIMES entity matching framework (pdf)</a> are probably close to the best there is. For other parts, the SQL-based products with hundreds of person years invested in user interaction are simply unbeatable.</p> <p>In these cases, the world can continue to talk SQL. If the regular part of the data is in fact tables already, so much the better. You connect to Virtuoso via SQL, just like to PostGIS or <a href="http://www.oracle.com/technetwork/database/options/spatialandgraph/overview/index.html" id="link-id0x2aabdd807168">Oracle Spatial</a>, and talk <a href="http://en.wikipedia.org/wiki/Simple_Features" id="link-id0x2aabdd807268">SQL MM</a>. The triples, in the sense of flexible annotation and integrated metadata, stay there; you just do not see them if you do not want them.</p> <p> There are possibilities all right. In the coming months I will showcase some of the progress, starting with a detailed look at the OpenStreetMap experiments we have made in GeoKnow.</p> <h3>Linked Geospatial Data 2014 Workshop posts:</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1781" id="link-id0x2aabdd807428">Linked Geospatial Data 2014 Workshop, Part 1: Web Services or SPARQL Modeling?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1782" id="link-id0x2aabdd8074e8">Linked Geospatial Data 2014 Workshop, Part 2: Is SPARQL Slow?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1783" id="link-id0x2aabdd8075a8">Linked Geospatial Data 2014 Workshop, Part 3: The Stellar Reach of OKFN</a> </li> <li> Linked Geospatial Data 2014 Workshop, Part 4: GeoKnow, London, Brussels, The Message<i> (this post)</i> </li> </ul>
2014-03-18T10:52:22.000013-04:00
Linked Geospatial Data 2014 Workshop, Part 3: The Stellar Reach of OKFN
http://www.openlinksw.com:443/weblog/oerling/?date=2014-03-18#1783
2014-03-18T14:20:28Z
<p>The <a href="http://okfn.org/" id="link-id0x2aabc38976f8">Open Knowledge Foundation</a> (OKFN) held a <a href="http://www.eventbrite.co.uk/e/london-open-data-meetup-march-2014-tickets-10574052275" id="link-id0x2aabdd08bb18">London Open Data Meetup</a> in the evening of the first day of the <a href="http://www.w3.org/2014/03/lgd/" id="link-id0x127f1648">Linked Geospatial Data 2014</a> workshop. The event was, as they themselves put it, at the amazing open concept office of OKFN at the <a href="http://blog.okfn.org/2011/06/23/okfns-london-office-opens-at-the-centre-for-creative-collaboration/" id="link-id0x2aabc2ffb288">Center for Creative Collaboration</a> in Central London. What could sound cooler? True, OKFN threw a good party, with ever engaging and charismatic founder <a href="http://okfn.org/about/team/#rufus-pollock-8212-founder" id="link-id0x2aabc2bbdd68">Rufus Pollock</a> presiding. <a href="http://philarcher.org/" id="link-id0x2aabc37081d8">Phil Archer</a> noted, only half in jest, that OKFN was so influential, visible, had the ear of government and public alike, etc., that it put <a href="http://www.w3.org/" id="link-id0x2aabc3cabb58">W3C</a> to shame.</p> <p>Now, OKFN is a party in the <a href="http://lod2.ec" id="link-id0x2aabdd5e9ab8">LOD2</a> FP7 project, so I have over the years met people from there on and off. In LOD2, OKFN is praised to the skies for its visibility and influence and outreach and sometimes, in passing, critiqued for not publishing enough <a href="http://dbpedia.org/page/Resource_Description_Framework" id="link-id0x2aabdd60e5b8">RDF</a>, let alone five star linked data.</p> <p>As it happens, <a href="http://dbpedia.org/page/Comma-separated_values" id="link-id0x128d93d8">CSV</a> rules, and even the W3C will, it appears, undertake to standardize a CSV-to-RDF mapping. As far as I am concerned, as long as there is no alignment of identifiers or vocabulary, whether a thing is CSV or exactly equivalent RDF, there is no great difference, except that CSV is smaller and loads into Excel.</p> <p>For OKFN, which has a mission of opening data, insisting on any particular format would just hinder the cause.</p> <p>What do we learn from this? OKFN is praised not only for government relations but also for developer friendliness. Lobbying for open data is something I can understand, but how do you do developer relations? This is not like talking to customers, where the customer wants to do something and it is usually possible to give some kind of advice or recommendation on how they can use our technology for the purpose.</p> <p>Are <a href="http://dbpedia.org/page/JSON" id="link-id0x2aabc1987188">JSON</a> and <a href="https://www.mongodb.org/" id="link-id0x2aabc10ba4e8">Mongo DB</a> the key? A well renowned database guy once said that to be with the times, JSON is your data model, <a href="http://hadoop.apache.org/" id="link-id0x2aabdd60e488">Hadoop</a> your file system, Mongo DB your database, and <a href="http://en.wikipedia.org/wiki/JavaScript" id="link-id0x2aabdcc71938">JavaScript</a> your language, and failing this, you are an old fart, a legacy suit, well, some uncool fossil.</p> <p>The key is not limited to JSON. More generally, it is zero time to some result and no learning curve. Some people will sacrifice almost anything for this, such as the possibility of doing arbitrary joins. People will even write code, even lots of it, if it only happens to be in their framework of choice.</p> <p>Phil again deplored the early fiasco of RDF messaging. "Triples are not so difficult. It is not true that RDF has a very steep learning curve." I would have to agree. The earlier gaffes of the <a href="http://dbpedia.org/resource/RDF/XML" id="link-id0x2aabc316f4f8">RDF/XML</a> syntax and the infamous <a href="http://dbpedia.org/resource/Semantic_Web_Stack" id="link-id0x2aabc1a85bb8">semantic web layer cake</a> <a href="https://en.wikipedia.org/wiki/File:Semantic-web-stack.png" id="link-id0x2aabdcdbb248">diagram</a> now lie buried and unlamented; let them be.</p> <p>Generating user experience from data or schema is an old mirage that has never really worked out. The imagined gain from eliminating application writing has however continued to fascinate IT minds and attempts in this direction have never really ceased. The lesson of history seems to be that coding is not to be eliminated, but that it should have fast turnaround time and immediately visible results.</p> <p>And since this is the age of data, databases should follow this lead. Schema-last is a good point, maybe adding JSON alongside <a href="http://en.wikipedia.org/wiki/XML" id="link-id0x1b8e638">XML</a> as an object type in RDF might not be so bad. There are already XML functions, so why not the analog for JSON? Just don't mention XML to the JSON folks...</p> <p>How does this relate to OKFN? Well, in the first instance this is the cultural impression I received from the meetup, but in a broader sense these factors are critical to realizing the full potential of OKFN's successes so far. OKFN is a data opening advocacy group; it is not a domain-specific think tank or special interest group. The data owners and their consultants will do analytics and even data integration if they see enough benefit in this, all in the established ways. However, the widespread opening of data does create possibilities that did not exist before. Actual benefits depend in great part on constant lowering of access barriers, and on a commitment by publishers to keep the data up to date, so that developers can build more than just a one-off mashup.</p> <p>True, there are government users of open data, since there is a productivity gain in already having the neighboring department's data opened to a point; one does no longer have to go through red tape to gain access to it.</p> <p>For an application ecosystem to keep growing on the base of tens of thousands of very heterogeneous datasets coming into the open, continuing to lower barriers is key. This is a very different task from making faster and faster databases or of optimizing a particular business process, and it demands different thinking.</p> <h3>Linked Geospatial Data 2014 Workshop posts:</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1781" id="link-id0x2aabdd4ac8c8">Linked Geospatial Data 2014 Workshop, Part 1: Web Services or SPARQL Modeling?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1782" id="link-id0x19a93298">Linked Geospatial Data 2014 Workshop, Part 2: Is SPARQL Slow?</a> </li> <li> Linked Geospatial Data 2014 Workshop, Part 3: The Stellar Reach of OKFN<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1784" id="link-id0x2aabc353fe38">Linked Geospatial Data 2014 Workshop, Part 4: GeoKnow, London, Brussels, The Message</a> </li> </ul>
2014-03-18T10:22:46-04:00
Linked Geospatial Data 2014 Workshop, Part 2: Is SPARQL Slow?
http://www.openlinksw.com:443/weblog/oerling/?date=2014-03-18#1782
2014-03-18T14:20:25Z
<p>I had a conversation with <a href="http://uk.linkedin.com/in/andyseaborne" id="link-id0x2aabc2be7ef8">Andy Seaborne</a> of <a href="http://www.epimorphics.com/" id="link-id0x2aabdcdec268">Epimorphics</a>, initial founder of the <a href="http://jena.apache.org/" id="link-id0x12a38db8">Jena RDF Framework</a> tool chain and editor of many <a href="https://www.w3.org/" id="link-id0x2aabdd5cdcd8">W3C</a> recommendations, among which the two <a href="http://dbpedia.org/page/SPARQL" id="link-id0x2aabc3cabb58">SPARQL</a>s. We exchanged some news; I told Andy about our progress in cutting the <a href="http://dbpedia.org/page/Resource_Description_Framework" id="link-id0x2aabdc894518">RDF</a>-to-<a href="http://dbpedia.org/page/SQL" id="link-id0x2aabc33b0138">SQL</a> performance penalty and doing more and better SQL tricks. Andy asked me if there were use cases doing analytics over RDF, not in the business intelligence sense, but in the sense of machine learning or discovery of structure. There is, in effect, such work, notably in data set summarization and description. A part of this has to do with learning the schema, like one would if wanting to put triples into tables when appropriate. <a href="http://www.cwi.nl/" id="link-id0x2aabc0ef1518">CWI</a> in <a href="http://lod2.ec" id="link-id0x2aabdc620178">LOD2</a> has worked in this direction, as has <a href="http://www.deri.ie/" id="link-id0x2aabc2df7e88">DERI</a> (<a href="http://www.deri.ie/users/giovanni-tummarello" id="link-id0x2aabc8076d58">Giovanni Tummarello</a>'s team), in the context of giving hints to SPARQL query writers. I would also mention <a href="http://dws.informatik.uni-mannheim.de/en/people/professors/prof-dr-christian-bizer/" id="link-id0x2aabc2a41468">Chris Bizer</a> <i>et al.,</i> at University of Mannheim, with their data integration work, which is all about similarity detection in a schema-less world, e.g., the 150M HTML tables in the Common Crawl, briefly mentioned in the <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x2aabc2c90ee8">previous blog</a>. <a href="http://jens-lehmann.org/" id="link-id0x993aea8">Jens Lehmann</a> from University of Leipzig has also done work in learning a schema from the data, this time in OWL.</p> <p>Andy was later on a panel where <a href="http://philarcher.org/" id="link-id0x2aabc10baba8">Phil Archer</a> asked him whether SPARQL was slow by nature or whether this was a matter of bad implementations. Andy answered approximately as follows: "If you allow for arbitrary <i>ad hoc</i> structure, you will always pay something for this. However, if you tell the engine what your data is like, it is no different from executing SQL." This is essentially the gist of our conversation. Most likely we will make this happen via adaptive schema for the regular part and exceptions as quads.</p> <p>Later I talked with Phil about the "SPARQL is slow" meme. The fact is that <a href="http://virtuoso.openlinksw.com" id="link-id0x2aabdd227d98">Virtuoso</a> SPARQL will outperform or match <a href="http://postgis.net/" id="link-id0x2aabdd41f238">PostGIS</a> SQL for Geospatial lookups against the <a href="http://www.openstreetmap.org/" id="link-id0x2aabdc7fbae8">OpenStreetMap</a> dataset. Virtuoso SQL will win by a factor of 5 to 10. Still, the SPARQL is slow meme is not entirely without a basis in fact. I would say that the really blatant cases that give SPARQL a bad name are query optimization problems. With 50 triple patterns in a query there are 50-factorial ways of getting a bad plan. This is where the catastrophic failures of 100+ times worse than SQL come from. The regular penalty of doing triples vs tables is somewhere between 2.5 (<a href="http://people.aifb.kit.edu/bka/ssb-benchmark/" id="link-id0x13912118">Star Schema Benchmark</a>) and 10 (lookups with many literals), quite acceptable for many applications. Some really bad cases can occur with regular expressions on URI strings or literals, but then, if this is the core of the application, it should use a different data model or an <a href="http://dbpedia.org/resource/N-gram" id="link-id0x2aabca30d8d8"><i>n</i>-gram</a> index.</p> <p>The solutions, including more dependable query plan choice, will flow from adaptive schema which essentially reduces RDF back into relational, however without forcing schema first and with accommodation for exceptions in the data.</p> <p>Phil noted here that there already exist many (so far, proprietary) ways of describing the shape of a graph. He said there would be a W3C activity for converging these. If so, a vocabulary that can express relationships, the types of related entities, their cardinalities, etc., comes close to a SQL schema and its statistics. Such a thing can be the output of data analysis, or the input to a query optimizer or storage engine, for using a schema where one in fact exists. Like this, there is no reason why things would be less predictable than with SQL. The idea of a re-convergence of data models is definitely in the air; this is in no sense limited to us.</p> <h3>Linked Geospatial Data 2014 Workshop posts:</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1781" id="link-id0x2aabc13c08e8">Linked Geospatial Data 2014 Workshop, Part 1: Web Services or SPARQL Modeling?</a> </li> <li> Linked Geospatial Data 2014 Workshop, Part 2: Is SPARQL Slow?<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1783" id="link-id0x2aabc2bd21e8">Linked Geospatial Data 2014 Workshop, Part 3: The Stellar Reach of OKFN</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1784" id="link-id0x2aabdd6840e8">Linked Geospatial Data 2014 Workshop, Part 4: GeoKnow, London, Brussels, The Message</a> </li> </ul>
2015-08-20T17:35:43.158051-04:00
Linked Geospatial Data 2014 Workshop, Part 1: Web Services or SPARQL Modeling?
http://www.openlinksw.com:443/weblog/oerling/?date=2014-03-18#1781
2014-03-18T14:20:20Z
<p>The <a href="http://www.w3.org/" id="link-id0x2aabdd526b38">W3C</a> (World Wide Web Consortium) and <a href="http://www.opengeospatial.org/" id="link-id0x2aabdd526af8">OGC</a> (Open Geospatial Consortium) organized the <a href="http://www.w3.org/2014/03/lgd/" id="link-id0x2aab69bcf828">Linked Geospatial Data 2014</a> workshop in London this week. The <a href="http://geoknow.eu" id="link-id0x2aab69bcf7e8">GeoKnow project</a> was represented by <a href="http://aksw.org/ClausStadler.html" id="link-id0x2aab69bcf7a8">Claus Stadler</a> of <a href="http://www.zv.uni-leipzig.de/" id="link-id0x2aabc3d44088">Universität Leipzig</a>, and Hugh Williams and myself (Orri Erling) from <a href="http://www.openlinksw.com" id="link-id0x2aabc3d44068">OpenLink Software</a>. The <a href="http://okfn.org/" id="link-id0x2aabc36fe418">Open Knowledge Foundation</a> (OKFN) also held an <a href="http://www.eventbrite.co.uk/e/london-open-data-meetup-march-2014-tickets-10574052275" id="link-id0x2aabc36fe3f8">Open Data Meetup</a> in the evening of the first day of the workshop.</p> <p>Reporting on each talk and the many highly diverse topics addressed is beyond the scope of this article; for this you can go to the program and the slides that will be online. Instead, I will talk about questions that to me seemed to be in the air, and about some conversations I had with the relevant people.</p> <p>The trend in events like this is towards shorter and shorter talks and more and more interaction. In this workshop, talks were given in series of three talks with all questions at the end, with all the presenters on stage. This is not a bad idea since we get a panel-like effect where many presenters can address the same question. If the subject matter allows, a panel is my preferred format.</p> <h3>Web services or SPARQL? Is GeoSPARQL good? Is it about Linked Data or about ontologies?</h3> <p>Geospatial data tends to be exposed via web services, e.g., WFS (<a href="http://dbpedia.org/page/Web_Feature_Service" id="link-id0x2aabdc7781e8">Web Feature Service</a>). This allows item retrieval on a lookup basis and some predefined filtering, transformation, and content negotiation. Capabilities vary; OGC now has WFS 2.0, and there are open source implementations that do a fair job of providing the functionality.</p> <p>Of course, a real query language is much more expressive, but a service API is more scalable, as people say. What they mean is that an API is more predictable. For pretty much any complex data task, a query language is near-infinitely more efficient than going back-and-forth, often on a wide area network, via an API. So, as <a href="http://www.aifb.kit.edu/web/Andreas_Harth/en" id="link-id0x2aabdc778188">Andreas Harth</a> put it: for data publishers, make an API; an open <a href="http://dbpedia.org/page/SPARQL" id="link-id0x2aabdd45cbd8">SPARQL</a> endpoint is too "brave," [Andreas' word, with the meaning of foolhardy]. When you analyze, he continued, then you load it into a endpoint, but you use your own. Any <a href="http://dbpedia.org/page/Quality_of_service" id="link-id0x2aabdd45cbb8">quality of service</a> terms must be formulated with respect to a fixed workload, this is not meaningful with <i>ad hoc</i> queries in an expressive language. Things like anytime semantics (return whatever is found within a time limit) are only good for a first interactive look, not for applications.</p> <p>Should the application go to the data or the reverse? Some data is big and moving it is not self-evident. A culture of datasets being hosted on a cloud may be forming. Of course some linked data like DBpedia has for a long time been available as <a href="http://kidehen.blogspot.co.uk/2014/03/preloaded-preconfigured-dbpedia.html" id="link-id0x2aabdc6fbd48">Amazon images</a>. Recently, <a href="http://www.sindicetech.com/" id="link-id0x2aabdc6fbce8">SindiceTech</a> has made a <a href="http://www.sindicetech.com/2/post/2014/02/sindicetech-freebase-distribution-launched-on-google-developers-live.html" id="link-id0x2aabdc6fbca8">similar packaging</a> of <a href="http://dbpedia.org/resource/Freebase" id="link-id0x2aabdd1b5c58">Freebase</a>. The data of interest here is larger and its target audience is more specific, on the e-science side.</p> <p>How should geometries be modeled? I have met the <a href="http://dbpedia.org/page/GeoSPARQL" id="link-id0x2aabdd1b5bf8">GeoSPARQL</a> and the <a href="http://dbpedia.org/page/Simple_Features" id="link-id0x2aabdc6fcaf8">SQL MM</a> on which it is based with a sense of relief, as these are reasonable things that can be efficiently implemented. There are proposals where points have URIs, and linestrings are ordered sets of points, and collections are actual trees with <a href="http://dbpedia.org/page/Resource_Description_Framework" id="link-id0x2aabdd661c28">RDF</a> subjects as nodes. As a standard, such a thing is beyond horrible, as it hits all the RDF penalties and overheads full force, and promises easily 10x worse space consumption and 100x worse run times compared to the sweetly reasonable GeoSPARQL. One presenter said that cases of actually hanging attributes off points of complex geometries had been heard of but were, in his words, anecdotal. He posed a question to the audience about use cases where points in fact needed separately addressable identities. Several cases did emerge, involving, for example, different measurement certainties for different points on on a trajectory trace obtained by radar. Applications that need data of this sort will perforce be very domain specific. <a href="http://www.openstreetmap.org/" id="link-id0x2aabdd581008">OpenStreetMap</a> (OSM) itself is a bit like this, but there the points that have individual identity also have predominantly non-geometry attributes and stand for actually-distinct entities. OSM being a practical project, these are then again collapsed into linestrings for cases where this is more efficient. The OGC data types themselves have up to 4 dimensions, of which the 4th could be used as an identifier of a point in the event this really were needed. If so, this would likely be empty for most points and would compress away if the data representation were done right.</p> <p>For data publishing, Andreas proposed to give OGC geometries URIs, i.e., the borders of a country can be more or less precisely modeled, and the large polygon may have different versions and provenances. This is reasonable enough, as long as the geometries are big. For applications, one will then collapse the <code>1:n</code> between entity and its geometry into a <code>1:1</code>. In the end, when you make an application, even an RDF one, you do not just throw all the data in a bucket and write queries against that. Some alignment and transformation is generally involved.</p> <h3>Linked Geospatial Data 2014 Workshop posts:</h3> <ul> <li> Linked Geospatial Data 2014 Workshop, Part 1: Web Services or SPARQL Modeling?<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1782" id="link-id0x2aabc02fa778">Linked Geospatial Data 2014 Workshop, Part 2: Is SPARQL Slow?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1783" id="link-id0x2aabdd11e238">Linked Geospatial Data 2014 Workshop, Part 3: The Stellar Reach of OKFN</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1784" id="link-id0x2aabdd59c448">Linked Geospatial Data 2014 Workshop, Part 4: GeoKnow, London, Brussels, The Message</a> </li> </ul>
2014-03-18T10:22:43.000001-04:00
In Hoc Signo Vinces (part 9 of n): TPC-H Q18, Ordered Aggregation, and Top K
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-03-17#1780
2014-03-17T16:41:32Z
<p>We will here return to polishing the cutting edge, the high geekcraft of database. We will look at more of the wonders of <a href="http://www.tpc.org/tpch/" id="link-id0x256ad6b8">TPC-H</a> and cover two more tricks. The experts can skip the preliminaries and go to the query profiles; for the others, there is some explanation first.</p> <p>From the TPC-H specification:</p> <blockquote> <code><pre> SELECT TOP 100 c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice, SUM ( l_quantity ) FROM customer, orders, lineitem WHERE o_orderkey IN ( SELECT l_orderkey FROM lineitem GROUP BY l_orderkey HAVING SUM ( l_quantity ) > 312 ) AND c_custkey = o_custkey AND o_orderkey = l_orderkey GROUP BY c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice ORDER BY o_totalprice DESC, o_orderdate </pre> </code> </blockquote> <p>The intent of the query is to return <code>order</code> and <code>customer</code> information for cases where an <code>order</code> involves a large quantity of <code>items</code>, with highest-value <code>orders</code> first. </p> <p>We note that the only restriction in the query is the one on the <code>SUM</code> of <code>l_quantity</code> in the <code>IN</code> subquery. Everything else is a full scan or a <code><a href="http://dbpedia.org/resource/Join_(SQL)" id="link-id0x19919558">JOIN</a></code> on a <a href="http://dbpedia.org/page/Foreign_key" id="link-id0x1fc089f8">foreign key</a>.</p> <p>Now, the first query optimization rule of thumb could be summarized as <i>start from the small.</i> <i>Small</i> here means something that is restricted; it does not mean small table. Smallest is the one from which the highest percentage is dropped via a condition that does not depend on other tables.</p> <p>The next rule of thumb is to try starting from the large, if the large has a restricting <code>join</code>; for example, scan all the <code>lineitems</code> and hash join to <code>parts</code> that are green and of a given brand. In this case, the idea is to make a hash table from the small side and sequentially scan the large side, dropping everything that does not match something in the hash table.</p> <p>The only restriction here is on <code>orders</code> via a <code>join</code> on <code>lineitem</code>. So, the <code>IN</code> subquery can be flattened, so as to read like --</p> <blockquote> <code><pre> SELECT ... FROM ( SELECT l_orderkey, SUM ( l_quantity ) FROM lineitem GROUP BY l_orderkey HAVING SUM ( l_quantity ) > 312 ) f, orders, customer, lineitem WHERE f.l_orderkey = o_orderkey .... </pre> </code> </blockquote> <p>The above (left to right) is the best <code>JOIN</code> order for this type of plan. We start from the restriction, and for all the rest the <code>JOIN</code> is foreign key to primary key, sometimes <code>n:1</code> (<code>orders</code> to <code>customer</code>), sometimes <code>1:n</code> (<code>orders</code> to <code>lineitem</code>). A <code>1:n</code> is usually best by index; an <code>n:1</code> can be better by hash if there are enough tuples on the n side to make it worthwhile to build the hash table.</p> <p>We note that the first <code>GROUP BY</code> makes a very large number of groups, e.g., 150M at 100 Gtriple scale. We also note that if <code>lineitem</code> is ordered so that the <code>lineitems</code> of a single <code>order</code> are together, the <code>GROUP BY</code> is ordered. In other words, once you have seen a specific value of <code>l_orderkey</code> change to the next, you will not see the old value again. In this way, the groups do not have to be remembered for all time. The <code>GROUP BY</code> produces a stream of results as the scan of <code>lineitem</code> proceeds.</p> <p>Considering vectored execution, the <code>GROUP BY</code> does remember a bunch of groups, up to a vector size worth, so that output from the <code>GROUP BY</code> is done in large enough batches, not a tuple at a time.</p> <p>Considering parallelization, the scan of <code>lineitem</code> must be split in such a way that all <code>lineitems</code> with the same <code>l_orderkey</code> get processed by the same thread. If this is the case, all threads will produce an independent stream of results that is guaranteed to need no merge with the output of another thread.</p> <p>So, we can try this:</p> <blockquote> <code><pre> { time 6e-06% fanout 1 input 1 rows time 4.5% fanout 1 input 1 rows { hash filler </pre> </code> </blockquote> <p> <i>-- Make a hash table from <code>c_custkey</code> to <code>c_name</code> </i> </p> <blockquote> <code><pre> time 0.99% fanout 1.5e+07 input 1 rows CUSTOMER 1.5e+07 rows(.C_CUSTKEY, .C_NAME) time 0.81% fanout 0 input 1.5e+07 rows Sort hf 35 (.C_CUSTKEY) -> (.C_NAME) } time 2.2e-05% fanout 1 input 1 rows time 1.6e-05% fanout 1 input 1 rows { fork time 5.2e-06% fanout 1 input 1 rows { fork </pre> </code> </blockquote> <p> <i>-- Scan <code>lineitem</code> </i> </p> <blockquote> <code><pre> time 10% fanout 6.00038e+08 input 1 rows LINEITEM 6e+08 rows(t5.L_ORDERKEY, t5.L_QUANTITY) </pre> </code> </blockquote> <p> <i>-- Ordered <code>GROUP BY</code> (streaming with duplicates) </i> </p> <blockquote> <code><pre> time 73% fanout 1.17743e-05 input 6.00038e+08 rows Sort streaming with duplicates (t5.L_ORDERKEY) -> (t5.L_QUANTITY) </pre> </code> </blockquote> <p> <i>-- The ordered aggregation above emits a batch of results every so often, having accumulated 20K or so groups (<code>DISTINCT l_orderkey</code>'s)</i> </p> <p> <i>-- The operator below reads the batch and sends it onward, the <code>GROUP BY</code> hash table for the next batch.</i> </p> <blockquote> <code><pre> time 10% fanout 21231.4 input 7065 rows group by read node (t5.L_ORDERKEY, aggregate) END Node After test: 0: if (aggregate > 312 ) then 4 else 5 unkn 5 4: BReturn 1 5: BReturn 0 After code: 0: L_ORDERKEY := := artm t5.L_ORDERKEY 4: BReturn 0 </pre> </code> </blockquote> <p> <i>-- This marks the end of the flattened <code>IN</code> subquery. 1063 out of 150M groups survive the test on the <code>SUM</code> of <code>l_quantity</code>.</i> </p> <p> <i>-- The main difficulty of Q18 is guessing that this condition is this selective.</i> </p> <blockquote> <code><pre> time 0.0013% fanout 1 input 1063 rows Subquery Select(L_ORDERKEY) time 0.058% fanout 1 input 1063 rows ORDERS unq 0.97 rows (.O_CUSTKEY, .O_ORDERKEY, .O_ORDERDATE, .O_TOTALPRICE) inlined O_ORDERKEY = L_ORDERKEY hash partition+bloom by 42 (tmp)hash join merged always card 0.99 -> (.C_NAME) time 0.0029% fanout 1 input 1063 rows Hash source 35 merged into ts 0.99 rows(.O_CUSTKEY) -> (.C_NAME) After code: 0: .C_CUSTKEY := := artm .O_CUSTKEY 4: BReturn 0 time 0.018% fanout 7 input 1063 rows LINEITEM 4.3 rows(.L_QUANTITY) inlined L_ORDERKEY = .O_ORDERKEY time 0.011% fanout 0 input 7441 rows Sort (.C_CUSTKEY, .O_ORDERKEY) -> (.L_QUANTITY, .O_TOTALPRICE, .O_ORDERDATE, .C_NAME) } time 0.00026% fanout 1063 input 1 rows group by read node (.C_CUSTKEY, .O_ORDERKEY, aggregate, .O_TOTALPRICE, .O_ORDERDATE, .C_NAME) time 0.00061% fanout 0 input 1063 rows Sort (.O_TOTALPRICE, .O_ORDERDATE) -> (.C_NAME, .C_CUSTKEY, .O_ORDERKEY, aggregate) } time 1.7e-05% fanout 100 input 1 rows top order by read (.C_NAME, .C_CUSTKEY, .O_ORDERKEY, .O_ORDERDATE, .O_TOTALPRICE, aggregate) time 1.2e-06% fanout 0 input 100 rows Select (.C_NAME, .C_CUSTKEY, .O_ORDERKEY, .O_ORDERDATE, .O_TOTALPRICE, aggregate) } 6351 msec 1470% cpu, 2151 rnd 6.14898e+08 seq 0.185874% same seg 1.57993% same pg </pre> </code> </blockquote> <p>What is wrong with this? The result is not bad, in the ballpark with <a href="http://dbpedia.org/resource/Vectorwise" id="link-id0x257913e8">VectorWise</a> published results (4.9s on a slightly faster box), but better is possible. We note that there is a hash join from <code>orders</code> to <code>customer</code>. Only 1K <code>customers</code> of 15M get hit. The whole hash table of 15M entries is built in vain. Let's cheat and declare the <code>join</code> to be by <code>index</code>. Cheats like this are not allowed in an official run but here we are just looking. So we change the mention of the <code>customer</code> table in the <code>FROM</code> clause from <code>FROM ... customer, ...</code> to <code>FROM ... customer TABLE OPTION (loop), ...</code> </p> <blockquote> <code><pre> { time 1.4e-06% fanout 1 input 1 rows time 9e-07% fanout 1 input 1 rows </pre> </code> </blockquote> <p> <i>-- Here was the hash build in the previous plan; now we start direct with the scan of <code>lineitem</code>.</i> </p> <blockquote> <code><pre> time 2.2e-06% fanout 1 input 1 rows { fork time 2.3e-06% fanout 1 input 1 rows { fork time 11% fanout 6.00038e+08 input 1 rows LINEITEM 6e+08 rows(t5.L_ORDERKEY, t5.L_QUANTITY) time 78% fanout 1.17743e-05 input 6.00038e+08 rows Sort streaming with duplicates (t5.L_ORDERKEY) -> (t5.L_QUANTITY) time 11% fanout 21231.4 input 7065 rows group by read node (t5.L_ORDERKEY, aggregate) END Node After test: 0: if (aggregate > 312 ) then 4 else 5 unkn 5 4: BReturn 1 5: BReturn 0 After code: 0: L_ORDERKEY := := artm t5.L_ORDERKEY 4: BReturn 0 time 0.0014% fanout 1 input 1063 rows Subquery Select(L_ORDERKEY) time 0.051% fanout 1 input 1063 rows ORDERS unq 0.97 rows (.O_CUSTKEY, .O_ORDERKEY, .O_ORDERDATE, .O_TOTALPRICE) inlined O_ORDERKEY = L_ORDERKEY </pre> </code> </blockquote> <p> <i>-- We note that getting the 1063 <code>customers</code> by index takes no time, and there is no hash table to build </i> </p> <blockquote> <code><pre> time 0.023% fanout 1 input 1063 rows CUSTOMER unq 0.99 rows (.C_CUSTKEY, .C_NAME) inlined C_CUSTKEY = .O_CUSTKEY time 0.021% fanout 7 input 1063 rows LINEITEM 4.3 rows(.L_QUANTITY) inlined L_ORDERKEY = k_.O_ORDERKEY </pre> </code> </blockquote> <p> <i>-- The rest is identical to the previous plan, cut for brevity </i> </p> <blockquote> <code><pre> 3852 msec 2311% cpu, 3213 rnd 5.99907e+08 seq 0.124456% same seg 1.08899% same pg Compilation: 1 msec 0 reads 0% read 0 messages 0% clw </pre> </code> </blockquote> <p>We save over 2s of real time. But the problem is how to know that very few <code>customers</code> will be hit. One could make a calculation that <code>l_quantity</code> is between 1 and 50, and that an <code>order</code> has an average of 4 <code>lineitems</code> with a maximum of 7. For the <code>SUM</code> to be over 312, only <code>orders</code> with 7 <code>lineitems</code> are eligible, and even so the <code>l_quantities</code> must all be high. Assuming flat distributions, which here happens to be the case, one could estimate that the condition selects very few <code>orders</code>. The problem is that real data with this kind of regularity is sight unseen, so such a trick, while allowed, would just work for benchmarks.</p> <p align="center">* * * * *</p> <p>As it happens, there is a better way. We also note that the query selects the <code>TOP 100 orders</code> with the highest <code>o_totalprice</code>. This is a very common pattern; there is almost always a <code>TOP k</code> clause in analytics queries unless they <code>GROUP BY</code> something that is known to be of low cardinality, like nation or year.</p> <p>If the ordering falls on a grouping column, as soon as there are enough groups generated to fill a <code>TOP 100</code>, one can take the lowest <code>o_totalprice</code> as a limit and add this into the query as an extra restriction. Every time the <code>TOP 100</code> changes, the condition becomes more selective, as the 100th highest <code>o_totalprice</code> increases.</p> <p>Sometimes the ordering falls on the aggregation result, which is not known until the aggregation is finished. However, in lookup-style queries, it is common to take the latest-so-many events or just the <code>TOP k</code> items by some metric. In these cases, pushing the <code>TOP k</code> restriction down into the selection always works.</p> <p>So, we try this:</p> <blockquote> <code><pre> { time 4e-06% fanout 1 input 1 rows time 6.1e-06% fanout 1 input 1 rows { fork </pre> </code> </blockquote> <p> <i>-- The plan begins with <code>orders</code>, as we now expect a selection on <code>o_totalprice</code> </i> </p> <p> <i>-- We see that out of 150M <code>orders</code>, a little over 10M survive the <code>o_totalprice</code> selection, which gets more restrictive as the query proceeds. </i> </p> <blockquote> <code><pre> time 33% fanout 1.00628e+07 input 1 rows ORDERS 4.3e+04 rows(.O_TOTALPRICE, .O_ORDERKEY, .O_CUSTKEY, .O_ORDERDATE) top k on O_TOTALPRICE time 32% fanout 3.50797e-05 input 1.00628e+07 rows END Node After test: 0: if ({ </pre> </code> </blockquote> <p> <i>-- The <code>IN</code> subquery is here kept as a subquery, not flattened. </i> </p> <blockquote> <code> <pre> time 0.42% fanout 1 input 1.00628e+07 rows time 11% fanout 4.00136 input 1.00628e+07 rows LINEITEM 4 rows(.L_ORDERKEY, .L_QUANTITY) inlined L_ORDERKEY = k_.O_ORDERKEY time 21% fanout 2.55806e-05 input 4.02649e+07 rows Sort streaming with duplicates (set_ctr, .L_ORDERKEY) -> (.L_QUANTITY) time 2.4% fanout 9769.72 input 1030 rows group by read node (gb_set_no, .L_ORDERKEY, aggregate) END Node After test: 0: if (aggregate > 312 ) then 4 else 5 unkn 5 4: BReturn 1 5: BReturn 0 time 0.00047% fanout 0 input 353 rows Subquery Select( <none> ) } ) then 4 else 5 unkn 5 4: BReturn 1 5: BReturn 0 </none> </pre> </code> </blockquote> <p> <i>-- Here we see that fewer <code>customers</code> are accessed than in the non-<code>TOP k</code> plans, since there is an extra cut on <code>o_totalprice</code> that takes effect earlier </i> </p> <blockquote> <code><pre> time 0.013% fanout 1 input 353 rows CUSTOMER unq 1 rows (.C_CUSTKEY, .C_NAME) inlined C_CUSTKEY = k_.O_CUSTKEY time 0.0079% fanout 7 input 353 rows LINEITEM 4 rows(.L_QUANTITY) inlined L_ORDERKEY = k_.O_ORDERKEY time 0.0063% fanout 0.0477539 input 2471 rows Sort streaming with duplicates (.C_CUSTKEY, .O_ORDERKEY) -> (.L_QUANTITY, .O_TOTALPRICE, .O_ORDERDATE, .C_NAME) time 0.0088% fanout 2.99153 input 118 rows group by read node (.C_CUSTKEY, .O_ORDERKEY, aggregate, .O_TOTALPRICE, .O_ORDERDATE, .C_NAME) time 0.0063% fanout 0 input 353 rows Sort (.O_TOTALPRICE, .O_ORDERDATE) -> (.C_NAME, .C_CUSTKEY, .O_ORDERKEY, aggregate) } time 8.5e-05% fanout 100 input 1 rows top order by read (.C_NAME, .C_CUSTKEY, .O_ORDERKEY, .O_ORDERDATE, .O_TOTALPRICE, aggregate) time 2.7e-06% fanout 0 input 100 rows Select (.C_NAME, .C_CUSTKEY, .O_ORDERKEY, .O_ORDERDATE, .O_TOTALPRICE, aggregate) } 949 msec 2179% cpu, 1.00486e+07 rnd 4.71013e+07 seq 99.9267% same seg 0.0318055% same pg </pre> </code> </blockquote> <p>Here we see that the time is about 4x better than with the cheat version. We note that about 10M of 1.5e8 <code>orders</code> get considered. After going through the first 10% or so of <code>orders</code>, there is a <code>TOP 100</code>, and a condition on <code>o_totalprice</code> that will drop most <code>orders</code> can be introduced.</p> <p>If we set the condition on the <code>SUM</code> of <code>quantity</code> so that no <code>orders</code> match, there is no <code>TOP k</code> at any point, and we get a time of 6.8s, which is a little worse than the initial time with the flattened <code>IN</code>. But since the <code>TOP k</code> trick does not allocate memory, it is relatively safe even in cases where it does not help.</p> <p>We can argue that the <code>TOP k</code> pushdown trick is more robust than guessing the selectivity of a <code>SUM</code> of <code>l_quantity</code>. Further, it applies to a broad range of lookup queries, while the <code>SUM</code> trick applies to only TPC-H Q18, or close enough. Thus, the <code>TOP k</code> trick is safer and more generic. </p> <p>We are approaching the end of the TPC-H blog series, with still two families of tricks to consider, namely, moving predicates between subqueries, and late projection. After this we will look at results and the overall picture.</p> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2248e18"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x241bd578"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x5dcfd38"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x7fe33c8"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x23036cb8"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x113fd508"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x23446f28"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x148aee58"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> In Hoc Signo Vinces (part 9): TPC-H: TPC-H Q18, Ordered Aggregation, and Top K<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aabf0e42c58"> In Hoc Signo Vinces (part 10): TPC-H: TPC-H Q9, Q17, Q20 - Predicate Games</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac1e6c0338"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac2bb7d2f8"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac6eb08118"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac304469f8"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x1a8e5d48"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x17f75d48"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0xfe9fe98"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aac657476a8"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aac0407cd98"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:10:34.804262-04:00
In Hoc Signo Vinces (part 9 of n): TPC-H Q18, Ordered Aggregation, and Top K
http://www.openlinksw.com:443/weblog/oerling/?date=2014-03-17#1779
2014-03-17T16:41:20Z
<p>We will here return to polishing the cutting edge, the high geekcraft of database. We will look at more of the wonders of <a href="http://www.tpc.org/tpch/" id="link-id0x256ad6b8">TPC-H</a> and cover two more tricks. The experts can skip the preliminaries and go to the query profiles; for the others, there is some explanation first.</p> <p>From the TPC-H specification:</p> <blockquote> <code><pre> SELECT TOP 100 c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice, SUM ( l_quantity ) FROM customer, orders, lineitem WHERE o_orderkey IN ( SELECT l_orderkey FROM lineitem GROUP BY l_orderkey HAVING SUM ( l_quantity ) > 312 ) AND c_custkey = o_custkey AND o_orderkey = l_orderkey GROUP BY c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice ORDER BY o_totalprice DESC, o_orderdate </pre> </code> </blockquote> <p>The intent of the query is to return <code>order</code> and <code>customer</code> information for cases where an <code>order</code> involves a large quantity of <code>items</code>, with highest-value <code>orders</code> first. </p> <p>We note that the only restriction in the query is the one on the <code>SUM</code> of <code>l_quantity</code> in the <code>IN</code> subquery. Everything else is a full scan or a <code><a href="http://dbpedia.org/resource/Join_(SQL)" id="link-id0x19919558">JOIN</a></code> on a <a href="http://dbpedia.org/page/Foreign_key" id="link-id0x1fc089f8">foreign key</a>.</p> <p>Now, the first query optimization rule of thumb could be summarized as <i>start from the small.</i> <i>Small</i> here means something that is restricted; it does not mean small table. Smallest is the one from which the highest percentage is dropped via a condition that does not depend on other tables.</p> <p>The next rule of thumb is to try starting from the large, if the large has a restricting <code>join</code>; for example, scan all the <code>lineitems</code> and hash join to <code>parts</code> that are green and of a given brand. In this case, the idea is to make a hash table from the small side and sequentially scan the large side, dropping everything that does not match something in the hash table.</p> <p>The only restriction here is on <code>orders</code> via a <code>join</code> on <code>lineitem</code>. So, the <code>IN</code> subquery can be flattened, so as to read like --</p> <blockquote> <code><pre> SELECT ... FROM ( SELECT l_orderkey, SUM ( l_quantity ) FROM lineitem GROUP BY l_orderkey HAVING SUM ( l_quantity ) > 312 ) f, orders, customer, lineitem WHERE f.l_orderkey = o_orderkey .... </pre> </code> </blockquote> <p>The above (left to right) is the best <code>JOIN</code> order for this type of plan. We start from the restriction, and for all the rest the <code>JOIN</code> is foreign key to primary key, sometimes <code>n:1</code> (<code>orders</code> to <code>customer</code>), sometimes <code>1:n</code> (<code>orders</code> to <code>lineitem</code>). A <code>1:n</code> is usually best by index; an <code>n:1</code> can be better by hash if there are enough tuples on the n side to make it worthwhile to build the hash table.</p> <p>We note that the first <code>GROUP BY</code> makes a very large number of groups, e.g., 150M at 100 Gtriple scale. We also note that if <code>lineitem</code> is ordered so that the <code>lineitems</code> of a single <code>order</code> are together, the <code>GROUP BY</code> is ordered. In other words, once you have seen a specific value of <code>l_orderkey</code> change to the next, you will not see the old value again. In this way, the groups do not have to be remembered for all time. The <code>GROUP BY</code> produces a stream of results as the scan of <code>lineitem</code> proceeds.</p> <p>Considering vectored execution, the <code>GROUP BY</code> does remember a bunch of groups, up to a vector size worth, so that output from the <code>GROUP BY</code> is done in large enough batches, not a tuple at a time.</p> <p>Considering parallelization, the scan of <code>lineitem</code> must be split in such a way that all <code>lineitems</code> with the same <code>l_orderkey</code> get processed by the same thread. If this is the case, all threads will produce an independent stream of results that is guaranteed to need no merge with the output of another thread.</p> <p>So, we can try this:</p> <blockquote> <code><pre> { time 6e-06% fanout 1 input 1 rows time 4.5% fanout 1 input 1 rows { hash filler </pre> </code> </blockquote> <p> <i>-- Make a hash table from <code>c_custkey</code> to <code>c_name</code> </i> </p> <blockquote> <code><pre> time 0.99% fanout 1.5e+07 input 1 rows CUSTOMER 1.5e+07 rows(.C_CUSTKEY, .C_NAME) time 0.81% fanout 0 input 1.5e+07 rows Sort hf 35 (.C_CUSTKEY) -> (.C_NAME) } time 2.2e-05% fanout 1 input 1 rows time 1.6e-05% fanout 1 input 1 rows { fork time 5.2e-06% fanout 1 input 1 rows { fork </pre> </code> </blockquote> <p> <i>-- Scan <code>lineitem</code> </i> </p> <blockquote> <code><pre> time 10% fanout 6.00038e+08 input 1 rows LINEITEM 6e+08 rows(t5.L_ORDERKEY, t5.L_QUANTITY) </pre> </code> </blockquote> <p> <i>-- Ordered <code>GROUP BY</code> (streaming with duplicates) </i> </p> <blockquote> <code><pre> time 73% fanout 1.17743e-05 input 6.00038e+08 rows Sort streaming with duplicates (t5.L_ORDERKEY) -> (t5.L_QUANTITY) </pre> </code> </blockquote> <p> <i>-- The ordered aggregation above emits a batch of results every so often, having accumulated 20K or so groups (<code>DISTINCT l_orderkey</code>'s)</i> </p> <p> <i>-- The operator below reads the batch and sends it onward, the <code>GROUP BY</code> hash table for the next batch.</i> </p> <blockquote> <code><pre> time 10% fanout 21231.4 input 7065 rows group by read node (t5.L_ORDERKEY, aggregate) END Node After test: 0: if (aggregate > 312 ) then 4 else 5 unkn 5 4: BReturn 1 5: BReturn 0 After code: 0: L_ORDERKEY := := artm t5.L_ORDERKEY 4: BReturn 0 </pre> </code> </blockquote> <p> <i>-- This marks the end of the flattened <code>IN</code> subquery. 1063 out of 150M groups survive the test on the <code>SUM</code> of <code>l_quantity</code>.</i> </p> <p> <i>-- The main difficulty of Q18 is guessing that this condition is this selective.</i> </p> <blockquote> <code><pre> time 0.0013% fanout 1 input 1063 rows Subquery Select(L_ORDERKEY) time 0.058% fanout 1 input 1063 rows ORDERS unq 0.97 rows (.O_CUSTKEY, .O_ORDERKEY, .O_ORDERDATE, .O_TOTALPRICE) inlined O_ORDERKEY = L_ORDERKEY hash partition+bloom by 42 (tmp)hash join merged always card 0.99 -> (.C_NAME) time 0.0029% fanout 1 input 1063 rows Hash source 35 merged into ts 0.99 rows(.O_CUSTKEY) -> (.C_NAME) After code: 0: .C_CUSTKEY := := artm .O_CUSTKEY 4: BReturn 0 time 0.018% fanout 7 input 1063 rows LINEITEM 4.3 rows(.L_QUANTITY) inlined L_ORDERKEY = .O_ORDERKEY time 0.011% fanout 0 input 7441 rows Sort (.C_CUSTKEY, .O_ORDERKEY) -> (.L_QUANTITY, .O_TOTALPRICE, .O_ORDERDATE, .C_NAME) } time 0.00026% fanout 1063 input 1 rows group by read node (.C_CUSTKEY, .O_ORDERKEY, aggregate, .O_TOTALPRICE, .O_ORDERDATE, .C_NAME) time 0.00061% fanout 0 input 1063 rows Sort (.O_TOTALPRICE, .O_ORDERDATE) -> (.C_NAME, .C_CUSTKEY, .O_ORDERKEY, aggregate) } time 1.7e-05% fanout 100 input 1 rows top order by read (.C_NAME, .C_CUSTKEY, .O_ORDERKEY, .O_ORDERDATE, .O_TOTALPRICE, aggregate) time 1.2e-06% fanout 0 input 100 rows Select (.C_NAME, .C_CUSTKEY, .O_ORDERKEY, .O_ORDERDATE, .O_TOTALPRICE, aggregate) } 6351 msec 1470% cpu, 2151 rnd 6.14898e+08 seq 0.185874% same seg 1.57993% same pg </pre> </code> </blockquote> <p>What is wrong with this? The result is not bad, in the ballpark with <a href="http://dbpedia.org/resource/Vectorwise" id="link-id0x257913e8">VectorWise</a> published results (4.9s on a slightly faster box), but better is possible. We note that there is a hash join from <code>orders</code> to <code>customer</code>. Only 1K <code>customers</code> of 15M get hit. The whole hash table of 15M entries is built in vain. Let's cheat and declare the <code>join</code> to be by <code>index</code>. Cheats like this are not allowed in an official run but here we are just looking. So we change the mention of the <code>customer</code> table in the <code>FROM</code> clause from <code>FROM ... customer, ...</code> to <code>FROM ... customer TABLE OPTION (loop), ...</code> </p> <blockquote> <code><pre> { time 1.4e-06% fanout 1 input 1 rows time 9e-07% fanout 1 input 1 rows </pre> </code> </blockquote> <p> <i>-- Here was the hash build in the previous plan; now we start direct with the scan of <code>lineitem</code>.</i> </p> <blockquote> <code><pre> time 2.2e-06% fanout 1 input 1 rows { fork time 2.3e-06% fanout 1 input 1 rows { fork time 11% fanout 6.00038e+08 input 1 rows LINEITEM 6e+08 rows(t5.L_ORDERKEY, t5.L_QUANTITY) time 78% fanout 1.17743e-05 input 6.00038e+08 rows Sort streaming with duplicates (t5.L_ORDERKEY) -> (t5.L_QUANTITY) time 11% fanout 21231.4 input 7065 rows group by read node (t5.L_ORDERKEY, aggregate) END Node After test: 0: if (aggregate > 312 ) then 4 else 5 unkn 5 4: BReturn 1 5: BReturn 0 After code: 0: L_ORDERKEY := := artm t5.L_ORDERKEY 4: BReturn 0 time 0.0014% fanout 1 input 1063 rows Subquery Select(L_ORDERKEY) time 0.051% fanout 1 input 1063 rows ORDERS unq 0.97 rows (.O_CUSTKEY, .O_ORDERKEY, .O_ORDERDATE, .O_TOTALPRICE) inlined O_ORDERKEY = L_ORDERKEY </pre> </code> </blockquote> <p> <i>-- We note that getting the 1063 <code>customers</code> by index takes no time, and there is no hash table to build </i> </p> <blockquote> <code><pre> time 0.023% fanout 1 input 1063 rows CUSTOMER unq 0.99 rows (.C_CUSTKEY, .C_NAME) inlined C_CUSTKEY = .O_CUSTKEY time 0.021% fanout 7 input 1063 rows LINEITEM 4.3 rows(.L_QUANTITY) inlined L_ORDERKEY = k_.O_ORDERKEY </pre> </code> </blockquote> <p> <i>-- The rest is identical to the previous plan, cut for brevity </i> </p> <blockquote> <code><pre> 3852 msec 2311% cpu, 3213 rnd 5.99907e+08 seq 0.124456% same seg 1.08899% same pg Compilation: 1 msec 0 reads 0% read 0 messages 0% clw </pre> </code> </blockquote> <p>We save over 2s of real time. But the problem is how to know that very few <code>customers</code> will be hit. One could make a calculation that <code>l_quantity</code> is between 1 and 50, and that an <code>order</code> has an average of 4 <code>lineitems</code> with a maximum of 7. For the <code>SUM</code> to be over 312, only <code>orders</code> with 7 <code>lineitems</code> are eligible, and even so the <code>l_quantities</code> must all be high. Assuming flat distributions, which here happens to be the case, one could estimate that the condition selects very few <code>orders</code>. The problem is that real data with this kind of regularity is sight unseen, so such a trick, while allowed, would just work for benchmarks.</p> <p align="center">* * * * *</p> <p>As it happens, there is a better way. We also note that the query selects the <code>TOP 100 orders</code> with the highest <code>o_totalprice</code>. This is a very common pattern; there is almost always a <code>TOP k</code> clause in analytics queries unless they <code>GROUP BY</code> something that is known to be of low cardinality, like nation or year.</p> <p>If the ordering falls on a grouping column, as soon as there are enough groups generated to fill a <code>TOP 100</code>, one can take the lowest <code>o_totalprice</code> as a limit and add this into the query as an extra restriction. Every time the <code>TOP 100</code> changes, the condition becomes more selective, as the 100th highest <code>o_totalprice</code> increases.</p> <p>Sometimes the ordering falls on the aggregation result, which is not known until the aggregation is finished. However, in lookup-style queries, it is common to take the latest-so-many events or just the <code>TOP k</code> items by some metric. In these cases, pushing the <code>TOP k</code> restriction down into the selection always works.</p> <p>So, we try this:</p> <blockquote> <code><pre> { time 4e-06% fanout 1 input 1 rows time 6.1e-06% fanout 1 input 1 rows { fork </pre> </code> </blockquote> <p> <i>-- The plan begins with <code>orders</code>, as we now expect a selection on <code>o_totalprice</code> </i> </p> <p> <i>-- We see that out of 150M <code>orders</code>, a little over 10M survive the <code>o_totalprice</code> selection, which gets more restrictive as the query proceeds. </i> </p> <blockquote> <code><pre> time 33% fanout 1.00628e+07 input 1 rows ORDERS 4.3e+04 rows(.O_TOTALPRICE, .O_ORDERKEY, .O_CUSTKEY, .O_ORDERDATE) top k on O_TOTALPRICE time 32% fanout 3.50797e-05 input 1.00628e+07 rows END Node After test: 0: if ({ </pre> </code> </blockquote> <p> <i>-- The <code>IN</code> subquery is here kept as a subquery, not flattened. </i> </p> <blockquote> <code> <pre> time 0.42% fanout 1 input 1.00628e+07 rows time 11% fanout 4.00136 input 1.00628e+07 rows LINEITEM 4 rows(.L_ORDERKEY, .L_QUANTITY) inlined L_ORDERKEY = k_.O_ORDERKEY time 21% fanout 2.55806e-05 input 4.02649e+07 rows Sort streaming with duplicates (set_ctr, .L_ORDERKEY) -> (.L_QUANTITY) time 2.4% fanout 9769.72 input 1030 rows group by read node (gb_set_no, .L_ORDERKEY, aggregate) END Node After test: 0: if (aggregate > 312 ) then 4 else 5 unkn 5 4: BReturn 1 5: BReturn 0 time 0.00047% fanout 0 input 353 rows Subquery Select( <none> ) } ) then 4 else 5 unkn 5 4: BReturn 1 5: BReturn 0 </none> </pre> </code> </blockquote> <p> <i>-- Here we see that fewer <code>customers</code> are accessed than in the non-<code>TOP k</code> plans, since there is an extra cut on <code>o_totalprice</code> that takes effect earlier </i> </p> <blockquote> <code><pre> time 0.013% fanout 1 input 353 rows CUSTOMER unq 1 rows (.C_CUSTKEY, .C_NAME) inlined C_CUSTKEY = k_.O_CUSTKEY time 0.0079% fanout 7 input 353 rows LINEITEM 4 rows(.L_QUANTITY) inlined L_ORDERKEY = k_.O_ORDERKEY time 0.0063% fanout 0.0477539 input 2471 rows Sort streaming with duplicates (.C_CUSTKEY, .O_ORDERKEY) -> (.L_QUANTITY, .O_TOTALPRICE, .O_ORDERDATE, .C_NAME) time 0.0088% fanout 2.99153 input 118 rows group by read node (.C_CUSTKEY, .O_ORDERKEY, aggregate, .O_TOTALPRICE, .O_ORDERDATE, .C_NAME) time 0.0063% fanout 0 input 353 rows Sort (.O_TOTALPRICE, .O_ORDERDATE) -> (.C_NAME, .C_CUSTKEY, .O_ORDERKEY, aggregate) } time 8.5e-05% fanout 100 input 1 rows top order by read (.C_NAME, .C_CUSTKEY, .O_ORDERKEY, .O_ORDERDATE, .O_TOTALPRICE, aggregate) time 2.7e-06% fanout 0 input 100 rows Select (.C_NAME, .C_CUSTKEY, .O_ORDERKEY, .O_ORDERDATE, .O_TOTALPRICE, aggregate) } 949 msec 2179% cpu, 1.00486e+07 rnd 4.71013e+07 seq 99.9267% same seg 0.0318055% same pg </pre> </code> </blockquote> <p>Here we see that the time is about 4x better than with the cheat version. We note that about 10M of 1.5e8 <code>orders</code> get considered. After going through the first 10% or so of <code>orders</code>, there is a <code>TOP 100</code>, and a condition on <code>o_totalprice</code> that will drop most <code>orders</code> can be introduced.</p> <p>If we set the condition on the <code>SUM</code> of <code>quantity</code> so that no <code>orders</code> match, there is no <code>TOP k</code> at any point, and we get a time of 6.8s, which is a little worse than the initial time with the flattened <code>IN</code>. But since the <code>TOP k</code> trick does not allocate memory, it is relatively safe even in cases where it does not help.</p> <p>We can argue that the <code>TOP k</code> pushdown trick is more robust than guessing the selectivity of a <code>SUM</code> of <code>l_quantity</code>. Further, it applies to a broad range of lookup queries, while the <code>SUM</code> trick applies to only TPC-H Q18, or close enough. Thus, the <code>TOP k</code> trick is safer and more generic. </p> <p>We are approaching the end of the TPC-H blog series, with still two families of tricks to consider, namely, moving predicates between subqueries, and late projection. After this we will look at results and the overall picture.</p> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> (TPC-H) Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2248e18"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x241bd578"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x5dcfd38"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x7fe33c8"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x23036cb8"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x113fd508"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x23446f28"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x148aee58"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs</a> </li> <li> In Hoc Signo Vinces (part 9): TPC-H: TPC-H Q18, Ordered Aggregation, and Top K<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac0ff1b258"> In Hoc Signo Vinces (part 10): TPC-H: TPC-H Q9, Q17, Q20 - Predicate Games</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac1ed06068"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac112545d8"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac6ffdaa48"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0xb138588"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x18371518"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aabe9ec1af8"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aac00f9a958"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aabcac19228"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aabd0bbd438"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:07:23.062020-04:00
LOD2 Plenary and Open Data Meet-up in Mannheim
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-02-26#1778
2014-02-26T20:38:14Z
<p>There was a <a href="http://lod2.eu/" id="link-id0x2d6bfc28">LOD2</a> plenary meeting hosted by <a href="http://www.bizer.de/" id="link-id0x296a4c18">Chris Bizer</a> from the <a href="http://lod2.eu/Partner/fub.html" id="link-id0x2ec8dd58">University of Mannheim</a> this week.</p> <p>The plenary meeting was preceded by a <a href="http://www.meetup.com/OpenKnowledgeFoundation/Mannheim-DE/1092882/" id="link-id0x2aac23993228">Linked Open Data Meetup</a> with talks from <a href="http://dbpedia.org/page/Springer_Science+Business_Media" id="link-id0x2bc7f2a8">Springer</a>, <a href="http://dbpedia.org/resource/Fluid_Operations" id="link-id0x1b077a48">fluid Operations</a>, and several LOD2 partners (<a href="http://lod2.eu/Partner/ulei.html" id="link-id0x2aac269eb678">Universität Leipzig</a>, <a href="http://lod2.eu/Partner/fub.html" id="link-id0x2aac26cf1fb8">University of Mannheim</a>, the <a href="http://lod2.eu/Partner/swc.html" id="link-id0x2aac25abb8f8">Semantic Web Company</a>, and <a href="http://lod2.eu/Partner/wkd.html" id="link-id0x300c2308">Wolters Kluwer Deutschland GmbH (WKD)</a>).</p> <p> <a href="http://lod2.eu/Partner/wkd.html" id="link-id0x300c2308">Wolters Kluwer Deutschland GmbH (WKD)</a> gave a presentation on the content production pipeline of their legal publications and their experiences in incorporating LOD2 technologies for content enrichment. This is a very successful LOD2 use case and demonstrates the value of <a href="http://kingsley.idehen.net/describe/?url=http%3A%2F%2Fkingsley.idehen.net%2FDAV%2Fhome%2Fkidehen%2FPublic%2FLinked%2520Data%2520Documents%2FGlossaryOfTerms.ttl%23LinkedData" id="link-id0x10705738">linked data</a> for the information industry.</p> <p>Springer gave a talk about their interest in linked data for enriching the Lecture Notes in Computer Science product. Also conference proceedings could be enhanced with structured <a href="http://www.openlinksw.com/href" id="link-id0x19279a98">metadata</a> in <a href="http://dbpedia.org/page/Resource_Description_Framework" id="link-id0x2c1e8c38">RDF</a>. I asked about nanopublications. The comment was that content authors might perceive nanopublications as an extra imposition. On the other hand, in the life sciences field there is a lot of enthusiasm for the idea. We will see; anyway, biology will likely lead the way for nanopublications. I referred <a href="http://www.linkedin.com/in/aliaksandrbirukou" id="link-id0x2ba061b8">Aliaksandr Birukou</a> of Springer to the companies <a href="http://www.linkedin.com/company/euretos" id="link-id0x2bcc5dc8">Euretos</a> and its parent S&T in Delft, Netherlands, and to <a href="http://dbpedia.org/resource/Barend_Mons" id="link-id0x2e4a9918">Barend Mons</a>, scientific director of <a href="http://dbpedia.org/page/Netherlands_Bioinformatics_Centre" id="link-id0x2b2dc518">NBIC</a>, the Netherlands Bioinformatics Centre. These are among the founding fathers of the Nano Republic, as they themselves put it.</p> <p> <a href="http://bis.informatik.uni-leipzig.de/SebastianHellmann" id="link-id0x2aac3c5cbcb8">Sebastian Hellman</a> gave a talk on efforts to set up the <i>DBpedia Foundation</i> as a not-for-profit organization, hopefully in the next 10 days, to aid in the sustainability and growth of the <a href="http://dbpedia.org/" id="link-id0x2aac3ceb18e8">DBpedia project</a>. The Foundation would identify stakeholders, their interests, and ways to generate income to further improve DBpedia. Planned areas of improvement include the development of high-availability value-added DBpedia services with <a href="http://dbpedia.org/page/Quality_of_service" id="link-id0x2aac3c86f938">quality of service (QoS)</a> agreements for enterprise users; additional tools in the DBpedia stack to support improved and cost-efficient data curation and <a href="http://wiki.dbpedia.org/Internationalization/" id="link-id0x2aac27b1f388">internationalization</a>; and improved documentation, tutorials, and support to speed uptake.</p> <p>I had a word with <a href="http://www.linkedin.com/pub/peter-haase/0/100/90b" id="link-id0x2b5ae2c8">Peter Haase</a> of fluid Operations about the <a href="http://www.optique-project.eu/" id="link-id0x2abfee08">Optique project</a> and their cloud management offerings. The claim is to do ontology-directed querying over thousands of terabytes of heterogenous data. This turns out to be a full-force attempt at large scale <a href="http://dbpedia.org/page/SQL" id="link-id0x2e1dac08">SQL</a> federation with ontology-directed query rewriting for covering <a href="http://www.w3.org/TR/owl2-profiles/#OWL_2_QL" id="link-id0x31883b48">OWL 2 QL</a> semantics. With <a href="http://dbpedia.org/page/Ian_Horrocks" id="link-id0x321f9f48">Ian Horrocks</a> of Oxford leading the ontology side, the matter is in good hands. Still the matter is not without its problems. Simple lookups can be directed to the data but if there are terabytes of it, it is more likely that <a href="http://dbpedia.org/page/Aggregate_function" id="link-id0x1c8bf838">aggregations</a> are what is desired. Federated aggregation tends to move a lot of data. So the problems are as they ever were. However, if the analytics are already done and stored in the relational space, finding these based on ontologies is a worthwhile thing for streamlining end user access to information.</p> <p>The LOD2 plenary itself was structured in the usual way, covering the work packages in two parallel tracks. </p> <p align="center"> <a href="http://lod2.eu/BlogPost/wp-content/uploads/2014/02/group-LOD2-Plenary-Mannheim.jpg" id="link-id0x2aac2605aab8"><img title="LOD2 Plenary Group Photo, Mannheim, February 2014" src="http://lod2.eu/BlogPost/wp-content/uploads/2014/02/group-LOD2-Plenary-Mannheim.jpg" alt="LOD2 Plenary Group Photo, Mannheim, February 2014" height="320" /> </a> <br /> LOD2 Plenary Group Photo, Mannheim, February 2014</p> <p>On the database side, the final victory will be won by going to adaptive schema for RDF. We brought the RDF penalty against relational to a factor of 2.5 for common analytics style queries, e.g., Star Schema Benchmark. This is a comparison to <a href="http://dbpedia.org/page/Virtuoso_Universal_Server" id="link-id0x2f1aef48">Virtuoso</a> SQL, which offers very high performance in this workload, over 2x the speed of column store pioneer <a href="http://dbpedia.org/page/MonetDB" id="link-id0x25e019f8">MonetDB</a> and 300x <a href="http://dbpedia.org/page/MySQL" id="link-id0x26670158">MySQL</a>. So this is where matters stand. To move them significantly forward, exploitation of structure for guiding physical storage will be needed. Also the project still has to deliver the 500 Gtriple results. The <a href="http://www.openlinksw.com/weblog/oerling/?id=1763" id="link-id0x300bca28">experiments around Christmas</a> at CWI support the possibility, but they are not final. Putting triples into tables when the triples in fact form table-shaped structures, which is the case most of the time, may turn out to be necessary for this. At least, this will be a significant help.</p> <p>Be the case as it may, using a table schema for regularly shaped data, while preserving the RDF quad flexibility, would essentially abolish the RDF tax and bring the LOD2 project to a glorious conclusion in August.</p> <p>I took the poetic license to compare the data journey into RDF and back to the Egyptian myth of Osiris: The data gets shut in a silo and then gets cut into 14 pieces; and subsequently thrown into the Nile (i.e., the LOD cloud, or the CKAN catalog). Grief-stricken Isis sees what is become of her love: She patiently reassembles the pieces, reconstructing Osiris in fact so well that he sires her a child, hawk-headed Horus, who proceeds to reclaim his father’s honor. (See, Isis means Intelligent Structured Information Storage.)</p> <p>I had many interesting conversations with Chris Bizer about his research in data integration, working with the 150M HTML tables in the common crawl. The idea is to resolve references and combine data from the tables. Interestingly enough, the data model in these situations is basically triples, while these are generally not stored as RDF but in Lucene. This makes sense due to the string-matching nature of the task. There appears to be opportunity in bringing together the state of the art in database, meaning the very highly optimized column-store and vectored execution in Virtuoso with the search-style workload found in instance matching and other data integration tasks. The promise goes in the direction of very fast <a href="http://dbpedia.org/resource/Extract,_transform,_load" id="link-id0x2f21cfb8">ETL</a> and subsequent discovery of structural commonalities and enrichment possibilities. This is also not infinitely far from the schema discovery that one may do in order to adaptively optimize storage based on the data.</p> <p> <a href="http://www.linkedin.com/pub/volha-bryl/10/451/333" id="link-id0x2f0cceb8">Volha Bryl</a> gave a very good overview of the Mannheim work in the data integration domain. For example, learning data fusion rules from examples of successful conflict resolution seems very promising. Learning text extraction rules from examples is also interesting. The problem of data integration is that the tasks are very heterogenous and therefore data integration suites have very large numbers of distinct tools. This is labor intensive but there is progress in automation. An error-free, or near enough, data product remains case by case and has human curation but automatic methods seem, based on Volha’s and Chris’ presentation, to be in the ballpark for statistics.</p> <p> <a href="http://www.linkedin.com/in/giovannitummarello" id="link-id0x24ebe6d8">Giovanni Tummarello</a> of <a href="http://insight-centre.org/" id="link-id0x312ab898">Insight</a>/<a href="http://www.sindicetech.com/" id="link-id0x300ba528">SindiceTech</a>, always the life of the party, presented his <a href="http://dbpedia.org/page/Apache_Solr" id="link-id0x2c299518">Solr</a>-based relational faceted browser. The idea is to show and drill down by facets over a set of related tables; in the demo, this was investments, investment targets, and investors. You can look at the data from any of the points and restrict the search based on attributes of any. Well, this is what a database does, right? That is so, but the Sindice tool is on top of Solr and actually materializes joins into a document. This blows up the data but has all the things colocated so it can even run from disk. We also talked about the <a href="http://www.sindicetech.com/overview.html" id="link-id0x2e061af8">Knowledge Graph</a> package Sindice offers on the Google cloud, this time a Virtuoso application. </p> <p>We hope that negotiations between SindiceTech and Insight (formerly <a href="http://dbpedia.org/page/Digital_Enterprise_Research_Institute" id="link-id0x2ac45b48">DERI</a>) around open sourcing the <a href="http://dbpedia.org/page/SPARQL" id="link-id0x256ad4f8">SPARQL</a> editor and other items come to a successful conclusion. The SPARQL editor especially would be of general interest to the RDF community. It is noteworthy that there is no SPARQL query builder in common use out there (even <a href="http://www.openlinksw.com/" id="link-id0x26a4acb8">OpenLink</a>'s own open source <a href="http://oat.openlinksw.com/index.html?isparql" id="link-id0x2faa4398">iSPARQL</a> has been largely (but <a href="http://answers.semanticweb.com/answer_link/13502/" id="link-id0x2dedb8a8">not entirely!</a>) overlooked and misunderstood, though it's been available as part of the <a href="http://www.openlinksw.com/RPC2" id="link-id0x17cab798">OpenLink Ajax Toolkit</a> for several years). OK, a query builder is useful when there is schema. But if the schema is an SQL one, as will be the case if RDF is adaptively stored, then any SQL query builder can be applied to the regular portion of the data. 40 years of calendar time and millennia of person years have gone into making SQL front ends and these will become applicable overnight; Virtuoso does speak SQL, as you may know.</p> <p>I had the breakout session about the database work in LOD2. What will be done is clear enough, the execution side is very good, and our coverage of the infinite space of query optimization continues to grow. One more revolution for storage may come about, as suggested above. There is not very much to discuss, just to execute. So I used the time to explain how you run </p> <blockquote> <code><pre>SELECT SUM ( l_extendedprice ) FROM lineitem , part WHERE l_partkey = p_partkey AND p_name LIKE '%green%'</pre> </code> </blockquote> <p>Simple query, right? Sure, but application guys or sem-heads generally have no clue about how these in fact need to be done. I have the probably foolish belief that a little understanding of database, especially in the RDF space which does get hit by every query optimization problem, would be helpful. At least one would know what goes wrong. So I explained to Giovanni, who is in fact a good geek, that this is a hash join, and with only a little prompting he suggested that you should also put a Bloom filter in front of the hash. Good. So in the bar after dinner I was told I ought to teach. Maybe. But the students would have to be very fast and motivated. Anyway, the take-home message is that the DBMS must figure it out. In the SQL space this is easier, and of course, if most of RDF reduces to this, then RDF too will be more predictable in this department.</p> <p>I talked with <a href="http://www.linkedin.com/in/martinkaltenboeck" id="link-id0x2303ff58">Martin Kaltenböck</a> of the <a href="http://www.linkedin.com/company/919714" id="link-id0x2e5a5908">Semantic Web Company</a> about his brilliant networking accomplishments around organizing the <a href="http://www.data-forum.eu/" id="link-id0x2d39f418">European Data Forum</a> and other activities. Martin is a great ambassador and lobbyist for linked data across Europe. Great work, also in generating visibility for LOD2.</p> <p>The <a href="http://dbpedia.org/page/European_Union" id="link-id0x2ef83908">EU</a> in general, thanks in great part to <a href="http://www.linkedin.com/in/stefanobertolo" id="link-id0x25ef47c8">Stefano Bertolo</a>’s long term push in this direction, is putting increasing emphasis on measuring progress in the research it funds. This is one of the messages from the LOD2 review also. Database is the domain of performance race <i>par excellence;</i> the matters on that side are well attended to by <a href="http://www.ldbc.eu/" id="link-id0xe514b98">LDBC</a> and, of course, the unimpeachably authoritative <a href="http://dbpedia.org/page/TPC" id="link-id0x2b4a1ac8">TPC</a>, among others. In other domains, measurement is harder, as it involves a human-curated ground truth for any extraction, linking, or other integration. There is good work in both Mannheim and Leipzig in these areas, and I may at some point take a closer look, but for now it is appropriate to stick to core database.</p>
2014-02-27T10:57:43.000001-05:00
LOD2 Plenary and Open Data Meet-up in Mannheim
http://www.openlinksw.com:443/weblog/oerling/?date=2014-02-26#1777
2014-02-26T20:37:52Z
<p>There was a <a href="http://lod2.eu/" id="link-id0x2d6bfc28">LOD2</a> plenary meeting hosted by <a href="http://www.bizer.de/" id="link-id0x296a4c18">Chris Bizer</a> from the <a href="http://lod2.eu/Partner/fub.html" id="link-id0x2ec8dd58">University of Mannheim</a> this week.</p> <p>The plenary meeting was preceded by a <a href="http://www.meetup.com/OpenKnowledgeFoundation/Mannheim-DE/1092882/" id="link-id0x3301ff08">Linked Open Data Meetup</a> with talks from <a href="http://dbpedia.org/page/Springer_Science+Business_Media" id="link-id0x2bc7f2a8">Springer</a>, <a href="http://dbpedia.org/resource/Fluid_Operations" id="link-id0x1b077a48">fluid Operations</a>, and several LOD2 partners (<a href="http://lod2.eu/Partner/ulei.html" id="link-id0x341e6678">Universität Leipzig</a>, <a href="http://lod2.eu/Partner/fub.html" id="link-id0x316e4998">University of Mannheim</a>, the <a href="http://lod2.eu/Partner/swc.html" id="link-id0x353bbe58">Semantic Web Company</a>, and <a href="http://lod2.eu/Partner/wkd.html" id="link-id0x300c2308">Wolters Kluwer Deutschland GmbH (WKD)</a>).</p> <p> <a href="http://lod2.eu/Partner/wkd.html" id="link-id0x300c2308">Wolters Kluwer Deutschland GmbH (WKD)</a> gave a presentation on the content production pipeline of their legal publications and their experiences in incorporating LOD2 technologies for content enrichment. This is a very successful LOD2 use case and demonstrates the value of <a href="http://kingsley.idehen.net/describe/?url=http%3A%2F%2Fkingsley.idehen.net%2FDAV%2Fhome%2Fkidehen%2FPublic%2FLinked%2520Data%2520Documents%2FGlossaryOfTerms.ttl%23LinkedData" id="link-id0x10705738">linked data</a> for the information industry.</p> <p>Springer gave a talk about their interest in linked data for enriching the Lecture Notes in Computer Science product. Also conference proceedings could be enhanced with structured <a href="http://www.openlinksw.com/href" id="link-id0x19279a98">metadata</a> in <a href="http://dbpedia.org/page/Resource_Description_Framework" id="link-id0x2c1e8c38">RDF</a>. I asked about nanopublications. The comment was that content authors might perceive nanopublications as an extra imposition. On the other hand, in the life sciences field there is a lot of enthusiasm for the idea. We will see; anyway, biology will likely lead the way for nanopublications. I referred <a href="http://www.linkedin.com/in/aliaksandrbirukou" id="link-id0x2ba061b8">Aliaksandr Birukou</a> of Springer to the companies <a href="http://www.linkedin.com/company/euretos" id="link-id0x2bcc5dc8">Euretos</a> and its parent S&T in Delft, Netherlands, and to <a href="http://dbpedia.org/resource/Barend_Mons" id="link-id0x2e4a9918">Barend Mons</a>, scientific director of <a href="http://dbpedia.org/page/Netherlands_Bioinformatics_Centre" id="link-id0x2b2dc518">NBIC</a>, the Netherlands Bioinformatics Centre. These are among the founding fathers of the Nano Republic, as they themselves put it.</p> <p> <a href="http://bis.informatik.uni-leipzig.de/SebastianHellmann" id="link-id0x355af288">Sebastian Hellman</a> gave a talk on efforts to set up the <i>DBpedia Foundation</i> as a not-for-profit organization, hopefully in the next 10 days, to aid in the sustainability and growth of the <a href="http://dbpedia.org/" id="link-id0x30931d88">DBpedia project</a>. The Foundation would identify stakeholders, their interests, and ways to generate income to further improve DBpedia. Planned areas of improvement include the development of high-availability value-added DBpedia services with <a href="http://dbpedia.org/page/Quality_of_service" id="link-id0x2c4cbe88">quality of service (QoS)</a> agreements for enterprise users; additional tools in the DBpedia stack to support improved and cost-efficient data curation and <a href="http://wiki.dbpedia.org/Internationalization/" id="link-id0x339f6168">internationalization</a>; and improved documentation, tutorials, and support to speed uptake.</p> <p>I had a word with <a href="http://www.linkedin.com/pub/peter-haase/0/100/90b" id="link-id0x2b5ae2c8">Peter Haase</a> of fluid Operations about the <a href="http://www.optique-project.eu/" id="link-id0x2abfee08">Optique project</a> and their cloud management offerings. The claim is to do ontology-directed querying over thousands of terabytes of heterogenous data. This turns out to be a full-force attempt at large scale <a href="http://dbpedia.org/page/SQL" id="link-id0x2e1dac08">SQL</a> federation with ontology-directed query rewriting for covering <a href="http://www.w3.org/TR/owl2-profiles/#OWL_2_QL" id="link-id0x31883b48">OWL 2 QL</a> semantics. With <a href="http://dbpedia.org/page/Ian_Horrocks" id="link-id0x321f9f48">Ian Horrocks</a> of Oxford leading the ontology side, the matter is in good hands. Still the matter is not without its problems. Simple lookups can be directed to the data but if there are terabytes of it, it is more likely that <a href="http://dbpedia.org/page/Aggregate_function" id="link-id0x1c8bf838">aggregations</a> are what is desired. Federated aggregation tends to move a lot of data. So the problems are as they ever were. However, if the analytics are already done and stored in the relational space, finding these based on ontologies is a worthwhile thing for streamlining end user access to information.</p> <p>The LOD2 plenary itself was structured in the usual way, covering the work packages in two parallel tracks. </p> <p align="center"> <a href="http://lod2.eu/BlogPost/wp-content/uploads/2014/02/group-LOD2-Plenary-Mannheim.jpg" id="link-id0x9e18678"><img title="LOD2 Plenary Group Photo, Mannheim, February 2014" src="http://lod2.eu/BlogPost/wp-content/uploads/2014/02/group-LOD2-Plenary-Mannheim.jpg" alt="LOD2 Plenary Group Photo, Mannheim, February 2014" height="320" /> </a> <br /> LOD2 Plenary Group Photo, Mannheim, February 2014</p> <p>On the database side, the final victory will be won by going to adaptive schema for RDF. We brought the RDF penalty against relational to a factor of 2.5 for common analytics style queries, e.g., Star Schema Benchmark. This is a comparison to <a href="http://dbpedia.org/page/Virtuoso_Universal_Server" id="link-id0x2f1aef48">Virtuoso</a> SQL, which offers very high performance in this workload, over 2x the speed of column store pioneer <a href="http://dbpedia.org/page/MonetDB" id="link-id0x25e019f8">MonetDB</a> and 300x <a href="http://dbpedia.org/page/MySQL" id="link-id0x26670158">MySQL</a>. So this is where matters stand. To move them significantly forward, exploitation of structure for guiding physical storage will be needed. Also the project still has to deliver the 500 Gtriple results. The <a href="http://www.openlinksw.com/weblog/oerling/?id=1763" id="link-id0x300bca28">experiments around Christmas</a> at CWI support the possibility, but they are not final. Putting triples into tables when the triples in fact form table-shaped structures, which is the case most of the time, may turn out to be necessary for this. At least, this will be a significant help.</p> <p>Be the case as it may, using a table schema for regularly shaped data, while preserving the RDF quad flexibility, would essentially abolish the RDF tax and bring the LOD2 project to a glorious conclusion in August.</p> <p>I took the poetic license to compare the data journey into RDF and back to the Egyptian myth of Osiris: The data gets shut in a silo and then gets cut into 14 pieces; and subsequently thrown into the Nile (i.e., the LOD cloud, or the CKAN catalog). Grief-stricken Isis sees what is become of her love: She patiently reassembles the pieces, reconstructing Osiris in fact so well that he sires her a child, hawk-headed Horus, who proceeds to reclaim his father’s honor. (See, Isis means Intelligent Structured Information Storage.)</p> <p>I had many interesting conversations with Chris Bizer about his research in data integration, working with the 150M HTML tables in the common crawl. The idea is to resolve references and combine data from the tables. Interestingly enough, the data model in these situations is basically triples, while these are generally not stored as RDF but in Lucene. This makes sense due to the string-matching nature of the task. There appears to be opportunity in bringing together the state of the art in database, meaning the very highly optimized column-store and vectored execution in Virtuoso with the search-style workload found in instance matching and other data integration tasks. The promise goes in the direction of very fast <a href="http://dbpedia.org/resource/Extract,_transform,_load" id="link-id0x2f21cfb8">ETL</a> and subsequent discovery of structural commonalities and enrichment possibilities. This is also not infinitely far from the schema discovery that one may do in order to adaptively optimize storage based on the data.</p> <p> <a href="http://www.linkedin.com/pub/volha-bryl/10/451/333" id="link-id0x2f0cceb8">Volha Bryl</a> gave a very good overview of the Mannheim work in the data integration domain. For example, learning data fusion rules from examples of successful conflict resolution seems very promising. Learning text extraction rules from examples is also interesting. The problem of data integration is that the tasks are very heterogenous and therefore data integration suites have very large numbers of distinct tools. This is labor intensive but there is progress in automation. An error-free, or near enough, data product remains case by case and has human curation but automatic methods seem, based on Volha’s and Chris’ presentation, to be in the ballpark for statistics.</p> <p> <a href="http://www.linkedin.com/in/giovannitummarello" id="link-id0x24ebe6d8">Giovanni Tummarello</a> of <a href="http://insight-centre.org/" id="link-id0x312ab898">Insight</a>/<a href="http://www.sindicetech.com/" id="link-id0x300ba528">SindiceTech</a>, always the life of the party, presented his <a href="http://dbpedia.org/page/Apache_Solr" id="link-id0x2c299518">Solr</a>-based relational faceted browser. The idea is to show and drill down by facets over a set of related tables; in the demo, this was investments, investment targets, and investors. You can look at the data from any of the points and restrict the search based on attributes of any. Well, this is what a database does, right? That is so, but the Sindice tool is on top of Solr and actually materializes joins into a document. This blows up the data but has all the things colocated so it can even run from disk. We also talked about the <a href="http://www.sindicetech.com/overview.html" id="link-id0x2e061af8">Knowledge Graph</a> package Sindice offers on the Google cloud, this time a Virtuoso application. </p> <p>We hope that negotiations between SindiceTech and Insight (formerly <a href="http://dbpedia.org/page/Digital_Enterprise_Research_Institute" id="link-id0x2ac45b48">DERI</a>) around open sourcing the <a href="http://dbpedia.org/page/SPARQL" id="link-id0x256ad4f8">SPARQL</a> editor and other items come to a successful conclusion. The SPARQL editor especially would be of general interest to the RDF community. It is noteworthy that there is no SPARQL query builder in common use out there (even <a href="http://www.openlinksw.com/" id="link-id0x26a4acb8">OpenLink</a>'s own open source <a href="http://oat.openlinksw.com/index.html?isparql" id="link-id0x2faa4398">iSPARQL</a> has been largely (but <a href="http://answers.semanticweb.com/answer_link/13502/" id="link-id0x2dedb8a8">not entirely!</a>) overlooked and misunderstood, though it's been available as part of the <a href="http://www.openlinksw.com/RPC2" id="link-id0x17cab798">OpenLink Ajax Toolkit</a> for several years). OK, a query builder is useful when there is schema. But if the schema is an SQL one, as will be the case if RDF is adaptively stored, then any SQL query builder can be applied to the regular portion of the data. 40 years of calendar time and millennia of person years have gone into making SQL front ends and these will become applicable overnight; Virtuoso does speak SQL, as you may know.</p> <p>I had the breakout session about the database work in LOD2. What will be done is clear enough, the execution side is very good, and our coverage of the infinite space of query optimization continues to grow. One more revolution for storage may come about, as suggested above. There is not very much to discuss, just to execute. So I used the time to explain how you run </p> <blockquote> <code><pre>SELECT SUM ( l_extendedprice ) FROM lineitem , part WHERE l_partkey = p_partkey AND p_name LIKE '%green%'</pre> </code> </blockquote> <p>Simple query, right? Sure, but application guys or sem-heads generally have no clue about how these in fact need to be done. I have the probably foolish belief that a little understanding of database, especially in the RDF space which does get hit by every query optimization problem, would be helpful. At least one would know what goes wrong. So I explained to Giovanni, who is in fact a good geek, that this is a hash join, and with only a little prompting he suggested that you should also put a Bloom filter in front of the hash. Good. So in the bar after dinner I was told I ought to teach. Maybe. But the students would have to be very fast and motivated. Anyway, the take-home message is that the DBMS must figure it out. In the SQL space this is easier, and of course, if most of RDF reduces to this, then RDF too will be more predictable in this department.</p> <p>I talked with <a href="http://www.linkedin.com/in/martinkaltenboeck" id="link-id0x2303ff58">Martin Kaltenböck</a> of the <a href="http://www.linkedin.com/company/919714" id="link-id0x2e5a5908">Semantic Web Company</a> about his brilliant networking accomplishments around organizing the <a href="http://www.data-forum.eu/" id="link-id0x2d39f418">European Data Forum</a> and other activities. Martin is a great ambassador and lobbyist for linked data across Europe. Great work, also in generating visibility for LOD2.</p> <p>The <a href="http://dbpedia.org/page/European_Union" id="link-id0x2ef83908">EU</a> in general, thanks in great part to <a href="http://www.linkedin.com/in/stefanobertolo" id="link-id0x25ef47c8">Stefano Bertolo</a>’s long term push in this direction, is putting increasing emphasis on measuring progress in the research it funds. This is one of the messages from the LOD2 review also. Database is the domain of performance race <i>par excellence;</i> the matters on that side are well attended to by <a href="http://www.ldbc.eu/" id="link-id0xe514b98">LDBC</a> and, of course, the unimpeachably authoritative <a href="http://dbpedia.org/page/TPC" id="link-id0x2b4a1ac8">TPC</a>, among others. In other domains, measurement is harder, as it involves a human-curated ground truth for any extraction, linking, or other integration. There is good work in both Mannheim and Leipzig in these areas, and I may at some point take a closer look, but for now it is appropriate to stick to core database.</p>
2014-02-27T10:57:35-05:00
Initiating V7 Fast Track: "File Tables" feature now in Virtuoso Open Source!
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-02-20#1776
2014-02-20T19:43:22Z
<p>The "file tables" feature first introduced in the <a href="http://www.tpc.org/tpch/" id="link-id0x2aac3994bd88">TPC-H</a> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac39531e48">bulk load article</a> is now available on the <a href="https://github.com/v7fasttrack/virtuoso-opensource" id="link-id0x2aac3a229148">v7fasttrack clone</a> of the <a href="https://github.com/openlink/virtuoso-opensource/" id="link-id0x2aac3a8bee58">Virtuoso repository on GitHub</a>.</p> <p>To check out —</p> <blockquote> <code><pre>$ git clone https://github.com/v7fasttrack/virtuoso-opensource.git v7fasttrack </pre> </code> </blockquote> <p>The v7fasttrack tree <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VOSBuild" id="link-id0x2aac3abeaac8">compiles just like the main Virtuoso tree</a>. Its content is substantially the same today, except for the file tables feature. There is a diff with the main tree which now consists mostly of white space, since the Fast Track tree is automatically indented with the Linux indent utility each time it is updated, and the Virtuoso tree is not.</p> <p>Ongoing maintenance and previews of new features will be added to this tree as and when they become available.</p> <p>Let's now look at the "file tables" feature.</p> <p>You can use any <a href="http://dbpedia.org/page/Comma-separated_values" id="link-id0x2aac3abeaa68">CSV file</a> like a table, as <a href="http://docs.openlinksw.com/virtuoso/sqlbulkloadoperations.html" id="link-id0x2aac39cf71a8">described in the documentation</a>. The TPC-H data generator (<a href="http://www.tpc.org/tpch/spec/tpch_2_16_0.zip" id="link-id0x2aac3abeab88">complete source ZIP</a>; <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFilesInitiatingV7FastTrack/dbgen2.16.zip" id="link-id0x2aac3abea9e8">ZIP of just the dbgen source</a>) is a convenient place to start to try things out.</p> <p>To generate the qualification database, run —</p> <blockquote> <code><pre>dbgen -s 1</pre> </code> </blockquote> <p>This makes 8 CSV files called <code>*.tbl</code>. You can use these scripts to load them into Virtuoso —</p> <ul> <li> <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFilesInitiatingV7FastTrack/schema.sql" id="link-id0x2aabf5535268">schema.sql </a> </code> </li> <li> <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFilesInitiatingV7FastTrack/ldschema.sql" id="link-id0x2aac388381c8">ldschema.sql </a> </code> </li> <li> <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFilesInitiatingV7FastTrack/ld.sql" id="link-id0x2aac3a59c258">ld.sql </a> </code> </li> </ul> <p>To verify the load, do —</p> <blockquote> <code><pre>SELECT COUNT (*) FROM lineitem_f ; SELECT COUNT (*) FROM lineitem ;</pre> </code> </blockquote> <p>To try different combinations of tables and CSV files, you can, for example, do —</p> <blockquote> <code><pre>SELECT COUNT (*) FROM lineitem, part_f WHERE l_partkey = p_partkey AND p_name LIKE '%green%' ;</pre> </code> </blockquote> <p>This counts shipments of green parts, using the file as the <code>part</code> table. You can then replace the <code>part_f</code> with <code>part</code> to join against the database. The database will be a little faster but the file is also pretty fast since the smaller table (<code>part</code>) is on the build side of a hash join and the scan of <code>lineitem</code> is the same in either case.</p> <p>You can now replace <code>lineitem</code> with <code>lineitem_f</code> and you will see a larger difference. This is still reasonably fast since the <code>lineitem</code> file is scanned in parallel.</p> <p>You can try the different TPC-H queries against tables and files. To get the perfect plans you will need the analytics branch which will be made available shortly via this same GitHub channel.</p> <p>You can also try RDFizing the files using the scripts in <a href="http://www.openlinksw.com/weblog/oerling/?id=1769" id="link-id0x2aac39cb7b48">the Enterprise Linked Data article</a> from earlier this year. The qualification database should go in about 15 minutes on a commodity server and make some 120M triples. In the article, the data came from another server, but it can just as well come from files. These two scripts from that article have been adapted for loading from files —</p> <ul> <li> <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFilesInitiatingV7FastTrack/ldfile.sql" id="link-id0x2aac3973b038">ldfile.sql </a> </code> </li> <li> <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFilesInitiatingV7FastTrack/sql_rdf.sql" id="link-id0x2aac39199f98">sql_rdf.sql </a> </code> </li> </ul> <p>To try this, execute the following commands in iSQL —</p> <blockquote> <code><pre>LOAD sql_rdf.sql; RDF_VIEW_SYNC_TO_PHYSICAL ( 'http://example.com/tpcd' , 1 , , "urn:example.com:tpcd" , 2 , 0 ) ;</pre> </code> </blockquote> <p>To verify the result —</p> <blockquote> <code><pre>sparql SELECT ?c COUNT (*) WHERE { ?s ?p ?o } GROUP BY ?p ORDER BY DESC 2 ;</pre> </code> </blockquote>
2014-02-20T14:43:22.000003-05:00
Initiating V7 Fast Track: "File Tables" feature now in Virtuoso Open Source!
http://www.openlinksw.com:443/weblog/oerling/?date=2014-02-20#1775
2014-02-20T19:43:10Z
<p>The "file tables" feature first introduced in the <a href="http://www.tpc.org/tpch/" id="link-id0x2aac3994bd88">TPC-H</a> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac39531e48">bulk load article</a> is now available on the <a href="https://github.com/v7fasttrack/virtuoso-opensource" id="link-id0x2aac3a229148">v7fasttrack clone</a> of the <a href="https://github.com/openlink/virtuoso-opensource/" id="link-id0x2aac3a8bee58">Virtuoso repository on GitHub</a>.</p> <p>To check out —</p> <blockquote> <code><pre>$ git clone https://github.com/v7fasttrack/virtuoso-opensource.git v7fasttrack </pre> </code> </blockquote> <p>The v7fasttrack tree <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VOSBuild" id="link-id0x2aac3abeaac8">compiles just like the main Virtuoso tree</a>. Its content is substantially the same today, except for the file tables feature. There is a diff with the main tree which now consists mostly of white space, since the Fast Track tree is automatically indented with the Linux indent utility each time it is updated, and the Virtuoso tree is not.</p> <p>Ongoing maintenance and previews of new features will be added to this tree as and when they become available.</p> <p>Let's now look at the "file tables" feature.</p> <p>You can use any <a href="http://dbpedia.org/page/Comma-separated_values" id="link-id0x2aac3abeaa68">CSV file</a> like a table, as <a href="http://docs.openlinksw.com/virtuoso/sqlbulkloadoperations.html" id="link-id0x2aac39cf71a8">described in the documentation</a>. The TPC-H data generator (<a href="http://www.tpc.org/tpch/spec/tpch_2_16_0.zip" id="link-id0x2aac3abeab88">complete source ZIP</a>; <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFilesInitiatingV7FastTrack/dbgen2.16.zip" id="link-id0x2aac3abea9e8">ZIP of just the dbgen source</a>) is a convenient place to start to try things out.</p> <p>To generate the qualification database, run —</p> <blockquote> <code><pre>dbgen -s 1</pre> </code> </blockquote> <p>This makes 8 CSV files called <code>*.tbl</code>. You can use these scripts to load them into Virtuoso —</p> <ul> <li> <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFilesInitiatingV7FastTrack/schema.sql" id="link-id0x2aabf5535268">schema.sql </a> </code> </li> <li> <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFilesInitiatingV7FastTrack/ldschema.sql" id="link-id0x2aac388381c8">ldschema.sql </a> </code> </li> <li> <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFilesInitiatingV7FastTrack/ld.sql" id="link-id0x2aac3a59c258">ld.sql </a> </code> </li> </ul> <p>To verify the load, do —</p> <blockquote> <code><pre>SELECT COUNT (*) FROM lineitem_f ; SELECT COUNT (*) FROM lineitem ;</pre> </code> </blockquote> <p>To try different combinations of tables and CSV files, you can, for example, do —</p> <blockquote> <code><pre>SELECT COUNT (*) FROM lineitem, part_f WHERE l_partkey = p_partkey AND p_name LIKE '%green%' ;</pre> </code> </blockquote> <p>This counts shipments of green parts, using the file as the <code>part</code> table. You can then replace the <code>part_f</code> with <code>part</code> to join against the database. The database will be a little faster but the file is also pretty fast since the smaller table (<code>part</code>) is on the build side of a hash join and the scan of <code>lineitem</code> is the same in either case.</p> <p>You can now replace <code>lineitem</code> with <code>lineitem_f</code> and you will see a larger difference. This is still reasonably fast since the <code>lineitem</code> file is scanned in parallel.</p> <p>You can try the different TPC-H queries against tables and files. To get the perfect plans you will need the analytics branch which will be made available shortly via this same GitHub channel.</p> <p>You can also try RDFizing the files using the scripts in <a href="http://www.openlinksw.com/weblog/oerling/?id=1769" id="link-id0x2aac39cb7b48">the Enterprise Linked Data article</a> from earlier this year. The qualification database should go in about 15 minutes on a commodity server and make some 120M triples. In the article, the data came from another server, but it can just as well come from files. These two scripts from that article have been adapted for loading from files —</p> <ul> <li> <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFilesInitiatingV7FastTrack/ldfile.sql" id="link-id0x2aac3973b038">ldfile.sql </a> </code> </li> <li> <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFilesInitiatingV7FastTrack/sql_rdf.sql" id="link-id0x2aac39199f98">sql_rdf.sql </a> </code> </li> </ul> <p>To try this, execute the following commands in iSQL —</p> <blockquote> <code><pre>LOAD sql_rdf.sql; RDF_VIEW_SYNC_TO_PHYSICAL ( 'http://example.com/tpcd' , 1 , , "urn:example.com:tpcd" , 2 , 0 ) ;</pre> </code> </blockquote> <p>To verify the result —</p> <blockquote> <code><pre>sparql SELECT ?c COUNT (*) WHERE { ?s ?p ?o } GROUP BY ?p ORDER BY DESC 2 ;</pre> </code> </blockquote>
2014-02-20T14:43:10.000018-05:00
Virtuoso Fast Track and the Ways of Open Source
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-02-14#1774
2014-02-14T14:40:55Z
<p>I recall a couple of years back, in a LOD2 project review, one of the reviewers made a comment to the effect that OpenLink's open source strategy was in essence autistic. More precisely, that OpenLink made some things, released these at unanticipatable times, and lacked community engagement.</p> <p>Since then the situation has improved somewhat with the <a href="https://github.com/openlink/virtuoso-opensource/" id="link-id0x2c6d45f8">code being on github</a> and kept more or less up to date with maintenance.</p> <p>There is no reason why users should not get bug fixes when they are made, nor why users should not get functionality previews.</p> <p>Introducing v7fasttrack, a clone of the git repository, where maintenance goes real time, and where early releases of functionality are available for experimentation.</p> <p>It is open kitchen, folks. You will see us slice the fish and mix the curry right in the dining room. You will see epic feats of pizza flipping -- something to regale your grandchildren with.</p> <p>You will still have to place your orders with your waiter; we still won't have time to be on most lists and such but at least you will get the dish while it's hot.</p> <p>In the immediate future, the "file-as-table" feature will be introduced via this channel. This is the capability to make CSV files look like tables, which was mentioned in the <a href="http://www.tpc.org/tpch/" id="link-id0x2d780628">TPC-H</a> <a href="http://www.openlinksw.com/weblog/oerling/?id=1767" id="link-id0x2aac3ade8538">bulk load article.</a> This is key to bulk loading 100GB in 15 minutes on a single commodity server.</p> <p>The query caching feature mentioned in <a href="http://www.openlinksw.com/weblog/oerling/?id=1771" id="link-id0x2daa4898">the previous blog post</a> is the next candidate for availability in this channel. If you have repeating long queries that do lookups with under a million triples or so, compile times will dominate and this will give significant acceleration.</p> <p>The largest item to come is the availability of the analytics branch that has been discussed in <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac3875e4f8">the TPC-H series</a>. It is basically complete, with implementations of most tricks discussed in the TPC TC paper, <i><a href="http://www.tpc.org/tpctc/tpctc2013/slides_and_papers/005.pdf" id="link-id0x2c3c94e8">TPC-H Analyzed: Hidden Messages and Lessons Learned from an Influential Benchmark</a>.</i> </p> <p>These are structured as branches, with safe maintenance to one side, and new features which potentially break things in their own branch. For example, with a much smarter optimizer in the analytics branch, it is conceivable some applications will get worse plans at first even if the benchmarks are good. Anyway, experimentation is safe.</p> <p>These will be introduced in future blog posts, and the relevant documentation will be in the archive. The <a href="http://dbpedia.org/resource/Docbook" id="link-id0x2c0bcf98">DocBook</a> XML form will come later.</p> <p>These features will migrate to the regular git in time. But you will not have to hold your breath waiting for this.</p> <p>The next action is the publishing of the file table feature on the fast track. There will be a post on this next week.</p>
2014-02-14T09:46:10.000001-05:00
Virtuoso Fast Track and the Ways of Open Source
http://www.openlinksw.com:443/weblog/oerling/?date=2014-02-14#1773
2014-02-14T14:40:48Z
<p>I recall a couple of years back, in a LOD2 project review, one of the reviewers made a comment to the effect that OpenLink's open source strategy was in essence autistic. More precisely, that OpenLink made some things, released these at unanticipatable times, and lacked community engagement.</p> <p>Since then the situation has improved somewhat with the <a href="https://github.com/openlink/virtuoso-opensource/" id="link-id0x2c6d45f8">code being on github</a> and kept more or less up to date with maintenance.</p> <p>There is no reason why users should not get bug fixes when they are made, nor why users should not get functionality previews.</p> <p>Introducing v7fasttrack, a clone of the git repository, where maintenance goes real time, and where early releases of functionality are available for experimentation.</p> <p>It is open kitchen, folks. You will see us slice the fish and mix the curry right in the dining room. You will see epic feats of pizza flipping -- something to regale your grandchildren with.</p> <p>You will still have to place your orders with your waiter; we still won't have time to be on most lists and such but at least you will get the dish while it's hot.</p> <p>In the immediate future, the "file-as-table" feature will be introduced via this channel. This is the capability to make CSV files look like tables, which was mentioned in the <a href="http://www.tpc.org/tpch/" id="link-id0x2d780628">TPC-H</a> <a href="http://www.openlinksw.com/weblog/oerling/?id=1767" id="link-id0x2aac3ade8538">bulk load article.</a> This is key to bulk loading 100GB in 15 minutes on a single commodity server.</p> <p>The query caching feature mentioned in <a href="http://www.openlinksw.com/weblog/oerling/?id=1771" id="link-id0x2daa4898">the previous blog post</a> is the next candidate for availability in this channel. If you have repeating long queries that do lookups with under a million triples or so, compile times will dominate and this will give significant acceleration.</p> <p>The largest item to come is the availability of the analytics branch that has been discussed in <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac3875e4f8">the TPC-H series</a>. It is basically complete, with implementations of most tricks discussed in the TPC TC paper, <i><a href="http://www.tpc.org/tpctc/tpctc2013/slides_and_papers/005.pdf" id="link-id0x2c3c94e8">TPC-H Analyzed: Hidden Messages and Lessons Learned from an Influential Benchmark</a>.</i> </p> <p>These are structured as branches, with safe maintenance to one side, and new features which potentially break things in their own branch. For example, with a much smarter optimizer in the analytics branch, it is conceivable some applications will get worse plans at first even if the benchmarks are good. Anyway, experimentation is safe.</p> <p>These will be introduced in future blog posts, and the relevant documentation will be in the archive. The <a href="http://dbpedia.org/resource/Docbook" id="link-id0x2c0bcf98">DocBook</a> XML form will come later.</p> <p>These features will migrate to the regular git in time. But you will not have to hold your breath waiting for this.</p> <p>The next action is the publishing of the file table feature on the fast track. There will be a post on this next week.</p>
2014-02-14T09:43:24.000001-05:00
Query Plan Cache and Lookups
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-02-03#1772
2014-02-03T16:13:20Z
<p>Performance of <a href="http://kingsley.idehen.net/describe/?url=http%3A%2F%2Fkingsley.idehen.net%2FDAV%2Fhome%2Fkidehen%2FPublic%2FLinked%2520Data%2520Documents%2FGlossaryOfTerms5.ttl%23RDF" id="link-id0x21debd98">RDF</a> and other graph workloads may easily be limited by query compilation time. Query compilation time tends to increase as more complex optimizations are added, thus simply streamlining compilation is not the answer. All applications we have come across make queries by instantiating templates and plugging in different literals and different combinations of predefined search conditions. The <a href="http://dbpedia.org/resource/DBMS" id="link-id0x26f5c928">DBMS</a> will see a few hundred distinct queries, but they may come each time with different literals.</p> <p>So reusing query plans between invocations is a natural optimization. This works especially well with lookup workloads, or when the data is small. Analytics tends to be dominated by run time, but lookups that touch at most a million or so triples will often be bound by compilation time, especially if these have tens of triple patterns.</p> <p>Let's consider the following from <a href="http://www.openphacts.org/about-open-phacts/about-open-phacts" id="link-id0x24336408">Open PHACTS</a>:</p> <blockquote> <code><pre>sparql PREFIX chembl: <http://rdf.ebi.ac.uk/terms/chembl#> PREFIX dcterms: <http://purl.org/dc/terms/> PREFIX skos: <http://www.w3.org/2004/02/skos/core#> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> PREFIX owl: <http://www.w3.org/2002/07/owl#> PREFIX bibo: <http://purl.org/ontology/bibo/> PREFIX cheminf: <http://semanticscience.org/resource/> PREFIX obo: <http://purl.obolibrary.org/obo/> PREFIX qudt: <http://qudt.org/1.1/schema/qudt#> SELECT DISTINCT ?item WHERE { VALUES ?chembl_target_uri { <http://rdf.ebi.ac.uk/resource/chembl/target/CHEMBL5451> } GRAPH <http://www.ebi.ac.uk/chembl> { ?assay_uri chembl:hasTarget ?chembl_target_uri . ?assay_uri chembl:hasActivity ?item . ?item chembl:hasMolecule ?compound_chembl . ?chembl_target_uri a ?target_type . OPTIONAL { ?chembl_target_uri dcterms:title ?target_name_chembl } OPTIONAL { ?chembl_target_uri chembl:organismName ?target_organism } OPTIONAL { ?chembl_target_uri chembl:hasTargetComponent ?protein . GRAPH <http://www.conceptwiki.org> { ?cw_target skos:exactMatch ?protein ; skos:prefLabel ?protein_name } } OPTIONAL { ?assay_uri chembl:organismName ?assay_organism } OPTIONAL { ?assay_uri dcterms:description ?assay_description } OPTIONAL { ?assay_uri chembl:assayTestType ?assay_type } OPTIONAL { ?item chembl:publishedType ?published_type } OPTIONAL { ?item chembl:publishedRelation ?published_relation } OPTIONAL { ?item chembl:publishedValue ?published_value } OPTIONAL { ?item chembl:publishedUnits ?published_unit } OPTIONAL { ?item chembl:standardType ?activity_type } OPTIONAL { ?item chembl:standardRelation ?activity_relation } OPTIONAL { ?item chembl:standardValue ?standard_value . BIND ( xsd:decimal( ?standard_value ) AS ?activity_value ) } OPTIONAL { ?item chembl:standardUnits ?activity_unit } OPTIONAL { ?item chembl:hasQUDT ?qudt_uri } OPTIONAL { ?item chembl:pChembl ?pChembl } OPTIONAL { ?item chembl:activityComment ?act_comment } OPTIONAL { ?item chembl:hasDocument ?doc_uri . OPTIONAL { ?doc_uri owl:sameAs ?doi } OPTIONAL { ?doc_uri bibo:pmid ?pmid } } } GRAPH <http://ops.rsc.org> { ?compound_ocrs skos:exactMatch ?compound_chembl . ?compound_ocrs cheminf:CHEMINF_000396 ?inchi ; cheminf:CHEMINF_000399 ?inchi_key ; cheminf:CHEMINF_000018 ?smiles . OPTIONAL { [] obo:IAO_0000136 ?compound_ocrs ; a cheminf:CHEMINF_000484 ; qudt:numericValue ?molweight . } OPTIONAL { [] obo:IAO_0000136 ?compound_ocrs ; a cheminf:CHEMINF_000367 ; qudt:numericValue ?num_ro5_violations . } } ?compound_cw skos:exactMatch ?compound_ocrs ; skos:prefLabel ?compound_name } ORDER BY ?item LIMIT 10 OFFSET 0 ;</pre> </code> </blockquote> <p>This is quite typical. More complex ones have been seen, with many unions on top.</p> <p>We run this with profile on warm cache, no plan reuse in effect. The database is Open PHACTS OPS from this January.</p> <blockquote> <code><pre>profile ('sparql prefix ....'); ... 15 msec 4% cpu, 15688 rnd 9547 seq 91.8669% same seg 4.62107% same pg Compilation: 313 msec 0 reads 0% read 0 messages 0% clw </pre> </code> </blockquote> <p>The compilation time is over 20x longer than the execution. We see from the top line that the execution did 15K random lookups and retrieved 9K rows sequentially.</p> <p>We enable plan reuse and rerun:</p> <blockquote> <code><pre> 14 msec 58% cpu, 15688 rnd 9547 seq 91.8669% same seg 4.62107% same pg Compilation: 0 msec 0 reads 0% read 0 messages 0% clw </pre> </code> </blockquote> <p>The compile time is now gone. This is an especially large win. With a set of 31 queries from Open PHACTS, each repeated over many different parameter bindings, the gains from query caching are a speedup of 1.5x. More details may be published after VU Amsterdam, which does the data management for Open PHACTS, following publication of the benchmark data and queries. The present figures are an order of magnitude better than the figures from last fall, which will also be in the publication.</p> <p>With query plan caching, the same plan will be reused as long as the literals in the new query have approximately the same selectivity as the ones which were present when the plan was first made. In this way, if a different plan is in fact needed, one will be made. The same query text can have many alternative plans for different selectivities of search conditions.</p> <p>In this way, plan reuse may work better than prepared statements. Anyway, prepared statements do not exist in the <a href="http://dbpedia.org/page/SPARQL" id="link-id0x26869b68">SPARQL</a> query language. In <a href="http://dbpedia.org/page/SQL" id="link-id0x2540acc8">SQL</a> they do, but then the optimizer does not know the values the parameters will have.</p> <p>The overhead of plan reuse, as opposed to parameterized prepared statements, is relatively low. The cache remembers the sampling that was done when the plan was first made. The same samples are taken with the new literals plugged in. If the cardinalities are within a settable percentage (e.g., 20% of the original), the plan is assumed to be applicable. On the other hand, with prepared parameterized statements, there is no sampling at all, but then the plan might be worse due to less information being available to the optimizer. </p> <p>Publishing is another type of workload where compile times easily form a large percentage of the total. The queries are shorter than in the biology case, since the modeling tends to be simpler and there are less distinct sources being queried. The frequency of the queries is higher though, and each might touch only some tens of triples.</p> <p>The query caching feature will be included in forthcoming <a href="http://virtuoso.openlinksw.com/" id="link-id0x26292b98">Virtuoso</a> updates, and will not require operator intervention nor changes to configurations or applications. The feature will be controlled by a few settings in <a href="http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#VIRTINI" id="link-id0x20edd4c8">the configuration file</a>, but the defaults will work for almost all cases.</p>
2014-02-03T11:13:20.000002-05:00
Query Plan Cache and Lookups
http://www.openlinksw.com:443/weblog/oerling/?date=2014-02-03#1771
2014-02-03T16:13:08Z
<p>Performance of <a href="http://kingsley.idehen.net/describe/?url=http%3A%2F%2Fkingsley.idehen.net%2FDAV%2Fhome%2Fkidehen%2FPublic%2FLinked%2520Data%2520Documents%2FGlossaryOfTerms5.ttl%23RDF" id="link-id0x21debd98">RDF</a> and other graph workloads may easily be limited by query compilation time. Query compilation time tends to increase as more complex optimizations are added, thus simply streamlining compilation is not the answer. All applications we have come across make queries by instantiating templates and plugging in different literals and different combinations of predefined search conditions. The <a href="http://dbpedia.org/resource/DBMS" id="link-id0x26f5c928">DBMS</a> will see a few hundred distinct queries, but they may come each time with different literals.</p> <p>So reusing query plans between invocations is a natural optimization. This works especially well with lookup workloads, or when the data is small. Analytics tends to be dominated by run time, but lookups that touch at most a million or so triples will often be bound by compilation time, especially if these have tens of triple patterns.</p> <p>Let's consider the following from <a href="http://www.openphacts.org/about-open-phacts/about-open-phacts" id="link-id0x24336408">Open PHACTS</a>:</p> <blockquote> <code><pre>sparql PREFIX chembl: <http://rdf.ebi.ac.uk/terms/chembl#> PREFIX dcterms: <http://purl.org/dc/terms/> PREFIX skos: <http://www.w3.org/2004/02/skos/core#> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> PREFIX owl: <http://www.w3.org/2002/07/owl#> PREFIX bibo: <http://purl.org/ontology/bibo/> PREFIX cheminf: <http://semanticscience.org/resource/> PREFIX obo: <http://purl.obolibrary.org/obo/> PREFIX qudt: <http://qudt.org/1.1/schema/qudt#> SELECT DISTINCT ?item WHERE { VALUES ?chembl_target_uri { <http://rdf.ebi.ac.uk/resource/chembl/target/CHEMBL5451> } GRAPH <http://www.ebi.ac.uk/chembl> { ?assay_uri chembl:hasTarget ?chembl_target_uri . ?assay_uri chembl:hasActivity ?item . ?item chembl:hasMolecule ?compound_chembl . ?chembl_target_uri a ?target_type . OPTIONAL { ?chembl_target_uri dcterms:title ?target_name_chembl } OPTIONAL { ?chembl_target_uri chembl:organismName ?target_organism } OPTIONAL { ?chembl_target_uri chembl:hasTargetComponent ?protein . GRAPH <http://www.conceptwiki.org> { ?cw_target skos:exactMatch ?protein ; skos:prefLabel ?protein_name } } OPTIONAL { ?assay_uri chembl:organismName ?assay_organism } OPTIONAL { ?assay_uri dcterms:description ?assay_description } OPTIONAL { ?assay_uri chembl:assayTestType ?assay_type } OPTIONAL { ?item chembl:publishedType ?published_type } OPTIONAL { ?item chembl:publishedRelation ?published_relation } OPTIONAL { ?item chembl:publishedValue ?published_value } OPTIONAL { ?item chembl:publishedUnits ?published_unit } OPTIONAL { ?item chembl:standardType ?activity_type } OPTIONAL { ?item chembl:standardRelation ?activity_relation } OPTIONAL { ?item chembl:standardValue ?standard_value . BIND ( xsd:decimal( ?standard_value ) AS ?activity_value ) } OPTIONAL { ?item chembl:standardUnits ?activity_unit } OPTIONAL { ?item chembl:hasQUDT ?qudt_uri } OPTIONAL { ?item chembl:pChembl ?pChembl } OPTIONAL { ?item chembl:activityComment ?act_comment } OPTIONAL { ?item chembl:hasDocument ?doc_uri . OPTIONAL { ?doc_uri owl:sameAs ?doi } OPTIONAL { ?doc_uri bibo:pmid ?pmid } } } GRAPH <http://ops.rsc.org> { ?compound_ocrs skos:exactMatch ?compound_chembl . ?compound_ocrs cheminf:CHEMINF_000396 ?inchi ; cheminf:CHEMINF_000399 ?inchi_key ; cheminf:CHEMINF_000018 ?smiles . OPTIONAL { [] obo:IAO_0000136 ?compound_ocrs ; a cheminf:CHEMINF_000484 ; qudt:numericValue ?molweight . } OPTIONAL { [] obo:IAO_0000136 ?compound_ocrs ; a cheminf:CHEMINF_000367 ; qudt:numericValue ?num_ro5_violations . } } ?compound_cw skos:exactMatch ?compound_ocrs ; skos:prefLabel ?compound_name } ORDER BY ?item LIMIT 10 OFFSET 0 ;</pre> </code> </blockquote> <p>This is quite typical. More complex ones have been seen, with many unions on top.</p> <p>We run this with profile on warm cache, no plan reuse in effect. The database is Open PHACTS OPS from this January.</p> <blockquote> <code><pre>profile ('sparql prefix ....'); ... 15 msec 4% cpu, 15688 rnd 9547 seq 91.8669% same seg 4.62107% same pg Compilation: 313 msec 0 reads 0% read 0 messages 0% clw </pre> </code> </blockquote> <p>The compilation time is over 20x longer than the execution. We see from the top line that the execution did 15K random lookups and retrieved 9K rows sequentially.</p> <p>We enable plan reuse and rerun:</p> <blockquote> <code><pre> 14 msec 58% cpu, 15688 rnd 9547 seq 91.8669% same seg 4.62107% same pg Compilation: 0 msec 0 reads 0% read 0 messages 0% clw </pre> </code> </blockquote> <p>The compile time is now gone. This is an especially large win. With a set of 31 queries from Open PHACTS, each repeated over many different parameter bindings, the gains from query caching are a speedup of 1.5x. More details may be published after VU Amsterdam, which does the data management for Open PHACTS, following publication of the benchmark data and queries. The present figures are an order of magnitude better than the figures from last fall, which will also be in the publication.</p> <p>With query plan caching, the same plan will be reused as long as the literals in the new query have approximately the same selectivity as the ones which were present when the plan was first made. In this way, if a different plan is in fact needed, one will be made. The same query text can have many alternative plans for different selectivities of search conditions.</p> <p>In this way, plan reuse may work better than prepared statements. Anyway, prepared statements do not exist in the <a href="http://dbpedia.org/page/SPARQL" id="link-id0x26869b68">SPARQL</a> query language. In <a href="http://dbpedia.org/page/SQL" id="link-id0x2540acc8">SQL</a> they do, but then the optimizer does not know the values the parameters will have.</p> <p>The overhead of plan reuse, as opposed to parameterized prepared statements, is relatively low. The cache remembers the sampling that was done when the plan was first made. The same samples are taken with the new literals plugged in. If the cardinalities are within a settable percentage (e.g., 20% of the original), the plan is assumed to be applicable. On the other hand, with prepared parameterized statements, there is no sampling at all, but then the plan might be worse due to less information being available to the optimizer. </p> <p>Publishing is another type of workload where compile times easily form a large percentage of the total. The queries are shorter than in the biology case, since the modeling tends to be simpler and there are less distinct sources being queried. The frequency of the queries is higher though, and each might touch only some tens of triples.</p> <p>The query caching feature will be included in forthcoming <a href="http://virtuoso.openlinksw.com/" id="link-id0x26292b98">Virtuoso</a> updates, and will not require operator intervention nor changes to configurations or applications. The feature will be controlled by a few settings in <a href="http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#VIRTINI" id="link-id0x20edd4c8">the configuration file</a>, but the defaults will work for almost all cases.</p>
2014-02-03T11:13:08-05:00
Meeting the Challenges of Linked Data in the Enterprise
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-01-17#1770
2014-01-17T16:49:12Z
<p> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x1af2a938">A few blogs back</a>, I was talking about the comeback of <a href="http://dbpedia.org/resource/SQL" id="link-id0x2aabf9d7bd18">SQL</a> (i.e., relational) <a href="http://dbpedia.org/resource/Federated_database_system" id="link-id0x2aabf97e4228">federation</a>. True, <a href="http://virtuoso.openlinksw.com/middleware/" id="link-id0x18276d18">Virtuoso has always had that</a>, but in the general perception, it is primarily known for <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x97fad48">RDF</a> and <a href="http://kingsley.idehen.net/describe/?url=http%3A%2F%2Fkingsley.idehen.net%2FDAV%2Fhome%2Fkidehen%2FPublic%2FLinked%2520Data%2520Documents%2FGlossaryOfTerms4.ttl%23LinkedData" id="link-id0x1c21d6f8">Linked Data</a>. Now these two areas of functionality are combined in a way that makes sense and can provide real performance, often surpassing the relational installed base while running <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1c1f9138">SPARQL</a> with no predefined schema.</p> <p>So, we take our standard <a href="http://www.tpc.org/tpch/" id="link-id0x1b1f6828">TPC-H</a> 100 GB dataset and turn it into RDF. We adjust the schema a little, so that each <code>customer</code>, its <code>orders</code>, and the <code>lineitems</code> of those orders go into a per-customer graph. The <code>part</code>, <code>partsupp</code>, and other tables will all go into a public graph. The per-customer graph can be used as a security label; for example, if there is customer self-service access to the warehouse, or if the access is compartmentalized by areas of responsibility (e.g., customer countries or market segments).</p> <p>We use two Virtuoso processes. The first contains the TPC-H 100G dataset, the same that was discussed in <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aabe462fe78">the TPC-H bulk load article</a>. The second process attaches the tables from the first via SQL federation, and constructs an RDF translation into its RDF store. The mapping is made with an RDF view, also known as a <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtRdb2RDFViewsGeneration" id="link-id0x1b637708">Linked Data View</a>. The initial RDF view can be generated from the relational schema, then edited for the selection of properties. If there are modeling or unit changes in the mapping, these are easiest done with SQL views, in which case the RDF mapping is made on top of the views, not the actual tables. The SQL views reside on the same server that has the RDF views, so no write access to the source database is needed.</p> <p>The server configuration is found in <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFilesEnterpriseLinkedDataMadePossible/virtuoso.ini" id="link-id0x181e9df8">virtuoso.ini</a></code>. This is for 4 disks and 192 GB RAM, so if you try this, make sure you have at least this much or use an accordingly scaled down dataset.</p> <p>The data is defined by loading the scripts:</p> <blockquote> <code><pre>SQL> LOAD <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFilesEnterpriseLinkedDataMadePossible/att2.sql" id="link-id0x191fb5d8">att2.sql</a> ; SQL> LOAD <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFilesEnterpriseLinkedDataMadePossible/sql_rdf_rdfh11.sql" id="link-id0x1b63ef68">sql_rdf_rdfh11.sql</a> ;</pre> </code> </blockquote> <p>The first attaches the tables from the source server; the second defines the mapping from tables to triples. The final script (below) starts the actual ETL.</p> <p>We set the default vector size to 200,000. Experience shows this is good for this sort of operation, and may save some 20% of time.</p> <blockquote> <code><pre>SQL> __dbf_set ('dc_batch_sz', 200000) ;</pre> </code> </blockquote> <p>We run the transformation: The commands are in the <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFilesEnterpriseLinkedDataMadePossible/rdb2rdf_rdfh11_1.sql" id="link-id0x2aabfa9dd5e8">rdb2rdf_rdfh11_1.sql</a></code> script, discussed below. The <code>ld_meter_run</code> starts a thread to record the load rate every 30 seconds.</p> <blockquote> <code><pre>SQL> LOAD /mvi/te/suite/tpc-d/rdb2rdf_rdfh11_1.sql & SQL> ld_meter_run (30) & Done. -- 40686670 msec.</pre> </code> </blockquote> <p> <i>-- The load is non-transactional bulk load, so needs an explicit checkpoint to make the result durable.</i> </p> <blockquote> <code><pre>SQL> checkpoint ; Done. -- 1858934 msec.</pre> </code> </blockquote> <p> <i>-- we check the result </i> </p> <blockquote> <code><pre>SQL> sparql SELECT COUNT (*) WHERE { ?s ?p ?o } ; 11869611740 SQL> sparql SELECT ?p COUNT (*) WHERE { ?s ?p ?o } GROUP BY ?p ORDER BY DESC 2 LIMIT 200 ;</pre> </code> </blockquote> <table> <tr> <th align="center">Predicate URI</th> <th align="center">number of triples</th> </tr> <tr> <td>http://www.w3.org/1999/02/22-rdf-syntax-ns#type</td> <td align="right">881,038,747</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_has_order</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_has_part</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_number</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_discount</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_linestatus</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_shipdate</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_quantity</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_extendedprice</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_commitdate</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_has_supplier</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_tax</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_returnflag</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_receiptdate</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_shipinstruct</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_shipmode</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#o_clerk</td> <td align="right">150,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#o_comment</td> <td align="right">150,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#c_customer_of</td> <td align="right">150,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#o_orderstatus</td> <td align="right">150,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#o_totalprice</td> <td align="right">150,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#o_orderpriority</td> <td align="right">150,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#o_orderkey</td> <td align="right">150,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#o_orderdate</td> <td align="right">150,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#o_shippriority</td> <td align="right">150,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#ps_has_supplier</td> <td align="right">80,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#ps_availqty</td> <td align="right">80,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#ps_supplycost</td> <td align="right">80,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#ps_has_part</td> <td align="right">80,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#p_type</td> <td align="right">20,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#p_size</td> <td align="right">20,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#p_container</td> <td align="right">20,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#p_mfgr</td> <td align="right">20,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#p_partkey</td> <td align="right">20,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#p_name</td> <td align="right">20,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#p_brand</td> <td align="right">20,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#p_comment</td> <td align="right">20,000,000</td> </tr> <tr> <td>http://xmlns.com/foaf/0.1/phone</td> <td align="right">16,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#c_comment</td> <td align="right">15,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#c_acctbal</td> <td align="right">15,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#c_mktsegment</td> <td align="right">15,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#c_custkey</td> <td align="right">15,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#c_name</td> <td align="right">15,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#c_has_nation</td> <td align="right">15,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#c_address</td> <td align="right">15,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#c_phone</td> <td align="right">15,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#n_nation_of</td> <td align="right">15,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#s_has_nation</td> <td align="right">1,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#s_acctbal</td> <td align="right">1,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#s_name</td> <td align="right">1,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#s_address</td> <td align="right">1,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#s_comment</td> <td align="right">1,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#s_suppkey</td> <td align="right">1,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#s_phone</td> <td align="right">1,000,000</td> </tr> <tr> <td colspan="2" align="center">...</td> </tr> </table> <p> <i>We calculate the ETL speed:</i> </p> <blockquote> <code><pre>SQL> SELECT 11869611740 / ( ( 1858934 + 40686670 ) / 1000.0 ) ; 278985.620700084549276</pre> </code> </blockquote> <p>11.8 hours, at just under 280 Kt/s, end-to-end. Worse has been heard of. This is a small single-server speed, on the usual test system with dual Xeon E5 2630, 192G RAM. A single-server for double the price might get double throughput. Beyond this, scale out is clearly the better deal. An elastic cluster will get throughput linear to the count of machines for this type of workload.</p> <p>This shows that deploying mid-size enterprise data as RDF is a job that goes easily overnight with a commodity box, reading directly from the source system; no file-system-based staging areas are needed.</p> <p>The dataset is 600M order lines; 150M orders; 15M customers; 20M parts, each with 4 suppliers; 1M total suppliers. You can contrast this to what you have in-house to get a rough estimate of what your own DW would come to.</p> <p>Later, we will use this dataset to illustrate how to scope queries to security categories with graph-level security. Of course, this dataset also provides a point of SQL-to-SPARQL comparison for the <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x1c477548">ongoing TPC-H series</a>. There will be more installments in not too long.</p>
2014-01-17T16:52:06-05:00
Meeting the Challenges of Linked Data in the Enterprise
http://www.openlinksw.com:443/weblog/oerling/?date=2014-01-17#1769
2014-01-17T16:21:27Z
<p> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x1af2a938">A few blogs back</a>, I was talking about the comeback of <a href="http://dbpedia.org/resource/SQL" id="link-id0x2aabf9d7bd18">SQL</a> (i.e., relational) <a href="http://dbpedia.org/resource/Federated_database_system" id="link-id0x2aabf97e4228">federation</a>. True, <a href="http://virtuoso.openlinksw.com/middleware/" id="link-id0x18276d18">Virtuoso has always had that</a>, but in the general perception, it is primarily known for <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x97fad48">RDF</a> and <a href="http://kingsley.idehen.net/describe/?url=http%3A%2F%2Fkingsley.idehen.net%2FDAV%2Fhome%2Fkidehen%2FPublic%2FLinked%2520Data%2520Documents%2FGlossaryOfTerms4.ttl%23LinkedData" id="link-id0x1c21d6f8">Linked Data</a>. Now these two areas of functionality are combined in a way that makes sense and can provide real performance, often surpassing the relational installed base while running <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1c1f9138">SPARQL</a> with no predefined schema.</p> <p>So, we take our standard <a href="http://www.tpc.org/tpch/" id="link-id0x1b1f6828">TPC-H</a> 100 GB dataset and turn it into RDF. We adjust the schema a little, so that each <code>customer</code>, its <code>orders</code>, and the <code>lineitems</code> of those orders go into a per-customer graph. The <code>part</code>, <code>partsupp</code>, and other tables will all go into a public graph. The per-customer graph can be used as a security label; for example, if there is customer self-service access to the warehouse, or if the access is compartmentalized by areas of responsibility (e.g., customer countries or market segments).</p> <p>We use two Virtuoso processes. The first contains the TPC-H 100G dataset, the same that was discussed in <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aabe462fe78">the TPC-H bulk load article</a>. The second process attaches the tables from the first via SQL federation, and constructs an RDF translation into its RDF store. The mapping is made with an RDF view, also known as a <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtRdb2RDFViewsGeneration" id="link-id0x1b637708">Linked Data View</a>. The initial RDF view can be generated from the relational schema, then edited for the selection of properties. If there are modeling or unit changes in the mapping, these are easiest done with SQL views, in which case the RDF mapping is made on top of the views, not the actual tables. The SQL views reside on the same server that has the RDF views, so no write access to the source database is needed.</p> <p>The server configuration is found in <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFilesEnterpriseLinkedDataMadePossible/virtuoso.ini" id="link-id0x181e9df8">virtuoso.ini</a></code>. This is for 4 disks and 192 GB RAM, so if you try this, make sure you have at least this much or use an accordingly scaled down dataset.</p> <p>The data is defined by loading the scripts:</p> <blockquote> <code><pre>SQL> LOAD <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFilesEnterpriseLinkedDataMadePossible/att2.sql" id="link-id0x191fb5d8">att2.sql</a> ; SQL> LOAD <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFilesEnterpriseLinkedDataMadePossible/sql_rdf_rdfh11.sql" id="link-id0x1b63ef68">sql_rdf_rdfh11.sql</a> ;</pre> </code> </blockquote> <p>The first attaches the tables from the source server; the second defines the mapping from tables to triples. The final script (below) starts the actual ETL.</p> <p>We set the default vector size to 200,000. Experience shows this is good for this sort of operation, and may save some 20% of time.</p> <blockquote> <code><pre>SQL> __dbf_set ('dc_batch_sz', 200000) ;</pre> </code> </blockquote> <p>We run the transformation: The commands are in the <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/BlogFilesEnterpriseLinkedDataMadePossible/rdb2rdf_rdfh11_1.sql" id="link-id0x2aabfa9dd5e8">rdb2rdf_rdfh11_1.sql</a></code> script, discussed below. The <code>ld_meter_run</code> starts a thread to record the load rate every 30 seconds.</p> <blockquote> <code><pre>SQL> LOAD /mvi/te/suite/tpc-d/rdb2rdf_rdfh11_1.sql & SQL> ld_meter_run (30) & Done. -- 40686670 msec.</pre> </code> </blockquote> <p> <i>-- The load is non-transactional bulk load, so needs an explicit checkpoint to make the result durable.</i> </p> <blockquote> <code><pre>SQL> checkpoint ; Done. -- 1858934 msec.</pre> </code> </blockquote> <p> <i>-- we check the result </i> </p> <blockquote> <code><pre>SQL> sparql SELECT COUNT (*) WHERE { ?s ?p ?o } ; 11869611740 SQL> sparql SELECT ?p COUNT (*) WHERE { ?s ?p ?o } GROUP BY ?p ORDER BY DESC 2 LIMIT 200 ;</pre> </code> </blockquote> <table> <tr> <th align="center">Predicate URI</th> <th align="center">number of triples</th> </tr> <tr> <td>http://www.w3.org/1999/02/22-rdf-syntax-ns#type</td> <td align="right">881,038,747</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_has_order</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_has_part</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_number</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_discount</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_linestatus</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_shipdate</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_quantity</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_extendedprice</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_commitdate</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_has_supplier</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_tax</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_returnflag</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_receiptdate</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_shipinstruct</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#l_shipmode</td> <td align="right">600,037,902</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#o_clerk</td> <td align="right">150,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#o_comment</td> <td align="right">150,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#c_customer_of</td> <td align="right">150,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#o_orderstatus</td> <td align="right">150,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#o_totalprice</td> <td align="right">150,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#o_orderpriority</td> <td align="right">150,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#o_orderkey</td> <td align="right">150,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#o_orderdate</td> <td align="right">150,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#o_shippriority</td> <td align="right">150,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#ps_has_supplier</td> <td align="right">80,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#ps_availqty</td> <td align="right">80,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#ps_supplycost</td> <td align="right">80,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#ps_has_part</td> <td align="right">80,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#p_type</td> <td align="right">20,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#p_size</td> <td align="right">20,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#p_container</td> <td align="right">20,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#p_mfgr</td> <td align="right">20,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#p_partkey</td> <td align="right">20,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#p_name</td> <td align="right">20,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#p_brand</td> <td align="right">20,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#p_comment</td> <td align="right">20,000,000</td> </tr> <tr> <td>http://xmlns.com/foaf/0.1/phone</td> <td align="right">16,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#c_comment</td> <td align="right">15,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#c_acctbal</td> <td align="right">15,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#c_mktsegment</td> <td align="right">15,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#c_custkey</td> <td align="right">15,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#c_name</td> <td align="right">15,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#c_has_nation</td> <td align="right">15,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#c_address</td> <td align="right">15,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#c_phone</td> <td align="right">15,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#n_nation_of</td> <td align="right">15,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#s_has_nation</td> <td align="right">1,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#s_acctbal</td> <td align="right">1,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#s_name</td> <td align="right">1,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#s_address</td> <td align="right">1,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#s_comment</td> <td align="right">1,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#s_suppkey</td> <td align="right">1,000,000</td> </tr> <tr> <td>http://lod2.eu/schemas/rdfh#s_phone</td> <td align="right">1,000,000</td> </tr> <tr> <td colspan="2" align="center">...</td> </tr> </table> <p> <i>We calculate the ETL speed:</i> </p> <blockquote> <code><pre>SQL> SELECT 11869611740 / ( ( 1858934 + 40686670 ) / 1000.0 ) ; 278985.620700084549276</pre> </code> </blockquote> <p>11.8 hours, at just under 280 Kt/s, end-to-end. Worse has been heard of. This is a small single-server speed, on the usual test system with dual Xeon E5 2630, 192G RAM. A single-server for double the price might get double throughput. Beyond this, scale out is clearly the better deal. An elastic cluster will get throughput linear to the count of machines for this type of workload.</p> <p>This shows that deploying mid-size enterprise data as RDF is a job that goes easily overnight with a commodity box, reading directly from the source system; no file-system-based staging areas are needed.</p> <p>The dataset is 600M order lines; 150M orders; 15M customers; 20M parts, each with 4 suppliers; 1M total suppliers. You can contrast this to what you have in-house to get a rough estimate of what your own DW would come to.</p> <p>Later, we will use this dataset to illustrate how to scope queries to security categories with graph-level security. Of course, this dataset also provides a point of SQL-to-SPARQL comparison for the <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x1c477548">ongoing TPC-H series</a>. There will be more installments in not too long.</p>
2014-01-17T16:52:01-05:00
500 Billion Triples Bulk Load Experiment
http://www.openlinksw.com:443/blog/vdb/blog/?date=2014-01-06#1768
2014-01-06T16:50:15Z
<p>Before Christmas, <a href="http://www.openlinksw.com/weblog/oerling/?id=1763" id="link-id0x2aac7cd09d38">I wrote about a rerun of BSBM</a> to check how it works before doing it on 500 Giga-triples ("Gt") at <a href="http://www.cwi.nl/" id="link-id0x2aac69ec7a88">CWI</a>. Now we can talk about the experiment with CWI's <a href="http://scilens.project.cwi.nl/" id="link-id0x2aac7c8c2778">Scilens</a> cluster. The specs are in <a href="http://www.openlinksw.com/weblog/oerling/?id=1763" id="link-id0x37286678">the previous post</a>. This is a cautionary tale about large data on one hand, and about high load speed on the other.</p> <p>The BSBM generator and driver are an ever-fresh source of new terrors. The nature of the gig is that you have a window to do your experiment in, and that involves first generating the test data. It is somewhere around 3 TB of gzipped files. It took a whole week to make the files. During that time of course you want to anticipate what's going to break with the queries. So while the generator was going, we loaded 50 renamed replicas of the 10 Gt dataset. At partial capacity, we may add, because 4 boxes had half memory taken by the BSBM generator. We hate that program. Of course nobody gives a damn about it so it has been maintained in the worst way possible; for example, the way its cluster version generates slices of data is by having every instance actually generate the full data set, but only write 1 out of so many items to the output. So no amount of capacity will make it faster. As for BSBM itself, if you generate 10 Gt once and occasionally use this as test data, it does not inconvenience you so much. Then, of course, the test driver was patched to generate queries against renamed replicas of a dataset. But then the new driver would not read the dataset summary files made by the previous driver, because of Java class versions. 8 hours to regenerate 10 Gt. A real train wreck. This is by far not the end of it but we are out of space. So on with it; may that program be buried.</p> <p>In the end, the 2000 gz files with the 500 Gt in them were complete. Then it turns out each file has tens of millions of namespace prefixes at the beginning. So, starting to load a file grows the process by some 9 GB just for the prefixes. So, out of 256 GB of RAM per box, there are about 72 GB taken by the prefixes, if you load 8 files in parallel on each. Well, one could do a <code>sed</code> script to unzip, expand the prefixes, and rezip, and the file would not be any bigger; but it would be a day to run.</p> <p>So, anyway with 12 boxes, 24 processes, and (in principle) 384 threads, the load rate is between 3 and 4 million triples per second ("Mt/s"). With 2 boxes, it is 630 Kt/s, so you would say this is scalable. Near enough to linear; the 2 boxes have 12 cores and 2.3GHz, Scilens has 16 at 2.0GHz; close enough.</p> <p>For the 3-4 Mt rate, there is an average of 200 threads running. This is not full platform, as there's the 2nd thread of each core idle for the most part. Adding the second thread usually adds some 30% throughput. A high of 5 Mt/s could be had if going to full CPU, but doubling the files being loaded would run out of memory because of the namespace prefixes. See, it is sheer luck that the BSBM thing, inept as it is, is still marginally usable, despite the prefixes and the horrible generator. A bit worse still, and it would have been a non-starter. It comes from the times when RDF just meant inept database, so scalability clearly was not in its design objectives.</p> <p>With 96 files being loaded across the cluster, we got the run stats below for a couple of 4 minute windows. In each, the data size at time of the sample is between 50 Gt and 100 Gt. The long line is the cluster status summary; the tables below are load rates in the windows between timestamps, so, growth in triple count as triples per second (tps) since the previous sample.</p> <blockquote> <code>Cluster 24 nodes, 240 s. 18866 m/s 692017 KB/s 21842% cpu 7% read 95% clw threads 356r 0w 114i buffers 99250961 97503789 d 2275 w 0 pfs </code> </blockquote> <table align="center" cellspacing="5px"> <tr> <th align="center">load rate <br /> (tps)</th> <th align="center">timestamp</th> </tr> <tr> <td>3,853,915.028323892 </td> <td>2014-01-04 08:38:36 +0000</td> </tr> <tr> <td>4,245,681.678456353</td> <td>2014-01-04 08:38:33 +0000</td> </tr> <tr> <td>3,680,757.080973009</td> <td>2014-01-04 08:38:06 +0000</td> </tr> <tr> <td>4,138,599.125958298</td> <td>2014-01-04 08:38:03 +0000</td> </tr> <tr> <td>4,887,272.575808064</td> <td>2014-01-04 08:37:36 +0000</td> </tr> <tr> <td>4,093,772.082515462</td> <td>2014-01-04 08:37:33 +0000</td> </tr> <tr> <td>4,399,343.552149284</td> <td>2014-01-04 08:37:06 +0000</td> </tr> <tr> <td>4,184,758.045998296</td> <td>2014-01-04 08:37:03 +0000</td> </tr> <tr> <td>3,884,665.444851716</td> <td>2014-01-04 08:36:36 +0000</td> </tr> <tr> <td>4,197,270.027036035</td> <td>2014-01-04 08:36:33 +0000</td> </tr> </table> <p>Some hours later --</p> <blockquote> <code>Cluster 24 nodes, 240 s. 14601 m/s 506784 KB/s 19721% cpu 61% read 1310% clw threads 374r 0w 126i buffers 189886490 107378792 d 1983 w 18 pfs</code> </blockquote> <table align="center" cellspacing="5px"> <tr> <th align="center">load rate <br /> (tps)</th> <th align="center">timestamp</th> </tr> <tr> <td>3,273,757.708076397 </td> <td>2014-01-04 11:49:53 +0000</td> </tr> <tr> <td>3,274,119.596013466</td> <td>2014-01-04 11:49:53 +0000</td> </tr> <tr> <td>3,318,539.715342822</td> <td>2014-01-04 11:49:23 +0000</td> </tr> <tr> <td>3,318,701.609946335</td> <td>2014-01-04 11:49:23 +0000</td> </tr> <tr> <td>3,127,730.142328589</td> <td>2014-01-04 11:48:53 +0000</td> </tr> <tr> <td>3,127,731.608946369</td> <td>2014-01-04 11:48:53 +0000</td> </tr> <tr> <td>3,273,572.647578414</td> <td>2014-01-04 11:48:23 +0000</td> </tr> <tr> <td>3,273,622.779240692</td> <td>2014-01-04 11:48:23 +0000</td> </tr> <tr> <td>2,872,466.21779274</td> <td>2014-01-04 11:47:53 +0000</td> </tr> <tr> <td>2,872,495.383487217</td> <td>2014-01-04 11:47:53 +0000</td> </tr> </table> <p>Pretty good. I don't know of others coming even close.</p> <p>Next we will look at query plans and scalability in query processing.</p>
2014-01-06T17:47:28-05:00
500 Billion Triples Bulk Load Experiment
http://www.openlinksw.com:443/weblog/oerling/?date=2014-01-06#1767
2014-01-06T16:49:45Z
<p>Before Christmas, <a href="http://www.openlinksw.com/weblog/oerling/?id=1763" id="link-id0x2aac7cd09d38">I wrote about a rerun of BSBM</a> to check how it works before doing it on 500 Giga-triples ("Gt") at <a href="http://www.cwi.nl/" id="link-id0x2aac69ec7a88">CWI</a>. Now we can talk about the experiment with CWI's <a href="http://scilens.project.cwi.nl/" id="link-id0x2aac7c8c2778">Scilens</a> cluster. The specs are in <a href="http://www.openlinksw.com/weblog/oerling/?id=1763" id="link-id0x37286678">the previous post</a>. This is a cautionary tale about large data on one hand, and about high load speed on the other.</p> <p>The BSBM generator and driver are an ever-fresh source of new terrors. The nature of the gig is that you have a window to do your experiment in, and that involves first generating the test data. It is somewhere around 3 TB of gzipped files. It took a whole week to make the files. During that time of course you want to anticipate what's going to break with the queries. So while the generator was going, we loaded 50 renamed replicas of the 10 Gt dataset. At partial capacity, we may add, because 4 boxes had half memory taken by the BSBM generator. We hate that program. Of course nobody gives a damn about it so it has been maintained in the worst way possible; for example, the way its cluster version generates slices of data is by having every instance actually generate the full data set, but only write 1 out of so many items to the output. So no amount of capacity will make it faster. As for BSBM itself, if you generate 10 Gt once and occasionally use this as test data, it does not inconvenience you so much. Then, of course, the test driver was patched to generate queries against renamed replicas of a dataset. But then the new driver would not read the dataset summary files made by the previous driver, because of Java class versions. 8 hours to regenerate 10 Gt. A real train wreck. This is by far not the end of it but we are out of space. So on with it; may that program be buried.</p> <p>In the end, the 2000 gz files with the 500 Gt in them were complete. Then it turns out each file has tens of millions of namespace prefixes at the beginning. So, starting to load a file grows the process by some 9 GB just for the prefixes. So, out of 256 GB of RAM per box, there are about 72 GB taken by the prefixes, if you load 8 files in parallel on each. Well, one could do a <code>sed</code> script to unzip, expand the prefixes, and rezip, and the file would not be any bigger; but it would be a day to run.</p> <p>So, anyway with 12 boxes, 24 processes, and (in principle) 384 threads, the load rate is between 3 and 4 million triples per second ("Mt/s"). With 2 boxes, it is 630 Kt/s, so you would say this is scalable. Near enough to linear; the 2 boxes have 12 cores and 2.3GHz, Scilens has 16 at 2.0GHz; close enough.</p> <p>For the 3-4 Mt rate, there is an average of 200 threads running. This is not full platform, as there's the 2nd thread of each core idle for the most part. Adding the second thread usually adds some 30% throughput. A high of 5 Mt/s could be had if going to full CPU, but doubling the files being loaded would run out of memory because of the namespace prefixes. See, it is sheer luck that the BSBM thing, inept as it is, is still marginally usable, despite the prefixes and the horrible generator. A bit worse still, and it would have been a non-starter. It comes from the times when RDF just meant inept database, so scalability clearly was not in its design objectives.</p> <p>With 96 files being loaded across the cluster, we got the run stats below for a couple of 4 minute windows. In each, the data size at time of the sample is between 50 Gt and 100 Gt. The long line is the cluster status summary; the tables below are load rates in the windows between timestamps, so, growth in triple count as triples per second (tps) since the previous sample.</p> <blockquote> <code>Cluster 24 nodes, 240 s. 18866 m/s 692017 KB/s 21842% cpu 7% read 95% clw threads 356r 0w 114i buffers 99250961 97503789 d 2275 w 0 pfs </code> </blockquote> <table align="center" cellspacing="5px"> <tr> <th align="center">load rate <br /> (tps)</th> <th align="center">timestamp</th> </tr> <tr> <td>3,853,915.028323892 </td> <td>2014-01-04 08:38:36 +0000</td> </tr> <tr> <td>4,245,681.678456353</td> <td>2014-01-04 08:38:33 +0000</td> </tr> <tr> <td>3,680,757.080973009</td> <td>2014-01-04 08:38:06 +0000</td> </tr> <tr> <td>4,138,599.125958298</td> <td>2014-01-04 08:38:03 +0000</td> </tr> <tr> <td>4,887,272.575808064</td> <td>2014-01-04 08:37:36 +0000</td> </tr> <tr> <td>4,093,772.082515462</td> <td>2014-01-04 08:37:33 +0000</td> </tr> <tr> <td>4,399,343.552149284</td> <td>2014-01-04 08:37:06 +0000</td> </tr> <tr> <td>4,184,758.045998296</td> <td>2014-01-04 08:37:03 +0000</td> </tr> <tr> <td>3,884,665.444851716</td> <td>2014-01-04 08:36:36 +0000</td> </tr> <tr> <td>4,197,270.027036035</td> <td>2014-01-04 08:36:33 +0000</td> </tr> </table> <p>Some hours later --</p> <blockquote> <code>Cluster 24 nodes, 240 s. 14601 m/s 506784 KB/s 19721% cpu 61% read 1310% clw threads 374r 0w 126i buffers 189886490 107378792 d 1983 w 18 pfs</code> </blockquote> <table align="center" cellspacing="5px"> <tr> <th align="center">load rate <br /> (tps)</th> <th align="center">timestamp</th> </tr> <tr> <td>3,273,757.708076397 </td> <td>2014-01-04 11:49:53 +0000</td> </tr> <tr> <td>3,274,119.596013466</td> <td>2014-01-04 11:49:53 +0000</td> </tr> <tr> <td>3,318,539.715342822</td> <td>2014-01-04 11:49:23 +0000</td> </tr> <tr> <td>3,318,701.609946335</td> <td>2014-01-04 11:49:23 +0000</td> </tr> <tr> <td>3,127,730.142328589</td> <td>2014-01-04 11:48:53 +0000</td> </tr> <tr> <td>3,127,731.608946369</td> <td>2014-01-04 11:48:53 +0000</td> </tr> <tr> <td>3,273,572.647578414</td> <td>2014-01-04 11:48:23 +0000</td> </tr> <tr> <td>3,273,622.779240692</td> <td>2014-01-04 11:48:23 +0000</td> </tr> <tr> <td>2,872,466.21779274</td> <td>2014-01-04 11:47:53 +0000</td> </tr> <tr> <td>2,872,495.383487217</td> <td>2014-01-04 11:47:53 +0000</td> </tr> </table> <p>Pretty good. I don't know of others coming even close.</p> <p>Next we will look at query plans and scalability in query processing.</p>
2014-01-06T17:47:21-05:00
RDF Meets Security
http://www.openlinksw.com:443/blog/vdb/blog/?date=2013-12-17#1766
2013-12-17T17:16:04Z
<p>We regularly encounter user scenarios where there is a need for selective access to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x2aac345ac178">RDF</a> and other <a href="http://dbpedia.org/resource/Graph_database" id="link-id0x2aac3539c228">graph-model</a> data. In a <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x2aac1c9d2218">schema</a>-less situation, the graph is the natural unit of access control. A common design pattern is to put all the triples that represent one logical entity, possibly including immediately dependent entities, into the same graph. The application will then usually replace the whole graph when some part of it changes.</p> <p>In this usage pattern, the graph is akin to a <a href="http://dbpedia.org/resource/Relational_model" id="link-id0x2aac250bc708">relational</a> row, plus maybe items of one to many tables; for example, an order and its order lines would make a single graph. In a publishing setting, what goes into a graph is whatever is approved for publication as a unit. So, a graph is conceptually somewhere between a row and a document.</p> <p>The access control situations further fall into two principal types.</p> <ul> <li>In a <i>hosted application</i> setting, the user will see his own data. Most of the database is invisible.</li> <li>In a <i>data publishing</i> setting, there will typically be a large quantity of data visible to all, plus some premium content visible only for users who pay for this specific content.</li> </ul> <p> <a href="http://www.openphacts.org/about-open-phacts/about-open-phacts" id="link-id0x2aac3514df98">OpenPHACTS</a>, for example, would fall into the latter category. There the matter is not so much about charging for premium content but about keeping proprietary separate from public.</p> <p>The graph is thus a combined <a href="http://dbpedia.org/resource/Provenance" id="link-id0x2aac34a75538">provenance</a> and <a href="http://dbpedia.org/resource/Security" id="link-id0x2aac3071eb28">security</a> label. We sometimes hear about applications that would want quints (a quad with an extra field) for separating concerns, but so far no actionable need has materialized. One could do quints if necessary. We did also <a href="http://www.openlinksw.com/weblog/oerling/?id=1705" id="link-id0x2aac342956f8">in past times</a> talk to <a href="http://www.systap.com/" id="link-id0x2aabf113f258">Systap</a>, the vendor of the <a href="http://www.systap.com/bigdata.htm" id="link-id0x2aac30a3c468">Bigdata®</a> RDF store, about doing <a href="http://dbpedia.org/resource/Reification_%28computer_science%29" id="link-id0x2aac3451dbc8">reification</a> right, which would resolve many of these things in one go.</p> <p>So anyway, it is time to do graph-level security right. The technical approaches depend on how many grantable objects one has, and how many distinct security principals ("grantees") there are. If the separately grantable objects are numerous (e.g., a billion distinct graphs), and if the typical principal has access to most, it makes sense to enumerate for each principal the items that are denied instead of enumerating the granted ones.</p> <p>There is a related case of scoping queries to a variable set of graphs. This corresponds to cases like "give me information from sources with an over-average reliability rating." There is also the security-related special case of scoping to non-classified material. This has the special feature that one part of the query must know what the classified material is, so as to exclude that. However, since from the user's viewpoint the classified material does not exist, the query must run at two different access control levels and must prevent leakage between the two. This is routinely done with SQL views and SQL policy functions so there is nothing new here, but this is just not something the RDF people have thought much of.</p> <p>In the hosted-application scenario there is a slowly-changing, relatively-small set of graphs that are in scope for a user session. Queries will only consider data where the graph is in this set. This set is typically different for each principal. The distinct end users are not extremely numerous; maybe in the thousands.</p> <p>In a publishing setting, there is a small set of restricted content and a large number of principals; however, many principals will have exactly the same exclude list (e.g.: no special access; paid access to content class A or B or both; etc.). The number of access compartments is small relative to the number of principals, and many independent principals will share the same set of premium content. As a result, the number of distinct exclusion lists will not be extremely large, even if the lists may be mid-size.</p> <p>How does one build this into the database? This is done by a selective <a href="http://dbpedia.org/resource/Hash_join" id="link-id0x2aac277fb6c8">hash join</a> for granted lists, and a hash anti-join (<code>NOT EXISTs</code> operator) for the exclusion lists. From the <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac0fc0f288">TPC-H series</a>, we recall that there is an invisible hash join operation that can be merged into a table scan/index lookup. This is the ideal building block for this task, except that it needs to be extended to also have a negative form. The negative form also occurs in <a href="http://www.tpc.org/tpch/" id="link-id0x2aac1e2e89a8">TPC-H</a> -- for example, in Q16, where one considers only stock from suppliers with no customer complaints.</p> <p>There are a few distinct cases of application behavior where the cost of enforcing graph-level access will be different. For example, in OpenPHACTS, a query is often a union/join between graph patterns where the graph is fixed. In this way, each triple pattern has exactly one named graph where it can be matched, but many named graphs are mentioned in the query. In this case, the access check has no cost, or the cost is a vanishingly small constant. In the case of the hosted application, where the named graph corresponds to a unit of update ("row" or "document," as discussed above), the query typically does not specify a named graph, so each triple pattern can match anywhere within the visible graphs. There the check is enforced at each triple pattern, but this is always against the same list (i.e., hash table), and the hash table will most often fit in CPU cache. Most of the accesses will find a match in the hash table.</p> <p>In the publishing case, where relatively small premium content is to be excluded, the named graph will also generally not be specified; but it can happen that some parts of the query are required to match within the same named graph -- i.e., there is a pattern like </p> <blockquote> <code><pre> graph ?g { ?x xx:about <stuff> . ?x xx:date ?dt . filter ( ?dt > ... ) }</pre> </code> </blockquote> <p>Here, the named graph needs to be checked only once for the two patterns. We also note that since most of the content is not restricted, the check against the restriction will nearly always fail -- i.e., the hash lookup will miss. Missing is cheaper than hitting, as mentioned in the TPC-H series. One can use a <a href="http://dbpedia.org/resource/Bloom_filter" id="link-id0x2aac02b4c848">Bloom filter</a> for low cost miss detection. So it follows that an exclusion list that seldom hits can be maybe 10x bigger than a near-always-hitting inclusion list with the same cost of checking.</p> <p>So, if inclusion lists are in the 100,000-entry range and exclusion lists in the 1,000,000-item range, we are set. What if this is not so? In that case, we will rely on two-level tricks and encoding of application information into supposedly meaning-free identifiers. This is not sem-head material but is well within the domain of database science. For example, if the graph is a marker of provenance or update locality, then all graphs of the same owner or security classification may have an identifier in a certain range or ranges. Then we may only check for range, i.e., omit the low few bits from the hash lookup, knowing that any graph id in the range will have the same security behavior. This does introduce some more burden on the application, and will add some special cases when moving assets between security principals, but then life is like that. If there is a big-ticket application that depends on this, then such things can be provided as custom development.</p> <p>I will here refer you to <a href="http://www.linkedin.com/in/davidkarger" id="link-id0x2aac0d749ba8">David Karger</a>'s immortal insight at the <a href="http://2013.eswc-conferences.org/program/panel" id="link-id0x2aac268822f8">Semantic Big Data panel</a> at <a href="http://2013.eswc-conferences.org/" id="link-id0x2aac27327ef8">ESWC 2013</a>, where <a href="http://www.openlinksw.com/weblog/oerling/?id=1730" id="link-id0x2aac3529f308">he said that</a> "big data is very much about performance, and performance is usually obtained by sacrificing the general for the specific." So it is.</p> <p>So far, I have made a summary where the many people with whom this matter has been touched upon may recognize their specific case. Next I will show some experiments. We use a copy of the <a href="http://www.openlinksw.com/RPC2" id="link-id0x2aac35527f38">URIBurner</a> dataset, as this is a collection with large numbers of tiny graphs that can be assigned different security attributes.</p> <p>To be continued.</p>
2013-12-17T12:16:04.000017-05:00
RDF Meets Security
http://www.openlinksw.com:443/weblog/oerling/?date=2013-12-17#1765
2013-12-17T17:15:51Z
<p>We regularly encounter user scenarios where there is a need for selective access to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x2aac345ac178">RDF</a> and other <a href="http://dbpedia.org/resource/Graph_database" id="link-id0x2aac3539c228">graph-model</a> data. In a <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x2aac1c9d2218">schema</a>-less situation, the graph is the natural unit of access control. A common design pattern is to put all the triples that represent one logical entity, possibly including immediately dependent entities, into the same graph. The application will then usually replace the whole graph when some part of it changes.</p> <p>In this usage pattern, the graph is akin to a <a href="http://dbpedia.org/resource/Relational_model" id="link-id0x2aac250bc708">relational</a> row, plus maybe items of one to many tables; for example, an order and its order lines would make a single graph. In a publishing setting, what goes into a graph is whatever is approved for publication as a unit. So, a graph is conceptually somewhere between a row and a document.</p> <p>The access control situations further fall into two principal types.</p> <ul> <li>In a <i>hosted application</i> setting, the user will see his own data. Most of the database is invisible.</li> <li>In a <i>data publishing</i> setting, there will typically be a large quantity of data visible to all, plus some premium content visible only for users who pay for this specific content.</li> </ul> <p> <a href="http://www.openphacts.org/about-open-phacts/about-open-phacts" id="link-id0x2aac3514df98">OpenPHACTS</a>, for example, would fall into the latter category. There the matter is not so much about charging for premium content but about keeping proprietary separate from public.</p> <p>The graph is thus a combined <a href="http://dbpedia.org/resource/Provenance" id="link-id0x2aac34a75538">provenance</a> and <a href="http://dbpedia.org/resource/Security" id="link-id0x2aac3071eb28">security</a> label. We sometimes hear about applications that would want quints (a quad with an extra field) for separating concerns, but so far no actionable need has materialized. One could do quints if necessary. We did also <a href="http://www.openlinksw.com/weblog/oerling/?id=1705" id="link-id0x2aac342956f8">in past times</a> talk to <a href="http://www.systap.com/" id="link-id0x2aabf113f258">Systap</a>, the vendor of the <a href="http://www.systap.com/bigdata.htm" id="link-id0x2aac30a3c468">Bigdata®</a> RDF store, about doing <a href="http://dbpedia.org/resource/Reification_%28computer_science%29" id="link-id0x2aac3451dbc8">reification</a> right, which would resolve many of these things in one go.</p> <p>So anyway, it is time to do graph-level security right. The technical approaches depend on how many grantable objects one has, and how many distinct security principals ("grantees") there are. If the separately grantable objects are numerous (e.g., a billion distinct graphs), and if the typical principal has access to most, it makes sense to enumerate for each principal the items that are denied instead of enumerating the granted ones.</p> <p>There is a related case of scoping queries to a variable set of graphs. This corresponds to cases like "give me information from sources with an over-average reliability rating." There is also the security-related special case of scoping to non-classified material. This has the special feature that one part of the query must know what the classified material is, so as to exclude that. However, since from the user's viewpoint the classified material does not exist, the query must run at two different access control levels and must prevent leakage between the two. This is routinely done with SQL views and SQL policy functions so there is nothing new here, but this is just not something the RDF people have thought much of.</p> <p>In the hosted-application scenario there is a slowly-changing, relatively-small set of graphs that are in scope for a user session. Queries will only consider data where the graph is in this set. This set is typically different for each principal. The distinct end users are not extremely numerous; maybe in the thousands.</p> <p>In a publishing setting, there is a small set of restricted content and a large number of principals; however, many principals will have exactly the same exclude list (e.g.: no special access; paid access to content class A or B or both; etc.). The number of access compartments is small relative to the number of principals, and many independent principals will share the same set of premium content. As a result, the number of distinct exclusion lists will not be extremely large, even if the lists may be mid-size.</p> <p>How does one build this into the database? This is done by a selective <a href="http://dbpedia.org/resource/Hash_join" id="link-id0x2aac277fb6c8">hash join</a> for granted lists, and a hash anti-join (<code>NOT EXISTs</code> operator) for the exclusion lists. From the <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac0fc0f288">TPC-H series</a>, we recall that there is an invisible hash join operation that can be merged into a table scan/index lookup. This is the ideal building block for this task, except that it needs to be extended to also have a negative form. The negative form also occurs in <a href="http://www.tpc.org/tpch/" id="link-id0x2aac1e2e89a8">TPC-H</a> -- for example, in Q16, where one considers only stock from suppliers with no customer complaints.</p> <p>There are a few distinct cases of application behavior where the cost of enforcing graph-level access will be different. For example, in OpenPHACTS, a query is often a union/join between graph patterns where the graph is fixed. In this way, each triple pattern has exactly one named graph where it can be matched, but many named graphs are mentioned in the query. In this case, the access check has no cost, or the cost is a vanishingly small constant. In the case of the hosted application, where the named graph corresponds to a unit of update ("row" or "document," as discussed above), the query typically does not specify a named graph, so each triple pattern can match anywhere within the visible graphs. There the check is enforced at each triple pattern, but this is always against the same list (i.e., hash table), and the hash table will most often fit in CPU cache. Most of the accesses will find a match in the hash table.</p> <p>In the publishing case, where relatively small premium content is to be excluded, the named graph will also generally not be specified; but it can happen that some parts of the query are required to match within the same named graph -- i.e., there is a pattern like </p> <blockquote> <code><pre> graph ?g { ?x xx:about <stuff> . ?x xx:date ?dt . filter ( ?dt > ... ) }</pre> </code> </blockquote> <p>Here, the named graph needs to be checked only once for the two patterns. We also note that since most of the content is not restricted, the check against the restriction will nearly always fail -- i.e., the hash lookup will miss. Missing is cheaper than hitting, as mentioned in the TPC-H series. One can use a <a href="http://dbpedia.org/resource/Bloom_filter" id="link-id0x2aac02b4c848">Bloom filter</a> for low cost miss detection. So it follows that an exclusion list that seldom hits can be maybe 10x bigger than a near-always-hitting inclusion list with the same cost of checking.</p> <p>So, if inclusion lists are in the 100,000-entry range and exclusion lists in the 1,000,000-item range, we are set. What if this is not so? In that case, we will rely on two-level tricks and encoding of application information into supposedly meaning-free identifiers. This is not sem-head material but is well within the domain of database science. For example, if the graph is a marker of provenance or update locality, then all graphs of the same owner or security classification may have an identifier in a certain range or ranges. Then we may only check for range, i.e., omit the low few bits from the hash lookup, knowing that any graph id in the range will have the same security behavior. This does introduce some more burden on the application, and will add some special cases when moving assets between security principals, but then life is like that. If there is a big-ticket application that depends on this, then such things can be provided as custom development.</p> <p>I will here refer you to <a href="http://www.linkedin.com/in/davidkarger" id="link-id0x2aac0d749ba8">David Karger</a>'s immortal insight at the <a href="http://2013.eswc-conferences.org/program/panel" id="link-id0x2aac268822f8">Semantic Big Data panel</a> at <a href="http://2013.eswc-conferences.org/" id="link-id0x2aac27327ef8">ESWC 2013</a>, where <a href="http://www.openlinksw.com/weblog/oerling/?id=1730" id="link-id0x2aac3529f308">he said that</a> "big data is very much about performance, and performance is usually obtained by sacrificing the general for the specific." So it is.</p> <p>So far, I have made a summary where the many people with whom this matter has been touched upon may recognize their specific case. Next I will show some experiments. We use a copy of the <a href="http://www.openlinksw.com/RPC2" id="link-id0x2aac35527f38">URIBurner</a> dataset, as this is a collection with large numbers of tiny graphs that can be assigned different security attributes.</p> <p>To be continued.</p>
2013-12-17T12:15:51.000001-05:00
BSBM Benchmark Season, 2013-2014
http://www.openlinksw.com:443/blog/vdb/blog/?date=2013-12-16#1764
2013-12-16T16:33:43Z
<p>We are getting ready to run <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x2a77ece8">BSBM</a> benchmarks at a 500 billion triple scale over the holidays. This will be done on the <a href="http://scilens.project.cwi.nl/" id="link-id0x23335c88">Scilens</a> cluster at <a href="http://www.cwi.nl/" id="link-id0x2a490868">CWI</a>. The system has 16 nodes, each with 2x Xeon E5-2650, 256 GB RAM, and QDR InfiniBand. We will use 12 of these, for a total of 3 TB RAM. I will blog about the experiences in January, after the experiment is done. This is the final <a href="http://lod2.eu/" id="link-id0x2c1b90d8">LOD2 (EU FP7 project)</a> benchmarking piece.</p> <p>I will here give results for practice runs on my desktop, with 1/50th the data and 1/8th the capacity. This is 10 billion triples on a system of two machines, each with 2x Xeon E5-2630, 192 GB RAM, and QDR InfiniBand.</p> <p>We start with Explore with 16 clients. The clients are evenly divided over 4 server processes, with 2 processes per machine. We do a run of 100 -- </p> <blockquote> <code><pre> % ./bibm/bsbmdriver -seed 1287654 -dg http://bsbm.org -t 300000 \ -idir /1d4/bsbm_10000/td_data -uqp query -uc bsbm/explore \ -mt 16 -runs 500 http://madras:8604/sparql \ http://madras:8605/sparql http://masala-i:8606/sparql \ http://masala-i:8607/sparql </pre> </code> </blockquote> <p>The QMPH (query mixes per hour) is 12683.046. The run is 1500 query mixes; it takes 425s. The warmup is about 3000 query mixes with a different seed. The details are in <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/Holidays2013BSBMBackup/10gc16e.xml" id="link-id0x2b071908"><code>10gc16e.xml</code></a>. The sample configurations are as in <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/Holidays2013BSBMBackup/virtuoso.global.ini" id="link-id0x1bc61a18"><code>virtuoso.global.ini</code></a>, <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/Holidays2013BSBMBackup/cluster.global.ini" id="link-id0x1ebe7ef8"><code>cluster.global.ini</code></a>, and <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/Holidays2013BSBMBackup/virtuoso.ini" id="link-id0x29ec7988"><code>virtuoso.ini</code></a>.</p> <p>We note that there is a total of 20M 8 KB buffer pages, and after running 1000 or so different query mixes, about 16M get used. So the working set is about 16M * 8 KB = 128 GB, for 10 Gt ("Gigatriples"). The quads themselves take less space than that, but the benchmark also accesses some literals. At 50x the size and at best 2.5 TB worth of buffers, there may be a problem.</p> <p>The total database files are around 800K pages * 8 KB/slice * 48 slices = 272 GB. This times 50 is 15.3 TB. I do not think the system has that much SSD space, and it has about 3 TB per node in 3-way striped RAID 0 disk. There will be some disk access during the explore run. So we will report one number with steady state from disk, and another for a rerun of a set of queries where data is known to come from memory.</p> <p>We note that there is speculative read, taking whole extents in when not all pages get used. Whether one reads 8 KB or 2 MB (the extent) makes little difference, so may as well do whole extents. Subtracting the speculatively-read pages that are not in fact accessed, we get 1.5M working set per box, which would indicate that we will make it into a RAM-based steady-state on the 500 Gt Scilens experiment. We shall see.</p> <p>Loading may present some problems, since last time we had two boxes with significantly worse disk-write throughput than the rest. The Virtuoso I/O system is now different, with more emphasis on writing contiguous sequential ranges of pages, irrespective of the time the page became dirty. But there is nothing that a bad disk will not screw up.</p> <p>We go to BI. First single user (power) run. This is preceded by one single user BI run with a different seed, for warmup. The power run has 4 consecutive query mixes; the throughput run has the same 4 query mixes concurrently.</p> <p>Power query mix run time: 229 (arithmetic mean)</p> <p>Throughput query mix run time: 269s (arithmetic mean)</p> <p>The test driver output follows, the full result summaries are in <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/Holidays2013BSBMBackup/10gc-4pwer.xml" id="link-id0x2aef4b98"><code>10gc-4pwer.xml</code></a> and <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/Holidays2013BSBMBackup/10gc-4tp.xml" id="link-id0x2b04fe98"><code>10gc-4tp.xml</code></a>.</p> <h3>Power Results</h3> <blockquote> <code><pre> % ./bibm/bsbmdriver -drill -t 300000 -dg http://bsbm.org -idir \ /1d4/bsbm_10000/td_data -uqp query -uc bsbm/bi -mt 1 -runs 4 \ http://madras:8604/sparql % java -Xmx256M com.openlinksw.bibm.bsbm.TestDriver -qrd ./bibm \ -dg http://bsbm/ -drill -t 300000 -dg http://bsbm.org -idir \ /1d4/bsbm_10000/td_data -uqp query -uc bsbm/bi -mt 1 -runs 4 \ http://madras:8604/sparql Thread 1: query mix: 0 255.074 s, total: 255.195 s Thread 1: query mix: 1 170.622 s, total: 170.667 s Thread 1: query mix: 2 295.642 s, total: 295.691 s Thread 1: query mix: 3 188.885 s, total: 188.935 s Benchmark run completed in 910.493 s </pre> </code> </blockquote> <table id="PowerResults" title="PowerResults" align="center" cellspacing="5px"> <thead> <th align="center">Query Number</th> <th align="center">Execute Count</th> <th align="center">Timeshare</th> <th align="center">aqet</th> <th align="center">aqetg</th> <th align="center">aps</th> <th align="center">minqet</th> <th align="center">maxqet</th> <th align="center">Average Results</th> <th align="center">Min Results</th> <th align="center">Max Results</th> <th align="center">Timeout Count</th> </thead> <tr> <th align="center">1</th> <td align="center">4</td> <td align="right">4.079</td> <td align="right">9.281250</td> <td align="right">7.848979</td> <td align="right">0.108</td> <td align="right">4.141000</td> <td align="right">18.836000</td> <td align="right">10.000</td> <td align="center">10</td> <td align="center">10</td> <td align="center">0</td> </tr> <tr> <th align="center">2</th> <td align="center">4</td> <td align="right">2.358</td> <td align="right">5.365000</td> <td align="right">4.907928</td> <td align="right">0.186</td> <td align="right">2.539000</td> <td align="right">8.182000</td> <td align="right">10.000</td> <td align="center">10</td> <td align="center">10</td> <td align="center">0</td> </tr> <tr> <th align="center">3</th> <td align="center">4</td> <td align="right">25.639</td> <td align="right">58.343500</td> <td align="right">24.902196</td> <td align="right">0.017</td> <td align="right">2.640000</td> <td align="right">111.680000</td> <td align="right">10.000</td> <td align="center">10</td> <td align="center">10</td> <td align="center">0</td> </tr> <tr> <th align="center">4</th> <td align="center">20</td> <td align="right">28.908</td> <td align="right">13.156450</td> <td align="right">1.725366</td> <td align="right">0.076</td> <td align="right">0.130000</td> <td align="right">73.728000</td> <td align="right">92.650</td> <td align="center">55</td> <td align="center">100</td> <td align="center">0</td> </tr> <tr> <th align="center">5</th> <td align="center">20</td> <td align="right">12.229</td> <td align="right">5.565350</td> <td align="right">2.550107</td> <td align="right">0.180</td> <td align="right">0.202000</td> <td align="right">16.251000</td> <td align="right">30.250</td> <td align="center">14</td> <td align="center">58</td> <td align="center">0</td> </tr> <tr> <th align="center">6</th> <td align="center">4</td> <td align="right">0.277</td> <td align="right">0.631250</td> <td align="right">0.574319</td> <td align="right">1.584</td> <td align="right">0.269000</td> <td align="right">0.950000</td> <td align="right">49.250</td> <td align="center">14</td> <td align="center">72</td> <td align="center">0</td> </tr> <tr> <th align="center">7</th> <td align="center">24</td> <td align="right">3.875</td> <td align="right">1.469625</td> <td align="right">0.268809</td> <td align="right">0.680</td> <td align="right">0.056000</td> <td align="right">7.961000</td> <td align="right">54.875</td> <td align="center">0</td> <td align="center">413</td> <td align="center">0</td> </tr> <tr> <th align="center">8</th> <td align="center">20</td> <td align="right">22.635</td> <td align="right">10.301600</td> <td align="right">5.450917</td> <td align="right">0.097</td> <td align="right">0.626000</td> <td align="right">36.908000</td> <td align="right">10.000</td> <td align="center">10</td> <td align="center">10</td> <td align="center">0</td> </tr> </table> <h3>Throughput Results</h3> <blockquote> <code><pre> % ./bibm/bsbmdriver -drill -t 300000 -dg http://bsbm.org -idir \ /1d4/bsbm_10000/td_data -uqp query -uc bsbm/bi -mt 4 -runs 4 \ http://madras:8604/sparql http://madras:8605/sparql \ http://masala-i:8606/sparql http://masala-i:8607/sparql % java -Xmx256M com.openlinksw.bibm.bsbm.TestDriver -qrd ./bibm \ -dg http://bsbm/ -drill -t 300000 -dg http://bsbm.org -idir \ /1d4/bsbm_10000/td_data -uqp query -uc bsbm/bi -mt 4 -runs 4 \ http://madras:8604/sparql http://madras:8605/sparql \ http://masala-i:8606/sparql http://masala-i:8607/sparql Thread 2: query mix: 1 474.435 s, total: 474.498 s Thread 1: query mix: 0 669.552 s, total: 669.663 s Thread 3: query mix: 3 914.943 s, total: 915.009 s Thread 4: query mix: 2 1077.138 s, total: 1077.283 s Benchmark run completed in 1077.285 s % </pre> </code> </blockquote> <table id="ThroughputResults" title="ThroughputResults" align="center" cellspacing="5px"> <thead> <th align="center">Query Number</th> <th align="center">Execute Count</th> <th align="center">Timeshare</th> <th align="center">aqet</th> <th align="center">aqetg</th> <th align="center">aps</th> <th align="center">minqet</th> <th align="center">maxqet</th> <th align="center">Average Results</th> <th align="center">Min Results</th> <th align="center">Max Results</th> <th align="center">Timeout Count</th> </thead> <tr> <th align="center">1</th> <td align="center">4</td> <td align="right">2.478</td> <td align="right">19.424500</td> <td align="right">12.522236</td> <td align="right">0.150</td> <td align="right">4.207000</td> <td align="right">52.115000</td> <td align="right">10.000</td> <td align="center">10</td> <td align="center">10</td> <td align="center">0</td> </tr> <tr> <th align="center">2</th> <td align="center">4</td> <td align="right">1.188</td> <td align="right">9.312250</td> <td align="right">7.525659</td> <td align="right">0.313</td> <td align="right">2.116000</td> <td align="right">14.486000</td> <td align="right">10.000</td> <td align="center">10</td> <td align="center">10</td> <td align="center">0</td> </tr> <tr> <th align="center">3</th> <td align="center">4</td> <td align="right">17.822</td> <td align="right">139.727250</td> <td align="right">79.815584</td> <td align="right">0.021</td> <td align="right">19.953000</td> <td align="right">268.652000</td> <td align="right">10.000</td> <td align="center">10</td> <td align="center">10</td> <td align="center">0</td> </tr> <tr> <th align="center">4</th> <td align="center">20</td> <td align="right">47.737</td> <td align="right">74.853750</td> <td align="right">4.014962</td> <td align="right">0.039</td> <td align="right">0.132000</td> <td align="right">728.240000</td> <td align="right">92.650</td> <td align="center">55</td> <td align="center">100</td> <td align="center">0</td> </tr> <tr> <th align="center">5</th> <td align="center">20</td> <td align="right">10.393</td> <td align="right">16.296000</td> <td align="right">6.905212</td> <td align="right">0.179</td> <td align="right">0.406000</td> <td align="right">76.083000</td> <td align="right">30.250</td> <td align="center">14</td> <td align="center">58</td> <td align="center">0</td> </tr> <tr> <th align="center">6</th> <td align="center">4</td> <td align="right">1.557</td> <td align="right">12.209000</td> <td align="right">2.304804</td> <td align="right">0.238</td> <td align="right">0.364000</td> <td align="right">45.836000</td> <td align="right">49.250</td> <td align="center">14</td> <td align="center">72</td> <td align="center">0</td> </tr> <tr> <th align="center">7</th> <td align="center">24</td> <td align="right">3.795</td> <td align="right">4.958458</td> <td align="right">0.876038</td> <td align="right">0.587</td> <td align="right">0.049000</td> <td align="right">33.648000</td> <td align="right">54.708</td> <td align="center">0</td> <td align="center">413</td> <td align="center">0</td> </tr> <tr> <th align="center">8</th> <td align="center">20</td> <td align="right">15.031</td> <td align="right">23.568900</td> <td align="right">9.495984</td> <td align="right">0.124</td> <td align="right">0.614000</td> <td align="right">157.444000</td> <td align="right">10.000</td> <td align="center">10</td> <td align="center">10</td> <td align="center">0</td> </tr> </table> <p>We notice quite a bit of variability between the different query mixes. This comes from parameter choices, and runs with different seeds are therefore not comparable, unless they be very long.</p> <p>What does this promise for the 500 Gt runs? The complexities are <i><code>n·log(n)</code></i>, with the log pretty constant. There will be some loss of speed from less locality of reference. I expect run times that are 8x or so higher, 50x more data, and about 8x more CPU. The dataset does not scale that linearly throughout, as the product hierarchies may have different depth.</p> <p>The working set will be OK; on each of the 4 processes, there are 3.1M buffers used, of which 2M are just read ahead, not really hit. When you read, read the whole extent of 256 pages while at it; costs the same and may prefetch. So actually, 4.4M buffers used is 34 GB, times 50 is 1.7 TB. Will fit.</p> <p>In the interest of advancing standards of disclosure, I am also providing the <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/Holidays2013BSBMBackup/exrun.txt" id="link-id0x2c291f88">test driver output</a> for the runs, and an excerpt of the server query log for an <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/Holidays2013BSBMBackup/qlogexp.txt" id="link-id0x2aac0c24bfe8">interactive query mix</a> and a <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/Holidays2013BSBMBackup/qlogbi.txt" id="link-id0x2415b618">BI query mix</a>. The query texts and plans are there, with per operator time and cardinality.</p> <h2>Related</h2> <ul> <li> <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/Holidays2013BSBMBackup" id="link-id0x2aac117db2e8">All supporting files</a> </li> </ul>
2013-12-16T14:02:20.000001-05:00
BSBM Benchmark Season, 2013-2014
http://www.openlinksw.com:443/weblog/oerling/?date=2013-12-16#1763
2013-12-16T16:30:39Z
<p>We are getting ready to run <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x2a77ece8">BSBM</a> benchmarks at a 500 billion triple scale over the holidays. This will be done on the <a href="http://scilens.project.cwi.nl/" id="link-id0x23335c88">Scilens</a> cluster at <a href="http://www.cwi.nl/" id="link-id0x2a490868">CWI</a>. The system has 16 nodes, each with 2x Xeon E5-2650, 256 GB RAM, and QDR InfiniBand. We will use 12 of these, for a total of 3 TB RAM. I will blog about the experiences in January, after the experiment is done. This is the final <a href="http://lod2.eu/" id="link-id0x2c1b90d8">LOD2 (EU FP7 project)</a> benchmarking piece.</p> <p>I will here give results for practice runs on my desktop, with 1/50th the data and 1/8th the capacity. This is 10 billion triples on a system of two machines, each with 2x Xeon E5-2630, 192 GB RAM, and QDR InfiniBand.</p> <p>We start with Explore with 16 clients. The clients are evenly divided over 4 server processes, with 2 processes per machine. We do a run of 100 -- </p> <blockquote> <code><pre> % ./bibm/bsbmdriver -seed 1287654 -dg http://bsbm.org -t 300000 \ -idir /1d4/bsbm_10000/td_data -uqp query -uc bsbm/explore \ -mt 16 -runs 500 http://madras:8604/sparql \ http://madras:8605/sparql http://masala-i:8606/sparql \ http://masala-i:8607/sparql </pre> </code> </blockquote> <p>The QMPH (query mixes per hour) is 12683.046. The run is 1500 query mixes; it takes 425s. The warmup is about 3000 query mixes with a different seed. The details are in <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/Holidays2013BSBMBackup/10gc16e.xml" id="link-id0x2b071908"><code>10gc16e.xml</code></a>. The sample configurations are as in <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/Holidays2013BSBMBackup/virtuoso.global.ini" id="link-id0x1bc61a18"><code>virtuoso.global.ini</code></a>, <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/Holidays2013BSBMBackup/cluster.global.ini" id="link-id0x1ebe7ef8"><code>cluster.global.ini</code></a>, and <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/Holidays2013BSBMBackup/virtuoso.ini" id="link-id0x29ec7988"><code>virtuoso.ini</code></a>.</p> <p>We note that there is a total of 20M 8 KB buffer pages, and after running 1000 or so different query mixes, about 16M get used. So the working set is about 16M * 8 KB = 128 GB, for 10 Gt ("Gigatriples"). The quads themselves take less space than that, but the benchmark also accesses some literals. At 50x the size and at best 2.5 TB worth of buffers, there may be a problem.</p> <p>The total database files are around 800K pages * 8 KB/slice * 48 slices = 272 GB. This times 50 is 15.3 TB. I do not think the system has that much SSD space, and it has about 3 TB per node in 3-way striped RAID 0 disk. There will be some disk access during the explore run. So we will report one number with steady state from disk, and another for a rerun of a set of queries where data is known to come from memory.</p> <p>We note that there is speculative read, taking whole extents in when not all pages get used. Whether one reads 8 KB or 2 MB (the extent) makes little difference, so may as well do whole extents. Subtracting the speculatively-read pages that are not in fact accessed, we get 1.5M working set per box, which would indicate that we will make it into a RAM-based steady-state on the 500 Gt Scilens experiment. We shall see.</p> <p>Loading may present some problems, since last time we had two boxes with significantly worse disk-write throughput than the rest. The Virtuoso I/O system is now different, with more emphasis on writing contiguous sequential ranges of pages, irrespective of the time the page became dirty. But there is nothing that a bad disk will not screw up.</p> <p>We go to BI. First single user (power) run. This is preceded by one single user BI run with a different seed, for warmup. The power run has 4 consecutive query mixes; the throughput run has the same 4 query mixes concurrently.</p> <p>Power query mix run time: 229 (arithmetic mean)</p> <p>Throughput query mix run time: 269s (arithmetic mean)</p> <p>The test driver output follows, the full result summaries are in <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/Holidays2013BSBMBackup/10gc-4pwer.xml" id="link-id0x2aef4b98"><code>10gc-4pwer.xml</code></a> and <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/Holidays2013BSBMBackup/10gc-4tp.xml" id="link-id0x2b04fe98"><code>10gc-4tp.xml</code></a>.</p> <h3>Power Results</h3> <blockquote> <code><pre> % ./bibm/bsbmdriver -drill -t 300000 -dg http://bsbm.org -idir \ /1d4/bsbm_10000/td_data -uqp query -uc bsbm/bi -mt 1 -runs 4 \ http://madras:8604/sparql % java -Xmx256M com.openlinksw.bibm.bsbm.TestDriver -qrd ./bibm \ -dg http://bsbm/ -drill -t 300000 -dg http://bsbm.org -idir \ /1d4/bsbm_10000/td_data -uqp query -uc bsbm/bi -mt 1 -runs 4 \ http://madras:8604/sparql Thread 1: query mix: 0 255.074 s, total: 255.195 s Thread 1: query mix: 1 170.622 s, total: 170.667 s Thread 1: query mix: 2 295.642 s, total: 295.691 s Thread 1: query mix: 3 188.885 s, total: 188.935 s Benchmark run completed in 910.493 s </pre> </code> </blockquote> <table id="PowerResults" title="PowerResults" align="center" cellspacing="5px"> <thead> <th align="center">Query Number</th> <th align="center">Execute Count</th> <th align="center">Timeshare</th> <th align="center">aqet</th> <th align="center">aqetg</th> <th align="center">aps</th> <th align="center">minqet</th> <th align="center">maxqet</th> <th align="center">Average Results</th> <th align="center">Min Results</th> <th align="center">Max Results</th> <th align="center">Timeout Count</th> </thead> <tr> <th align="center">1</th> <td align="center">4</td> <td align="right">4.079</td> <td align="right">9.281250</td> <td align="right">7.848979</td> <td align="right">0.108</td> <td align="right">4.141000</td> <td align="right">18.836000</td> <td align="right">10.000</td> <td align="center">10</td> <td align="center">10</td> <td align="center">0</td> </tr> <tr> <th align="center">2</th> <td align="center">4</td> <td align="right">2.358</td> <td align="right">5.365000</td> <td align="right">4.907928</td> <td align="right">0.186</td> <td align="right">2.539000</td> <td align="right">8.182000</td> <td align="right">10.000</td> <td align="center">10</td> <td align="center">10</td> <td align="center">0</td> </tr> <tr> <th align="center">3</th> <td align="center">4</td> <td align="right">25.639</td> <td align="right">58.343500</td> <td align="right">24.902196</td> <td align="right">0.017</td> <td align="right">2.640000</td> <td align="right">111.680000</td> <td align="right">10.000</td> <td align="center">10</td> <td align="center">10</td> <td align="center">0</td> </tr> <tr> <th align="center">4</th> <td align="center">20</td> <td align="right">28.908</td> <td align="right">13.156450</td> <td align="right">1.725366</td> <td align="right">0.076</td> <td align="right">0.130000</td> <td align="right">73.728000</td> <td align="right">92.650</td> <td align="center">55</td> <td align="center">100</td> <td align="center">0</td> </tr> <tr> <th align="center">5</th> <td align="center">20</td> <td align="right">12.229</td> <td align="right">5.565350</td> <td align="right">2.550107</td> <td align="right">0.180</td> <td align="right">0.202000</td> <td align="right">16.251000</td> <td align="right">30.250</td> <td align="center">14</td> <td align="center">58</td> <td align="center">0</td> </tr> <tr> <th align="center">6</th> <td align="center">4</td> <td align="right">0.277</td> <td align="right">0.631250</td> <td align="right">0.574319</td> <td align="right">1.584</td> <td align="right">0.269000</td> <td align="right">0.950000</td> <td align="right">49.250</td> <td align="center">14</td> <td align="center">72</td> <td align="center">0</td> </tr> <tr> <th align="center">7</th> <td align="center">24</td> <td align="right">3.875</td> <td align="right">1.469625</td> <td align="right">0.268809</td> <td align="right">0.680</td> <td align="right">0.056000</td> <td align="right">7.961000</td> <td align="right">54.875</td> <td align="center">0</td> <td align="center">413</td> <td align="center">0</td> </tr> <tr> <th align="center">8</th> <td align="center">20</td> <td align="right">22.635</td> <td align="right">10.301600</td> <td align="right">5.450917</td> <td align="right">0.097</td> <td align="right">0.626000</td> <td align="right">36.908000</td> <td align="right">10.000</td> <td align="center">10</td> <td align="center">10</td> <td align="center">0</td> </tr> </table> <h3>Throughput Results</h3> <blockquote> <code><pre> % ./bibm/bsbmdriver -drill -t 300000 -dg http://bsbm.org -idir \ /1d4/bsbm_10000/td_data -uqp query -uc bsbm/bi -mt 4 -runs 4 \ http://madras:8604/sparql http://madras:8605/sparql \ http://masala-i:8606/sparql http://masala-i:8607/sparql % java -Xmx256M com.openlinksw.bibm.bsbm.TestDriver -qrd ./bibm \ -dg http://bsbm/ -drill -t 300000 -dg http://bsbm.org -idir \ /1d4/bsbm_10000/td_data -uqp query -uc bsbm/bi -mt 4 -runs 4 \ http://madras:8604/sparql http://madras:8605/sparql \ http://masala-i:8606/sparql http://masala-i:8607/sparql Thread 2: query mix: 1 474.435 s, total: 474.498 s Thread 1: query mix: 0 669.552 s, total: 669.663 s Thread 3: query mix: 3 914.943 s, total: 915.009 s Thread 4: query mix: 2 1077.138 s, total: 1077.283 s Benchmark run completed in 1077.285 s % </pre> </code> </blockquote> <table id="ThroughputResults" title="ThroughputResults" align="center" cellspacing="5px"> <thead> <th align="center">Query Number</th> <th align="center">Execute Count</th> <th align="center">Timeshare</th> <th align="center">aqet</th> <th align="center">aqetg</th> <th align="center">aps</th> <th align="center">minqet</th> <th align="center">maxqet</th> <th align="center">Average Results</th> <th align="center">Min Results</th> <th align="center">Max Results</th> <th align="center">Timeout Count</th> </thead> <tr> <th align="center">1</th> <td align="center">4</td> <td align="right">2.478</td> <td align="right">19.424500</td> <td align="right">12.522236</td> <td align="right">0.150</td> <td align="right">4.207000</td> <td align="right">52.115000</td> <td align="right">10.000</td> <td align="center">10</td> <td align="center">10</td> <td align="center">0</td> </tr> <tr> <th align="center">2</th> <td align="center">4</td> <td align="right">1.188</td> <td align="right">9.312250</td> <td align="right">7.525659</td> <td align="right">0.313</td> <td align="right">2.116000</td> <td align="right">14.486000</td> <td align="right">10.000</td> <td align="center">10</td> <td align="center">10</td> <td align="center">0</td> </tr> <tr> <th align="center">3</th> <td align="center">4</td> <td align="right">17.822</td> <td align="right">139.727250</td> <td align="right">79.815584</td> <td align="right">0.021</td> <td align="right">19.953000</td> <td align="right">268.652000</td> <td align="right">10.000</td> <td align="center">10</td> <td align="center">10</td> <td align="center">0</td> </tr> <tr> <th align="center">4</th> <td align="center">20</td> <td align="right">47.737</td> <td align="right">74.853750</td> <td align="right">4.014962</td> <td align="right">0.039</td> <td align="right">0.132000</td> <td align="right">728.240000</td> <td align="right">92.650</td> <td align="center">55</td> <td align="center">100</td> <td align="center">0</td> </tr> <tr> <th align="center">5</th> <td align="center">20</td> <td align="right">10.393</td> <td align="right">16.296000</td> <td align="right">6.905212</td> <td align="right">0.179</td> <td align="right">0.406000</td> <td align="right">76.083000</td> <td align="right">30.250</td> <td align="center">14</td> <td align="center">58</td> <td align="center">0</td> </tr> <tr> <th align="center">6</th> <td align="center">4</td> <td align="right">1.557</td> <td align="right">12.209000</td> <td align="right">2.304804</td> <td align="right">0.238</td> <td align="right">0.364000</td> <td align="right">45.836000</td> <td align="right">49.250</td> <td align="center">14</td> <td align="center">72</td> <td align="center">0</td> </tr> <tr> <th align="center">7</th> <td align="center">24</td> <td align="right">3.795</td> <td align="right">4.958458</td> <td align="right">0.876038</td> <td align="right">0.587</td> <td align="right">0.049000</td> <td align="right">33.648000</td> <td align="right">54.708</td> <td align="center">0</td> <td align="center">413</td> <td align="center">0</td> </tr> <tr> <th align="center">8</th> <td align="center">20</td> <td align="right">15.031</td> <td align="right">23.568900</td> <td align="right">9.495984</td> <td align="right">0.124</td> <td align="right">0.614000</td> <td align="right">157.444000</td> <td align="right">10.000</td> <td align="center">10</td> <td align="center">10</td> <td align="center">0</td> </tr> </table> <p>We notice quite a bit of variability between the different query mixes. This comes from parameter choices, and runs with different seeds are therefore not comparable, unless they be very long.</p> <p>What does this promise for the 500 Gt runs? The complexities are <i><code>n·log(n)</code></i>, with the log pretty constant. There will be some loss of speed from less locality of reference. I expect run times that are 8x or so higher, 50x more data, and about 8x more CPU. The dataset does not scale that linearly throughout, as the product hierarchies may have different depth.</p> <p>The working set will be OK; on each of the 4 processes, there are 3.1M buffers used, of which 2M are just read ahead, not really hit. When you read, read the whole extent of 256 pages while at it; costs the same and may prefetch. So actually, 4.4M buffers used is 34 GB, times 50 is 1.7 TB. Will fit.</p> <p>In the interest of advancing standards of disclosure, I am also providing the <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/Holidays2013BSBMBackup/exrun.txt" id="link-id0x2c291f88">test driver output</a> for the runs, and an excerpt of the server query log for an <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/Holidays2013BSBMBackup/qlogexp.txt" id="link-id0x2aac0c24bfe8">interactive query mix</a> and a <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/Holidays2013BSBMBackup/qlogbi.txt" id="link-id0x2415b618">BI query mix</a>. The query texts and plans are there, with per operator time and cardinality.</p> <h2>Related</h2> <ul> <li> <a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/Holidays2013BSBMBackup" id="link-id0x2aac117db2e8">All supporting files</a> </li> </ul>
2013-12-16T14:02:14.000002-05:00
Virtuoso Meets OpenStreetMap: Explorations in ETL
http://www.openlinksw.com:443/blog/vdb/blog/?date=2013-12-06#1762
2013-12-06T17:53:35Z
<p> <a href="http://dbpedia.org/resource/OpenStreetMap" id="link-id0x2aac4d351368">OpenStreetMap</a> (OSM) is one of the largest open datasets. The <a href="http://geoknow.eu/" id="link-id0x2aac4c531528">GeoKnow</a> project of which OpenLink is a part is, among other things, drawing on this resource. The EU FP7 project <a href="http://lod2.eu/" id="link-id0x2aac4d430f48">LOD2</a> has developed a <a href="http://dbpedia.org/resource/Geospatial_analysis" id="link-id0x2aac4c9837f8">geospatial</a> drill-down benchmark for <a href="http://dbpedia.org/resource/RDF" id="link-id0x2aac4d1cbbc8">RDF</a> and here we will be applying it to <a href="http://dbpedia.org/resource/PostGIS" id="link-id0x2aac4c7a0b88">PostGIS</a> and <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aac4d252e48">Virtuoso</a> in plain <a href="http://dbpedia.org/resource/SQL" id="link-id0x2aac4d2874f8">SQL</a>.</p> <p>Generic relational performance is the necessary predecessor of any graph performance, so we will talk SQL at first; the Linked Geodata RDFisms will come later.</p> <p>In this article, we will look in detail at <a href="http://dbpedia.org/resource/Extract,_transform,_load" id="link-id0x2aac4c40d7a8">ETL> from PostGIS to Virtuoso via <a href="http://dbpedia.org/resource/Federated_database_system" id="link-id0x2aac4c7cec68">SQL federation</a>. In the <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac4d32c3f8">TPC-H series</a>, we looked at bulk loading from files which are 1:1 shaped like the tables, but life is seldom this simple.</a> </p> <p>Here we will see how to change normalization in schemas, from a denormalized key-value pair-structure in PostGIS, to a normalized "triple table" in Virtuoso. We will also look at data type conversion, overall data transfer speed, and automatic parallelization.</p> <p>ETL, even with medium data sizes, like with OSM at a little under 600 GB in <a href="http://dbpedia.org/resource/PostgreSQL" id="link-id0x2aac4d4beaf8">PostgreSQL</a> files, is a performance game, like everything in database. Data must move fast, expressing the transformation logic must be compact, and parallelism must be automatic. Next to nobody can write parallel code and the few that can are needed somewhere else.</p> <p>I suppose, without insider knowledge, I would dump the data into <a href="http://dbpedia.org/resource/Comma-separated_values" id="link-id0x2aac4cfb2828">CSV</a>; do some <code>sed</code> scripts or the like for the transformation, maybe in Hadoop if the data were really large; and then I would use the target database's bulk load utility. This makes the steps so simple that they can be delegated with some possibility of success. This is what data integration tends to be like. As we saw with the <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac4c955718">TPC-H bulk load</a>, CSV loading is foolproof, easy, and fast.</p> <p>Further, I would not make a JDBC program to first read one database and write into another because this would have to be explicitly multithreaded, would have loops, would require use of array parameters in order not to get killed by client server latency, would be liable to run into oddities of JDBC implementations, and so forth. Plus, this could be a few hundred lines long, and the developer would come back with questions like, "Why is it slow?" Well, it is slow because of lock contention, because transactions are not turned off, or something of the sort. No. Shell scripts and bulk load anytime.</p> <p>Now we will explore a third possibility: vectored stored procedures. It is true that nobody uses stored procedures. They are sooo nineties -- where's the client side Javascript? I will introduce a design pattern that runs table-to-table copy and normalization changes, with perfect parallelism and scale-out, in SQL procedures. This will work from the file system as well, since a CSV file can be accessed as a table. For number of code lines, time-to-solution, as well as run-time performance, this is unbeatable.</p> <h2>Schema Choices</h2> <p>The LOD2 project developed a <a href="http://svn.aksw.org/lod2/D5.1.2/LOD2_D5%202%201_Initial_GEO_Benchmark_Definition.pdf" id="link-id0x2aabc63dc0b8">benchmark for geo retrieval in SPARQL</a>. We have adapted the benchmark to work in SQL against the PostgreSQL OSM schema and a Virtuoso SQL equivalent.</p> <p>The intent is to run the LOD2 geobench against the planet-wide OSM dataset in PostgreSQL and Virtuoso. With Virtuoso we will also compare scale-out and single server versions.</p> <p>The PostgreSQL OSM implementation exists in both normalized and denormalized variants. The denormalized variant uses an <code><a href="http://dbpedia.org/resource/H-Store" id="link-id0x2aac4d29db78">H-Store</a></code> column type, which is a built-in non-first-normal-form set of key-value pairs that can occur as a column value. In Virtuoso, the equivalent would be to use an array in a column value, but this is not very efficient. Rather, we will go the normalized route, getting outstanding <code>JOIN</code> performance and space efficiency from the column store. Since this is a freestyle race, we take the liberty of borrowing the <a href="http://www.openlinksw.com/dataspace/doc/dav/wiki/Main/VOSSQLSPARQL" id="link-id0x2aac4d7baa58"><code>IRI</code> data type</a> from the RDF side of Virtuoso. This offers a fast mapping between names and integer identifiers. This is especially handy for tags. PostgreSQL likely has some similar encoding as part of the H-Store implementation.</p> <p>The geometry types are transferred as strings, and then re-parsed into the Virtuoso equivalents. The <code>EWKT</code> syntax is compatible between the systems. The potentially long geometries are stored in a <code>LONG ANY</code> column, and the always short ones (e.g., bounding boxes and points) into an <code>ANY</code> column. In both implementations, there is an <a href="http://dbpedia.org/resource/R-tree" id="link-id0x2aac4c1a7348">R-tree index</a> on the points but not on the linestrings.</p> <p>We will later look at space consumption and access locality in more detail.</p> <p>To ETL the PostgreSQL based dataset, we attach the OSM tables as remote tables using <a href="http://virtuoso.openlinksw.com/" id="link-id0x2aac4c338f18">Virtuoso</a>'s SQL federation (<a href="http://virtuoso.openlinksw.com/middleware/" id="link-id0x2aac4d36c3c8">VDB</a>) feature. This is <a href="http://virtuoso.openlinksw.com/features-comparison-matrix/" id="link-id0x2aac4d2a6d98">not in the Open Source Edition</a> (<a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSIndex" id="link-id0x2aac4c9dca68">VOS</a>) but you can get the same effect by dumping the tables into files, and defining the files as tables with the file-table feature.</p> <p>The tables which have no need of special transformation go with just an <code>INSERT ... SELECT</code>, like this:</p> <blockquote> <code><pre> log_enable (2); INSERT INTO users SELECT * FROM users1 & </pre> </code> </blockquote> <p>The tables which have special datatypes (like geometries or <code>H-Stores</code>) need a little application logic, like this:</p> <blockquote> <code><pre> CREATE PROCEDURE copy_ways () { log_enable (2); RETURN ( SELECT COUNT (ins_ways ( id, version, user_id, tstamp, changeset_id, tags, linestring_wkt, bbox_wkt ) ) FROM ways1) ; } </pre> </code> </blockquote> <p>The first line disables logging and makes inserts non-transactional. The rest does the copy. The scan of the remote table is automatically split by ranges of its primary key, so there is no need for explicit parallelism. The <code>ins_ways</code> function is called on each thread, on a whole vector of values for each column. In this way operations are batched together, gaining by locality, and eliminating interpretation overhead.</p> <p>The ins_ways procedure follows:</p> <blockquote> <code><pre> CREATE PROCEDURE ins_ways ( IN id BIGINT, IN version INT, IN user_id INT, IN tstamp DATETIME, IN changeset_id BIGINT, IN tags ANY ARRAY, IN linestring VARCHAR, IN bbox VARCHAR ) { </pre> </code> </blockquote> <p> <i>-- The vectored declaration means that each statement is run on the full input before going to the next.</i> </p> <p> <i>-- Thus, by default, the insert gets 10K consecutive rows to insert. The conversion functions like <code>st_ewkt_read</code> are also run in a tight loop over a large number of values.</i> </p> <blockquote> <code><pre> VECTORED; INSERT INTO ways VALUES ( id, version, user_id, tstamp, changeset_id, st_ewkt_read ( charset_recode ( linestring, '_WIDE_', 'UTF-8' ) ), st_ewkt_read ( charset_recode ( bbox, '_WIDE_', 'UTF-8' ) ) ) ; </pre> </code> </blockquote> <p> <i>-- The <code>tags</code> is a vector of strings where each string is a serialization of the H-Store content. <code>split_and_decode</code> splits each string into an array at the delimiter.</i> </p> <blockquote> <code><pre> tags := split_and_decode ( TRIM ( REPLACE ( REPLACE ( REPLACE ( REPLACE ( tags, '"=>"', '!!!' ), '&', '%26' ), '", "', '&' ), '=', '%3D' ), '"' ) ); NOT VECTORED { DECLARE a1, b1 VARCHAR ; DECLARE ws, vs, ts ANY ARRAY ; DECLARE n_sets, n_tags, set_no, wid, inx, pos, fill INT ; </pre> </code> </blockquote> <p> <i>-- We insert triples of the form <code>tag, way_id, tag_value</code>. For each of these, we reserve an array of 100K elements. We put the values into the array, and insert when full or when all rows of input are done. An insert of 100K values in one go is much faster than inserting 100K values singly, especially on a cluster.</i> </p> <blockquote> <code><pre> ws := make_array (100000, 'ANY'); ts := make_array (100000, 'ANY'); vs := make_array (100000, 'ANY'); fill := 0; DECLARE tag_arr, str ANY ARRAY; n_sets := vec_length (tags); </pre> </code> </blockquote> <p> <i>-- For each row of input to the vectored function:</i> </p> <blockquote> <code><pre> FOR ( set_no := 0 ; set_no < n_sets ; set_no := set_no + 1 ) { wid := vec_ref (id, set_no); tag_arr := vec_ref (tags, set_no); n_tags := LENGTH (tag_arr); </pre> </code> </blockquote> <p> <i>-- for each tag in the H-Store string:</i> </p> <blockquote> <code><pre> FOR ( inx := 0; inx < n_tags; inx := inx + 2) { </pre> </code> </blockquote> <p> <i>-- split the tag into a key and a value at the <code>!!!</code> delimiter</i> </p> <blockquote> <code><pre> str := tag_arr[inx]; pos := strstr(str, '!!!'); a1 := substring(str, 1, pos); b1 := subseq(str, pos + 3); </pre> </code> </blockquote> <p> <i>-- add to the array of key-value pairs to insert </i> </p> <blockquote> <code><pre> way_tag_add (ws, ts, vs, fill, wid, a1, b1); } } way_tag_ins (ws, ts, vs); } } </pre> </code> </blockquote> <p>Now we define the functions for adding a <code>way, key, value</code> triple into the batch, and for inserting the batch.</p> <blockquote> <code><pre> CREATE PROCEDURE way_tag_ins ( INOUT ws ANY ARRAY, INOUT ts ANY ARRAY, INOUT vs ANY ARRAY ) { </pre> </code> </blockquote> <p> <i>-- given an array of way ids, tag names, and tag values, insert all rows where the tag is not <code>0</code>. If the tag is empty, call it unknown instead.</i> </p> <p> <i>-- The <code>__i2id</code> function replaces the tag name with an <code>IRI ID</code> that is persistently mapped to the name. The insert and the tag name-to-id mapping are done as a single operation. This is a single network round trip for each in a cluster setting. </i> </p> <blockquote> <code><pre> FOR VECTORED ( IN wid INT := ws, IN tag ANY := ts, IN val VARCHAR := vs ) { IF (tag <> 0) { IF ('' = tag) tag := 'unknown'; INSERT INTO ways_tags VALUES ( __i2id (tag), wid, val ); } } } CREATE PROCEDURE way_tag_add ( INOUT ws ANY ARRAY, INOUT ts ANY ARRAY, INOUT vs ANY ARRAY, INOUT fill INT, IN wid INT, INOUT tg VARCHAR, INOUT val VARCHAR ) { </pre> </code> </blockquote> <p> <i>-- Add at the end of the arrays; if full, insert the content and replace with fresh arrays.</i> </p> <p> <i>-- The <code>INOUT</code> keyword means call by reference, which is important; you do not want to copy larger arrays, and you want to return new ones to the caller. </i> </p> <blockquote> <code><pre> ws[fill] := wid; ts[fill] := tg; vs[fill] := val; fill := fill + 1; IF (100000 = fill) { way_tag_ins (ws, ts, vs); fill := 0; ws := make_array (100000, 'ANY'); ts := make_array (100000, 'ANY'); vs := make_array (100000, 'ANY'); } } </pre> </code> </blockquote> <p>The same logic can be applied to any simple data transformation task. Vectoring and automatic parallelism make sure that there is full platform utilization without explicitly working with threads. The <code>NOT VECTORED {}</code> section allows the procedure to aggregate over all the values in a vector. The <code>FOR VECTORED</code> construct in the <code>INSERT</code> function switches back into running on a vector composed in the scalar part so as to get the insert throughput and cluster-friendly message pattern.</p> <p>Because every non-1MF hack in every application is different, it is not possible to make this fully declarative. But the code is very repetitive and a skeleton could be easily generated from the schema.</p> <p>In the next installment, we we will analyze the performance of copying the full Open Street Map dataset from PostgreSQL to Virtuoso. To be continued...</p>
2013-12-07T12:38:57-05:00
Virtuoso Meets OpenStreetMap: Explorations in ETL
http://www.openlinksw.com:443/weblog/oerling/?date=2013-12-06#1761
2013-12-06T17:53:23Z
<p> <a href="http://dbpedia.org/resource/OpenStreetMap" id="link-id0x2aac4d351368">OpenStreetMap</a> (OSM) is one of the largest open datasets. The <a href="http://geoknow.eu/" id="link-id0x2aac4c531528">GeoKnow</a> project of which OpenLink is a part is, among other things, drawing on this resource. The EU FP7 project <a href="http://lod2.eu/" id="link-id0x2aac4d430f48">LOD2</a> has developed a <a href="http://dbpedia.org/resource/Geospatial_analysis" id="link-id0x2aac4c9837f8">geospatial</a> drill-down benchmark for <a href="http://dbpedia.org/resource/RDF" id="link-id0x2aac4d1cbbc8">RDF</a> and here we will be applying it to <a href="http://dbpedia.org/resource/PostGIS" id="link-id0x2aac4c7a0b88">PostGIS</a> and <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aac4d252e48">Virtuoso</a> in plain <a href="http://dbpedia.org/resource/SQL" id="link-id0x2aac4d2874f8">SQL</a>.</p> <p>Generic relational performance is the necessary predecessor of any graph performance, so we will talk SQL at first; the Linked Geodata RDFisms will come later.</p> <p>In this article, we will look in detail at <a href="http://dbpedia.org/resource/Extract,_transform,_load" id="link-id0x2aac4c40d7a8">ETL> from PostGIS to Virtuoso via <a href="http://dbpedia.org/resource/Federated_database_system" id="link-id0x2aac4c7cec68">SQL federation</a>. In the <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac4d32c3f8">TPC-H series</a>, we looked at bulk loading from files which are 1:1 shaped like the tables, but life is seldom this simple.</a> </p> <p>Here we will see how to change normalization in schemas, from a denormalized key-value pair-structure in PostGIS, to a normalized "triple table" in Virtuoso. We will also look at data type conversion, overall data transfer speed, and automatic parallelization.</p> <p>ETL, even with medium data sizes, like with OSM at a little under 600 GB in <a href="http://dbpedia.org/resource/PostgreSQL" id="link-id0x2aac4d4beaf8">PostgreSQL</a> files, is a performance game, like everything in database. Data must move fast, expressing the transformation logic must be compact, and parallelism must be automatic. Next to nobody can write parallel code and the few that can are needed somewhere else.</p> <p>I suppose, without insider knowledge, I would dump the data into <a href="http://dbpedia.org/resource/Comma-separated_values" id="link-id0x2aac4cfb2828">CSV</a>; do some <code>sed</code> scripts or the like for the transformation, maybe in Hadoop if the data were really large; and then I would use the target database's bulk load utility. This makes the steps so simple that they can be delegated with some possibility of success. This is what data integration tends to be like. As we saw with the <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac4c955718">TPC-H bulk load</a>, CSV loading is foolproof, easy, and fast.</p> <p>Further, I would not make a JDBC program to first read one database and write into another because this would have to be explicitly multithreaded, would have loops, would require use of array parameters in order not to get killed by client server latency, would be liable to run into oddities of JDBC implementations, and so forth. Plus, this could be a few hundred lines long, and the developer would come back with questions like, "Why is it slow?" Well, it is slow because of lock contention, because transactions are not turned off, or something of the sort. No. Shell scripts and bulk load anytime.</p> <p>Now we will explore a third possibility: vectored stored procedures. It is true that nobody uses stored procedures. They are sooo nineties -- where's the client side Javascript? I will introduce a design pattern that runs table-to-table copy and normalization changes, with perfect parallelism and scale-out, in SQL procedures. This will work from the file system as well, since a CSV file can be accessed as a table. For number of code lines, time-to-solution, as well as run-time performance, this is unbeatable.</p> <h2>Schema Choices</h2> <p>The LOD2 project developed a <a href="http://svn.aksw.org/lod2/D5.1.2/LOD2_D5%202%201_Initial_GEO_Benchmark_Definition.pdf" id="link-id0x2aab6ed32c68">benchmark for geo retrieval in SPARQL</a>. We have adapted the benchmark to work in SQL against the PostgreSQL OSM schema and a Virtuoso SQL equivalent.</p> <p>The intent is to run the LOD2 geobench against the planet-wide OSM dataset in PostgreSQL and Virtuoso. With Virtuoso we will also compare scale-out and single server versions.</p> <p>The PostgreSQL OSM implementation exists in both normalized and denormalized variants. The denormalized variant uses an <code><a href="http://dbpedia.org/resource/H-Store" id="link-id0x2aac4d29db78">H-Store</a></code> column type, which is a built-in non-first-normal-form set of key-value pairs that can occur as a column value. In Virtuoso, the equivalent would be to use an array in a column value, but this is not very efficient. Rather, we will go the normalized route, getting outstanding <code>JOIN</code> performance and space efficiency from the column store. Since this is a freestyle race, we take the liberty of borrowing the <a href="http://www.openlinksw.com/dataspace/doc/dav/wiki/Main/VOSSQLSPARQL" id="link-id0x2aac4d7baa58"><code>IRI</code> data type</a> from the RDF side of Virtuoso. This offers a fast mapping between names and integer identifiers. This is especially handy for tags. PostgreSQL likely has some similar encoding as part of the H-Store implementation.</p> <p>The geometry types are transferred as strings, and then re-parsed into the Virtuoso equivalents. The <code>EWKT</code> syntax is compatible between the systems. The potentially long geometries are stored in a <code>LONG ANY</code> column, and the always short ones (e.g., bounding boxes and points) into an <code>ANY</code> column. In both implementations, there is an <a href="http://dbpedia.org/resource/R-tree" id="link-id0x2aac4c1a7348">R-tree index</a> on the points but not on the linestrings.</p> <p>We will later look at space consumption and access locality in more detail.</p> <p>To ETL the PostgreSQL based dataset, we attach the OSM tables as remote tables using <a href="http://virtuoso.openlinksw.com/" id="link-id0x2aac4c338f18">Virtuoso</a>'s SQL federation (<a href="http://virtuoso.openlinksw.com/middleware/" id="link-id0x2aac4d36c3c8">VDB</a>) feature. This is <a href="http://virtuoso.openlinksw.com/features-comparison-matrix/" id="link-id0x2aac4d2a6d98">not in the Open Source Edition</a> (<a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSIndex" id="link-id0x2aac4c9dca68">VOS</a>) but you can get the same effect by dumping the tables into files, and defining the files as tables with the file-table feature.</p> <p>The tables which have no need of special transformation go with just an <code>INSERT ... SELECT</code>, like this:</p> <blockquote> <code><pre> log_enable (2); INSERT INTO users SELECT * FROM users1 & </pre> </code> </blockquote> <p>The tables which have special datatypes (like geometries or <code>H-Stores</code>) need a little application logic, like this:</p> <blockquote> <code><pre> CREATE PROCEDURE copy_ways () { log_enable (2); RETURN ( SELECT COUNT (ins_ways ( id, version, user_id, tstamp, changeset_id, tags, linestring_wkt, bbox_wkt ) ) FROM ways1) ; } </pre> </code> </blockquote> <p>The first line disables logging and makes inserts non-transactional. The rest does the copy. The scan of the remote table is automatically split by ranges of its primary key, so there is no need for explicit parallelism. The <code>ins_ways</code> function is called on each thread, on a whole vector of values for each column. In this way operations are batched together, gaining by locality, and eliminating interpretation overhead.</p> <p>The ins_ways procedure follows:</p> <blockquote> <code><pre> CREATE PROCEDURE ins_ways ( IN id BIGINT, IN version INT, IN user_id INT, IN tstamp DATETIME, IN changeset_id BIGINT, IN tags ANY ARRAY, IN linestring VARCHAR, IN bbox VARCHAR ) { </pre> </code> </blockquote> <p> <i>-- The vectored declaration means that each statement is run on the full input before going to the next.</i> </p> <p> <i>-- Thus, by default, the insert gets 10K consecutive rows to insert. The conversion functions like <code>st_ewkt_read</code> are also run in a tight loop over a large number of values.</i> </p> <blockquote> <code><pre> VECTORED; INSERT INTO ways VALUES ( id, version, user_id, tstamp, changeset_id, st_ewkt_read ( charset_recode ( linestring, '_WIDE_', 'UTF-8' ) ), st_ewkt_read ( charset_recode ( bbox, '_WIDE_', 'UTF-8' ) ) ) ; </pre> </code> </blockquote> <p> <i>-- The <code>tags</code> is a vector of strings where each string is a serialization of the H-Store content. <code>split_and_decode</code> splits each string into an array at the delimiter.</i> </p> <blockquote> <code><pre> tags := split_and_decode ( TRIM ( REPLACE ( REPLACE ( REPLACE ( REPLACE ( tags, '"=>"', '!!!' ), '&', '%26' ), '", "', '&' ), '=', '%3D' ), '"' ) ); NOT VECTORED { DECLARE a1, b1 VARCHAR ; DECLARE ws, vs, ts ANY ARRAY ; DECLARE n_sets, n_tags, set_no, wid, inx, pos, fill INT ; </pre> </code> </blockquote> <p> <i>-- We insert triples of the form <code>tag, way_id, tag_value</code>. For each of these, we reserve an array of 100K elements. We put the values into the array, and insert when full or when all rows of input are done. An insert of 100K values in one go is much faster than inserting 100K values singly, especially on a cluster.</i> </p> <blockquote> <code><pre> ws := make_array (100000, 'ANY'); ts := make_array (100000, 'ANY'); vs := make_array (100000, 'ANY'); fill := 0; DECLARE tag_arr, str ANY ARRAY; n_sets := vec_length (tags); </pre> </code> </blockquote> <p> <i>-- For each row of input to the vectored function:</i> </p> <blockquote> <code><pre> FOR ( set_no := 0 ; set_no < n_sets ; set_no := set_no + 1 ) { wid := vec_ref (id, set_no); tag_arr := vec_ref (tags, set_no); n_tags := LENGTH (tag_arr); </pre> </code> </blockquote> <p> <i>-- for each tag in the H-Store string:</i> </p> <blockquote> <code><pre> FOR ( inx := 0; inx < n_tags; inx := inx + 2) { </pre> </code> </blockquote> <p> <i>-- split the tag into a key and a value at the <code>!!!</code> delimiter</i> </p> <blockquote> <code><pre> str := tag_arr[inx]; pos := strstr(str, '!!!'); a1 := substring(str, 1, pos); b1 := subseq(str, pos + 3); </pre> </code> </blockquote> <p> <i>-- add to the array of key-value pairs to insert </i> </p> <blockquote> <code><pre> way_tag_add (ws, ts, vs, fill, wid, a1, b1); } } way_tag_ins (ws, ts, vs); } } </pre> </code> </blockquote> <p>Now we define the functions for adding a <code>way, key, value</code> triple into the batch, and for inserting the batch.</p> <blockquote> <code><pre> CREATE PROCEDURE way_tag_ins ( INOUT ws ANY ARRAY, INOUT ts ANY ARRAY, INOUT vs ANY ARRAY ) { </pre> </code> </blockquote> <p> <i>-- given an array of way ids, tag names, and tag values, insert all rows where the tag is not <code>0</code>. If the tag is empty, call it unknown instead.</i> </p> <p> <i>-- The <code>__i2id</code> function replaces the tag name with an <code>IRI ID</code> that is persistently mapped to the name. The insert and the tag name-to-id mapping are done as a single operation. This is a single network round trip for each in a cluster setting. </i> </p> <blockquote> <code><pre> FOR VECTORED ( IN wid INT := ws, IN tag ANY := ts, IN val VARCHAR := vs ) { IF (tag <> 0) { IF ('' = tag) tag := 'unknown'; INSERT INTO ways_tags VALUES ( __i2id (tag), wid, val ); } } } CREATE PROCEDURE way_tag_add ( INOUT ws ANY ARRAY, INOUT ts ANY ARRAY, INOUT vs ANY ARRAY, INOUT fill INT, IN wid INT, INOUT tg VARCHAR, INOUT val VARCHAR ) { </pre> </code> </blockquote> <p> <i>-- Add at the end of the arrays; if full, insert the content and replace with fresh arrays.</i> </p> <p> <i>-- The <code>INOUT</code> keyword means call by reference, which is important; you do not want to copy larger arrays, and you want to return new ones to the caller. </i> </p> <blockquote> <code><pre> ws[fill] := wid; ts[fill] := tg; vs[fill] := val; fill := fill + 1; IF (100000 = fill) { way_tag_ins (ws, ts, vs); fill := 0; ws := make_array (100000, 'ANY'); ts := make_array (100000, 'ANY'); vs := make_array (100000, 'ANY'); } } </pre> </code> </blockquote> <p>The same logic can be applied to any simple data transformation task. Vectoring and automatic parallelism make sure that there is full platform utilization without explicitly working with threads. The <code>NOT VECTORED {}</code> section allows the procedure to aggregate over all the values in a vector. The <code>FOR VECTORED</code> construct in the <code>INSERT</code> function switches back into running on a vector composed in the scalar part so as to get the insert throughput and cluster-friendly message pattern.</p> <p>Because every non-1MF hack in every application is different, it is not possible to make this fully declarative. But the code is very repetitive and a skeleton could be easily generated from the schema.</p> <p>In the next installment, we we will analyze the performance of copying the full Open Street Map dataset from PostgreSQL to Virtuoso. To be continued...</p>
2013-12-07T12:38:31.000002-05:00
LDBC Technical User Community (TUC), 3rd Meeting
http://www.openlinksw.com:443/blog/vdb/blog/?date=2013-11-27#1760
2013-11-27T17:21:05Z
<p>Last week the RDF and graph DB benchmarking project, <a href="http://ldbc.eu/" id="link-id0x2aac4eb7ad38">LDBC</a>, had its <a href="http://ldbc.eu:8090/display/TUC/Third%20TUC%20Meeting%2C%20November%202013" id="link-id0x2aac4f0c0318">3rd Technical User Community meeting</a> in London, held in collaboration with the <a href="http://www.graphconnect.com/london/" id="link-id0x2aac4f3aa708">GraphConnect event</a>. This meeting marks the official launch of the LDBC non-profit company which is the successor of the present EU FP7 project.</p> <p>The meeting was very well attended, along with most of the new advisory board. <a href="http://www.linkedin.com/pub/xavier-lopez/b/871/788" id="link-id0x2aac4eff7128">Xavier Lopez</a> from <a href="http://www.linkedin.com/company/1028" id="link-id0x2aac4e939db8">Oracle</a>, <a href="http://homes.cs.washington.edu/~luisceze/" id="link-id0x2aac4e601588">Luis Ceze</a> from the <a href="http://dbpedia.org/resource/University_of_Washington" id="link-id0x2aac4ec862d8">University of Washington</a>, and <a href="http://www.ifi.uzh.ch/ddis/people/bernstein.html" id="link-id0x2aac4c0e5978">Abraham Bernstein</a> of the <a href="http://dbpedia.org/resource/University_of_Zurich" id="link-id0x2aac37e0b168">University of Zurich</a> were present. <a href="http://www.linkedin.com/pub/jans-aasman/0/107/272" id="link-id0x2aac4f353f68">Jans Aasman</a> of <a href="http://www.linkedin.com/company/167333" id="link-id0x2aac4d786768">Franz, Inc.</a>, and <a href="http://www.linkedin.com/pub/karl-huppler/8/ba9/8b4" id="link-id0x2aac4e9cca38">Karl Huppler</a>, former chairman of the <a href="http://dbpedia.org/resource/Transaction_Processing_Performance_Council" id="link-id0x2aac4e1df248">TPC</a>, were not present but are signed up as advisory board members.</p> <p>We had great talks by the new board members and invited graph and RDF DB users. </p> <p> <a href="http://www.gsd.inesc-id.pt/~nonius/" id="link-id0x2aac4f893878">Nuno Carvalho</a> of <a href="http://www.fujitsu.com/emea/about/fle/" id="link-id0x2aac4f893798">Fujitsu Labs</a> presented on the <a href="http://ldbc.eu:8090/download/attachments/4325436/LDBC-TUC-Fujitsu-Final.pdf?version=1&modificationDate=1385291853000" id="link-id0x2aac4f893778">Fujitsu RDF use cases and benchmarking requirements</a>, based around analytics streaming on time series of streaming data. The technology platform is diverse, with anything from RDF stores to <a href="http://dbpedia.org/resource/HBase" id="link-id0x2aac4f893738">HBase</a>. The challenge is integration. I pointed out that with <a href="http://virtuoso.openlinksw.com/" id="link-id0x2aac4f483818">Virtuoso</a> column store, you could now efficiently host time series data alongside RDF. Sure, a relational format is more efficient with time series data, but it can be co-located with RDF, and queries can join between the two. This is especially so after our stellar bulk-load speed measured with the TPC-H dataset.</p> <p>Luis Ceze of Washington University presented <a href="http://sampa.cs.washington.edu/grappa/" id="link-id0x2aac4f8936d8">Grappa</a>, a C++ graph programming framework that in his words would be like <a href="http://dbpedia.org/resource/Cray_XMT" id="link-id0x2aac4f284308">Cray XMT</a>, later <a href="http://www.linkedin.com/company/yarcdata" id="link-id0x2aac4f06a098">Yarc Data</a>, in software. The idea is to have a graph algorithm divided into small executable steps, millions in number, and to have very efficient scheduling and switching between these, building latency tolerance into every step of the application. Commodity interconnects like <a href="http://dbpedia.org/resource/InfiniBand" id="link-id0x2aac4f83ee28">InfiniBand</a> deliver bad throughput with small messages, but with endless message combination opportunities from millions of mini work units, the overall throughput stays good. We know the same from all the Virtuoso scale-out work. Luis is presently working on <a href="http://labs.oracle.com/pls/apex/f?p=labs:50:0:::::#ero-400" id="link-id0x2aac4f0e38a8">GraphBench</a>, a research project at Washington State funded by Oracle for graph algorithm benchmarking. The major interest for LDBC is in having a library of common graph analytics as a starting point. Having these, the data generation can further evolve so as to create challenges for the algorithms. One issue that came up is the question of validating graph algorithm results: Unlike in SQL queries, there is not necessarily a single correct answer. If the algorithm to use and the count of iterations to run is not fully specified, response times will vary widely. Random walks will anyway create variation between consecutive runs.</p> <p>Abraham Bernstein presented about the work on his <a href="http://uzh.github.io/signal-collect/" id="link-id0x2aac4ea858d8">Signal/Collect</a> graph programming framework and its applications in fraud detection. He also talked about the EU FP7 project <a href="http://vista-tv.eu/" id="link-id0x2aac4f2cb818">ViSTA-TV</a> which does massive stream processing around the real time behavior of internet TV users. Again, Abraham gave very direct suggestions for what to include in the LDBC graph analytics workload.</p> <p> <a href="http://www.linkedin.com/pub/andreas-both/22/672/942" id="link-id0x2aac4e733c18">Andreas Both</a> of <a href="http://www.linkedin.com/company/124775" id="link-id0x2aac4ea3c158">Unister</a> presented on RDF ontology-driven applications in an e-commerce context. Unister is Germany’s leading e-commerce portal operator with a large number of properties ranging across travel to most B2C. The RDF use cases are many, in principle down to final content distribution but high online demand often calls for specialized solutions like bit field intersections for combining conditions. Sufficiently advanced database technology may also offer this but this is not a guarantee. Selecting travel destinations based on attributes like sports opportunities, culture, etc., can be made into efficient query plans, but this also requires perfect query plans for short queries. I expect to learn more about this when visiting on site. There is clear input for LDBC in these workloads.</p> <p>There were three talks on semantic applications in cultural heritage. <a href="http://www.linkedin.com/pub/robina-clayphan/24/290/131" id="link-id0x247565f8">Robina Clayphan</a> of <a href="http://pro.europeana.eu/web/guest/foundation" id="link-id0x22ae6c18">Europeana</a> talked about this pan-European digital museum and library, and the <a href="http://pro.europeana.eu/edm-documentation" id="link-id0x1e12dba8">Europeana Data Model</a> (EDM). <a href="http://no.linkedin.com/pub/christian-emil-ore/8/b7/215" id="link-id0x2465f718">C.E. Ore</a> of the University of Oslo talked about the <a href="http://cidoc-crm.org/" id="link-id0x24714ac8">CIDOC CRM</a> (Conceptual Reference Model) ontology (<a href="http://www.iso.org/iso/catalogue_detail?csnumber=34424" id="link-id0x2472dfa8">ISO standard 21127:2006</a>) and its role in representing cultural, historic, and archaeological information. <a href="http://www.linkedin.com/pub/atanas-kiryakov/0/465/62a" id="link-id0xc2bbe28">Atanas Kiryakov</a> of <a href="http://www.linkedin.com/company/208070" id="link-id0x2471d1e8">Ontotext</a> gave a talk on a possible benchmark around <a href="http://dbpedia.org/resource/CIDOC_Conceptual_Reference_Model" id="link-id0x24741f88">CIDOC CRM</a> reasoning. In the present LDBC work, RDF inference plays a minor role, but reasoning would be emphasized with this CRM workload, in which the inference needed revolves around abbreviating unions between many traversal paths of different lengths between modeled objects. The data is not very large but the ontology has a lot of detail. This still is not the elusive use case which would really require all the <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x2aac4f7af718">OWL</a> complexities. We will first see how the semantic publishing benchmark work led by Ontotext in LDBC plays out. There is anyhow work enough there.</p> <p>The most concrete result was that the graph analytics part of the LDBC agenda starts to take shape. The LDBC organization is getting formed, and its processes and policies are getting defined. I visited <a href="http://www3.in.tum.de/~neumann/" id="link-id0x2aac4e70c318">Thomas Neumann</a>’s group in Munich just prior to the TUC meeting to work on this. Nowadays <a href="http://homepages.cwi.nl/~boncz/" id="link-id0x2aac4f47ae08">Peter Boncz</a>, who was recently awarded the <a href="http://dbpedia.org/resource/Humboldt_Prize" id="link-id0x2aac4eb7add8">Humboldt Prize</a>, goes to Munich on a weekly basis, so Munich is the favored destination for much LDBC-related work.</p> <p>The first workload of the <a href="http://ldbc.eu:8090/download/attachments/4325436/LDBC_SPB_Report_Nov2013.pdf?version=2&modificationDate=1384778837000" id="link-id0x2aac4eb7adb8">Social Network Benchmark</a> is taking shape, and there is good advance also in the <a href="http://ldbc.eu:8090/download/attachments/4325436/LDBC_SPB_Report_Nov2013.pdf?version=2&modificationDate=1384778837000" id="link-id0x2aac4f1a89e8">Semantic Publishing Benchmark</a>. I will in a future post give more commentary on these workloads, now that the initial drafts from the respective task forces are out.</p>
2013-12-11T16:51:35.000001-05:00
LDBC Technical User Community (TUC), 3rd Meeting
http://www.openlinksw.com:443/weblog/oerling/?date=2013-11-27#1759
2013-11-27T17:20:56Z
<p>Last week the RDF and graph DB benchmarking project, <a href="http://ldbc.eu/" id="link-id0x2aac4eb7ad38">LDBC</a>, had its <a href="http://ldbc.eu:8090/display/TUC/Third%20TUC%20Meeting%2C%20November%202013" id="link-id0x2aac4f0c0318">3rd Technical User Community meeting</a> in London, held in collaboration with the <a href="http://www.graphconnect.com/london/" id="link-id0x2aac4f3aa708">GraphConnect event</a>. This meeting marks the official launch of the LDBC non-profit company which is the successor of the present EU FP7 project.</p> <p>The meeting was very well attended, along with most of the new advisory board. <a href="http://www.linkedin.com/pub/xavier-lopez/b/871/788" id="link-id0x2aac4eff7128">Xavier Lopez</a> from <a href="http://www.linkedin.com/company/1028" id="link-id0x2aac4e939db8">Oracle</a>, <a href="http://homes.cs.washington.edu/~luisceze/" id="link-id0x2aac4e601588">Luis Ceze</a> from the <a href="http://dbpedia.org/resource/University_of_Washington" id="link-id0x2aac4ec862d8">University of Washington</a>, and <a href="http://www.ifi.uzh.ch/ddis/people/bernstein.html" id="link-id0x2aac4c0e5978">Abraham Bernstein</a> of the <a href="http://dbpedia.org/resource/University_of_Zurich" id="link-id0x2aac37e0b168">University of Zurich</a> were present. <a href="http://www.linkedin.com/pub/jans-aasman/0/107/272" id="link-id0x2aac4f353f68">Jans Aasman</a> of <a href="http://www.linkedin.com/company/167333" id="link-id0x2aac4d786768">Franz, Inc.</a>, and <a href="http://www.linkedin.com/pub/karl-huppler/8/ba9/8b4" id="link-id0x2aac4e9cca38">Karl Huppler</a>, former chairman of the <a href="http://dbpedia.org/resource/Transaction_Processing_Performance_Council" id="link-id0x2aac4e1df248">TPC</a>, were not present but are signed up as advisory board members.</p> <p>We had great talks by the new board members and invited graph and RDF DB users. </p> <p> <a href="http://www.gsd.inesc-id.pt/~nonius/" id="link-id0x2aac4f893878">Nuno Carvalho</a> of <a href="http://www.fujitsu.com/emea/about/fle/" id="link-id0x2aac4f893798">Fujitsu Labs</a> presented on the <a href="http://ldbc.eu:8090/download/attachments/4325436/LDBC-TUC-Fujitsu-Final.pdf?version=1&modificationDate=1385291853000" id="link-id0x2aac4f893778">Fujitsu RDF use cases and benchmarking requirements</a>, based around analytics streaming on time series of streaming data. The technology platform is diverse, with anything from RDF stores to <a href="http://dbpedia.org/resource/HBase" id="link-id0x2aac4f893738">HBase</a>. The challenge is integration. I pointed out that with <a href="http://virtuoso.openlinksw.com/" id="link-id0x2aac4f483818">Virtuoso</a> column store, you could now efficiently host time series data alongside RDF. Sure, a relational format is more efficient with time series data, but it can be co-located with RDF, and queries can join between the two. This is especially so after our stellar bulk-load speed measured with the TPC-H dataset.</p> <p>Luis Ceze of Washington University presented <a href="http://sampa.cs.washington.edu/grappa/" id="link-id0x2aac4f8936d8">Grappa</a>, a C++ graph programming framework that in his words would be like <a href="http://dbpedia.org/resource/Cray_XMT" id="link-id0x2aac4f284308">Cray XMT</a>, later <a href="http://www.linkedin.com/company/yarcdata" id="link-id0x2aac4f06a098">Yarc Data</a>, in software. The idea is to have a graph algorithm divided into small executable steps, millions in number, and to have very efficient scheduling and switching between these, building latency tolerance into every step of the application. Commodity interconnects like <a href="http://dbpedia.org/resource/InfiniBand" id="link-id0x2aac4f83ee28">InfiniBand</a> deliver bad throughput with small messages, but with endless message combination opportunities from millions of mini work units, the overall throughput stays good. We know the same from all the Virtuoso scale-out work. Luis is presently working on <a href="http://labs.oracle.com/pls/apex/f?p=labs:50:0:::::#ero-400" id="link-id0x2aac4f0e38a8">GraphBench</a>, a research project at Washington State funded by Oracle for graph algorithm benchmarking. The major interest for LDBC is in having a library of common graph analytics as a starting point. Having these, the data generation can further evolve so as to create challenges for the algorithms. One issue that came up is the question of validating graph algorithm results: Unlike in SQL queries, there is not necessarily a single correct answer. If the algorithm to use and the count of iterations to run is not fully specified, response times will vary widely. Random walks will anyway create variation between consecutive runs.</p> <p>Abraham Bernstein presented about the work on his <a href="http://uzh.github.io/signal-collect/" id="link-id0x2aac4ea858d8">Signal/Collect</a> graph programming framework and its applications in fraud detection. He also talked about the EU FP7 project <a href="http://vista-tv.eu/" id="link-id0x2aac4f2cb818">ViSTA-TV</a> which does massive stream processing around the real time behavior of internet TV users. Again, Abraham gave very direct suggestions for what to include in the LDBC graph analytics workload.</p> <p> <a href="http://www.linkedin.com/pub/andreas-both/22/672/942" id="link-id0x2aac4e733c18">Andreas Both</a> of <a href="http://www.linkedin.com/company/124775" id="link-id0x2aac4ea3c158">Unister</a> presented on RDF ontology-driven applications in an e-commerce context. Unister is Germany’s leading e-commerce portal operator with a large number of properties ranging across travel to most B2C. The RDF use cases are many, in principle down to final content distribution but high online demand often calls for specialized solutions like bit field intersections for combining conditions. Sufficiently advanced database technology may also offer this but this is not a guarantee. Selecting travel destinations based on attributes like sports opportunities, culture, etc., can be made into efficient query plans, but this also requires perfect query plans for short queries. I expect to learn more about this when visiting on site. There is clear input for LDBC in these workloads.</p> <p>There were three talks on semantic applications in cultural heritage. <a href="http://www.linkedin.com/pub/robina-clayphan/24/290/131" id="link-id0x2aac3463e858">Robina Clayphan</a> of <a href="http://pro.europeana.eu/web/guest/foundation" id="link-id0x2aac3486e998">Europeana</a> talked about this pan-European digital museum and library, and the <a href="http://pro.europeana.eu/edm-documentation" id="link-id0x2aabe7afafa8">Europeana Data Model</a> (EDM). <a href="http://no.linkedin.com/pub/christian-emil-ore/8/b7/215" id="link-id0x2aac35e62f58">C.E. Ore</a> of the University of Oslo talked about the <a href="http://cidoc-crm.org/" id="link-id0x2aabcfdf2a88">CIDOC CRM</a> (Conceptual Reference Model) ontology (<a href="http://www.iso.org/iso/catalogue_detail?csnumber=34424" id="link-id0x2aac4c5d63e8">ISO standard 21127:2006</a>) and its role in representing cultural, historic, and archaeological information. <a href="http://www.linkedin.com/pub/atanas-kiryakov/0/465/62a" id="link-id0x2aac3542aef8">Atanas Kiryakov</a> of <a href="http://www.linkedin.com/company/208070" id="link-id0x2aab69c82b28">Ontotext</a> gave a talk on a possible benchmark around <a href="http://dbpedia.org/resource/CIDOC_Conceptual_Reference_Model" id="link-id0x2aac37cd2518">CIDOC CRM</a> reasoning. In the present LDBC work, RDF inference plays a minor role, but reasoning would be emphasized with this CRM workload, in which the inference needed revolves around abbreviating unions between many traversal paths of different lengths between modeled objects. The data is not very large but the ontology has a lot of detail. This still is not the elusive use case which would really require all the <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x2aac4f7af718">OWL</a> complexities. We will first see how the semantic publishing benchmark work led by Ontotext in LDBC plays out. There is anyhow work enough there.</p> <p>The most concrete result was that the graph analytics part of the LDBC agenda starts to take shape. The LDBC organization is getting formed, and its processes and policies are getting defined. I visited <a href="http://www3.in.tum.de/~neumann/" id="link-id0x2aac4e70c318">Thomas Neumann</a>’s group in Munich just prior to the TUC meeting to work on this. Nowadays <a href="http://homepages.cwi.nl/~boncz/" id="link-id0x2aac4f47ae08">Peter Boncz</a>, who was recently awarded the <a href="http://dbpedia.org/resource/Humboldt_Prize" id="link-id0x2aac4eb7add8">Humboldt Prize</a>, goes to Munich on a weekly basis, so Munich is the favored destination for much LDBC-related work.</p> <p>The first workload of the <a href="http://ldbc.eu:8090/download/attachments/4325436/LDBC_SPB_Report_Nov2013.pdf?version=2&modificationDate=1384778837000" id="link-id0x2aac4eb7adb8">Social Network Benchmark</a> is taking shape, and there is good advance also in the <a href="http://ldbc.eu:8090/download/attachments/4325436/LDBC_SPB_Report_Nov2013.pdf?version=2&modificationDate=1384778837000" id="link-id0x2aac4f1a89e8">Semantic Publishing Benchmark</a>. I will in a future post give more commentary on these workloads, now that the initial drafts from the respective task forces are out.</p>
2013-12-11T16:51:17.000001-05:00
In Hoc Signo Vinces (part 8 of n) -- TPC-H: INs, Expressions, ORs
http://www.openlinksw.com:443/blog/vdb/blog/?date=2013-11-25#1758
2013-11-25T17:22:46Z
<p>In this installment, we look at <a href="http://www.tpc.org/tpch/" id="link-id0x2aabd9458d78">TPC-H</a> Q7, Q12, Q19, Q21, and Q22. These have complex expressions and require some tricks with <code>OR</code> and <code>IN</code>. Also proper order between <code>JOINs</code> and expressions is tested in Q21.</p> <h2> <code>IN</code> Predicates</h2> <p>One of the choke points mentioned in the <i><a href="http://www.tpc.org/tpctc/tpctc2013/slides_and_papers/005.pdf" id="link-id0x2aac0c0d7d98">TPC-H Analyzed</a></i> paper is the <code>IN</code> predicate with a list of constants. This occurs in Q12 and Q22. Q12 is the simpler of the two:</p> <blockquote> <code><pre> SELECT l_shipmode , SUM ( CASE WHEN o_orderpriority = '1-URGENT' OR o_orderpriority = '2-HIGH' THEN 1 ELSE 0 END ) AS high_line_count, SUM ( CASE WHEN o_orderpriority <> '1-URGENT' AND o_orderpriority <> '2-HIGH' THEN 1 ELSE 0 END ) AS low_line_count FROM orders, lineitem WHERE o_orderkey = l_orderkey AND l_shipmode IN ('MAIL', 'SHIP') AND l_commitdate < l_receiptdate AND l_shipdate < l_commitdate AND l_receiptdate >= CAST ('1994-01-01' AS DATE) AND l_receiptdate < DATEADD ('year', 1, CAST ('1994-01-01' AS DATE)) GROUP BY l_shipmode ORDER BY l_shipmode</pre> </code> </blockquote> <p>The execution profile:</p> <blockquote> <code><pre>{ time 2.2e-05% fanout 1 input 1 rows time 0.00023% fanout 1 input 1 rows Precode: 0: chash_in_init := Call chash_in_init ( 182 , $29 "chash_in_tree", 0 , 0 , <c MAIL>, <c SHIP>) 5: BReturn 0 { fork time 1.2e-05% fanout 1 input 1 rows { fork time 86% fanout 2.60039e+07 input 1 rows LINEITEM 2.4e+06 rows(.L_COMMITDATE, .L_RECEIPTDATE, .L_SHIPDATE, .L_ORDERKEY, .L_SHIPMODE) L_RECEIPTDATE >= <c 1994-01-01> < <c 1995-01-01> hash partition+bloom by 0 () time 4.4% fanout 0.119803 input 2.60039e+07 rows END Node After test: 0: if (.L_COMMITDATE < .L_RECEIPTDATE) then 4 else 9 unkn 9 4: if (.L_SHIPDATE < .L_COMMITDATE) then 8 else 9 unkn 9 8: BReturn 1 9: BReturn 0 time 7.9% fanout 1 input 3.11534e+06 rows ORDERS unq 1 rows (.O_ORDERPRIORITY) inlined O_ORDERKEY = k_.L_ORDERKEY After code: 0: if (.O_ORDERPRIORITY = <c 1-URGENT>) then 13 else 4 unkn 13 4: if (.O_ORDERPRIORITY = <c 2-HIGH>) then 13 else 8 unkn 13 8: callretSearchedCASE := := artm 1 12: Jump 17 (level=0) 13: callretSearchedCASE := := artm 0 17: if (.O_ORDERPRIORITY = <c 1-URGENT>) then 25 else 21 unkn 21 21: if (.O_ORDERPRIORITY = <c 2-HIGH>) then 25 else 30 unkn 30 25: callretSearchedCASE := := artm 1 29: Jump 34 (level=0) 30: callretSearchedCASE := := artm 0 34: BReturn 0 time 1.3% fanout 0 input 3.11534e+06 rows Sort (.L_SHIPMODE) -> (callretSearchedCASE, callretSearchedCASE) }</pre> </code> </blockquote> <p> <i>-- rest left out </i> </p> <blockquote> <code><pre> 1077 msec 2014% cpu, 3.11419e+06 rnd 5.99896e+08 seq 97.9872% same seg 1.74106% same pg </pre> </code> </blockquote> <p>The top item in the profile is the predicate on <code>l_receiptdate</code>. <i>TPC-H Analyzed</i> correctly points out that a <code>lineitem</code> table in date order is best here, because zone maps will work on <code>l_receiptdate</code> since this is correlated with <code>l_shipdate</code>, which is the best date ordering column, as it is the most used. The date compare is done first in the scan, as it selects 1/7 and is fast, whereas the <code>IN</code> selects 2/7 and has more instructions on the execution path.</p> <h2>Q22</h2> <blockquote> <code><pre> SELECT cntrycode, COUNT(*) AS numcust, SUM(c_acctbal) AS totacctbal FROM ( SELECT SUBSTRING(c_phone, 1, 2) AS cntrycode, c_acctbal FROM customer WHERE SUBSTRING(c_phone, 1, 2) IN ('13', '31', '23', '29', '30', '18', '17') AND c_acctbal > ( SELECT AVG(c_acctbal) FROM customer WHERE c_acctbal > 0.00 AND SUBSTRING(c_phone, 1, 2) IN ('13', '31', '23', '29', '30', '18', '17') ) AND NOT EXISTS ( SELECT * FROM orders WHERE o_custkey = c_custkey ) ) AS custsale GROUP BY cntrycode ORDER BY cntrycode</pre> </code> </blockquote> <p>Q22 has a condition on a substring of a string column. This merits a separate trick, namely merging the substring extraction into the scan, so the invisible hash-join predicate reads the column, cuts the substring in place, calculates a hash number, <a href="http://dbpedia.org/resource/Bloom_filter" id="link-id0x2aac1df3c948">Bloom filters</a> this against the <code>IN</code> set, then finally outputs the row numbers which match. This operation is run-time re-orderable with other conditions, like the test on <code>c_acctbal</code>. <a href="http://www.tpc.org/tpcds/" id="link-id0x2aac0e99d2b8">TPC-DS</a> has a similar pattern in some queries.</p> <p>The profile follows.</p> <p>Q22 is one of the rare queries that clearly benefit from having an index on a foreign key column. The <code>NOT EXISTS</code> with <code>orders</code> could be done by hash, but then the hash would have to have every <code>DISTINCT o_custkey</code> and the hash build could be filtered by a <code>JOIN</code> on <code>customer</code> with the conditions on <code>c_acctbal</code> and <code>c_phone</code> repeated. Being inside an existence, the number of <code>orders</code> would not have to be retained, so the hash table would not end up larger than the probe side.</p> <p>The payoff in Q22 makes it worthwhile to maintain an index on <code>o_custkey</code> in the refresh functions.</p> <blockquote> <code><pre>{ time 1.4e-05% fanout 1 input 1 rows time 1.2% fanout 1 input 1 rows Precode: 0: $27 "chash_in_init" := Call chash_in_init ( 182 , $29 "chash_in_tree", 0 , 0 , <c 13>, <c 31>, <c 23>, <c 29>, <c 30>, <c 18>, <c 17>) 5: { time 7.9e-06% fanout 1 input 1 rows time 6.7e-05% fanout 1 input 1 rows { fork</pre> </code> </blockquote> <p> <i>-- Read <code>customer</code> once, filter with the <code>IN</code> predicate, and add up <code>c_acctbal</code> and <code>count</code> for the average.</i> </p> <blockquote> <code><pre>time 26% fanout 0 input 1 rows CUSTOMER 1.4e+07 rows(t4.C_ACCTBAL) C_ACCTBAL > 0 hash partition+bloom by 0 () After code: 0: sum count 1 5: sum sumt4.C_ACCTBAL 10: BReturn 0 } After code: 0: temp := artm sum / count 4: aggregate := := artm temp 8: BReturn 0 time 2.6e-05% fanout 0 input 1 rows Subquery Select(aggregate) } 13: BReturn 0 { fork time 1.9e-05% fanout 1 input 1 rows { fork</pre> </code> </blockquote> <p> <i>-- second scan of customer with the test on c_acctbal > average and the same in predicate.</i> </p> <blockquote> <code><pre>time 26% fanout 1.90967e+06 input 1 rows CUSTOMER 2.7e+05 rows(t2.C_CUSTKEY, t2.C_PHONE, t2.C_ACCTBAL) C_ACCTBAL > k_scalar hash partition+bloom by 0 () time 7% fanout 0.333434 input 1.90967e+06 rows END Node After test: 0: if ({ time 0.26% fanout 1 input 1.90967e+06 rows</pre> </code> </blockquote> <p> <i>-- See if the customer has orders. The index lookup is very fast since the keys come in order from the scan of customer.</i> </p> <blockquote> <code><pre>time 5% fanout 9.9955 input 1.90967e+06 rows O_CK 9.9 rows() inlined O_CUSTKEY = k_t2.C_CUSTKEY time 1.6% fanout 0 input 1.90881e+07 rows Subquery Select( <none> ) } ) then 5 else 4 unkn 5 4: BReturn 1 5: BReturn 0 time 3.1% fanout 1 input 636749 rows Precode: 0: cntrycode := Call substring (t2.C_PHONE, 1 , 2 ) 5: BReturn 0 Stage 2 time 0.5% fanout 0 input 636749 rows Sort (q_cntrycode) -> (t2.C_ACCTBAL, inc) } time 0.011% fanout 7 input 1 rows group by read node (cntrycode, totacctbal, numcust)in each partition slice time 0.0013% fanout 0 input 7 rows Sort (cntrycode) -> (numcust, totacctbal) } time 0.00016% fanout 7 input 1 rows Key from temp (cntrycode, numcust, totacctbal) time 1.4e-05% fanout 0 input 7 rows Select (cntrycode, numcust, totacctbal) } 306 msec 2107% cpu, 1.90678e+06 rnd 4.77961e+07 seq 99.5612% same seg 0.419137% same pg </pre> </code> </blockquote> <h2>Q21 - Join Order and complex tests</h2> <p>This identifies <code>suppliers</code> from a given country that have kept <code>orders</code> waiting, i.e., they supply a delayed <code>lineitem</code> and nobody else in the <code>order</code> supplies a delayed <code>lineitem</code>.</p> <blockquote> <code><pre> SELECT TOP 100 s_name, COUNT(*) AS numwait FROM supplier, lineitem l1, orders, nation WHERE s_suppkey = l1.l_suppkey AND o_orderkey = l1.l_orderkey AND o_orderstatus = 'F' AND l1.l_receiptdate > l1.l_commitdate AND EXISTS ( SELECT * FROM lineitem l2 WHERE l2.l_orderkey = l1.l_orderkey AND l2.l_suppkey <> l1.l_suppkey ) AND NOT EXISTS ( SELECT * FROM lineitem l3 WHERE l3.l_orderkey = l1.l_orderkey AND l3.l_suppkey <> l1.l_suppkey AND l3.l_receiptdate > l3.l_commitdate ) AND s_nationkey = n_nationkey AND n_name = 'SAUDI ARABIA' GROUP BY s_name ORDER BY numwait desc, s_name ;</pre> </code> </blockquote> <p>The crucial thing the optimizer must realize is that conditions that depend on a table should not always be placed right after the table because there could be a selective join that costs less. In this case, the plan comes out as a scan of <code>orders</code> selecting 1/2; then an index lookup on <code>lineitem</code> which is very efficient because the keys come in order and are tightly local; then the selective hash join with <code>suppliers</code> from the country is merged into the index lookup. After this, 1/50 of <code>lineitems</code> are left. Only for these does the system actually fetch the dates. After this comes the series of tests, ordered similarly to how joins are ordered. First the cheap date comparison, then the subqueries. The <code>lineitems</code> can be fetched by their primary key, which again comes in order. Doing this by hash would duplicate most of the query on the build side and would be a lot of trouble.</p> <p>Another join order would be <code>lineitem</code> first, selecting 1/25 with the merged hash-join with <code>supplier</code>, then <code>orders</code> by index, selecting 1/2, then the existences. Experiment shows there is no great difference.</p> <blockquote> <code><pre>{ time 4.5e-06% fanout 1 input 1 rows time 0.0036% fanout 1 input 1 rows { hash filler Subquery 27 { time 3.8e-05% fanout 1 input 1 rows NATION 1 rows(t4.N_NATIONKEY) N_NAME = <c SAUDI ARABIA> time 0.01% fanout 39953 input 1 rows SUPPLIER 4.2e+04 rows(t1.S_SUPPKEY, t1.S_NAME) S_NATIONKEY = t4.N_NATIONKEY After code: 0: t1.S_SUPPKEY := := artm t1.S_SUPPKEY 4: t1.S_NAME := := artm t1.S_NAME 8: BReturn 0 time 0.0019% fanout 0 input 39953 rows Sort hf 48 (t1.S_SUPPKEY) -> (t1.S_NAME) } } time 2.8e-06% fanout 1 input 1 rows { fork time 2.6e-06% fanout 1 input 1 rows { fork time 6.2% fanout 7.30725e+07 input 1 rows ORDERS 7.3e+07 rows(.O_ORDERKEY) O_ORDERSTATUS = <c F> time 33% fanout 0.142002 input 7.30725e+07 rows LINEITEM 0.49 rows(l1.L_RECEIPTDATE, l1.L_COMMITDATE, l1.L_ORDERKEY, l1.L_SUPPKEY) inlined L_ORDERKEY = .O_ORDERKEY hash partition+bloom by 58 (tmp)hash join merged always card 0.04 -> (.S_NAME) time 9.7% fanout 0.0341517 input 1.03764e+07 rows END Node After test: 0: if (l1.L_RECEIPTDATE > l1.L_COMMITDATE) then 4 else 13 unkn 13 4: if ({ time 0.18% fanout 0.630264 input 1.03764e+07 rows time 6.4% fanout 4.99133 input 6.53989e+06 rows LINEITEM 1.1 rows(l3.L_SUPPKEY, l3.L_RECEIPTDATE, l3.L_COMMITDATE) inlined L_ORDERKEY = k_l1.L_ORDERKEY time 1.4% fanout 0.504963 input 3.26427e+07 rows END Node After test: 0: if (l3.L_RECEIPTDATE > l3.L_COMMITDATE) then 4 else 9 unkn 9 4: if (l3.L_SUPPKEY = l1.L_SUPPKEY) then 9 else 8 unkn 9 8: BReturn 1 9: BReturn 0 time 0.17% fanout 0 input 1.64834e+07 rows Subquery Select( <none> ) } ) then 13 else 8 unkn 13 8: if ({ time 0.052% fanout 0.0570441 input 1.03764e+07 rows time 1.1% fanout 2.13264 input 591914 rows LINEITEM 3.7 rows(l2.L_SUPPKEY) inlined L_ORDERKEY = k_l1.L_ORDERKEY time 0.047% fanout 0.531095 input 1.26234e+06 rows END Node After test: 0: if (l2.L_SUPPKEY = l1.L_SUPPKEY) then 5 else 4 unkn 5 4: BReturn 1 5: BReturn 0 time 0.013% fanout 0 input 670421 rows Subquery Select( <none> ) } ) then 12 else 13 unkn 13 12: BReturn 1 13: BReturn 0 time 0.0086% fanout 1 input 354373 rows Hash source 48 merged into ts 0.04 rows(k_l1.L_SUPPKEY) -> (.S_NAME) time 1.5% fanout 1 input 354373 rows Stage 2 time 0.18% fanout 0 input 354373 rows Sort (q_.S_NAME) -> (inc) }</pre> </code> </blockquote> <p> <i>-- rest left out </i> </p> <blockquote> <code><pre> 2301 msec 2092% cpu, 8.01116e+07 rnd 3.95017e+08 seq 99.4284% same seg 0.475267% same pg Compilation: 2 msec 0 reads 0% read 0 messages 0% clw</pre> </code> </blockquote> <h2>Q19 Complex Expressions</h2> <blockquote> <code><pre>SELECT SUM(l_extendedprice* (1 - l_discount)) AS revenue FROM lineitem, part WHERE ( p_partkey = l_partkey AND p_brand = 'Brand#12' AND p_container IN ('SM CASE', 'SM BOX', 'SM PACK', 'SM PKG') AND l_quantity >= 1 AND l_quantity <= 1 + 10 AND p_size BETWEEN 1 AND 5 AND l_shipmode IN ('AIR', 'AIR REG') AND l_shipinstruct = 'DELIVER IN PERSON' ) OR ( p_partkey = l_partkey AND p_brand = 'Brand#23' AND p_container IN ('MED BAG', 'MED BOX', 'MED PKG', 'MED PACK') AND l_quantity >= 10 AND l_quantity <= 10 + 10 AND p_size BETWEEN 1 AND 10 AND l_shipmode IN ('AIR', 'AIR REG') AND l_shipinstruct = 'DELIVER IN PERSON' ) OR ( p_partkey = l_partkey AND p_brand = 'Brand#34' AND p_container IN ('LG CASE', 'LG BOX', 'LG PACK', 'LG PKG') AND l_quantity >= 20 AND l_quantity <= 20 + 10 AND p_size BETWEEN 1 AND 15 AND l_shipmode in ('AIR', 'AIR REG') AND l_shipinstruct = 'DELIVER IN PERSON' )</pre> </code> </blockquote> <p>The essential trick is to recognize that each of the terms of the <code>OR</code> have the join condition between <code>lineitem</code> and <code>part</code> and the <code>l_shipmode</code> and <code>l_shipinstruct</code> conditions in common. After extracting these, the <code>OR</code> is split into two more <code>ORs</code>, one with conditions on <code>part</code> and the other with conditions on <code>lineitem</code> only. A hash is made of the matching parts where parts that correspond to none of the 3 <code>ORed</code> <code>ANDs</code> are left out. Then there is a scan of <code>lineitem</code> with the hash lookup merged. The merged hash lookup does in this case produce result columns, which are further tested later in the query.</p> <blockquote> <code><pre>{ time 1.7e-05% fanout 1 input 1 rows time 0.018% fanout 1 input 1 rows Precode: 0: chash_in_init := Call chash_in_init ( 182 , $29 "chash_in_tree", 0 , 0 , <c AIR>, <c AIR REG>) 5: temp := artm 1 + 10 9: temp := artm 10 + 10 13: temp := artm 20 + 10 17: BReturn 0 { hash filler time 4.5% fanout 2e+07 input 1 rows PART 7.2e+04 rows(.P_BRAND, .P_CONTAINER, .P_SIZE, .P_PARTKEY) P_SIZE >= 1 time 16% fanout 0.00240925 input 2e+07 rows END Node After test: 0: if (.P_BRAND = <c Brand#12>) then 4 else 17 unkn 17 4: if (.P_SIZE <= 5 ) then 8 else 17 unkn 17 8: one_of_these := Call one_of_these (.P_CONTAINER, <c SM CASE>, <c SM BOX>, <c SM PACK>, <c SM PKG>) 13: if ( 0 < one_of_these) then 51 else 17 unkn 17 17: if (.P_BRAND = <c Brand#23>) then 21 else 34 unkn 34 21: if (.P_SIZE <= 10 ) then 25 else 34 unkn 34 25: one_of_these := Call one_of_these (.P_CONTAINER, <c MED BAG>, <c MED BOX>, <c MED PKG>, <c MED PACK>) 30: if ( 0 < one_of_these) then 51 else 34 unkn 34 34: if (.P_BRAND = <c Brand#34>) then 38 else 52 unkn 52 38: if (.P_SIZE <= 15 ) then 42 else 52 unkn 52 42: one_of_these := Call one_of_these (.P_CONTAINER, <c LG CASE>, <c LG BOX>, <c LG PACK>, <c LG PKG>) 47: if ( 0 < one_of_these) then 51 else 52 unkn 52 51: BReturn 1 52: BReturn 0 time 0.058% fanout 0 input 48185 rows Sort hf 52 (.P_PARTKEY) -> (.P_SIZE, .P_CONTAINER, .P_BRAND) } time 2.4e-05% fanout 1 input 1 rows { fork time 79% fanout 46004 input 1 rows LINEITEM 1.1e+07 rows(.L_QUANTITY, .L_PARTKEY, .L_EXTENDEDPRICE, .L_DISCOUNT, .L_SHIPMODE) L_SHIPINSTRUCT = <c DELIVER IN PERSON> hash partition+bloom by 0 () hash partition+bloom by 59 (tmp)hash join merged always card 0.00032 -> (.P_SIZE, .P_CONTAINER, .P_BRAND) time 0.011% fanout 0.599796 input 46004 rows END Node After test: 0: if (.L_QUANTITY <= temp) then 4 else 8 unkn 8 4: if (.L_QUANTITY >= 1 ) then 24 else 8 unkn 8 8: if (.L_QUANTITY <= temp) then 12 else 16 unkn 16 12: if ( 10 <= .L_QUANTITY) then 24 else 16 unkn 16 16: if (temp >= .L_QUANTITY) then 20 else 25 unkn 25 20: if (.L_QUANTITY >= 20 ) then 24 else 25 unkn 25 24: BReturn 1 25: BReturn 0 time 0.002% fanout 1 input 27593 rows Precode: 0: temp := artm 1 - .L_DISCOUNT 4: temp := artm .L_EXTENDEDPRICE * temp 8: BReturn 0 Hash source 52 merged into ts 0.00032 rows(k_.L_PARTKEY) -> (.P_SIZE, .P_CONTAINER, .P_BRAND) time 0.053% fanout 0 input 27593 rows END Node After test: 0: if (.P_BRAND = <c Brand#12>) then 4 else 25 unkn 25 4: if (.L_QUANTITY >= 1 ) then 8 else 25 unkn 25 8: if (.L_QUANTITY <= temp) then 12 else 25 unkn 25 12: if (.P_SIZE <= 5 ) then 16 else 25 unkn 25 16: one_of_these := Call one_of_these (.P_CONTAINER, <c SM CASE>, <c SM BOX>, <c SM PACK>, <c SM PKG>) 21: if ( 0 < one_of_these) then 75 else 25 unkn 25 25: if (.P_BRAND = <c Brand#23>) then 29 else 50 unkn 50 29: if ( 10 <= .L_QUANTITY) then 33 else 50 unkn 50 33: if (.L_QUANTITY <= temp) then 37 else 50 unkn 50 37: if (.P_SIZE <= 10 ) then 41 else 50 unkn 50 41: one_of_these := Call one_of_these (.P_CONTAINER, <c MED BAG>, <c MED BOX>, <c MED PKG>, <c MED PACK>) 46: if ( 0 < one_of_these) then 75 else 50 unkn 50 50: if (.P_BRAND = <c Brand#34>) then 54 else 76 unkn 76 54: if (.L_QUANTITY >= 20 ) then 58 else 76 unkn 76 58: if (temp >= .L_QUANTITY) then 62 else 76 unkn 76 62: if (.P_SIZE <= 15 ) then 66 else 76 unkn 76 66: one_of_these := Call one_of_these (.P_CONTAINER, <c LG CASE>, <c LG BOX>, <c LG PACK>, <c LG PKG>) 71: if ( 0 < one_of_these) then 75 else 76 unkn 76 75: BReturn 1 76: BReturn 0 After code: 0: sum revenuetemp 5: BReturn 0 } time 8.7e-06% fanout 0 input 1 rows Select (revenue) } 1315 msec 1889% cpu, 2 rnd 1.62319e+08 seq 0% same seg 0% same pg </pre> </code> </blockquote> <h2>Q7 More ORs</h2> <p>We find a similar pattern in Q7, where an implementation is expected to extract conditions from an <code>OR</code> and to restrict hash build sides with these. For </p> <blockquote> <code><pre>SELECT supp_nation, cust_nation, l_year, SUM(volume) AS revenue FROM ( SELECT n1.n_name AS supp_nation, n2.n_name AS cust_nation, extract(year FROM l_shipdate) AS l_year, l_extendedprice * (1 - l_discount) AS volume FROM supplier, lineitem, orders, customer, nation n1, nation n2 WHERE s_suppkey = l_suppkey AND o_orderkey = l_orderkey AND c_custkey = o_custkey AND s_nationkey = n1.n_nationkey AND c_nationkey = n2.n_nationkey AND ( ( n1.n_name = 'FRANCE' AND n2.n_name = 'GERMANY' ) OR ( n1.n_name = 'GERMANY' AND n2.n_name = 'FRANCE' ) ) AND l_shipdate BETWEEN CAST ('1995-01-01' AS DATE) AND CAST ('1996-12-31' AS DATE) ) AS shipping GROUP BY supp_nation, cust_nation, l_year ORDER BY supp_nation, cust_nation, l_year</pre> </code> </blockquote> <p>The plan builds a hash with <code>customers</code> from either France or Germany, then of <code>suppliers</code> from either France or Germany. Then it scans <code>lineitem</code> for 2/7 <code>years</code> and selects 2/25 based on the <code>supplier</code>. The name of the <code>supplier</code> country is also returned from the merged hash lookup. Then the corresponding <code>order</code> is fetched by primary key, which is fast since the <code>lineitem</code> produces keys in order. A similar hash condition is on the <code>customer</code>. Finally, there is code to check that the countries are different between <code>supplier</code> and <code>customer</code>. We leave out the plan in the interest of space. A single execution is between 1.7 and 1.9s; 5 concurrent executions are 7.5s for the slowest. </p> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x233f6158"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac078514a8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac12d42c08"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac16346118"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aab6b129918"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac1ee7e2b8"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aac154f7aa8"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2771a578"> In Hoc Signo Vinces (part 9): TPC-H: TPC-H Q18, Ordered Aggregation, and Top K </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aabf129f5a8"> In Hoc Signo Vinces (part 10): TPC-H: TPC-H Q9, Q17, Q20 - Predicate Games</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac1ed2ae08"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac1c9f2d08"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac752dde68"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x19d27968"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x195d2b18"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x108f2558"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0xeb62dd8"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x101630b8"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aac1458cf48"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:10:45.438978-04:00
In Hoc Signo Vinces (part 7 of n) -- TPC-H Q13: The Good and the Bad Plans
http://www.openlinksw.com:443/blog/vdb/blog/?date=2013-11-25#1757
2013-11-25T17:22:39Z
<p>Q13 is one of the longest running of the 22 queries. The <a href="http://www.tpc.org/tpch/" id="link-id0x23e81828">TPC-H</a> metric is a geometric mean of two scores, power and throughput, where the throughput score is the elapsed time of the multiuser part of the test divided by the number of queries executed. In this part of the score, Q13 can be up to 1/5 of the total. The power score on the other hand is a geometric mean of the run times of all the queries, scaled into queries per hour. There all queries have equal importance. A bad Q13 will sink a whole result.</p> <p>Q13 counts the <code>orders</code> of each <code>customer</code> and then shows, for each distinct <code>count</code> of <code>orders</code>, how many <code>customers</code> have this number of <code>orders</code>. 1/3 of the <code>customers</code> have no <code>orders</code>; hence this is an outer join between <code>customers</code> and <code>orders</code>, as follows:</p> <blockquote> <code><pre> SELECT c_count, COUNT(*) AS custdist FROM ( SELECT c_custkey, COUNT(o_orderkey) AS c_count FROM ( SELECT * FROM customer LEFT OUTER JOIN orders ON c_custkey = o_custkey AND o_comment NOT LIKE '%special%requests%' ) c_customer GROUP BY c_custkey ) c_orders GROUP BY c_count ORDER BY custdist DESC, c_count DESC ; </pre> </code> </blockquote> <p>The only parameter of the query is the pattern in the <code>NOT LIKE</code> condition. The <code>NOT LIKE</code> is very unselective, so almost all <code>orders</code> will be considered.</p> <p>The Virtuoso run time for Q13 is 6.7s, which we can consider a good result. Running 5 of these at the same time has the fastest execution finishing in 23.7s and the slowest in 35.3s. Doing 5x the work takes 5.2x the time. This is not bad, considering that the query has a high transient memory consumption. A second execution of 5 concurrent Q13s has the fastest finishing in 22.s and the slowest in 29.8s. The difference comes from already having the needed memory blocks cached, so there are no calls to the OS for mapping more memory.</p> <p>To measure the peak memory consumption, which is a factor with this query, there is the <code>mp_max_large_in_use</code> counter. To reset: </p> <blockquote> <code>__dbf_set ('mp_max_large_in_use', 0);</code> </blockquote> <p>To read:</p> <blockquote> <code>SELECT sys_stat ('mp_max_large_in_use');</code> </blockquote> <p>For the 5 concurrent executions of Q13, the counter goes to 10GB. This is easily accommodated at 100 GB; but at ten times the scale, this will be a significant quantity, even in a scale out setting. The memory allocation time is recorded in the counter <code>mp_mmap_clocks</code>, read with <code>sys_stat</code>. This is a count of cycles spent waiting for <code>mmap</code> or <code>munmap</code> and allows tracking if the process is being slowed down by transient memory allocation.</p> <p>Let us consider how this works. The plan is as follows:</p> <blockquote> <code><pre> { { hash filler CUSTOMER 1.5e+07 rows(t3.C_CUSTKEY) </pre> </code> </blockquote> <p> <i>-- Make a hash table of the 150M customers. The stage 2 operator below means that the <code>customers</code> are partitioned in a number of distinct partitions based on the <code>c_custkey</code>, which is the key in the hash table. This means that a number of disjoint hash tables are built, as many as there are concurrent threads. This corresponds to the <code>ThreadsPerQuery</code> ini file setting of the <code>enable_qp</code> setting with <code>__dbf_set</code> and <code>sys_stat</code>.</i> </p> <blockquote> <code><pre> Stage 2 Sort hf 34 (q_t3.C_CUSTKEY) } { fork { fork { fork END Node outer { </pre> </code> </blockquote> <p> <i>-- Here we start a <code>RIGHT OUTER JOIN</code> block. The below operator scans the <code>orders</code> table and picks out the <code>orders</code> which do not contain the mentioned <code>LIKE</code> pattern.</i> </p> <blockquote> <code> <pre> ORDERS 1.5e+08 rows(t4.O_CUSTKEY, t4.O_ORDERKEY) O_COMMENT LIKE <c special="special" requests="requests"> LIKE <c> hash partition+bloom by 80 () </c></c> </pre> </code> </blockquote> <p> <i>-- Below is a partitioning operator, also known as an exchange operator, which will divide the stream of <code>o_custkeys</code> from the previous scan into different partitions, each served by a different thread.</i> </p> <blockquote> <code><pre> Stage 2 </pre> </code> </blockquote> <p> <i>-- Below is a lookup in the <code>customer</code> hash table. The lookup takes place in the partition determined by the <code>o_custkey</code> being looked up.</i> </p> <blockquote> <code><pre> Hash source 34 not partitionable 1 rows(q_t4.O_CUSTKEY) -> () right oj, key out ssls: (t3.C_CUSTKEY) After code: 0: t3.C_CUSTKEY := := artm t4.O_CUSTKEY 4: BReturn 0 </pre> </code> </blockquote> <p> <i>-- The below is a <code>RIGHT OUTER JOIN</code> end operator; see below for further description </i> </p> <blockquote> <code><pre> end of outer} set_ctr out: (t4.O_ORDERKEY, t4.O_CUSTKEY) shadow: (t4.O_ORDERKEY, t4.O_CUSTKEY) Precode: 0: isnotnull := Call isnotnull (t4.O_ORDERKEY) 5: BReturn 0 </pre> </code> </blockquote> <p> <i>-- The below sort is the innermost <code>GROUP BY</code>. The <code>ISNOTNULL</code> above makes a <code>0</code> or a <code>1</code>, depending on whether there was a found <code>o_custkey</code> for the <code>c_custkey</code> of the <code>customer</code>.</i> </p> <blockquote> <code><pre> Sort (t3.C_CUSTKEY) -> (isnotnull) } </pre> </code> </blockquote> <p> <i>-- The below operators start after the above have executed to completion on every partition. We read the first aggregation, containing for each <code>customer</code> the <code>COUNT</code> of <code>orders</code>. </i> </p> <blockquote> <code><pre> group by read node (t3.C_CUSTKEY, aggregate)in each partition slice After code: 0: c_custkey := := artm t3.C_CUSTKEY 4: c_count := := artm aggregate 8: BReturn 0 Subquery Select(c_custkey, c_count) </pre> </code> </blockquote> <p> <i>-- Below is the second <code>GROUP BY</code>; for each <code>COUNT</code>, we count how many <code>customers</code> have this many <code>orders</code>.</i> </p> <blockquote> <code><pre> Sort (c_count) -> (inc) } group by read node (c_count, custdist) </pre> </code> </blockquote> <p> <i>-- Below is the final <code>ORDER BY</code>.</i> </p> <blockquote> <code><pre> Sort (custdist, c_count) } Key from temp (c_count, custdist) Select (c_count, custdist) } </pre> </code> </blockquote> <p>The CPU profile starts as follows:</p> <blockquote> <code><pre> 971537 31.8329 setp_chash_run 494300 16.1960 hash_source_chash_input_1i_n 262218 8.5917 clrg_partition_dc 162773 5.3333 strstr_sse42 68049 2.2297 memcpy_16 65883 2.1587 cha_insert_1i_n 57515 1.8845 hs_send_output 56093 1.8379 cmp_like_const 53752 1.7612 gb_aggregate 51274 1.6800 cha_rehash_ents ... </pre> </code> </blockquote> <p>The <code>GROUP BY</code> is on top, with 31%. This is the first <code>GROUP BY</code>, which has one group per customer, for a total of 150M groups. Below the <code>GROUP BY</code> is the hash lookup of the hash join from <code>orders</code> to <code>customer</code>. The third item is partitioning of a data column (<code>dc</code>, or vectored query variable). The partitioning refers to the operator labeled <b>stage 2</b> above. From one column of values, it makes several. In the 4th place, we have the <code>NOT LIKE</code> predicate on <code>o_comment</code>. This is a substring search implemented using SSE 4.2 instructions. Finally, in the last place, there is a function for resizing a hash table; in the present case, the hash table for the innermost <code>GROUP BY</code>.</p> <p>At this point, we have to explain the <code>RIGHT OUTER JOIN</code>: Generally when making a hash join, the larger table is on the probe side and the smaller on the build side. This means that the rows on the build side get put in a hash table and then for each row on the probe side there is a lookup to see if there is a match in the hash table.</p> <p>However, here the bigger table is on the right side of <code>LEFT OUTER JOIN</code>. Normally, one would have to make the hash table from the <code>orders</code> table and then probe it with <code>customer</code>, so that one would find no match for the <code>customers</code> with no <code>orders</code> and several matches for <code>customers</code> with many <code>orders</code>. However, this would be much slower. So there is a trick for reversing the process: You still build the hash from the smaller set in the <code>JOIN</code>, but now for each key that does get probed, you set a bit in a bit mask, in addition to sending the match as output. After all outputs have been generated, you look in the hash table for the entries where the bit is not set. These correspond to the <code>customers</code> with no <code>orders</code>. For these, you send the <code>c_custkey</code> with a null <code>o_orderkey</code> to the next operator in the pipeline, which is the <code>GROUP BY</code> on <code>c_custkey</code> with the count of non-null <code>o_orderkeys</code>.</p> <p>One might at first think that such a backwards way of doing an outer join is good for nothing but this benchmark and should be considered a benchmark special. This is not so, though, as there are accepted implementations that do this very thing.</p> <p>Furthermore, getting a competitive score in any other way is impossible, as we shall see below.</p> <p>We further note that the the grouping key in the innermost <code>GROUP BY</code> is the same as the hash key in the last hash join, i.e., <code>o_custkey</code>. This means that the <code>GROUP BY</code> and the hash join could be combined in a single operator called <code>GROUPJOIN</code>. If this were done, the hash would be built from <code>customer</code> with extra space left for the counters. This would in fact remove the hash join from the profile as well as the rehash of the group by hash table, for a gain of about 20%. The outer join behavior is not a problem here since untouched buckets, e.g., <code>customers</code> without <code>orders</code>, would be inited with a <code>COUNT</code> of <code>0</code>. For an inner join behavior, one would simply leave out the zero counts when reading the <code>GROUP BY</code>. At the end of the series, we will see what the DBT3 score will be. We remember that there is a 1.5s savings to be had here for the throughput score if the score is not high enough otherwise. The effect on the power score will be less because that only cares about relative speedup, not absolute time.</p> <p>Next, we disable the <code>RIGHT OUTER JOIN</code> optimization and force the <code>JOIN</code> to build a hash on <code>orders</code> and to probe it with <code>customer</code>. The execution time is 25s. Most of the time goes into building the hash table of <code>orders</code>. The memory consumption also goes up to around 8G. Then we try the <code>JOIN</code> by index with a scan of <code>customer</code>, and for each an index lookup of <code>orders</code> based on an index on <code>o_custkey</code>. Here we note that there is a condition on a dependent part of the primary key, namely <code>o_comment</code>, which requires joining to the main row from the <code>o_ck</code> index. There is a gain however because the <code>GROUP BY</code> becomes ordered; i.e., there is no need to keep groups around for <code>customers</code> that have already been seen since we know they will not come again, the outer scan being in order of <code>c_custkey</code>. For this reason, the memory consumption for the <code>GROUP BY</code> goes away. However, the index-based plan is extremely sensitive to vector size: The execution takes 29.4s if vector size is allowed to grow to 1MB, but 413s if it stays at the default of 10KB. The difference is in the 1MB vector hitting 1/150 (1 million lookups for a 150 million row table), whereas the 10KB vector hits 1/15000. Thus, benefits from vectoring lookups are largely lost, since there are hardly ever hits in the same segment; in this case, within 2000 rows. But this is not the main problem: The condition on the main row is a <code>LIKE</code> on a long column. Thus, the whole column for the segment in question must be accessed for read, meaning 2000 or so <code>o_comments</code>, of which one will be checked. If instead of a condition on <code>o_comment</code>, we have one on <code>o_totalprice > 0</code>, we get 93s with 10KB vector size and 15s with dynamic up to 1MB.</p> <p>If we now remove the condition on dependent columns of <code>orders</code>, the index plan becomes faster, since the whole condition is resolved within the <code>o_custkey</code> index -- 2.5s with 10KB vector size, 2.6s with dynamic vector size up to 1MB. The point here is that the access from <code>customer</code> to <code>orders</code> on the <code>o_custkey</code> index is ordered, like a merge join.</p> <h3>Q13 Conclusions</h3> <p>Q13 is a combo of many choke points in the <i>TPC-H Analyzed</i> paper. The most important is special <code>JOIN</code> types, i.e., <code>RIGHT OUTER JOIN</code> and <code>GROUPJOIN</code>. Then there is string operation performance for the substring matching with <code>LIKE</code>. This needs to be implemented with the SSE 4.2 string instructions; otherwise there is a hit of about 0.5s on query speed.</p> <p>The <i>TPC-H Analyzed</i> paper was written against the background of analytical DB tradition where the dominant <code>JOIN</code> type is hash, except when there is a merge between two sets that are ordered or at least clustered on the same key. Clustered here means physical order but without the need to be strictly in key order.</p> <p>Here I have added some index based variants to show that hash join indeed wins and to point out the sensitivity of random access to vector size. As column stores go, Virtuoso is especially good at random access. This must be so since it was optimized to do RDF well, which entails a lot of lookup. Also note how a big string column goes with great ease in a sequential scan, but kills in a non-local random access pattern.</p> <h3> <i>In Hoc Signo Vinces</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x233f6158"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac1c7b8ca8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac1c79fb58"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac1e3a3c58"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac1cd4e8a8"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac0eb77df8"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aac1e4e0988"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x4bc6b48"> In Hoc Signo Vinces (part 9): TPC-H: TPC-H Q18, Ordered Aggregation, and Top K </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aab6d3ab518"> In Hoc Signo Vinces (part 10): TPC-H: TPC-H Q9, Q17, Q20 - Predicate Games</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x1eb9a678"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aabd308be88"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x177580f8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac3124cce8"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x2aabde079648"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aabc90bb398"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aabdcf12238"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aabd54ff898"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aabd0d4cfc8"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:10:51.444495-04:00
In Hoc Signo Vinces (part 8 of n) -- TPC-H: INs, Expressions, ORs
http://www.openlinksw.com:443/weblog/oerling/?date=2013-11-25#1756
2013-11-25T16:58:38Z
<p>In this installment, we look at <a href="http://www.tpc.org/tpch/" id="link-id0x2aabd9458d78">TPC-H</a> Q7, Q12, Q19, Q21, and Q22. These have complex expressions and require some tricks with <code>OR</code> and <code>IN</code>. Also proper order between <code>JOINs</code> and expressions is tested in Q21.</p> <h2> <code>IN</code> Predicates</h2> <p>One of the choke points mentioned in the <i><a href="http://www.tpc.org/tpctc/tpctc2013/slides_and_papers/005.pdf" id="link-id0x2aac0c0d7d98">TPC-H Analyzed</a></i> paper is the <code>IN</code> predicate with a list of constants. This occurs in Q12 and Q22. Q12 is the simpler of the two:</p> <blockquote> <code><pre> SELECT l_shipmode , SUM ( CASE WHEN o_orderpriority = '1-URGENT' OR o_orderpriority = '2-HIGH' THEN 1 ELSE 0 END ) AS high_line_count, SUM ( CASE WHEN o_orderpriority <> '1-URGENT' AND o_orderpriority <> '2-HIGH' THEN 1 ELSE 0 END ) AS low_line_count FROM orders, lineitem WHERE o_orderkey = l_orderkey AND l_shipmode IN ('MAIL', 'SHIP') AND l_commitdate < l_receiptdate AND l_shipdate < l_commitdate AND l_receiptdate >= CAST ('1994-01-01' AS DATE) AND l_receiptdate < DATEADD ('year', 1, CAST ('1994-01-01' AS DATE)) GROUP BY l_shipmode ORDER BY l_shipmode</pre> </code> </blockquote> <p>The execution profile:</p> <blockquote> <code><pre>{ time 2.2e-05% fanout 1 input 1 rows time 0.00023% fanout 1 input 1 rows Precode: 0: chash_in_init := Call chash_in_init ( 182 , $29 "chash_in_tree", 0 , 0 , <c MAIL>, <c SHIP>) 5: BReturn 0 { fork time 1.2e-05% fanout 1 input 1 rows { fork time 86% fanout 2.60039e+07 input 1 rows LINEITEM 2.4e+06 rows(.L_COMMITDATE, .L_RECEIPTDATE, .L_SHIPDATE, .L_ORDERKEY, .L_SHIPMODE) L_RECEIPTDATE >= <c 1994-01-01> < <c 1995-01-01> hash partition+bloom by 0 () time 4.4% fanout 0.119803 input 2.60039e+07 rows END Node After test: 0: if (.L_COMMITDATE < .L_RECEIPTDATE) then 4 else 9 unkn 9 4: if (.L_SHIPDATE < .L_COMMITDATE) then 8 else 9 unkn 9 8: BReturn 1 9: BReturn 0 time 7.9% fanout 1 input 3.11534e+06 rows ORDERS unq 1 rows (.O_ORDERPRIORITY) inlined O_ORDERKEY = k_.L_ORDERKEY After code: 0: if (.O_ORDERPRIORITY = <c 1-URGENT>) then 13 else 4 unkn 13 4: if (.O_ORDERPRIORITY = <c 2-HIGH>) then 13 else 8 unkn 13 8: callretSearchedCASE := := artm 1 12: Jump 17 (level=0) 13: callretSearchedCASE := := artm 0 17: if (.O_ORDERPRIORITY = <c 1-URGENT>) then 25 else 21 unkn 21 21: if (.O_ORDERPRIORITY = <c 2-HIGH>) then 25 else 30 unkn 30 25: callretSearchedCASE := := artm 1 29: Jump 34 (level=0) 30: callretSearchedCASE := := artm 0 34: BReturn 0 time 1.3% fanout 0 input 3.11534e+06 rows Sort (.L_SHIPMODE) -> (callretSearchedCASE, callretSearchedCASE) }</pre> </code> </blockquote> <p> <i>-- rest left out </i> </p> <blockquote> <code><pre> 1077 msec 2014% cpu, 3.11419e+06 rnd 5.99896e+08 seq 97.9872% same seg 1.74106% same pg </pre> </code> </blockquote> <p>The top item in the profile is the predicate on <code>l_receiptdate</code>. <i>TPC-H Analyzed</i> correctly points out that a <code>lineitem</code> table in date order is best here, because zone maps will work on <code>l_receiptdate</code> since this is correlated with <code>l_shipdate</code>, which is the best date ordering column, as it is the most used. The date compare is done first in the scan, as it selects 1/7 and is fast, whereas the <code>IN</code> selects 2/7 and has more instructions on the execution path.</p> <h2>Q22</h2> <blockquote> <code><pre> SELECT cntrycode, COUNT(*) AS numcust, SUM(c_acctbal) AS totacctbal FROM ( SELECT SUBSTRING(c_phone, 1, 2) AS cntrycode, c_acctbal FROM customer WHERE SUBSTRING(c_phone, 1, 2) IN ('13', '31', '23', '29', '30', '18', '17') AND c_acctbal > ( SELECT AVG(c_acctbal) FROM customer WHERE c_acctbal > 0.00 AND SUBSTRING(c_phone, 1, 2) IN ('13', '31', '23', '29', '30', '18', '17') ) AND NOT EXISTS ( SELECT * FROM orders WHERE o_custkey = c_custkey ) ) AS custsale GROUP BY cntrycode ORDER BY cntrycode</pre> </code> </blockquote> <p>Q22 has a condition on a substring of a string column. This merits a separate trick, namely merging the substring extraction into the scan, so the invisible hash-join predicate reads the column, cuts the substring in place, calculates a hash number, <a href="http://dbpedia.org/resource/Bloom_filter" id="link-id0x2aac1df3c948">Bloom filters</a> this against the <code>IN</code> set, then finally outputs the row numbers which match. This operation is run-time re-orderable with other conditions, like the test on <code>c_acctbal</code>. <a href="http://www.tpc.org/tpcds/" id="link-id0x2aac0e99d2b8">TPC-DS</a> has a similar pattern in some queries.</p> <p>The profile follows.</p> <p>Q22 is one of the rare queries that clearly benefit from having an index on a foreign key column. The <code>NOT EXISTS</code> with <code>orders</code> could be done by hash, but then the hash would have to have every <code>DISTINCT o_custkey</code> and the hash build could be filtered by a <code>JOIN</code> on <code>customer</code> with the conditions on <code>c_acctbal</code> and <code>c_phone</code> repeated. Being inside an existence, the number of <code>orders</code> would not have to be retained, so the hash table would not end up larger than the probe side.</p> <p>The payoff in Q22 makes it worthwhile to maintain an index on <code>o_custkey</code> in the refresh functions.</p> <blockquote> <code><pre>{ time 1.4e-05% fanout 1 input 1 rows time 1.2% fanout 1 input 1 rows Precode: 0: $27 "chash_in_init" := Call chash_in_init ( 182 , $29 "chash_in_tree", 0 , 0 , <c 13>, <c 31>, <c 23>, <c 29>, <c 30>, <c 18>, <c 17>) 5: { time 7.9e-06% fanout 1 input 1 rows time 6.7e-05% fanout 1 input 1 rows { fork</pre> </code> </blockquote> <p> <i>-- Read <code>customer</code> once, filter with the <code>IN</code> predicate, and add up <code>c_acctbal</code> and <code>count</code> for the average.</i> </p> <blockquote> <code><pre>time 26% fanout 0 input 1 rows CUSTOMER 1.4e+07 rows(t4.C_ACCTBAL) C_ACCTBAL > 0 hash partition+bloom by 0 () After code: 0: sum count 1 5: sum sumt4.C_ACCTBAL 10: BReturn 0 } After code: 0: temp := artm sum / count 4: aggregate := := artm temp 8: BReturn 0 time 2.6e-05% fanout 0 input 1 rows Subquery Select(aggregate) } 13: BReturn 0 { fork time 1.9e-05% fanout 1 input 1 rows { fork</pre> </code> </blockquote> <p> <i>-- second scan of customer with the test on c_acctbal > average and the same in predicate.</i> </p> <blockquote> <code><pre>time 26% fanout 1.90967e+06 input 1 rows CUSTOMER 2.7e+05 rows(t2.C_CUSTKEY, t2.C_PHONE, t2.C_ACCTBAL) C_ACCTBAL > k_scalar hash partition+bloom by 0 () time 7% fanout 0.333434 input 1.90967e+06 rows END Node After test: 0: if ({ time 0.26% fanout 1 input 1.90967e+06 rows</pre> </code> </blockquote> <p> <i>-- See if the customer has orders. The index lookup is very fast since the keys come in order from the scan of customer.</i> </p> <blockquote> <code><pre>time 5% fanout 9.9955 input 1.90967e+06 rows O_CK 9.9 rows() inlined O_CUSTKEY = k_t2.C_CUSTKEY time 1.6% fanout 0 input 1.90881e+07 rows Subquery Select( <none> ) } ) then 5 else 4 unkn 5 4: BReturn 1 5: BReturn 0 time 3.1% fanout 1 input 636749 rows Precode: 0: cntrycode := Call substring (t2.C_PHONE, 1 , 2 ) 5: BReturn 0 Stage 2 time 0.5% fanout 0 input 636749 rows Sort (q_cntrycode) -> (t2.C_ACCTBAL, inc) } time 0.011% fanout 7 input 1 rows group by read node (cntrycode, totacctbal, numcust)in each partition slice time 0.0013% fanout 0 input 7 rows Sort (cntrycode) -> (numcust, totacctbal) } time 0.00016% fanout 7 input 1 rows Key from temp (cntrycode, numcust, totacctbal) time 1.4e-05% fanout 0 input 7 rows Select (cntrycode, numcust, totacctbal) } 306 msec 2107% cpu, 1.90678e+06 rnd 4.77961e+07 seq 99.5612% same seg 0.419137% same pg </pre> </code> </blockquote> <h2>Q21 - Join Order and complex tests</h2> <p>This identifies <code>suppliers</code> from a given country that have kept <code>orders</code> waiting, i.e., they supply a delayed <code>lineitem</code> and nobody else in the <code>order</code> supplies a delayed <code>lineitem</code>.</p> <blockquote> <code><pre> SELECT TOP 100 s_name, COUNT(*) AS numwait FROM supplier, lineitem l1, orders, nation WHERE s_suppkey = l1.l_suppkey AND o_orderkey = l1.l_orderkey AND o_orderstatus = 'F' AND l1.l_receiptdate > l1.l_commitdate AND EXISTS ( SELECT * FROM lineitem l2 WHERE l2.l_orderkey = l1.l_orderkey AND l2.l_suppkey <> l1.l_suppkey ) AND NOT EXISTS ( SELECT * FROM lineitem l3 WHERE l3.l_orderkey = l1.l_orderkey AND l3.l_suppkey <> l1.l_suppkey AND l3.l_receiptdate > l3.l_commitdate ) AND s_nationkey = n_nationkey AND n_name = 'SAUDI ARABIA' GROUP BY s_name ORDER BY numwait desc, s_name ;</pre> </code> </blockquote> <p>The crucial thing the optimizer must realize is that conditions that depend on a table should not always be placed right after the table because there could be a selective join that costs less. In this case, the plan comes out as a scan of <code>orders</code> selecting 1/2; then an index lookup on <code>lineitem</code> which is very efficient because the keys come in order and are tightly local; then the selective hash join with <code>suppliers</code> from the country is merged into the index lookup. After this, 1/50 of <code>lineitems</code> are left. Only for these does the system actually fetch the dates. After this comes the series of tests, ordered similarly to how joins are ordered. First the cheap date comparison, then the subqueries. The <code>lineitems</code> can be fetched by their primary key, which again comes in order. Doing this by hash would duplicate most of the query on the build side and would be a lot of trouble.</p> <p>Another join order would be <code>lineitem</code> first, selecting 1/25 with the merged hash-join with <code>supplier</code>, then <code>orders</code> by index, selecting 1/2, then the existences. Experiment shows there is no great difference.</p> <blockquote> <code><pre>{ time 4.5e-06% fanout 1 input 1 rows time 0.0036% fanout 1 input 1 rows { hash filler Subquery 27 { time 3.8e-05% fanout 1 input 1 rows NATION 1 rows(t4.N_NATIONKEY) N_NAME = <c SAUDI ARABIA> time 0.01% fanout 39953 input 1 rows SUPPLIER 4.2e+04 rows(t1.S_SUPPKEY, t1.S_NAME) S_NATIONKEY = t4.N_NATIONKEY After code: 0: t1.S_SUPPKEY := := artm t1.S_SUPPKEY 4: t1.S_NAME := := artm t1.S_NAME 8: BReturn 0 time 0.0019% fanout 0 input 39953 rows Sort hf 48 (t1.S_SUPPKEY) -> (t1.S_NAME) } } time 2.8e-06% fanout 1 input 1 rows { fork time 2.6e-06% fanout 1 input 1 rows { fork time 6.2% fanout 7.30725e+07 input 1 rows ORDERS 7.3e+07 rows(.O_ORDERKEY) O_ORDERSTATUS = <c F> time 33% fanout 0.142002 input 7.30725e+07 rows LINEITEM 0.49 rows(l1.L_RECEIPTDATE, l1.L_COMMITDATE, l1.L_ORDERKEY, l1.L_SUPPKEY) inlined L_ORDERKEY = .O_ORDERKEY hash partition+bloom by 58 (tmp)hash join merged always card 0.04 -> (.S_NAME) time 9.7% fanout 0.0341517 input 1.03764e+07 rows END Node After test: 0: if (l1.L_RECEIPTDATE > l1.L_COMMITDATE) then 4 else 13 unkn 13 4: if ({ time 0.18% fanout 0.630264 input 1.03764e+07 rows time 6.4% fanout 4.99133 input 6.53989e+06 rows LINEITEM 1.1 rows(l3.L_SUPPKEY, l3.L_RECEIPTDATE, l3.L_COMMITDATE) inlined L_ORDERKEY = k_l1.L_ORDERKEY time 1.4% fanout 0.504963 input 3.26427e+07 rows END Node After test: 0: if (l3.L_RECEIPTDATE > l3.L_COMMITDATE) then 4 else 9 unkn 9 4: if (l3.L_SUPPKEY = l1.L_SUPPKEY) then 9 else 8 unkn 9 8: BReturn 1 9: BReturn 0 time 0.17% fanout 0 input 1.64834e+07 rows Subquery Select( <none> ) } ) then 13 else 8 unkn 13 8: if ({ time 0.052% fanout 0.0570441 input 1.03764e+07 rows time 1.1% fanout 2.13264 input 591914 rows LINEITEM 3.7 rows(l2.L_SUPPKEY) inlined L_ORDERKEY = k_l1.L_ORDERKEY time 0.047% fanout 0.531095 input 1.26234e+06 rows END Node After test: 0: if (l2.L_SUPPKEY = l1.L_SUPPKEY) then 5 else 4 unkn 5 4: BReturn 1 5: BReturn 0 time 0.013% fanout 0 input 670421 rows Subquery Select( <none> ) } ) then 12 else 13 unkn 13 12: BReturn 1 13: BReturn 0 time 0.0086% fanout 1 input 354373 rows Hash source 48 merged into ts 0.04 rows(k_l1.L_SUPPKEY) -> (.S_NAME) time 1.5% fanout 1 input 354373 rows Stage 2 time 0.18% fanout 0 input 354373 rows Sort (q_.S_NAME) -> (inc) }</pre> </code> </blockquote> <p> <i>-- rest left out </i> </p> <blockquote> <code><pre> 2301 msec 2092% cpu, 8.01116e+07 rnd 3.95017e+08 seq 99.4284% same seg 0.475267% same pg Compilation: 2 msec 0 reads 0% read 0 messages 0% clw</pre> </code> </blockquote> <h2>Q19 Complex Expressions</h2> <blockquote> <code><pre>SELECT SUM(l_extendedprice* (1 - l_discount)) AS revenue FROM lineitem, part WHERE ( p_partkey = l_partkey AND p_brand = 'Brand#12' AND p_container IN ('SM CASE', 'SM BOX', 'SM PACK', 'SM PKG') AND l_quantity >= 1 AND l_quantity <= 1 + 10 AND p_size BETWEEN 1 AND 5 AND l_shipmode IN ('AIR', 'AIR REG') AND l_shipinstruct = 'DELIVER IN PERSON' ) OR ( p_partkey = l_partkey AND p_brand = 'Brand#23' AND p_container IN ('MED BAG', 'MED BOX', 'MED PKG', 'MED PACK') AND l_quantity >= 10 AND l_quantity <= 10 + 10 AND p_size BETWEEN 1 AND 10 AND l_shipmode IN ('AIR', 'AIR REG') AND l_shipinstruct = 'DELIVER IN PERSON' ) OR ( p_partkey = l_partkey AND p_brand = 'Brand#34' AND p_container IN ('LG CASE', 'LG BOX', 'LG PACK', 'LG PKG') AND l_quantity >= 20 AND l_quantity <= 20 + 10 AND p_size BETWEEN 1 AND 15 AND l_shipmode in ('AIR', 'AIR REG') AND l_shipinstruct = 'DELIVER IN PERSON' )</pre> </code> </blockquote> <p>The essential trick is to recognize that each of the terms of the <code>OR</code> have the join condition between <code>lineitem</code> and <code>part</code> and the <code>l_shipmode</code> and <code>l_shipinstruct</code> conditions in common. After extracting these, the <code>OR</code> is split into two more <code>ORs</code>, one with conditions on <code>part</code> and the other with conditions on <code>lineitem</code> only. A hash is made of the matching parts where parts that correspond to none of the 3 <code>ORed</code> <code>ANDs</code> are left out. Then there is a scan of <code>lineitem</code> with the hash lookup merged. The merged hash lookup does in this case produce result columns, which are further tested later in the query.</p> <blockquote> <code><pre>{ time 1.7e-05% fanout 1 input 1 rows time 0.018% fanout 1 input 1 rows Precode: 0: chash_in_init := Call chash_in_init ( 182 , $29 "chash_in_tree", 0 , 0 , <c AIR>, <c AIR REG>) 5: temp := artm 1 + 10 9: temp := artm 10 + 10 13: temp := artm 20 + 10 17: BReturn 0 { hash filler time 4.5% fanout 2e+07 input 1 rows PART 7.2e+04 rows(.P_BRAND, .P_CONTAINER, .P_SIZE, .P_PARTKEY) P_SIZE >= 1 time 16% fanout 0.00240925 input 2e+07 rows END Node After test: 0: if (.P_BRAND = <c Brand#12>) then 4 else 17 unkn 17 4: if (.P_SIZE <= 5 ) then 8 else 17 unkn 17 8: one_of_these := Call one_of_these (.P_CONTAINER, <c SM CASE>, <c SM BOX>, <c SM PACK>, <c SM PKG>) 13: if ( 0 < one_of_these) then 51 else 17 unkn 17 17: if (.P_BRAND = <c Brand#23>) then 21 else 34 unkn 34 21: if (.P_SIZE <= 10 ) then 25 else 34 unkn 34 25: one_of_these := Call one_of_these (.P_CONTAINER, <c MED BAG>, <c MED BOX>, <c MED PKG>, <c MED PACK>) 30: if ( 0 < one_of_these) then 51 else 34 unkn 34 34: if (.P_BRAND = <c Brand#34>) then 38 else 52 unkn 52 38: if (.P_SIZE <= 15 ) then 42 else 52 unkn 52 42: one_of_these := Call one_of_these (.P_CONTAINER, <c LG CASE>, <c LG BOX>, <c LG PACK>, <c LG PKG>) 47: if ( 0 < one_of_these) then 51 else 52 unkn 52 51: BReturn 1 52: BReturn 0 time 0.058% fanout 0 input 48185 rows Sort hf 52 (.P_PARTKEY) -> (.P_SIZE, .P_CONTAINER, .P_BRAND) } time 2.4e-05% fanout 1 input 1 rows { fork time 79% fanout 46004 input 1 rows LINEITEM 1.1e+07 rows(.L_QUANTITY, .L_PARTKEY, .L_EXTENDEDPRICE, .L_DISCOUNT, .L_SHIPMODE) L_SHIPINSTRUCT = <c DELIVER IN PERSON> hash partition+bloom by 0 () hash partition+bloom by 59 (tmp)hash join merged always card 0.00032 -> (.P_SIZE, .P_CONTAINER, .P_BRAND) time 0.011% fanout 0.599796 input 46004 rows END Node After test: 0: if (.L_QUANTITY <= temp) then 4 else 8 unkn 8 4: if (.L_QUANTITY >= 1 ) then 24 else 8 unkn 8 8: if (.L_QUANTITY <= temp) then 12 else 16 unkn 16 12: if ( 10 <= .L_QUANTITY) then 24 else 16 unkn 16 16: if (temp >= .L_QUANTITY) then 20 else 25 unkn 25 20: if (.L_QUANTITY >= 20 ) then 24 else 25 unkn 25 24: BReturn 1 25: BReturn 0 time 0.002% fanout 1 input 27593 rows Precode: 0: temp := artm 1 - .L_DISCOUNT 4: temp := artm .L_EXTENDEDPRICE * temp 8: BReturn 0 Hash source 52 merged into ts 0.00032 rows(k_.L_PARTKEY) -> (.P_SIZE, .P_CONTAINER, .P_BRAND) time 0.053% fanout 0 input 27593 rows END Node After test: 0: if (.P_BRAND = <c Brand#12>) then 4 else 25 unkn 25 4: if (.L_QUANTITY >= 1 ) then 8 else 25 unkn 25 8: if (.L_QUANTITY <= temp) then 12 else 25 unkn 25 12: if (.P_SIZE <= 5 ) then 16 else 25 unkn 25 16: one_of_these := Call one_of_these (.P_CONTAINER, <c SM CASE>, <c SM BOX>, <c SM PACK>, <c SM PKG>) 21: if ( 0 < one_of_these) then 75 else 25 unkn 25 25: if (.P_BRAND = <c Brand#23>) then 29 else 50 unkn 50 29: if ( 10 <= .L_QUANTITY) then 33 else 50 unkn 50 33: if (.L_QUANTITY <= temp) then 37 else 50 unkn 50 37: if (.P_SIZE <= 10 ) then 41 else 50 unkn 50 41: one_of_these := Call one_of_these (.P_CONTAINER, <c MED BAG>, <c MED BOX>, <c MED PKG>, <c MED PACK>) 46: if ( 0 < one_of_these) then 75 else 50 unkn 50 50: if (.P_BRAND = <c Brand#34>) then 54 else 76 unkn 76 54: if (.L_QUANTITY >= 20 ) then 58 else 76 unkn 76 58: if (temp >= .L_QUANTITY) then 62 else 76 unkn 76 62: if (.P_SIZE <= 15 ) then 66 else 76 unkn 76 66: one_of_these := Call one_of_these (.P_CONTAINER, <c LG CASE>, <c LG BOX>, <c LG PACK>, <c LG PKG>) 71: if ( 0 < one_of_these) then 75 else 76 unkn 76 75: BReturn 1 76: BReturn 0 After code: 0: sum revenuetemp 5: BReturn 0 } time 8.7e-06% fanout 0 input 1 rows Select (revenue) } 1315 msec 1889% cpu, 2 rnd 1.62319e+08 seq 0% same seg 0% same pg </pre> </code> </blockquote> <h2>Q7 More ORs</h2> <p>We find a similar pattern in Q7, where an implementation is expected to extract conditions from an <code>OR</code> and to restrict hash build sides with these. For </p> <blockquote> <code><pre>SELECT supp_nation, cust_nation, l_year, SUM(volume) AS revenue FROM ( SELECT n1.n_name AS supp_nation, n2.n_name AS cust_nation, extract(year FROM l_shipdate) AS l_year, l_extendedprice * (1 - l_discount) AS volume FROM supplier, lineitem, orders, customer, nation n1, nation n2 WHERE s_suppkey = l_suppkey AND o_orderkey = l_orderkey AND c_custkey = o_custkey AND s_nationkey = n1.n_nationkey AND c_nationkey = n2.n_nationkey AND ( ( n1.n_name = 'FRANCE' AND n2.n_name = 'GERMANY' ) OR ( n1.n_name = 'GERMANY' AND n2.n_name = 'FRANCE' ) ) AND l_shipdate BETWEEN CAST ('1995-01-01' AS DATE) AND CAST ('1996-12-31' AS DATE) ) AS shipping GROUP BY supp_nation, cust_nation, l_year ORDER BY supp_nation, cust_nation, l_year</pre> </code> </blockquote> <p>The plan builds a hash with <code>customers</code> from either France or Germany, then of <code>suppliers</code> from either France or Germany. Then it scans <code>lineitem</code> for 2/7 <code>years</code> and selects 2/25 based on the <code>supplier</code>. The name of the <code>supplier</code> country is also returned from the merged hash lookup. Then the corresponding <code>order</code> is fetched by primary key, which is fast since the <code>lineitem</code> produces keys in order. A similar hash condition is on the <code>customer</code>. Finally, there is code to check that the countries are different between <code>supplier</code> and <code>customer</code>. We leave out the plan in the interest of space. A single execution is between 1.7 and 1.9s; 5 concurrent executions are 7.5s for the slowest. </p> <p>To be continued...</p> <h3> <i>In Hoc Signo Vinces</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x233f6158"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac078514a8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac12d42c08"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac16346118"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aab6b129918"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac1ee7e2b8"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aac154f7aa8"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aac5901ca68"> In Hoc Signo Vinces (part 9): TPC-H: TPC-H Q18, Ordered Aggregation, and Top K </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac16ce0798"> In Hoc Signo Vinces (part 10): TPC-H: TPC-H Q9, Q17, Q20 - Predicate Games</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2101b868"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac29bfd458"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac6f23cdb8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0xb150b38"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x2aab75cd0128"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aac00458af8"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aabe9162e58"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aac374d1128"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aabd089f4c8"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:07:31.805346-04:00
In Hoc Signo Vinces (part 7 of n) -- TPC-H Q13: The Good and the Bad Plans
http://www.openlinksw.com:443/weblog/oerling/?date=2013-11-25#1755
2013-11-25T16:58:10Z
<p>Q13 is one of the longest running of the 22 queries. The <a href="http://www.tpc.org/tpch/" id="link-id0x23e81828">TPC-H</a> metric is a geometric mean of two scores, power and throughput, where the throughput score is the elapsed time of the multiuser part of the test divided by the number of queries executed. In this part of the score, Q13 can be up to 1/5 of the total. The power score on the other hand is a geometric mean of the run times of all the queries, scaled into queries per hour. There all queries have equal importance. A bad Q13 will sink a whole result.</p> <p>Q13 counts the <code>orders</code> of each <code>customer</code> and then shows, for each distinct <code>count</code> of <code>orders</code>, how many <code>customers</code> have this number of <code>orders</code>. 1/3 of the <code>customers</code> have no <code>orders</code>; hence this is an outer join between <code>customers</code> and <code>orders</code>, as follows:</p> <blockquote> <code><pre> SELECT c_count, COUNT(*) AS custdist FROM ( SELECT c_custkey, COUNT(o_orderkey) AS c_count FROM ( SELECT * FROM customer LEFT OUTER JOIN orders ON c_custkey = o_custkey AND o_comment NOT LIKE '%special%requests%' ) c_customer GROUP BY c_custkey ) c_orders GROUP BY c_count ORDER BY custdist DESC, c_count DESC ; </pre> </code> </blockquote> <p>The only parameter of the query is the pattern in the <code>NOT LIKE</code> condition. The <code>NOT LIKE</code> is very unselective, so almost all <code>orders</code> will be considered.</p> <p>The Virtuoso run time for Q13 is 6.7s, which we can consider a good result. Running 5 of these at the same time has the fastest execution finishing in 23.7s and the slowest in 35.3s. Doing 5x the work takes 5.2x the time. This is not bad, considering that the query has a high transient memory consumption. A second execution of 5 concurrent Q13s has the fastest finishing in 22.s and the slowest in 29.8s. The difference comes from already having the needed memory blocks cached, so there are no calls to the OS for mapping more memory.</p> <p>To measure the peak memory consumption, which is a factor with this query, there is the <code>mp_max_large_in_use</code> counter. To reset: </p> <blockquote> <code>__dbf_set ('mp_max_large_in_use', 0);</code> </blockquote> <p>To read:</p> <blockquote> <code>SELECT sys_stat ('mp_max_large_in_use');</code> </blockquote> <p>For the 5 concurrent executions of Q13, the counter goes to 10GB. This is easily accommodated at 100 GB; but at ten times the scale, this will be a significant quantity, even in a scale out setting. The memory allocation time is recorded in the counter <code>mp_mmap_clocks</code>, read with <code>sys_stat</code>. This is a count of cycles spent waiting for <code>mmap</code> or <code>munmap</code> and allows tracking if the process is being slowed down by transient memory allocation.</p> <p>Let us consider how this works. The plan is as follows:</p> <blockquote> <code><pre> { { hash filler CUSTOMER 1.5e+07 rows(t3.C_CUSTKEY) </pre> </code> </blockquote> <p> <i>-- Make a hash table of the 150M customers. The stage 2 operator below means that the <code>customers</code> are partitioned in a number of distinct partitions based on the <code>c_custkey</code>, which is the key in the hash table. This means that a number of disjoint hash tables are built, as many as there are concurrent threads. This corresponds to the <code>ThreadsPerQuery</code> ini file setting of the <code>enable_qp</code> setting with <code>__dbf_set</code> and <code>sys_stat</code>.</i> </p> <blockquote> <code><pre> Stage 2 Sort hf 34 (q_t3.C_CUSTKEY) } { fork { fork { fork END Node outer { </pre> </code> </blockquote> <p> <i>-- Here we start a <code>RIGHT OUTER JOIN</code> block. The below operator scans the <code>orders</code> table and picks out the <code>orders</code> which do not contain the mentioned <code>LIKE</code> pattern.</i> </p> <blockquote> <code> <pre> ORDERS 1.5e+08 rows(t4.O_CUSTKEY, t4.O_ORDERKEY) O_COMMENT LIKE <c special="special" requests="requests"> LIKE <c> hash partition+bloom by 80 () </c></c> </pre> </code> </blockquote> <p> <i>-- Below is a partitioning operator, also known as an exchange operator, which will divide the stream of <code>o_custkeys</code> from the previous scan into different partitions, each served by a different thread.</i> </p> <blockquote> <code><pre> Stage 2 </pre> </code> </blockquote> <p> <i>-- Below is a lookup in the <code>customer</code> hash table. The lookup takes place in the partition determined by the <code>o_custkey</code> being looked up.</i> </p> <blockquote> <code><pre> Hash source 34 not partitionable 1 rows(q_t4.O_CUSTKEY) -> () right oj, key out ssls: (t3.C_CUSTKEY) After code: 0: t3.C_CUSTKEY := := artm t4.O_CUSTKEY 4: BReturn 0 </pre> </code> </blockquote> <p> <i>-- The below is a <code>RIGHT OUTER JOIN</code> end operator; see below for further description </i> </p> <blockquote> <code><pre> end of outer} set_ctr out: (t4.O_ORDERKEY, t4.O_CUSTKEY) shadow: (t4.O_ORDERKEY, t4.O_CUSTKEY) Precode: 0: isnotnull := Call isnotnull (t4.O_ORDERKEY) 5: BReturn 0 </pre> </code> </blockquote> <p> <i>-- The below sort is the innermost <code>GROUP BY</code>. The <code>ISNOTNULL</code> above makes a <code>0</code> or a <code>1</code>, depending on whether there was a found <code>o_custkey</code> for the <code>c_custkey</code> of the <code>customer</code>.</i> </p> <blockquote> <code><pre> Sort (t3.C_CUSTKEY) -> (isnotnull) } </pre> </code> </blockquote> <p> <i>-- The below operators start after the above have executed to completion on every partition. We read the first aggregation, containing for each <code>customer</code> the <code>COUNT</code> of <code>orders</code>. </i> </p> <blockquote> <code><pre> group by read node (t3.C_CUSTKEY, aggregate)in each partition slice After code: 0: c_custkey := := artm t3.C_CUSTKEY 4: c_count := := artm aggregate 8: BReturn 0 Subquery Select(c_custkey, c_count) </pre> </code> </blockquote> <p> <i>-- Below is the second <code>GROUP BY</code>; for each <code>COUNT</code>, we count how many <code>customers</code> have this many <code>orders</code>.</i> </p> <blockquote> <code><pre> Sort (c_count) -> (inc) } group by read node (c_count, custdist) </pre> </code> </blockquote> <p> <i>-- Below is the final <code>ORDER BY</code>.</i> </p> <blockquote> <code><pre> Sort (custdist, c_count) } Key from temp (c_count, custdist) Select (c_count, custdist) } </pre> </code> </blockquote> <p>The CPU profile starts as follows:</p> <blockquote> <code><pre> 971537 31.8329 setp_chash_run 494300 16.1960 hash_source_chash_input_1i_n 262218 8.5917 clrg_partition_dc 162773 5.3333 strstr_sse42 68049 2.2297 memcpy_16 65883 2.1587 cha_insert_1i_n 57515 1.8845 hs_send_output 56093 1.8379 cmp_like_const 53752 1.7612 gb_aggregate 51274 1.6800 cha_rehash_ents ... </pre> </code> </blockquote> <p>The <code>GROUP BY</code> is on top, with 31%. This is the first <code>GROUP BY</code>, which has one group per customer, for a total of 150M groups. Below the <code>GROUP BY</code> is the hash lookup of the hash join from <code>orders</code> to <code>customer</code>. The third item is partitioning of a data column (<code>dc</code>, or vectored query variable). The partitioning refers to the operator labeled <b>stage 2</b> above. From one column of values, it makes several. In the 4th place, we have the <code>NOT LIKE</code> predicate on <code>o_comment</code>. This is a substring search implemented using SSE 4.2 instructions. Finally, in the last place, there is a function for resizing a hash table; in the present case, the hash table for the innermost <code>GROUP BY</code>.</p> <p>At this point, we have to explain the <code>RIGHT OUTER JOIN</code>: Generally when making a hash join, the larger table is on the probe side and the smaller on the build side. This means that the rows on the build side get put in a hash table and then for each row on the probe side there is a lookup to see if there is a match in the hash table.</p> <p>However, here the bigger table is on the right side of <code>LEFT OUTER JOIN</code>. Normally, one would have to make the hash table from the <code>orders</code> table and then probe it with <code>customer</code>, so that one would find no match for the <code>customers</code> with no <code>orders</code> and several matches for <code>customers</code> with many <code>orders</code>. However, this would be much slower. So there is a trick for reversing the process: You still build the hash from the smaller set in the <code>JOIN</code>, but now for each key that does get probed, you set a bit in a bit mask, in addition to sending the match as output. After all outputs have been generated, you look in the hash table for the entries where the bit is not set. These correspond to the <code>customers</code> with no <code>orders</code>. For these, you send the <code>c_custkey</code> with a null <code>o_orderkey</code> to the next operator in the pipeline, which is the <code>GROUP BY</code> on <code>c_custkey</code> with the count of non-null <code>o_orderkeys</code>.</p> <p>One might at first think that such a backwards way of doing an outer join is good for nothing but this benchmark and should be considered a benchmark special. This is not so, though, as there are accepted implementations that do this very thing.</p> <p>Furthermore, getting a competitive score in any other way is impossible, as we shall see below.</p> <p>We further note that the the grouping key in the innermost <code>GROUP BY</code> is the same as the hash key in the last hash join, i.e., <code>o_custkey</code>. This means that the <code>GROUP BY</code> and the hash join could be combined in a single operator called <code>GROUPJOIN</code>. If this were done, the hash would be built from <code>customer</code> with extra space left for the counters. This would in fact remove the hash join from the profile as well as the rehash of the group by hash table, for a gain of about 20%. The outer join behavior is not a problem here since untouched buckets, e.g., <code>customers</code> without <code>orders</code>, would be inited with a <code>COUNT</code> of <code>0</code>. For an inner join behavior, one would simply leave out the zero counts when reading the <code>GROUP BY</code>. At the end of the series, we will see what the DBT3 score will be. We remember that there is a 1.5s savings to be had here for the throughput score if the score is not high enough otherwise. The effect on the power score will be less because that only cares about relative speedup, not absolute time.</p> <p>Next, we disable the <code>RIGHT OUTER JOIN</code> optimization and force the <code>JOIN</code> to build a hash on <code>orders</code> and to probe it with <code>customer</code>. The execution time is 25s. Most of the time goes into building the hash table of <code>orders</code>. The memory consumption also goes up to around 8G. Then we try the <code>JOIN</code> by index with a scan of <code>customer</code>, and for each an index lookup of <code>orders</code> based on an index on <code>o_custkey</code>. Here we note that there is a condition on a dependent part of the primary key, namely <code>o_comment</code>, which requires joining to the main row from the <code>o_ck</code> index. There is a gain however because the <code>GROUP BY</code> becomes ordered; i.e., there is no need to keep groups around for <code>customers</code> that have already been seen since we know they will not come again, the outer scan being in order of <code>c_custkey</code>. For this reason, the memory consumption for the <code>GROUP BY</code> goes away. However, the index-based plan is extremely sensitive to vector size: The execution takes 29.4s if vector size is allowed to grow to 1MB, but 413s if it stays at the default of 10KB. The difference is in the 1MB vector hitting 1/150 (1 million lookups for a 150 million row table), whereas the 10KB vector hits 1/15000. Thus, benefits from vectoring lookups are largely lost, since there are hardly ever hits in the same segment; in this case, within 2000 rows. But this is not the main problem: The condition on the main row is a <code>LIKE</code> on a long column. Thus, the whole column for the segment in question must be accessed for read, meaning 2000 or so <code>o_comments</code>, of which one will be checked. If instead of a condition on <code>o_comment</code>, we have one on <code>o_totalprice > 0</code>, we get 93s with 10KB vector size and 15s with dynamic up to 1MB.</p> <p>If we now remove the condition on dependent columns of <code>orders</code>, the index plan becomes faster, since the whole condition is resolved within the <code>o_custkey</code> index -- 2.5s with 10KB vector size, 2.6s with dynamic vector size up to 1MB. The point here is that the access from <code>customer</code> to <code>orders</code> on the <code>o_custkey</code> index is ordered, like a merge join.</p> <h3>Q13 Conclusions</h3> <p>Q13 is a combo of many choke points in the <i>TPC-H Analyzed</i> paper. The most important is special <code>JOIN</code> types, i.e., <code>RIGHT OUTER JOIN</code> and <code>GROUPJOIN</code>. Then there is string operation performance for the substring matching with <code>LIKE</code>. This needs to be implemented with the SSE 4.2 string instructions; otherwise there is a hit of about 0.5s on query speed.</p> <p>The <i>TPC-H Analyzed</i> paper was written against the background of analytical DB tradition where the dominant <code>JOIN</code> type is hash, except when there is a merge between two sets that are ordered or at least clustered on the same key. Clustered here means physical order but without the need to be strictly in key order.</p> <p>Here I have added some index based variants to show that hash join indeed wins and to point out the sensitivity of random access to vector size. As column stores go, Virtuoso is especially good at random access. This must be so since it was optimized to do RDF well, which entails a lot of lookup. Also note how a big string column goes with great ease in a sequential scan, but kills in a non-local random access pattern.</p> <h3> <i>In Hoc Signo Vinces</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x233f6158"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac1c7b8ca8"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac1c79fb58"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac1e3a3c58"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac1cd4e8a8"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2aac0eb77df8"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aac1e4e0988"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x1fa4f7b8"> In Hoc Signo Vinces (part 9): TPC-H: TPC-H Q18, Ordered Aggregation, and Top K </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aab6d0057c8"> In Hoc Signo Vinces (part 10): TPC-H: TPC-H Q9, Q17, Q20 - Predicate Games</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac1eefbb78"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac29fa7b28"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac7aa7ae58"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac345c6278"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x17b3e448"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aabebab0578"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aabe95c4c88"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0xb88bcb8"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aabd149c4e8"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:07:36.781653-04:00
In Hoc Signo Vinces (part 6 of n) -- TPC-H Q1 and Q3: An Introduction to Query Plans
http://www.openlinksw.com:443/blog/vdb/blog/?date=2013-11-18#1754
2013-11-18T21:57:13Z
<p>Here we look at two <a href="http://www.tpc.org/tpch/" id="link-id0x29fbb0e8">TPC-H</a> queries whose execution plans are relatively straightforward and look at <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x1b09ee88">Virtuoso</a> performance metrics. On one hand; this is an introduction to query plans; on the other hand, a case study in tuning Virtuoso and understanding what goes on. The choke points outlined in <i><a href="http://www.tpc.org/tpctc/tpctc2013/slides_and_papers/005.pdf" id="link-id0x9052148">TPC-H Analyzed</a></i> are mentioned where applicable, with some extra commentary.</p> <h2>Q1 -- Scan, I/O, Aggregation </h2> <p>The query is below. The date is a parameter (a value near the end of the <code>l_shipdate</code> range is used), so most of the columns get read.</p> <blockquote> <code><pre> SELECT l_returnflag, l_linestatus, SUM(l_quantity) AS sum_qty, SUM(l_extendedprice) AS sum_base_price, SUM(l_extendedprice * (1 - l_discount)) AS sum_disc_price, SUM(l_extendedprice * (1 - l_discount) * (1 + l_tax)) AS sum_charge, AVG(l_quantity) AS avg_qty, AVG(l_extendedprice) AS avg_price, AVG(l_discount) AS avg_disc, COUNT(*) AS count_order FROM lineitem WHERE l_shipdate <= dateadd('DAY', -90, CAST ('1998-12-01' AS DATE)) GROUP BY l_returnflag, l_linestatus ORDER BY l_returnflag, l_linestatus ;</pre> </code> </blockquote> <p>We note that a count of a non-nullable column is the same as <code>COUNT (*)</code> and that <code>AVG</code> of a non-null column is <code>SUM (column) / COUNT (*)</code>. So <code>COUNT (*)</code> occurs 4 times, and <code>SUM (l_extendedprice)</code> and <code>SUM (l_quantity)</code> each occur twice. The grouping columns have few distinct values.</p> <p> <i>TPC-H Analyzed</i> suggests to use an array-based <code>GROUP BY</code>, because there can only be 64K combinations of 2 single-character values. The grouping keys are declared <code>CHAR (1)</code> and non-nullable. The Virtuoso implementation does not do this, though.</p> <p>This query has been treated in many papers because it cannot be implemented in very many ways, it is easy to understand, and it still illustrates some basic metrics.</p> <p>One execution on warm cache is between 3.9s and 4.7s. One execution with the data coming from OS disk cache is 11.8s. One execution with the data coming from 2 SSDs is 22s. Five concurrent executions from warm cache are 17.3s for the fastest and 20.5s for the slowest. A single threaded execution from warm cache is 58.4s.</p> <p>We see that scaling is linear; i.e., 5 times the work takes a little under 5x longer. The parallelism is reasonable, with 14.6 speedup from 24 threads on 12 cores. Splitting the work into 48 software threads, time-sliced on 24 hardware threads, does not affect execution time. The work thus appears to be evenly spread on the threads.</p> <p>It may be interesting to see how much data is transferred. To see the space consumption per column --</p> <blockquote> <code><pre> SELECT TOP 20 * FROM sys_index_space_stats ORDER BY iss_pages DESC ; </pre> </code> </blockquote> <p> -- followed by --</p> <blockquote> <code><pre> SELECT coi_column, SUM (coi_pages) / 128 FROM sys_col_info GROUP BY coi_column ORDER BY 2 DESC ; </pre> </code> </blockquote> <p>-- gives us the following --</p> <blockquote> <code><pre> L_COMMENT 17893 PS_COMMENT 10371 O_COMMENT 7824 L_EXTENDEDPRICE 4771 O_CLERK 2744 L_PARTKEY 2432 L_SUPPKEY 2432 L_COMMITDATE 1784 L_SHIPDATE 1551 L_RECEIPTDATE 1537 O_TOTALPRICE 1181 C_COMMENT 1150 L_QUANTITY 960 O_ORDERKEY 736 P_NAME 729 PS_SUPPLYCOST 647 O_CUSTKEY 624 C_ADDRESS 427 L_DISCOUNT 424 L_TAX 419 L_SHIPINSTRUCT 412 L_SHIPMODE 410 L_LINENUMBER 394 L_RETURNFLAG 394 L_LINESTATUS 394 O_ORDERDATE 389 P_COMMENT 341 PS_SUPPKEY 323 P_TYPE 293 C_PHONE 274 L_ORDERKEY 268 PS_AVAILQTY 201 P_RETAILPRICE 161 C_ACCTBAL 123 O_ORDERPRIORITY 95 O_ORDERSTATUS 94 S_COMMENT 66 ... </pre> </code> </blockquote> <p>The total in allocated pages is 65.6 GB, of which 34.1 GB are accessed by the workload. The comment strings could be stream-compressed, bringing some speedup in load time due to less I/O. Also <code>l_extendedprice</code>, a frequently accessed column, could be represented with 4 bytes instead of 8. The working set could thus be cut down to about 28 GB, which may offer some benefit at larger scales. At any rate, for system sizing, the space utilization report is very useful.</p> <p>The query execution profile is as below, with comments inline. The profile here is obvious, but we show this as a guide to reading future profiles which will be more interesting.</p> <blockquote> <code><pre> { time 2.6e-06% fanout 1 input 1 rows time 1.7e-06% fanout 1 input 1 rows { fork time 2.1e-06% fanout 1 input 1 rows { fork </pre> </code> </blockquote> <p>The <code>time xx%</code> line above each operator is the actual percentage of execution time taken by it, followed by the count of rows of output per row of input, followed by the actual rows of input. The below produced 591M rows of output for one row of input --</p> <blockquote> <code><pre> time 34% fanout 5.91599e+08 input 1 rows LINEITEM 5.9e+08 rows(.L_RETURNFLAG, .L_LINESTATUS, .L_DISCOUNT, .L_EXTENDEDPRICE, .L_QUANTITY, .L_TAX) L_SHIPDATE <= <c 1998-09-02> </pre> </code> </blockquote> <p>Below is the arithmetic of the query, followed by a sort (<code>GROUP BY</code>) operator.</p> <blockquote> <code><pre> After code: 0: temp := artm 1 - .L_DISCOUNT 4: temp := artm .L_EXTENDEDPRICE * temp 8: temp := artm 1 + .L_TAX 12: temp := artm temp * temp 16: BReturn 0 </pre> </code> </blockquote> <p>Most of the time is spent below, in the <code>GROUP BY</code>. We notice that each needed aggregation is done once, so the common subexpressions are correctly detected.</p> <blockquote> <code><pre> time 66% fanout 0 input 5.91599e+08 rows Sort (.L_RETURNFLAG, .L_LINESTATUS) -> (inc, .L_DISCOUNT, .L_EXTENDEDPRICE, .L_QUANTITY, temp, temp) } time 4e-05% fanout 4 input 1 rows group by read node (.L_RETURNFLAG, .L_LINESTATUS, count_order, aggregate, sum_base_price, sum_qty, sum_charge, sum_disc_price) time 6.2e-05% fanout 0 input 4 rows </pre> </code> </blockquote> <p>The <code>SUMs</code> are divided by the <code>COUNTs</code>, and the rows are sorted.</p> <blockquote> <code><pre> Precode: 0: avg_qty := artm sum_qty / count_order 4: avg_price := artm sum_base_price / count_order 8: avg_disc := artm aggregate / count_order 12: BReturn 0 Sort (.L_RETURNFLAG, .L_LINESTATUS) -> (sum_qty, sum_base_price, sum_disc_price, sum_charge, avg_qty, avg_price, avg_disc, count_order) } time 1.2e-05% fanout 4 input 1 rows Key from temp (.L_RETURNFLAG, .L_LINESTATUS, sum_qty, sum_base_price, sum_disc_price, sum_charge, avg_qty, avg_price, avg_disc, count_order) </pre> </code> </blockquote> <p>The data is returned to the client.</p> <blockquote> <code><pre> time 4.4e-06% fanout 0 input 4 rows Select (.L_RETURNFLAG, .L_LINESTATUS, sum_qty, sum_base_price, sum_disc_price, sum_charge, avg_qty, avg_price, avg_disc, count_order) } </pre> </code> </blockquote> <p>Elapsed time and CPU%.</p> <blockquote> <code><pre> 3947 msec 2365% cpu, 6 rnd 5.99841e+08 seq 0% same seg 0% same pg Compilation: 0 msec 0 reads 0% read 0 messages 0% clw </pre> </code> </blockquote> <p>This output is produced by the following sequence on the iSQL command line --</p> <blockquote> <code><pre> SQL> SET blobs ON; SQL> PROFILE ('SELECT .... FROM .....'); </pre> </code> </blockquote> <p>We will next consider the CPU profile:</p> <blockquote> <code><pre> 704173 33.0087 setp_chash_run 275039 12.8927 gb_aggregate 252751 11.8479 ce_dict_any_sets_decode 178304 8.3581 cha_cmp_2a 170819 8.0073 ce_dict_int64_sets_decode 127827 5.9920 chash_array_0 120994 5.6717 chash_array 60902 2.8548 ce_intd_any_range_lte 47865 2.2437 artm_mpy_double 38634 1.8110 ce_vec_int64_sets_decode 26411 1.2380 artm_sub_double 24794 1.1622 artm_add_double 13600 0.6375 cs_decode </pre> </code> </blockquote> <p>For hardcore aficionados, the code may be found in the Virtuoso <code>develop/7.x</code> branch on <a href="https://github.com/openlink/virtuoso-opensource/" id="link-id0x22ea68d8">github.com</a>. The version is not exactly the same but close enough for the parts above. <code>artm_*</code> is arithmetic on typed vectors. As pointed out before, the arithmetic is with <code>DOUBLEs</code>, although users would prefer fixed point. There is, I believe, a MS SQL Server result with <code>DOUBLEs</code>, so using <code>DOUBLEs</code> would not disqualify a 100 GB TPC-H result.</p> <p>The moral of the story is that an array-based aggregation without the <code>chash_array*</code> and <code>cha_cmp*</code> and only 1/3 of the <code>setp_chash_run</code> function would save upwards of a second of real time. The <code>setp_</code> and <code>cha_</code> are aggregation; the <code>ce_*</code> are column decompression and filtering. The arithmetic is not high in the sample but it could be sped up by 2-4x by SIMD, specially since AVX on Sandy Bridge and later does 4 <code>DOUBLEs</code> in a single instruction.</p> <p>We note that the <code>ce_filter_*</code> function would drop off if the table were stored in date order, as then the top level index would show that all the values in the column matched, thus making it unnecessary to even read the <code>l_shipdate</code> column, except for the last part of the table. However this is a marginal slice of the time even now.</p> <h3>Q1 Conclusions</h3> <p>We have demonstrated good load balance and passed the required common sub-expressions exam. The array-based <code>GROUP BY</code> trick is unused but would save over 1s of real time, hence will be good value for only 100-200 lines of code.</p> <h2>Q3 -- Hash and Merge Joins</h2> <p>Next we look at <code><a href="http://dbpedia.org/resource/Join_(SQL)" id="link-id0x29bee6a8">JOINs</a></code> by hash and index. Q3 is a relatively straightforward example, so we will go over the basics of <code>JOIN</code> type (i.e., whether by index or hash) and <code>JOIN</code> order. This will also show some scheduling effects.</p> <p>The definition is:</p> <blockquote> <code><pre> SELECT TOP 10 l_orderkey, SUM(l_extendedprice * (1 - l_discount)) AS revenue, o_orderdate, o_shippriority FROM customer, orders, lineitem WHERE c_mktsegment = 'BUILDING' AND c_custkey = o_custkey AND l_orderkey = o_orderkey AND o_orderdate < CAST ('1995-03-15' AS DATE) AND l_shipdate > CAST ('1995-03-15' AS DATE) GROUP BY l_orderkey, o_orderdate, o_shippriority ORDER BY revenue desc, o_orderdate </pre> </code> </blockquote> <p>The profile, comments inline, is:</p> <blockquote> <code><pre> { time 6.7e-06% fanout 1 input 1 rows </pre> </code> </blockquote> <p>Make a <a href="http://dbpedia.org/resource/Hash_table" id="link-id0x1168e208">hash table</a> with <code>c_custkey</code> for all customers with <code>c_mktsegment</code> building. For a <a href="http://dbpedia.org/resource/Hash_join" id="link-id0x125017f8">hash join</a> build side, the time above the hash filler line is the time for making the hash table from the buffered rows. The time above the <code>sort ... hf ...</code> line is the time for buffering the rows that go into the hash table. The other times in the hash filler block are for the operators for getting the data, and are not related to making the hash table.</p> <blockquote> <code><pre> time 0.72% fanout 1 input 1 rows { hash filler </pre> </code> </blockquote> <p>We see that the actual cardinality of <code>customer</code> is close to what was predicted. The actual number is on the line with time; the predicted is on the line with the index name <code>customer</code>.</p> <blockquote> <code> <pre> time 0.63% fanout 3.00019e+06 input 1 rows CUSTOMER 3e+06 rows(.C_CUSTKEY) C_MKTSEGMENT = <c building="building"> time 0.089% fanout 0 input 3.00019e+06 rows Sort hf 34 (.C_CUSTKEY) } time 9.2e-06% fanout 1 input 1 rows { fork time 5.2e-06% fanout 1 input 1 rows { fork </c> </pre> </code> </blockquote> <p>The below is a merge of a scan of <code>orders</code> and a hash join to <code>customer</code>. The <code>orders</code> table is scanned, first reading <code>o_orderdate</code> and <code>o_custkey</code>, on which there are selections. The <code>o_orderdate</code> is a range check that is true of about 1/2 of the rows. The other condition is an invisible hash join against the <code>customer</code> hash table built above. This selects 1/5 of the rows on the average. So we see that for a total of 150M orders, the fanout is 14.5M, about 1/10, as predicted. The 6.1e7 rows on the line with <code>orders</code> represents the estimate based on the <code>orderdate</code> condition. The card 0.2 on the hash filter line is the prediction for the hash join selectivity.</p> <p>We note that since no <code>order</code> has more than one <code>customer</code>, the <code>JOIN</code> is always cardinality-restricting, hence can be merged into a scan. Being merged into a scan, it becomes run-time re-orderable with the condition on <code>o_orderdate</code>. The conditions are evaluated and arranged at run time in the order of rows eliminated per unit of time.</p> <p>The expression "hash partition + bloom" means that the hash join could be partitioned if the hash table did not fit in memory; i.e., there could be several passes over the data. This is not here the case, nor is this generally desirable. The bloom means that the hash is pre-filtered with a Bloom filter, which we will see in the CPU profile.</p> <blockquote> <code><pre> time 30% fanout 1.45679e+07 input 1 rows ORDERS 6.1e+07 rows(.O_CUSTKEY, .O_ORDERKEY, .O_ORDERDATE, .O_SHIPPRIORITY) O_ORDERDATE < <c 1995-03-15> hash partition+bloom by 41 (tmp)hash join merged always card 0.2 -> () </pre> </code> </blockquote> <p>The below is the hash join operator that in fact was merged into the table scan above.</p> <blockquote> <code><pre> time 0.0016% fanout 1 input 1.45679e+07 rows Hash source 34 merged into ts 0.2 rows(.O_CUSTKEY) -> () </pre> </code> </blockquote> <p>Below is the index-based access to <code>lineitem</code>. This is a <i>de facto</i> <a href="http://dbpedia.org/resource/Sort-merge_join" id="link-id0xc2a6528">merge-join</a> since the <code>o_orderkeys</code> are generated in order by the scan. One in 10 <code>l_orderkeys</code> is selected. Each of these has an average of 4 <code>lineitems</code>. Of these 4, the cost model predicts that 2.7 will be selected based on the additional condition on <code>l_shipdate</code>. The actual number of rows matched is in fact much lower since the date selection is heavily anti-correlated with the date selection on <code>orders</code>. In other words, an <code>order</code> tends to be shipped soon after the <code>orderdate</code>.</p> <blockquote> <code> <pre> time 16% fanout 0.20508 input 1.45679e+07 rows LINEITEM 2.6 rows(.L_ORDERKEY, .L_EXTENDEDPRICE, .L_DISCOUNT) inlined L_ORDERKEY = .O_ORDERKEY L_SHIPDATE > <c> After code: 0: temp := artm 1 - .L_DISCOUNT 4: temp := artm .L_EXTENDEDPRICE * temp 8: BReturn 0 </c> </pre> </code> </blockquote> <p>The query has a <code>GROUP BY</code> that includes the high cardinality column, <code>l_orderkey</code>, with 150M distinct values. The <code>GROUP BY</code> is therefore partitioned.</p> <p>This means that the previous part of the query is run on multiple threads, so that each thread gets an approximately equal number of lines of orders. For the <code>GROUP BY</code>, the threads pass each other chunks of data so that each grouping key can only end up in one partition. This means that at the end of the <code>GROUP BY</code>, there are multiple hash tables with grouping results that are guaranteed non-overlapping, hence there is no need to add up (re-aggregate) the per-thread results. The stage operator passes data between the threads. This is also known as an exchange operator.</p> <blockquote> <code><pre> time 1% fanout 1 input 2.98758e+06 rows Stage 2 time 1.4% fanout 0 input 2.98758e+06 rows Sort (q_.L_ORDERKEY, .O_ORDERDATE, .O_SHIPPRIORITY) -> (temp) } time 0.4% fanout 1.13104e+06 input 1 rows group by read node (.L_ORDERKEY, .O_ORDERDATE, .O_SHIPPRIORITY, revenue)in each partition slice time 0.36% fanout 0 input 1.13104e+06 rows Sort (revenue, .O_ORDERDATE) -> (.L_ORDERKEY, .O_SHIPPRIORITY) } time 3.1e-05% fanout 10 input 1 rows top order by read (.L_ORDERKEY, revenue, .O_ORDERDATE, .O_SHIPPRIORITY) time 6.2e-06% fanout 0 input 10 rows Select (.L_ORDERKEY, revenue, .O_ORDERDATE, .O_SHIPPRIORITY) } 1189 msec 2042% cpu, 1.45513e+07 rnd 2.08588e+08 seq 98.9053% same seg 0.952364% same pg Compilation: 1 msec 0 reads 0% read 0 messages 0% clw </pre> </code> </blockquote> <p>This query also illustrates the meaning of the random and sequential access meters in the profile: For 1/10 of the <code>orders</code>, there is a random lookup from <code>lineitem</code>, hence 14.5M random lookups. The sequential scan is 150M rows of <code>orders</code> plus an average of 3 extra rows for each of the 14M random accesses of <code>lineitem</code>. The locality metric, <i>98.9% same segment</i>, means that the <code>JOIN</code> has a merge-join pattern, since 99% of lookups fall in the same segment as the previous one. A segment is a column store structure that, in the case of <code>lineitem</code>, corresponds to about 4500 consecutive rows.</p> <p>This is one of the queries where storing the data in date order would be advantageous. A zone map on date would eliminate half the second half of the <code>orders</code> without even looking at the columns. A zone map is a summary data structure that keeps, for example, a minimum and a maximum value of an attribute for a range of consecutive rows. Also, for all but the <code>lineitems</code> at the end of the range of <code>orders</code>, a zone map would also disqualify the items without looking at the column. VectorWise, for example, profits from this. However, the CPU profile below shows that the time spent in date compares is not very long even now.</p> <p>On further analysis, we see that the query is run in the order of <code>o_orderkey</code>, so that each <code>o_orderkey</code> is seen once. Hence the partitioned <code>GROUP BY</code> can be changed into an ordered <code>GROUP BY</code>, as all the grouping columns are further functionally dependent on <code>o_orderkey</code>. An ordered <code>GROUP BY</code> is more efficient than a partitioned or re-aggregated one, since it does not have to remember grouping keys: Once a new key comes in, the previous key will not be seen again, and the aggregation for it can be sent onwards in the pipeline.</p> <p>However, this last transformation has little effect here, as the count of rows passing to the aggregation is small. Use of ordered aggregation has much higher impact in other queries and will be visited there. There is also a chance for late projection, as the <code>o_shippriority</code> is in fact only needed for the top 10 rows returned. The impact is small in this case, though. This too will be visited later.</p> <p>We now consider the CPU profile:</p> <blockquote> <code><pre> 93087 20.9350 cha_inline_1i_n 63224 14.2189 cha_bloom_unroll 30162 6.7834 cha_insert_1i_n 29178 6.5621 ce_search_rld 28519 6.4139 ce_intd_range_ltgt 17319 3.8950 cs_decode 15848 3.5642 ce_intd_sets_ltgt 11731 2.6383 ce_skip_bits_2 8212 1.8469 ce_vec_int_sets_decode 7946 1.7870 itc_single_row_opt 7886 1.7735 ce_intd_any_sets_decode 7072 1.5905 itc_fetch_col_vec 6474 1.4560 setp_chash_run 5304 1.1929 itc_ce_value_offset 5263 1.1836 itc_col_seg </pre> </code> </blockquote> <p>The top 3 items are for the <code>orders</code> x <code>customer</code> hash join -- the top 2 for the probe, and the 3rd for the build. The 4th item is the index lookup on <code>lineitem</code>. The one below that is the date condition on <code>orders</code>; below this is the condition on the date of <code>lineitem</code>.</p> <p>The functions working on a compressed column are usually called <code>ce_<compression type>_<sets or range>_<filter or decode></code>. <code>ce</code> means compression entry; the compression types are <code>rl</code> (run length), <code>rld</code> (run length with delta), <code>bits</code> (densely ascending values as bitmap), <code>intd</code> (16 bit deltas on a base), and <code>dict</code> (dictionary). The <code>sets</code> vs <code>range</code> determines whether the operation works on a set of contiguous values in the entry, or takes a vector of row numbers as context. The first predicate works on a range; the next one on the sets (row numbers) selected by the previous. <code>Filter</code> means selection, and <code>decode</code> means extracting a value for processing by a downstream operator.</p> <p>We run 5 of these concurrently: the fastest returns in 2.8s; the slowest in 5.4s. The executions are staggered, so that each divides into up to 24 independent fragments which are then multiplexed on 48 worker threads, with each fragment guaranteed at least one thread. The slices of the first query are prioritized, so that when a worker thread has a choice of next unit of work, it will prefer one from an older queue. Each query in this setting has one queue of independently executable fragments. Thus the first to come in gets the most threads and finishes sooner. The rationale for this is that a query may have large transient memory consumption, e.g., <code>GROUP BYs</code> or hash join build sides. The sooner such a query finishes, the less likely it is that there will be many concurrent queries with the high peak-memory demand. This does not block short queries since in any case a runnable query will have one thread which will get scheduled by the OS from time to time.</p> <h3>Q3 Conclusions</h3> <p>The balance is that unused tricks (ordered aggregation, late projection) would gain little. Date order would gain about 0.4s from 1.3s, but would lose in other queries.</p> <p>We have treated Q1 and Q3 at some length in order to introduce reading of query profiles and the meaning of some meters. For the handful of people who are deep into this sport, the information is rather obvious, but will still give an idea of the specific feature mix of Virtuoso. Column stores are similar to a point, but not all make the exact same choices.</p> <p>If you are a developer, what, if anything, should you remember of this? Never mind the finesses of column store science -- if you understand join order and join type, then there is the possibility of understanding why some queries are fast and some slow. Most support questions are about this. If you know what the DBMS does or should do you are in control. This is why the metrics and concepts here are also of some interest outside the very small group that actually makes DBMS.</p> <h3> <i>In Hoc Signo Vinces</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x233f6158"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac0d9ccf98"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aabda3ead98"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac1ee21098"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac1c7a12a8"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aabe81d5468"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aabc7308158"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aac49385648"> In Hoc Signo Vinces (part 9): TPC-H: TPC-H Q18, Ordered Aggregation, and Top K </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x247daab8"> In Hoc Signo Vinces (part 10): TPC-H: TPC-H Q9, Q17, Q20 - Predicate Games</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x156b4528"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac2b8e4118"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac66827ee8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac32c2b4c8"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x133b7aa8"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aab7749ef18"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aabe8078ee8"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aabd76f3528"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aab521d3d78"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:10:58.209795-04:00
In Hoc Signo Vinces (part 6 of n) -- TPC-H Q1 and Q3: An Introduction to Query Plans
http://www.openlinksw.com:443/weblog/oerling/?date=2013-11-18#1753
2013-11-18T21:57:04Z
<p>Here we look at two <a href="http://www.tpc.org/tpch/" id="link-id0x29fbb0e8">TPC-H</a> queries whose execution plans are relatively straightforward and look at <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x1b09ee88">Virtuoso</a> performance metrics. On one hand; this is an introduction to query plans; on the other hand, a case study in tuning Virtuoso and understanding what goes on. The choke points outlined in <i><a href="http://www.tpc.org/tpctc/tpctc2013/slides_and_papers/005.pdf" id="link-id0x9052148">TPC-H Analyzed</a></i> are mentioned where applicable, with some extra commentary.</p> <h2>Q1 -- Scan, I/O, Aggregation </h2> <p>The query is below. The date is a parameter (a value near the end of the <code>l_shipdate</code> range is used), so most of the columns get read.</p> <blockquote> <code><pre> SELECT l_returnflag, l_linestatus, SUM(l_quantity) AS sum_qty, SUM(l_extendedprice) AS sum_base_price, SUM(l_extendedprice * (1 - l_discount)) AS sum_disc_price, SUM(l_extendedprice * (1 - l_discount) * (1 + l_tax)) AS sum_charge, AVG(l_quantity) AS avg_qty, AVG(l_extendedprice) AS avg_price, AVG(l_discount) AS avg_disc, COUNT(*) AS count_order FROM lineitem WHERE l_shipdate <= dateadd('DAY', -90, CAST ('1998-12-01' AS DATE)) GROUP BY l_returnflag, l_linestatus ORDER BY l_returnflag, l_linestatus ;</pre> </code> </blockquote> <p>We note that a count of a non-nullable column is the same as <code>COUNT (*)</code> and that <code>AVG</code> of a non-null column is <code>SUM (column) / COUNT (*)</code>. So <code>COUNT (*)</code> occurs 4 times, and <code>SUM (l_extendedprice)</code> and <code>SUM (l_quantity)</code> each occur twice. The grouping columns have few distinct values.</p> <p> <i>TPC-H Analyzed</i> suggests to use an array-based <code>GROUP BY</code>, because there can only be 64K combinations of 2 single-character values. The grouping keys are declared <code>CHAR (1)</code> and non-nullable. The Virtuoso implementation does not do this, though.</p> <p>This query has been treated in many papers because it cannot be implemented in very many ways, it is easy to understand, and it still illustrates some basic metrics.</p> <p>One execution on warm cache is between 3.9s and 4.7s. One execution with the data coming from OS disk cache is 11.8s. One execution with the data coming from 2 SSDs is 22s. Five concurrent executions from warm cache are 17.3s for the fastest and 20.5s for the slowest. A single threaded execution from warm cache is 58.4s.</p> <p>We see that scaling is linear; i.e., 5 times the work takes a little under 5x longer. The parallelism is reasonable, with 14.6 speedup from 24 threads on 12 cores. Splitting the work into 48 software threads, time-sliced on 24 hardware threads, does not affect execution time. The work thus appears to be evenly spread on the threads.</p> <p>It may be interesting to see how much data is transferred. To see the space consumption per column --</p> <blockquote> <code><pre> SELECT TOP 20 * FROM sys_index_space_stats ORDER BY iss_pages DESC ; </pre> </code> </blockquote> <p> -- followed by --</p> <blockquote> <code><pre> SELECT coi_column, SUM (coi_pages) / 128 FROM sys_col_info GROUP BY coi_column ORDER BY 2 DESC ; </pre> </code> </blockquote> <p>-- gives us the following --</p> <blockquote> <code><pre> L_COMMENT 17893 PS_COMMENT 10371 O_COMMENT 7824 L_EXTENDEDPRICE 4771 O_CLERK 2744 L_PARTKEY 2432 L_SUPPKEY 2432 L_COMMITDATE 1784 L_SHIPDATE 1551 L_RECEIPTDATE 1537 O_TOTALPRICE 1181 C_COMMENT 1150 L_QUANTITY 960 O_ORDERKEY 736 P_NAME 729 PS_SUPPLYCOST 647 O_CUSTKEY 624 C_ADDRESS 427 L_DISCOUNT 424 L_TAX 419 L_SHIPINSTRUCT 412 L_SHIPMODE 410 L_LINENUMBER 394 L_RETURNFLAG 394 L_LINESTATUS 394 O_ORDERDATE 389 P_COMMENT 341 PS_SUPPKEY 323 P_TYPE 293 C_PHONE 274 L_ORDERKEY 268 PS_AVAILQTY 201 P_RETAILPRICE 161 C_ACCTBAL 123 O_ORDERPRIORITY 95 O_ORDERSTATUS 94 S_COMMENT 66 ... </pre> </code> </blockquote> <p>The total in allocated pages is 65.6 GB, of which 34.1 GB are accessed by the workload. The comment strings could be stream-compressed, bringing some speedup in load time due to less I/O. Also <code>l_extendedprice</code>, a frequently accessed column, could be represented with 4 bytes instead of 8. The working set could thus be cut down to about 28 GB, which may offer some benefit at larger scales. At any rate, for system sizing, the space utilization report is very useful.</p> <p>The query execution profile is as below, with comments inline. The profile here is obvious, but we show this as a guide to reading future profiles which will be more interesting.</p> <blockquote> <code><pre> { time 2.6e-06% fanout 1 input 1 rows time 1.7e-06% fanout 1 input 1 rows { fork time 2.1e-06% fanout 1 input 1 rows { fork </pre> </code> </blockquote> <p>The <code>time xx%</code> line above each operator is the actual percentage of execution time taken by it, followed by the count of rows of output per row of input, followed by the actual rows of input. The below produced 591M rows of output for one row of input --</p> <blockquote> <code><pre> time 34% fanout 5.91599e+08 input 1 rows LINEITEM 5.9e+08 rows(.L_RETURNFLAG, .L_LINESTATUS, .L_DISCOUNT, .L_EXTENDEDPRICE, .L_QUANTITY, .L_TAX) L_SHIPDATE <= <c 1998-09-02> </pre> </code> </blockquote> <p>Below is the arithmetic of the query, followed by a sort (<code>GROUP BY</code>) operator.</p> <blockquote> <code><pre> After code: 0: temp := artm 1 - .L_DISCOUNT 4: temp := artm .L_EXTENDEDPRICE * temp 8: temp := artm 1 + .L_TAX 12: temp := artm temp * temp 16: BReturn 0 </pre> </code> </blockquote> <p>Most of the time is spent below, in the <code>GROUP BY</code>. We notice that each needed aggregation is done once, so the common subexpressions are correctly detected.</p> <blockquote> <code><pre> time 66% fanout 0 input 5.91599e+08 rows Sort (.L_RETURNFLAG, .L_LINESTATUS) -> (inc, .L_DISCOUNT, .L_EXTENDEDPRICE, .L_QUANTITY, temp, temp) } time 4e-05% fanout 4 input 1 rows group by read node (.L_RETURNFLAG, .L_LINESTATUS, count_order, aggregate, sum_base_price, sum_qty, sum_charge, sum_disc_price) time 6.2e-05% fanout 0 input 4 rows </pre> </code> </blockquote> <p>The <code>SUMs</code> are divided by the <code>COUNTs</code>, and the rows are sorted.</p> <blockquote> <code><pre> Precode: 0: avg_qty := artm sum_qty / count_order 4: avg_price := artm sum_base_price / count_order 8: avg_disc := artm aggregate / count_order 12: BReturn 0 Sort (.L_RETURNFLAG, .L_LINESTATUS) -> (sum_qty, sum_base_price, sum_disc_price, sum_charge, avg_qty, avg_price, avg_disc, count_order) } time 1.2e-05% fanout 4 input 1 rows Key from temp (.L_RETURNFLAG, .L_LINESTATUS, sum_qty, sum_base_price, sum_disc_price, sum_charge, avg_qty, avg_price, avg_disc, count_order) </pre> </code> </blockquote> <p>The data is returned to the client.</p> <blockquote> <code><pre> time 4.4e-06% fanout 0 input 4 rows Select (.L_RETURNFLAG, .L_LINESTATUS, sum_qty, sum_base_price, sum_disc_price, sum_charge, avg_qty, avg_price, avg_disc, count_order) } </pre> </code> </blockquote> <p>Elapsed time and CPU%.</p> <blockquote> <code><pre> 3947 msec 2365% cpu, 6 rnd 5.99841e+08 seq 0% same seg 0% same pg Compilation: 0 msec 0 reads 0% read 0 messages 0% clw </pre> </code> </blockquote> <p>This output is produced by the following sequence on the iSQL command line --</p> <blockquote> <code><pre> SQL> SET blobs ON; SQL> PROFILE ('SELECT .... FROM .....'); </pre> </code> </blockquote> <p>We will next consider the CPU profile:</p> <blockquote> <code><pre> 704173 33.0087 setp_chash_run 275039 12.8927 gb_aggregate 252751 11.8479 ce_dict_any_sets_decode 178304 8.3581 cha_cmp_2a 170819 8.0073 ce_dict_int64_sets_decode 127827 5.9920 chash_array_0 120994 5.6717 chash_array 60902 2.8548 ce_intd_any_range_lte 47865 2.2437 artm_mpy_double 38634 1.8110 ce_vec_int64_sets_decode 26411 1.2380 artm_sub_double 24794 1.1622 artm_add_double 13600 0.6375 cs_decode </pre> </code> </blockquote> <p>For hardcore aficionados, the code may be found in the Virtuoso <code>develop/7.x</code> branch on <a href="https://github.com/openlink/virtuoso-opensource/" id="link-id0x22ea68d8">github.com</a>. The version is not exactly the same but close enough for the parts above. <code>artm_*</code> is arithmetic on typed vectors. As pointed out before, the arithmetic is with <code>DOUBLEs</code>, although users would prefer fixed point. There is, I believe, a MS SQL Server result with <code>DOUBLEs</code>, so using <code>DOUBLEs</code> would not disqualify a 100 GB TPC-H result.</p> <p>The moral of the story is that an array-based aggregation without the <code>chash_array*</code> and <code>cha_cmp*</code> and only 1/3 of the <code>setp_chash_run</code> function would save upwards of a second of real time. The <code>setp_</code> and <code>cha_</code> are aggregation; the <code>ce_*</code> are column decompression and filtering. The arithmetic is not high in the sample but it could be sped up by 2-4x by SIMD, specially since AVX on Sandy Bridge and later does 4 <code>DOUBLEs</code> in a single instruction.</p> <p>We note that the <code>ce_filter_*</code> function would drop off if the table were stored in date order, as then the top level index would show that all the values in the column matched, thus making it unnecessary to even read the <code>l_shipdate</code> column, except for the last part of the table. However this is a marginal slice of the time even now.</p> <h3>Q1 Conclusions</h3> <p>We have demonstrated good load balance and passed the required common sub-expressions exam. The array-based <code>GROUP BY</code> trick is unused but would save over 1s of real time, hence will be good value for only 100-200 lines of code.</p> <h2>Q3 -- Hash and Merge Joins</h2> <p>Next we look at <code><a href="http://dbpedia.org/resource/Join_(SQL)" id="link-id0x29bee6a8">JOINs</a></code> by hash and index. Q3 is a relatively straightforward example, so we will go over the basics of <code>JOIN</code> type (i.e., whether by index or hash) and <code>JOIN</code> order. This will also show some scheduling effects.</p> <p>The definition is:</p> <blockquote> <code><pre> SELECT TOP 10 l_orderkey, SUM(l_extendedprice * (1 - l_discount)) AS revenue, o_orderdate, o_shippriority FROM customer, orders, lineitem WHERE c_mktsegment = 'BUILDING' AND c_custkey = o_custkey AND l_orderkey = o_orderkey AND o_orderdate < CAST ('1995-03-15' AS DATE) AND l_shipdate > CAST ('1995-03-15' AS DATE) GROUP BY l_orderkey, o_orderdate, o_shippriority ORDER BY revenue desc, o_orderdate </pre> </code> </blockquote> <p>The profile, comments inline, is:</p> <blockquote> <code><pre> { time 6.7e-06% fanout 1 input 1 rows </pre> </code> </blockquote> <p>Make a <a href="http://dbpedia.org/resource/Hash_table" id="link-id0x1168e208">hash table</a> with <code>c_custkey</code> for all customers with <code>c_mktsegment</code> building. For a <a href="http://dbpedia.org/resource/Hash_join" id="link-id0x125017f8">hash join</a> build side, the time above the hash filler line is the time for making the hash table from the buffered rows. The time above the <code>sort ... hf ...</code> line is the time for buffering the rows that go into the hash table. The other times in the hash filler block are for the operators for getting the data, and are not related to making the hash table.</p> <blockquote> <code><pre> time 0.72% fanout 1 input 1 rows { hash filler </pre> </code> </blockquote> <p>We see that the actual cardinality of <code>customer</code> is close to what was predicted. The actual number is on the line with time; the predicted is on the line with the index name <code>customer</code>.</p> <blockquote> <code> <pre> time 0.63% fanout 3.00019e+06 input 1 rows CUSTOMER 3e+06 rows(.C_CUSTKEY) C_MKTSEGMENT = <c building="building"> time 0.089% fanout 0 input 3.00019e+06 rows Sort hf 34 (.C_CUSTKEY) } time 9.2e-06% fanout 1 input 1 rows { fork time 5.2e-06% fanout 1 input 1 rows { fork </c> </pre> </code> </blockquote> <p>The below is a merge of a scan of <code>orders</code> and a hash join to <code>customer</code>. The <code>orders</code> table is scanned, first reading <code>o_orderdate</code> and <code>o_custkey</code>, on which there are selections. The <code>o_orderdate</code> is a range check that is true of about 1/2 of the rows. The other condition is an invisible hash join against the <code>customer</code> hash table built above. This selects 1/5 of the rows on the average. So we see that for a total of 150M orders, the fanout is 14.5M, about 1/10, as predicted. The 6.1e7 rows on the line with <code>orders</code> represents the estimate based on the <code>orderdate</code> condition. The card 0.2 on the hash filter line is the prediction for the hash join selectivity.</p> <p>We note that since no <code>order</code> has more than one <code>customer</code>, the <code>JOIN</code> is always cardinality-restricting, hence can be merged into a scan. Being merged into a scan, it becomes run-time re-orderable with the condition on <code>o_orderdate</code>. The conditions are evaluated and arranged at run time in the order of rows eliminated per unit of time.</p> <p>The expression "hash partition + bloom" means that the hash join could be partitioned if the hash table did not fit in memory; i.e., there could be several passes over the data. This is not here the case, nor is this generally desirable. The bloom means that the hash is pre-filtered with a Bloom filter, which we will see in the CPU profile.</p> <blockquote> <code><pre> time 30% fanout 1.45679e+07 input 1 rows ORDERS 6.1e+07 rows(.O_CUSTKEY, .O_ORDERKEY, .O_ORDERDATE, .O_SHIPPRIORITY) O_ORDERDATE < <c 1995-03-15> hash partition+bloom by 41 (tmp)hash join merged always card 0.2 -> () </pre> </code> </blockquote> <p>The below is the hash join operator that in fact was merged into the table scan above.</p> <blockquote> <code><pre> time 0.0016% fanout 1 input 1.45679e+07 rows Hash source 34 merged into ts 0.2 rows(.O_CUSTKEY) -> () </pre> </code> </blockquote> <p>Below is the index-based access to <code>lineitem</code>. This is a <i>de facto</i> <a href="http://dbpedia.org/resource/Sort-merge_join" id="link-id0xc2a6528">merge-join</a> since the <code>o_orderkeys</code> are generated in order by the scan. One in 10 <code>l_orderkeys</code> is selected. Each of these has an average of 4 <code>lineitems</code>. Of these 4, the cost model predicts that 2.7 will be selected based on the additional condition on <code>l_shipdate</code>. The actual number of rows matched is in fact much lower since the date selection is heavily anti-correlated with the date selection on <code>orders</code>. In other words, an <code>order</code> tends to be shipped soon after the <code>orderdate</code>.</p> <blockquote> <code> <pre> time 16% fanout 0.20508 input 1.45679e+07 rows LINEITEM 2.6 rows(.L_ORDERKEY, .L_EXTENDEDPRICE, .L_DISCOUNT) inlined L_ORDERKEY = .O_ORDERKEY L_SHIPDATE > <c> After code: 0: temp := artm 1 - .L_DISCOUNT 4: temp := artm .L_EXTENDEDPRICE * temp 8: BReturn 0 </c> </pre> </code> </blockquote> <p>The query has a <code>GROUP BY</code> that includes the high cardinality column, <code>l_orderkey</code>, with 150M distinct values. The <code>GROUP BY</code> is therefore partitioned.</p> <p>This means that the previous part of the query is run on multiple threads, so that each thread gets an approximately equal number of lines of orders. For the <code>GROUP BY</code>, the threads pass each other chunks of data so that each grouping key can only end up in one partition. This means that at the end of the <code>GROUP BY</code>, there are multiple hash tables with grouping results that are guaranteed non-overlapping, hence there is no need to add up (re-aggregate) the per-thread results. The stage operator passes data between the threads. This is also known as an exchange operator.</p> <blockquote> <code><pre> time 1% fanout 1 input 2.98758e+06 rows Stage 2 time 1.4% fanout 0 input 2.98758e+06 rows Sort (q_.L_ORDERKEY, .O_ORDERDATE, .O_SHIPPRIORITY) -> (temp) } time 0.4% fanout 1.13104e+06 input 1 rows group by read node (.L_ORDERKEY, .O_ORDERDATE, .O_SHIPPRIORITY, revenue)in each partition slice time 0.36% fanout 0 input 1.13104e+06 rows Sort (revenue, .O_ORDERDATE) -> (.L_ORDERKEY, .O_SHIPPRIORITY) } time 3.1e-05% fanout 10 input 1 rows top order by read (.L_ORDERKEY, revenue, .O_ORDERDATE, .O_SHIPPRIORITY) time 6.2e-06% fanout 0 input 10 rows Select (.L_ORDERKEY, revenue, .O_ORDERDATE, .O_SHIPPRIORITY) } 1189 msec 2042% cpu, 1.45513e+07 rnd 2.08588e+08 seq 98.9053% same seg 0.952364% same pg Compilation: 1 msec 0 reads 0% read 0 messages 0% clw </pre> </code> </blockquote> <p>This query also illustrates the meaning of the random and sequential access meters in the profile: For 1/10 of the <code>orders</code>, there is a random lookup from <code>lineitem</code>, hence 14.5M random lookups. The sequential scan is 150M rows of <code>orders</code> plus an average of 3 extra rows for each of the 14M random accesses of <code>lineitem</code>. The locality metric, <i>98.9% same segment</i>, means that the <code>JOIN</code> has a merge-join pattern, since 99% of lookups fall in the same segment as the previous one. A segment is a column store structure that, in the case of <code>lineitem</code>, corresponds to about 4500 consecutive rows.</p> <p>This is one of the queries where storing the data in date order would be advantageous. A zone map on date would eliminate half the second half of the <code>orders</code> without even looking at the columns. A zone map is a summary data structure that keeps, for example, a minimum and a maximum value of an attribute for a range of consecutive rows. Also, for all but the <code>lineitems</code> at the end of the range of <code>orders</code>, a zone map would also disqualify the items without looking at the column. VectorWise, for example, profits from this. However, the CPU profile below shows that the time spent in date compares is not very long even now.</p> <p>On further analysis, we see that the query is run in the order of <code>o_orderkey</code>, so that each <code>o_orderkey</code> is seen once. Hence the partitioned <code>GROUP BY</code> can be changed into an ordered <code>GROUP BY</code>, as all the grouping columns are further functionally dependent on <code>o_orderkey</code>. An ordered <code>GROUP BY</code> is more efficient than a partitioned or re-aggregated one, since it does not have to remember grouping keys: Once a new key comes in, the previous key will not be seen again, and the aggregation for it can be sent onwards in the pipeline.</p> <p>However, this last transformation has little effect here, as the count of rows passing to the aggregation is small. Use of ordered aggregation has much higher impact in other queries and will be visited there. There is also a chance for late projection, as the <code>o_shippriority</code> is in fact only needed for the top 10 rows returned. The impact is small in this case, though. This too will be visited later.</p> <p>We now consider the CPU profile:</p> <blockquote> <code><pre> 93087 20.9350 cha_inline_1i_n 63224 14.2189 cha_bloom_unroll 30162 6.7834 cha_insert_1i_n 29178 6.5621 ce_search_rld 28519 6.4139 ce_intd_range_ltgt 17319 3.8950 cs_decode 15848 3.5642 ce_intd_sets_ltgt 11731 2.6383 ce_skip_bits_2 8212 1.8469 ce_vec_int_sets_decode 7946 1.7870 itc_single_row_opt 7886 1.7735 ce_intd_any_sets_decode 7072 1.5905 itc_fetch_col_vec 6474 1.4560 setp_chash_run 5304 1.1929 itc_ce_value_offset 5263 1.1836 itc_col_seg </pre> </code> </blockquote> <p>The top 3 items are for the <code>orders</code> x <code>customer</code> hash join -- the top 2 for the probe, and the 3rd for the build. The 4th item is the index lookup on <code>lineitem</code>. The one below that is the date condition on <code>orders</code>; below this is the condition on the date of <code>lineitem</code>.</p> <p>The functions working on a compressed column are usually called <code>ce_<compression type>_<sets or range>_<filter or decode></code>. <code>ce</code> means compression entry; the compression types are <code>rl</code> (run length), <code>rld</code> (run length with delta), <code>bits</code> (densely ascending values as bitmap), <code>intd</code> (16 bit deltas on a base), and <code>dict</code> (dictionary). The <code>sets</code> vs <code>range</code> determines whether the operation works on a set of contiguous values in the entry, or takes a vector of row numbers as context. The first predicate works on a range; the next one on the sets (row numbers) selected by the previous. <code>Filter</code> means selection, and <code>decode</code> means extracting a value for processing by a downstream operator.</p> <p>We run 5 of these concurrently: the fastest returns in 2.8s; the slowest in 5.4s. The executions are staggered, so that each divides into up to 24 independent fragments which are then multiplexed on 48 worker threads, with each fragment guaranteed at least one thread. The slices of the first query are prioritized, so that when a worker thread has a choice of next unit of work, it will prefer one from an older queue. Each query in this setting has one queue of independently executable fragments. Thus the first to come in gets the most threads and finishes sooner. The rationale for this is that a query may have large transient memory consumption, e.g., <code>GROUP BYs</code> or hash join build sides. The sooner such a query finishes, the less likely it is that there will be many concurrent queries with the high peak-memory demand. This does not block short queries since in any case a runnable query will have one thread which will get scheduled by the OS from time to time.</p> <h3>Q3 Conclusions</h3> <p>The balance is that unused tricks (ordered aggregation, late projection) would gain little. Date order would gain about 0.4s from 1.3s, but would lose in other queries.</p> <p>We have treated Q1 and Q3 at some length in order to introduce reading of query profiles and the meaning of some meters. For the handful of people who are deep into this sport, the information is rather obvious, but will still give an idea of the specific feature mix of Virtuoso. Column stores are similar to a point, but not all make the exact same choices.</p> <p>If you are a developer, what, if anything, should you remember of this? Never mind the finesses of column store science -- if you understand join order and join type, then there is the possibility of understanding why some queries are fast and some slow. Most support questions are about this. If you know what the DBMS does or should do you are in control. This is why the metrics and concepts here are also of some interest outside the very small group that actually makes DBMS.</p> <h3> <i>In Hoc Signo Vinces</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x233f6158"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac0d9ccf98"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aabda3ead98"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac1ee21098"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac1c7a12a8"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x2aabe81d5468"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2aabc7308158"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x20d67a48"> In Hoc Signo Vinces (part 9): TPC-H: TPC-H Q18, Ordered Aggregation, and Top K </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x946a618"> In Hoc Signo Vinces (part 10): TPC-H: TPC-H Q9, Q17, Q20 - Predicate Games</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aabcb9b8548"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aabf9ffcc98"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac64094c08"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac3414f9f8"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x175a0328"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x14714e58"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x1804d4d8"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aac65219ce8"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aabd1c1b8e8"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:07:41.796973-04:00
In Hoc Signo Vinces (part 5 of n) -- The Return of SQL Federation
http://www.openlinksw.com:443/blog/vdb/blog/?date=2013-11-13#1752
2013-11-14T00:15:53Z
<p>In past years, <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aabfb321848">Virtuoso</a> has mostly been known as an <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x2aabc345c6e8">RDF</a> store. Some of you will recall that Virtuoso has always had <a href="http://dbpedia.org/resource/SQL" id="link-id0x2aabf8ffe268">SQL</a> and SQL federation capabilities.</p> <p>With the coming of age of the Virtuoso column store, where this becomes a strong contender for SQL warehousing, the SQL federation aspect is also revitalized.</p> <p>In <a href="http://www.openlinksw.com/weblogs/oerling/?id=1744" id="link-id0x2aabc35b67a8">the previous article</a>, we saw that Virtuoso can load files at well over gigabit-ethernet wire speed. The same of course applies to SQL federation. We can copy the 100 GB <a href="http://www.tpc.org/tpch/" id="link-id0x2aabc39864a8">TPC-H</a> dataset between two Virtuoso instances in only slightly more time than it takes to load the data from files. In a network situation, the network is likely to be the slowest link when extracting data from other SQL stores into Virtuoso. So, to be "semantically elastic," federating has become warehousing. The articles to follow will show excellent query speed for analytics. The combination of this with connectivity to any existing SQL infrastructure makes Virtuoso an easy-to-deploy accelerator cache for almost any data integration situation. This in fact also simplifies query execution, because the more data one can have locally, the more query optimization choices there are, and performance becomes much more predictable than in situations where queries execute across many heterogenous systems. The win is compounded by reducing loads on the line-of-business databases. The missing link in this case becomes heterogenous log shipping. One can usually not modify a line of business system; for example, adding triggers for tracking changes is generally not done. Being able to read transaction logs of all the most common DBMS would offer a solution.</p> <p>The barrier to having one's own extract of data for analysis has become much lower. Even the ETL step can be easily streamlined by the SQL federation. For very time-sensitive applications, one can always keep a local copy of a history in a union with the most recent data accessed from the line-of-business system. At the end of the TPC-H series, we will show examples of a near real-time analytics system that keeps up to date with an Oracle database.</p> <p>For RDF users, this means we have the capacity to extract RDF at bulk load speed from any relational source, whether local or remote. For the test system discussed in the TPC-H series, RDF load shows a sustained throughput of around 320K triples per second. This means that an RDF materialization of the 100 GB TPC-H dataset, about 12.5 billion triples, is done in under 11 hours. This is a vast improvement over the present, and we will show the details in a forthcoming article.</p> <h3> <i>In Hoc Signo Vinces</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x233ad048"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x23d3f248"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x23f8f828"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x22fc65b8"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> In Hoc Signo Vinces (part 5): The Return of SQL Federation<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aac02b6fa88"> In Hoc Signo Vinces (part 9): TPC-H: TPC-H Q18, Ordered Aggregation, and Top K </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aab6ebab028"> In Hoc Signo Vinces (part 10): TPC-H: TPC-H Q9, Q17, Q20 - Predicate Games</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2189c888"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aab707c59d8"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac678110c8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac34b5bfb8"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0xedee4c8"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aabded2a2e8"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aabeb7ff588"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aabf7a5a068"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aab52dc50f8"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:11:04.768627-04:00
In Hoc Signo Vinces (part 4 of n) -- Bulk Load and Refresh
http://www.openlinksw.com:443/blog/vdb/blog/?date=2013-11-13#1751
2013-11-14T00:15:40Z
<p>The server configuration is as in <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/InHocSignoVincesSuppDocs/virtuoso.ini" id="link-id0x2aabe3352788">virtuoso.ini</a></code>, discussed in <a href="http://www.openlinksw.com/weblogs/oerling/?id=1742" id="link-id0x2aabe25b6168">the previous post</a>. The schema is created by loading the file <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/InHocSignoVincesSuppDocs/schema.sql" id="link-id0x2aac18efb818">schema.sql</a></code>, attached. All the tables are stored column-wise. The file contains declarations for hash partitioning in a cluster, but these have no effect on the single-server case. The file tables are declared in <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/InHocSignoVincesSuppDocs/ldschema.sql" id="link-id0x9491548">ldschema.sql</a></code> and bound to files in <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/InHocSignoVincesSuppDocs/ldfile.sql" id="link-id0x2aabe218aee8">ldfile.sql</a></code>. The refresh functions are in <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/InHocSignoVincesSuppDocs/rf.sql" id="link-id0x2aabe0a12128">rf.sql</a></code>.</p> <p>The source data is created with the <code>dbgen</code> utility. One file is generated per table.</p> <p>Twelve refresh datasets are created in order to do the prescribed two runs; each consists of one power test, and one five-stream throughput test. Five streams is the minimum for the 100 GB scale.</p> <h2>Bulk Load</h2> <p>The bulk load script, <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/InHocSignoVincesSuppDocs/ld.sql" id="link-id0x2aabe155fc28">ld.sql</a></code> specifies the CSV files from which the data is loaded as file tables. The load command is simply --</p> <blockquote> <code><pre>log_enable (2); INSERT INTO lineitem SELECT * FROM lineitem_f ;</pre> </code> </blockquote> <p>The <code>log_enable (2)</code> turns off transaction logging, and enables non-transactional inserts. The <code>lineitem</code> table is a column-wise stored database table; the <code>lineitem_f</code> is a table view on the <code>lineitem.tbl</code> CSV file. The load script launches one statement like the above for each table, all in parallel, and then waits for their completion. It then makes an explicit <code>CHECKPOINT</code> to make the data durable. No foreign keys are declared; hence the load does not have to occur in any particular order. Each file is loaded in 24 parallel chunks; the file table facility splits the scan automatically inTO as many chunks as are specified by <code>ThreadsPerQuery</code> in the ini file.</p> <p>The last of the load statements, that for <code>lineitem</code>, completes in 849s of real time. At this point, the data is loaded, and the database is ready for query. There are 3.4M dirty buffers yet to be flushed before the database state is durable. Thus, we must include the checkpoint time in the load result, which adds another 169s. The total load time is hence 16m58s.</p> <p>By the TPC-H rules the timed portion of load must include any gathering of database statistics. We do not do any; rather the queries will derive any needed statistics by sampling at run time.</p> <p>The bulk load has a sustained read rate around 120 MB/s from the source files. The average rate of writing is 60 MB/s. The writing continues long after the read has finished, so we have a truly I/O-bound situation. This can be improved by adding more SSDs. The CPU profile shows a possible gain of around 10%. Thus, with a better I/O system and some more optimization, a load time of about 11m should be possible with this CPU/memory configuration.</p> <p>TPC-H specifies two data refresh operations: one inserting 1/1000th of the <code>orders</code>/<code>lineitem</code> combination; and another deleting the same. The rules leave the implementation largely open; they only specify that the <code>order</code> and its <code>lineitem</code>s must be inserted or deleted within the same transaction.</p> <p>Most implementations bulk-load a staging table, and then do an <code>INSERT ... SELECT</code> statement for the <code>INSERT</code> or a <code>DELETE WHERE IN (SELECT ...)</code> for the <code>DELETE</code>.</p> <p>In <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aabe13333f8">Virtuoso</a>, the refreshes are implemented as SQL procedures that read file tables, as follows:</p> <blockquote> <code><pre>CREATE PROCEDURE rf1 ( IN dir VARCHAR , IN nth INT , IN no_pk INT := 0 , IN rb INT := 0 , IN qp INT := NULL ) { INSERT INTO orders SELECT * FROM orders_f TABLE OPTION ( FROM sprintf ('%s/orders.tbl.u%d', dir, nth) ) ; INSERT INTO lineitem SELECT * FROM lineitem_f TABLE OPTION ( FROM sprintf ('%s/lineitem.tbl.u%d', dir, nth) ) ; COMMIT WORK ; } CREATE PROCEDURE del_batch ( IN d_orderkey INT ) { VECTORED; DELETE FROM lineitem WHERE l_orderkey = d_orderkey ; DELETE FROM orders WHERE o_orderkey = d_orderkey ; } CREATE PROCEDURE rf2 ( IN dir VARCHAR , IN nth INT ) { DECLARE cnt INT ; cnt := ( SELECT COUNT (del_batch (d_orderkey) ) FROM delete_f TABLE OPTION ( FROM sprintf ( '%s/delete.%d', dir, nth ) ) ) ; COMMIT WORK ; RETURN cnt ; }</pre> </code> </blockquote> <p>Things can hardly be simpler. The delete uses a vectored stored procedure to delete a batch of rows in one statement. The parallelization is done automatically, dividing the file table being read into equal size chunks.</p> <p>The performance is as follows. These refreshes are run on the database right after the bulk load.</p> <blockquote> <code><pre>-- Line 33: rf1 ('/1s1/tpch100src', 1) Done. -- 3909 msec. -- Line 34: rf2 ('/1s1/tpch100src', 1) Done. -- 863 msec. -- Line 36: rf1 ('/1s1/tpch100src', 2) Done. -- 2269 msec. -- Line 37: rf2 ('/1s1/tpch100src', 2) Done. -- 871 msec. -- Line 40: rf1 ('/1s1/tpch100src', 3) Done. -- 2315 msec. -- Line 41: rf2 ('/1s1/tpch100src', 3) Done. -- 906 msec. -- Line 43: rf1 ('/1s1/tpch100src', 4) Done. -- 2337 msec. -- Line 44: rf2 ('/1s1/tpch100src', 4) Done. -- 913 msec. -- Line 46: rf1 ('/1s1/tpch100src', 5) Done. -- 2429 msec. -- Line 47: rf2 ('/1s1/tpch100src', 5) Done. -- 1970 msec. -- Line 49: rf1 ('/1s1/tpch100src', 6) Done. -- 2467 msec. -- Line 50: rf2 ('/1s1/tpch100src', 6) Done. -- 888 msec.</pre> </code> </blockquote> <p>The performance is very competitive. Some improvement remains possible but load and refresh are already strong.</p> <p>In the next installment we will look at some queries and explain how to interpret query plans and profiles.</p> <h3> <i>In Hoc Signo Vinces</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x1d1b2148"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aabeb0a0178"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac0adf7ba8"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> In Hoc Signo Vinces (part 4): Bulk Load and Refresh<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x1db753e8"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aac024dd748"> In Hoc Signo Vinces (part 9): TPC-H: TPC-H Q18, Ordered Aggregation, and Top K </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x23286de8"> In Hoc Signo Vinces (part 10): TPC-H: TPC-H Q9, Q17, Q20 - Predicate Games</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x200696a8"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac15301358"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac667b72c8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x18b11288"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x194b8578"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aab74730668"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aabdd4c3558"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aac660a9f88"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x10f4e6ee8"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:11:11.424985-04:00
In Hoc Signo Vinces (part 3 of n) -- Benchmark Configuration Settings
http://www.openlinksw.com:443/blog/vdb/blog/?date=2013-11-13#1750
2013-11-14T00:14:59Z
<p>In this section, we cover the system configuration for running an analytics workload. If you are running a Virtuoso database with even moderate data size, or are interested in reproducing the results presented here, the below will be relevant. If you are interested in the science of query, you can skip to the next installment.</p> <p>The relevant sections of the <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/InHocSignoVincesSuppDocs/virtuoso.ini" id="link-id0x2aabcfc16b98">virtuoso.ini</a></code> file are below, with commentary inline. The actual ini file has many more settings but these do not influence the benchmark.</p> <p>The test file system layout has two SSD file systems, mounted on <code>/1s1</code> and <code>/1s2</code>. The database is striped across the two file systems.</p> <blockquote> <code><pre>[Database] DatabaseFile = virtuoso.db TransactionFile = /1s2/dbs/virtuoso.trx Striping = 1</pre> </code> </blockquote> <p>This sets the log to be on the second SSD, and the database to be striped; the files are declared in the <code>[Striping]</code> section further below.</p> <blockquote> <code><pre>[TempDatabase] DatabaseFile = virtuoso.tdb TransactionFile = virtuoso.ttr [Parameters] ServerPort = 1209 ServerThreads = 100 CheckpointInterval = 0 NumberOfBuffers = 8000000 MaxDirtyBuffers = 1000000</pre> </code> </blockquote> <p>The thread count is set to 100. This is not significant, since the test will only have a few concurrent connections, but this should be at least as high as the number of concurrent user connections expected.</p> <p>The 100 GB TPC-H working set is about 38 GB for the queries. The full database is about 80 GB. Eight million buffers at 8 KB each means that up to 64 GB of database pages will be resident in memory. This should be set higher than the expected working set if possible, but the database process size should also not exceed 80% of physical memory.</p> <p>The max dirty buffers limit is set to a small fraction of the total buffers for faster bulk load. The bulk load is limited by writing to secondary storage, so we want the writing to start early, and continue through the bulk load. Otherwise the checkpoint at the end of the bulk load would be oversized, because of high numbers of un-flushed buffers.</p> <p>The checkpoint interval is set to 0, meaning no automatic checkpoints. There will be one at the end of the bulk load, as required by the rules, but the rules do not require checkpoints for the refresh functions.</p> <blockquote> <code><pre>ColumnStore = 1</pre> </code> </blockquote> <p>This sets all tables to be created column-wise. No special DDL directives are needed for column store operation.</p> <blockquote> <code><pre>MaxCheckpointRemap = 2500000 DefaultIsolation = 2</pre> </code> </blockquote> <p>The default isolation is set to <code>READ COMMITTED</code>. Running large queries with locking on reads would have a very high overhead.</p> <blockquote> <code><pre>DirsAllowed = / TransactionAfterImageLimit = 1500000000</pre> </code> </blockquote> <p>This is set to an arbitrarily high number. The measure is the count of bytes to be written to log at commit (1.5 GB, here). If the amount of data to be logged exceeds this, the transaction aborts. The RF1 transaction at 100 GB scale will log about 100 MB.</p> <blockquote> <code><pre>FDsPerFile = 4 MaxMemPoolSize = 40000000</pre> </code> </blockquote> <p>This is the maximum number of bytes of transient memory to be used for query optimization (40 MB, here). The number is adequate for TPC-H, since the queries only have a few joins each. For RDF workloads, the number should be higher, since there are more joins.</p> <blockquote> <code><pre>AdjustVectorSize = 0</pre> </code> </blockquote> <p>The workload will run at the default vector size. Index operations can be accelerated by switching to a larger vector size, trading memory for locality. But since this workload is mostly by hash join, there is no benefit in changing this.</p> <blockquote> <code><pre>ThreadsPerQuery = 24</pre> </code> </blockquote> <p>Each query is divided into up to 24 parallel fragments. 24 is the number of threads on the test system.</p> <blockquote> <code><pre>AsyncQueueMaxThreads = 48</pre> </code> </blockquote> <p>Queries are run by a pool of 48 worker threads. Each session has one thread of its own. If a query parallelizes, the first fragment runs on the session's thread and the remaining fragments run on a thread from this pool. Thus the core threads are oversubscribed by a factor of slightly over 2 in the throughput run: 6 sessions plus 48 threads makes up to 53 runnable threads at any point in the throughput test.</p> <blockquote> <code><pre>MaxQueryMem = 30G</pre> </code> </blockquote> <p>This is a cap on query execution memory. If memory would exceed this, optimizations that would increase space consumption are not used. The memory may still transiently exceed this limit.</p> <blockquote> <code><pre>HashJoinSpace = 30G</pre> </code> </blockquote> <p>This is the maximum memory to be used for hash tables during hash joins. If a hash join causes this amount to be exceeded, it will be run in multiple passes, so as to have a cap on the hash table size. Not all hash joins may be partitioned, and the test must not do multi-pass hash joins, hence a high number here. We will see actual space consumption figures when looking at the queries. This parameter may be increased for analytics performance, especially in multiuser situations.</p> <blockquote> <code><pre>[Client] SQL_QUERY_TIMEOUT = 0 SQL_TXN_TIMEOUT = 0 SQL_ROWSET_SIZE = 10 SQL_PREFETCH_BYTES = 120000</pre> </code> </blockquote> <p>120 KB of results is to be sent to clients in a single window. This is enough for the relatively short result sets in this benchmark.</p> <blockquote> <code><pre>[Striping] Segment1 = 1024, /1s1/dbs/tpch100cp-1.db = q1, /1s2/dbs/tpch100cp-2.db = q2</pre> </code> </blockquote> <p>The database is set to stripe in two files, each on a different SSD. Each file has its own background I/O thread; this is the meaning of the <code>= q1</code> and <code>= q2</code> declaration. All files on each separately-seekable device should share the same <code>q</code>.</p> <blockquote> <code><pre>[Flags] enable_mt_txn = 1 enable_mt_transact = 1</pre> </code> </blockquote> <p>The first setting enables multithreading DML statement execution. The second setting enables multithreading of <code>COMMIT</code> or <code>ROLLBACK</code> operations. This is important for the refresh function performance. A column store <code>COMMIT</code> of a <code>DELETE</code> will especially benefit from multithreading, since this may involve re-compression.</p> <blockquote> <code><pre>hash_join_enable = 2</pre> </code> </blockquote> <p>Will use hash joins for SQL and SPARQL (even though SPARQL is not used in this experiment).</p> <blockquote> <code><pre>dbf_explain_level = 0</pre> </code> </blockquote> <p>Specifies less verbose query plan formatting for logging of query execution.</p> <blockquote> <code><pre>dbf_log_fsync = 1</pre> </code> </blockquote> <p>Specify that <code>fsync</code> is to be called after each write to the transaction log. The ACID qualification procedure specifies that the system is to be powered down in mid-run, hence this setting is required by the test.</p> <h3> <i>In Hoc Signo Vinces</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac10367338"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac10369458"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac09638778"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac1036f9d8"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x8b44fc8"> In Hoc Signo Vinces (part 9): TPC-H: TPC-H Q18, Ordered Aggregation, and Top K </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x204a3a98"> In Hoc Signo Vinces (part 10): TPC-H: TPC-H Q9, Q17, Q20 - Predicate Games</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac09d78ed8"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac2b23a6d8"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac50e18368"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0xa5a8188"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x2aab7962cef8"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aabe9842418"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aac0103faf8"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aac4b265be8"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x10c987558"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:11:17.921466-04:00
In Hoc Signo Vinces (part 2 of n) -- TPC-H Schema Choices
http://www.openlinksw.com:443/blog/vdb/blog/?date=2013-11-13#1749
2013-11-14T00:14:47Z
<p> <a href="http://www.tpc.org/tpch/" id="link-id0x2aac182bc168">TPC-H</a> gives a strict definition of the table structure and content, but leaves implementations some latitude in defining indices and data layout. A TPC-H run consists of a power test with a single user submitting queries in sequence, and of a throughput test where many concurrent users are simulated. An implementation may choose how many concurrent users are simulated, as long as this is more than a set minimum.</p> <p>Tables do not always have to be stored in the order of their primary key. A clustered index is a structure where the index tree leaves contain the whole row of data. TPC-H explicitly forbids materializing multiple copies of a table in different sort orders for speeding up different queries. That said, an implementation may pick one primary order for each table which does not have to be the primary key order.</p> <p>The present champion in core-for-core speed, Actian <a href="http://dbpedia.org/resource/Vectorwise" id="link-id0x2aab8cb5f898">VectorWise</a>, organizes the main tables, <code>lineitem</code> and <code>orders</code>, in date order. The <a href="http://dbpedia.org/page/Microsoft_SQL_Server" id="link-id0x2aab8d146198">Microsoft SQL Server</a> implementations use clustered indices on <code>lineitem</code> and <code>orders</code>, where the table is stored in non-primary key order; in this case, the major ordering columns are <code>l_shipdate</code> and <code>o_orderdate</code>, respectively.</p> <p>These index schemes create locality on the date dimension.</p> <p>In the present discussion, we take a different tack: We keep <code>lineitem</code> and <code>orders</code> in primary key order, and sacrifice locality on date in favor of a fast merge join between these two tables, and for faster load and data maintenance.</p> <p>TPC-H rules allow indices on foreign keys and dates. In this implementation, we only define one on <code>o_custkey</code>; <code>lineitem</code> is only indexed on its primary key. These are the two largest tables, and the only ones that change during the benchmark. Whether indices are defined on other tables makes little or no difference, since joins between these tend to perform better by hash than by index in any case.</p> <h2>Scale and Test System</h2> <p>All the runs are done at 100 GB scale. The test machine is a dual E5-2630 (2x6 cores, 2x12 threads, 2.3 GHz) with 192 GB of 1066 MHz RAM. Two Crucial 512 GB SSDs are used both for database and staging of the files. The disks are independent file systems with no RAID. (Note that RAID would be required for an official result.) The operating system is CentOS 6.2.</p> <p>For cluster results, two machines with the above spec are used with a QDR InfiniBand interconnect.</p> <p>The Virtuoso used is a current internal development version, to be available at the end of this series or by special request. This is neither the publicly available open nor closed source version.</p> <h3> <i>In Hoc Signo Vinces</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x233f6158"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> In Hoc Signo Vinces (part 2): TPC-H Schema Choices<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x22454708"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2213e628"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x239b07e8"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x71bc8e8"> In Hoc Signo Vinces (part 9): TPC-H: TPC-H Q18, Ordered Aggregation, and Top K </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x9419bc8"> In Hoc Signo Vinces (part 10): TPC-H: TPC-H Q9, Q17, Q20 - Predicate Games</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x21bff8c8"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac2ab001f8"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac65da3028"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x18fd1cb8"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0xfa85898"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aabeb7ad658"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aabea871078"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aac645fd4e8"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aaad8672ca8"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:11:24.547353-04:00
In Hoc Signo Vinces (part 1 of n) -- Virtuoso meets TPC-H
http://www.openlinksw.com:443/blog/vdb/blog/?date=2013-11-13#1748
2013-11-14T00:14:37Z
<p> <a href="http://www.tpc.org/tpch/" id="link-id0x14305a58">TPC-H</a> is <i>the</i> data warehousing benchmark to date. I will here cover the <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aac18ee59d8">Virtuoso</a> implementation of this in detail. The primary audience is database experts. This will also be very educational for DBAs and advanced application developers: Life becomes much more predictable if one knows a good query plan from a bad one. Alongside a commentary on database science, you will also find here a guided tour of Virtuoso performance tuning and diagnostics. To follow this, it is useful to have the official TPC-H spec at hand (download links are on the far-right of <a href="http://www.tpc.org/tpch/" id="link-id0x1418dc38">this page</a>).</p> <p>By now, TPC-H is an old game and it is safe to say that pretty much any player in the analytics database domain has had a go at it, even though some have never published a result. So, the bar for new entrants is very high.</p> <p>Especially, <a href="http://dbpedia.org/page/Vectorwise" id="link-id0x8f6ba08">VectorWise</a> and <a href="http://dbpedia.org/resource/EXASolution" id="link-id0x2aac18ccdfc8">EXASolution</a> have taken performance in this workload close to the limits of the achievable. A challenger has to do everything right in order to win. One wrong move will lose the whole race.</p> <p>This presentation has many objectives:</p> <ul> <li> <p>To illustrate how Virtuoso is an excellent SQL analytics engine</p> </li> <li> <p>To provide an in-depth discussion on the science of query optimization and execution</p> </li> <li> <p>To outline avenues of future development, specifically as concerns analytics with schema-less data</p> </li> </ul> <p>In the <a href="http://www.tpc.org/tpctc/tpctc2013/" id="link-id0x2aac182d14d8">TPC TC workshop</a> at <a href="http://www.vldb.org/2013" id="link-id0x14ae6bb8">VLDB 2013</a> there was a paper <i><a href="http://www.tpc.org/tpctc/tpctc2013/slides_and_papers/005.pdf" id="link-id0xa9316e8">TPC-H Analyzed: Hidden Messages and Lessons Learned from an Influential Benchmark</a></i>, by <a href="http://homepages.cwi.nl/~boncz/" id="link-id0x127527a8">Peter Boncz</a>, <a href="http://www-db.in.tum.de/~neumann/" id="link-id0x14a12d18">Thomas Neumann</a>, and <a href="http://www.linkedin.com/in/orrierling" id="link-id0x14a7a308">myself</a> concerning what the database world has learned from this very tough exercise. Peter Boncz is the original architect of Actian VectorWise, the current champion in TPC-H performance per core. Thomas Neumann is the author of <a href="http://www-db.in.tum.de/research/projects/HyPer/" id="link-id0x14a99168">HyPer</a>, most likely the best entry in DBMS research for simultaneously supporting analytics and OLTP. Peter and Thomas are among the most renowned in database science. I am the Program Manager of the Virtuoso column store, overseeing core engineering tasks such as SQL query optimization, execution, storage, and scale out. </p> <p>In this series I will go over the Virtuoso implementation of TPC-H and will elaborate further on the points discussed in the paper. The subject is broader than any single paper can cover in detail, although there are plenty of papers only addressing one or two of the 22 queries.</p> <p>Virtuoso is mostly known for <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x955fb38">RDF</a>. Here we will cover the whole benchmark in <a href="http://dbpedia.org/resource/SQL" id="link-id0xaca8ae8">SQL</a> first, with both single-server and cluster implementations, and discussion of where these differ. A state-of-the-art SQL implementation is the necessary basis for discussing how the same can be accomplished in RDF. Comparing good RDF to bad SQL is not interesting.</p> <p>The earlier articles on the Star Schema Benchmark (SSB) (<a href="http://www.cs.umb.edu/~poneil/StarSchemaB.PDF" id="link-id0x14aef2a8">PDF</a>) -- <i><a href="http://www.openlinksw.com/weblogs/oerling/?id=1732" id="link-id0x13a8f708">Annuit Coeptis, or, Star Schema and The Cost of Freedom</a></i> and <i><a href="http://www.openlinksw.com/weblogs/oerling/?id=1733" id="link-id0x14aa6b58">E Pluribus Unum, or, Star Schema Meets Cluster</a></i> -- demonstrated how the most basic analytical database operations perform in Virtuoso. All the techniques used there are also directly applicable to TPC-H, but the latter adds a good 20 more tricks one needs to see through.</p> <p>Future installments will discuss TPC-H query by query. We conclude with a full run of <a href="http://sourceforge.net/apps/mediawiki/osdldbt/" id="link-id0x14abca48">OSDL-DBT-3</a>. DBT-3™ is an unofficial TPC-H without auditing but with the same workload.</p> <h3> <i>In Hoc Signo Vinces</i> Series</h3> <ul> <li> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac0b114898"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac06d43db8"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac0bf375e8"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac06d24268"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aac4b54c208"> In Hoc Signo Vinces (part 9): TPC-H: TPC-H Q18, Ordered Aggregation, and Top K </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2251ead8"> In Hoc Signo Vinces (part 10): TPC-H: TPC-H Q9, Q17, Q20 - Predicate Games</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac08234e28"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac2ab89598"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac74e794a8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac332bb958"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x2aab6f50a6c8"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aabe94b29a8"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aabe97d03b8"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aac9c3dc178"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aab53c91a78"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:11:29.522883-04:00
In Hoc Signo Vinces (part 5 of n) -- The Return of SQL Federation
http://www.openlinksw.com:443/weblog/oerling/?date=2013-11-13#1747
2013-11-14T00:06:08Z
<p>In past years, <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aabfb321848">Virtuoso</a> has mostly been known as an <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x2aabc345c6e8">RDF</a> store. Some of you will recall that Virtuoso has always had <a href="http://dbpedia.org/resource/SQL" id="link-id0x2aabf8ffe268">SQL</a> and SQL federation capabilities.</p> <p>With the coming of age of the Virtuoso column store, where this becomes a strong contender for SQL warehousing, the SQL federation aspect is also revitalized.</p> <p>In <a href="http://www.openlinksw.com/weblogs/oerling/?id=1744" id="link-id0x2aabc35b67a8">the previous article</a>, we saw that Virtuoso can load files at well over gigabit-ethernet wire speed. The same of course applies to SQL federation. We can copy the 100 GB <a href="http://www.tpc.org/tpch/" id="link-id0x2aabc39864a8">TPC-H</a> dataset between two Virtuoso instances in only slightly more time than it takes to load the data from files. In a network situation, the network is likely to be the slowest link when extracting data from other SQL stores into Virtuoso. So, to be "semantically elastic," federating has become warehousing. The articles to follow will show excellent query speed for analytics. The combination of this with connectivity to any existing SQL infrastructure makes Virtuoso an easy-to-deploy accelerator cache for almost any data integration situation. This in fact also simplifies query execution, because the more data one can have locally, the more query optimization choices there are, and performance becomes much more predictable than in situations where queries execute across many heterogenous systems. The win is compounded by reducing loads on the line-of-business databases. The missing link in this case becomes heterogenous log shipping. One can usually not modify a line of business system; for example, adding triggers for tracking changes is generally not done. Being able to read transaction logs of all the most common DBMS would offer a solution.</p> <p>The barrier to having one's own extract of data for analysis has become much lower. Even the ETL step can be easily streamlined by the SQL federation. For very time-sensitive applications, one can always keep a local copy of a history in a union with the most recent data accessed from the line-of-business system. At the end of the TPC-H series, we will show examples of a near real-time analytics system that keeps up to date with an Oracle database.</p> <p>For RDF users, this means we have the capacity to extract RDF at bulk load speed from any relational source, whether local or remote. For the test system discussed in the TPC-H series, RDF load shows a sustained throughput of around 320K triples per second. This means that an RDF materialization of the 100 GB TPC-H dataset, about 12.5 billion triples, is done in under 11 hours. This is a vast improvement over the present, and we will show the details in a forthcoming article.</p> <h3> <i>In Hoc Signo Vinces</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x233ad048"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x23d3f248"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x23f8f828"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x22fc65b8"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> In Hoc Signo Vinces (part 5): The Return of SQL Federation<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2cbf92d8"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x14609048"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x9685b48"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x101b44c8"> In Hoc Signo Vinces (part 9): TPC-H: TPC-H Q18, Ordered Aggregation, and Top K </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aac14a29ab8"> In Hoc Signo Vinces (part 10): TPC-H: TPC-H Q9, Q17, Q20 - Predicate Games</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac3672dc98"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac2a8654e8"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x1f028528"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac3477e948"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x2aabdbe3c0f8"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aab6b98bfa8"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x1698e778"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aabc9187e78"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aabd2b02798"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:07:47.154096-04:00
In Hoc Signo Vinces (part 4 of n) -- Bulk Load and Refresh
http://www.openlinksw.com:443/weblog/oerling/?date=2013-11-13#1744
2013-11-14T00:01:39Z
<p>The server configuration is as in <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/InHocSignoVincesSuppDocs/virtuoso.ini" id="link-id0x2aabe3352788">virtuoso.ini</a></code>, discussed in <a href="http://www.openlinksw.com/weblogs/oerling/?id=1742" id="link-id0x2aabe25b6168">the previous post</a>. The schema is created by loading the file <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/InHocSignoVincesSuppDocs/schema.sql" id="link-id0x2aac18efb818">schema.sql</a></code>, attached. All the tables are stored column-wise. The file contains declarations for hash partitioning in a cluster, but these have no effect on the single-server case. The file tables are declared in <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/InHocSignoVincesSuppDocs/ldschema.sql" id="link-id0x9491548">ldschema.sql</a></code> and bound to files in <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/InHocSignoVincesSuppDocs/ldfile.sql" id="link-id0x2aabe218aee8">ldfile.sql</a></code>. The refresh functions are in <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/InHocSignoVincesSuppDocs/rf.sql" id="link-id0x2aabe0a12128">rf.sql</a></code>.</p> <p>The source data is created with the <code>dbgen</code> utility. One file is generated per table.</p> <p>Twelve refresh datasets are created in order to do the prescribed two runs; each consists of one power test, and one five-stream throughput test. Five streams is the minimum for the 100 GB scale.</p> <h2>Bulk Load</h2> <p>The bulk load script, <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/InHocSignoVincesSuppDocs/ld.sql" id="link-id0x2aabe155fc28">ld.sql</a></code> specifies the CSV files from which the data is loaded as file tables. The load command is simply --</p> <blockquote> <code><pre>log_enable (2); INSERT INTO lineitem SELECT * FROM lineitem_f ;</pre> </code> </blockquote> <p>The <code>log_enable (2)</code> turns off transaction logging, and enables non-transactional inserts. The <code>lineitem</code> table is a column-wise stored database table; the <code>lineitem_f</code> is a table view on the <code>lineitem.tbl</code> CSV file. The load script launches one statement like the above for each table, all in parallel, and then waits for their completion. It then makes an explicit <code>CHECKPOINT</code> to make the data durable. No foreign keys are declared; hence the load does not have to occur in any particular order. Each file is loaded in 24 parallel chunks; the file table facility splits the scan automatically inTO as many chunks as are specified by <code>ThreadsPerQuery</code> in the ini file.</p> <p>The last of the load statements, that for <code>lineitem</code>, completes in 849s of real time. At this point, the data is loaded, and the database is ready for query. There are 3.4M dirty buffers yet to be flushed before the database state is durable. Thus, we must include the checkpoint time in the load result, which adds another 169s. The total load time is hence 16m58s.</p> <p>By the TPC-H rules the timed portion of load must include any gathering of database statistics. We do not do any; rather the queries will derive any needed statistics by sampling at run time.</p> <p>The bulk load has a sustained read rate around 120 MB/s from the source files. The average rate of writing is 60 MB/s. The writing continues long after the read has finished, so we have a truly I/O-bound situation. This can be improved by adding more SSDs. The CPU profile shows a possible gain of around 10%. Thus, with a better I/O system and some more optimization, a load time of about 11m should be possible with this CPU/memory configuration.</p> <p>TPC-H specifies two data refresh operations: one inserting 1/1000th of the <code>orders</code>/<code>lineitem</code> combination; and another deleting the same. The rules leave the implementation largely open; they only specify that the <code>order</code> and its <code>lineitem</code>s must be inserted or deleted within the same transaction.</p> <p>Most implementations bulk-load a staging table, and then do an <code>INSERT ... SELECT</code> statement for the <code>INSERT</code> or a <code>DELETE WHERE IN (SELECT ...)</code> for the <code>DELETE</code>.</p> <p>In <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aabe13333f8">Virtuoso</a>, the refreshes are implemented as SQL procedures that read file tables, as follows:</p> <blockquote> <code><pre>CREATE PROCEDURE rf1 ( IN dir VARCHAR , IN nth INT , IN no_pk INT := 0 , IN rb INT := 0 , IN qp INT := NULL ) { INSERT INTO orders SELECT * FROM orders_f TABLE OPTION ( FROM sprintf ('%s/orders.tbl.u%d', dir, nth) ) ; INSERT INTO lineitem SELECT * FROM lineitem_f TABLE OPTION ( FROM sprintf ('%s/lineitem.tbl.u%d', dir, nth) ) ; COMMIT WORK ; } CREATE PROCEDURE del_batch ( IN d_orderkey INT ) { VECTORED; DELETE FROM lineitem WHERE l_orderkey = d_orderkey ; DELETE FROM orders WHERE o_orderkey = d_orderkey ; } CREATE PROCEDURE rf2 ( IN dir VARCHAR , IN nth INT ) { DECLARE cnt INT ; cnt := ( SELECT COUNT (del_batch (d_orderkey) ) FROM delete_f TABLE OPTION ( FROM sprintf ( '%s/delete.%d', dir, nth ) ) ) ; COMMIT WORK ; RETURN cnt ; }</pre> </code> </blockquote> <p>Things can hardly be simpler. The delete uses a vectored stored procedure to delete a batch of rows in one statement. The parallelization is done automatically, dividing the file table being read into equal size chunks.</p> <p>The performance is as follows. These refreshes are run on the database right after the bulk load.</p> <blockquote> <code><pre>-- Line 33: rf1 ('/1s1/tpch100src', 1) Done. -- 3909 msec. -- Line 34: rf2 ('/1s1/tpch100src', 1) Done. -- 863 msec. -- Line 36: rf1 ('/1s1/tpch100src', 2) Done. -- 2269 msec. -- Line 37: rf2 ('/1s1/tpch100src', 2) Done. -- 871 msec. -- Line 40: rf1 ('/1s1/tpch100src', 3) Done. -- 2315 msec. -- Line 41: rf2 ('/1s1/tpch100src', 3) Done. -- 906 msec. -- Line 43: rf1 ('/1s1/tpch100src', 4) Done. -- 2337 msec. -- Line 44: rf2 ('/1s1/tpch100src', 4) Done. -- 913 msec. -- Line 46: rf1 ('/1s1/tpch100src', 5) Done. -- 2429 msec. -- Line 47: rf2 ('/1s1/tpch100src', 5) Done. -- 1970 msec. -- Line 49: rf1 ('/1s1/tpch100src', 6) Done. -- 2467 msec. -- Line 50: rf2 ('/1s1/tpch100src', 6) Done. -- 888 msec.</pre> </code> </blockquote> <p>The performance is very competitive. Some improvement remains possible but load and refresh are already strong.</p> <p>In future installments we will look at some queries and explain how to interpret query plans and profiles.</p> <h3> <i>In Hoc Signo Vinces</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x1d1b2148"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aabeb0a0178"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac0adf7ba8"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> In Hoc Signo Vinces (part 4): Bulk Load and Refresh<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x1db753e8"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0xb553bc8"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x8fa5c08"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2dea7aa8"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x22ec08b8"> In Hoc Signo Vinces (part 9): TPC-H: TPC-H Q18, Ordered Aggregation, and Top K </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aabc519d998"> In Hoc Signo Vinces (part 10): TPC-H: TPC-H Q9, Q17, Q20 - Predicate Games</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac1f18a6e8"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac1c239438"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac66b1b9e8"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aac3485abf8"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x2aabda0958f8"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0xa470658"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x1601c018"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aac64d7e928"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aabd1c61bb8"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:07:51.714822-04:00
In Hoc Signo Vinces (part 3 of n) -- Benchmark Configuration Settings
http://www.openlinksw.com:443/weblog/oerling/?date=2013-11-13#1742
2013-11-13T23:51:17Z
<p>In this section, we cover the system configuration for running an analytics workload. If you are running a Virtuoso database with even moderate data size, or are interested in reproducing the results presented here, the below will be relevant. If you are interested in the science of query, you can skip to the next installment.</p> <p>The relevant sections of the <code><a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/InHocSignoVincesSuppDocs/virtuoso.ini" id="link-id0x2aabcfc16b98">virtuoso.ini</a></code> file are below, with commentary inline. The actual ini file has many more settings but these do not influence the benchmark.</p> <p>The test file system layout has two SSD file systems, mounted on <code>/1s1</code> and <code>/1s2</code>. The database is striped across the two file systems.</p> <blockquote> <code><pre>[Database] DatabaseFile = virtuoso.db TransactionFile = /1s2/dbs/virtuoso.trx Striping = 1</pre> </code> </blockquote> <p>This sets the log to be on the second SSD, and the database to be striped; the files are declared in the <code>[Striping]</code> section further below.</p> <blockquote> <code><pre>[TempDatabase] DatabaseFile = virtuoso.tdb TransactionFile = virtuoso.ttr [Parameters] ServerPort = 1209 ServerThreads = 100 CheckpointInterval = 0 NumberOfBuffers = 8000000 MaxDirtyBuffers = 1000000</pre> </code> </blockquote> <p>The thread count is set to 100. This is not significant, since the test will only have a few concurrent connections, but this should be at least as high as the number of concurrent user connections expected.</p> <p>The 100 GB TPC-H working set is about 38 GB for the queries. The full database is about 80 GB. Eight million buffers at 8 KB each means that up to 64 GB of database pages will be resident in memory. This should be set higher than the expected working set if possible, but the database process size should also not exceed 80% of physical memory.</p> <p>The max dirty buffers limit is set to a small fraction of the total buffers for faster bulk load. The bulk load is limited by writing to secondary storage, so we want the writing to start early, and continue through the bulk load. Otherwise the checkpoint at the end of the bulk load would be oversized, because of high numbers of un-flushed buffers.</p> <p>The checkpoint interval is set to 0, meaning no automatic checkpoints. There will be one at the end of the bulk load, as required by the rules, but the rules do not require checkpoints for the refresh functions.</p> <blockquote> <code><pre>ColumnStore = 1</pre> </code> </blockquote> <p>This sets all tables to be created column-wise. No special DDL directives are needed for column store operation.</p> <blockquote> <code><pre>MaxCheckpointRemap = 2500000 DefaultIsolation = 2</pre> </code> </blockquote> <p>The default isolation is set to <code>READ COMMITTED</code>. Running large queries with locking on reads would have a very high overhead.</p> <blockquote> <code><pre>DirsAllowed = / TransactionAfterImageLimit = 1500000000</pre> </code> </blockquote> <p>This is set to an arbitrarily high number. The measure is the count of bytes to be written to log at commit (1.5 GB, here). If the amount of data to be logged exceeds this, the transaction aborts. The RF1 transaction at 100 GB scale will log about 100 MB.</p> <blockquote> <code><pre>FDsPerFile = 4 MaxMemPoolSize = 40000000</pre> </code> </blockquote> <p>This is the maximum number of bytes of transient memory to be used for query optimization (40 MB, here). The number is adequate for TPC-H, since the queries only have a few joins each. For RDF workloads, the number should be higher, since there are more joins.</p> <blockquote> <code><pre>AdjustVectorSize = 0</pre> </code> </blockquote> <p>The workload will run at the default vector size. Index operations can be accelerated by switching to a larger vector size, trading memory for locality. But since this workload is mostly by hash join, there is no benefit in changing this.</p> <blockquote> <code><pre>ThreadsPerQuery = 24</pre> </code> </blockquote> <p>Each query is divided into up to 24 parallel fragments. 24 is the number of threads on the test system.</p> <blockquote> <code><pre>AsyncQueueMaxThreads = 48</pre> </code> </blockquote> <p>Queries are run by a pool of 48 worker threads. Each session has one thread of its own. If a query parallelizes, the first fragment runs on the session's thread and the remaining fragments run on a thread from this pool. Thus the core threads are oversubscribed by a factor of slightly over 2 in the throughput run: 6 sessions plus 48 threads makes up to 53 runnable threads at any point in the throughput test.</p> <blockquote> <code><pre>MaxQueryMem = 30G</pre> </code> </blockquote> <p>This is a cap on query execution memory. If memory would exceed this, optimizations that would increase space consumption are not used. The memory may still transiently exceed this limit.</p> <blockquote> <code><pre>HashJoinSpace = 30G</pre> </code> </blockquote> <p>This is the maximum memory to be used for hash tables during hash joins. If a hash join causes this amount to be exceeded, it will be run in multiple passes, so as to have a cap on the hash table size. Not all hash joins may be partitioned, and the test must not do multi-pass hash joins, hence a high number here. We will see actual space consumption figures when looking at the queries. This parameter may be increased for analytics performance, especially in multiuser situations.</p> <blockquote> <code><pre>[Client] SQL_QUERY_TIMEOUT = 0 SQL_TXN_TIMEOUT = 0 SQL_ROWSET_SIZE = 10 SQL_PREFETCH_BYTES = 120000</pre> </code> </blockquote> <p>120 KB of results is to be sent to clients in a single window. This is enough for the relatively short result sets in this benchmark.</p> <blockquote> <code><pre>[Striping] Segment1 = 1024, /1s1/dbs/tpch100cp-1.db = q1, /1s2/dbs/tpch100cp-2.db = q2</pre> </code> </blockquote> <p>The database is set to stripe in two files, each on a different SSD. Each file has its own background I/O thread; this is the meaning of the <code>= q1</code> and <code>= q2</code> declaration. All files on each separately-seekable device should share the same <code>q</code>.</p> <blockquote> <code><pre>[Flags] enable_mt_txn = 1 enable_mt_transact = 1</pre> </code> </blockquote> <p>The first setting enables multithreading DML statement execution. The second setting enables multithreading of <code>COMMIT</code> or <code>ROLLBACK</code> operations. This is important for the refresh function performance. A column store <code>COMMIT</code> of a <code>DELETE</code> will especially benefit from multithreading, since this may involve re-compression.</p> <blockquote> <code><pre>hash_join_enable = 2</pre> </code> </blockquote> <p>Will use hash joins for SQL and SPARQL (even though SPARQL is not used in this experiment).</p> <blockquote> <code><pre>dbf_explain_level = 0</pre> </code> </blockquote> <p>Specifies less verbose query plan formatting for logging of query execution.</p> <blockquote> <code><pre>dbf_log_fsync = 1</pre> </code> </blockquote> <p>Specify that <code>fsync</code> is to be called after each write to the transaction log. The ACID qualification procedure specifies that the system is to be powered down in mid-run, hence this setting is required by the test.</p> <h3> <i>In Hoc Signo Vinces</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x2aac10367338"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac10369458"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac09638778"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac1036f9d8"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2ad0fa98"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x1e578158"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x2b9af358"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x26189d58"> In Hoc Signo Vinces (part 9): TPC-H: TPC-H Q18, Ordered Aggregation, and Top K </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aab6ccef668"> In Hoc Signo Vinces (part 10): TPC-H: TPC-H Q9, Q17, Q20 - Predicate Games</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac1e991478"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac1c94cf78"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac7a629e08"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aabd42e3db8"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x199a6778"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aabe9f5fb48"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x1612e3a8"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x1f93c7c8"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aab521b3f48"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:07:56.580435-04:00
In Hoc Signo Vinces (part 2 of n) -- TPC-H Schema Choices
http://www.openlinksw.com:443/weblog/oerling/?date=2013-11-13#1741
2013-11-13T23:01:21Z
<p> <a href="http://www.tpc.org/tpch/" id="link-id0x2aac182bc168">TPC-H</a> gives a strict definition of the table structure and content, but leaves implementations some latitude in defining indices and data layout. A TPC-H run consists of a power test with a single user submitting queries in sequence, and of a throughput test where many concurrent users are simulated. An implementation may choose how many concurrent users are simulated, as long as this is more than a set minimum.</p> <p>Tables do not always have to be stored in the order of their primary key. A clustered index is a structure where the index tree leaves contain the whole row of data. TPC-H explicitly forbids materializing multiple copies of a table in different sort orders for speeding up different queries. That said, an implementation may pick one primary order for each table which does not have to be the primary key order.</p> <p>The present champion in core-for-core speed, Actian <a href="http://dbpedia.org/resource/Vectorwise" id="link-id0x2aabd7066a58">VectorWise</a>, organizes the main tables, <code>lineitem</code> and <code>orders</code>, in date order. The <a href="http://dbpedia.org/page/Microsoft_SQL_Server" id="link-id0x2aabd5c9aaa8">Microsoft SQL Server</a> implementations use clustered indices on <code>lineitem</code> and <code>orders</code>, where the table is stored in non-primary key order; in this case, the major ordering columns are <code>l_shipdate</code> and <code>o_orderdate</code>, respectively.</p> <p>These index schemes create locality on the date dimension.</p> <p>In the present discussion, we take a different tack: We keep <code>lineitem</code> and <code>orders</code> in primary key order, and sacrifice locality on date in favor of a fast merge join between these two tables, and for faster load and data maintenance.</p> <p>TPC-H rules allow indices on foreign keys and dates. In this implementation, we only define one on <code>o_custkey</code>; <code>lineitem</code> is only indexed on its primary key. These are the two largest tables, and the only ones that change during the benchmark. Whether indices are defined on other tables makes little or no difference, since joins between these tend to perform better by hash than by index in any case.</p> <h2>Scale and Test System</h2> <p>All the runs are done at 100 GB scale. The test machine is a dual E5-2630 (2x6 cores, 2x12 threads, 2.3 GHz) with 192 GB of 1066 MHz RAM. Two Crucial 512 GB SSDs are used both for database and staging of the files. The disks are independent file systems with no RAID. (Note that RAID would be required for an official result.) The operating system is CentOS 6.2.</p> <p>For cluster results, two machines with the above spec are used with a QDR InfiniBand interconnect.</p> <p>The Virtuoso used is a current internal development version, to be available at the end of this series or by special request. This is neither the publicly available open nor closed source version.</p> <h3> <i>In Hoc Signo Vinces</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1739" id="link-id0x233f6158"> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H</a> </li> <li> In Hoc Signo Vinces (part 2): TPC-H Schema Choices<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x22454708"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2213e628"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x239b07e8"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2f13e298"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x303a238"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x27399d58"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x25c910c8"> In Hoc Signo Vinces (part 9): TPC-H: TPC-H Q18, Ordered Aggregation, and Top K </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aabb6e74ab8"> In Hoc Signo Vinces (part 10): TPC-H: TPC-H Q9, Q17, Q20 - Predicate Games</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac1dd2fa98"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac28f74518"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac79cfa248"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aabbad88b38"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x17368908"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0xfc6bd48"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aabe89eea68"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aac37bd05b8"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aaadb262348"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:08:01.180051-04:00
In Hoc Signo Vinces (part 1 of n) -- Virtuoso meets TPC-H
http://www.openlinksw.com:443/weblog/oerling/?date=2013-11-13#1739
2013-11-13T22:56:22Z
<p> <a href="http://www.tpc.org/tpch/" id="link-id0x14305a58">TPC-H</a> is <i>the</i> data warehousing benchmark to date. I will here cover the <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id0x2aac18ee59d8">Virtuoso</a> implementation of this in detail. The primary audience is database experts. This will also be very educational for DBAs and advanced application developers: Life becomes much more predictable if one knows a good query plan from a bad one. Alongside a commentary on database science, you will also find here a guided tour of Virtuoso performance tuning and diagnostics. To follow this, it is useful to have the official TPC-H spec at hand (download links are on the far-right of <a href="http://www.tpc.org/tpch/" id="link-id0x1418dc38">this page</a>).</p> <p>By now, TPC-H is an old game and it is safe to say that pretty much any player in the analytics database domain has had a go at it, even though some have never published a result. So, the bar for new entrants is very high.</p> <p>Especially, <a href="http://dbpedia.org/page/Vectorwise" id="link-id0x8f6ba08">VectorWise</a> and <a href="http://dbpedia.org/resource/EXASolution" id="link-id0x2aac18ccdfc8">EXASolution</a> have taken performance in this workload close to the limits of the achievable. A challenger has to do everything right in order to win. One wrong move will lose the whole race.</p> <p>This presentation has many objectives:</p> <ul> <li> <p>To illustrate how Virtuoso is an excellent SQL analytics engine</p> </li> <li> <p>To provide an in-depth discussion on the science of query optimization and execution</p> </li> <li> <p>To outline avenues of future development, specifically as concerns analytics with schema-less data</p> </li> </ul> <p>In the <a href="http://www.tpc.org/tpctc/tpctc2013/" id="link-id0x2aac182d14d8">TPC TC workshop</a> at <a href="http://www.vldb.org/2013" id="link-id0x14ae6bb8">VLDB 2013</a> there was a paper <i><a href="http://www.tpc.org/tpctc/tpctc2013/slides_and_papers/005.pdf" id="link-id0xa9316e8">TPC-H Analyzed: Hidden Messages and Lessons Learned from an Influential Benchmark</a></i>, by <a href="http://homepages.cwi.nl/~boncz/" id="link-id0x127527a8">Peter Boncz</a>, <a href="http://www-db.in.tum.de/~neumann/" id="link-id0x14a12d18">Thomas Neumann</a>, and <a href="http://www.linkedin.com/in/orrierling" id="link-id0x14a7a308">myself</a> concerning what the database world has learned from this very tough exercise. Peter Boncz is the original architect of Actian VectorWise, the current champion in TPC-H performance per core. Thomas Neumann is the author of <a href="http://www-db.in.tum.de/research/projects/HyPer/" id="link-id0x14a99168">HyPer</a>, most likely the best entry in DBMS research for simultaneously supporting analytics and OLTP. Peter and Thomas are among the most renowned in database science. I am the Program Manager of the Virtuoso column store, overseeing core engineering tasks such as SQL query optimization, execution, storage, and scale out. </p> <p>In this series I will go over the Virtuoso implementation of TPC-H and will elaborate further on the points discussed in the paper. The subject is broader than any single paper can cover in detail, although there are plenty of papers only addressing one or two of the 22 queries.</p> <p>Virtuoso is mostly known for <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x955fb38">RDF</a>. Here we will cover the whole benchmark in <a href="http://dbpedia.org/resource/SQL" id="link-id0xaca8ae8">SQL</a> first, with both single-server and cluster implementations, and discussion of where these differ. A state-of-the-art SQL implementation is the necessary basis for discussing how the same can be accomplished in RDF. Comparing good RDF to bad SQL is not interesting.</p> <p>The earlier articles on the Star Schema Benchmark (SSB) (<a href="http://www.cs.umb.edu/~poneil/StarSchemaB.PDF" id="link-id0x14aef2a8">PDF</a>) -- <i><a href="http://www.openlinksw.com/weblogs/oerling/?id=1732" id="link-id0x13a8f708">Annuit Coeptis, or, Star Schema and The Cost of Freedom</a></i> and <i><a href="http://www.openlinksw.com/weblogs/oerling/?id=1733" id="link-id0x14aa6b58">E Pluribus Unum, or, Star Schema Meets Cluster</a></i> -- demonstrated how the most basic analytical database operations perform in Virtuoso. All the techniques used there are also directly applicable to TPC-H, but the latter adds a good 20 more tricks one needs to see through.</p> <p>Future installments will discuss TPC-H query by query. We conclude with a full run of <a href="http://sourceforge.net/apps/mediawiki/osdldbt/" id="link-id0x14abca48">OSDL-DBT-3</a>. DBT-3™ is an unofficial TPC-H without auditing but with the same workload.</p> <h3> <i>In Hoc Signo Vinces</i> Series</h3> <ul> <li> In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1741" id="link-id0x2aac0b114898"> In Hoc Signo Vinces (part 2): TPC-H Schema Choices</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1742" id="link-id0x2aac06d43db8"> In Hoc Signo Vinces (part 3): Benchmark Configuration Settings</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1744" id="link-id0x2aac0bf375e8"> In Hoc Signo Vinces (part 4): Bulk Load and Refresh</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1747" id="link-id0x2aac06d24268"> In Hoc Signo Vinces (part 5): The Return of SQL Federation</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1753" id="link-id0x2e2d6458"> In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1755" id="link-id0x16bbc8a8"> In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1756" id="link-id0x22fac5a8"> In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1779" id="link-id0x2aac4b7c7d98"> In Hoc Signo Vinces (part 9): TPC-H: TPC-H Q18, Ordered Aggregation, and Top K </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1789" id="link-id0x2aabb40541a8"> In Hoc Signo Vinces (part 10): TPC-H: TPC-H Q9, Q17, Q20 - Predicate Games</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1793" id="link-id0x2aac37edea48"> In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1796" id="link-id0x2aac2af3f398"> In Hoc Signo Vinces (part 12): TPC-H: Result Preview </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1798" id="link-id0x2aac30f6b918"> In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1800" id="link-id0x2aabd55e11d8"> In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1802" id="link-id0x2aac344da368"> In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1816" id="link-id0x18f4fa58"> In Hoc Signo Vinces (part 16): Introduction to Scale-Out </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1818" id="link-id0x2aabeb095408"> In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1819" id="link-id0x2aabebd63928"> In Hoc Signo Vinces (part 18): Cluster Dynamics </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1822" id="link-id0x2aac342c0d98"> In Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1824" id="link-id0x2aab517a6398"> In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1845" id="link-id0x2aaad84c2ae8"> In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 </a> </li> </ul>
2015-06-10T12:08:06.398597-04:00
E Pluribus Unum, or, Star Schema Meets Cluster
http://www.openlinksw.com:443/blog/vdb/blog/?date=2013-07-01#1735
2013-07-01T15:38:35Z
<p>In <a href="http://www.openlinksw.com/weblogs/oerling/?id=1732" id="link-id0x2aabd8ca4e38">the <i>Cost of Freedom</i> article</a>, we looked at the implications of doing standard analytics without a schema. Here we will look at scalability.</p> <p>One could claim that a 30GB scale is trivial. Here, we multiply the scale by 10 and move from one to two servers, going from shared-memory multicore to distributed-memory scale-out with partitioned data, i.e. each machine holds a distinct fragment of the database.</p> <p>Again, we run the same workload in SQL and in SPARQL. The RDB schema is the same as before, so for SQL tables only primary keys are declared, there are no indices or special declarations about data placement, and the table partitioning is on the first part of the primary key. The RDF data for the SPARQL runs is quads with the default index scheme, partitioned on subject or object, whichever is first in key order. The test system is two machines, each with dual Xeon E5-2630 and 192GB RAM. The queries are from warm cache.</p> <p>There are 1.79 billion line-order rows, giving a total of 32.5 billion RDF triples.</p> <table id="ResultsTable" title="Comparative Results, Virtuoso SQL vs Virtuoso SPARQL" align="center" cellspacing="5px" cellpadding="2px" width="350px"> <caption>Comparative Results, 300GB, in seconds<br />Virtuoso Cluster, SQL <i>Vs.</i> SPARQL</caption> <tr> <th align="center">Query</th> <th align="center">Virtuoso <br />SQL</th> <th align="center">Virtuoso <br />SPARQL</th> <th>SPARQL-to-SQL <br />ratio</th> </tr> <tr> <th align="center">Q1</th> <td align="center">2.285</td> <td align="center">7.767</td> <td align="center">3.4x</td> </tr> <tr> <th align="center">Q2</th> <td align="center">1.530</td> <td align="center">3.535</td> <td align="center">2.3x</td> </tr> <tr> <th align="center">Q3</th> <td align="center">1.237</td> <td align="center">1.457</td> <td align="center">1.2x</td> </tr> <tr> <th align="center">Q4</th> <td align="center">3.459</td> <td align="center">6.978</td> <td align="center">2.0x</td> </tr> <tr> <th align="center">Q5</th> <td align="center">3.065</td> <td align="center">8.710</td> <td align="center">2.8x</td> </tr> <tr> <th align="center">Q6</th> <td align="center">2.901</td> <td align="center">8.454</td> <td align="center">2.9x</td> </tr> <tr> <th align="center">Q7</th> <td align="center">5.733</td> <td align="center">15.939</td> <td align="center">2.8x</td> </tr> <tr> <th align="center">Q8</th> <td align="center">2.267</td> <td align="center">6.759</td> <td align="center">3.0x</td> </tr> <tr> <th align="center">Q9</th> <td align="center">1.773</td> <td align="center">4.217</td> <td align="center">2.4x</td> </tr> <tr> <th align="center">Q10</th> <td align="center">1.440</td> <td align="center">4.342</td> <td align="center">3.0x</td> </tr> <tr> <th align="center">Q11</th> <td align="center">5.031</td> <td align="center">12.608</td> <td align="center">2.5x</td> </tr> <tr> <th align="center">Q12</th> <td align="center">4.464</td> <td align="center">15.497</td> <td align="center">3.5x</td> </tr> <tr> <th align="center">Q13</th> <td align="center">2.807</td> <td align="center">4.467</td> <td align="center">1.6x</td> </tr> <tr> <th align="center">Total</th> <td align="center">37.992</td> <td align="center">100.730</td> <td align="center">2.7x</td> </tr> </table> <p>The SQL run takes 38s, and the SPARQL run takes 101s. Comparing to the single server, we see a little better than linear scaling, i.e., double the gear is a little over 2x faster. In the present case, 10x the data takes a little under 5x the time.</p> <p>This is due to slightly better load balancing. The single server splits the workload into chunks based on a run-time approximation; the cluster stores the partitions separately and uses this to determine the parallelism. In the latter case, the chunks are of more equal size.</p> <p>The SPARQL penalty is here 2.7x, essentially the same as in the single server case.</p> <p>It is no secret that a star schema is "embarrassingly parallel." In other words, when there is one big table (fact table) that references many smaller tables (dimension tables), and query conditions are expressed on properties of the dimension tables, the correct query plan nearly always consists of putting the interesting foreign key values into hash tables, and then scanning the fact table from beginning to end and picking the rows where the values in the foreign key columns are found in the hash tables. This is called a <i>selective hash join</i> or <i>invisible hash join</i> if the hash join operation is merged in the table scan itself. <a href="http://cs-www.cs.yale.edu/homes/dna/" id="link-id0x2aabcc04ee68">Daniel Abadi</a>’s <a href="http://dspace.mit.edu/handle/1721.1/43043" id="link-id0xf3953e8">well known thesis</a> explains this matter. In the case of a cluster, supposing all tables are partitioned, identical hash tables are made on all participating servers, and after this each server gets to scan its fraction of the fact table independently of any other. This nearly always works because the dimension tables are typically orders of magnitude smaller than the fact table.</p> <p>The complexity of the queries is close to linear to the data size. The factor that makes this deviate from linear is the fact that as hash tables get larger, they will miss the CPU cache more frequently; hence they get slower to probe. We here assume that a bigger fact table means bigger dimension tables. This is often the case, i.e., the more sales records there are, the larger the number of distinct customers or distinct items in the catalogue is likely to be. This is not always so, though, as the number of days in the history does not scale in the same way.</p> <p>Any decent analytics oriented RDBMS with scale out will give near-linear performance with a star schema, at least up to the point where the hash tables can no longer be replicated on all servers, or the hash join must do multiple passes over the fact table.</p> <p>Doing the same with schema-less data is harder, even though the principle is exactly the same. The difficulty lies again in detecting that a scan filtered by selective hash joins will give the most locality in access pattern, visiting each cache-line-worth of data once at most and doing so almost always in sequential order. This triggers memory prefetching on any modern CPU and significantly reduces memory latency.</p> <p>As the scale grows there are some details of query plan that become significant. For example, in Q7 of SSB, there is a report on sales between Asian customers and Asian suppliers, country by country. In SQL:</p> <blockquote> <code><pre>SELECT d_year, c_nation, s_nation, SUM (...), ... FROM lineorder, customer, supplier, dwdate WHERE lo_custkey = c_custkey AND lo_suppkey = s_suppkey AND c_region = 'ASIA' AND s_region = 'ASIA' ... </pre> </code> </blockquote> <p>In this case, one builds a hash table from <code>c_custkey</code> to <code>c_nation</code> and from <code>s_suppkey</code> to <code>s_nation</code>, including only suppliers and customers where the <code>region = 'ASIA'</code>.</p> <p>In SPARQL, the situation is the same, except that we say:</p> <blockquote> <code><pre>SELECT * WHERE { li rdfh:lo_custkey ?lo_custkey ; rdfh:lo_suppkey ?lo_suppkey . ?lo_custkey rdfh:c_nation ?c_nation ; rdfh:c_region "ASIA" . ?lo_suppkey rdfh:s_nation ?s_nation ; rdfh:s_region "ASIA" . }</pre> </code> </blockquote> <p>This is an equivalent expression, or near enough. We note that now it makes sense to build the hash table from the join of two patterns, i.e., the <code>rdfh:s_nation</code> and <code>rdfh:s_nregion</code>, which together are the same as the <code>supplier</code> table in the SQL variant. But now, a supplier might have more than one <code>rdfh:s_nation</code> triple; hence the hash table is no longer known to be <i>a priori</i> unique -- i.e., its key is the URI of a supplier, but it is only known that suppliers usually have one nation; it is not known that they never have two.</p> <p>However, if we build the hash table from only one pattern, i.e., <code>{ ?s_suppkey rdfh:s_region "ASIA" }</code>, and the graph is specified, then we know that the subject will be unique, as stating that X has region "ASIA" twice has no effect beyond stating it once. However, if we do this, then there needs to be another hash table built for mapping the supplier to its <code>s_nation</code>. There is no way to know that this is unique. Making two hash tables instead of one has been seen to slow down the query by a factor of two at scale 300GB whereas the effect is hardly noticeable at 30GB.</p> <p>Thus recognizing these special cases with SPARQL is crucial if one is to come anywhere near the performance the SQL world attains just by following the schema. What used to be basic becomes trickier. It is true that the same or similar tricks are also needed in pure SQL workloads, but then not within the SSBM queries. What is a star in SQL is quite often a snowflake in SPARQL. A star schema has a single table-per-dimension, e.g., <code>customer</code>, whereas a snowflake has more structure, e.g., <code>customer</code>, the customer’s <code>country</code> and the country’s <code>region</code>, all in different tables.</p> <p>The message to the SPARQL public is that now, for the first time, if there is natural parallelism and locality in the data, you know that the database will exploit this correctly and derive the same benefits from this as any SQL-only system would. Of course, there have been and are parallel RDF databases, including previous versions of Virtuoso, which do distributed index-based operations with various levels of concurrency and distributed coordination, but experience shows that these query strategies are not very good for queries that touch a large fraction (over 3%) of the database.</p> <p>For basic data warehouse workloads, whether in SQL or SPARQL, Virtuoso offers linear scaling where clusters pay off from the start. While a handful of relational column stores have offered such capabilities for SQL for some time, now the same is also available for schema-less data: Entirely declarative querying; no explicit data partitioning; no schema restrictions; no map-reduce programs or the like.</p> <p>The cluster evolution path is clear: as with single server, we strive for more speed on a broader range of operations. The star and snowflake functionality discussed here is the core piece for any analytics, so if this is not right, the rest is also compromised. The other side of cluster is operational, i.e., flexibility of deployment -- for example, flexibly resizing cloud-based databases.</p> <p>This is the qualitative jump. Incremental performance gains will follow for both SQL and SPARQL.</p> <p>Stars and snowflakes are common but the world does not end with these. As we will see in subsequent articles, there is a whole world of graphs, e.g., social media analytics, as well as more complex relational schemas.</p>
2015-08-21T14:19:48.733946-04:00
Annuit Coeptis, or, Star Schema and The Cost of Freedom
http://www.openlinksw.com:443/blog/vdb/blog/?date=2013-07-01#1734
2013-07-01T15:38:22Z
<p>Around 2009, we set out to find out whether one can do analytics without a schema. The rationale is of course that data without query is not useful and that large data often has variation of schema, whether due to multiple sources, multiple versions of schema at different points in history, changes in applications and business model, or any number of other factors.</p> <p>In order to meaningfully answer this question, one has to have a top notch analytics engine. Comparing "schema" and "no schema" on anything except the state-of-the-art in analytics database is not interesting. So we made Virtuoso 7 and its Column Store Module, and implemented the features common in dedicated analytics databases, plus suitable SPARQL adaptations of same.</p> <p>The present beach head is the Star Schema Benchmark (SSB) (<a href="http://www.openlinksw.com/www.cs.umb.edu/~poneil/StarSchemaB.PDF" id="link-id0x2aabcd1f4f08">PDF</a>), which represents the core of most data warehouse workloads (i.e., big scans, selective hash joins, and aggregation). These same patterns are also found at the core of <a href="http://www.tpc.org/tpch/" id="link-id0x2aabce8838c8">TPC-H</a> and the new <a href="http://www.tpc.org/tpcds/" id="link-id0x2aabce202ca8">TPC-DS</a>.</p> <p>At present, the cost of having no schema is a 2.7x increase in run time, as evidenced by SSB. Virtuoso SQL does the run in 8.4s, <a href="http://dbpedia.org/page/MonetDB" id="link-id0xe89eba8">MonetDB</a> SQL in 17s, Virtuoso SPARQL in 22.5s. Virtuoso outperforms column store pioneer MonetDB by a fair margin. MonetDB is probably the fastest open source column store, although there exists faster ones in closed source.</p> <p>SPARQL in Virtuoso comes close behind SQL in MonetDB, only a factor of 1.3. <a href="http://dbpedia.org/resource/MySQL" id="link-id0x2aabd81ee648">MySQL</a> with <a href="http://dbpedia.org/resource/InnoDB" id="link-id0x161e22f8">InnoDB</a>, which is not an analytical database, does the run in 2391s.</p> <p>Initially, we were aiming at a slowdown factor of 2 when comparing SPARQL to SQL on Virtuoso. Of course, this is only worth something if the SQL performance is generally on a level with relational column stores. These goals have now been substantially attained.</p> <p>This is a major beachhead for RDF as a branch of graph database technology, and for Virtuoso as a product. For the first time one can run a real database workload based on SPARQL at a speed that is comparable with SQL, and without any compromises on the schema flexibility that is the principal reason to use an RDF-based graph model in the first place. No SPARQL-to-SQL mapping, property tables, or such.</p> <h2>Execution and Query Optimization</h2> <p>The technical accomplishment is divided in two parts. For query execution, there is a compressed column store with vectored execution and a good implementation of query parallelization, hash join, and aggregation. With a schema, there is a multicolumn table. and with RDF, there is a quads table. Both are compressed based on the characteristics of actual data. The SQL execution consists of a scan of the fact table, taking a few columns of this and applying one or more hash joins to column values, where the hash joins are usually selective. This is the same in SPARQL, except that instead of getting another column of the table there is a self-join. The self-join has a fairly dense access pattern and is in the same order as the previous one, thus it is not very expensive. This constitutes the difference.</p> <p>While in SQL, the plan is obvious, especially since there are no indices on the fact table; however, getting the right plan with SPARQL is quite difficult. There are up to 12 triple-patterns in a query, leading to 12! (twelve-factorial, i.e., 12*11*...*2*1 = 479,001,600) possible join orders, multiplied by index and hash based variants of each join, where the hash-based variants further multiply the space by considering different combinations of patterns on the build side. Experiments confirm that the same plan is best for both SQL and SPARQL but getting this as the outcome of query optimization is far from self-evident. This requires a very precise cost model that correctly takes ordering of intermediate results and density of hits in index lookups into account, as well as the variability in hash join performance when the hash table size varies, i.e., CPU cache effects. All this modeling is of course also valid for SQL, but is not really required there because a much coarser model will also deliver the right plan, as there is much less choice. Further, making the right plan must be fast. Now, the longest query optimization time is 240ms for an execution time of 3.7s (Q12 at 30G in SPARQL). This is pretty good.</p> <p>Using hash join in SPARQL has anyway been problematical because knowing when to use one requires high confidence in the cost model. If one ends up building a large hash table that is used only a few times, there is a steep penalty. Index-based plans, especially since RDF data tends to be indexed in both S-to-O and O-to-S directions, do not have bad worst-cases, but then a good hash-based plan is easily 5x faster than an index-based one.</p> <p>So, for SPARQL the results are game changing. Finally up-to-date database. For SQL, Virtuoso 7 performs like an analytics column store is supposed to, but then that is what it is. For the SQL space, it is interesting that this is also open source, so Virtuoso may well be the best performing open source SQL analytics engine out there. We will later see how the comparison with other SQL column stores goes.</p> <p>In summary, the central core of the Virtuoso 7 agenda has been accomplished. Incremental progress will continue around addressing more complex benchmarks like TPC-H and TPC-DS, both in SQL and in SPARQL translation, plus full-scale <a href="http://www.openstreetmap.org/" id="link-id0x2aabbaa86d48">Open Streetmap</a> in both SQL and SPARQL, and of course the benchmarks being developed in <a href="http://www.ldbc.eu/" id="link-id0x2aabdcef94e8">LDBC</a>.</p> <h2>Experiments</h2> <p>The scale of the dataset is 30GB, with 187M line-order rows. This comes to 3.2Gt (3.2 billion triples). The runs are from warm cache on all systems. The test system for all runs is a dual Xeon E5-2630 with 192GB RAM.</p> <p>The scripts for duplicating the experiment on the Virtuoso 7 Open Source cut will be published later on this blog, when the open source cut incorporates the query optimization improvements discussed herein.</p> <p>In the table below, the times are elapsed real times in milliseconds, with one query at a time, all from warm cache (i.e. the second run of the query set is reported and we make sure that all databases are running from memory). The MySQL is configured with InnoDB and 40GB of buffer pool, which should be enough for the 30GB dataset. The SQL versions do not declare any explicit indices, but do declare primary keys. The SPARQL version is with the default Virtuoso index scheme, but the query plans end up scanning each predicate in order of S, except for the build phases of hash-joins where an index from O-to-S is occasionally used.</p> <table id="SSBResultsTable" title="Comparative Results" align="center" cellspacing="5px" cellpadding="2px" width="450px"> <caption>Comparative Results, 30GB dataset<br />Virtuoso SQL <i>Vs.</i> Virtuoso SPARQL <i>Vs.</i> MonetDB SQL <i>Vs.</i> MySQL SQL</caption> <tr> <th align="center">Query</th> <th align="center">Virtuoso <br /> SQL</th> <th align="center">Virtuoso <br /> SPARQL</th> <th align="center">MonetDB <br /> SQL</th> <th align="center">MySQL <br /> SQL</th> </tr> <tr> <th align="center">Q1</th> <td align="center" bgcolor="#33CC33"> <b><i>413</i> </b></td> <td align="center" bgcolor="#99FF66">1101</td> <td align="center">1659</td> <td align="center">82477</td> </tr> <tr> <th align="center">Q2</th> <td align="center" bgcolor="#33CC33"> <b><i>282</i> </b></td> <td align="center" bgcolor="#99FF66">416</td> <td align="center">500</td> <td align="center">74436</td> </tr> <tr> <th align="center">Q3</th> <td align="center" bgcolor="#33CC33"> <b><i>253</i> </b></td> <td align="center" bgcolor="#99FF66">295</td> <td align="center">494</td> <td align="center">75411</td> </tr> <tr> <th align="center">Q4</th> <td align="center" bgcolor="#33CC33"> <b><i>828</i> </b></td> <td align="center">2484</td> <td align="center" bgcolor="#99FF66">958</td> <td align="center">226604</td> </tr> <tr> <th align="center">Q5</th> <td align="center" bgcolor="#99FF66">837</td> <td align="center">1915</td> <td align="center" bgcolor="#33CC33"> <b><i>648</i> </b></td> <td align="center">222782</td> </tr> <tr> <th align="center">Q6</th> <td align="center" bgcolor="#33CC33"> <b><i>419</i> </b></td> <td align="center">1813</td> <td align="center" bgcolor="#99FF66">541</td> <td align="center">219656</td> </tr> <tr> <th align="center">Q7</th> <td align="center" bgcolor="#33CC33"> <b><i>1062</i> </b></td> <td align="center" bgcolor="#99FF66">2330</td> <td align="center">5658</td> <td align="center">237730</td> </tr> <tr> <th align="center">Q8</th> <td align="center" bgcolor="#99FF66">617</td> <td align="center">2182</td> <td align="center" bgcolor="#33CC33"> <b><i>521</i> </b></td> <td align="center">194918</td> </tr> <tr> <th align="center">Q9</th> <td align="center" bgcolor="#99FF66">547</td> <td align="center">1290</td> <td align="center" bgcolor="#33CC33"> <b><i>381</i> </b></td> <td align="center">186112</td> </tr> <tr> <th align="center">Q10</th> <td align="center" bgcolor="#99FF66">499</td> <td align="center">639</td> <td align="center" bgcolor="#33CC33"> <b><i>370</i> </b></td> <td align="center">186123</td> </tr> <tr> <th align="center">Q11</th> <td align="center" bgcolor="#33CC33"> <b><i>1132</i> </b></td> <td align="center" bgcolor="#99FF66">2142</td> <td align="center">2760</td> <td align="center">241045</td> </tr> <tr> <th align="center">Q12</th> <td align="center" bgcolor="#33CC33"> <b><i>863</i> </b></td> <td align="center">3770</td> <td align="center" bgcolor="#99FF66">2127</td> <td align="center">241439</td> </tr> <tr> <th align="center">Q13</th> <td align="center" bgcolor="#33CC33"> <b><i>653</i> </b></td> <td align="center">1612</td> <td align="center" bgcolor="#99FF66">1005</td> <td align="center">202817</td> </tr> </table> <p>One could argue that comparing against MySQL is unjustified, as MySQL is certainly not optimized for this workload, e.g., it does not do hash joins and does not parallelize queries. On the other hand, the previous work on SSBM for SPARQL, <i><a href="http://eswc-conferences.org/sites/default/files/papers2013/kaempgen.pdf" id="link-id0x2aabcddc7bd8">No Size Fits All</a></i> by <a href="http://www.aifb.kit.edu/web/Benedikt_Kaempgen/en" id="link-id0x2aabcdee77d8">Benedikt Kaempgen</a> and <a href="http://www.aifb.kit.edu/web/Andreas_Harth/en" id="link-id0x2aabdb8285b8">Andreas Harth</a>, published at the <a href="http://2013.eswc-conferences.org/" id="link-id0x2aabcc627198">2013 ESWC</a>, did make the comparison between MySQL and Virtuoso 6; thus we think it informative to include MySQL. To summarize <i>No Size Fits All,</i> SPARQL in Virtuoso 6 lost by a factor of 12 against MySQL, but in the present case, Virtuoso 7 SPARQL wins by a factor of 106 against MySQL.</p> <p>The ESWC paper used a scale of 1G, while the present test uses a scale of 30G. One should remember that the MySQL times are single-threaded, and all other times are multi-threaded. The test system has 12 cores and 24 threads, so running at full platform utilization is at best 16x faster than single threaded. Running SPARQL single-threaded instead of 24 threads-per-query gives a total time of 175s, still over 10x better than MySQL. Compared to the multi-threaded time of 22.5s, the parallelism yields an average acceleration of 7.7x. Running with SPARQL with full threading but no hash-join gives a time of 80s, 3.5x worse than with hash-join. We note that SSB is a very hash-join intensive workload.</p> <p>MonetDB is the more relevant comparison, as it does use the full CPU, with load peaks up to the theoretical 2400% (12 dual-threaded cores) and is the platform on which a lot of the science of the hash join was refined. MonetDB does relatively best with queries that can start by a very selective join (e.g., Q9) where it outperforms Virtuoso. This is probably due to more even splitting of the work among threads and to not having to deal with data compression. Virtuoso wins the most on queries that select a large fraction of the fact table, where MonetDB is penalized due to its policy of full materialization of intermediate results. Joins without any selection (e.g., adding up the <code>lo_extendedprice</code>, and grouping by the <code>d_year</code> of <code>lo_orderdate</code>) show MonetDB at its worst. Such queries do not occur in SSB though, so SSB is a relatively MonetDB-friendly benchmark.</p> <h2>Conclusions</h2> <p>The back of the problem is broken. Big query without schema can be done. One could of course see this coming by experimenting with explicit plans. But nobody out there can manually optimize a query plan. Thus the final step consisted of having a good-enough cost model and a smart-enough search order to get the right plan fast. This will be in the next update of Virtuoso Open Source. At that time, we will publish the full queries and configuration files.</p> <p>This is the breakthrough for RDF analytics. Incremental progress will follow, with more tricks being incorporated, like the ones known to be needed by TPC-H.</p>
2013-07-01T13:55:17.000001-04:00
E Pluribus Unum, or, Star Schema Meets Cluster
http://www.openlinksw.com:443/weblog/oerling/?date=2013-07-01#1733
2013-07-01T15:38:05Z
<p>In <a href="http://www.openlinksw.com/weblogs/oerling/?id=1732" id="link-id0x2aabd8ca4e38">the <i>Cost of Freedom</i> article</a>, we looked at the implications of doing standard analytics without a schema. Here we will look at scalability.</p> <p>One could claim that a 30GB scale is trivial. Here, we multiply the scale by 10 and move from one to two servers, going from shared-memory multicore to distributed-memory scale-out with partitioned data, i.e. each machine holds a distinct fragment of the database.</p> <p>Again, we run the same workload in SQL and in SPARQL. The RDB schema is the same as before, so for SQL tables only primary keys are declared, there are no indices or special declarations about data placement, and the table partitioning is on the first part of the primary key. The RDF data for the SPARQL runs is quads with the default index scheme, partitioned on subject or object, whichever is first in key order. The test system is two machines, each with dual Xeon E5-2630 and 192GB RAM. The queries are from warm cache.</p> <p>There are 1.79 billion line-order rows, giving a total of 32.5 billion RDF triples.</p> <table id="ResultsTable" title="Comparative Results, Virtuoso SQL vs Virtuoso SPARQL" align="center" cellspacing="5px" cellpadding="2px" width="350px"> <caption>Comparative Results, 300GB, in seconds<br />Virtuoso Cluster, SQL <i>Vs.</i> SPARQL</caption> <tr> <th align="center">Query</th> <th align="center">Virtuoso <br />SQL</th> <th align="center">Virtuoso <br />SPARQL</th> <th>SPARQL-to-SQL <br />ratio</th> </tr> <tr> <th align="center">Q1</th> <td align="center">2.285</td> <td align="center">7.767</td> <td align="center">3.4x</td> </tr> <tr> <th align="center">Q2</th> <td align="center">1.530</td> <td align="center">3.535</td> <td align="center">2.3x</td> </tr> <tr> <th align="center">Q3</th> <td align="center">1.237</td> <td align="center">1.457</td> <td align="center">1.2x</td> </tr> <tr> <th align="center">Q4</th> <td align="center">3.459</td> <td align="center">6.978</td> <td align="center">2.0x</td> </tr> <tr> <th align="center">Q5</th> <td align="center">3.065</td> <td align="center">8.710</td> <td align="center">2.8x</td> </tr> <tr> <th align="center">Q6</th> <td align="center">2.901</td> <td align="center">8.454</td> <td align="center">2.9x</td> </tr> <tr> <th align="center">Q7</th> <td align="center">5.733</td> <td align="center">15.939</td> <td align="center">2.8x</td> </tr> <tr> <th align="center">Q8</th> <td align="center">2.267</td> <td align="center">6.759</td> <td align="center">3.0x</td> </tr> <tr> <th align="center">Q9</th> <td align="center">1.773</td> <td align="center">4.217</td> <td align="center">2.4x</td> </tr> <tr> <th align="center">Q10</th> <td align="center">1.440</td> <td align="center">4.342</td> <td align="center">3.0x</td> </tr> <tr> <th align="center">Q11</th> <td align="center">5.031</td> <td align="center">12.608</td> <td align="center">2.5x</td> </tr> <tr> <th align="center">Q12</th> <td align="center">4.464</td> <td align="center">15.497</td> <td align="center">3.5x</td> </tr> <tr> <th align="center">Q13</th> <td align="center">2.807</td> <td align="center">4.467</td> <td align="center">1.6x</td> </tr> <tr> <th align="center">Total</th> <td align="center">37.992</td> <td align="center">100.730</td> <td align="center">2.7x</td> </tr> </table> <p>The SQL run takes 38s, and the SPARQL run takes 101s. Comparing to the single server, we see a little better than linear scaling, i.e., double the gear is a little over 2x faster. In the present case, 10x the data takes a little under 5x the time.</p> <p>This is due to slightly better load balancing. The single server splits the workload into chunks based on a run-time approximation; the cluster stores the partitions separately and uses this to determine the parallelism. In the latter case, the chunks are of more equal size.</p> <p>The SPARQL penalty is here 2.7x, essentially the same as in the single server case.</p> <p>It is no secret that a star schema is "embarrassingly parallel." In other words, when there is one big table (fact table) that references many smaller tables (dimension tables), and query conditions are expressed on properties of the dimension tables, the correct query plan nearly always consists of putting the interesting foreign key values into hash tables, and then scanning the fact table from beginning to end and picking the rows where the values in the foreign key columns are found in the hash tables. This is called a <i>selective hash join</i> or <i>invisible hash join</i> if the hash join operation is merged in the table scan itself. <a href="http://cs-www.cs.yale.edu/homes/dna/" id="link-id0x2aabcc04ee68">Daniel Abadi</a>’s <a href="http://dspace.mit.edu/handle/1721.1/43043" id="link-id0xf3953e8">well known thesis</a> explains this matter. In the case of a cluster, supposing all tables are partitioned, identical hash tables are made on all participating servers, and after this each server gets to scan its fraction of the fact table independently of any other. This nearly always works because the dimension tables are typically orders of magnitude smaller than the fact table.</p> <p>The complexity of the queries is close to linear to the data size. The factor that makes this deviate from linear is the fact that as hash tables get larger, they will miss the CPU cache more frequently; hence they get slower to probe. We here assume that a bigger fact table means bigger dimension tables. This is often the case, i.e., the more sales records there are, the larger the number of distinct customers or distinct items in the catalogue is likely to be. This is not always so, though, as the number of days in the history does not scale in the same way.</p> <p>Any decent analytics oriented RDBMS with scale out will give near-linear performance with a star schema, at least up to the point where the hash tables can no longer be replicated on all servers, or the hash join must do multiple passes over the fact table.</p> <p>Doing the same with schema-less data is harder, even though the principle is exactly the same. The difficulty lies again in detecting that a scan filtered by selective hash joins will give the most locality in access pattern, visiting each cache-line-worth of data once at most and doing so almost always in sequential order. This triggers memory prefetching on any modern CPU and significantly reduces memory latency.</p> <p>As the scale grows there are some details of query plan that become significant. For example, in Q7 of SSB, there is a report on sales between Asian customers and Asian suppliers, country by country. In SQL:</p> <blockquote> <code><pre>SELECT d_year, c_nation, s_nation, SUM (...), ... FROM lineorder, customer, supplier, dwdate WHERE lo_custkey = c_custkey AND lo_suppkey = s_suppkey AND c_region = 'ASIA' AND s_region = 'ASIA' ... </pre> </code> </blockquote> <p>In this case, one builds a hash table from <code>c_custkey</code> to <code>c_nation</code> and from <code>s_suppkey</code> to <code>s_nation</code>, including only suppliers and customers where the <code>region = 'ASIA'</code>.</p> <p>In SPARQL, the situation is the same, except that we say:</p> <blockquote> <code><pre>SELECT * WHERE { li rdfh:lo_custkey ?lo_custkey ; rdfh:lo_suppkey ?lo_suppkey . ?lo_custkey rdfh:c_nation ?c_nation ; rdfh:c_region "ASIA" . ?lo_suppkey rdfh:s_nation ?s_nation ; rdfh:s_region "ASIA" . }</pre> </code> </blockquote> <p>This is an equivalent expression, or near enough. We note that now it makes sense to build the hash table from the join of two patterns, i.e., the <code>rdfh:s_nation</code> and <code>rdfh:s_nregion</code>, which together are the same as the <code>supplier</code> table in the SQL variant. But now, a supplier might have more than one <code>rdfh:s_nation</code> triple; hence the hash table is no longer known to be <i>a priori</i> unique -- i.e., its key is the URI of a supplier, but it is only known that suppliers usually have one nation; it is not known that they never have two.</p> <p>However, if we build the hash table from only one pattern, i.e., <code>{ ?s_suppkey rdfh:s_region "ASIA" }</code>, and the graph is specified, then we know that the subject will be unique, as stating that X has region "ASIA" twice has no effect beyond stating it once. However, if we do this, then there needs to be another hash table built for mapping the supplier to its <code>s_nation</code>. There is no way to know that this is unique. Making two hash tables instead of one has been seen to slow down the query by a factor of two at scale 300GB whereas the effect is hardly noticeable at 30GB.</p> <p>Thus recognizing these special cases with SPARQL is crucial if one is to come anywhere near the performance the SQL world attains just by following the schema. What used to be basic becomes trickier. It is true that the same or similar tricks are also needed in pure SQL workloads, but then not within the SSBM queries. What is a star in SQL is quite often a snowflake in SPARQL. A star schema has a single table-per-dimension, e.g., <code>customer</code>, whereas a snowflake has more structure, e.g., <code>customer</code>, the customer’s <code>country</code> and the country’s <code>region</code>, all in different tables.</p> <p>The message to the SPARQL public is that now, for the first time, if there is natural parallelism and locality in the data, you know that the database will exploit this correctly and derive the same benefits from this as any SQL-only system would. Of course, there have been and are parallel RDF databases, including previous versions of Virtuoso, which do distributed index-based operations with various levels of concurrency and distributed coordination, but experience shows that these query strategies are not very good for queries that touch a large fraction (over 3%) of the database.</p> <p>For basic data warehouse workloads, whether in SQL or SPARQL, Virtuoso offers linear scaling where clusters pay off from the start. While a handful of relational column stores have offered such capabilities for SQL for some time, now the same is also available for schema-less data: Entirely declarative querying; no explicit data partitioning; no schema restrictions; no map-reduce programs or the like.</p> <p>The cluster evolution path is clear: as with single server, we strive for more speed on a broader range of operations. The star and snowflake functionality discussed here is the core piece for any analytics, so if this is not right, the rest is also compromised. The other side of cluster is operational, i.e., flexibility of deployment -- for example, flexibly resizing cloud-based databases.</p> <p>This is the qualitative jump. Incremental performance gains will follow for both SQL and SPARQL.</p> <p>Stars and snowflakes are common but the world does not end with these. As we will see in subsequent articles, there is a whole world of graphs, e.g., social media analytics, as well as more complex relational schemas.</p>
2013-07-01T11:38:05-04:00
Annuit Coeptis, or, Star Schema and The Cost of Freedom
http://www.openlinksw.com:443/weblog/oerling/?date=2013-06-28#1732
2013-06-28T14:59:22Z
<p>Around 2009, we set out to find out whether one can do analytics without a schema. The rationale is of course that data without query is not useful and that large data often has variation of schema, whether due to multiple sources, multiple versions of schema at different points in history, changes in applications and business model, or any number of other factors.</p> <p>In order to meaningfully answer this question, one has to have a top notch analytics engine. Comparing "schema" and "no schema" on anything except the state-of-the-art in analytics database is not interesting. So we made Virtuoso 7 and its Column Store Module, and implemented the features common in dedicated analytics databases, plus suitable SPARQL adaptations of same.</p> <p>The present beach head is the Star Schema Benchmark (SSB) (<a href="http://www.openlinksw.com/www.cs.umb.edu/~poneil/StarSchemaB.PDF" id="link-id0x2aabcd1f4f08">PDF</a>), which represents the core of most data warehouse workloads (i.e., big scans, selective hash joins, and aggregation). These same patterns are also found at the core of <a href="http://www.tpc.org/tpch/" id="link-id0x2aabce8838c8">TPC-H</a> and the new <a href="http://www.tpc.org/tpcds/" id="link-id0x2aabce202ca8">TPC-DS</a>.</p> <p>At present, the cost of having no schema is a 2.7x increase in run time, as evidenced by SSB. Virtuoso SQL does the run in 8.4s, <a href="http://dbpedia.org/page/MonetDB" id="link-id0xe89eba8">MonetDB</a> SQL in 17s, Virtuoso SPARQL in 22.5s. Virtuoso outperforms column store pioneer MonetDB by a fair margin. MonetDB is probably the fastest open source column store, although there exists faster ones in closed source.</p> <p>SPARQL in Virtuoso comes close behind SQL in MonetDB, only a factor of 1.3. <a href="http://dbpedia.org/resource/MySQL" id="link-id0x2aabd81ee648">MySQL</a> with <a href="http://dbpedia.org/resource/InnoDB" id="link-id0x161e22f8">InnoDB</a>, which is not an analytical database, does the run in 2391s.</p> <p>Initially, we were aiming at a slowdown factor of 2 when comparing SPARQL to SQL on Virtuoso. Of course, this is only worth something if the SQL performance is generally on a level with relational column stores. These goals have now been substantially attained.</p> <p>This is a major beachhead for RDF as a branch of graph database technology, and for Virtuoso as a product. For the first time one can run a real database workload based on SPARQL at a speed that is comparable with SQL, and without any compromises on the schema flexibility that is the principal reason to use an RDF-based graph model in the first place. No SPARQL-to-SQL mapping, property tables, or such.</p> <h2>Execution and Query Optimization</h2> <p>The technical accomplishment is divided in two parts. For query execution, there is a compressed column store with vectored execution and a good implementation of query parallelization, hash join, and aggregation. With a schema, there is a multicolumn table. and with RDF, there is a quads table. Both are compressed based on the characteristics of actual data. The SQL execution consists of a scan of the fact table, taking a few columns of this and applying one or more hash joins to column values, where the hash joins are usually selective. This is the same in SPARQL, except that instead of getting another column of the table there is a self-join. The self-join has a fairly dense access pattern and is in the same order as the previous one, thus it is not very expensive. This constitutes the difference.</p> <p>While in SQL, the plan is obvious, especially since there are no indices on the fact table; however, getting the right plan with SPARQL is quite difficult. There are up to 12 triple-patterns in a query, leading to 12! (twelve-factorial, i.e., 12*11*...*2*1 = 479,001,600) possible join orders, multiplied by index and hash based variants of each join, where the hash-based variants further multiply the space by considering different combinations of patterns on the build side. Experiments confirm that the same plan is best for both SQL and SPARQL but getting this as the outcome of query optimization is far from self-evident. This requires a very precise cost model that correctly takes ordering of intermediate results and density of hits in index lookups into account, as well as the variability in hash join performance when the hash table size varies, i.e., CPU cache effects. All this modeling is of course also valid for SQL, but is not really required there because a much coarser model will also deliver the right plan, as there is much less choice. Further, making the right plan must be fast. Now, the longest query optimization time is 240ms for an execution time of 3.7s (Q12 at 30G in SPARQL). This is pretty good.</p> <p>Using hash join in SPARQL has anyway been problematical because knowing when to use one requires high confidence in the cost model. If one ends up building a large hash table that is used only a few times, there is a steep penalty. Index-based plans, especially since RDF data tends to be indexed in both S-to-O and O-to-S directions, do not have bad worst-cases, but then a good hash-based plan is easily 5x faster than an index-based one.</p> <p>So, for SPARQL the results are game changing. Finally up-to-date database. For SQL, Virtuoso 7 performs like an analytics column store is supposed to, but then that is what it is. For the SQL space, it is interesting that this is also open source, so Virtuoso may well be the best performing open source SQL analytics engine out there. We will later see how the comparison with other SQL column stores goes.</p> <p>In summary, the central core of the Virtuoso 7 agenda has been accomplished. Incremental progress will continue around addressing more complex benchmarks like TPC-H and TPC-DS, both in SQL and in SPARQL translation, plus full-scale <a href="http://www.openstreetmap.org/" id="link-id0x2aabbaa86d48">Open Streetmap</a> in both SQL and SPARQL, and of course the benchmarks being developed in <a href="http://www.ldbc.eu/" id="link-id0x2aabdcef94e8">LDBC</a>.</p> <h2>Experiments</h2> <p>The scale of the dataset is 30GB, with 187M line-order rows. This comes to 3.2Gt (3.2 billion triples). The runs are from warm cache on all systems. The test system for all runs is a dual Xeon E5-2630 with 192GB RAM.</p> <p>The scripts for duplicating the experiment on the Virtuoso 7 Open Source cut will be published later on this blog, when the open source cut incorporates the query optimization improvements discussed herein.</p> <p>In the table below, the times are elapsed real times in milliseconds, with one query at a time, all from warm cache (i.e. the second run of the query set is reported and we make sure that all databases are running from memory). The MySQL is configured with InnoDB and 40GB of buffer pool, which should be enough for the 30GB dataset. The SQL versions do not declare any explicit indices, but do declare primary keys. The SPARQL version is with the default Virtuoso index scheme, but the query plans end up scanning each predicate in order of S, except for the build phases of hash-joins where an index from O-to-S is occasionally used.</p> <table id="SSBResultsTable" title="Comparative Results" align="center" cellspacing="5px" cellpadding="2px" width="450px"> <caption>Comparative Results, 30GB dataset<br />Virtuoso SQL <i>Vs.</i> Virtuoso SPARQL <i>Vs.</i> MonetDB SQL <i>Vs.</i> MySQL SQL</caption> <tr> <th align="center">Query</th> <th align="center">Virtuoso <br /> SQL</th> <th align="center">Virtuoso <br /> SPARQL</th> <th align="center">MonetDB <br /> SQL</th> <th align="center">MySQL <br /> SQL</th> </tr> <tr> <th align="center">Q1</th> <td align="center" bgcolor="#33CC33"> <b><i>413</i> </b></td> <td align="center" bgcolor="#99FF66">1101</td> <td align="center">1659</td> <td align="center">82477</td> </tr> <tr> <th align="center">Q2</th> <td align="center" bgcolor="#33CC33"> <b><i>282</i> </b></td> <td align="center" bgcolor="#99FF66">416</td> <td align="center">500</td> <td align="center">74436</td> </tr> <tr> <th align="center">Q3</th> <td align="center" bgcolor="#33CC33"> <b><i>253</i> </b></td> <td align="center" bgcolor="#99FF66">295</td> <td align="center">494</td> <td align="center">75411</td> </tr> <tr> <th align="center">Q4</th> <td align="center" bgcolor="#33CC33"> <b><i>828</i> </b></td> <td align="center">2484</td> <td align="center" bgcolor="#99FF66">958</td> <td align="center">226604</td> </tr> <tr> <th align="center">Q5</th> <td align="center" bgcolor="#99FF66">837</td> <td align="center">1915</td> <td align="center" bgcolor="#33CC33"> <b><i>648</i> </b></td> <td align="center">222782</td> </tr> <tr> <th align="center">Q6</th> <td align="center" bgcolor="#33CC33"> <b><i>419</i> </b></td> <td align="center">1813</td> <td align="center" bgcolor="#99FF66">541</td> <td align="center">219656</td> </tr> <tr> <th align="center">Q7</th> <td align="center" bgcolor="#33CC33"> <b><i>1062</i> </b></td> <td align="center" bgcolor="#99FF66">2330</td> <td align="center">5658</td> <td align="center">237730</td> </tr> <tr> <th align="center">Q8</th> <td align="center" bgcolor="#99FF66">617</td> <td align="center">2182</td> <td align="center" bgcolor="#33CC33"> <b><i>521</i> </b></td> <td align="center">194918</td> </tr> <tr> <th align="center">Q9</th> <td align="center" bgcolor="#99FF66">547</td> <td align="center">1290</td> <td align="center" bgcolor="#33CC33"> <b><i>381</i> </b></td> <td align="center">186112</td> </tr> <tr> <th align="center">Q10</th> <td align="center" bgcolor="#99FF66">499</td> <td align="center">639</td> <td align="center" bgcolor="#33CC33"> <b><i>370</i> </b></td> <td align="center">186123</td> </tr> <tr> <th align="center">Q11</th> <td align="center" bgcolor="#33CC33"> <b><i>1132</i> </b></td> <td align="center" bgcolor="#99FF66">2142</td> <td align="center">2760</td> <td align="center">241045</td> </tr> <tr> <th align="center">Q12</th> <td align="center" bgcolor="#33CC33"> <b><i>863</i> </b></td> <td align="center">3770</td> <td align="center" bgcolor="#99FF66">2127</td> <td align="center">241439</td> </tr> <tr> <th align="center">Q13</th> <td align="center" bgcolor="#33CC33"> <b><i>653</i> </b></td> <td align="center">1612</td> <td align="center" bgcolor="#99FF66">1005</td> <td align="center">202817</td> </tr> </table> <p>One could argue that comparing against MySQL is unjustified, as MySQL is certainly not optimized for this workload, e.g., it does not do hash joins and does not parallelize queries. On the other hand, the previous work on SSBM for SPARQL, <i><a href="http://eswc-conferences.org/sites/default/files/papers2013/kaempgen.pdf" id="link-id0x2aabcddc7bd8">No Size Fits All</a></i> by <a href="http://www.aifb.kit.edu/web/Benedikt_Kaempgen/en" id="link-id0x2aabcdee77d8">Benedikt Kaempgen</a> and <a href="http://www.aifb.kit.edu/web/Andreas_Harth/en" id="link-id0x2aabdb8285b8">Andreas Harth</a>, published at the <a href="http://2013.eswc-conferences.org/" id="link-id0x2aabcc627198">2013 ESWC</a>, did make the comparison between MySQL and Virtuoso 6; thus we think it informative to include MySQL. To summarize <i>No Size Fits All,</i> SPARQL in Virtuoso 6 lost by a factor of 12 against MySQL, but in the present case, Virtuoso 7 SPARQL wins by a factor of 106 against MySQL.</p> <p>The ESWC paper used a scale of 1G, while the present test uses a scale of 30G. One should remember that the MySQL times are single-threaded, and all other times are multi-threaded. The test system has 12 cores and 24 threads, so running at full platform utilization is at best 16x faster than single threaded. Running SPARQL single-threaded instead of 24 threads-per-query gives a total time of 175s, still over 10x better than MySQL. Compared to the multi-threaded time of 22.5s, the parallelism yields an average acceleration of 7.7x. Running with SPARQL with full threading but no hash-join gives a time of 80s, 3.5x worse than with hash-join. We note that SSB is a very hash-join intensive workload.</p> <p>MonetDB is the more relevant comparison, as it does use the full CPU, with load peaks up to the theoretical 2400% (12 dual-threaded cores) and is the platform on which a lot of the science of the hash join was refined. MonetDB does relatively best with queries that can start by a very selective join (e.g., Q9) where it outperforms Virtuoso. This is probably due to more even splitting of the work among threads and to not having to deal with data compression. Virtuoso wins the most on queries that select a large fraction of the fact table, where MonetDB is penalized due to its policy of full materialization of intermediate results. Joins without any selection (e.g., adding up the <code>lo_extendedprice</code>, and grouping by the <code>d_year</code> of <code>lo_orderdate</code>) show MonetDB at its worst. Such queries do not occur in SSB though, so SSB is a relatively MonetDB-friendly benchmark.</p> <h2>Conclusions</h2> <p>The back of the problem is broken. Big query without schema can be done. One could of course see this coming by experimenting with explicit plans. But nobody out there can manually optimize a query plan. Thus the final step consisted of having a good-enough cost model and a smart-enough search order to get the right plan fast. This will be in the next update of Virtuoso Open Source. At that time, we will publish the full queries and configuration files.</p> <p>This is the breakthrough for RDF analytics. Incremental progress will follow, with more tricks being incorporated, like the ones known to be needed by TPC-H.</p>
2015-08-20T17:35:04.895699-04:00
ESWC 2013 Panel - Semantic Technologies for Big Data Analytics: Opportunities and Challenges
http://www.openlinksw.com:443/blog/vdb/blog/?date=2013-06-04#1731
2013-06-04T14:05:28Z
<p>I was invited to the <a href="http://2013.eswc-conferences.org/program/panel" id="link-id0x7f5118cd8118">ESWC 2013 "Semantic Technologies for Big Data Analytics: Opportunities and Challenges"</a> panel on 29th May 2013 in Montpellier, France. The panel was moderated by <a href="http://www.linkedin.com/in/markogrobelnik" id="link-id0x7f51ce728038">Marko Grobelnik</a> (JSI), with panelists <a href="http://kmi.open.ac.uk/people/member/enrico-motta" id="link-id0x7f51188422d8">Enrico Motta</a> (KMi), <a href="http://www.nuigalway.ie/our-research/people/manfredhauswirth/" id="link-id0x7f50c2ad7568">Manfred Hauswirth</a> (NUIG), <a href="http://www.linkedin.com/in/davidkarger" id="link-id0x7f511aefa138">David Karger</a> (MIT), <a href="http://ngwr.labs.bt.com/FB/JohnDavies.htm" id="link-id0x7f50c37e1f98">John Davies</a> (British Telecom), <a href="http://www.linkedin.com/pub/jose-manuel-gomez-perez/4/66a/3a3" id="link-id0x7f50c0ba56e8">José Manuel Gómez Pérez</a> (ISOCO) and <a href="http://www.openlinksw.com/weblog/oerling/" id="link-id0x7f51ce5da9a8">Orri Erling</a> (myself).</p> <p> Marko opened the panel by looking at the Google Trends search statistics for big data, semantics, business intelligence, data mining, and other such terms. Big data keeps climbing its hype-cycle hill, now above semantics and most of the other terms. But what do these in fact mean? In the leading books about big data, the word semantics does not occur.</p> <p>I will first recap my 5 minute intro, and then summarize some questions and answers. This is from memory and is in no sense a full transcript.</p> <h2> <a name="Presentation" id="Presentation"></a>Presentation</h2> <p>Over the years we have maintained that what the RDF community most needs is good database. Indeed, RDF is relational in essence and, while it requires some new datatypes and other adaptations, there is nothing in it that is fundamentally foreign to RDBMS technology.</p> <p>This spring, we came through on the promise, delivering Virtuoso 7, packed full of all the state-of-the-art tricks in analytics-oriented databasing, column-wise compressed storage, vectored execution, great parallelism, and flexible scale-out.</p> <p>At this same ESWC, <a href="http://www.aifb.kit.edu/web/Benedikt_Kaempgen/en" id="link-id0x7f51cd4e7698">Benedikt Kaempgen</a> and <a href="http://www.aifb.kit.edu/web/Andreas_Harth/en" id="link-id0x7f51c3667b68">Andreas Harth</a> presented a paper (<i><a href="http://eswc-conferences.org/sites/default/files/papers2013/kaempgen.pdf" id="link-id0x7f50c25703d8">No Size Fits All -- Running the Star Schema Benchmark with SPARQL and RDF Aggregate Views</a></i>) comparing Virtuoso and MySQL on the star schema benchmark at 1G scale. We redid their experiments with Virtuoso 7 at 30x and at 300x the scale.</p> <p>At present, when running the star schema benchmark in SQL, we outperform column-store pioneer MonetDB by a factor of 2. When running the same star schema benchmark in SPARQL against triples as opposed to tables, we see a slowdown of 5x. When scaling from 30 to 300G and from one to two machines, we get linear increase in throughput, 5x longer for 10x more data.</p> <p> Coming back to MySQL, the run with 1G takes about 60 seconds. Virtuoso SPARQL does the same on 30x the data in 45 seconds. Well, you could say that we should go pick on somebody in our series and not MySQL, being not relevant for this. Comparing with MonetDB and other analytics column stores is of course more relevant.</p> <p>For cluster scaling, one could say that star schema benchmark is easy, and so it is, but even with harder ones, which do joins across partitions all the time, like the BSBM BI workload, we get scaling that is close to linear.</p> <p>So, for analytics, you can use SPARQL in Virtuoso, and run circles around some common SQL databases.</p> <p>The difference between SQL and SPARQL comes from having no schema. Instead of scanning aligned columns in a table, you do an index lookup for each column. This is not too slow if there is locality, as there is, but still a lot more than when talking about a multicolumn column-compressed table. With more execution tricks, we can maybe cut this to 3x.</p> <p>The beach-head of workable RDF-based analytics on schema-less data has been attained. Medium-scale data, to the single-digit terabytes, is OK on small clusters.</p> <h3> <a name="What about the future?" id="What about the future?"></a>What about the future?</h3> <p>First, Big Data means more than querying. Before meaningful analytics can be done, the data must generally be prepared and massaged. This means fast bulk load and fast database-resident transformation. We have that via flexible, expressive, parallelizable stored procedures and run time hosting. One can do everything one does in <a href="http://dbpedia.org/resource/MapReduce" id="link-id0x7f51cf753658">MapReduce</a> right inside the database.</p> <p>Some analytics cannot be expressed in a query language. For example, graph algorithms like clustering generate large intermediate states and run in many passes. For this, bulk synchronous processing frameworks like Giraph are becoming popular. We can again do this right inside the DBMS, on RDF or SQL tables. There is great platform utilization and more flexibility than in strict BSP, while being able to do any BSP algorithm.</p> <p>The history of technology is one of divergence followed by reintegration. New trends, like Column stores, RDF databases, key value stores, or MapReduce, start as one-off special-purpose products, and the technologies then find their way back into platforms addressing a broader functionality.</p> <p>The whole semantic experiment might be seen as a break-away from the web, if also a little from database, for the specific purpose of exploring schemaless-ness, universal referenceability of data, self-describing data, and some inference.</p> <p>With RDF, we see lasting value in globally consistent identifiers. The URI "superkey" is the ultimate silo-breaker. The future is in integrating more and more varied data and a schema-first approach is cost-prohibitive. If data is to be preserved over extended lengths of time, self-description is essential; the applications and people that produced the data might not be around. Same for publishing data for outside reuse.</p> <p>In fact, many of these things are right now being pursued in mainstream IT. Everybody is reinventing the triple, whether by using non-first normal form key-value pairs in an RDB, tagging each row of a table with the name of the table, using XML documents, etc. The RDF model provides all these desirable features, but most applications that need these things do not run on RDF infrastructure.</p> <p>Anyway, by revolutionizing RDF store performance, we make this technology a cost-effective alternative in places where it was not such before.</p> <p>To get much further in performance, physical storage needs to adapt to the data. Thus, in the long term, we see RDF as a <i>lingua franca</i> of data interchange and publishing, supported by highly scalable and adaptive databases that exploit the structure implicit in the data to deliver performance equal to the best in SQL data warehousing. When we get the schema from the data, we have schema-last flexibility and schema-first performance. The genie is back in the bottle, and data models are unified.</p> <iframe src="http://www.slideshare.net/slideshow/embed_code/22434050" width="476" height="400" frameborder="0" marginwidth="0" marginheight="0" scrolling="no"></iframe> <h2> <a name="Questions and Answers" id="Questions and Answers"></a>Questions and Answers</h2> <h3> <i>Q: Is the web big data?</i> </h3> <p> <b>David Karger:</b> No, the shallow web (i.e., static web pages for purposes of search) is not big data. One can put it in a box and search. But for purposes of more complex processing, like analytics on the structure of the whole web, this is still big data.</p> <h3> <i>Q: I bet you still can't do analytics on a fast stream.</i> </h3> <p> <b>Orri Erling:</b> I am not sure about that, because when you have a stream -- whether this is network management and denial of service detection, or managing traffic in a city -- you know ahead of time what peak volume you are looking at, so you can size the system accordingly. And streams have a schema. So you can play all the database tricks. Vectored execution will work there just as it does for query processing, for example.</p> <h3> <i>Q: I did not mean storage, I meant analysis.</i> </h3> <p> <b>Orri Erling:</b> Here we mean sliding windows and constant queries. The triple vs. row issue also seems the same. There will be some overhead from schema-lastness, but for streams, I would say each has a regular structure.</p> <p> <b>John Davies:</b> For example, we gather gigabytes a minute of traffic data from sensors in the road network and all this data is very regular, with a fixed schema.</p> <p> <b>Manfred Hauswirth:</b> Or is this always so? The internet of things has potentially huge diversity in schema, with everything producing a stream. The user of the stream has no control whatever on the schema.</p> <p> <b>Marko Grobelnik:</b> Yes, we have had streams for a long time -- on Wall Street, for example, where these make a lot of money. But high frequency trading is a very specific application. There is a stream, some analytics, not very complicated, just fast. This is one specific solution, with fixed schema and very specific scope, no explicit semantics.</p> <h3> <i>Q: What is big data, in fact?</i> </h3> <p> <b>David Karger:</b> Computer science has always been about big data; it is just the definition of big that changes. Big data is something one cannot conveniently process on a computer system. Not without unusual tricks, where something trivial, like shortest path, becomes difficult just because of volume. So it is that big data is very much about performance, and performance is usually obtained by sacrificing the general for the specific. The semantic world on the other hand is after something very general and about complex and expressive schema. When data gets big, the schema is vanishingly small in comparison with the data, and the schema work gets done by hand; the schema is not the problem there. Big data is not very internetty either, because the 40 TB produced by the telescope are centrally stored and you do not download them or otherwise transport them very much.</p> <h3> <i>Q: Now, what do each of you understand with semantics?</i> </h3> <p> <b>Manfred Hauswirth:</b> The essential aspect is that data is machine interpretable, with sufficient machine readable context.</p> <p> <b>David Karger:</b> Semantics has to do with complexity or heterogeneity in the schema. Big data has to do with large volume. Maybe semantic big data would be all the databases in the world with a million different schemas. But today we do not see such applications. If the volume is high, the schema is usually not very large.</p> <p> <b>Manfred Hauswirth:</b> This is not so far as that, for example a telco has over a hundred distinct network management systems and each has a different schema.</p> <p> <b>Orri Erling:</b> From the data angle, we have come to associate semantic with </p> <ul> <li>schema-lastness</li> <li>globally-resolvable identifiers</li> <li>self-description</li> </ul> When people use RDF as a storage model, they mostly do so because of schema flexibility, not because of expressive schemas or inference. Some use a little inference, but inference or logics or considerations of knowledge representation do not in our experience drive the choice. <h2> <a name="Conclusion" id="Conclusion"></a>Conclusion</h2> <p>In conclusion, the event was rather peaceful, with a good deal of agreement between the panelists and audience and no heated controversy. I hoped to get some reaction when I said that semantics was schema flexibility, but apparently this has become a politically acceptable stance. In the golden days of AI this would not have been so. But then Marko Grobelnik did point out that the whole landscape has become data driven. Even in fields like natural language, one looks more at statistics than deep structure: For example, if a phrase is often found on Google, it is proper usage.</p>
2015-08-21T14:15:05.521272-04:00
ESWC 2013 Panel - Semantic Technologies for Big Data Analytics: Opportunities and Challenges
http://www.openlinksw.com:443/weblog/oerling/?date=2013-06-04#1730
2013-06-04T14:04:45Z
<p>I was invited to the <a href="http://2013.eswc-conferences.org/program/panel" id="link-id0x7f5118cd8118">ESWC 2013 "Semantic Technologies for Big Data Analytics: Opportunities and Challenges"</a> panel on 29th May 2013 in Montpellier, France. The panel was moderated by <a href="http://www.linkedin.com/in/markogrobelnik" id="link-id0x7f51ce728038">Marko Grobelnik</a> (JSI), with panelists <a href="http://kmi.open.ac.uk/people/member/enrico-motta" id="link-id0x7f51188422d8">Enrico Motta</a> (KMi), <a href="http://www.nuigalway.ie/our-research/people/manfredhauswirth/" id="link-id0x7f50c2ad7568">Manfred Hauswirth</a> (NUIG), <a href="http://www.linkedin.com/in/davidkarger" id="link-id0x7f511aefa138">David Karger</a> (MIT), <a href="http://ngwr.labs.bt.com/FB/JohnDavies.htm" id="link-id0x7f50c37e1f98">John Davies</a> (British Telecom), <a href="http://www.linkedin.com/pub/jose-manuel-gomez-perez/4/66a/3a3" id="link-id0x7f50c0ba56e8">José Manuel Gómez Pérez</a> (ISOCO) and <a href="http://www.openlinksw.com/weblog/oerling/" id="link-id0x7f51ce5da9a8">Orri Erling</a> (myself).</p> <p> Marko opened the panel by looking at the Google Trends search statistics for big data, semantics, business intelligence, data mining, and other such terms. Big data keeps climbing its hype-cycle hill, now above semantics and most of the other terms. But what do these in fact mean? In the leading books about big data, the word semantics does not occur.</p> <p>I will first recap my 5 minute intro, and then summarize some questions and answers. This is from memory and is in no sense a full transcript.</p> <h2> <a name="Presentation" id="Presentation"></a>Presentation</h2> <p>Over the years we have maintained that what the RDF community most needs is good database. Indeed, RDF is relational in essence and, while it requires some new datatypes and other adaptations, there is nothing in it that is fundamentally foreign to RDBMS technology.</p> <p>This spring, we came through on the promise, delivering Virtuoso 7, packed full of all the state-of-the-art tricks in analytics-oriented databasing, column-wise compressed storage, vectored execution, great parallelism, and flexible scale-out.</p> <p>At this same ESWC, <a href="http://www.aifb.kit.edu/web/Benedikt_Kaempgen/en" id="link-id0x7f51cd4e7698">Benedikt Kaempgen</a> and <a href="http://www.aifb.kit.edu/web/Andreas_Harth/en" id="link-id0x7f51c3667b68">Andreas Harth</a> presented a paper (<i><a href="http://eswc-conferences.org/sites/default/files/papers2013/kaempgen.pdf" id="link-id0x7f50c25703d8">No Size Fits All -- Running the Star Schema Benchmark with SPARQL and RDF Aggregate Views</a></i>) comparing Virtuoso and MySQL on the star schema benchmark at 1G scale. We redid their experiments with Virtuoso 7 at 30x and at 300x the scale.</p> <p>At present, when running the star schema benchmark in SQL, we outperform column-store pioneer MonetDB by a factor of 2. When running the same star schema benchmark in SPARQL against triples as opposed to tables, we see a slowdown of 5x. When scaling from 30 to 300G and from one to two machines, we get linear increase in throughput, 5x longer for 10x more data.</p> <p> Coming back to MySQL, the run with 1G takes about 60 seconds. Virtuoso SPARQL does the same on 30x the data in 45 seconds. Well, you could say that we should go pick on somebody in our series and not MySQL, being not relevant for this. Comparing with MonetDB and other analytics column stores is of course more relevant.</p> <p>For cluster scaling, one could say that star schema benchmark is easy, and so it is, but even with harder ones, which do joins across partitions all the time, like the BSBM BI workload, we get scaling that is close to linear.</p> <p>So, for analytics, you can use SPARQL in Virtuoso, and run circles around some common SQL databases.</p> <p>The difference between SQL and SPARQL comes from having no schema. Instead of scanning aligned columns in a table, you do an index lookup for each column. This is not too slow if there is locality, as there is, but still a lot more than when talking about a multicolumn column-compressed table. With more execution tricks, we can maybe cut this to 3x.</p> <p>The beach-head of workable RDF-based analytics on schema-less data has been attained. Medium-scale data, to the single-digit terabytes, is OK on small clusters.</p> <h3> <a name="What about the future?" id="What about the future?"></a>What about the future?</h3> <p>First, Big Data means more than querying. Before meaningful analytics can be done, the data must generally be prepared and massaged. This means fast bulk load and fast database-resident transformation. We have that via flexible, expressive, parallelizable stored procedures and run time hosting. One can do everything one does in <a href="http://dbpedia.org/resource/MapReduce" id="link-id0x7f51cf753658">MapReduce</a> right inside the database.</p> <p>Some analytics cannot be expressed in a query language. For example, graph algorithms like clustering generate large intermediate states and run in many passes. For this, bulk synchronous processing frameworks like Giraph are becoming popular. We can again do this right inside the DBMS, on RDF or SQL tables. There is great platform utilization and more flexibility than in strict BSP, while being able to do any BSP algorithm.</p> <p>The history of technology is one of divergence followed by reintegration. New trends, like Column stores, RDF databases, key value stores, or MapReduce, start as one-off special-purpose products, and the technologies then find their way back into platforms addressing a broader functionality.</p> <p>The whole semantic experiment might be seen as a break-away from the web, if also a little from database, for the specific purpose of exploring schemaless-ness, universal referenceability of data, self-describing data, and some inference.</p> <p>With RDF, we see lasting value in globally consistent identifiers. The URI "superkey" is the ultimate silo-breaker. The future is in integrating more and more varied data and a schema-first approach is cost-prohibitive. If data is to be preserved over extended lengths of time, self-description is essential; the applications and people that produced the data might not be around. Same for publishing data for outside reuse.</p> <p>In fact, many of these things are right now being pursued in mainstream IT. Everybody is reinventing the triple, whether by using non-first normal form key-value pairs in an RDB, tagging each row of a table with the name of the table, using XML documents, etc. The RDF model provides all these desirable features, but most applications that need these things do not run on RDF infrastructure.</p> <p>Anyway, by revolutionizing RDF store performance, we make this technology a cost-effective alternative in places where it was not such before.</p> <p>To get much further in performance, physical storage needs to adapt to the data. Thus, in the long term, we see RDF as a <i>lingua franca</i> of data interchange and publishing, supported by highly scalable and adaptive databases that exploit the structure implicit in the data to deliver performance equal to the best in SQL data warehousing. When we get the schema from the data, we have schema-last flexibility and schema-first performance. The genie is back in the bottle, and data models are unified.</p> <iframe src="http://www.slideshare.net/slideshow/embed_code/22434050" width="476" height="400" frameborder="0" marginwidth="0" marginheight="0" scrolling="no"></iframe> <h2> <a name="Questions and Answers" id="Questions and Answers"></a>Questions and Answers</h2> <h3> <i>Q: Is the web big data?</i> </h3> <p> <b>David Karger:</b> No, the shallow web (i.e., static web pages for purposes of search) is not big data. One can put it in a box and search. But for purposes of more complex processing, like analytics on the structure of the whole web, this is still big data.</p> <h3> <i>Q: I bet you still can't do analytics on a fast stream.</i> </h3> <p> <b>Orri Erling:</b> I am not sure about that, because when you have a stream -- whether this is network management and denial of service detection, or managing traffic in a city -- you know ahead of time what peak volume you are looking at, so you can size the system accordingly. And streams have a schema. So you can play all the database tricks. Vectored execution will work there just as it does for query processing, for example.</p> <h3> <i>Q: I did not mean storage, I meant analysis.</i> </h3> <p> <b>Orri Erling:</b> Here we mean sliding windows and constant queries. The triple vs. row issue also seems the same. There will be some overhead from schema-lastness, but for streams, I would say each has a regular structure.</p> <p> <b>John Davies:</b> For example, we gather gigabytes a minute of traffic data from sensors in the road network and all this data is very regular, with a fixed schema.</p> <p> <b>Manfred Hauswirth:</b> Or is this always so? The internet of things has potentially huge diversity in schema, with everything producing a stream. The user of the stream has no control whatever on the schema.</p> <p> <b>Marko Grobelnik:</b> Yes, we have had streams for a long time -- on Wall Street, for example, where these make a lot of money. But high frequency trading is a very specific application. There is a stream, some analytics, not very complicated, just fast. This is one specific solution, with fixed schema and very specific scope, no explicit semantics.</p> <h3> <i>Q: What is big data, in fact?</i> </h3> <p> <b>David Karger:</b> Computer science has always been about big data; it is just the definition of big that changes. Big data is something one cannot conveniently process on a computer system. Not without unusual tricks, where something trivial, like shortest path, becomes difficult just because of volume. So it is that big data is very much about performance, and performance is usually obtained by sacrificing the general for the specific. The semantic world on the other hand is after something very general and about complex and expressive schema. When data gets big, the schema is vanishingly small in comparison with the data, and the schema work gets done by hand; the schema is not the problem there. Big data is not very internetty either, because the 40 TB produced by the telescope are centrally stored and you do not download them or otherwise transport them very much.</p> <h3> <i>Q: Now, what do each of you understand with semantics?</i> </h3> <p> <b>Manfred Hauswirth:</b> The essential aspect is that data is machine interpretable, with sufficient machine readable context.</p> <p> <b>David Karger:</b> Semantics has to do with complexity or heterogeneity in the schema. Big data has to do with large volume. Maybe semantic big data would be all the databases in the world with a million different schemas. But today we do not see such applications. If the volume is high, the schema is usually not very large.</p> <p> <b>Manfred Hauswirth:</b> This is not so far as that, for example a telco has over a hundred distinct network management systems and each has a different schema.</p> <p> <b>Orri Erling:</b> From the data angle, we have come to associate semantic with </p> <ul> <li>schema-lastness</li> <li>globally-resolvable identifiers</li> <li>self-description</li> </ul> When people use RDF as a storage model, they mostly do so because of schema flexibility, not because of expressive schemas or inference. Some use a little inference, but inference or logics or considerations of knowledge representation do not in our experience drive the choice. <h2> <a name="Conclusion" id="Conclusion"></a>Conclusion</h2> <p>In conclusion, the event was rather peaceful, with a good deal of agreement between the panelists and audience and no heated controversy. I hoped to get some reaction when I said that semantics was schema flexibility, but apparently this has become a politically acceptable stance. In the golden days of AI this would not have been so. But then Marko Grobelnik did point out that the whole landscape has become data driven. Even in fields like natural language, one looks more at statistics than deep structure: For example, if a phrase is often found on Google, it is proper usage.</p>
2013-06-04T16:04:45+02:00
Virtuoso 7 Release
http://www.openlinksw.com:443/blog/vdb/blog/?date=2013-05-13#1729
2013-05-13T16:06:02Z
<p>The quest of <a href="http://www.openlinksw.com/" id="link-id0x7f51106137d8">OpenLink Software</a> is to bring flexibility, efficiency, and expressive power to people working with data. For the past several years, this has been focused on making graph data models viable for the enterprise. Flexibility in schema evolution is a central aspect of this, as is the ability to share identifiers across different information systems, i.e., giving things URIs instead of synthetic keys that are not interpretable outside of a particular application.</p> <p>With <a href="http://virtuoso.openlinksw.com/news/virtuoso-v7-press-release-20130423" id="link-id0x7f51d7c4e7a8">Virtuoso 7</a>, we dramatically improve the efficiency of all this. With databases in the billions of relations (also known as triples, or 3-tuples), we can fit about 3x as many relations in the same space (disk and RAM) as with Virtuoso 6. Single-threaded query speed is up to 3x better, plus there is intra-query parallelization even in single-server configurations. Graph data workloads are all about random lookups. With these, having data in RAM is all-important. With 3x space efficiency, you can run with 3x more data in the same space before starting to go to disk. In some benchmarks, this can make a 20x gain.</p> <p>Also the Virtuoso scale-out support is fundamentally reworked, with much more parallelism and better deployment flexibility.</p> <p>So, for graph data, Virtuoso 7 is a major step in the coming of age of the technology. Data keeps growing and time is getting scarcer, so we need more flexibility and more performance at the same time.</p> <p>So, let’s talk about how we accomplish this. Column stores have been the trend in relational data warehousing for over a decade. With column stores comes vectored execution, i.e., running any operation on a large number of values at one time. Instead of running one operation on one value, then the next operation on the result, and so forth, you run the first operation on thousands or hundreds-of-thousands of values, then the next one on the results of this, and so on.</p> <p>Column-wise storage brings space efficiency, since values in one column of a table tend to be alike -- whether repeating, sorted, within a specific range, or picked from a particular set of possible values. With graph data, where there are no columns as such, the situation is exactly the same -- just substitute the word <i>predicate</i> for <i>column.</i> Space efficiency brings speed -- first by keeping more of the data in memory; secondly by having less data travel between CPU and memory. Vectoring makes sure that data that are closely located get accessed in close temporal proximity, hence improving cache utilization. When there is no locality, there are a lot of operations pending at the same time, as things always get done on a set of values instead of on a single value. This is the crux of the science of columns and vectoring.</p> <p>Of the prior work in column stores, Virtuoso may most resemble <a href="http://www.vertica.com/" id="link-id0x7f51d66bfc88">Vertica</a>, well described in <a href="http://cs-www.cs.yale.edu/homes/dna/" id="link-id0x7f5110b17b68">Daniel Abadi</a>’s famous <a href="http://dspace.mit.edu/handle/1721.1/43043" id="link-id0x7f51d68fdfc8">PhD thesis</a>. Virtuoso itself is described in <a href="http://www.informatik.uni-trier.de/~ley/db/journals/debu/index.html" id="link-id0x7f51d73b8f28">IEEE Data Engineering Bulletin</a>, <a href="http://www.informatik.uni-trier.de/~ley/db/journals/debu/debu35.html" id="link-id0x7f51c6705998">March 2012</a> (<a href="http://bit.ly/166kEnC" id="link-id0x7f51f0862d28">PDF</a>). The first experiments in column store technology with Virtuoso were in 2009, published at the <a href="http://semdata.org/events/2010/vldb" id="link-id0x7f50d10c3508">SemData workshop</a> at <a href="http://www.vldb2010.org/" id="link-id0x7f51d73fbb98">VLDB 2010</a> in Singapore. We tried storing <a href="http://www.tpc.org/tpch/" id="link-id0x7f51d5f43b58">TPC H</a> as graph data and in relational tables, each with both rows and columns, and found that we could get 6 bytes per quad space utilization with the RDF-ization of TPC H, as opposed to 27 bytes with the row-wise compressed RDF storage model. The row-wise compression itself is 3x more compact than a row-wise representation with no compression.</p> <p>Memory is the key to speed, and space efficiency is the key to memory. Performance comes from two factors: locality and parallelism. Both are addressed by column store technology. This made me a convert.</p> <p>At this time, we also started the EU FP7 project, <a href="http://lod2.eu/" id="link-id0x7f51d53489d8">LOD2</a>, most specifically working with <a href="http://homepages.cwi.nl/~boncz/" id="link-id0x7f51d5ab9f18">Peter Boncz</a> of CWI, the king of the column store, famous for <a href="http://dbpedia.org/page/MonetDB" id="link-id0x7f51d5e97078">MonetDB</a> and <a href="http://dbpedia.org/page/Vectorwise" id="link-id0x7f51d60bfa88">VectorWise</a>. This cooperation goes on within LOD2 and has extended to <a href="http://www.ldbc.eu/" id="link-id0x7f51d5ccaad8">LDBC</a>, an FP7 for designing benchmarks for graph and RDF databases. Peter has given us a world of valuable insight and experience in all aspects of <i>avant garde</i> database, from adaptive techniques to query optimization and beyond. One thing that was recently published is the <a href="http://bit.ly/14ULX2F" id="link-id0x7f51d69a15e8">results for Virtuoso cluster at CWI</a>, running analytics on 150 billion relations on CWI’s SciLens cluster.</p> <p>The SQL relational table-oriented databases and property graph-oriented databases (Graph for short) are both rooted in relational database science. Graph management simply introduces extra challenges with regards to scalability. Hence, at OpenLink Software, having a good grounding in the best practices of relational columnar (or column-wise) database management technology is vital.</p> <p>Virtuoso is more prominently known for high-performance RDF-based graph database technology, but the entirety of its SQL relational data management functionality (which is the foundation for graph store) is vectored, and even allows users to choose between row-wise and column-wise physical layouts, index by index.</p> <p>It has been asked: is this a new NoSQL engine? Well, there isn’t really such a thing. There are of course database engines that do not have SQL support and it has become trendy to call them "NoSQL." So, in this space, Virtuoso is an engine that <i>does</i> support SQL, plus SPARQL, and is designed to do big joins and aggregation (i.e., analytics) and fast bulk load, as well as ACID transactions on small updates, all with column store space efficiency. It is not only for big scans, as people tend to think about column stores, since it can also be used in compact embedded form.</p> <p>Virtuoso also delivers great parallelism and throughput in a scale-out setting, with no restrictions on transactions and no limits on joining. The base is in relational database science, but all the adaptations that RDF and graph workloads need are built-in, with core level support for run-time data-typing, URIs as native Reference types, user-defined custom data types, etc.</p> <p>Now that the major milestone of releasing Virtuoso 7 (<a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VOSIndex" id="link-id0x7f51d6f78e88">open source</a> and <a href="http://virtuoso.openlinksw.com/download/" id="link-id0x7f5110cf58f8">commercial editions</a>) has been reached, the next steps include enabling our current and future customers to attain increased agility from big (linked) open data exploits. Technically, it will also include continued participation in DBMS industry benchmarks, such as those from the <a href="http://www.tpc.org/" id="link-id0x7f51d5088008">TPC</a>, and others under development via the Linked Data Benchmark Council (LDBC), plus other social-media-oriented challenges that arise in this exciting data access, integration, and management innovation continuum. Thus, continue to expect new optimization tricks to be introduced at frequent intervals through the open source development branch at <a href="https://github.com/openlink/" id="link-id0x7f51c4addd18">GitHub</a>, between major commercial releases.</p> <p> <i><b>Related</b> </i> </p> <ul> <li> <a href="http://bit.ly/17oSWk9" id="link-id0x7f51f0c46f18">Column Store Tutorial from Peter Boncz and Daniel Abadi</a> </li> <li> <a href="http://slidesha.re/13RzSfq" id="link-id0x7f511a9f1b98">NewSQL vs NoSQL session by Prof. Michael Stonebraker</a> </li> <li> <a href="http://bit.ly/166kEnC" id="link-id0x7f51e1d85de8">Virtuoso 7.0 White Paper by Orri Erling (ACM edition)</a> </li> </ul>
2015-08-21T14:17:45.353974-04:00
Virtuoso 7 Release
http://www.openlinksw.com:443/weblog/oerling/?date=2013-05-13#1728
2013-05-13T16:05:13Z
<p>The quest of <a href="http://www.openlinksw.com/" id="link-id0x7f51106137d8">OpenLink Software</a> is to bring flexibility, efficiency, and expressive power to people working with data. For the past several years, this has been focused on making graph data models viable for the enterprise. Flexibility in schema evolution is a central aspect of this, as is the ability to share identifiers across different information systems, i.e., giving things URIs instead of synthetic keys that are not interpretable outside of a particular application.</p> <p>With <a href="http://virtuoso.openlinksw.com/news/virtuoso-v7-press-release-20130423" id="link-id0x7f51d7c4e7a8">Virtuoso 7</a>, we dramatically improve the efficiency of all this. With databases in the billions of relations (also known as triples, or 3-tuples), we can fit about 3x as many relations in the same space (disk and RAM) as with Virtuoso 6. Single-threaded query speed is up to 3x better, plus there is intra-query parallelization even in single-server configurations. Graph data workloads are all about random lookups. With these, having data in RAM is all-important. With 3x space efficiency, you can run with 3x more data in the same space before starting to go to disk. In some benchmarks, this can make a 20x gain.</p> <p>Also the Virtuoso scale-out support is fundamentally reworked, with much more parallelism and better deployment flexibility.</p> <p>So, for graph data, Virtuoso 7 is a major step in the coming of age of the technology. Data keeps growing and time is getting scarcer, so we need more flexibility and more performance at the same time.</p> <p>So, let’s talk about how we accomplish this. Column stores have been the trend in relational data warehousing for over a decade. With column stores comes vectored execution, i.e., running any operation on a large number of values at one time. Instead of running one operation on one value, then the next operation on the result, and so forth, you run the first operation on thousands or hundreds-of-thousands of values, then the next one on the results of this, and so on.</p> <p>Column-wise storage brings space efficiency, since values in one column of a table tend to be alike -- whether repeating, sorted, within a specific range, or picked from a particular set of possible values. With graph data, where there are no columns as such, the situation is exactly the same -- just substitute the word <i>predicate</i> for <i>column.</i> Space efficiency brings speed -- first by keeping more of the data in memory; secondly by having less data travel between CPU and memory. Vectoring makes sure that data that are closely located get accessed in close temporal proximity, hence improving cache utilization. When there is no locality, there are a lot of operations pending at the same time, as things always get done on a set of values instead of on a single value. This is the crux of the science of columns and vectoring.</p> <p>Of the prior work in column stores, Virtuoso may most resemble <a href="http://www.vertica.com/" id="link-id0x7f51d66bfc88">Vertica</a>, well described in <a href="http://cs-www.cs.yale.edu/homes/dna/" id="link-id0x7f5110b17b68">Daniel Abadi</a>’s famous <a href="http://dspace.mit.edu/handle/1721.1/43043" id="link-id0x7f51d68fdfc8">PhD thesis</a>. Virtuoso itself is described in <a href="http://www.informatik.uni-trier.de/~ley/db/journals/debu/index.html" id="link-id0x7f51d73b8f28">IEEE Data Engineering Bulletin</a>, <a href="http://www.informatik.uni-trier.de/~ley/db/journals/debu/debu35.html" id="link-id0x7f51c6705998">March 2012</a> (<a href="http://bit.ly/166kEnC" id="link-id0x7f50d0f31ba8">PDF</a>). The first experiments in column store technology with Virtuoso were in 2009, published at the <a href="http://semdata.org/events/2010/vldb" id="link-id0x7f50d10c3508">SemData workshop</a> at <a href="http://www.vldb2010.org/" id="link-id0x7f51d73fbb98">VLDB 2010</a> in Singapore. We tried storing <a href="http://www.tpc.org/tpch/" id="link-id0x7f51d5f43b58">TPC H</a> as graph data and in relational tables, each with both rows and columns, and found that we could get 6 bytes per quad space utilization with the RDF-ization of TPC H, as opposed to 27 bytes with the row-wise compressed RDF storage model. The row-wise compression itself is 3x more compact than a row-wise representation with no compression.</p> <p>Memory is the key to speed, and space efficiency is the key to memory. Performance comes from two factors: locality and parallelism. Both are addressed by column store technology. This made me a convert.</p> <p>At this time, we also started the EU FP7 project, <a href="http://lod2.eu/" id="link-id0x7f51d53489d8">LOD2</a>, most specifically working with <a href="http://homepages.cwi.nl/~boncz/" id="link-id0x7f51d5ab9f18">Peter Boncz</a> of CWI, the king of the column store, famous for <a href="http://dbpedia.org/page/MonetDB" id="link-id0x7f51d5e97078">MonetDB</a> and <a href="http://dbpedia.org/page/Vectorwise" id="link-id0x7f51d60bfa88">VectorWise</a>. This cooperation goes on within LOD2 and has extended to <a href="http://www.ldbc.eu/" id="link-id0x7f51d5ccaad8">LDBC</a>, an FP7 for designing benchmarks for graph and RDF databases. Peter has given us a world of valuable insight and experience in all aspects of <i>avant garde</i> database, from adaptive techniques to query optimization and beyond. One thing that was recently published is the <a href="http://bit.ly/14ULX2F" id="link-id0x7f51d69a15e8">results for Virtuoso cluster at CWI</a>, running analytics on 150 billion relations on CWI’s SciLens cluster.</p> <p>The SQL relational table-oriented databases and property graph-oriented databases (Graph for short) are both rooted in relational database science. Graph management simply introduces extra challenges with regards to scalability. Hence, at OpenLink Software, having a good grounding in the best practices of relational columnar (or column-wise) database management technology is vital.</p> <p>Virtuoso is more prominently known for high-performance RDF-based graph database technology, but the entirety of its SQL relational data management functionality (which is the foundation for graph store) is vectored, and even allows users to choose between row-wise and column-wise physical layouts, index by index.</p> <p>It has been asked: is this a new NoSQL engine? Well, there isn’t really such a thing. There are of course database engines that do not have SQL support and it has become trendy to call them "NoSQL." So, in this space, Virtuoso is an engine that <i>does</i> support SQL, plus SPARQL, and is designed to do big joins and aggregation (i.e., analytics) and fast bulk load, as well as ACID transactions on small updates, all with column store space efficiency. It is not only for big scans, as people tend to think about column stores, since it can also be used in compact embedded form.</p> <p>Virtuoso also delivers great parallelism and throughput in a scale-out setting, with no restrictions on transactions and no limits on joining. The base is in relational database science, but all the adaptations that RDF and graph workloads need are built-in, with core level support for run-time data-typing, URIs as native Reference types, user-defined custom data types, etc.</p> <p>Now that the major milestone of releasing Virtuoso 7 (<a href="http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VOSIndex" id="link-id0x7f51d6f78e88">open source</a> and <a href="http://virtuoso.openlinksw.com/download/" id="link-id0x7f5110cf58f8">commercial editions</a>) has been reached, the next steps include enabling our current and future customers to attain increased agility from big (linked) open data exploits. Technically, it will also include continued participation in DBMS industry benchmarks, such as those from the <a href="http://www.tpc.org/" id="link-id0x7f51d5088008">TPC</a>, and others under development via the Linked Data Benchmark Council (LDBC), plus other social-media-oriented challenges that arise in this exciting data access, integration, and management innovation continuum. Thus, continue to expect new optimization tricks to be introduced at frequent intervals through the open source development branch at <a href="https://github.com/openlink/" id="link-id0x7f51c4addd18">GitHub</a>, between major commercial releases.</p> <p> <i><b>Related</b> </i> </p> <ul> <li> <a href="http://bit.ly/17oSWk9" id="link-id0x7f51d68ac7a8">Column Store Tutorial from Peter Boncz and Daniel Abadi</a> </li> <li> <a href="http://slidesha.re/13RzSfq" id="link-id0x7f51d68220f8">NewSQL vs NoSQL session by Prof. Michael Stonebraker</a> </li> <li> <a href="http://bit.ly/166kEnC" id="link-id0x7f51e27b37e8">Virtuoso 7.0 White Paper by Orri Erling (ACM edition)</a> </li> </ul>
2013-05-13T18:52:44.000002+02:00
LDBC: A Socio-technical Perspective
http://www.openlinksw.com:443/blog/vdb/blog/?date=2012-12-03#1727
2012-12-03T15:24:40Z
<p> <i>(Originally posted to <a href="http://www.ldbc.eu/blog/ldbc-socio-technical-perspective" id="link-id0x7f4529b06088">the LDBC blog</a>.)</i> </p> <p>In recent days, cyberspace has seen some discussion concerning the relationship of the EU FP7 project <a href="http://www.ldbc.eu/" id="link-id0x7f41908e6af8">LDBC (Linked Data Benchmark Council)</a> and sociotechnical considerations. It has been suggested that LDBC, to its own and the community’s detriment, ignores sociotechnical aspects.</p> <p>LDBC, as research projects go, actually has an unusually large, and as of this early date, successful and thriving sociotechnical aspect, i.e., involvement of users and vendors alike. I will here discuss why, insofar as the technical output of the project goes, sociotechnical metrics are in fact out of scope. Then yet again, to what degree the benefits potentially obtained from the use of LDBC outcomes are in fact realized does have a strong dependence on community building, a social process.</p> <p>One criticism of big data projects we sometimes encounter is the point that data without context is not useful. Further, one cannot just assume that one can throw several data sets together and get meaning from this, as there may be different semantics for similar looking things, just think of 7 different definitions of blood pressure.</p> <p>In its <a href="http://www.ldbc.eu/events/1st-ldbc-technical-user-community-meeting" id="link-id0x7f477e386598">initial user community meeting</a>, LDBC was, according to its charter, focusing mostly on cases where the data is already in existence and of sufficient quality for the application at hand.</p> <p>Michael Brodie, Chief Scientist at Verizon, is a well known advocate of focusing on meaning of data, not only on processing performance. There is a piece on this matter by him, Peter Boncz, Chris Bizer, and myself on the Sigmod Record: "<a href="http://www.sigmod.org/publications/sigmod-record/1112/pdfs/10.report.bizer.pdf" id="link-id0x7f43522f8398">The Meaningful Use of Big Data: Four Perspectives – Four Challenges</a>".</p> <p>I had a conversation with Michael at a DERI meeting a couple of years ago about measuring the total cost of technology adoption, thus including socio-technical aspects such as acceptance by users, learning curves of various stakeholders, whether in fact one could demonstrate an overall gain in productivity arising from semantic technologies. [in my words, paraphrased] </p> <blockquote> <p> <i>"Can one measure the effectiveness of different approaches to data integration?"</i> asked I. </p> <p> <i>"Of course one can,"</i> answered Michael, <i>"this only involves carrying out the same task with two different technologies, two different teams and then doing a double blind test with users. However, this never happens. Nobody does this because doing the task even once in a large organization is enormously costly and nobody will even seriously consider doubling the expense."</i> </p> </blockquote> <p>LDBC does in fact intend to address technical aspects of data integration, i.e., schema conversion, entity resolution, and the like. Addressing the sociotechnical aspects of this (whether one should integrate in the first place, whether the integration result adds value, whether it violates privacy or security concerns, whether users will understand the result, what the learning curves are, etc.) is simply too diverse and so totally domain dependent that a general purpose metric cannot be developed, at least not in the time and budget constraints of the project. Further, adding a large human element in the experimental setting (e.g., how skilled the developers are, how well the stakeholders can explain their needs, how often these needs change, etc.) will lead to experiments that are so expensive to carry out and whose results will have so many unquantifiable factors that these will constitute an insuperable barrier to adoption.</p> <p>Experience demonstrates that even agreeing on the relative importance of quantifiable metrics of database performance is hard enough. Overreaching would compromise the project's ability to deliver its core value. Let us next talk about this.</p> <p>It is only a natural part of the political landscape that the EC's research funding choices are criticized by some members of the public. Some criticism is about the emphasis on big data. Big data is a fact on the ground, and research and industry need to deal with it. Of course, there have been and will be critics of technology in general on moral or philosophical grounds. Instead of opening this topic, I will refer you to <a href="http://www.michaelbrodie.com/michael_brodie_statement.asp" id="link-id0x7f43528f65c8">an article by Michael Brodie</a>. In a world where big data is a given, lowering the entry threshold for big data applications, thus making them available not only to government agencies and the largest businesses, seems ethical to me, as per Brodie's checklist. LDBC will contribute to this by driving greater availability, better performance, and lower cost for these technologies.</p> <p>Once we accept that big data is there and is important, we arrive at the issue of deriving actionable meaning from it. A prerequisite of deriving actionable meaning from big data is the ability to flexibly process this data. LDBC is about creating metrics for this. The prerequisites for flexibly working with data are fairly independent of the specific use case, while the criteria of meaning, let alone actionable analysis, are very domain specific. Therefore, in order to provide the greatest service to the broadest constituency, LDBC focuses on measuring that which is most generic, yet will underlie any decision support or other data processing deployment that involves RDF or graph data.</p> <p>I would say that LDBC is an exceptionally effective use of taxpayer money. LDBC will produce metrics that will drive technological innovation for years to come. The total money spent towards pursuing goals set forth by LDBC is likely to vastly exceed the budget of LDBC. Only think of the person-centuries or even millennia that have gone into optimizing for TPC-C and TPC-H. The vast majority of the money spent for these pursuits is paid by industry, not by research funding. It is spent worldwide, not in Europe alone.</p> <p>Thus, if LDBC is successful, a limited amount of EC research money will influence how much greater product development budgets are spent in the future. This multiplier effect applies of course to highly successful research outcomes in general but is especially clear with LDBC.</p> <p>European research funding has played a significant role in creating the foundations of the RDF/Linked Data scene. LDBC is a continuation of this policy, however the focus has now shifted to reflect the greater maturity of the technology. LDBC is now about making the RDF and graph database sectors into mature industries whose products can predictably tackle the challenges out there.</p>
2012-12-03T16:24:40+01:00
LDBC: A Socio-technical Perspective
http://www.openlinksw.com:443/weblog/oerling/?date=2012-12-03#1726
2012-12-03T15:23:47Z
<p> <i>(Originally posted to <a href="http://www.ldbc.eu/blog/ldbc-socio-technical-perspective" id="link-id0x7f45f274e738">the LDBC blog</a>.)</i> </p> <p>In recent days, cyberspace has seen some discussion concerning the relationship of the EU FP7 project <a href="http://www.ldbc.eu/" id="link-id0x7f424b1e36b8">LDBC (Linked Data Benchmark Council)</a> and sociotechnical considerations. It has been suggested that LDBC, to its own and the community’s detriment, ignores sociotechnical aspects.</p> <p>LDBC, as research projects go, actually has an unusually large, and as of this early date, successful and thriving sociotechnical aspect, i.e., involvement of users and vendors alike. I will here discuss why, insofar as the technical output of the project goes, sociotechnical metrics are in fact out of scope. Then yet again, to what degree the benefits potentially obtained from the use of LDBC outcomes are in fact realized does have a strong dependence on community building, a social process.</p> <p>One criticism of big data projects we sometimes encounter is the point that data without context is not useful. Further, one cannot just assume that one can throw several data sets together and get meaning from this, as there may be different semantics for similar looking things, just think of 7 different definitions of blood pressure.</p> <p>In its <a href="http://www.ldbc.eu/events/1st-ldbc-technical-user-community-meeting" id="link-id0x7f4588b17f58">initial user community meeting</a>, LDBC was, according to its charter, focusing mostly on cases where the data is already in existence and of sufficient quality for the application at hand.</p> <p>Michael Brodie, Chief Scientist at Verizon, is a well known advocate of focusing on meaning of data, not only on processing performance. There is a piece on this matter by him, Peter Boncz, Chris Bizer, and myself on the Sigmod Record: "<a href="http://www.sigmod.org/publications/sigmod-record/1112/pdfs/10.report.bizer.pdf" id="link-id0x7f424baeba68">The Meaningful Use of Big Data: Four Perspectives – Four Challenges</a>".</p> <p>I had a conversation with Michael at a DERI meeting a couple of years ago about measuring the total cost of technology adoption, thus including socio-technical aspects such as acceptance by users, learning curves of various stakeholders, whether in fact one could demonstrate an overall gain in productivity arising from semantic technologies. [in my words, paraphrased] </p> <blockquote> <p> <i>"Can one measure the effectiveness of different approaches to data integration?"</i> asked I. </p> <p> <i>"Of course one can,"</i> answered Michael, <i>"this only involves carrying out the same task with two different technologies, two different teams and then doing a double blind test with users. However, this never happens. Nobody does this because doing the task even once in a large organization is enormously costly and nobody will even seriously consider doubling the expense."</i> </p> </blockquote> <p>LDBC does in fact intend to address technical aspects of data integration, i.e., schema conversion, entity resolution, and the like. Addressing the sociotechnical aspects of this (whether one should integrate in the first place, whether the integration result adds value, whether it violates privacy or security concerns, whether users will understand the result, what the learning curves are, etc.) is simply too diverse and so totally domain dependent that a general purpose metric cannot be developed, at least not in the time and budget constraints of the project. Further, adding a large human element in the experimental setting (e.g., how skilled the developers are, how well the stakeholders can explain their needs, how often these needs change, etc.) will lead to experiments that are so expensive to carry out and whose results will have so many unquantifiable factors that these will constitute an insuperable barrier to adoption.</p> <p>Experience demonstrates that even agreeing on the relative importance of quantifiable metrics of database performance is hard enough. Overreaching would compromise the project's ability to deliver its core value. Let us next talk about this.</p> <p>It is only a natural part of the political landscape that the EC's research funding choices are criticized by some members of the public. Some criticism is about the emphasis on big data. Big data is a fact on the ground, and research and industry need to deal with it. Of course, there have been and will be critics of technology in general on moral or philosophical grounds. Instead of opening this topic, I will refer you to <a href="http://www.michaelbrodie.com/michael_brodie_statement.asp" id="link-id0x7f45f2720538">an article by Michael Brodie</a>. In a world where big data is a given, lowering the entry threshold for big data applications, thus making them available not only to government agencies and the largest businesses, seems ethical to me, as per Brodie's checklist. LDBC will contribute to this by driving greater availability, better performance, and lower cost for these technologies.</p> <p>Once we accept that big data is there and is important, we arrive at the issue of deriving actionable meaning from it. A prerequisite of deriving actionable meaning from big data is the ability to flexibly process this data. LDBC is about creating metrics for this. The prerequisites for flexibly working with data are fairly independent of the specific use case, while the criteria of meaning, let alone actionable analysis, are very domain specific. Therefore, in order to provide the greatest service to the broadest constituency, LDBC focuses on measuring that which is most generic, yet will underlie any decision support or other data processing deployment that involves RDF or graph data.</p> <p>I would say that LDBC is an exceptionally effective use of taxpayer money. LDBC will produce metrics that will drive technological innovation for years to come. The total money spent towards pursuing goals set forth by LDBC is likely to vastly exceed the budget of LDBC. Only think of the person-centuries or even millennia that have gone into optimizing for TPC-C and TPC-H. The vast majority of the money spent for these pursuits is paid by industry, not by research funding. It is spent worldwide, not in Europe alone.</p> <p>Thus, if LDBC is successful, a limited amount of EC research money will influence how much greater product development budgets are spent in the future. This multiplier effect applies of course to highly successful research outcomes in general but is especially clear with LDBC.</p> <p>European research funding has played a significant role in creating the foundations of the RDF/Linked Data scene. LDBC is a continuation of this policy, however the focus has now shifted to reflect the greater maturity of the technology. LDBC is now about making the RDF and graph database sectors into mature industries whose products can predictably tackle the challenges out there.</p>
2012-12-03T16:23:47+01:00
LDBC - the Linked Data Benchmark Council
http://www.openlinksw.com:443/blog/vdb/blog/?date=2012-11-28#1725
2012-11-28T17:08:37Z
<p> <i>(This posting was inadvertently delayed from the time of its writing, 2012-11-21.)</i> </p> <p>The <a href="http://www.ldbc.eu/" id="link-id0x7f4780561218">Linked Data Benchmark Council (LDBC)</a> project is officially starting now.</p> <p>This represents a serious effort towards making relevant and well thought out <a href="http://dbpedia.org/page/Software_metric" id="link-id0x7f46086bb908">metrics</a> for <a href="http://dbpedia.org/page/Resource_Description_Framework" id="link-id0x7f4750f49728">RDF</a> and graph databases and defining protocols for measurement and publishing of well documented and reproducible results. This also entails the creation of a <a href="http://www.tpc.org/" id="link-id0x7f47805a1838">TPC</a>-analog for the graph and RDF domains.</p> <p>The project brings together leading vendors, with <a href="http://dbpedia.org/page/OpenLink_Software" id="link-id0x7f47801c9ce8">OpenLink</a> and <a href="http://dbpedia.org/page/Ontotext" id="link-id0x7f478059b2f8">Ontotext</a> representing the RDF side and <a href="http://dbpedia.org/resource/Neo_Technology" id="link-id0x7f47813a0968">Neo Technology</a> and <a href="http://www.sparsity-technologies.com/" id="link-id0x7f4780128c48">Sparsity Technologies</a> representing the graph database side. <a href="http://homepages.cwi.nl/~boncz/" id="link-id0x7f47808001c8">Peter Boncz</a> of <a href="http://dbpedia.org/page/MonetDB" id="link-id0x7f4629543fc8">MonetDB</a> and <a href="http://dbpedia.org/page/Vectorwise" id="link-id0x7f47800f84d8">Vectorwise</a> fame is the technical director, with participation from the <a href="http://dbpedia.org/page/Technical_University_Munich" id="link-id0x7f47812e0108">Technical University of Munich</a> with <a href="http://www-db.in.tum.de/~neumann/" id="link-id0x7f4780428808">Thomas Neumann</a>, known for <a href="http://code.google.com/p/rdf3x/" id="link-id0x7f478218ae38">RDF3X</a> and <a href="http://www-db.in.tum.de/research/projects/HyPer/" id="link-id0x7f47805eceb8">HyPer</a>. <a href="http://dbpedia.org/page/Polytechnic_University_of_Catalonia" id="link-id0x7f46295754c8">La Universitat Politècnica de Catalunya</a> coordinates the project and brings strong academic expertise in graph databasing, also representing their Sparsity Technologies spinoff. <a href="http://www.forth.gr/" id="link-id0x7f47805857e8">FORTH (Foundation for Research and Technology - Hellas) of Crete</a> contributes expertise in data integration and provenance. <a href="http://www.sti-innsbruck.at/" id="link-id0x7f47804aee08">STI Innsbruck</a> participates in community building and outreach.</p> <p>The consortium has second-to-none understanding of benchmarking and has sufficient time allotted to the task for producing world class work, comparable to the TPC benchmarks. This has to date never been realized in the RDF or graph space.</p> <p>History demonstrates that whenever something that is sufficiently important starts getting systematically measured, there is an improvement in the metric. The early days of the TPC saw a 40-fold increase in transaction processing speed. TPC-H continues to be, after 18 years, well used as a basis of quantifying advances in analytics databases.</p> <p>A serious initiative for well-thought-out benchmarks for guiding the emerging RDF and graph database markets is nothing short of a necessary precondition for the emergence of a serious market with several vendors offering mutually comparable products.</p> <p>Benchmarks are only as good as their credibility and adoption. For this reason, LDBC has been in touch with all graph and RDF vendors we could find, and has received a positive statement of intent from most, indicating that they would participate in a LDBC organization and contribute to shaping benchmarks.</p> <p>There is further a Technical User Community, with its initial meeting this week, where present-day end users of RDF and graph databases will voice their wishes for benchmark development. Thus benchmarks will be grounded in use cases contributed by real users.</p> <p>With these elements in place we have every reason to expect relevant benchmarks with broad adoption, with all the benefits this entails.</p>
2012-11-28T18:08:37+01:00
LDBC - the Linked Data Benchmark Council
http://www.openlinksw.com:443/weblog/oerling/?date=2012-11-28#1724
2012-11-28T17:06:19Z
<p> <i>(This posting was inadvertently delayed from the time of its writing, 2012-11-21.)</i> </p> <p>The <a href="http://www.ldbc.eu" id="link-id0x7f465a65c618">Linked Data Benchmark Council (LDBC)</a> project is officially starting now.</p> <p>This represents a serious effort towards making relevant and well thought out <a href="http://dbpedia.org/page/Software_metric" id="link-id0x7f4659a35e78">metrics</a> for <a href="http://dbpedia.org/page/Resource_Description_Framework" id="link-id0x7f4629429a48">RDF</a> and graph databases and defining protocols for measurement and publishing of well documented and reproducible results. This also entails the creation of a <a href="http://www.tpc.org/" id="link-id0x7f4629510fb8">TPC</a>-analog for the graph and RDF domains.</p> <p>The project brings together leading vendors, with <a href="http://dbpedia.org/page/OpenLink_Software" id="link-id0x7f465999de58">OpenLink</a> and <a href="http://dbpedia.org/page/Ontotext" id="link-id0x7f46294e3c68">Ontotext</a> representing the RDF side and <a href="http://dbpedia.org/resource/Neo_Technology" id="link-id0x7f4659a03e38">Neo Technology</a> and <a href="http://www.sparsity-technologies.com/" id="link-id0x7f46295bb358">Sparsity Technologies</a> representing the graph database side. <a href="http://homepages.cwi.nl/~boncz/" id="link-id0x7f4629362188">Peter Boncz</a> of <a href="http://dbpedia.org/page/MonetDB" id="link-id0x7f462931ee08">MonetDB</a> and <a href="http://dbpedia.org/page/Vectorwise" id="link-id0x7f4776358078">Vectorwise</a> fame is the technical director, with participation from the <a href="http://dbpedia.org/page/Technical_University_Munich" id="link-id0x7f465aa2aef8">Technical University of Munich</a> with <a href="http://www-db.in.tum.de/~neumann/" id="link-id0x7f4658d62378">Thomas Neumann</a>, known for <a href="http://code.google.com/p/rdf3x/" id="link-id0x7f465a4a9f98">RDF3X</a> and <a href="http://www-db.in.tum.de/research/projects/HyPer/" id="link-id0x7f465a1ce1f8">HyPer</a>. <a href="http://dbpedia.org/page/Polytechnic_University_of_Catalonia" id="link-id0x7f462945d2d8">La Universitat Politècnica de Catalunya</a> coordinates the project and brings strong academic expertise in graph databasing, also representing their Sparsity Technologies spinoff. <a href="http://www.forth.gr/" id="link-id0x7f476266ce78">FORTH (Foundation for Research and Technology - Hellas) of Crete</a> contributes expertise in data integration and provenance. <a href="http://www.sti-innsbruck.at/" id="link-id0x7f47624ae5f8">STI Innsbruck</a> participates in community building and outreach.</p> <p>The consortium has second-to-none understanding of benchmarking and has sufficient time allotted to the task for producing world class work, comparable to the TPC benchmarks. This has to date never been realized in the RDF or graph space.</p> <p>History demonstrates that whenever something that is sufficiently important starts getting systematically measured, there is an improvement in the metric. The early days of the TPC saw a 40-fold increase in transaction processing speed. TPC-H continues to be, after 18 years, well used as a basis of quantifying advances in analytics databases.</p> <p>A serious initiative for well-thought-out benchmarks for guiding the emerging RDF and graph database markets is nothing short of a necessary precondition for the emergence of a serious market with several vendors offering mutually comparable products.</p> <p>Benchmarks are only as good as their credibility and adoption. For this reason, LDBC has been in touch with all graph and RDF vendors we could find, and has received a positive statement of intent from most, indicating that they would participate in a LDBC organization and contribute to shaping benchmarks.</p> <p>There is further a Technical User Community, with its initial meeting this week, where present-day end users of RDF and graph databases will voice their wishes for benchmark development. Thus benchmarks will be grounded in use cases contributed by real users.</p> <p>With these elements in place we have every reason to expect relevant benchmarks with broad adoption, with all the benefits this entails.</p>
2012-11-28T18:06:19.000007+01:00
LDBC Technical User Community Meeting
http://www.openlinksw.com:443/blog/vdb/blog/?date=2012-11-27#1721
2012-11-27T22:18:24Z
<p>The <a href="http://www.ldbc.eu/" id="link-id0x7f4740087de8">LDBC</a> Technical User Community (TUC) had <a href="http://www.ldbc.eu/events/1st-ldbc-technical-user-community-meeting" id="link-id0x7f47800d4b48">its initial meeting</a> in Barcelona last week.</p> <p>First we wish to thank the many end user organizations that were present. This clearly validates the project's mission and demonstrates that there is acute awareness of the need for better metrics in the field. In the following, I will summarize the requirements that were brought forth.</p> <ul> <li> <p> <b>Scale out</b> - There was near unanimity among users that even if present workloads could be handled on single servers, a scale-out growth path was highly desirable. On the other hand, some applications were scale-out based from the get go. Even when not actually used, a scale-out capability is felt to be an insurance against future need.</p> </li> <li> <p> <b>Making limits explicit</b> - How far can this technology go? <a href="http://dbpedia.org/page/Benchmark_%28computing%29" id="link-id0x7f4748067b58">Benchmarks</a> need to demonstrate at what scales the products being considered work best, and where they will grind to a halt. Also, the impact of scale-out on performance needs to be made clear. The cost of solutions at different scales must be made explicit.</p> <p>Many of these requirements will be met by simply following <a href="http://www.tpc.org/" id="link-id0x7f4751c0fa08">TPC</a> practices. Now, vendors cannot be expected to publish numbers for cases where their products fail, but they do have incentives for publishing numbers on large data, and at least giving a price/performance point that exceeds most user needs.</p> </li> <li> <p> <b>Fault tolerance and operational characteristics</b> - Present day benchmarks (e.g., the TPC ones) hardly address operational aspects that most enterprise deployments will encounter. This was already stated by <a href="http://dbpedia.org/page/Michael_Stonebraker" id="link-id0x7f47800d5a98">Michael Stonebraker</a> at the first TPC performance evaluation workshop some years back at <a href="http://www.vldb.org/archives/website/2009/" id="link-id0x7f4748066bd8">VLDB in Lyon</a>. Users want to know the price/performance impact of making fault-tolerant systems and wish to have metrics for things like backup and bulk load under online conditions. A need to operate across multiple geographies was present in more than one use case, thus requiring a degree of asynchronous replication such as log shipping.</p> </li> <li> <p> <b>Update-intensive workloads</b> - Unlike one might think, <a href="http://dbpedia.org/page/Resource_Description_Framework" id="link-id0x7f4782546868">RDF</a> uses are not primarily load-once-plus-lookup. Freshness of data creates value, and databases, even if they are warehouses in character, need to be kept up to date much better than just by periodic reload. Online updates may be small, as for example refreshing news feeds or web crawls, where the unit of update is small but updates are many, but also replacing reference data sets of hundreds of millions of triples. The latter requirement exceeds what is practical in a single transaction. <a href="http://dbpedia.org/page/ACID" id="link-id0x7f4740070db8">ACID</a> was generally desired, with some interest also in eventual consistency. We did not get use cases with much repeatable read (e.g., updating account balances), but rather atomic and durable replacement of sets of statements.</p> </li> <li> <p> <b>Inference</b> - Class and property hierarchies were common, followed by use of transitivity. <code><a href="http://www.w3.org/TR/owl2-overview/" id="link-id0x7f47480c5b48">owl:sameAs</a></code> was not in much use, being <a href="http://events.linkeddata.org/ldow2010/papers/ldow2010_paper09.pdf" id="link-id0x7f474007a198">too dangerous</a>, i.e., a single statement may potentially have huge effect and produce unpredictable sets of properties for instances, for which applications are not prepared. Beyond these, the wishes for inference, with use cases ranging from medicine to forensics, were outside of the OWL domain. These typically involved probability scores adding up the joint occurrence of complex criteria with some numeric computation (e.g. time intervals, geography, etc.).</p> <p>As materialization of forward closure is the prevalent mode of implementing inference in RDF, users wished to have a measure of its cost in space and time, especially under online-update loads.</p> </li> <li> <p> <b>Text, XML, and Geospatial</b> - There is no online application that does not have text search. In publishing, this is hardly ever provided by an RDF store, even if there is one in the mix. Even so, there is an understandable desire to consolidate systems, i.e., to not have an XML database for content and a separate RDF database for metadata. Also, many applications have a geospatial element. One wish was to combine XPATH/XQuery with SPARQL, and it was implied that query optimization should create good plans under these conditions.</p> <p>There was extensive discussion especially on benchmarking full-text. Such a benchmark would need to address the quality of relevance ranking. Doing new work in this space is clearly out of scope for LDBC, but an IR benchmark could be reused as an add-on to provide a quality score. The performance score would come from the LDBC side of the benchmark. Now, many of the applications of text (e.g., news) might not even sort on text match score, but rather by time. Also if the text search is applied to metadata like labels or URI strings, the quality of a match is a non-issue, as there is no document context.</p> </li> <li> <p> <b>Data integration</b> - Almost all applications had some element of data integration. Indeed, if one uses RDF in the first place, the motivation usually has to do with schema flexibility. Having a relational schema for everything is often seen to be too hard to maintain and to lead to too much development time before an initial version of an application or answer of a business question. Data integration is everywhere but stays elusive for benchmarking. Every time it is different and most vendors present do not offer <a href="http://virtuoso.openlinksw.com/middleware/" id="link-id0x7f4728113608">products for this specific need</a>. Many ideas were presented, including using SPARQL for entity resolution, and for checking consistency of an integration result.</p> </li> </ul> <p>A central issue of benchmark design is having an understandable <a href="http://dbpedia.org/page/Software_metric" id="link-id0x7f47280fac58">metric</a>. People cannot make sense of more than a few figures. The TPC practice of throughput at scale and price per unit of throughput at scale is a successful example. However, it may be difficult to agree on relative weights of components if a metric is an aggregate of too many things. Also, if a benchmark has too many optional parts, metrics easily become too complicated. On the other hand, requiring too many features (e.g. XML, full text, geospatial) restricts the number of possible participants.</p> <p>To stimulate innovation, a benchmark needs to be difficult but restricted to a specific domain. <a href="http://www.tpc.org/tpch/" id="link-id0x7f47400b3ad8">TPC-H</a> is a good example, favoring specialized systems built for analytics alone. To be a predictor of total cost and performance in a complex application, a benchmark must include much more functionality, and will favor general purpose systems that do many things but are not necessarily outstanding in any single aspect.</p> <p>After 1-1/2 days with users, the project team met to discuss actual benchmark task forces to be started. The conclusion was that work would initially proceed around two use cases: publishing, and social networks. The present use of RDF by the <a href="http://dbpedia.org/page/BBC" id="link-id0x7f478009ee38">BBC</a> and the <a href="http://dbpedia.org/page/Press_Association" id="link-id0x7f478014c998">Press Association</a> provides the background scenario for the publishing benchmark, and the work carried out around the <a href="http://sourceforge.net/projects/sibenchmark/" id="link-id0x7f47480ae1d8">Social Intelligence Benchmark (SIB)</a> in <a href="http://lod2.eu/" id="link-id0x7f4780f71458">LOD2</a> will provide a starting point for the social network benchmark. Additionally, user scenarios from the <a href="http://dbpedia.org/page/DEX_%28Graph_database%29" id="link-id0x7f47400704e8">DEX graph database</a> user base will help shape the SN workload.</p> <p>A data integration task force needs more clarification, but work in this direction is in progress.</p> <p>In practice, driving progress needs well-focused benchmarks with special trick questions intended to stress specific aspects of a database engine. Providing an overall perspective on cost and online operations needs a broad mix of features to be covered.</p> These needs will be reconciled by having many metrics inside a single use case, i.e., a social network data set can be used for transactional updates, for lookup queries, for graph analytics, and for TPC-H style business intelligence questions, especially if integrated with another more-relational dataset. Thus there will be a mix of metrics, from transactions to analytics, with single and multiuser workloads. Whether these are packaged as separate benchmarks, or as optional sections of one, remains to be seen.
2012-11-27T23:18:24+01:00
LDBC Technical User Community Meeting
http://www.openlinksw.com:443/weblog/oerling/?date=2012-11-27#1720
2012-11-27T22:17:48Z
<p>The <a href="http://www.ldbc.eu/" id="link-id0x7f4740087de8">LDBC</a> Technical User Community (TUC) had <a href="http://www.ldbc.eu/events/1st-ldbc-technical-user-community-meeting" id="link-id0x7f47800d4b48">its initial meeting</a> in Barcelona last week.</p> <p>First we wish to thank the many end user organizations that were present. This clearly validates the project's mission and demonstrates that there is acute awareness of the need for better metrics in the field. In the following, I will summarize the requirements that were brought forth.</p> <ul> <li> <p> <b>Scale out</b> - There was near unanimity among users that even if present workloads could be handled on single servers, a scale-out growth path was highly desirable. On the other hand, some applications were scale-out based from the get go. Even when not actually used, a scale-out capability is felt to be an insurance against future need.</p> </li> <li> <p> <b>Making limits explicit</b> - How far can this technology go? <a href="http://dbpedia.org/page/Benchmark_%28computing%29" id="link-id0x7f4748067b58">Benchmarks</a> need to demonstrate at what scales the products being considered work best, and where they will grind to a halt. Also, the impact of scale-out on performance needs to be made clear. The cost of solutions at different scales must be made explicit.</p> <p>Many of these requirements will be met by simply following <a href="http://www.tpc.org/" id="link-id0x7f4751c0fa08">TPC</a> practices. Now, vendors cannot be expected to publish numbers for cases where their products fail, but they do have incentives for publishing numbers on large data, and at least giving a price/performance point that exceeds most user needs.</p> </li> <li> <p> <b>Fault tolerance and operational characteristics</b> - Present day benchmarks (e.g., the TPC ones) hardly address operational aspects that most enterprise deployments will encounter. This was already stated by <a href="http://dbpedia.org/page/Michael_Stonebraker" id="link-id0x7f47800d5a98">Michael Stonebraker</a> at the first TPC performance evaluation workshop some years back at <a href="http://www.vldb.org/archives/website/2009/" id="link-id0x7f4748066bd8">VLDB in Lyon</a>. Users want to know the price/performance impact of making fault-tolerant systems and wish to have metrics for things like backup and bulk load under online conditions. A need to operate across multiple geographies was present in more than one use case, thus requiring a degree of asynchronous replication such as log shipping.</p> </li> <li> <p> <b>Update-intensive workloads</b> - Unlike one might think, <a href="http://dbpedia.org/page/Resource_Description_Framework" id="link-id0x7f4782546868">RDF</a> uses are not primarily load-once-plus-lookup. Freshness of data creates value, and databases, even if they are warehouses in character, need to be kept up to date much better than just by periodic reload. Online updates may be small, as for example refreshing news feeds or web crawls, where the unit of update is small but updates are many, but also replacing reference data sets of hundreds of millions of triples. The latter requirement exceeds what is practical in a single transaction. <a href="http://dbpedia.org/page/ACID" id="link-id0x7f4740070db8">ACID</a> was generally desired, with some interest also in eventual consistency. We did not get use cases with much repeatable read (e.g., updating account balances), but rather atomic and durable replacement of sets of statements.</p> </li> <li> <p> <b>Inference</b> - Class and property hierarchies were common, followed by use of transitivity. <code><a href="http://www.w3.org/TR/owl2-overview/" id="link-id0x7f47480c5b48">owl:sameAs</a></code> was not in much use, being <a href="http://events.linkeddata.org/ldow2010/papers/ldow2010_paper09.pdf" id="link-id0x7f474007a198">too dangerous</a>, i.e., a single statement may potentially have huge effect and produce unpredictable sets of properties for instances, for which applications are not prepared. Beyond these, the wishes for inference, with use cases ranging from medicine to forensics, were outside of the OWL domain. These typically involved probability scores adding up the joint occurrence of complex criteria with some numeric computation (e.g. time intervals, geography, etc.).</p> <p>As materialization of forward closure is the prevalent mode of implementing inference in RDF, users wished to have a measure of its cost in space and time, especially under online-update loads.</p> </li> <li> <p> <b>Text, XML, and Geospatial</b> - There is no online application that does not have text search. In publishing, this is hardly ever provided by an RDF store, even if there is one in the mix. Even so, there is an understandable desire to consolidate systems, i.e., to not have an XML database for content and a separate RDF database for metadata. Also, many applications have a geospatial element. One wish was to combine XPATH/XQuery with SPARQL, and it was implied that query optimization should create good plans under these conditions.</p> <p>There was extensive discussion especially on benchmarking full-text. Such a benchmark would need to address the quality of relevance ranking. Doing new work in this space is clearly out of scope for LDBC, but an IR benchmark could be reused as an add-on to provide a quality score. The performance score would come from the LDBC side of the benchmark. Now, many of the applications of text (e.g., news) might not even sort on text match score, but rather by time. Also if the text search is applied to metadata like labels or URI strings, the quality of a match is a non-issue, as there is no document context.</p> </li> <li> <p> <b>Data integration</b> - Almost all applications had some element of data integration. Indeed, if one uses RDF in the first place, the motivation usually has to do with schema flexibility. Having a relational schema for everything is often seen to be too hard to maintain and to lead to too much development time before an initial version of an application or answer of a business question. Data integration is everywhere but stays elusive for benchmarking. Every time it is different and most vendors present do not offer <a href="http://virtuoso.openlinksw.com/middleware/" id="link-id0x7f4728113608">products for this specific need</a>. Many ideas were presented, including using SPARQL for entity resolution, and for checking consistency of an integration result.</p> </li> </ul> <p>A central issue of benchmark design is having an understandable <a href="http://dbpedia.org/page/Software_metric" id="link-id0x7f47280fac58">metric</a>. People cannot make sense of more than a few figures. The TPC practice of throughput at scale and price per unit of throughput at scale is a successful example. However, it may be difficult to agree on relative weights of components if a metric is an aggregate of too many things. Also, if a benchmark has too many optional parts, metrics easily become too complicated. On the other hand, requiring too many features (e.g. XML, full text, geospatial) restricts the number of possible participants.</p> <p>To stimulate innovation, a benchmark needs to be difficult but restricted to a specific domain. <a href="http://www.tpc.org/tpch/" id="link-id0x7f47400b3ad8">TPC-H</a> is a good example, favoring specialized systems built for analytics alone. To be a predictor of total cost and performance in a complex application, a benchmark must include much more functionality, and will favor general purpose systems that do many things but are not necessarily outstanding in any single aspect.</p> <p>After 1-1/2 days with users, the project team met to discuss actual benchmark task forces to be started. The conclusion was that work would initially proceed around two use cases: publishing, and social networks. The present use of RDF by the <a href="http://dbpedia.org/page/BBC" id="link-id0x7f478009ee38">BBC</a> and the <a href="http://dbpedia.org/page/Press_Association" id="link-id0x7f478014c998">Press Association</a> provides the background scenario for the publishing benchmark, and the work carried out around the <a href="http://sourceforge.net/projects/sibenchmark/" id="link-id0x7f47480ae1d8">Social Intelligence Benchmark (SIB)</a> in <a href="http://lod2.eu/" id="link-id0x7f4780f71458">LOD2</a> will provide a starting point for the social network benchmark. Additionally, user scenarios from the <a href="http://dbpedia.org/page/DEX_%28Graph_database%29" id="link-id0x7f47400704e8">DEX graph database</a> user base will help shape the SN workload.</p> <p>A data integration task force needs more clarification, but work in this direction is in progress.</p> <p>In practice, driving progress needs well-focused benchmarks with special trick questions intended to stress specific aspects of a database engine. Providing an overall perspective on cost and online operations needs a broad mix of features to be covered.</p> These needs will be reconciled by having many metrics inside a single use case, i.e., a social network data set can be used for transactional updates, for lookup queries, for graph analytics, and for TPC-H style business intelligence questions, especially if integrated with another more-relational dataset. Thus there will be a mix of metrics, from transactions to analytics, with single and multiuser workloads. Whether these are packaged as separate benchmarks, or as optional sections of one, remains to be seen.
2012-11-27T23:17:48+01:00
Developer Recruitment Exercise
http://www.openlinksw.com:443/blog/vdb/blog/?date=2012-08-16#1719
2012-08-16T19:28:03Z
<p>The specification of the exercise referred to in <a href="http://www.openlinksw.com/weblog/oerling/?id=1717" id="link-id0x1bc4d790">the previous post</a> may be found below.</p> <p>Questions on the exercise can be sent to <a href="mailto:hwilliams@openlinksw.com?subject=2012-08%20Virtuoso%20Developer%20Exercise" id="link-id0x1cb00cc0">the email specified in the previous post</a>. I may schedule a phone call to answer questions based on the initial email contact.</p> <p>We seek to have all applicants complete the exercise before October 1.</p> <h2>General</h2> <p>The exercise consists of implementing a part of the <a href="http://dbpedia.org/resource/TPC-C" id="link-id0x1c158760">TPC-C</a> workload in memory, in <a href="http://dbpedia.org/resource/C_(programming_language)" id="link-id0x1b38bec0"><code>C</code></a> or <a href="http://dbpedia.org/page/C++" id="link-id0x1119e720"><code>C++</code></a>. TPC-C is the long-time industry standard benchmark for transaction processing performance. We use this as a starting point for an exercise for assessing developer skill level in writing heavily multithreaded, performance-critical code.</p> <p>The application performs a series of transactions against an in-memory database, encountering lock contention and occasional deadlocks. The application needs to provide atomicity, consistency, and isolation for transactions. The task consists of writing the low-level data structures for storing the memory-resident database and for managing concurrency, including lock queueing, deadlock detection, and commit/rollback. The solutions are evaluated based on their actual measured multithreaded performance on commodity servers, e.g., 8- or 12-cores of Intel Xeon.</p> <p>OpenLink provides the code for data generation and driving the test. This is part of the TPC-C kit in <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSIndex" id="link-id0x1be9fab8">Virtuoso Open Source</a>. The task is to replace the SQL API calls with equivalent in-process function calls against the in-memory database developed as part of the exercise.</p> <h2>Rules</h2> <p>We are aware that the best solution to the problem may be running transactions single-threaded against in-memory hash tables without any concurrency control. The application data may be partitioned so that a single transaction can be in most cases assigned to a partition, which it will get for itself for the few microseconds it takes to do its job. <b>For this exercise, this solution is explicitly ruled out.</b> The application must demonstrate shared access to data, with a transaction holding multiple concurrent locks and being liable to deadlock.</p> <p>TPC-C can be written so as to avoid deadlocks by always locking in a certain order. <b>This is also expressly prohibited;</b> in specific, the stock rows of a new order transaction must be locked in the order they are specified in the invocation. In application terms this makes no sense, but for purposes of the exercise this will serve as a natural source of deadlocks.</p> <h2>Parameters</h2> <p>The application needs to offer an interactive or scripted interface (command line is OK) which provides the following operations:</p> <ul> <li> <p>Clear and initialize a database of n warehouses.</p> </li> <li> <p>Run <i>n</i> threads, each doing <i>m</i> new order transactions. Each thread has a home warehouse and occasionally accesses other warehouse's data. This reports the real time elapsed and the number of retries arising from deadlocks. </p> </li> <li> <p>Check the consistency between the <code>stock</code>, <code>orders</code>, and <code>order_line</code> data structures.</p> </li> <li> <p>Report system status such as clocks spent waiting for specific mutexes. This is supplied as part of the OpenLink library used by the data generator.</p> </li> </ul> <h2>Data Structures</h2> <p>The transactions are written as <code>C</code> functions. The data is represented as <code>C</code> structs, and tree indices or hash tables are used for value-based access to the structures by key. The application has no persistent storage. The structures reference each other by the key values as in the database, so no direct pointers. The key values are to be translated into pointers with a hash table or other index-like structure.</p> <p>The application must be thread-safe, and transactions must be able to roll back. Transactions will sometimes wait for each other in updating shared resources such as stock or district or warehouse balances. The application must be written so as to implement fine-grained locking, and each transaction must be able to hold multiple locks. The application must be able to detect deadlocks. For deadlock recovery, it is acceptable to abort the transaction that detects the deadlock.</p> <p> <code>C++</code> template libraries may be used but one must pay attention to their efficiency.</p> <p>The new order transaction is the only required transaction.</p> <p>All numbers can be represented as integers. This holds equally for key columns as for monetary amounts.</p> <p>All index structures (e.g., hash tables) in the application must be thread safe, so that an insert would be safe with concurrent access or concurrent inserts. This holds also for index structures for tables which do not get inserts in the test (e.g. item, customer, stock, etc.).</p> <p>A sequence object must not be used for assigning new values to the <code>O_ID</code> column of <code>ORDERS</code>. These values must come from the <code>D_NEXT_O_ID</code> column of the <code>DISTRICT</code> table. If a new order transaction rolls back, its update of <code>D_NEXT_O_ID</code> is also rolled back. This causes <code>O_ID</code> values to always be consecutive within a district.</p> <h2>TPC-C Functionality</h2> <p>The application must implement the TPC-C new order transaction in full. This must not avoid deadlocks by ordering locking on stock rows. See the rules section.</p> <p>The transaction must have the semantics specified in TPC-C, except for durability.</p> <h2>Supporting Files</h2> <p>The test driver calling the transaction procedures is in <code>tpccodbc.c</code>. This can be reused so as to call the transaction procedure in process instead of the ODBC exec.</p> <p>The user interface may be a command line menu with run options for different numbers of transactions with different thread counts and an option for integrity check.</p> <p>The integrity check consists of verifying <code>s_cnt_order</code> against the orders and checking that <code>max (O_ID)</code> and <code>D_NEXT_O_ID</code> match within each district.</p> <p>Running the application should give different statistics such as CPU%, cumulative time spent waiting for locks, etc. The <code>rdtsc</code> instruction can be used for getting clock counts for timing.</p> <h2>Points to Note</h2> <p>This section summarizes some of the design patterns and coding tricks we expect to see in a solution to the exercise. These may seem self-evident to some, but experience indicates that this is not universally so.</p> <ul> <li> <p>The TPC-C transaction profile for new order specifies a semantics for the operation. The order of locking is left to the implementation as long as the semantics are in effect. The application will be tested with many clients on the same warehouse, running as fast as they can. So lock contention is expected. Therefore, the transaction should be written so as to acquire the locks with the greatest contention as late as possible. No locks need be acquired for the item table since none of the transactions will update it.</p> </li> <li> <p>For implementing locks, using a mutex to serialize access to application resources is not enough. Many locks will be acquired by each transaction, in an unpredictable order. Unless explicit queueing for locks is implemented with deadlock detection, the application will not work.</p> </li> <li> <p>If waiting for a mutex causes the operating system to stop a thread, even when there are cores free, the latency is multiple microseconds, even if the mutex is released by its owner on the next cycle after the waiting thread is suspended. This will destroy any benefit from parallelism unless one is very careful. Programmers do not seem to instinctively know this.</p> </li> </ul> <p>Therefore any structure to which access must be serialized (e.g. hash tables, locks, etc.) needs to be protected by a mutex but must be partitioned so that there are tens or hundreds of mutexes depending on which section of the structure one is accessing.</p> <p>Submissions that protect a hash table or other index-like structure for a whole application table with a single mutex or <code>rw</code> lock will be discarded off the bat.</p> <p>Even while using many mutexes, one must hold them for a minimum of time. When accessing a hash table, do the invariant parts first; acquire the mutex after that. For example, if you calculate the hash number after acquiring the mutex for the hash table, the submission will be rejected.</p> <p>The TPC-C application has some local and some scattered access. Orders are local, and stock and item lines are scattered. When doing scattered memory accesses, the program should be written so that the CPU will, from a single thread, have multiple concurrent cache misses in flight at all times. So, when accessing 10 stock lines, calculate the hash numbers first; then access the memory, deferring any branches based on the accessed values. In this way, out of order execution will miss the CPU cache for many independent addresses in parallel. One can use the gcc <code>__builtin_prefetch</code> primitive, or simply write the program so as to have mutually data-independent memory accesses in close proximity.</p> <p>For detecting deadlocks, a global transaction wait graph may have to be maintained. This will need to be maintained in a serialized manner. If many threads access this, the accesses must be serialized on a global mutex. This may be very bad if the deadlock detection takes a long time. Alternately, the wait graph may be maintained on another thread. The thread will get notices of waits and transacts from worker threads with some delay. Having spotted a cycle, it may kill one or another party. This will require some inter-thread communication. The submission may address this matter in any number of ways.</p> <p>However, just acquiring a lock without wait must not involve getting a global mutex. Going to wait will have to do so, were it only for queueing a notice to a monitor thread. Using a socket-to-self might appear to circumvent this, but the communication stack will have mutexes inside so this is no better.</p> <h2>Evaluation Criteria</h2> <p>The exercise will be evaluated based on the run time performance, especially multicore scalability of the result.</p> <p>Extra points are not given for implementing interfaces or for being object oriented. Interfaces, templates, and objects are not forbidden as such, but their cost must not exceed the difference between getting an address from a virtual table and calling a function directly.</p> <p>The locking implementation must be correct. It can be limited to exclusive locks and need not support isolation other than <code>repeatable read</code>. Running the application must demonstrate deadlocks and working recovery from these.</p> <h2>Code and Libraries To Be Used</h2> <p>The TPC-C data generator and test driver are in the <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSGIT" id="link-id0x1be07088">Virtuoso Open Source distribution</a>, in the files <code>binsrc/tests/tpcc*.c</code> and files included from these. You can make the exercise in the same directory and just alter the files or make script. The application is standalone and has no other relation to the Virtuoso code. The <code>libsrc/Thread</code> threading wrappers may be used. If not using these, make a wrapper similar to <code>mutex_enter</code> when <code>MTX_METER</code> is defined so that it counts the waits and clocks spent during wait. Also have a report like that in <code>mutex_stat()</code> for the mutex wait frequency and duration.</p>
2015-08-21T14:24:58.140364-04:00
Developer Recruitment Exercise
http://www.openlinksw.com:443/weblog/oerling/?date=2012-08-16#1718
2012-08-16T19:26:37Z
<p>The specification of the exercise referred to in <a href="http://www.openlinksw.com/weblog/oerling/?id=1714" id="link-id0x2c0160c8">the previous post</a> may be found below.</p> <p>Questions on the exercise can be sent to <a href="mailto:hwilliams@openlinksw.com?subject=2012-08%20Virtuoso%20Developer%20Exercise" id="link-id0x2a668288">the email specified in the previous post</a>. I may schedule a phone call to answer questions based on the initial email contact.</p> <p>We seek to have all applicants complete the exercise before October 1.</p> <h2>General</h2> <p>The exercise consists of implementing a part of the <a href="http://dbpedia.org/resource/TPC-C" id="link-id0x2bbe9040">TPC-C</a> workload in memory, in <a href="http://dbpedia.org/resource/C_(programming_language)" id="link-id0x2a660ea0"><code>C</code></a> or <a href="http://dbpedia.org/page/C++" id="link-id0x29c29db0"><code>C++</code></a>. TPC-C is the long-time industry standard benchmark for transaction processing performance. We use this as a starting point for an exercise for assessing developer skill level in writing heavily multithreaded, performance-critical code.</p> <p>The application performs a series of transactions against an in-memory database, encountering lock contention and occasional deadlocks. The application needs to provide atomicity, consistency, and isolation for transactions. The task consists of writing the low-level data structures for storing the memory-resident database and for managing concurrency, including lock queueing, deadlock detection, and commit/rollback. The solutions are evaluated based on their actual measured multithreaded performance on commodity servers, e.g., 8- or 12-cores of Intel Xeon.</p> <p>OpenLink provides the code for data generation and driving the test. This is part of the TPC-C kit in <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSIndex" id="link-id0x2bf5dac8">Virtuoso Open Source</a>. The task is to replace the SQL API calls with equivalent in-process function calls against the in-memory database developed as part of the exercise.</p> <h2>Rules</h2> <p>We are aware that the best solution to the problem may be running transactions single-threaded against in-memory hash tables without any concurrency control. The application data may be partitioned so that a single transaction can be in most cases assigned to a partition, which it will get for itself for the few microseconds it takes to do its job. <b>For this exercise, this solution is explicitly ruled out.</b> The application must demonstrate shared access to data, with a transaction holding multiple concurrent locks and being liable to deadlock.</p> <p>TPC-C can be written so as to avoid deadlocks by always locking in a certain order. <b>This is also expressly prohibited;</b> in specific, the stock rows of a new order transaction must be locked in the order they are specified in the invocation. In application terms this makes no sense, but for purposes of the exercise this will serve as a natural source of deadlocks.</p> <h2>Parameters</h2> <p>The application needs to offer an interactive or scripted interface (command line is OK) which provides the following operations:</p> <ul> <li> <p>Clear and initialize a database of n warehouses.</p> </li> <li> <p>Run <i>n</i> threads, each doing <i>m</i> new order transactions. Each thread has a home warehouse and occasionally accesses other warehouse's data. This reports the real time elapsed and the number of retries arising from deadlocks. </p> </li> <li> <p>Check the consistency between the <code>stock</code>, <code>orders</code>, and <code>order_line</code> data structures.</p> </li> <li> <p>Report system status such as clocks spent waiting for specific mutexes. This is supplied as part of the OpenLink library used by the data generator.</p> </li> </ul> <h2>Data Structures</h2> <p>The transactions are written as <code>C</code> functions. The data is represented as <code>C</code> structs, and tree indices or hash tables are used for value-based access to the structures by key. The application has no persistent storage. The structures reference each other by the key values as in the database, so no direct pointers. The key values are to be translated into pointers with a hash table or other index-like structure.</p> <p>The application must be thread-safe, and transactions must be able to roll back. Transactions will sometimes wait for each other in updating shared resources such as stock or district or warehouse balances. The application must be written so as to implement fine-grained locking, and each transaction must be able to hold multiple locks. The application must be able to detect deadlocks. For deadlock recovery, it is acceptable to abort the transaction that detects the deadlock.</p> <p> <code>C++</code> template libraries may be used but one must pay attention to their efficiency.</p> <p>The new order transaction is the only required transaction.</p> <p>All numbers can be represented as integers. This holds equally for key columns as for monetary amounts.</p> <p>All index structures (e.g., hash tables) in the application must be thread safe, so that an insert would be safe with concurrent access or concurrent inserts. This holds also for index structures for tables which do not get inserts in the test (e.g. item, customer, stock, etc.).</p> <p>A sequence object must not be used for assigning new values to the <code>O_ID</code> column of <code>ORDERS</code>. These values must come from the <code>D_NEXT_O_ID</code> column of the <code>DISTRICT</code> table. If a new order transaction rolls back, its update of <code>D_NEXT_O_ID</code> is also rolled back. This causes <code>O_ID</code> values to always be consecutive within a district.</p> <h2>TPC-C Functionality</h2> <p>The application must implement the TPC-C new order transaction in full. This must not avoid deadlocks by ordering locking on stock rows. See the rules section.</p> <p>The transaction must have the semantics specified in TPC-C, except for durability.</p> <h2>Supporting Files</h2> <p>The test driver calling the transaction procedures is in <code>tpccodbc.c</code>. This can be reused so as to call the transaction procedure in process instead of the ODBC exec.</p> <p>The user interface may be a command line menu with run options for different numbers of transactions with different thread counts and an option for integrity check.</p> <p>The integrity check consists of verifying <code>s_cnt_order</code> against the orders and checking that <code>max (O_ID)</code> and <code>D_NEXT_O_ID</code> match within each district.</p> <p>Running the application should give different statistics such as CPU%, cumulative time spent waiting for locks, etc. The <code>rdtsc</code> instruction can be used for getting clock counts for timing.</p> <h2>Points to Note</h2> <p>This section summarizes some of the design patterns and coding tricks we expect to see in a solution to the exercise. These may seem self-evident to some, but experience indicates that this is not universally so.</p> <ul> <li> <p>The TPC-C transaction profile for new order specifies a semantics for the operation. The order of locking is left to the implementation as long as the semantics are in effect. The application will be tested with many clients on the same warehouse, running as fast as they can. So lock contention is expected. Therefore, the transaction should be written so as to acquire the locks with the greatest contention as late as possible. No locks need be acquired for the item table since none of the transactions will update it.</p> </li> <li> <p>For implementing locks, using a mutex to serialize access to application resources is not enough. Many locks will be acquired by each transaction, in an unpredictable order. Unless explicit queueing for locks is implemented with deadlock detection, the application will not work.</p> </li> <li> <p>If waiting for a mutex causes the operating system to stop a thread, even when there are cores free, the latency is multiple microseconds, even if the mutex is released by its owner on the next cycle after the waiting thread is suspended. This will destroy any benefit from parallelism unless one is very careful. Programmers do not seem to instinctively know this.</p> </li> </ul> <p>Therefore any structure to which access must be serialized (e.g. hash tables, locks, etc.) needs to be protected by a mutex but must be partitioned so that there are tens or hundreds of mutexes depending on which section of the structure one is accessing.</p> <p>Submissions that protect a hash table or other index-like structure for a whole application table with a single mutex or <code>rw</code> lock will be discarded off the bat.</p> <p>Even while using many mutexes, one must hold them for a minimum of time. When accessing a hash table, do the invariant parts first; acquire the mutex after that. For example, if you calculate the hash number after acquiring the mutex for the hash table, the submission will be rejected.</p> <p>The TPC-C application has some local and some scattered access. Orders are local, and stock and item lines are scattered. When doing scattered memory accesses, the program should be written so that the CPU will, from a single thread, have multiple concurrent cache misses in flight at all times. So, when accessing 10 stock lines, calculate the hash numbers first; then access the memory, deferring any branches based on the accessed values. In this way, out of order execution will miss the CPU cache for many independent addresses in parallel. One can use the gcc <code>__builtin_prefetch</code> primitive, or simply write the program so as to have mutually data-independent memory accesses in close proximity.</p> <p>For detecting deadlocks, a global transaction wait graph may have to be maintained. This will need to be maintained in a serialized manner. If many threads access this, the accesses must be serialized on a global mutex. This may be very bad if the deadlock detection takes a long time. Alternately, the wait graph may be maintained on another thread. The thread will get notices of waits and transacts from worker threads with some delay. Having spotted a cycle, it may kill one or another party. This will require some inter-thread communication. The submission may address this matter in any number of ways.</p> <p>However, just acquiring a lock without wait must not involve getting a global mutex. Going to wait will have to do so, were it only for queueing a notice to a monitor thread. Using a socket-to-self might appear to circumvent this, but the communication stack will have mutexes inside so this is no better.</p> <h2>Evaluation Criteria</h2> <p>The exercise will be evaluated based on the run time performance, especially multicore scalability of the result.</p> <p>Extra points are not given for implementing interfaces or for being object oriented. Interfaces, templates, and objects are not forbidden as such, but their cost must not exceed the difference between getting an address from a virtual table and calling a function directly.</p> <p>The locking implementation must be correct. It can be limited to exclusive locks and need not support isolation other than <code>repeatable read</code>. Running the application must demonstrate deadlocks and working recovery from these.</p> <h2>Code and Libraries To Be Used</h2> <p>The TPC-C data generator and test driver are in the <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSGIT" id="link-id0x2b5bd6b8">Virtuoso Open Source distribution</a>, in the files <code>binsrc/tests/tpcc*.c</code> and files included from these. You can make the exercise in the same directory and just alter the files or make script. The application is standalone and has no other relation to the Virtuoso code. The <code>libsrc/Thread</code> threading wrappers may be used. If not using these, make a wrapper similar to <code>mutex_enter</code> when <code>MTX_METER</code> is defined so that it counts the waits and clocks spent during wait. Also have a report like that in <code>mutex_stat()</code> for the mutex wait frequency and duration.</p>
2012-08-16T15:26:37.000001-04:00
Developer Opportunities at OpenLink Software
http://www.openlinksw.com:443/blog/vdb/blog/?date=2012-08-07#1717
2012-08-07T17:21:52Z
<p>If it is advanced database technology, you will get to do it with us.</p> <p>We are looking for exceptional talent to implement some of the hardest stuff in the industry. This ranges from new approaches to query optimization; to parallel execution (both scale up and scale out); to elastic cloud deployments and self-managing, self-tuning, fault-tolerant databases. We are most familiar to the RDF world, but also have full SQL support, and the present work will serve both use cases equally.</p> <p>We are best known in the realms of high-performance database connectivity middleware and massively-scalable Linked-Data-oriented graph-model DBMS technology.</p> <p>We have the basics -- SQL and SPARQL, column store, vectored execution, cost based optimization, parallel execution (local and cluster), and so forth. In short, we have everything you would expect from a DBMS. We do transactions as well as analytics, but the greater challenges at present are on the analytics side.</p> <p>You will be working with my team covering:</p> <ul> <li> <p>Adaptive query optimization -- interleaving execution and optimization, so as to always make the correct plan choices based on actual data characteristics</p> </li> <li> <p>Self-managing cloud deployments for elastic big data -- clusters that can grow themselves and redistribute load, recover from failures, etc.</p> </li> <li> <p>Developing and analyzing new benchmarks for RDF and graph databases</p> </li> <li> <p>Embedding complex geospatial reasoning inside the database engine. We have the basic R-tree and the OGC geometry data types; now we need to go beyond this</p> </li> <li> <p>Every type of SQL optimizer and execution engine trick that serves to optimize for TPC-H and DS.</p> </li> </ul> <p>What do I mean by really good? It boils down to being a smart and fast programmer. We have over the years talked to people, including many who have worked on DBMS programming, and found that they actually know next to nothing of database science. For example, they might not know what a hash join is. Or they might not know that interprocess latency is in the tens of microseconds even within one box, and that in that time one can do tens of index lookups. Or they might not know that blocking on a mutex kills.</p> <p>If you do core database work, we want you to know how many CPU cache misses you will have in flight at any point of the algorithm, and how many clocks will be spent waiting for them at what points. Same for distributed execution: The only way a cluster can perform is having max messages with max payload per message in flight at all times.</p> <p>These are things that can be learned. So I do not necessarily expect that you have in-depth experience of these, especially since most developer jobs are concerned with something else. You may have to unlearn the bad habit of putting interfaces where they do not belong, for example. Or to learn that if there is an interface, then it must pass as much data as possible in one go.</p> <p>Talent is the key. You need to be a self-starter with a passion for technology and have competitive drive. These can be found in many guises, so we place very few limits on the rest. If you show you can learn and code fast, we don't necessarily care about academic or career histories. You can be located anywhere in the world, and you can work from home. There may be some travel but not very much.</p> <p>In the context of <a href="http://lod2.eu/" id="link-id0x7719ea0">EU FP7 projects</a>, we are working with some of the best minds in database, including <a href="http://homepages.cwi.nl/~boncz/" id="link-id0x21d67d80">Peter Boncz</a> of CWI and VU Amsterdam (MonetDB, VectorWise) and <a href="http://www.mpi-inf.mpg.de/~neumann/" id="link-id0x2192d900">Thomas Neumann</a> of Technical University of Munich (RDF3X, HYPER). This is an extra guarantee that you will be working on the most relevant problems in database, informed by the results of the very best work to date.</p> <p>For more background, please see the IEEE Computer Society <i>Bulletin of the Technical Committee on Data Engineering,</i> <a href="http://sites.computer.org/debull/A12mar/issue1.htm" id="link-id0x7ca1d20">Special Issue on Column Store Systems</a>.</p> <p>All articles and references therein are relevant for the job. Be sure to read the CWI work on run time optimization (ROX), cracking, and recycling. Do not miss the many papers on architecture-conscious, cache-optimized algorithms; see the VectorWise and MonetDB articles in the bulletin for extensive references.</p> <p>If you are interested in an opportunity with us, we will ask you to do a little exercise in multithreaded, performance-critical coding, to be detailed in a blog post in a few days. If you have done similar work in research or industry, we can substitute the exercise with a suitable sample of this, but only if this is core database code.</p> <p>There is a dual message: The challenges will be the toughest a very tough race can offer. On the other hand, I do not want to scare you away prematurely. Nobody knows this stuff, except for the handful of people who actually do core database work. So we are not limiting this call to this small crowd and will teach you on the job if you just come with an aptitude to think in algorithms and code fast. Experience has pros and cons so we do not put formal bounds on this. "Just out of high school" may be good enough, if you are otherwise exceptional. Prior work in RDF or semantic web is not a factor. Sponsorship of your M.Sc. or Ph.D. thesis, if the topic is in our line of work and implementation can be done in our environment, is a further possibility. Seasoned pros are also welcome and will know the nature of the gig from the reading list.</p> <p>We are aiming to fill the position(s) between now and October.</p> <p>Resumes and inquiries can be sent to Hugh Williams, <a href="mailto:hwilliams@openlinksw.com?subject=2012-08%20Virtuoso%20Developer%20Application" id="link-id0x5dc1ac0">hwilliams@openlinksw.com</a>. We will contact applicants for interviews.</p>
2012-08-07T13:21:52.000005-04:00
Developer Opportunities at OpenLink Software
http://www.openlinksw.com:443/weblog/oerling/?date=2012-08-07#1716
2012-08-07T17:21:29Z
<p>If it is advanced database technology, you will get to do it with us.</p> <p>We are looking for exceptional talent to implement some of the hardest stuff in the industry. This ranges from new approaches to query optimization; to parallel execution (both scale up and scale out); to elastic cloud deployments and self-managing, self-tuning, fault-tolerant databases. We are most familiar to the RDF world, but also have full SQL support, and the present work will serve both use cases equally.</p> <p>We are best known in the realms of high-performance database connectivity middleware and massively-scalable Linked-Data-oriented graph-model DBMS technology.</p> <p>We have the basics -- SQL and SPARQL, column store, vectored execution, cost based optimization, parallel execution (local and cluster), and so forth. In short, we have everything you would expect from a DBMS. We do transactions as well as analytics, but the greater challenges at present are on the analytics side.</p> <p>You will be working with my team covering:</p> <ul> <li> <p>Adaptive query optimization -- interleaving execution and optimization, so as to always make the correct plan choices based on actual data characteristics</p> </li> <li> <p>Self-managing cloud deployments for elastic big data -- clusters that can grow themselves and redistribute load, recover from failures, etc.</p> </li> <li> <p>Developing and analyzing new benchmarks for RDF and graph databases</p> </li> <li> <p>Embedding complex geospatial reasoning inside the database engine. We have the basic R-tree and the OGC geometry data types; now we need to go beyond this</p> </li> <li> <p>Every type of SQL optimizer and execution engine trick that serves to optimize for TPC-H and DS.</p> </li> </ul> <p>What do I mean by really good? It boils down to being a smart and fast programmer. We have over the years talked to people, including many who have worked on DBMS programming, and found that they actually know next to nothing of database science. For example, they might not know what a hash join is. Or they might not know that interprocess latency is in the tens of microseconds even within one box, and that in that time one can do tens of index lookups. Or they might not know that blocking on a mutex kills.</p> <p>If you do core database work, we want you to know how many CPU cache misses you will have in flight at any point of the algorithm, and how many clocks will be spent waiting for them at what points. Same for distributed execution: The only way a cluster can perform is having max messages with max payload per message in flight at all times.</p> <p>These are things that can be learned. So I do not necessarily expect that you have in-depth experience of these, especially since most developer jobs are concerned with something else. You may have to unlearn the bad habit of putting interfaces where they do not belong, for example. Or to learn that if there is an interface, then it must pass as much data as possible in one go.</p> <p>Talent is the key. You need to be a self-starter with a passion for technology and have competitive drive. These can be found in many guises, so we place very few limits on the rest. If you show you can learn and code fast, we don't necessarily care about academic or career histories. You can be located anywhere in the world, and you can work from home. There may be some travel but not very much.</p> <p>In the context of <a href="http://lod2.eu/" id="link-id0x7719ea0">EU FP7 projects</a>, we are working with some of the best minds in database, including <a href="http://homepages.cwi.nl/~boncz/" id="link-id0x21d67d80">Peter Boncz</a> of CWI and VU Amsterdam (MonetDB, VectorWise) and <a href="http://www.mpi-inf.mpg.de/~neumann/" id="link-id0x2192d900">Thomas Neumann</a> of Technical University of Munich (RDF3X, HYPER). This is an extra guarantee that you will be working on the most relevant problems in database, informed by the results of the very best work to date.</p> <p>For more background, please see the IEEE Computer Society <i>Bulletin of the Technical Committee on Data Engineering,</i> <a href="http://sites.computer.org/debull/A12mar/issue1.htm" id="link-id0x7ca1d20">Special Issue on Column Store Systems</a>.</p> <p>All articles and references therein are relevant for the job. Be sure to read the CWI work on run time optimization (ROX), cracking, and recycling. Do not miss the many papers on architecture-conscious, cache-optimized algorithms; see the VectorWise and MonetDB articles in the bulletin for extensive references.</p> <p>If you are interested in an opportunity with us, we will ask you to do a little exercise in multithreaded, performance-critical coding, to be detailed in a blog post in a few days. If you have done similar work in research or industry, we can substitute the exercise with a suitable sample of this, but only if this is core database code.</p> <p>There is a dual message: The challenges will be the toughest a very tough race can offer. On the other hand, I do not want to scare you away prematurely. Nobody knows this stuff, except for the handful of people who actually do core database work. So we are not limiting this call to this small crowd and will teach you on the job if you just come with an aptitude to think in algorithms and code fast. Experience has pros and cons so we do not put formal bounds on this. "Just out of high school" may be good enough, if you are otherwise exceptional. Prior work in RDF or semantic web is not a factor. Sponsorship of your M.Sc. or Ph.D. thesis, if the topic is in our line of work and implementation can be done in our environment, is a further possibility. Seasoned pros are also welcome and will know the nature of the gig from the reading list.</p> <p>We are aiming to fill the position(s) between now and October.</p> <p>Resumes and inquiries can be sent to Hugh Williams, <a href="mailto:hwilliams@openlinksw.com?subject=2012-08%20Virtuoso%20Developer%20Application" id="link-id0x5dc1ac0">hwilliams@openlinksw.com</a>. We will contact applicants for interviews.</p>
2012-08-07T13:21:29-04:00
IEEE publication of ?Virtuoso, a Hybrid RDBMS/Graph Column Store?
http://www.openlinksw.com:443/blog/vdb/blog/?date=2012-04-23#1715
2012-04-23T14:55:31Z
<p>My article, <b><i>Virtuoso, a Hybrid RDBMS/Graph Column Store</i></b> (<a href="http://sites.computer.org/debull/A12mar/vicol.pdf" id="link-id0x1f456350">PDF</a>), can be found in <a href="http://www.informatik.uni-trier.de/~ley/db/journals/debu/debu35.html" id="link-id0x8bf15d8">Volume 35, Number 1, March 2012</a> (<a href="http://sites.computer.org/debull/A12mar/A12MAR-CD.pdf" id="link-id0x20adffb8">PDF</a>) of the <i><a href="http://www.informatik.uni-trier.de/~ley/db/journals/debu/index.html" id="link-id0x1e133b50">Bulletin</a> of the <a href="http://dbpedia.org/resource/IEEE_Computer_Society" id="link-id0x1f46c998">IEEE Computer Society</a> <a href="http://tab.computer.org/tcde/" id="link-id0x1c3fce40">Technical Committee on Data Engineering</a></i> (also known as the <i><a href="http://www.informatik.uni-trier.de/~ley/db/journals/debu/index.html" id="link-id0x17d03718">IEEE Data Engineering Bulletin</a>)</i>. </p> <p> <b>Abstract:</b> </p> <blockquote> <i>We discuss applying column store techniques to both graph (RDF) and relational data for mixed workloads ranging from lookup to analytics in the context of the OpenLink Virtuoso DBMS. In so doing, we need to obtain the excellent memory efficiency, locality and bulk read throughput that are the hallmark of column stores while retaining low-latency random reads and updates, under serializable isolation.</i> </blockquote> <p> <b>DBLP BibTeX Record 'journals/debu/Erling12'</b> (<a href="http://dblp.uni-trier.de/rec/bibtex/journals/debu/Erling12.xml" id="link-id0x20356268">XML</a>)</p> <blockquote> <pre><tt>@article{DBLP:journals/debu/Erling12, author = {Orri Erling}, title = {Virtuoso, a Hybrid RDBMS/Graph Column Store}, journal = {IEEE Data Eng. Bull.}, volume = {35}, number = {1}, year = {2012}, pages = {3-8}, ee = {http://sites.computer.org/debull/A12mar/vicol.pdf}, bibsource = {DBLP, http://dblp.uni-trier.de} } </tt> </pre></blockquote>
2012-04-23T10:55:31-04:00
IEEE publication of ?Virtuoso, a Hybrid RDBMS/Graph Column Store?
http://www.openlinksw.com:443/weblog/oerling/?date=2012-04-23#1714
2012-04-23T14:55:17Z
<p>My article, <b><i>Virtuoso, a Hybrid RDBMS/Graph Column Store</i></b> (<a href="http://sites.computer.org/debull/A12mar/vicol.pdf" id="link-id0x1f456350">PDF</a>), can be found in <a href="http://www.informatik.uni-trier.de/~ley/db/journals/debu/debu35.html" id="link-id0x8bf15d8">Volume 35, Number 1, March 2012</a> (<a href="http://sites.computer.org/debull/A12mar/A12MAR-CD.pdf" id="link-id0x20adffb8">PDF</a>) of the <i><a href="http://www.informatik.uni-trier.de/~ley/db/journals/debu/index.html" id="link-id0x1e133b50">Bulletin</a> of the <a href="http://dbpedia.org/resource/IEEE_Computer_Society" id="link-id0x1f46c998">IEEE Computer Society</a> <a href="http://tab.computer.org/tcde/" id="link-id0x1c3fce40">Technical Committee on Data Engineering</a></i> (also known as the <i><a href="http://www.informatik.uni-trier.de/~ley/db/journals/debu/index.html" id="link-id0x17d03718">IEEE Data Engineering Bulletin</a>)</i>. </p> <p> <b>Abstract:</b> </p> <blockquote> <i>We discuss applying column store techniques to both graph (RDF) and relational data for mixed workloads ranging from lookup to analytics in the context of the OpenLink Virtuoso DBMS. In so doing, we need to obtain the excellent memory efficiency, locality and bulk read throughput that are the hallmark of column stores while retaining low-latency random reads and updates, under serializable isolation.</i> </blockquote> <p> <b>DBLP BibTeX Record 'journals/debu/Erling12'</b> (<a href="http://dblp.uni-trier.de/rec/bibtex/journals/debu/Erling12.xml" id="link-id0x20356268">XML</a>)</p> <blockquote> <pre><tt>@article{DBLP:journals/debu/Erling12, author = {Orri Erling}, title = {Virtuoso, a Hybrid RDBMS/Graph Column Store}, journal = {IEEE Data Eng. Bull.}, volume = {35}, number = {1}, year = {2012}, pages = {3-8}, ee = {http://sites.computer.org/debull/A12mar/vicol.pdf}, bibsource = {DBLP, http://dblp.uni-trier.de} } </tt> </pre></blockquote>
2012-04-23T10:55:17.000001-04:00
ICDE 2012 (post 6 of 6) - Science Data Panel
http://www.openlinksw.com:443/blog/vdb/blog/?date=2012-04-17#1713
2012-04-17T19:38:20Z
<p> <a href="http://dbpedia.org/resource/Michael_Stonebraker" class="absuri" id="link-id0x26c05b28">Michael Stonebraker</a> chaired a panel on the future of science data at <a href="http://www.icde12.org/Site/" class="absuri" id="link-id0x26d5d658">ICDE 2012</a> last week. Other participants were <a href="http://www.mit.edu/~kepner/" class="absuri" id="link-id0x26c05360">Jeremy Kepner</a> from <a href="http://www.mit.edu/" class="absuri" id="link-id0x26b12608">MIT</a> <a href="http://dbpedia.org/resource/Lincoln_Laboratory" class="absuri" id="link-id0x27d4f370">Lincoln Labs</a>, <a href="http://people.epfl.ch/cgi-bin/people?id=177957" class="absuri" id="link-id0x25259440">Anastasia Ailamaki</a> from <a href="http://dbpedia.org/resource/%C3%89cole_Polytechnique_F%C3%A9d%C3%A9rale_de_Lausanne" class="absuri" id="link-id0x252593b8">EPFL</a>, and <a href="http://www.sdss.jhu.edu/~szalay/" class="absuri" id="link-id0x26c067e8">Alex Szalay</a> from <a href="http://dbpedia.org/resource/Johns_Hopkins_University" class="absuri" id="link-id0x26c06678">Johns Hopkins University</a>.</p> <p> This is the thrust of what was said, noted from memory. My comments follow after the synopsis.</p> <b>Jeremy Kepner:</b> When <a href="http://dbpedia.org/resource/Java_(programming_language)" class="absuri" id="link-id0x26c052d8">Java</a> was new we saw it as the coming thing and figured that in <a href="http://dbpedia.org/resource/High-performance_computing" class="absuri" id="link-id0x26c06900">HPC</a> we should find space for this. When <a href="http://dbpedia.org/resource/MapReduce" class="absuri" id="link-id0x26d5d518">MapReduce</a> and <a href="http://dbpedia.org/resource/Apache_Hadoop" class="absuri" id="link-id0x27d6a330">Hadoop</a> came along, we saw this as a sea change in parallel programming models. This was so simple literally anybody could make parallel algorithms whereas this was not so with <a href="http://dbpedia.org/resource/Message_Passing_Interface" class="absuri" id="link-id0x25290350">MPI</a>. Even parallel distributed arrays are harder. So MapReduce was a game changer, together with the cloud where anybody can get a cluster. Hardly a week passes without me having to explain to somebody in government what MapReduce and Hadoop are about.<p>We have a lot of arrays and a custom database for them. But the arrays are sparse so this is in fact a triple store. Our users like to work in <a href="http://dbpedia.org/resource/MATLAB" class="absuri" id="link-id0x26b11980">MATLAB</a>, and any data management must run together with that.</p> <p>Of course, MapReduce is not a real scheduler, and Hadoop is not a real file system. For deployment, we must integrate real schedulers and make <a href="http://dbpedia.org/resource/Hadoop_Distributed_File_System" class="absuri" id="link-id0x27d50258">HDFS</a> look like a file system to applications. The abstraction of a file system is something people like. Being able to skip a time-consuming data-ingestion process with a database is an advantage with file-based paradigms like Hadoop. If this is enhanced with the right scheduling features, this can be a good component in the HPC toolbox.</p> <p> <b>Michael Stonebraker:</b> Users of the data use math packages like R, MATLAB, SAS, SPSS, or similar. If business intelligence is about <tt>AVG</tt>, <tt>MIN</tt>, <tt>MAX</tt>, <tt>COUNT</tt>, and <tt>GROUP BY</tt>, science applications are much more diverse in their analytics. All science algorithms have an inner loop that resembles linear algebra operations like matrix multiplication. Data is more often than not a large array. There are some graphs in biology and chemistry, but the world is primarily rectangular. Relational databases can emulate sparse arrays but are 20x slower than a custom-made array database for dense arrays. And I will not finish without picking on MapReduce: I know of 2000-node MapReduce clusters. The work they do is maybe that of a 100-node parallel database. So if 2000 nodes is what you want to operate, be my guest.</p> <p> Science database is a zero billion dollar business. We do not expect to make money from the science market with <a href="http://www.scidb.org/" class="absuri" id="link-id0x25269080">SciDB</a>, which by now works and has commercial services supplied by Paradigm 4, while the code itself is open source, which is a must for the science community. The real business opportunity is in the analytics needed by insurance and financial services in general, which are next to identical with the science use cases SciDB tackles. This makes the vendors pay attention.</p> <p> <b>Alex Szalay:</b> The way astronomy is done today is through surveys: a telescope scans through the sky and produces data. We have now for 10 years operated the Sloane Sky Survey and kept the data online. We have all the data, and complete query logs, available for anyone interested. When we set out to do this with Jim Gray, everybody found this a crazy idea, but it has worked out.</p> <p> <b>Anastasia Ailamaki:</b> We do not use SciDB. We find a lot of spatial use cases. Researchers need access to simulation results which are usually over a spatial model, like in earthquake simulations and the brain. Off-the-shelf techniques like R trees do not work -- the objects overlap too much -- so we have made our own spatial indexing. We make custom software when it is necessary, and are not tied to vendors. In geospatial applications, we can create meshes of different shapes -- like tetrahedral or cubes for earthquakes, and cylinders for the brain -- and index these in a geospatial index. But since an R tree is inefficient when objects overlap too much, as these do, we just find one; and then because there is reachability from an object to neighboring ones, we use this to get all the objects in the area of interest.</p> <p align="center">* * *</p> <p>This is obviously a diverse field. Probably the message that we can synthesize out of this is that <b><i>flexibility and parallel programming models are what we need to pay attention to.</i></b> There is a need to go beyond what one can do in SQL while continuing to stay close to the data. Also, allowing for plug-in data types and index structures may be useful; we sometimes get requests for such anyway.</p> <p>The continuing argument around MapReduce and Hadoop is a lasting feature of the landscape. A parallel DB will beat MapReduce any day at joining across partitions; the problem is to overcome the mindset that sees Hadoop as the always-first answer to anything parallel. People will likely have to fail with this before they do anything else. For us, the matter is about having database-resident logic for <a href="http://dbpedia.org/resource/Extract,_transform,_load" class="absuri" id="link-id0x26c06558">extract-transform-load (ETL)</a> that can do data-integration type-transformations and maybe iterative graph algorithms that constantly join across partitions, better than a MapReduce job, while still allowing application logic to be written in Java. Teaching sem-web-heads to write SQL procedures and to know about join order, join type, and partition locality, has proven to be difficult. People do not understand latency, whether in client-server or cluster settings. This is why they do not see the point of stored procedures or of shipping functions to data. This sounds like a terrible indictment, like saying that people do not understand why rivers flow downhill. Yet, it is true. This is also why MapReduce is maybe the only parallel programming paradigm that can be successfully deployed in the absence of this understanding, since it is actually quite latency-tolerant, not having any synchronous cross-partition operations except for the succession of the map and reduce steps themselves.</p> <p>Maybe it is so that the database guys see MapReduce as an insult to their intelligence and the rest of the world sees it as the only understandable way of running <tt><a href="http://dbpedia.org/resource/Grep" class="absuri" id="link-id0x2507a000">grep</a></tt> and <tt><a href="http://dbpedia.org/resource/Sed" class="absuri" id="link-id0x25079ec0">sed</a></tt> (Unix commands for string search/replace) in parallel, with the super bonus of letting you reshuffle the outputs so that you can compare everything to everything else, which <tt>grep</tt> alone never let you do.</p> <p align="center">* * *</p> <p>Making a database that does not need data loading seems a nice idea, and CWI has actually done something in this direction in "<a href="http://infoscience.epfl.ch/record/161489" class="absuri" id="link-id0x250787f0">Here are my Data Files. Here are my Queries. Where are my Results?</a>"] However, there is another product called Algebra Data that claims to take in data without loading and to optimize storage based on access. We do not have immediate plans in this direction. Bulk load is already quite fast (take 100G TPC-H in 70 minutes or so), but faster is always possible.</p>
2015-08-21T15:47:03.467938-04:00
ICDE 2012 (post 5 of 6) - Graphs
http://www.openlinksw.com:443/blog/vdb/blog/?date=2012-04-17#1712
2012-04-17T19:38:15Z
<p>There were quite a few talks about graphs at <a href="http://www.icde12.org/Site/" class="absuri" id="link-id0x26d23c70">ICDE 2012</a>. Neither the representations of graphs, nor the differences between RDF and generic graph models, entered much into the discussion. On the other hand, graph similarity searches and related were addressed a fair bit.</p> <p>Graph DB and RDF/Linked Data are distinct, if neighboring disciplines. On one hand, graph problems predate Linked Data, and the RDF/Linked Data world is a web artifact, which graphs are not as such, so a slightly different cultural derivation also makes these disjoint. Besides, graphs may imply schema first whereas linked data basically cannot. Then another differentiation might be derived from edges not really being first class citizens in RDF, except for reification, at which the RDF reification vocabulary is miserably inadequate, as pointed out before.</p> <p>RDF is being driven by the web-style publishing of Linked Open Data (LOD), with some standardization and uptake by publishers; Graph DB is not standardized but driven by diverse graph-analytics use cases.</p> <p>There is no necessary reason why these could not converge, but it will be indefinitely long before any standards come to cover this, so best not hold one's breath. Communities are jealous of their borders, so if the neighbor does something similar one tends to emphasize the differences and not the commonalities.</p> <p>So for some things, one could warehouse the original RDF of the web microformats and LOD, and then ETL into some other graph model for specific tasks, or just do these in RDF. Of course, then RDF systems need to offer suitable capabilities. These seem to be about very fast edge traversal within a rather local working set, and about accommodating large, iteratively-updated intermediate results, e.g., edge weights.</p> <p>Judging by the benchmarks paper (<i>Benchmarking traversal operations over graph databases (<a href="http://www.cse.unsw.edu.au/~iwgdm/2012/Slides/Ciglan.pptx" class="absuri" id="link-id0x26e6cc68">Slidedeck (ppt)</a>, <a href="http://ups.savba.sk/~marek/papers/gdm12-ciglan.pdf" class="absuri" id="link-id0x28545128">paper (pdf)</a>);</i> Marek Ciglan, Alex Averbuch, and Ladialav Hluchy.) at the <a href="http://www.cse.unsw.edu.au/~iwgdm/2012/index.html" class="absuri" id="link-id0x26b17d70">GDM workshop</a>, the state of benchmarking in graph databases is even worse than in RDF, where the state is bad enough. The paper's premise was flawed to start, using application logic to do <tt>JOIN</tt>s instead of doing them in the DBMS. In this way, latency comes to dominate, and only the most blatant differences are seen. There is nothing like this style of benchmarking to make an industry look bad. The supercomputer Graph 500 benchmark, on the other hand, lets the contestants make their own implementations on a diversity of architectures with random traversal as well as loading and generating large intermediate results. It is somewhat limited, but still broader than the the graph database benchmarks paper at the GDM workshop.</p> <p> Returning to graphs, there were some papers on similarity search and clique detection. As players in this space, beyond just RDF, we might as well consider implementing necessary features for efficient expression of such problems. The algorithms discussed were expressed in procedural code against memory-based data structures; there is usually no query language or parallel/distributed processing involved.</p> <p> <a href="http://dbpedia.org/resource/MapReduce" class="absuri" id="link-id0x27d27f40">MapReduce</a> has become the default way in which people would tackle such problems at scale; in fact, people do not consider anything else, as far as I can tell. Well, they certainly do not consider MPI for example as a first choice. The parallel array things in Fortran do not at first sight seem very graphy, so this is likely not something that crosses one's mind either.</p> <p>We should try some of the similarity search and clustering in SQL with a parallel programming model. We have excellent expression-evaluation speed from vectoring and unrestricted recursion between partitions, and no file system latencies like <nop></nop>MapReduce. The initial test case will be some of the linking/data-integration/mapping workloads in LOD2.</p> <p> Having some sort-of-agreed-upon benchmark for these workloads would make this more worthwhile. Again, we will see what emerges.</p>
2012-04-17T15:38:15.000004-04:00
ICDE 2012 (post 4 of 6) - Graph Data Management Workshop
http://www.openlinksw.com:443/blog/vdb/blog/?date=2012-04-17#1711
2012-04-17T19:38:09Z
<p>I gave an invited talk ("Virtuoso 7 - Column Store and Adaptive Techniques for Graph" (<a href="http://www.cse.unsw.edu.au/~iwgdm/2012/Slides/Virtuoso.ppt" class="absuri" id="link-id0x27fce2f8">Slides (ppt)</a>)) at the <a href="http://www.cse.unsw.edu.au/~iwgdm/2012/" class="absuri" id="link-id0x27eb6cd8">Graph Data Management Workshop</a> at <a href="http://www.icde12.org/Site/" class="absuri" id="link-id0x26d89980">ICDE 2012</a>.</p> <p>Bryan Thompson of <a href="http://www.systap.com/" class="absuri" id="link-id0x2a893000">Systap</a> (<a href="http://www.systap.com/bigdata.htm" class="absuri" id="link-id0x26c36d70">Bigdata®</a> RDF store) was also invited, so we got to talk about our common interests. He told me about two cool things they have recently done, namely introducing tables to <a href="http://dbpedia.org/resource/SPARQL" class="absuri" id="link-id0x2a74b040">SPARQL</a>, and adding a way of <a href="http://dbpedia.org/resource/Reification_%28computer_science%29" class="absuri" id="link-id0x26c90790">reifying statements</a> that does not rely on extra columns. The table business is just about being able to store a multicolumn result set into a named persistent entity for subsequent processing. But this amounts to a SQL table, so the relational model has been re-arrived at, once more, from practical considerations. The reification just packs all the fields of a triple (or quad) into a single string and this string is then used as an RDF <tt>S</tt> or <tt>O</tt> (Subject or Object), less frequently a <tt>P</tt> or <tt>G</tt> (Predicate or Graph). This works because Bigdata® has variable length fields in all columns of the triple/quad table. The query notation then accepts a function-looking thing in a triple pattern to mark reification. Nice. Virtuoso has a variable length column in only the <tt>O</tt> but could of course have one in also <tt>S</tt> and even in <tt>P</tt> and <tt>G</tt>. The column store would still compress the same as long as reified values did not occur. These values on the other hand would be unlikely to compress very well but run length and dictionary would always work.</p> <p>So, we could do it like Bigdata®, or we could add a "quad ID" column to one of the indices, to give a reification ID to quads. Again no penalty in a column store, if you do not access the column. Or we could make an extra table of <tt>PSOG->R</tt>.</p> <p>Yet another variation would be to make the <tt>SPOG</tt> concatenation a literal that is interned in the RDF literal table, and then used as any literal would be in the <tt>O</tt>, and as an IRI in a special range when occurring as <tt>S</tt>. The relative merits depend on how often something will be reified and on whether one wishes to <tt>SELECT</tt> based on parts of reification. Whichever the case may be, the idea of a function-looking placeholder for a reification is a nice one and we should make a compatible syntax if we do special provenance/reification support. The model in the RDF reification vocabulary is a non-starter and a thing to discredit the sem web for anyone from database.</p> <p>I heard from Bryan that the new <a href="http://www.w3.org/2011/rdf-wg/" class="absuri" id="link-id0x28a628b0">W3 RDF WG</a> had declared provenance out of scope, unfortunately. The word on the street on the other hand is that provenance is increasingly found to be an issue. This is confirmed by the active work of the <a href="http://www.w3.org/2011/prov/" class="absuri" id="link-id0x2540a098">W3 Provenance Working Group</a>.</p>
2012-04-17T15:38:09.000007-04:00
ICDE 2012 (post 3 of 6) - What Is Timely LOD Search Worth?
http://www.openlinksw.com:443/blog/vdb/blog/?date=2012-04-17#1710
2012-04-17T19:38:03Z
<p>There was a talk (Linked Data and Live Querying for Enabling Support Platforms for Web Dataspaces (<a href="https://sites.google.com/site/desweb2012/parreira.pdf?attredirects=0" class="absuri" id="link-id0x27f2bbe0">Slides (PDF)</a>); Jürgen Umbrich, Marcel Karnstedt, Josiane Xavier Parreira, Axel Polleres and Manfred Hauswirth) at the <a href="https://sites.google.com/site/desweb2012/" class="absuri" id="link-id0x299b5230">Data Engineering Meets the Semantic Web (DESWEB)</a> workshop at <a href="http://www.icde12.org/Site/" class="absuri" id="link-id0x2928bfd8">ICDE</a> last week about the problems of caching LOD, whether attempted by <a href="http://sindice.com/" class="absuri" id="link-id0x278876b8">Sindice</a> or <a href="http://www.openlinksw.com" class="absuri" id="link-id0x2a4ba140">OpenLink</a>'s <a href="http://lod.openlinksw.com/" class="absuri" id="link-id0x299ed1f0">LOD Cloud Cache</a>. The conclusion was that OpenLink covered a bit more of the test data sets and that Sindice was maybe better up to date on the ones that it covered but that neither did it very well. The data sets were random graphs of user FOAF profiles and such collected from some Billion Triples Data set, thus not data that is likely to have commercial value, except in huge quantities maybe for some advertising, except that click streams and the like are much more valuable.</p> <p>Being involved with at least one of these, and being in the audience, I felt obligated to comment. The fact is, neither OpenLink's LOD Cloud Cache nor Sindice is a business, and there is not a business model which could justify keeping them timely on the web crawls they contain. Doing so is easy enough, if there is a good enough reason.</p> <p>The talk did make a couple of worthwhile points: The data does change; and if one queries entities, one encounters large variation in change-frequency across entities and their attributes.</p> <p>The authors suggested to have a piece of middleware decide what things can be safely retrieved from a copy and what have to be retrieved from the source. Not too much is in fact known about the change frequency of the data, except that it changes, as the authors pointed out.</p> <p>The crux of the matter is that the thing that ought to know this best is the query processor at the LOD warehouse. For client-side middleware to split the query, it needs access to statistics that it must get from the warehouse or keep by itself. Of course, in concrete application scenarios, you go to the source if you ask about the weather or traffic jams, and otherwise go to the warehouse based on application-level knowledge.</p> <p>But for actual business intelligence, one needs histories, so a search engine with only the present is not so interesting. At any rate, refreshing the data should leave a trail of past states. Exposing this for online query would just triple the price, so we forget about that for now. Just keeping an append-only table of history is not too much of a problem. One may make extracts from this table into a relational form for specific business questions. There is no point doing such analytics in RDF itself. One would have to just try to see if there is anything remotely exploitable in such histories. Making a history table is easy enough. Maybe I will add one.</p> <p>Let us now see what it would take to operate a web crawl cache that would be properly provisioned, kept fresh, and managed. We base this on the Sindice crawl sizes and our experiments on these; the non-web-crawl LOD Cloud Cache is not included.</p> <p>From previous experience we know the sizing: 5Gt/144GB RAM. Today's best price point is on 24-DIMM E5 boards, so 192GB RAM, or 6.67Gt. A unit like that (8TB HDD, 0.5TB SSD, 192GB RAM, 12 core E5, <a href="http://dbpedia.org/page/InfiniBand" class="absuri" id="link-id0x27eb8158">InfiniBand</a>) costs about $6800.</p> <p>The Sindice crawl is now about 20Gt, so $28K of gear (768GB RAM) is enough. Let us count this 4 times: 2x for anticipated growth; and 2x for running two copies -- one for online, and one for batch jobs. This is 3TB RAM. Power is 16x500W = 8KW, which we could round to 80A at 110V. Colocation comes to $500 for the space, and $1200 per month for power; make it $2500 per month with traffic included.</p> <p>At this rate, 3 year TCO is <tt>$120K + ( 36 * $2.5K ) = $210K</tt>. This takes one person half time to operate, so this is another $50K per year.</p> <p>We do not count software development in this, except some scripting that should be included in the yearly $50K DBA bill.</p> <p>Under what circumstances is such a thing profitable? Or can such a thing be seen as a marketing demo, to be paid for by license or service sales?</p> <p>A third party can operate a system of this sort, but then the cost will be dominated by software licenses if running on Virtuoso cluster.</p> <p>For comparison, the TB at EC2 costs <tt>((( 16 * $2 ) * 24 ) * 31 ) = $24,808</tt> per month. With reserved instances, it is <tt>( 16 * ( $2192 + ((( 0.7 * 24 ) * 365 ) * 3 ))) / 36 = $8938</tt> per month for a 3 year term. Counting at 3TB, the 3 year TCO is $965K at EC2. AWS has volume discounts but they start higher than this; <tt>( 3 * ( 16 * $2K )) = $96K</tt> reserved host premium is under $250K. So if you do not even exceed their first volume discount threshold, it does not look likely you can cut a special deal with AWS.</p> <p>(The AWS prices are calculated with the high memory instances, approximately 64GB usable RAM each. The slightly better CC2 instance is a bit more expensive.)</p> <p>Yet another experiment to make is whether a system as outlined will even run at anywhere close to the performance of physical equipment. This is uncertain; clouds are not for speed, based on what we have seen. They make the most sense when the monthly bill is negligible in relation to the cost of a couple of days of human time.</p>
2012-04-19T16:44:28-04:00
ICDE 2012 (post 2 of 6) - LOD Column Store Experiences and Sizing
http://www.openlinksw.com:443/blog/vdb/blog/?date=2012-04-17#1709
2012-04-17T19:37:58Z
<p>We have played around with <a href="http://lod.openlinksw.com/" class="absuri" id="link-id0x2a8a8df8">LOD data sets and Virtuoso Column Store</a> for the past several months. I will here give a few numbers and comment on some different platform comparisons that we have made. The answer at the end of this is how to size a system for often-changing web-style data. The conclusion is a data-to-RAM ratio that gives an acceptable working set without driving the price up by forcing 100% RAM residence.</p> <p>The experiment is loading <a href="http://sindice.com/" class="absuri" id="link-id0x28e1fb38">Sindice</a> web crawls. The platform is 2 x Xeon 5520 and 144G RAM. The initial load rate is 200-180Kt, and drops to 100Kt at 5Gt because of I/O. The system is Virtuoso Column Store configured to run as 4 processes and 32 partitions, all on the same box. After 5Gt, we see just more I/O and going further is not relevant; one runs CPU-bound or not at all.</p> <p>We use 4 Crucial SSDs in the setup. The hot structures like the RDF quad indices are on SSD, and the cold ones are on hard disk. A cold structure is a write-only index like the dictionary of literals (id to lit).</p> <p>For bulk load, SSDs turn out not to be particularly useful. For a cold start on the other hand, SSDs cut warmup time of 144G RAM from over half an hour to a couple of minutes. It is possible that Intel SSDs would also help with bulk load, but this has not been tried. The SSD problem during bulk load is that these do not write very fast, and while there are writes in queue, read latency goes up; so under a constant write load, the SSD's famous instantaneous random read no longer works.</p> <p>The fragment considered in the example is 4.95Gt: 8.1M pages worth of quads; 12.7M of literals and iris; and 4.71M of full text index. A page is 8KB. The files on disk contain empty pages, but these do not matter since they do not take up RAM. The quad indices take 13.4 bytes/quad. The row-wise equivalent used to be 38 or so bytes/quad with similar data. Two-thirds of the IRI and literal string data can benefit from column-wise stream compression. (This was not used but if it were, we could count on a 50% drop in size for the data affected, so instead of 12.7M pages, we could maybe get 8.5M on a good day. This could be worth doing but is not a priority.) The system was configured to have 12M database pages in RAM, so a little under half the database pages of the set fit in RAM at one time; thus one cannot call this a memory-only setup. Due to the locality in the unusually non-local data, this is as far as secondary storage can reach without becoming an over-2x slowdown. In practice, we are talking about under 1% of rows accessed coming from secondary storage, but that alone means half throughput.</p> <p>We note that this data set represents the worst that we have seen. It has 129M distinct graphs, 38 t/g. Regular data like the synthetic benchmark sets take half the space per quad. This is about a third of a Sindice crawl; the other two-thirds look the same as far as we looked.</p> <p>So if you are interested in hosting data like this, you can budget 144GB RAM for every 5Gt. Do not try it with anything less. Budgeting double this is wise, so that you have space to cook the data; this is important since in order to do things with it, one needs to at least copy things for materializing transformations.</p> <p>If you are budget-constrained and hosting very regular content like <a href="http://dbpedia.org/page/UniProt" class="absuri" id="link-id0x29b465b8">UniProt</a>, you can budget maybe 144GB RAM for every 10Gt.</p> <p>As for CPU, this does not matter as much as long as you do not go to disk. Just for load speed, Dbpedia is loaded in 300s on a cluster of eight (8) dual AMD 2378 boxes at 2.6GHz (total 8 cores per host, so 64 cores in the cluster), and in 945s on one (1) dual Xeon 5520 box at 2.26GHz (total 8 cores in the host). Intel makes much better CPUs, as we see. Both scenarios are 100% in RAM. For even more regular data, the load rates are a bit higher: 1.3Mt/s for the AMD cluster, and 300Kt/s for the Xeon host.</p> <p>The interconnect for the AMD cluster is 1 x gigE but this does not matter for load. For CPU-bound cross-partition <tt>JOIN</tt>s, 1 or 2 x gigE is insufficient; 4 x gigE might barely make it; <a href="http://dbpedia.org/page/InfiniBand" class="absuri" id="link-id0x26d4cec8">InfiniBand</a> should be safe. When running cross-partition <tt>JOIN</tt>s, a single 8-core Xeon box generates about 300MB/s of interconnect traffic; a gigE connection can maybe take 50MB/s with some luck.</p> <p> <a href="http://en.wikipedia.org/wiki/Xeon#E5-16xx.2F26xx-series_.22Sandy_Bridge-EP.22" class="absuri" id="link-id0x27df5b40">Intel E5</a> is not dramatically better than <a href="http://en.wikipedia.org/wiki/Xeon#Nehalem-based_Xeon" class="absuri" id="link-id0x293e9d50">Nehalem</a> but this is something we will see in a while when we make measurements with real equipment. Prior to the E5 release, we tried Amazon EC2 CC2 ("Cluster Compute Eight Extra Large Instance" -- 2x8 core E5, 2.66GHz). The results were inconclusive; it never did more than 1.9x better than Xeon 5520 even when running an empty loop (i.e., recursive <a href="http://dbpedia.org/resource/Fibonacci_function" class="absuri" id="link-id0x2a480aa0">Fibonacci function</a> in SQL, no cache misses, no I/O). With a database <tt>JOIN</tt>, 1.3x better is the best we saw. But this must be the fault of Amazon and not of E5.</p> <p>We also tried <a href="http://en.wikipedia.org/wiki/List_of_AMD_Opteron_microprocessors#Opteron_6100-series_.22Magny-Cours.22_.2845_nm.29" class="absuri" id="link-id0x27e262a0">AMD "Magny-Cours"</a>, but for 32 cores against 8 it never did over 2x better, more like 1.4x often enough, and and single thread speed was 50% worse, so not a good buy. We did not find a <a href="http://en.wikipedia.org/wiki/List_of_AMD_Opteron_microprocessors#Bulldozer_based_Opterons" class="absuri" id="link-id0x29936f10">Bulldozer</a> to try, and did not feel like buying one since the reviews did not promise more core speed over the Magny-Cours.</p> <p>It seems that especially with Column Store, we are truly CPU-bound and not memory-latency- or bandwidth-bound. This is based on the observation that a Xeon 5620 with 2 of 3 memory channels populated loads BSBM data only 10% faster than the same with 1 of 3 channels populated, with CPU affinity set on a dual socket system.</p> <p>So, if you have a choice between a $2K processor (E5-2690) and a $600 processor (E5-2630), buy the cheaper one and get RAM with the money saved. $1440 buys 128G in $90 8G DIMMs. Then buy E5 boards with 24 DIMMs -- one for every 7Gt of web crawl data. If your software licenses are priced per core, getting higher-clock 4-core E5’s might make sense.</p> <p>While on the subject of bytes and quads/triples, we note that <a href="http://www.systap.com/bigdata.htm" class="absuri" id="link-id0x2784a1d8">Bigdata®</a>'s <a href="http://www.bigdata.com/bigdata/blog/?p=423" class="absuri" id="link-id0x2ac80e68">recent announcement</a> says up to 50 billion triples per single server. Franz loaded at a good 800+ Kt/s rate up to <a href="http://franz.com/agraph/allegrograph/agraph_benchmarks.lhtml" class="absuri" id="link-id0x26c3bea0">a trillion triples</a>. One is led to think from the spec that this was with less than full cpu but still with highly local data, considering 1.5 bytes a triple would hit very heavy I/O otherwise. Their statement to the effect of <a href="http://swat.cse.lehigh.edu/projects/lubm/" class="absuri" id="link-id0x251d4180">LUBM</a>-like data corroborates this, so we are not talking about exactly the same thing.</p> <p>So if you compare the claims, I am talking about running CPU-bound on the worst data there is. Franz and Bigdata® do not specify, so it is hard to compare. LOD2 should in principle publish actual metrics with at least Bigdata®; Franz is not participating in these races.</p> <p>We may publish some more detailed measurements with more varied configurations later. The thing to remember is minimum 144GB RAM for every 5Gt of web crawls, if you want to load and refresh in RAM.</p>
2012-04-17T15:37:58-04:00
ICDE 2012 (post 1 of 6) - LOD2 Plenary
http://www.openlinksw.com:443/blog/vdb/blog/?date=2012-04-17#1708
2012-04-17T19:37:50Z
<p> <a href="http://lod2.eu/" class="absuri" id="link-id0x1d3c5620">LOD2</a>'s database contributions are, on one hand, Virtuoso Column Store and Elastic Cluster, and on the other, the demonstration and proof from <a href="http://dbpedia.org/resource/Centrum_Wiskunde_&_Informatica" class="absuri" id="link-id0x11356250">CWI</a> that indeed all of the <a href="http://dbpedia.org/resource/Relational_database" class="absuri" id="link-id0x1d6296d8">relational</a> innovations for which CWI is well known apply to <a href="http://dbpedia.org/resource/Graph_%28data_structure%29" class="absuri" id="link-id0x1bcd83d8">graph</a>/<a href="http://dbpedia.org/resource/Resource_Description_Framework" class="absuri" id="link-id0x1befd4a0">RDF</a> data as well.</p> <p>The value is unquestionable both to Virtuoso users in the short-term, and to the state of science and to all RDF users and vendors in the mid-term.</p> <p>The LOD2 claim of "<a href="http://www.openlinksw.com/weblog/oerling/?id=1649" class="absuri" id="link-id0x1c161d88">linking the universe</a>" (my words) will be tested soon enough, after we first put the universe in a bucket. This refers to a real-time quad store of <a href="http://sindice.com/" class="absuri" id="link-id0x1d3eb050">Sindice</a> crawls, plus a warehouse of the LOD data sets.</p> <p>This effort raises a few questions that I will treat in a number of posts to follow, such as --</p> <ul> <li>How do you size a real-time copy of LOD/web data? </li> <li>What does it cost to operate a properly provisioned warehouse of all RDF web crawls?</li> </ul> <p>What is done now is under-provisioned and not kept up to date. We are talking about all the RDF on the web in near real time with arbitrary queries. This is very far from the "billion triples" data sets or vertical portals, which are both easy by comparison.</p>
2012-04-17T15:37:50.000002-04:00
ICDE 2012 (post 6 of 6) - Science Data Panel
http://www.openlinksw.com:443/weblog/oerling/?date=2012-04-17#1707
2012-04-17T19:36:28Z
<p> <a href="http://dbpedia.org/resource/Michael_Stonebraker" class="absuri" id="link-id0x26c05b28">Michael Stonebraker</a> chaired a panel on the future of science data at <a href="http://www.icde12.org/Site/" class="absuri" id="link-id0x26d5d658">ICDE 2012</a> last week. Other participants were <a href="http://www.mit.edu/~kepner/" class="absuri" id="link-id0x26c05360">Jeremy Kepner</a> from <a href="http://www.mit.edu/" class="absuri" id="link-id0x26b12608">MIT</a> <a href="http://dbpedia.org/resource/Lincoln_Laboratory" class="absuri" id="link-id0x27d4f370">Lincoln Labs</a>, <a href="http://people.epfl.ch/cgi-bin/people?id=177957" class="absuri" id="link-id0x25259440">Anastasia Ailamaki</a> from <a href="http://dbpedia.org/resource/%C3%89cole_Polytechnique_F%C3%A9d%C3%A9rale_de_Lausanne" class="absuri" id="link-id0x252593b8">EPFL</a>, and <a href="http://www.sdss.jhu.edu/~szalay/" class="absuri" id="link-id0x26c067e8">Alex Szalay</a> from <a href="http://dbpedia.org/resource/Johns_Hopkins_University" class="absuri" id="link-id0x26c06678">Johns Hopkins University</a>.</p> <p> This is the thrust of what was said, noted from memory. My comments follow after the synopsis.</p> <b>Jeremy Kepner:</b> When <a href="http://dbpedia.org/resource/Java_(programming_language)" class="absuri" id="link-id0x26c052d8">Java</a> was new we saw it as the coming thing and figured that in <a href="http://dbpedia.org/resource/High-performance_computing" class="absuri" id="link-id0x26c06900">HPC</a> we should find space for this. When <a href="http://dbpedia.org/resource/MapReduce" class="absuri" id="link-id0x26d5d518">MapReduce</a> and <a href="http://dbpedia.org/resource/Apache_Hadoop" class="absuri" id="link-id0x27d6a330">Hadoop</a> came along, we saw this as a sea change in parallel programming models. This was so simple literally anybody could make parallel algorithms whereas this was not so with <a href="http://dbpedia.org/resource/Message_Passing_Interface" class="absuri" id="link-id0x25290350">MPI</a>. Even parallel distributed arrays are harder. So MapReduce was a game changer, together with the cloud where anybody can get a cluster. Hardly a week passes without me having to explain to somebody in government what MapReduce and Hadoop are about.<p>We have a lot of arrays and a custom database for them. But the arrays are sparse so this is in fact a triple store. Our users like to work in <a href="http://dbpedia.org/resource/MATLAB" class="absuri" id="link-id0x26b11980">MATLAB</a>, and any data management must run together with that.</p> <p>Of course, MapReduce is not a real scheduler, and Hadoop is not a real file system. For deployment, we must integrate real schedulers and make <a href="http://dbpedia.org/resource/Hadoop_Distributed_File_System" class="absuri" id="link-id0x27d50258">HDFS</a> look like a file system to applications. The abstraction of a file system is something people like. Being able to skip a time-consuming data-ingestion process with a database is an advantage with file-based paradigms like Hadoop. If this is enhanced with the right scheduling features, this can be a good component in the HPC toolbox.</p> <p> <b>Michael Stonebraker:</b> Users of the data use math packages like R, MATLAB, SAS, SPSS, or similar. If business intelligence is about <tt>AVG</tt>, <tt>MIN</tt>, <tt>MAX</tt>, <tt>COUNT</tt>, and <tt>GROUP BY</tt>, science applications are much more diverse in their analytics. All science algorithms have an inner loop that resembles linear algebra operations like matrix multiplication. Data is more often than not a large array. There are some graphs in biology and chemistry, but the world is primarily rectangular. Relational databases can emulate sparse arrays but are 20x slower than a custom-made array database for dense arrays. And I will not finish without picking on MapReduce: I know of 2000-node MapReduce clusters. The work they do is maybe that of a 100-node parallel database. So if 2000 nodes is what you want to operate, be my guest.</p> <p> Science database is a zero billion dollar business. We do not expect to make money from the science market with <a href="http://www.scidb.org/" class="absuri" id="link-id0x25269080">SciDB</a>, which by now works and has commercial services supplied by Paradigm 4, while the code itself is open source, which is a must for the science community. The real business opportunity is in the analytics needed by insurance and financial services in general, which are next to identical with the science use cases SciDB tackles. This makes the vendors pay attention.</p> <p> <b>Alex Szalay:</b> The way astronomy is done today is through surveys: a telescope scans through the sky and produces data. We have now for 10 years operated the Sloane Sky Survey and kept the data online. We have all the data, and complete query logs, available for anyone interested. When we set out to do this with Jim Gray, everybody found this a crazy idea, but it has worked out.</p> <p> <b>Anastasia Ailamaki:</b> We do not use SciDB. We find a lot of spatial use cases. Researchers need access to simulation results which are usually over a spatial model, like in earthquake simulations and the brain. Off-the-shelf techniques like R trees do not work -- the objects overlap too much -- so we have made our own spatial indexing. We make custom software when it is necessary, and are not tied to vendors. In geospatial applications, we can create meshes of different shapes -- like tetrahedral or cubes for earthquakes, and cylinders for the brain -- and index these in a geospatial index. But since an R tree is inefficient when objects overlap too much, as these do, we just find one; and then because there is reachability from an object to neighboring ones, we use this to get all the objects in the area of interest.</p> <p align="center">* * *</p> <p>This is obviously a diverse field. Probably the message that we can synthesize out of this is that <b><i>flexibility and parallel programming models are what we need to pay attention to.</i></b> There is a need to go beyond what one can do in SQL while continuing to stay close to the data. Also, allowing for plug-in data types and index structures may be useful; we sometimes get requests for such anyway.</p> <p>The continuing argument around MapReduce and Hadoop is a lasting feature of the landscape. A parallel DB will beat MapReduce any day at joining across partitions; the problem is to overcome the mindset that sees Hadoop as the always-first answer to anything parallel. People will likely have to fail with this before they do anything else. For us, the matter is about having database-resident logic for <a href="http://dbpedia.org/resource/Extract,_transform,_load" class="absuri" id="link-id0x26c06558">extract-transform-load (ETL)</a> that can do data-integration type-transformations and maybe iterative graph algorithms that constantly join across partitions, better than a MapReduce job, while still allowing application logic to be written in Java. Teaching sem-web-heads to write SQL procedures and to know about join order, join type, and partition locality, has proven to be difficult. People do not understand latency, whether in client-server or cluster settings. This is why they do not see the point of stored procedures or of shipping functions to data. This sounds like a terrible indictment, like saying that people do not understand why rivers flow downhill. Yet, it is true. This is also why MapReduce is maybe the only parallel programming paradigm that can be successfully deployed in the absence of this understanding, since it is actually quite latency-tolerant, not having any synchronous cross-partition operations except for the succession of the map and reduce steps themselves.</p> <p>Maybe it is so that the database guys see MapReduce as an insult to their intelligence and the rest of the world sees it as the only understandable way of running <tt><a href="http://dbpedia.org/resource/Grep" class="absuri" id="link-id0x2507a000">grep</a></tt> and <tt><a href="http://dbpedia.org/resource/Sed" class="absuri" id="link-id0x25079ec0">sed</a></tt> (Unix commands for string search/replace) in parallel, with the super bonus of letting you reshuffle the outputs so that you can compare everything to everything else, which <tt>grep</tt> alone never let you do.</p> <p align="center">* * *</p> <p>Making a database that does not need data loading seems a nice idea, and CWI has actually done something in this direction in "<a href="http://infoscience.epfl.ch/record/161489" class="absuri" id="link-id0x250787f0">Here are my Data Files. Here are my Queries. Where are my Results?</a>"] However, there is another product called Algebra Data that claims to take in data without loading and to optimize storage based on access. We do not have immediate plans in this direction. Bulk load is already quite fast (take 100G TPC-H in 70 minutes or so), but faster is always possible.</p>
2012-04-19T16:44:03.000001-04:00
ICDE 2012 (post 5 of 6) - Graphs
http://www.openlinksw.com:443/weblog/oerling/?date=2012-04-17#1706
2012-04-17T19:35:33Z
<p>There were quite a few talks about graphs at <a href="http://www.icde12.org/Site/" class="absuri" id="link-id0x26d23c70">ICDE 2012</a>. Neither the representations of graphs, nor the differences between RDF and generic graph models, entered much into the discussion. On the other hand, graph similarity searches and related were addressed a fair bit.</p> <p>Graph DB and RDF/Linked Data are distinct, if neighboring disciplines. On one hand, graph problems predate Linked Data, and the RDF/Linked Data world is a web artifact, which graphs are not as such, so a slightly different cultural derivation also makes these disjoint. Besides, graphs may imply schema first whereas linked data basically cannot. Then another differentiation might be derived from edges not really being first class citizens in RDF, except for reification, at which the RDF reification vocabulary is miserably inadequate, as pointed out before.</p> <p>RDF is being driven by the web-style publishing of Linked Open Data (LOD), with some standardization and uptake by publishers; Graph DB is not standardized but driven by diverse graph-analytics use cases.</p> <p>There is no necessary reason why these could not converge, but it will be indefinitely long before any standards come to cover this, so best not hold one's breath. Communities are jealous of their borders, so if the neighbor does something similar one tends to emphasize the differences and not the commonalities.</p> <p>So for some things, one could warehouse the original RDF of the web microformats and LOD, and then ETL into some other graph model for specific tasks, or just do these in RDF. Of course, then RDF systems need to offer suitable capabilities. These seem to be about very fast edge traversal within a rather local working set, and about accommodating large, iteratively-updated intermediate results, e.g., edge weights.</p> <p>Judging by the benchmarks paper (<i>Benchmarking traversal operations over graph databases (<a href="http://www.cse.unsw.edu.au/~iwgdm/2012/Slides/Ciglan.pptx" class="absuri" id="link-id0x26e6cc68">Slidedeck (ppt)</a>, <a href="http://ups.savba.sk/~marek/papers/gdm12-ciglan.pdf" class="absuri" id="link-id0x28545128">paper (pdf)</a>);</i> Marek Ciglan, Alex Averbuch, and Ladialav Hluchy.) at the <a href="http://www.cse.unsw.edu.au/~iwgdm/2012/index.html" class="absuri" id="link-id0x26b17d70">GDM workshop</a>, the state of benchmarking in graph databases is even worse than in RDF, where the state is bad enough. The paper's premise was flawed to start, using application logic to do <tt>JOIN</tt>s instead of doing them in the DBMS. In this way, latency comes to dominate, and only the most blatant differences are seen. There is nothing like this style of benchmarking to make an industry look bad. The supercomputer Graph 500 benchmark, on the other hand, lets the contestants make their own implementations on a diversity of architectures with random traversal as well as loading and generating large intermediate results. It is somewhat limited, but still broader than the the graph database benchmarks paper at the GDM workshop.</p> <p> Returning to graphs, there were some papers on similarity search and clique detection. As players in this space, beyond just RDF, we might as well consider implementing necessary features for efficient expression of such problems. The algorithms discussed were expressed in procedural code against memory-based data structures; there is usually no query language or parallel/distributed processing involved.</p> <p> <a href="http://dbpedia.org/resource/MapReduce" class="absuri" id="link-id0x27d27f40">MapReduce</a> has become the default way in which people would tackle such problems at scale; in fact, people do not consider anything else, as far as I can tell. Well, they certainly do not consider MPI for example as a first choice. The parallel array things in Fortran do not at first sight seem very graphy, so this is likely not something that crosses one's mind either.</p> <p>We should try some of the similarity search and clustering in SQL with a parallel programming model. We have excellent expression-evaluation speed from vectoring and unrestricted recursion between partitions, and no file system latencies like MapReduce. The initial test case will be some of the linking/data-integration/mapping workloads in LOD2.</p> <p> Having some sort-of-agreed-upon benchmark for these workloads would make this more worthwhile. Again, we will see what emerges.</p>
2012-04-19T16:43:38.000010-04:00
ICDE 2012 (post 4 of 6) - Graph Data Management Workshop
http://www.openlinksw.com:443/weblog/oerling/?date=2012-04-17#1705
2012-04-17T19:34:20Z
<p>I gave an invited talk ("Virtuoso 7 - Column Store and Adaptive Techniques for Graph" (<a href="http://www.cse.unsw.edu.au/~iwgdm/2012/Slides/Virtuoso.ppt" class="absuri" id="link-id0x27fce2f8">Slides (ppt)</a>)) at the <a href="http://www.cse.unsw.edu.au/~iwgdm/2012/" class="absuri" id="link-id0x27eb6cd8">Graph Data Management Workshop</a> at <a href="http://www.icde12.org/Site/" class="absuri" id="link-id0x26d89980">ICDE 2012</a>.</p> <p>Bryan Thompson of <a href="http://www.systap.com/" class="absuri" id="link-id0x2a893000">Systap</a> (<a href="http://www.systap.com/bigdata.htm" class="absuri" id="link-id0x26c36d70">Bigdata®</a> RDF store) was also invited, so we got to talk about our common interests. He told me about two cool things they have recently done, namely introducing tables to <a href="http://dbpedia.org/resource/SPARQL" class="absuri" id="link-id0x2a74b040">SPARQL</a>, and adding a way of <a href="http://dbpedia.org/resource/Reification_%28computer_science%29" class="absuri" id="link-id0x26c90790">reifying statements</a> that does not rely on extra columns. The table business is just about being able to store a multicolumn result set into a named persistent entity for subsequent processing. But this amounts to a SQL table, so the relational model has been re-arrived at, once more, from practical considerations. The reification just packs all the fields of a triple (or quad) into a single string and this string is then used as an RDF <tt>S</tt> or <tt>O</tt> (Subject or Object), less frequently a <tt>P</tt> or <tt>G</tt> (Predicate or Graph). This works because Bigdata® has variable length fields in all columns of the triple/quad table. The query notation then accepts a function-looking thing in a triple pattern to mark reification. Nice. Virtuoso has a variable length column in only the <tt>O</tt> but could of course have one in also <tt>S</tt> and even in <tt>P</tt> and <tt>G</tt>. The column store would still compress the same as long as reified values did not occur. These values on the other hand would be unlikely to compress very well but run length and dictionary would always work.</p> <p>So, we could do it like Bigdata®, or we could add a "quad ID" column to one of the indices, to give a reification ID to quads. Again no penalty in a column store, if you do not access the column. Or we could make an extra table of <tt>PSOG->R</tt>.</p> <p>Yet another variation would be to make the <tt>SPOG</tt> concatenation a literal that is interned in the RDF literal table, and then used as any literal would be in the <tt>O</tt>, and as an IRI in a special range when occurring as <tt>S</tt>. The relative merits depend on how often something will be reified and on whether one wishes to <tt>SELECT</tt> based on parts of reification. Whichever the case may be, the idea of a function-looking placeholder for a reification is a nice one and we should make a compatible syntax if we do special provenance/reification support. The model in the RDF reification vocabulary is a non-starter and a thing to discredit the sem web for anyone from database.</p> <p>I heard from Bryan that the new <a href="http://www.w3.org/2011/rdf-wg/" class="absuri" id="link-id0x28a628b0">W3 RDF WG</a> had declared provenance out of scope, unfortunately. The word on the street on the other hand is that provenance is increasingly found to be an issue. This is confirmed by the active work of the <a href="http://www.w3.org/2011/prov/" class="absuri" id="link-id0x2540a098">W3 Provenance Working Group</a>.</p>
2012-04-17T15:34:20.000004-04:00
ICDE 2012 (post 3 of 6) - What Is Timely LOD Search Worth?
http://www.openlinksw.com:443/weblog/oerling/?date=2012-04-17#1704
2012-04-17T19:33:21Z
<p>There was a talk (Linked Data and Live Querying for Enabling Support Platforms for Web Dataspaces (<a href="https://sites.google.com/site/desweb2012/parreira.pdf?attredirects=0" class="absuri" id="link-id0x27f2bbe0">Slides (PDF)</a>); Jürgen Umbrich, Marcel Karnstedt, Josiane Xavier Parreira, Axel Polleres and Manfred Hauswirth) at the <a href="https://sites.google.com/site/desweb2012/" class="absuri" id="link-id0x299b5230">Data Engineering Meets the Semantic Web (DESWEB)</a> workshop at <a href="http://www.icde12.org/Site/" class="absuri" id="link-id0x2928bfd8">ICDE</a> last week about the problems of caching LOD, whether attempted by <a href="http://sindice.com/" class="absuri" id="link-id0x278876b8">Sindice</a> or <a href="http://www.openlinksw.com" class="absuri" id="link-id0x2a4ba140">OpenLink</a>'s <a href="http://lod.openlinksw.com/" class="absuri" id="link-id0x299ed1f0">LOD Cloud Cache</a>. The conclusion was that OpenLink covered a bit more of the test data sets and that Sindice was maybe better up to date on the ones that it covered but that neither did it very well. The data sets were random graphs of user FOAF profiles and such collected from some Billion Triples Data set, thus not data that is likely to have commercial value, except in huge quantities maybe for some advertising, except that click streams and the like are much more valuable.</p> <p>Being involved with at least one of these, and being in the audience, I felt obligated to comment. The fact is, neither OpenLink's LOD Cloud Cache nor Sindice is a business, and there is not a business model which could justify keeping them timely on the web crawls they contain. Doing so is easy enough, if there is a good enough reason.</p> <p>The talk did make a couple of worthwhile points: The data does change; and if one queries entities, one encounters large variation in change-frequency across entities and their attributes.</p> <p>The authors suggested to have a piece of middleware decide what things can be safely retrieved from a copy and what have to be retrieved from the source. Not too much is in fact known about the change frequency of the data, except that it changes, as the authors pointed out.</p> <p>The crux of the matter is that the thing that ought to know this best is the query processor at the LOD warehouse. For client-side middleware to split the query, it needs access to statistics that it must get from the warehouse or keep by itself. Of course, in concrete application scenarios, you go to the source if you ask about the weather or traffic jams, and otherwise go to the warehouse based on application-level knowledge.</p> <p>But for actual business intelligence, one needs histories, so a search engine with only the present is not so interesting. At any rate, refreshing the data should leave a trail of past states. Exposing this for online query would just triple the price, so we forget about that for now. Just keeping an append-only table of history is not too much of a problem. One may make extracts from this table into a relational form for specific business questions. There is no point doing such analytics in RDF itself. One would have to just try to see if there is anything remotely exploitable in such histories. Making a history table is easy enough. Maybe I will add one.</p> <p>Let us now see what it would take to operate a web crawl cache that would be properly provisioned, kept fresh, and managed. We base this on the Sindice crawl sizes and our experiments on these; the non-web-crawl LOD Cloud Cache is not included.</p> <p>From previous experience we know the sizing: 5Gt/144GB RAM. Today's best price point is on 24-DIMM E5 boards, so 192GB RAM, or 6.67Gt. A unit like that (8TB HDD, 0.5TB SSD, 192GB RAM, 12 core E5, <a href="http://dbpedia.org/page/InfiniBand" class="absuri" id="link-id0x27eb8158">InfiniBand</a>) costs about $6800.</p> <p>The Sindice crawl is now about 20Gt, so $28K of gear (768GB RAM) is enough. Let us count this 4 times: 2x for anticipated growth; and 2x for running two copies -- one for online, and one for batch jobs. This is 3TB RAM. Power is 16x500W = 8KW, which we could round to 80A at 110V. Colocation comes to $500 for the space, and $1200 per month for power; make it $2500 per month with traffic included.</p> <p>At this rate, 3 year TCO is <tt>$120K + ( 36 * $2.5K ) = $210K</tt>. This takes one person half time to operate, so this is another $50K per year.</p> <p>We do not count software development in this, except some scripting that should be included in the yearly $50K DBA bill.</p> <p>Under what circumstances is such a thing profitable? Or can such a thing be seen as a marketing demo, to be paid for by license or service sales?</p> <p>A third party can operate a system of this sort, but then the cost will be dominated by software licenses if running on Virtuoso cluster.</p> <p>For comparison, the TB at EC2 costs <tt>((( 16 * $2 ) * 24 ) * 31 ) = $24,808</tt> per month. With reserved instances, it is <tt>( 16 * ( $2192 + ((( 0.7 * 24 ) * 365 ) * 3 ))) / 36 = $8938</tt> per month for a 3 year term. Counting at 3TB, the 3 year TCO is $965K at EC2. AWS has volume discounts but they start higher than this; <tt>( 3 * ( 16 * $2K )) = $96K</tt> reserved host premium is under $250K. So if you do not even exceed their first volume discount threshold, it does not look likely you can cut a special deal with AWS.</p> <p>(The AWS prices are calculated with the high memory instances, approximately 64GB usable RAM each. The slightly better CC2 instance is a bit more expensive.)</p> <p>Yet another experiment to make is whether a system as outlined will even run at anywhere close to the performance of physical equipment. This is uncertain; clouds are not for speed, based on what we have seen. They make the most sense when the monthly bill is negligible in relation to the cost of a couple of days of human time.</p>
2012-04-19T16:43:20-04:00
ICDE 2012 (post 2 of 6) - LOD Column Store Experiences and Sizing
http://www.openlinksw.com:443/weblog/oerling/?date=2012-04-17#1703
2012-04-17T19:31:04Z
<p>We have played around with <a href="http://lod.openlinksw.com/" class="absuri" id="link-id0x2a8a8df8">LOD data sets and Virtuoso Column Store</a> for the past several months. I will here give a few numbers and comment on some different platform comparisons that we have made. The answer at the end of this is how to size a system for often-changing web-style data. The conclusion is a data-to-RAM ratio that gives an acceptable working set without driving the price up by forcing 100% RAM residence.</p> <p>The experiment is loading <a href="http://sindice.com/" class="absuri" id="link-id0x28e1fb38">Sindice</a> web crawls. The platform is 2 x Xeon 5520 and 144G RAM. The initial load rate is 200-180Kt, and drops to 100Kt at 5Gt because of I/O. The system is Virtuoso Column Store configured to run as 4 processes and 32 partitions, all on the same box. After 5Gt, we see just more I/O and going further is not relevant; one runs CPU-bound or not at all.</p> <p>We use 4 Crucial SSDs in the setup. The hot structures like the RDF quad indices are on SSD, and the cold ones are on hard disk. A cold structure is a write-only index like the dictionary of literals (id to lit).</p> <p>For bulk load, SSDs turn out not to be particularly useful. For a cold start on the other hand, SSDs cut warmup time of 144G RAM from over half an hour to a couple of minutes. It is possible that Intel SSDs would also help with bulk load, but this has not been tried. The SSD problem during bulk load is that these do not write very fast, and while there are writes in queue, read latency goes up; so under a constant write load, the SSD's famous instantaneous random read no longer works.</p> <p>The fragment considered in the example is 4.95Gt: 8.1M pages worth of quads; 12.7M of literals and iris; and 4.71M of full text index. A page is 8KB. The files on disk contain empty pages, but these do not matter since they do not take up RAM. The quad indices take 13.4 bytes/quad. The row-wise equivalent used to be 38 or so bytes/quad with similar data. Two-thirds of the IRI and literal string data can benefit from column-wise stream compression. (This was not used but if it were, we could count on a 50% drop in size for the data affected, so instead of 12.7M pages, we could maybe get 8.5M on a good day. This could be worth doing but is not a priority.) The system was configured to have 12M database pages in RAM, so a little under half the database pages of the set fit in RAM at one time; thus one cannot call this a memory-only setup. Due to the locality in the unusually non-local data, this is as far as secondary storage can reach without becoming an over-2x slowdown. In practice, we are talking about under 1% of rows accessed coming from secondary storage, but that alone means half throughput.</p> <p>We note that this data set represents the worst that we have seen. It has 129M distinct graphs, 38 t/g. Regular data like the synthetic benchmark sets take half the space per quad. This is about a third of a Sindice crawl; the other two-thirds look the same as far as we looked.</p> <p>So if you are interested in hosting data like this, you can budget 144GB RAM for every 5Gt. Do not try it with anything less. Budgeting double this is wise, so that you have space to cook the data; this is important since in order to do things with it, one needs to at least copy things for materializing transformations.</p> <p>If you are budget-constrained and hosting very regular content like <a href="http://dbpedia.org/page/UniProt" class="absuri" id="link-id0x29b465b8">UniProt</a>, you can budget maybe 144GB RAM for every 10Gt.</p> <p>As for CPU, this does not matter as much as long as you do not go to disk. Just for load speed, Dbpedia is loaded in 300s on a cluster of eight (8) dual AMD 2378 boxes at 2.6GHz (total 8 cores per host, so 64 cores in the cluster), and in 945s on one (1) dual Xeon 5520 box at 2.26GHz (total 8 cores in the host). Intel makes much better CPUs, as we see. Both scenarios are 100% in RAM. For even more regular data, the load rates are a bit higher: 1.3Mt/s for the AMD cluster, and 300Kt/s for the Xeon host.</p> <p>The interconnect for the AMD cluster is 1 x gigE but this does not matter for load. For CPU-bound cross-partition <tt>JOIN</tt>s, 1 or 2 x gigE is insufficient; 4 x gigE might barely make it; <a href="http://dbpedia.org/page/InfiniBand" class="absuri" id="link-id0x26d4cec8">InfiniBand</a> should be safe. When running cross-partition <tt>JOIN</tt>s, a single 8-core Xeon box generates about 300MB/s of interconnect traffic; a gigE connection can maybe take 50MB/s with some luck.</p> <p> <a href="http://en.wikipedia.org/wiki/Xeon#E5-16xx.2F26xx-series_.22Sandy_Bridge-EP.22" class="absuri" id="link-id0x27df5b40">Intel E5</a> is not dramatically better than <a href="http://en.wikipedia.org/wiki/Xeon#Nehalem-based_Xeon" class="absuri" id="link-id0x293e9d50">Nehalem</a> but this is something we will see in a while when we make measurements with real equipment. Prior to the E5 release, we tried Amazon EC2 CC2 ("Cluster Compute Eight Extra Large Instance" -- 2x8 core E5, 2.66GHz). The results were inconclusive; it never did more than 1.9x better than Xeon 5520 even when running an empty loop (i.e., recursive <a href="http://dbpedia.org/resource/Fibonacci_function" class="absuri" id="link-id0x2a480aa0">Fibonacci function</a> in SQL, no cache misses, no I/O). With a database <tt>JOIN</tt>, 1.3x better is the best we saw. But this must be the fault of Amazon and not of E5.</p> <p>We also tried <a href="http://en.wikipedia.org/wiki/List_of_AMD_Opteron_microprocessors#Opteron_6100-series_.22Magny-Cours.22_.2845_nm.29" class="absuri" id="link-id0x27e262a0">AMD "Magny-Cours"</a>, but for 32 cores against 8 it never did over 2x better, more like 1.4x often enough, and and single thread speed was 50% worse, so not a good buy. We did not find a <a href="http://en.wikipedia.org/wiki/List_of_AMD_Opteron_microprocessors#Bulldozer_based_Opterons" class="absuri" id="link-id0x29936f10">Bulldozer</a> to try, and did not feel like buying one since the reviews did not promise more core speed over the Magny-Cours.</p> <p>It seems that especially with Column Store, we are truly CPU-bound and not memory-latency- or bandwidth-bound. This is based on the observation that a Xeon 5620 with 2 of 3 memory channels populated loads BSBM data only 10% faster than the same with 1 of 3 channels populated, with CPU affinity set on a dual socket system.</p> <p>So, if you have a choice between a $2K processor (E5-2690) and a $600 processor (E5-2630), buy the cheaper one and get RAM with the money saved. $1440 buys 128G in $90 8G DIMMs. Then buy E5 boards with 24 DIMMs -- one for every 7Gt of web crawl data. If your software licenses are priced per core, getting higher-clock 4-core E5’s might make sense.</p> <p>While on the subject of bytes and quads/triples, we note that <a href="http://www.systap.com/bigdata.htm" class="absuri" id="link-id0x2784a1d8">Bigdata®</a>'s <a href="http://www.bigdata.com/bigdata/blog/?p=423" class="absuri" id="link-id0x2ac80e68">recent announcement</a> says up to 50 billion triples per single server. Franz loaded at a good 800+ Kt/s rate up to <a href="http://franz.com/agraph/allegrograph/agraph_benchmarks.lhtml" class="absuri" id="link-id0x26c3bea0">a trillion triples</a>. One is led to think from the spec that this was with less than full cpu but still with highly local data, considering 1.5 bytes a triple would hit very heavy I/O otherwise. Their statement to the effect of <a href="http://swat.cse.lehigh.edu/projects/lubm/" class="absuri" id="link-id0x251d4180">LUBM</a>-like data corroborates this, so we are not talking about exactly the same thing.</p> <p>So if you compare the claims, I am talking about running CPU-bound on the worst data there is. Franz and Bigdata® do not specify, so it is hard to compare. LOD2 should in principle publish actual metrics with at least Bigdata®; Franz is not participating in these races.</p> <p>We may publish some more detailed measurements with more varied configurations later. The thing to remember is minimum 144GB RAM for every 5Gt of web crawls, if you want to load and refresh in RAM.</p>
2012-04-17T15:31:04-04:00
ICDE 2012 (post 1 of 6) - LOD2 Plenary
http://www.openlinksw.com:443/weblog/oerling/?date=2012-04-17#1702
2012-04-17T19:28:48Z
<p> <a href="http://lod2.eu/" class="absuri" id="link-id0x1d3c5620">LOD2</a>'s database contributions are, on one hand, Virtuoso Column Store and Elastic Cluster, and on the other, the demonstration and proof from <a href="http://dbpedia.org/resource/Centrum_Wiskunde_&_Informatica" class="absuri" id="link-id0x11356250">CWI</a> that indeed all of the <a href="http://dbpedia.org/resource/Relational_database" class="absuri" id="link-id0x1d6296d8">relational</a> innovations for which CWI is well known apply to <a href="http://dbpedia.org/resource/Graph_%28data_structure%29" class="absuri" id="link-id0x1bcd83d8">graph</a>/<a href="http://dbpedia.org/resource/Resource_Description_Framework" class="absuri" id="link-id0x1befd4a0">RDF</a> data as well.</p> <p>The value is unquestionable both to Virtuoso users in the short-term, and to the state of science and to all RDF users and vendors in the mid-term.</p> <p>The LOD2 claim of "<a href="http://www.openlinksw.com/weblog/oerling/?id=1649" class="absuri" id="link-id0x1c161d88">linking the universe</a>" (my words) will be tested soon enough, after we first put the universe in a bucket. This refers to a real-time quad store of <a href="http://sindice.com/" class="absuri" id="link-id0x1d3eb050">Sindice</a> crawls, plus a warehouse of the LOD data sets.</p> <p>This effort raises a few questions that I will treat in a number of posts to follow, such as --</p> <ul> <li>How do you size a real-time copy of LOD/web data? </li> <li>What does it cost to operate a properly provisioned warehouse of all RDF web crawls?</li> </ul> <p>What is done now is under-provisioned and not kept up to date. We are talking about all the RDF on the web in near real time with arbitrary queries. This is very far from the "billion triples" data sets or vertical portals, which are both easy by comparison.</p>
2012-04-17T15:28:48-04:00
LOD2 Plenary and Review: Semanticist, Think Database!
http://www.openlinksw.com:443/blog/vdb/blog/?date=2011-09-30#1700
2011-09-30T21:02:01Z
<p>Last week the <a href="http://lod2.eu/" id="link-id0x57d7368">LOD2 FP7 project</a> had its first review, preceded by its third plenary meeting.</p> <p>Before this, we did, <a href="http://www.openlinksw.com/weblog/oerling/?id=1683" id="link-id0x579c950">as promised</a>, get the column store and vectored execution capabilities of Virtuoso 7 Single-Server Edition extended to Virtuoso 7 Cluster Edition. More interesting still, we decoupled storage from the database server process, so now database files can migrate between server processes. This means that clusters are now elastic, i.e., new servers can be added to a cluster and the load can be redistributed without reloading the data.</p> <p>These things were long planned, but now are done. Measurements will be published in some weeks, as part of CWI's continued running of RDF store benchmarks, per the LOD2 plan.</p> <p>Doing the column store and elastic cluster is work enough, so I do not in general participate in support or consultancy or the like. This has some pros and cons. On the plus side, there is a relative lack of noise and a very clear idea of focus. Of course, this work is most highly applied, thus always informed by use cases, thus forgetting what ought to be done out there is not the problem. Rather, the problem is forgetting how things in fact <i>are</i> done as opposed to how they <i>could or should be</i> done.</p> <p>To cut a long story short, it has become clear to me that the DBMS must tell the application developer what to do. Of course, the application developer could also look at performance metrics, but they do not, and explaining these metrics is too much work and yields no lasting benefit. Developers will produce all kinds of performance diagnostic traces if requested, but going through this song and dance can also be avoided by the right automation.</p> <p>So, I will introduce two new product features called <i><b>Wazzup?</b></i> and <i><b>Saywhat?</b></i> </p> <p> <b>Wazzup?</b> is answered by a mood line, like "Heavily disk bound: 100G more memory will give 10x speedup" or "Network bound: Processing in larger batches will give 5x more throughput" and <b>Saywhat?</b> is answered by some commentary on the user's last action, for example "there is no ?order with o_totalprice < 0" or "there is no property O_misspelledtotallprrice."</p> <p> <b>Wazzup?</b> is about overall system state, and <b>Saywhat?</b> is about the user session, specifically query plans. But an explanation of a query plan is not understandable, so this will just point out some salient facts, like the reason why the answer comes out empty.</p> <p>The other thing that came to my attention is the fact that a user has no instinctive feel for <a href="http://dbpedia.org/page/Extract,_transform,_load" id="link-id0x527eb88">ETL</a>. A database person takes it for a self-evident truth that data is loaded in bulk, but the application developer does not think of that. Likewise, the line between warehousing and federating is not instinctively felt; actually the question is not even posed in these terms. So one will find Web protocols and end-points and glue code on the app server when one ought to have ETL and adequate hardware for running the consolidated database.</p> <p>Further, under-provisioning of equipment is endemic with semanticists. The Semantic Web gets a needlessly bad rap just because we find too much data on too little equipment. For example, I was surprised to learn that the Linked Geodata demo ran on only 16 GB RAM and 6 processor cores with 2 billion triples and 350 million points in a geo index. Now, even with our greatest space efficiency advances, there is no way this will run from memory.</p> <p>It is not that the Web 2.0 stack is necessarily efficient (we hear the wildest stories of lack of database understanding from that side too), but at least there is a culture of running with enough equipment. Surely when the web-scale data gear (e.g. Google Bigtable, Yahoo PNUTS, Amazon Dynamo) was new, by the operators' own admission there was no way for this to be particularly efficient, database-wise. Not if your eventual consistency is a client application to a shared MySQL back-end. For a lookup or single-record-update workload, who cares when there is enough hardware? For analytics, there is the <i>de facto</i> impossibility of doing big joins, but map reduce is for that, all offline. The big web houses have always known how to deal with data; it is the smaller Web 2.0 guys who patch systems together with duct tape and memcache. Even so, the online experience gets created.</p> <p>Semanticism has no part of this outlook, except maybe for Freebase, but then they are from California and now have been inside Google for a while.</p> <p>We quite understand that when one needs to get big data online, one makes a key-value store as a point solution, because this way one owns what one operates, and the time to market is a lot shorter than if one tried building all this inside a general-purpose DBMS. Besides, the people who can in fact do this almost do not exist, and even if one had a whole army of this rare breed, development is not very scalable in a tightly-integrated system like a high-performance DBMS. Still further, to even start, one needs to own the DBMS, meaning that the initial platform must be known through and through. This is an issue even though open source platforms exist.</p> <p>The graph data, semdata, schema-last, RDF, linked data enterprise -- whatever one calls it -- makes the bold proposition of bringing complex-query-at-scale to heterogeneous data. This is a database claim.</p> <p>In the meantime, test deployments are made in defiance of database best practices. This is a bit like test driving a race car in reverse gear and steering by looking in the rear-view mirror.</p> <p>There is also no short-term scalable way to educate people. At the LOD2 review, one comment was that an integrated project ought to clearly indicate how to set up the tool chain for good performance, specially as concerns interfaces between the tools. This is very true. Experience shows that developers of tools cannot accurately anticipate what usage patterns will emerge in the field. Therefore, we propose to do better than just documentation; we will make the server recognize the common sources of inefficiency and point the user to the right action.</p> <h3>Provisioning and usage patterns: The DBMS ought to know best.</h3> <p>Imagine the following conversation:</p> <p> <b>DBMS:</b> Your application does single-triple INSERTs over client-server protocol all day, from a single client. 57% of real time goes in client server latency, 40% in cluster interconnect latency, 2% in compiling the statements, and 1% in doing the work. Use array parameters or bulk load from a file.</p> <p> <b>Operator:</b> My developers use industry-standard Java class libraries with a service-oriented architecture and strictly enforced interfaces. This is called software engineering. Watch out ere you raise your voice against the canon.</p> <p> <i>[Some weeks later, after the load job has gone on for 10 days and gotten a third of the way, developers have discovered that JDBC has array parameters and are trying these.]</i> </p> <p> <b>DBMS:</b> 60% of real time goes into waiting for locks. 10% of transactions get aborted for deadlock. Transactions consist of an average of 10 client-server operations. Use stored procedures; acquire locks in predictable order; do SELECT FOR UPDATE. Throughput will be 4x higher if client-server operations are merged into a single operation. The transactions only INSERT; hence consider bulk load instead.</p> <p> <b>Operator</b>: We are using an enterprise-class three-tier architecture. It has "enterprise" in the name and all the big guys are using it, so it must be scalable. Besides, it is distributed transactions, and distributed computing is the wave of the future. You are a cluster yourself, so the pot's got no business calling the kettle black.</p> <p> <i>[After a while, the data gets loaded with bulk load, but now on a single stream.]</i> </p> <p> <b>DBMS</b>: CPU is at 400% for an INSERT workload; adding more parallel threads will get 4.5x better throughput.</p> <p> <i>[Some time has elapsed and there are Ajax client apps out there trying to use the data.]</i> </p> <p> <b>DBMS</b>: Will you really not give me another 140 GB RAM and 16 more cores?</p> <p> <b>Operator</b>: No, on general principles I will not, shut up.</p> <p> <b>DBMS</b>: Do you know that your page impression takes 3 seconds and anything over 0.25 seconds is visibly slow? 300 GB worth of distinct pages have been accessed in the last 24 hours for 160 GB of RAM. Latency will drop 10x by using SSD; 50x by increasing RAM.</p> <p> <b>Operator</b>: No dice, bucket. Shut up, besides, when I scroll through the data I always use for testing, I get it fast enough, you are just doing this out of greed and self-importance. You are a server among many, just like the mail server; you databases are just pretentious.</p> <p>Currently addressing any of the above sorts of issues takes a long time and involves mostly-avoidable support communication. Questions of this sort do occur. We can probably produce commentary like the above based on logging some 50 numbers, and making some 15 regularly-run reports over these. The patterns to watch out for are well known. No, we will not make a Zippy the Pinhead office assistant; a computer should not try to be cute. This one will talk only in terms of gains from adjusting the deployment or usage patterns.</p> <p>Now, suppose the operator said <i>yes</i> to the request for more cores and memory; then it would be up to the DBMS to deliver. This entails a capacity to redistribute itself automatically, and to give a quantitative report on the success of this measure. This means usage-based repartitioning of the data to equalize load over a cluster. The relevant metric in the above case is the drop in response time. On the other hand, the DBMS should also notice if there is clearly unused capacity.</p> <p>This all will be presented as a line in the status report, so there is no extra wizard or workload analyzer that one must remember to run. For programmatic use there are SQL views for the relevant reports.</p> <p>As for ETL, even if the DBMS can detect that it is not being done right, this does not mean that the user will know what to do. Therefore, for all the Web harvesting we support, as well as any import from local file system or Web services, with some RDF-ization, we will simply implement a proper ETL utility that will do things right. <b>Wazzup?</b> can just point the user to that if the workload looks like loading. This will have its own status report giving a load and transform rate and will point out what takes the longest, after everything is duly parallelized and made asynchronous.</p> <p>Beyond these lessons, there is more to say about the review and plenary, we will get to that a bit later. We did promise a new edition of the LOD cache in a couple of months, now on the clustered column-store platform. Look for advances in data discoverability.</p>
2015-08-21T15:43:29.091466-04:00
LOD2 Plenary and Review: Semanticist, Think Database!
http://www.openlinksw.com:443/weblog/oerling/?date=2011-09-29#1698
2011-09-29T14:50:47Z
<p>Last week the <a href="http://lod2.eu/" id="link-id0x57d7368">LOD2 FP7 project</a> had its first review, preceded by its third plenary meeting.</p> <p>Before this, we did, <a href="http://www.openlinksw.com/weblog/oerling/?id=1683" id="link-id0x579c950">as promised</a>, get the column store and vectored execution capabilities of Virtuoso 7 Single-Server Edition extended to Virtuoso 7 Cluster Edition. More interesting still, we decoupled storage from the database server process, so now database files can migrate between server processes. This means that clusters are now elastic, i.e., new servers can be added to a cluster and the load can be redistributed without reloading the data.</p> <p>These things were long planned, but now are done. Measurements will be published in some weeks, as part of CWI's continued running of RDF store benchmarks, per the LOD2 plan.</p> <p>Doing the column store and elastic cluster is work enough, so I do not in general participate in support or consultancy or the like. This has some pros and cons. On the plus side, there is a relative lack of noise and a very clear idea of focus. Of course, this work is most highly applied, thus always informed by use cases, thus forgetting what ought to be done out there is not the problem. Rather, the problem is forgetting how things in fact <i>are</i> done as opposed to how they <i>could or should be</i> done.</p> <p>To cut a long story short, it has become clear to me that the DBMS must tell the application developer what to do. Of course, the application developer could also look at performance metrics, but they do not, and explaining these metrics is too much work and yields no lasting benefit. Developers will produce all kinds of performance diagnostic traces if requested, but going through this song and dance can also be avoided by the right automation.</p> <p>So, I will introduce two new product features called <i><b>Wazzup?</b></i> and <i><b>Saywhat?</b></i> </p> <p> <b>Wazzup?</b> is answered by a mood line, like "Heavily disk bound: 100G more memory will give 10x speedup" or "Network bound: Processing in larger batches will give 5x more throughput" and <b>Saywhat?</b> is answered by some commentary on the user's last action, for example "there is no ?order with o_totalprice < 0" or "there is no property O_misspelledtotallprrice."</p> <p> <b>Wazzup?</b> is about overall system state, and <b>Saywhat?</b> is about the user session, specifically query plans. But an explanation of a query plan is not understandable, so this will just point out some salient facts, like the reason why the answer comes out empty.</p> <p>The other thing that came to my attention is the fact that a user has no instinctive feel for <a href="http://dbpedia.org/page/Extract,_transform,_load" id="link-id0x527eb88">ETL</a>. A database person takes it for a self-evident truth that data is loaded in bulk, but the application developer does not think of that. Likewise, the line between warehousing and federating is not instinctively felt; actually the question is not even posed in these terms. So one will find Web protocols and end-points and glue code on the app server when one ought to have ETL and adequate hardware for running the consolidated database.</p> <p>Further, under-provisioning of equipment is endemic with semanticists. The Semantic Web gets a needlessly bad rap just because we find too much data on too little equipment. For example, I was surprised to learn that the Linked Geodata demo ran on only 16 GB RAM and 6 processor cores with 2 billion triples and 350 million points in a geo index. Now, even with our greatest space efficiency advances, there is no way this will run from memory.</p> <p>It is not that the Web 2.0 stack is necessarily efficient (we hear the wildest stories of lack of database understanding from that side too), but at least there is a culture of running with enough equipment. Surely when the web-scale data gear (e.g. Google Bigtable, Yahoo PNUTS, Amazon Dynamo) was new, by the operators' own admission there was no way for this to be particularly efficient, database-wise. Not if your eventual consistency is a client application to a shared MySQL back-end. For a lookup or single-record-update workload, who cares when there is enough hardware? For analytics, there is the <i>de facto</i> impossibility of doing big joins, but map reduce is for that, all offline. The big web houses have always known how to deal with data; it is the smaller Web 2.0 guys who patch systems together with duct tape and memcache. Even so, the online experience gets created.</p> <p>Semanticism has no part of this outlook, except maybe for Freebase, but then they are from California and now have been inside Google for a while.</p> <p>We quite understand that when one needs to get big data online, one makes a key-value store as a point solution, because this way one owns what one operates, and the time to market is a lot shorter than if one tried building all this inside a general-purpose DBMS. Besides, the people who can in fact do this almost do not exist, and even if one had a whole army of this rare breed, development is not very scalable in a tightly-integrated system like a high-performance DBMS. Still further, to even start, one needs to own the DBMS, meaning that the initial platform must be known through and through. This is an issue even though open source platforms exist.</p> <p>The graph data, semdata, schema-last, RDF, linked data enterprise -- whatever one calls it -- makes the bold proposition of bringing complex-query-at-scale to heterogeneous data. This is a database claim.</p> <p>In the meantime, test deployments are made in defiance of database best practices. This is a bit like test driving a race car in reverse gear and steering by looking in the rear-view mirror.</p> <p>There is also no short-term scalable way to educate people. At the LOD2 review, one comment was that an integrated project ought to clearly indicate how to set up the tool chain for good performance, specially as concerns interfaces between the tools. This is very true. Experience shows that developers of tools cannot accurately anticipate what usage patterns will emerge in the field. Therefore, we propose to do better than just documentation; we will make the server recognize the common sources of inefficiency and point the user to the right action.</p> <h3>Provisioning and usage patterns: The DBMS ought to know best.</h3> <p>Imagine the following conversation:</p> <p> <b>DBMS:</b> Your application does single-triple INSERTs over client-server protocol all day, from a single client. 57% of real time goes in client server latency, 40% in cluster interconnect latency, 2% in compiling the statements, and 1% in doing the work. Use array parameters or bulk load from a file.</p> <p> <b>Operator:</b> My developers use industry-standard Java class libraries with a service-oriented architecture and strictly enforced interfaces. This is called software engineering. Watch out ere you raise your voice against the canon.</p> <p> <i>[Some weeks later, after the load job has gone on for 10 days and gotten a third of the way, developers have discovered that JDBC has array parameters and are trying these.]</i> </p> <p> <b>DBMS:</b> 60% of real time goes into waiting for locks. 10% of transactions get aborted for deadlock. Transactions consist of an average of 10 client-server operations. Use stored procedures; acquire locks in predictable order; do SELECT FOR UPDATE. Throughput will be 4x higher if client-server operations are merged into a single operation. The transactions only INSERT; hence consider bulk load instead.</p> <p> <b>Operator</b>: We are using an enterprise-class three-tier architecture. It has "enterprise" in the name and all the big guys are using it, so it must be scalable. Besides, it is distributed transactions, and distributed computing is the wave of the future. You are a cluster yourself, so the pot's got no business calling the kettle black.</p> <p> <i>[After a while, the data gets loaded with bulk load, but now on a single stream.]</i> </p> <p> <b>DBMS</b>: CPU is at 400% for an INSERT workload; adding more parallel threads will get 4.5x better throughput.</p> <p> <i>[Some time has elapsed and there are Ajax client apps out there trying to use the data.]</i> </p> <p> <b>DBMS</b>: Will you really not give me another 140 GB RAM and 16 more cores?</p> <p> <b>Operator</b>: No, on general principles I will not, shut up.</p> <p> <b>DBMS</b>: Do you know that your page impression takes 3 seconds and anything over 0.25 seconds is visibly slow? 300 GB worth of distinct pages have been accessed in the last 24 hours for 160 GB of RAM. Latency will drop 10x by using SSD; 50x by increasing RAM.</p> <p> <b>Operator</b>: No dice, bucket. Shut up, besides, when I scroll through the data I always use for testing, I get it fast enough, you are just doing this out of greed and self-importance. You are a server among many, just like the mail server; you databases are just pretentious.</p> <p>Currently addressing any of the above sorts of issues takes a long time and involves mostly-avoidable support communication. Questions of this sort do occur. We can probably produce commentary like the above based on logging some 50 numbers, and making some 15 regularly-run reports over these. The patterns to watch out for are well known. No, we will not make a Zippy the Pinhead office assistant; a computer should not try to be cute. This one will talk only in terms of gains from adjusting the deployment or usage patterns.</p> <p>Now, suppose the operator said <i>yes</i> to the request for more cores and memory; then it would be up to the DBMS to deliver. This entails a capacity to redistribute itself automatically, and to give a quantitative report on the success of this measure. This means usage-based repartitioning of the data to equalize load over a cluster. The relevant metric in the above case is the drop in response time. On the other hand, the DBMS should also notice if there is clearly unused capacity.</p> <p>This all will be presented as a line in the status report, so there is no extra wizard or workload analyzer that one must remember to run. For programmatic use there are SQL views for the relevant reports.</p> <p>As for ETL, even if the DBMS can detect that it is not being done right, this does not mean that the user will know what to do. Therefore, for all the Web harvesting we support, as well as any import from local file system or Web services, with some RDF-ization, we will simply implement a proper ETL utility that will do things right. <b>Wazzup?</b> can just point the user to that if the workload looks like loading. This will have its own status report giving a load and transform rate and will point out what takes the longest, after everything is duly parallelized and made asynchronous.</p> <p>Beyond these lessons, there is more to say about the review and plenary, we will get to that a bit later. We did promise a new edition of the LOD cache in a couple of months, now on the clustered column-store platform. Look for advances in data discoverability.</p>
2011-09-29T14:48:59-04:00
GDB for the Data Driven Age (STI Summit Position Paper)
http://www.openlinksw.com:443/blog/vdb/blog/?date=2011-07-26#1697
2011-07-26T13:37:26Z
<p> <i><b>Note:</b> The following was written prior to the event, but was not posted until later due to human error.</i> </p> <p>The <a href="http://sti2.org/" id="link-id0x261e3798">Semantic Technology Institute</a> (<a href="http://sti2.org/" id="link-id0x243dac30">STI</a>) is organizing <a href="http://summit2011.sti2.org/" id="link-id0x25fc4e68">a meeting</a> around the questions of making semantic technology deliver on its promise. We were asked to present a position paper (reproduced below). This is another recap of our position on making graph databasing come of age. While the database technology matters are getting tackled, we are drawing closer to the question of deciding actually what kind of inference will be needed close to the data. My personal wish is to use this summit for clarifying exactly what is needed from the database in order to extract value from the data explosion. We have a good idea of what to do with queries but what is the exact requirement for transformation and alignment of schema and identifiers? What is the actual use case of inference, OWL or other, in this? It is time to get very concrete in terms of applications. We expect a mixed requirement but it is time to look closely at the details.</p> <h3>GDB for the Data Driven Age</h3> <p>Databases and knowledge representation both have decades of history, but to date the exchange of ideas and techniques between these disciplines has been limited. The intuition that there would be value in greater cooperation has not failed to occur to researchers on either side; after all, both sides deal with data. From this, we have seen deductive databases emerge, as well as more recently "database friendly" profiles of OWL.</p> <p>In this position paper we will examine what, in the most concrete terms, is needed in order to bring leading edge database technology together with expressive querying and reasoning. This draws on our experience in building <a href="http://virtuoso.openlinksw.com/" id="link-id0x240cdd28">Virtuoso</a>, one of today's leading <a href="http://dbpedia.org/page/Graph_database" id="link-id0x24ceaae0">graph data stores</a>. Following this, we argue for the creation of benchmarks and challenges that in fact do reflect reality and facilitate open and fair comparison of products and technologies.</p> <p>Data integration is often mentioned as the motivating use case for GDB, commonly popularized today as RDF. Database research has over the past few years produced great advances for business intelligence (i.e., complex queries and read-mostly workloads). These advances are typified by compressed columnar storage and architecture-conscious execution models, mostly based on the idea of always processing multiple sets of values in each operation (vectoring). With these techniques, raw performance with relatively simple schemas and regular data (e.g., TPC-H) is no longer a barrier to extracting value from data.</p> <p>A similar breakthrough has not been seen on the semantics side. Data integration still requires manual labor. Publishing GDB datasets is a good and necessary intermediate stage, but producing these datasets from diverse sources is not fundamentally different from doing the same work without GDB or RDF. Even so, GDB and RDF serve as a catalyst for a culture of publishing datasets.</p> <p>GDB, as a base model for integration, offers the following benefits over a purely relational result format: </p> <ul> <li>All entities have globally unique identifiers.</li> <li>Any statements may be associated ad hoc to any entities.</li> <li>These statements can be scoped into graphs according to their provenance, time, validity, etc.</li> </ul> <p>Obtaining this flexibility on a relational basis would simply require moving to an graph-like representation with essentially one-row-per-attribute. Indeed, we see key-value stores being used in online applications with high volatility of schema (e.g., social networks, search); and we also see relational applications making provisions for post-hoc addition of per-entity attributes (i.e., associating a bag of mixed non-first normal form data with entities). The benefits of a schema-last approach are recognized in many places.</p> <p>GDB seems <i>a priori</i> a fit for all these requirements, thus how will it claim its place as a solution?</p> <p>The first part of the answer lies in learning all the relevant database lessons. The second part lies in eliminating the impedance mismatch between querying and reasoning. The third and most important part consists of substantiating these claims in a manner that is understandable to the relevant publics, finally leading to the creation of a semantics-aware segment of the database industry. We will address each of these aspects in turn.</p> <h4>GDB and RDB</h4> <p>The problem is divided into storage format, execution, and query optimization. For the first two, Daniel Abadi's <a href="http://cs-www.cs.yale.edu/homes/dna/papers/abadiphd.pdf" id="link-id0x25ebd568">renowned Ph.D. thesis</a> holds most of the keys. Space efficiency is specially important for Linked Data, since data is often voluminous, and many datasets have to be brought together for integration. Access patterns are also unpredictable, with indexed-random-access predominating, as opposed to RDB BI workloads where sequential scans and hash joins represent the bulk of the work. However, we find that a sorted column-wise compressed representation of Linked Data with a single quad table for all statements gives excellent space efficiency and good random access as well as random insert speed. The space efficiency is close to par with the equivalent column-wise relational format, since three of the four columns of the quad table compress to almost nothing. As many sort orders as are necessary may be maintained, but we find that two are enough, with some extra data structures for dealing with queries where the predicate is unspecified. The details are found in VLDB 2010 Semdata workshop paper, <i><a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtDirectionsChallengesSemdata" id="link-id0x244a8010">Directions and Challenges for Semantically Linked Data</a></i>. Since GDB/RDF is a model typed at run time, the engine must support an "<code>ANY</code>" data type for columns and query variables, where values on successive rows may be of different types. This is a straightforward enhancement.</p> <p>Vectored execution is traditionally associated with column stores because the per-row access cost is relatively high, thus needing to access many nearby rows at a time in order to amortize the overhead. Aside this, vectored execution provides many opportunities for parallelism, from the instruction level all the way to threading and distributed execution on clusters, thus some form of execution on large numbers of concurrent query states is needed for RDF stores, just as it is needed for RDBMS"s.</p> <p>Query optimization for GDBMS is similar to that for RDBMS, except that the statistics can no longer be collected by column and table, but must rather apply to individual entities and ranges of a single quad table. This can be provided through run-time sampling of the database based on constants in the query being optimized. This may take into account trivial inference such as expanding properties into the set of their sub-properties and the like. Beyond this, interleaving execution and optimization (as in <a href="http://oai.cwi.nl/oai/asset/14193/14193B.pdf" id="link-id0x264cfd20">ROX</a>) seems to offer limitless possibilities, especially when inference is introduced, making optimizer statistics less predictive. </p> <p>In summary, starting with an RDBMS and going to GDB entails changes to all parts of the engine, but these changes are not fundamental. One does need to own the engine; however, otherwise the expertise for efficiently implementing these changes will not exist. Essentially any DBMS technique may be translated to a GDB use case, if its application can be decided at run-time. GDB may be schema-less, yet most datasets have fairly regular structure; the question is simply to reconstruct the needed statistics and schema information from the data on an as you go basis. Techniques with high up-front cost, like constructing specially ordered materializations for optimizing specific queries, are harder to deploy but still conceivable for GDB also.</p> <h4>RDB and Inference</h4> <p>Compared to the straightforwardly performance oriented world of database engines, the contours of the landscape become less defined when moving to inference. Databases, whether relational or schema-less all perform roughly the same functions but inference is more diverse. We include here also techniques like machine learning and meta-reasoning for guiding reasoning, although these might not strictly fit the definition.</p> <p>As we posit that data integration is the motivating use case for GDB as opposed to RDB (Relational Database Model), we must ask which modes of inference are actually required for data integration. Further, we need to ask whether these inferences ought to be applied as a preprocessing step (ETL or forward chaining), or as needed (backward chaining). Some low-hanging fruit can be collected by simply constructing class or property hierarchies; e.g., in the data at hand, the following properties have the meaning of company name, and the following classes have the meaning of company. We have found that such techniques can be efficiently supported at run-time, without materialization, if the support is simply built into the engine, which is in itself straightforward as long as one controls the engine. The same applies to trivial identity resolution, such as <code>owl:sameAs</code> or resolution of identity based on sharing an inverse-functional property value. These things take longer at run-time, but if one caches and reuses the result, one can get around materialization.</p> <p>We do not believe in weak statements of identity, as in <i>X is similar to Y,</i> since the meaning of similarity is entirely contextual. X and Y may or may not be interchangeable depending on the application; thus the statement on identity needs to be strong, but it must be easy to modify the grounds on which such a statement is made. This is a further argument for why one should not automatically materialize consequences of identity, particularly if dealing with web data where identity is especially problematic.</p> <p>Real-world problems are however harder than just bundling properties, classes, or instances into sets of interchangeable equivalents, which is all we have mentioned thus far. There are differences of modeling ("address as many columns in customer table" vs. "address normalized away under a contact entity"), normalization ("first name" and "last name" as one or more properties; national conventions on person names; tags as comma-separated in a string or as a one-to-many), incomplete data (one customer table has family income bracket, the other does not), diversity in units of measurement (Imperial vs. metric), variability in the definition of units (seven different things all called blood pressure), variability in unit conversions (currency exchange rates), to name a few. What a world!</p> <p>If data exists, the conversion questions are often answerable but their answer depends on context -- e.g., date of transaction for currency exchange rate; source of data for the definition of blood pressure.</p> <p>Alongside these, there remain issues of identity, e.g., depending on the perspective, a national subsidiary is or is not the same entity as the parent company, companies with the same name can be entirely unrelated in different jurisdictions.</p> <p>It appears that we may need a multi-level approach, combining different techniques for different phases of the integration process. We do not <i>a priori</i> believe that using SQL VIEWs for unit and modeling conversion, and then OWL for unifying terminology on top of this, were the whole solution. Even if this were the solution, the pipeline from the relational sources to SPARQL and OWL needs to be optimized for real-world BI information volumes, and the query language needs to be able to express the business questions and needs to interface with the reporting tools the analyst has come to expect.</p> <p>Our answer so far consists of a SPARQL extension with non-recursive rules, roughly equivalent to SQL VIEWs in expressive power, tightly integrated to the query engine. There is also limited support for recursion through transitive subqueries; thus one can compactly express things like "all parts of all assemblies and subassemblies must satisfy applicable safety requirements, where the requirements depend on the type of the part in question."</p> <p>This is only an intermediate step. We believe that a database-scale generic inference engine with at least Datalog power, with second-order extensions like computed predicates, is needed, executing inside the DBMS, benefiting from the whole array of optimizations database-science expects of execution engines, as part of the answer.</p> <p>This will not relieve the analyst of having to consider that the currency rates in effect at the time of conversion must be taken into account when calculating profits, but this will at least make expressing this and similar pieces of context more compact.</p> <p>We note that time-to-answer has historically won over raw performance. This was also the case for RDBMS when these were the fresh challenger to the CODASYL incumbents, just as was the case with the adoption of high-level languages. The key is that the raw performance must be sufficient for the real world task. With the adoption of the database lessons outlined in the previous section, we believe this to be the case for GDB (and thus, RDF).</p> <h4>Substantiating the Claims</h4> <p>Benchmarks have a stellar record for improving any metric they measure. The question is, how can we make a metric that measures GDB's ability to deliver on its claim to fame -- time-to-answer for big data -- with all the integration and other complexities this entails?</p> <p>So far, GDB benchmarks have consisted of workloads where RDBMS are clearly better (e.g., LUBM, or the Berlin SPARQL Benchmark). This does not remove their usefulness for GDB, but does not constitute a GDB selling point, either.</p> <p>We suggest a dual approach. The first part is demonstrating that GDB is scalable for BI: We take the industry standard decision support benchmark TPC-H, which is very favorable to RDB and quite unfavorable to GDB, and show that we can tackle the workload at reasonable cost. If TPC-H is all one wants, an RDBMS will stay a better fit, but then this benchmark does not capture any of the heterogeneity, schema evolution, or other such requirements faced by real-world data warehouses. This is still a qualification test, not the selling point.</p> <p>The issue of benchmark is inextricably tied to the issue of messaging. There must be a compelling story, with which the IT community can identify. Further, the benchmark must capture real-world challenges in the area of interest. With all this, the benchmark should not be too expensive to run. Here too, a multistage approach suggests itself.</p> <p>Our tentative answer to this question is the Social Intelligence Benchmark (SIB), developed together with CWI and other partners in the LOD2 consortium. This simulates a social network and combines an online workload with complex analytics. This benchmark should cover all of the target areas of the LOD2 project, so that the project itself generates its own metric of success. The project has clear data integration targets, especially as applies to Web and Linked Data. Questions of integration with enterprise sources need to be further developed; for example, comparing CRM data with extractions from the online conversation space for market research.</p> <p>Data integration will invariably involve human effort, and the area cannot be satisfactorily covered with metrics of scale and throughput alone. Development time, accuracy of results, and cost of maintenance are all factors. Furthermore, the task being modeled must correspond to reality, still without being too domain-specific or prohibitively time-consuming to implement.</p> <h4>Conclusions</h4> <p>The data driven world will increase rewards for efficiency in data integration. We believe that such efficiency crucially depends on semantics. Real world requirements just might throw the database and AI communities together with enough heat and pressure for fusion to ignite, allegorically speaking. Without a clear and present need, the geek world analog of electrostatic repulsion will keep the communities separate, as has been the case thus far, and no new, qualitatively-different element will arise.</p> <p>Efforts such as this STI Summit and the LOD2 Project are needed for setting directions and communicating the requirement to the research world. In our fusion analogy, this is the field which directs the nuclei to collide.</p> <p>Once there is an actual reaction that produces more than it consumes by a sufficient margin, regular business dynamics will take over, and we will have an industry with several products of comparable capability, as well as a set of metrics, all to the benefit of the end user.</p> <h4>References</h4> <ul> <li> <p>TPC-H <a href="http://www.tpc.org/tpch/" id="link-id0x25b910e8">results pages</a> </p> </li> <li> <p>Daniel Abadi's Ph.D. Thesis, <i>Query Execution in Column-Oriented Database Systems</i> ( <a href="http://cs-www.cs.yale.edu/homes/dna/papers/abadiphd.pdf" id="link-id0x25f8eeb0">PDF</a> )</p> </li> <li> <p>Our VLDB 2010 Semdata workshop paper, <i>Directions and Challenges for Semantically Linked Data</i> ( <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtDirectionsChallengesSemdata" id="link-id0x25f88520">HTML</a> | <a href="http://virtuoso.openlinksw.com/whitepapers/Directions_and_Challenges_for_Semantically_Linked_Data.pdf" id="link-id0x271416b8">PDF</a> )</p> </li> <li> <p>CWI's <i>ROX: Run-time Optimization of XQueries</i> ( <a href="http://oai.cwi.nl/oai/asset/14193/14193B.pdf" id="link-id0x2699ac78">PDF</a> )</p> </li> <li> <p>The <a href="http://lod2.eu/" id="link-id0x25856fc8">LOD2 Project web site</a> </p> </li> </ul>
2011-07-26T09:40:34.000002-04:00
GDB for the Data Driven Age (STI Summit Position Paper)
http://www.openlinksw.com:443/weblog/oerling/?date=2011-07-26#1696
2011-07-26T13:36:58Z
<p> <i><b>Note:</b> The following was written prior to the event, but was not posted until later due to human error.</i> </p> <p>The <a href="http://sti2.org/" id="link-id0x261e3798">Semantic Technology Institute</a> (<a href="http://sti2.org/" id="link-id0x243dac30">STI</a>) is organizing <a href="http://summit2011.sti2.org/" id="link-id0x25fc4e68">a meeting</a> around the questions of making semantic technology deliver on its promise. We were asked to present a position paper (reproduced below). This is another recap of our position on making graph databasing come of age. While the database technology matters are getting tackled, we are drawing closer to the question of deciding actually what kind of inference will be needed close to the data. My personal wish is to use this summit for clarifying exactly what is needed from the database in order to extract value from the data explosion. We have a good idea of what to do with queries but what is the exact requirement for transformation and alignment of schema and identifiers? What is the actual use case of inference, OWL or other, in this? It is time to get very concrete in terms of applications. We expect a mixed requirement but it is time to look closely at the details.</p> <h3>GDB for the Data Driven Age</h3> <p>Databases and knowledge representation both have decades of history, but to date the exchange of ideas and techniques between these disciplines has been limited. The intuition that there would be value in greater cooperation has not failed to occur to researchers on either side; after all, both sides deal with data. From this, we have seen deductive databases emerge, as well as more recently "database friendly" profiles of OWL.</p> <p>In this position paper we will examine what, in the most concrete terms, is needed in order to bring leading edge database technology together with expressive querying and reasoning. This draws on our experience in building <a href="http://virtuoso.openlinksw.com/" id="link-id0x240cdd28">Virtuoso</a>, one of today's leading <a href="http://dbpedia.org/page/Graph_database" id="link-id0x24ceaae0">graph data stores</a>. Following this, we argue for the creation of benchmarks and challenges that in fact do reflect reality and facilitate open and fair comparison of products and technologies.</p> <p>Data integration is often mentioned as the motivating use case for GDB, commonly popularized today as RDF. Database research has over the past few years produced great advances for business intelligence (i.e., complex queries and read-mostly workloads). These advances are typified by compressed columnar storage and architecture-conscious execution models, mostly based on the idea of always processing multiple sets of values in each operation (vectoring). With these techniques, raw performance with relatively simple schemas and regular data (e.g., TPC-H) is no longer a barrier to extracting value from data.</p> <p>A similar breakthrough has not been seen on the semantics side. Data integration still requires manual labor. Publishing GDB datasets is a good and necessary intermediate stage, but producing these datasets from diverse sources is not fundamentally different from doing the same work without GDB or RDF. Even so, GDB and RDF serve as a catalyst for a culture of publishing datasets.</p> <p>GDB, as a base model for integration, offers the following benefits over a purely relational result format: </p> <ul> <li>All entities have globally unique identifiers.</li> <li>Any statements may be associated ad hoc to any entities.</li> <li>These statements can be scoped into graphs according to their provenance, time, validity, etc.</li> </ul> <p>Obtaining this flexibility on a relational basis would simply require moving to an graph-like representation with essentially one-row-per-attribute. Indeed, we see key-value stores being used in online applications with high volatility of schema (e.g., social networks, search); and we also see relational applications making provisions for post-hoc addition of per-entity attributes (i.e., associating a bag of mixed non-first normal form data with entities). The benefits of a schema-last approach are recognized in many places.</p> <p>GDB seems <i>a priori</i> a fit for all these requirements, thus how will it claim its place as a solution?</p> <p>The first part of the answer lies in learning all the relevant database lessons. The second part lies in eliminating the impedance mismatch between querying and reasoning. The third and most important part consists of substantiating these claims in a manner that is understandable to the relevant publics, finally leading to the creation of a semantics-aware segment of the database industry. We will address each of these aspects in turn.</p> <h4>GDB and RDB</h4> <p>The problem is divided into storage format, execution, and query optimization. For the first two, Daniel Abadi's <a href="http://cs-www.cs.yale.edu/homes/dna/papers/abadiphd.pdf" id="link-id0x25ebd568">renowned Ph.D. thesis</a> holds most of the keys. Space efficiency is specially important for Linked Data, since data is often voluminous, and many datasets have to be brought together for integration. Access patterns are also unpredictable, with indexed-random-access predominating, as opposed to RDB BI workloads where sequential scans and hash joins represent the bulk of the work. However, we find that a sorted column-wise compressed representation of Linked Data with a single quad table for all statements gives excellent space efficiency and good random access as well as random insert speed. The space efficiency is close to par with the equivalent column-wise relational format, since three of the four columns of the quad table compress to almost nothing. As many sort orders as are necessary may be maintained, but we find that two are enough, with some extra data structures for dealing with queries where the predicate is unspecified. The details are found in VLDB 2010 Semdata workshop paper, <i><a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtDirectionsChallengesSemdata" id="link-id0x244a8010">Directions and Challenges for Semantically Linked Data</a></i>. Since GDB/RDF is a model typed at run time, the engine must support an "<code>ANY</code>" data type for columns and query variables, where values on successive rows may be of different types. This is a straightforward enhancement.</p> <p>Vectored execution is traditionally associated with column stores because the per-row access cost is relatively high, thus needing to access many nearby rows at a time in order to amortize the overhead. Aside this, vectored execution provides many opportunities for parallelism, from the instruction level all the way to threading and distributed execution on clusters, thus some form of execution on large numbers of concurrent query states is needed for RDF stores, just as it is needed for RDBMS"s.</p> <p>Query optimization for GDBMS is similar to that for RDBMS, except that the statistics can no longer be collected by column and table, but must rather apply to individual entities and ranges of a single quad table. This can be provided through run-time sampling of the database based on constants in the query being optimized. This may take into account trivial inference such as expanding properties into the set of their sub-properties and the like. Beyond this, interleaving execution and optimization (as in <a href="http://oai.cwi.nl/oai/asset/14193/14193B.pdf" id="link-id0x264cfd20">ROX</a>) seems to offer limitless possibilities, especially when inference is introduced, making optimizer statistics less predictive. </p> <p>In summary, starting with an RDBMS and going to GDB entails changes to all parts of the engine, but these changes are not fundamental. One does need to own the engine; however, otherwise the expertise for efficiently implementing these changes will not exist. Essentially any DBMS technique may be translated to a GDB use case, if its application can be decided at run-time. GDB may be schema-less, yet most datasets have fairly regular structure; the question is simply to reconstruct the needed statistics and schema information from the data on an as you go basis. Techniques with high up-front cost, like constructing specially ordered materializations for optimizing specific queries, are harder to deploy but still conceivable for GDB also.</p> <h4>RDB and Inference</h4> <p>Compared to the straightforwardly performance oriented world of database engines, the contours of the landscape become less defined when moving to inference. Databases, whether relational or schema-less all perform roughly the same functions but inference is more diverse. We include here also techniques like machine learning and meta-reasoning for guiding reasoning, although these might not strictly fit the definition.</p> <p>As we posit that data integration is the motivating use case for GDB as opposed to RDB (Relational Database Model), we must ask which modes of inference are actually required for data integration. Further, we need to ask whether these inferences ought to be applied as a preprocessing step (ETL or forward chaining), or as needed (backward chaining). Some low-hanging fruit can be collected by simply constructing class or property hierarchies; e.g., in the data at hand, the following properties have the meaning of company name, and the following classes have the meaning of company. We have found that such techniques can be efficiently supported at run-time, without materialization, if the support is simply built into the engine, which is in itself straightforward as long as one controls the engine. The same applies to trivial identity resolution, such as <code>owl:sameAs</code> or resolution of identity based on sharing an inverse-functional property value. These things take longer at run-time, but if one caches and reuses the result, one can get around materialization.</p> <p>We do not believe in weak statements of identity, as in <i>X is similar to Y,</i> since the meaning of similarity is entirely contextual. X and Y may or may not be interchangeable depending on the application; thus the statement on identity needs to be strong, but it must be easy to modify the grounds on which such a statement is made. This is a further argument for why one should not automatically materialize consequences of identity, particularly if dealing with web data where identity is especially problematic.</p> <p>Real-world problems are however harder than just bundling properties, classes, or instances into sets of interchangeable equivalents, which is all we have mentioned thus far. There are differences of modeling ("address as many columns in customer table" vs. "address normalized away under a contact entity"), normalization ("first name" and "last name" as one or more properties; national conventions on person names; tags as comma-separated in a string or as a one-to-many), incomplete data (one customer table has family income bracket, the other does not), diversity in units of measurement (Imperial vs. metric), variability in the definition of units (seven different things all called blood pressure), variability in unit conversions (currency exchange rates), to name a few. What a world!</p> <p>If data exists, the conversion questions are often answerable but their answer depends on context -- e.g., date of transaction for currency exchange rate; source of data for the definition of blood pressure.</p> <p>Alongside these, there remain issues of identity, e.g., depending on the perspective, a national subsidiary is or is not the same entity as the parent company, companies with the same name can be entirely unrelated in different jurisdictions.</p> <p>It appears that we may need a multi-level approach, combining different techniques for different phases of the integration process. We do not <i>a priori</i> believe that using SQL VIEWs for unit and modeling conversion, and then OWL for unifying terminology on top of this, were the whole solution. Even if this were the solution, the pipeline from the relational sources to SPARQL and OWL needs to be optimized for real-world BI information volumes, and the query language needs to be able to express the business questions and needs to interface with the reporting tools the analyst has come to expect.</p> <p>Our answer so far consists of a SPARQL extension with non-recursive rules, roughly equivalent to SQL VIEWs in expressive power, tightly integrated to the query engine. There is also limited support for recursion through transitive subqueries; thus one can compactly express things like "all parts of all assemblies and subassemblies must satisfy applicable safety requirements, where the requirements depend on the type of the part in question."</p> <p>This is only an intermediate step. We believe that a database-scale generic inference engine with at least Datalog power, with second-order extensions like computed predicates, is needed, executing inside the DBMS, benefiting from the whole array of optimizations database-science expects of execution engines, as part of the answer.</p> <p>This will not relieve the analyst of having to consider that the currency rates in effect at the time of conversion must be taken into account when calculating profits, but this will at least make expressing this and similar pieces of context more compact.</p> <p>We note that time-to-answer has historically won over raw performance. This was also the case for RDBMS when these were the fresh challenger to the CODASYL incumbents, just as was the case with the adoption of high-level languages. The key is that the raw performance must be sufficient for the real world task. With the adoption of the database lessons outlined in the previous section, we believe this to be the case for GDB (and thus, RDF).</p> <h4>Substantiating the Claims</h4> <p>Benchmarks have a stellar record for improving any metric they measure. The question is, how can we make a metric that measures GDB's ability to deliver on its claim to fame -- time-to-answer for big data -- with all the integration and other complexities this entails?</p> <p>So far, GDB benchmarks have consisted of workloads where RDBMS are clearly better (e.g., LUBM, or the Berlin SPARQL Benchmark). This does not remove their usefulness for GDB, but does not constitute a GDB selling point, either.</p> <p>We suggest a dual approach. The first part is demonstrating that GDB is scalable for BI: We take the industry standard decision support benchmark TPC-H, which is very favorable to RDB and quite unfavorable to GDB, and show that we can tackle the workload at reasonable cost. If TPC-H is all one wants, an RDBMS will stay a better fit, but then this benchmark does not capture any of the heterogeneity, schema evolution, or other such requirements faced by real-world data warehouses. This is still a qualification test, not the selling point.</p> <p>The issue of benchmark is inextricably tied to the issue of messaging. There must be a compelling story, with which the IT community can identify. Further, the benchmark must capture real-world challenges in the area of interest. With all this, the benchmark should not be too expensive to run. Here too, a multistage approach suggests itself.</p> <p>Our tentative answer to this question is the Social Intelligence Benchmark (SIB), developed together with CWI and other partners in the LOD2 consortium. This simulates a social network and combines an online workload with complex analytics. This benchmark should cover all of the target areas of the LOD2 project, so that the project itself generates its own metric of success. The project has clear data integration targets, especially as applies to Web and Linked Data. Questions of integration with enterprise sources need to be further developed; for example, comparing CRM data with extractions from the online conversation space for market research.</p> <p>Data integration will invariably involve human effort, and the area cannot be satisfactorily covered with metrics of scale and throughput alone. Development time, accuracy of results, and cost of maintenance are all factors. Furthermore, the task being modeled must correspond to reality, still without being too domain-specific or prohibitively time-consuming to implement.</p> <h4>Conclusions</h4> <p>The data driven world will increase rewards for efficiency in data integration. We believe that such efficiency crucially depends on semantics. Real world requirements just might throw the database and AI communities together with enough heat and pressure for fusion to ignite, allegorically speaking. Without a clear and present need, the geek world analog of electrostatic repulsion will keep the communities separate, as has been the case thus far, and no new, qualitatively-different element will arise.</p> <p>Efforts such as this STI Summit and the LOD2 Project are needed for setting directions and communicating the requirement to the research world. In our fusion analogy, this is the field which directs the nuclei to collide.</p> <p>Once there is an actual reaction that produces more than it consumes by a sufficient margin, regular business dynamics will take over, and we will have an industry with several products of comparable capability, as well as a set of metrics, all to the benefit of the end user.</p> <h4>References</h4> <ul> <li> <p>TPC-H <a href="http://www.tpc.org/tpch/" id="link-id0x25b910e8">results pages</a> </p> </li> <li> <p>Daniel Abadi's Ph.D. Thesis, <i>Query Execution in Column-Oriented Database Systems</i> ( <a href="http://cs-www.cs.yale.edu/homes/dna/papers/abadiphd.pdf" id="link-id0x25f8eeb0">PDF</a> )</p> </li> <li> <p>Our VLDB 2010 Semdata workshop paper, <i>Directions and Challenges for Semantically Linked Data</i> ( <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtDirectionsChallengesSemdata" id="link-id0x25f88520">HTML</a> | <a href="http://virtuoso.openlinksw.com/whitepapers/Directions_and_Challenges_for_Semantically_Linked_Data.pdf" id="link-id0x271416b8">PDF</a> )</p> </li> <li> <p>CWI's <i>ROX: Run-time Optimization of XQueries</i> ( <a href="http://oai.cwi.nl/oai/asset/14193/14193B.pdf" id="link-id0x2699ac78">PDF</a> )</p> </li> <li> <p>The <a href="http://lod2.eu/" id="link-id0x25856fc8">LOD2 Project web site</a> </p> </li> </ul>
2011-07-26T09:39:43-04:00
The 2011 STI Semantic Summit
http://www.openlinksw.com:443/blog/vdb/blog/?date=2011-07-22#1695
2011-07-22T15:49:15Z
<p>I was recently at the <a href="http://www.openlinksw.com:80/www.sti2.org/events/2011-sti-semantic-summit" id="link-id0x2308d838">STI 2011 summit in Riga, Latvia</a>. This is a meeting of senior participants in the semantic web and sem tech scene, organized by <a href="http://www.openlinksw.com:80/www.sti2.org/" id="link-id0x25076168">STI</a> of <a href="http://dbpedia.org/page/Dieter_Fensel" id="link-id0x24d2e998">Dieter Fensel</a> fame, with board members like <a href="http://www.michaelbrodie.com/" id="link-id0x224b4b58">Michael Brodie</a>, <a href="http://www.iks-project.eu/community/people/mark-greaves" id="link-id0x2308d4a8">Mark Greaves</a>, and <a href="http://dbpedia.org/page/James_Hendler" id="link-id0x24c192d0">Jim Hendler</a>.</p> <p>This is substantially about the intersection of AI, knowledge representation, and databases. As we have said before, the database side has not been very prominent in these meetings in the past, but this time we had <a href="http://homepages.cwi.nl/~boncz/" id="link-id0x26654260">Peter Boncz</a> of CWI, of MonetDB and VectorWise fame, attending the proceedings.</p> <p>Will DB and AI finally meet? Well, they have met, but how do they get along? Before I try to answer this, let us look at some background.</p> <p>At present, CWI and <a href="http://www.openlinksw.com/" id="link-id0x24724fe0">OpenLink</a> are working together in the <a href="http://lod2.eu/" id="link-id0x24e20d90">LOD2 EU FP7 project</a>, around the general topic of bringing the best of <a href="http://dbpedia.org/page/Relational_database" id="link-id0x2475f128">Relational Database</a> (RDB) science to the <a href="http://dbpedia.org/page/Graph_database" id="link-id0x2474e988">Graph Database</a> (GDB) world. Virtuoso has for a few months had a column store capability (which is about to be made available for public preview). CWI has a long history of column store work, with MonetDB and Ingres VectorWise as results. OpenLink's column store implementation is separate in terms of code but is of course influenced by the work at CWI and other published column store results. The plan is to transplant the applicable CWI innovations into the graph context within Virtuoso. These improvements naturally also benefit Virtuoso RDB (SQL), but the LOD2 project is primarily concerned with GDB applications. The RDB yardstick for much of this work is <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x22a96588">TPC-H</a>, of which we have made a GDB translation. CWI is uniquely qualified as concerns this in light of VectorWise holding some of the top places in the TPC-H charts.</p> <p>Even now, we do in fact run the 22 TPC-H queries in SPARQL against the Virtuoso column store. True, these run faster in SQL against relational tables but we have established a beach head. From this initial position, we can incrementally improve the GDB/SPARQL and RDB/SQL functions, and see how close to SQL we get with SPARQL. I will make a separate post commenting on the differences between SQL and SPARQL.</p> <p>So let's get back to Riga. Mark Greaves said in his opening comments that he would be sick if he once again heard complaining about how bad and un-scalable the tools were. From all the talks, I did get the overall impression that just better databasing for Graph Data is still needed. OK, we have 1-1/2 years of unreleased work just for that about to hit the street; advances are substantial. Along these lines, the people from <a href="http://www.bio2rdf.org/" id="link-id0x2315c088">Bio2RDF</a> pointed out that there still is a cost to publishing query services, specially for complex queries. Well, this cost will be substantially reduced.</p> <p>The takeaway from the meeting is that the most useful thing, for both our public and ourselves, is simply to keep advancing database tech for graph data. In the first instance, this is about launching what we already have; in the second, about going through the CWI record of innovation and adapting this to GDB.</p> <p>The thinking is that once query-answering on some tens-of-billions of triples is easily interactive no matter what question one asks, a tipping point will be reached, and GDB can efficiently play the role of data-melting-pot that has been envisioned for it.</p> <p>This is just a beginning, though. Michael Brodie has on a number of occasions pointed out that that (relational) database guys are only about performance with little or no regard to meaning or even questions of the applicability of the relational model. Peter Boncz then comments back that it can well be that the bulk of IT expenditure worldwide in fact goes into data integration. However, data integration is an "<a href="http://dbpedia.org/page/AI-complete" id="link-id0x24754170">AI-complete</a>" problem with infinite variety and consequent difficulty of measurement. So, making better database engines stands a much greater chance of success and has the nicety of relatively unambiguous metrics. </p> <p>Quite so. We are somewhere in the middle. I'd say that GDB is still at the stage where making better databases is a matter of make-or-break and not a matter of cutting already vanishingly-short response times just for the sake of it. We will have progress if we just keep at it; for now, performance is still a basic need and not a luxury.</p> <p>Now that there is all this potentially integrable data published as graphs (most commonly as RDF serializations), what do we do? Someone at the Riga meeting suggested we take a look across the tracks to the RDB world to see what is being done there for data integration. The question is raised, what does GDB have for data integration? The automatic answer that GDB and RDF have OWL is not adequate, as was rightly pointed out by many. Having schema-last, global identifiers, and some culture of vocabulary reuse is nice, but this is only a start. To cite an example, <code>owl:sameAs</code> will not work when entities simply do not align: One database models a product as a parts hierarchy; another does the same but now based on the materials used in the parts. One tree just has a node that is not in the other. Besides, things like string matching (as in extracting area codes from phone numbers) are common, and OWL specifically excludes any such functions.</p> <p>It is now time to look at what will come after all the database advances. In my talk I outlined some things that have or are about to get solutions:</p> <ul> <li> <p> <b>Database technology:</b> Applying advances from RDB (specifically columns, vectoring, and some adaptive query execution) will make GDB a possibility for data warehousing at some scale.</p> </li> <li> <p> <b>Benchmarks:</b> These advances will be demonstrable through benchmarking. There is a better suite of benchmarks with many variations of BSBM, an GDB-modified TPC-H, and the upcoming Social Intelligence Benchmark (SIBB) with actual graph data. There are the beginnings of an auditing process for result publishing, and a fair chance the semdata world will get its analog of the TPC.</p> </li> </ul> <p>After these basics are more or less in hand, we have a vista of more diverse questions:</p> <ul> <li> <p>What to do about inference? We do not want OWL or RIF for their own sake; instead we want whatever will declaratively facilitate making sense of data. This is an entirely use-case-driven question. If this can have a reasonably generic answer, we will build it into the engine. </p> </li> <li> <p>Data integration is highly diverse, and tool sets like IBM Infosphere have thousands of modules and functions for different aspects of the problem. To what degree does it make sense to put DI-oriented capabilities into a DBMS? </p> </li> <li> <p>Is it the case that SQL or SPARQL, plus or minus a few details, is as powerful as a language can be while staying application domain-agnostic? In other words, if more powerful reasoning is built into the query language, will the requirements vary so much between application domains that the work is not generally applicable? <a href="http://dbpedia.org/page/Datalog" id="link-id0x2403b2f0">Datalog</a> is general enough, but can we demonstrate substantially reduced time to answer with big data if this is built into the engine? <a href="http://boom.cs.berkeley.edu/" id="link-id0x23ed5730">Berkeley Orders Of Magnitude</a> claims this, even though their claim is not exactly in a database context. We need use cases to refine the actual requirement for inference.</p> </li> </ul> <p>In all these questions, we of necessity turn to the user community. In fact we do not follow the usage of these technologies as much as we ought to. One outcome of the Riga summit is a set of public challenges that will hopefully ameliorate this state of matters, to be released soon.</p> <p>The general feeling was that there is more going on on the data side than the AI side. The LOD movement proceeds and lightweight everything predominates, also for knowledge representation. There was some discussion about "pay as you go" integration. On the one hand, there is no up-front integration of information systems just for its own sake, so pay as you go is the only kind that exists, system by system, as the need becomes sufficient. On the other hand, each such integration is a process which has its distinct steps and maintenance and within itself it is planned, and thus pre-paid, so to speak. We need more work with the data itself to better understand the matter. The open government data should offer a playground for this and there will be a special challenge around this.</p> <p> <a href="http://schema.org/" id="link-id0x2475a708">Schema.org</a> and <a href="http://www.w3.org/TR/microdata/" id="link-id0x2a6f8b40">Microdata</a> got their share of discussion. As we see it, it is good that search engines make their pre-competitive data open. This is better than, for example, Google wanting retailers to put their catalogs in Google Base. We do not care about the specific syntax in which data is embedded; we support them all. Microdata converts easily to triples, and if one wants to make a tabular extraction for use with relational tools, this too is simple enough. Applications will have to do their own entity resolution, but this is independent of data publication format. </p> <p>All in all, the mood was positive. Mark Greaves noted in his closing remarks that there has been a 1000x increase in published GDB data over a few years. There is in fact a large quantity of technology for tackling almost any aspect of the LOD value chain, but people do not necessarily know about this nor is it easy to integrate. Still there would be great value in integration. Getting software to interoperate in a meaningful way is manual labor, so it might make sense to organize hackathons around this. While the STI Summit is for the senior people, there could be a parallel track of events for bringing the coders together to actually practice tool integration and interoperation.</p>
2011-07-22T11:49:15.000046-04:00
The 2011 STI Semantic Summit
http://www.openlinksw.com:443/weblog/oerling/?date=2011-07-22#1694
2011-07-22T15:12:34Z
<p>I was recently at the <a href="http://www.openlinksw.com:80/www.sti2.org/events/2011-sti-semantic-summit" id="link-id0x2308d838">STI 2011 summit in Riga, Latvia</a>. This is a meeting of senior participants in the semantic web and sem tech scene, organized by <a href="http://www.openlinksw.com:80/www.sti2.org/" id="link-id0x25076168">STI</a> of <a href="http://dbpedia.org/page/Dieter_Fensel" id="link-id0x24d2e998">Dieter Fensel</a> fame, with board members like <a href="http://www.michaelbrodie.com/" id="link-id0x224b4b58">Michael Brodie</a>, <a href="http://www.iks-project.eu/community/people/mark-greaves" id="link-id0x2308d4a8">Mark Greaves</a>, and <a href="http://dbpedia.org/page/James_Hendler" id="link-id0x24c192d0">Jim Hendler</a>.</p> <p>This is substantially about the intersection of AI, knowledge representation, and databases. As we have said before, the database side has not been very prominent in these meetings in the past, but this time we had <a href="http://homepages.cwi.nl/~boncz/" id="link-id0x26654260">Peter Boncz</a> of CWI, of MonetDB and VectorWise fame, attending the proceedings.</p> <p>Will DB and AI finally meet? Well, they have met, but how do they get along? Before I try to answer this, let us look at some background.</p> <p>At present, CWI and <a href="http://www.openlinksw.com/" id="link-id0x24724fe0">OpenLink</a> are working together in the <a href="http://lod2.eu/" id="link-id0x24e20d90">LOD2 EU FP7 project</a>, around the general topic of bringing the best of <a href="http://dbpedia.org/page/Relational_database" id="link-id0x2475f128">Relational Database</a> (RDB) science to the <a href="http://dbpedia.org/page/Graph_database" id="link-id0x2474e988">Graph Database</a> (GDB) world. Virtuoso has for a few months had a column store capability (which is about to be made available for public preview). CWI has a long history of column store work, with MonetDB and Ingres VectorWise as results. OpenLink's column store implementation is separate in terms of code but is of course influenced by the work at CWI and other published column store results. The plan is to transplant the applicable CWI innovations into the graph context within Virtuoso. These improvements naturally also benefit Virtuoso RDB (SQL), but the LOD2 project is primarily concerned with GDB applications. The RDB yardstick for much of this work is <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x22a96588">TPC-H</a>, of which we have made a GDB translation. CWI is uniquely qualified as concerns this in light of VectorWise holding some of the top places in the TPC-H charts.</p> <p>Even now, we do in fact run the 22 TPC-H queries in SPARQL against the Virtuoso column store. True, these run faster in SQL against relational tables but we have established a beach head. From this initial position, we can incrementally improve the GDB/SPARQL and RDB/SQL functions, and see how close to SQL we get with SPARQL. I will make a separate post commenting on the differences between SQL and SPARQL.</p> <p>So let's get back to Riga. Mark Greaves said in his opening comments that he would be sick if he once again heard complaining about how bad and un-scalable the tools were. From all the talks, I did get the overall impression that just better databasing for Graph Data is still needed. OK, we have 1-1/2 years of unreleased work just for that about to hit the street; advances are substantial. Along these lines, the people from <a href="http://www.bio2rdf.org/" id="link-id0x2315c088">Bio2RDF</a> pointed out that there still is a cost to publishing query services, specially for complex queries. Well, this cost will be substantially reduced.</p> <p>The takeaway from the meeting is that the most useful thing, for both our public and ourselves, is simply to keep advancing database tech for graph data. In the first instance, this is about launching what we already have; in the second, about going through the CWI record of innovation and adapting this to GDB.</p> <p>The thinking is that once query-answering on some tens-of-billions of triples is easily interactive no matter what question one asks, a tipping point will be reached, and GDB can efficiently play the role of data-melting-pot that has been envisioned for it.</p> <p>This is just a beginning, though. Michael Brodie has on a number of occasions pointed out that that (relational) database guys are only about performance with little or no regard to meaning or even questions of the applicability of the relational model. Peter Boncz then comments back that it can well be that the bulk of IT expenditure worldwide in fact goes into data integration. However, data integration is an "<a href="http://dbpedia.org/page/AI-complete" id="link-id0x24754170">AI-complete</a>" problem with infinite variety and consequent difficulty of measurement. So, making better database engines stands a much greater chance of success and has the nicety of relatively unambiguous metrics. </p> <p>Quite so. We are somewhere in the middle. I'd say that GDB is still at the stage where making better databases is a matter of make-or-break and not a matter of cutting already vanishingly-short response times just for the sake of it. We will have progress if we just keep at it; for now, performance is still a basic need and not a luxury.</p> <p>Now that there is all this potentially integrable data published as graphs (most commonly as RDF serializations), what do we do? Someone at the Riga meeting suggested we take a look across the tracks to the RDB world to see what is being done there for data integration. The question is raised, what does GDB have for data integration? The automatic answer that GDB and RDF have OWL is not adequate, as was rightly pointed out by many. Having schema-last, global identifiers, and some culture of vocabulary reuse is nice, but this is only a start. To cite an example, <code>owl:sameAs</code> will not work when entities simply do not align: One database models a product as a parts hierarchy; another does the same but now based on the materials used in the parts. One tree just has a node that is not in the other. Besides, things like string matching (as in extracting area codes from phone numbers) are common, and OWL specifically excludes any such functions.</p> <p>It is now time to look at what will come after all the database advances. In my talk I outlined some things that have or are about to get solutions:</p> <ul> <li> <p> <b>Database technology:</b> Applying advances from RDB (specifically columns, vectoring, and some adaptive query execution) will make GDB a possibility for data warehousing at some scale.</p> </li> <li> <p> <b>Benchmarks:</b> These advances will be demonstrable through benchmarking. There is a better suite of benchmarks with many variations of BSBM, an GDB-modified TPC-H, and the upcoming Social Intelligence Benchmark (SIBB) with actual graph data. There are the beginnings of an auditing process for result publishing, and a fair chance the semdata world will get its analog of the TPC.</p> </li> </ul> <p>After these basics are more or less in hand, we have a vista of more diverse questions:</p> <ul> <li> <p>What to do about inference? We do not want OWL or RIF for their own sake; instead we want whatever will declaratively facilitate making sense of data. This is an entirely use-case-driven question. If this can have a reasonably generic answer, we will build it into the engine. </p> </li> <li> <p>Data integration is highly diverse, and tool sets like IBM Infosphere have thousands of modules and functions for different aspects of the problem. To what degree does it make sense to put DI-oriented capabilities into a DBMS? </p> </li> <li> <p>Is it the case that SQL or SPARQL, plus or minus a few details, is as powerful as a language can be while staying application domain-agnostic? In other words, if more powerful reasoning is built into the query language, will the requirements vary so much between application domains that the work is not generally applicable? <a href="http://dbpedia.org/page/Datalog" id="link-id0x2403b2f0">Datalog</a> is general enough, but can we demonstrate substantially reduced time to answer with big data if this is built into the engine? <a href="http://boom.cs.berkeley.edu/" id="link-id0x23ed5730">Berkeley Orders Of Magnitude</a> claims this, even though their claim is not exactly in a database context. We need use cases to refine the actual requirement for inference.</p> </li> </ul> <p>In all these questions, we of necessity turn to the user community. In fact we do not follow the usage of these technologies as much as we ought to. One outcome of the Riga summit is a set of public challenges that will hopefully ameliorate this state of matters, to be released soon.</p> <p>The general feeling was that there is more going on on the data side than the AI side. The LOD movement proceeds and lightweight everything predominates, also for knowledge representation. There was some discussion about "pay as you go" integration. On the one hand, there is no up-front integration of information systems just for its own sake, so pay as you go is the only kind that exists, system by system, as the need becomes sufficient. On the other hand, each such integration is a process which has its distinct steps and maintenance and within itself it is planned, and thus pre-paid, so to speak. We need more work with the data itself to better understand the matter. The open government data should offer a playground for this and there will be a special challenge around this.</p> <p> <a href="http://schema.org/" id="link-id0x2475a708">Schema.org</a> and <a href="http://www.w3.org/TR/microdata/" id="link-id0x2a6f8b40">Microdata</a> got their share of discussion. As we see it, it is good that search engines make their pre-competitive data open. This is better than, for example, Google wanting retailers to put their catalogs in Google Base. We do not care about the specific syntax in which data is embedded; we support them all. Microdata converts easily to triples, and if one wants to make a tabular extraction for use with relational tools, this too is simple enough. Applications will have to do their own entity resolution, but this is independent of data publication format. </p> <p>All in all, the mood was positive. Mark Greaves noted in his closing remarks that there has been a 1000x increase in published GDB data over a few years. There is in fact a large quantity of technology for tackling almost any aspect of the LOD value chain, but people do not necessarily know about this nor is it easy to integrate. Still there would be great value in integration. Getting software to interoperate in a meaningful way is manual labor, so it might make sense to organize hackathons around this. While the STI Summit is for the senior people, there could be a parallel track of events for bringing the coders together to actually practice tool integration and interoperation.</p>
2011-07-22T11:12:34.000002-04:00
Transaction Semantics in RDF and Relational Models
http://www.openlinksw.com:443/blog/vdb/blog/?date=2011-03-22#1692
2011-03-22T23:55:43Z
<p>As a part of defining benchmark audit for testing <a class="auto-href" href="http://dbpedia.org/resource/ACID" id="link-id0x1cfc6e38">ACID</a> properties on <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1f1302b8">RDF</a> stores, we will here examine different RDF scenarios where lack of concurrency control causes inconsistent results. In so doing, we consider common implementation techniques and implications as concern locking (pessimistic) and multi-version (optimistic) concurrency control schemes.</p> <p>In the following, we will talk in terms of triples, but the discussion can be trivially generalized to quads. We will use numbers for IRIs and literals. In most implementations, the internal representation for these is indeed a number (or at least some <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x1728a9a8">data</a> type that has a well defined collation order). For ease of presentation, we consider a single index with key parts <code>SPO</code>. Any other index-like setting with any possible key order will have similar issues. </p> <h2>Insert (Create) and Delete </h2> <p> <code>INSERT</code> and <code>DELETE</code> as defined in <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x16dee7f8">SPARQL</a> are queries which generate a result set which is then used for instantiating triple patterns. We note that a <code>DELETE</code> may delete a triple which the <code>DELETE</code> has not read; thus the delete set is not a subset of the read set. The <a class="auto-href" href="http://dbpedia.org/resource/SQL" id="link-id0x1e3afb78">SQL</a> equivalent is the </p> <blockquote> <code><pre>DELETE FROM table WHERE key IN ( SELECT key1 FROM other_table )</pre> </code> </blockquote> <p>expression, supposing it were implemented as a scan of <code>other_table</code> and an index lookup followed by <code>DELETE</code> on table. </p> <p>The meaning of <code>INSERT</code> is that the triples in question exist after the operation, and the meaning of <code>DELETE</code> is that said triples do not exist. In a transactional context, this means that the after-image of the transaction is guaranteed either to have or not-have said triples. </p> <p>Suppose that the triples <code>{ 1 0 0 }</code>, <code>{ 1 5 6 }</code>, and <code>{ 1 5 7 }</code> exist in the beginning. If we <code>DELETE { 1 ?x ?y }</code> and concurrently <code>INSERT { 1 2 4 . 1 2 3 . 1 3 5 }</code>, then whichever was considered to be first by the concurrency control of the DBMS would complete first, and the other after that. Thus the end state would either have no triples with subject <code>1</code> or would have the three just inserted. </p> <p>Suppose the <code>INSERT</code> inserts the first triple, <code>{ 1 2 4 }</code>. The <code>DELETE</code> at the same time reads all triples with subject <code>1</code>. The exclusive read waits for the uncommitted <code>INSERT</code>. The <code>INSERT</code> then inserts the second triple, <code>{ 1 2 3 }</code>. Depending on the isolation of the read, this either succeeds, since no <code>{ 1 2 3 }</code> was read, or causes a deadlock. The first corresponds to <code>REPEATABLE READ</code> isolation; the second to <code>SERIALIZABLE</code>.</p> <p>We would not get the desired end-state of either <i>all the inserted triples</i> or <i>no triples with subject <code>1</code></i> if the read or the <code>DELETE</code> were not serializable.</p> <p>Furthermore if a <code>DELETE</code> template produced a triple that did not exist in the pre-image, the <code>DELETE</code> semantics still imply that this also does not exist in the after-image, which implies serializability.</p> <h2>Read and Update</h2> <p>Let us consider the prototypical transaction example of transferring funds from one account to another. Two balances are updated, and a history record is inserted.</p> <p>The initial state is </p> <blockquote> <code><pre>a balance 10 b balance 10</pre></code> </blockquote> <p>We transfer <code>1</code> from <code>a</code> to <code>b</code>, and at the same time transfer <code>2</code> from <code>b</code> to <code>a</code>. The end state must have <code>a</code> at <code>11</code> and <code>b</code> at <code>9</code>.</p> <p>A relational database needs <code>REPEATABLE READ</code> isolation for this.</p> <p>With RDF, <code>txn1</code> reads that <code>a</code> has a <code>balance</code> of <code>10</code>. At the same time, <code>txn1</code> reads the <code>balance</code> of <code>a</code>. <code>txn2</code> waits because the read of <code>txn1</code> is exclusive. <code>txn1</code> proceeds and read the <code>balance</code> of <code>b</code>. It then updates the <code>balance</code> of <code>a</code> and <code>b</code>. </p> <p>All goes without the deadlock which is always cited in this scenario, because the locks are acquired in the same order. The act of updating the balance of <code>a</code>, since RDF does not really have an update-in-place, consists of deleting <code>{ a balance 10 }</code> and inserting <code>{ a balance 9 }</code>. This gets done and <code>txn1</code> commits. At this point, <code>txn2</code> proceeds after its wait on the row that stated <code>{ a balance 10 }</code>. This row is now gone, and <code>txn2</code> sees that <code>a</code> has no balance, which is quite possible in RDF's <a class="auto-href" href="http://dbpedia.org/resource/Database_schema" id="link-id0x1ebb94c8">schema</a>-less model.</p> <p>We see that <code>REPEATABLE READ</code> is not adequate with RDF, even though it is with relational. The reason why there is no <code>UPDATE</code>-in-place is that the <code>PRIMARY KEY</code> of the triple includes all the parts, including the object. Even in a <a class="auto-href" href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1ca86578">RDBMS</a>, an <code>UPDATE</code> of a primary key part amounts to a <code>DELETE</code>-plus-<code>INSERT</code>. One could here argue that an implementation might still <code>UPDATE</code>-in-place if the key order were not changed. This would resolve the special case of the accounts but not a more general case.</p> <p>Thus we see that the read of the balance must be <code>SERIALIZABLE</code>. This means that the read locks the space before the first balance, so that no insertion may take place. In this way the read of <code>txn2</code> waits on the lock that is conceptually before the first possible match of <code>{ a balance ?x }</code>.</p> <h2>locking order and OLTP </h2> <p>To implement <a class="auto-href" href="http://www.tpc.org/" id="link-id0x1e811d68">TPC</a>-<a class="auto-href" href="http://dbpedia.org/resource/C%2B%2B" id="link-id0x1df9c990">C</a>, I would update the table with the highest cardinality first, and then all tables in descending order of cardinality. In this way, the locks with the highest likelihood for contention are held for the least time. If locking multiple rows of a table, these should be locked in a deterministic order, e.g., lowest key-value first. In this way, the workload would not deadlock. In actual fact, with clusters and parallel execution, the lock acquisition will not be guaranteed to be serial, so deadlocks do not entirely go away, but still may get fewer. Besides, any outside transaction might still lock in the wrong order and cause deadlocks, which is why the OLTP application must in any case be built to deal with the possibility of deadlock.</p> <p>This is the conventional relational view of the matter. In more recent times, in-memory schemes with deterministic lock acquisition (<a href="http://cs-www.cs.yale.edu/homes/dna/papers/determinism-vldb10.pdf" id="link-id0x1c5d9340">Abadi VLDB 2010</a>) or single-threaded atomic execution of transactions (<a href="http://bird.cs.tu-berlin.de:8008/birte2010/" id="link-id0x1ec0ed18">Uni Munich BIRTE workshop at VLDB2010</a>, <a href="http://www.voltdb.com/" id="link-id0x1ab6e380">VoltDB</a>) have been proposed. There the transaction is described as a stored procedure, possibly with extra annotations. These techniques might apply to RDF also. RDF is however an unlikely model for transaction-intensive applications, so we will not for now examine these further.</p> <p>RDBMS usually implement row-level locking. This means that once a column of a row has an uncommitted state, any other transaction is prevented from changing the row. This has no ready RDF equivalent. RDF is usually implemented as a row-per-triple system and applying row-level locking to this does not give the semantic one expects of a relational row. </p> <p>I would argue that it is not essential to enforce transactional guarantees in units of rows. The guarantees must apply between data that is <i>read</i> and <i>written</i> by a transaction. It does not need to apply to columns that the transaction does not reference. To take the TPC-C example, the <i>new order</i> transaction updates the stock level and the <i>delivery</i> transaction updates the delivery count on the stock table. In practice, a <i>delivery</i> and a <i>new order</i> falling on the same row of stock will lock each other out, but nothing in the semantics of the workload mandates this.</p> <p>It does not seem <i>a priori</i> necessary to recreate the row as a unit of concurrency control in RDF. One could say that a multi-attribute whole (such as an address) ought to be atomic for concurrency control, but then applications updating addresses will most likely read and update all the fields together even if only the street name changes.</p> <h2>Pessimistic Vs. Optimistic Concurrency Control </h2> <p>We have so far spoken only in terms of row-level locking, which is to my <a class="auto-href" href="http://dbpedia.org/resource/Knowledge" id="link-id0x1ebbf3f8">knowledge</a> the most widely used model in RDBMS, and one we implement ourselves. Some databases (e.g., <a class="auto-href" href="http://dbpedia.org/resource/MonetDB" id="link-id0x1e771f48">MonetDB</a> and <a class="auto-href" href="http://www.ingres.com/vectorwise/" id="link-id0x1f3b4830">VectorWise</a>) implement optimistic concurrency control. The general idea is that each transaction has a read and write set and when a transaction commits, any other transactions whose read or write set intersects with the write set of the committing transaction are marked un-committable. Once a transaction thus becomes un-committable, it may presumably continue reading indefinitely but may no longer commit its updates. Optimistic concurrency is generally coupled with multi-version semantics where the pre-image of a transaction is a clean committed state of the database as of a specific point in time, i.e., snapshot isolation. </p> <p>To implement <code>SERIALIZABLE</code> isolation, i.e., the guarantee that if a transaction twice performs a <code>COUNT</code> the result will be the same, one locks also the row that precedes the set of selected rows and marks each lock so as to prevent an insert to the right of the lock in key order. The same thing may be done in an optimistic setting.</p> <p> <a href="http://event.cwi.nl/SIGMOD-RWE/2010/22-7f15a1/paper.pdf" id="link-id0x1d5de810">Positional Handling of Updates in Column Stores</a> [Heman, Zukowski, <a class="auto-href" href="http://dbpedia.org/resource/National_Research_Institute_for_Mathematics_and_Computer_Science" id="link-id0x1e7644d8">CWI</a> science library] discusses management of multiple consecutive snapshots in some detail. The paper does not go into the details of different levels of isolation but nothing there suggests that serializability could not be supported. There is some complexity in marking the space between ordered rows as non-insertable across multiple versions but this should be feasible enough. </p> <p>The issue of optimistic Vs. pessimistic concurrency does not seem to be affected by the differences between RDF and relational models. We note that an OLTP workload can be made to run with very few transaction aborts (deadlocks) by properly ordering operations when using a locking scheme. The same does not work with optimistic concurrency since updates happen immediately and transaction aborts occur whenever the writes of one intersect the reads or writes of another, regardless of the order in which these were made.</p> <p>Developers seldom understand transactions; therefore DBMS should, within the limits of the possible, optimize locking order for locking schemes. A simple example is locking in key order when doing an operation on a set of values. A more complex variant would consist of analyzing data dependencies in stored procedures and reordering updates so as to get the highest cardinality tables first. We note that this latter trick also benefits optimistic schemes.</p> <p>In RDF, the same principles apply but distinguishing cardinality of an updated set will have to rely on statistics of predicate cardinality. Such are anyhow needed for query <a class="auto-href" href="http://dbpedia.org/resource/Program_optimization" id="link-id0x1f05c1a8">optimization</a>.</p> <h2>Eventual Consistency </h2> <p>Web scale systems that need to maintain consistent state across multiple data centers sometimes use "eventual consistency" schemes. <a class="auto-href" href="http://dbpedia.org/resource/Two-phase_commit_protocol" id="link-id0x1cebd340">Two-phase-commit</a> becomes very inefficient as latency increases, thus strict transactional semantics have prohibitive cost if the system is more distributed than a cluster with a fast interconnect.</p> <p>Eventual consistency schemes (<a href="http://dbpedia.org/page/Dynamo_(storage_system)" id="link-id0x1f9db8f8">Amazon Dynamo</a>, <a href="http://research.yahoo.com/project/212" id="link-id0x1da3db80">Yahoo! PNUTS</a>) maintain history <a class="auto-href" href="http://dbpedia.org/resource/Information" id="link-id0x1ec4dbc8">information</a> on the record which is the unit of concurrency control. The record is typically a non-first normal form chunk of related data that it makes sense to store together from the application's viewpoint. Application logic can then be applied to reconciling differing copies of the same logical record. </p> <p>Such a scheme seems <i>a priori</i> ill-suited for RDF, where the natural unit of concurrency control would seem to be the quad. We first note that only recently changed (i.e., <code>DELETEd + INSERTed</code> quads, as there is no <code>UPDATE</code>-in-place) need history information. This history information can be stored away from the quad itself, thus not disrupting compression. When detecting that one site has <code>INSERTed</code> a quad that another has <code>DELETEd</code> in the same general time period, application logic can still be applied for reading related quads in order to arrive at a decision on how to reconcile two databases that have diverged. The same can apply to conflicting values of properties that for the application should be single-valued. Comparing time-stamped transaction logs on quads is not fundamentally different from comparing record histories in Dynamo or PNUTS.</p> <p>As we overcome the data size penalties that have until recently been associated with RDF, RDF becomes even more interesting as a data model for large online systems such as social network platforms where frequent application changes lead to volatility of schema. Key value stores are currently found in such applications, but they generally do not provide the query flexibility at which RDF excels. </p> <h2>Conclusions </h2> <p>We have gone over basic aspects of the endlessly complex and variable topic of transactions, and drawn parallels as well as outlined two basic differences between relational and RDF systems: What used to be <code>REPEATABLE READ</code> becomes <code>SERIALIZABLE</code>; and row-level locking becomes locking at the level of a single attribute value. For the rest, we see that the optimistic and pessimistic modes of concurrency control, as well as guidelines for writing transaction procedures, remain much the same.</p> <p>Based on this overview, it should be possible to design an ACID test for describing the ACID behavior of benchmarked systems. We do not intend to make transaction support a qualification requirement for an RDF benchmark, but information on transaction support will still be valuable in comparing different systems.</p>
2011-03-22T18:24:26-04:00
Transaction Semantics in RDF and Relational Models
http://www.openlinksw.com:443/weblog/oerling/?date=2011-03-22#1691
2011-03-22T23:55:43Z
<p>As a part of defining benchmark audit for testing <a class="auto-href" href="http://dbpedia.org/resource/ACID" id="link-id0x1eae93c0">ACID</a> properties on <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1f0adbe0">RDF</a> stores, we will here examine different RDF scenarios where lack of concurrency control causes inconsistent results. In so doing, we consider common implementation techniques and implications as concern locking (pessimistic) and multi-version (optimistic) concurrency control schemes.</p> <p>In the following, we will talk in terms of triples, but the discussion can be trivially generalized to quads. We will use numbers for IRIs and literals. In most implementations, the internal representation for these is indeed a number (or at least some <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x1ea79630">data</a> type that has a well defined collation order). For ease of presentation, we consider a single index with key parts <code>SPO</code>. Any other index-like setting with any possible key order will have similar issues. </p> <h2>Insert (Create) and Delete </h2> <p> <code>INSERT</code> and <code>DELETE</code> as defined in <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x1f335c60">SPARQL</a> are queries which generate a result set which is then used for instantiating triple patterns. We note that a <code>DELETE</code> may delete a triple which the <code>DELETE</code> has not read; thus the delete set is not a subset of the read set. The <a class="auto-href" href="http://dbpedia.org/resource/SQL" id="link-id0x1e99b4b8">SQL</a> equivalent is the </p> <blockquote> <code><pre>DELETE FROM table WHERE key IN ( SELECT key1 FROM other_table )</pre> </code> </blockquote> <p>expression, supposing it were implemented as a scan of <code>other_table</code> and an index lookup followed by <code>DELETE</code> on table. </p> <p>The meaning of <code>INSERT</code> is that the triples in question exist after the operation, and the meaning of <code>DELETE</code> is that said triples do not exist. In a transactional context, this means that the after-image of the transaction is guaranteed either to have or not-have said triples. </p> <p>Suppose that the triples <code>{ 1 0 0 }</code>, <code>{ 1 5 6 }</code>, and <code>{ 1 5 7 }</code> exist in the beginning. If we <code>DELETE { 1 ?x ?y }</code> and concurrently <code>INSERT { 1 2 4 . 1 2 3 . 1 3 5 }</code>, then whichever was considered to be first by the concurrency control of the DBMS would complete first, and the other after that. Thus the end state would either have no triples with subject <code>1</code> or would have the three just inserted. </p> <p>Suppose the <code>INSERT</code> inserts the first triple, <code>{ 1 2 4 }</code>. The <code>DELETE</code> at the same time reads all triples with subject <code>1</code>. The exclusive read waits for the uncommitted <code>INSERT</code>. The <code>INSERT</code> then inserts the second triple, <code>{ 1 2 3 }</code>. Depending on the isolation of the read, this either succeeds, since no <code>{ 1 2 3 }</code> was read, or causes a deadlock. The first corresponds to <code>REPEATABLE READ</code> isolation; the second to <code>SERIALIZABLE</code>.</p> <p>We would not get the desired end-state of either <i>all the inserted triples</i> or <i>no triples with subject <code>1</code></i> if the read or the <code>DELETE</code> were not serializable.</p> <p>Furthermore if a <code>DELETE</code> template produced a triple that did not exist in the pre-image, the <code>DELETE</code> semantics still imply that this also does not exist in the after-image, which implies serializability.</p> <h2>Read and Update</h2> <p>Let us consider the prototypical transaction example of transferring funds from one account to another. Two balances are updated, and a history record is inserted.</p> <p>The initial state is </p> <blockquote> <code><pre>a balance 10 b balance 10</pre></code> </blockquote> <p>We transfer <code>1</code> from <code>a</code> to <code>b</code>, and at the same time transfer <code>2</code> from <code>b</code> to <code>a</code>. The end state must have <code>a</code> at <code>11</code> and <code>b</code> at <code>9</code>.</p> <p>A relational database needs <code>REPEATABLE READ</code> isolation for this.</p> <p>With RDF, <code>txn1</code> reads that <code>a</code> has a <code>balance</code> of <code>10</code>. At the same time, <code>txn1</code> reads the <code>balance</code> of <code>a</code>. <code>txn2</code> waits because the read of <code>txn1</code> is exclusive. <code>txn1</code> proceeds and read the <code>balance</code> of <code>b</code>. It then updates the <code>balance</code> of <code>a</code> and <code>b</code>. </p> <p>All goes without the deadlock which is always cited in this scenario, because the locks are acquired in the same order. The act of updating the balance of <code>a</code>, since RDF does not really have an update-in-place, consists of deleting <code>{ a balance 10 }</code> and inserting <code>{ a balance 9 }</code>. This gets done and <code>txn1</code> commits. At this point, <code>txn2</code> proceeds after its wait on the row that stated <code>{ a balance 10 }</code>. This row is now gone, and <code>txn2</code> sees that <code>a</code> has no balance, which is quite possible in RDF's <a class="auto-href" href="http://dbpedia.org/resource/Database_schema" id="link-id0x1c933cf0">schema</a>-less model.</p> <p>We see that <code>REPEATABLE READ</code> is not adequate with RDF, even though it is with relational. The reason why there is no <code>UPDATE</code>-in-place is that the <code>PRIMARY KEY</code> of the triple includes all the parts, including the object. Even in a <a class="auto-href" href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1f3cc3c8">RDBMS</a>, an <code>UPDATE</code> of a primary key part amounts to a <code>DELETE</code>-plus-<code>INSERT</code>. One could here argue that an implementation might still <code>UPDATE</code>-in-place if the key order were not changed. This would resolve the special case of the accounts but not a more general case.</p> <p>Thus we see that the read of the balance must be <code>SERIALIZABLE</code>. This means that the read locks the space before the first balance, so that no insertion may take place. In this way the read of <code>txn2</code> waits on the lock that is conceptually before the first possible match of <code>{ a balance ?x }</code>.</p> <h2>locking order and OLTP </h2> <p>To implement <a class="auto-href" href="http://www.tpc.org/" id="link-id0x1e20f2e8">TPC</a>-<a class="auto-href" href="http://dbpedia.org/resource/C%2B%2B" id="link-id0x1fa46718">C</a>, I would update the table with the highest cardinality first, and then all tables in descending order of cardinality. In this way, the locks with the highest likelihood for contention are held for the least time. If locking multiple rows of a table, these should be locked in a deterministic order, e.g., lowest key-value first. In this way, the workload would not deadlock. In actual fact, with clusters and parallel execution, the lock acquisition will not be guaranteed to be serial, so deadlocks do not entirely go away, but still may get fewer. Besides, any outside transaction might still lock in the wrong order and cause deadlocks, which is why the OLTP application must in any case be built to deal with the possibility of deadlock.</p> <p>This is the conventional relational view of the matter. In more recent times, in-memory schemes with deterministic lock acquisition (<a href="http://cs-www.cs.yale.edu/homes/dna/papers/determinism-vldb10.pdf" id="link-id0x1c5d9340">Abadi VLDB 2010</a>) or single-threaded atomic execution of transactions (<a href="http://bird.cs.tu-berlin.de:8008/birte2010/" id="link-id0x1ec0ed18">Uni Munich BIRTE workshop at VLDB2010</a>, <a href="http://www.voltdb.com/" id="link-id0x1ab6e380">VoltDB</a>) have been proposed. There the transaction is described as a stored procedure, possibly with extra annotations. These techniques might apply to RDF also. RDF is however an unlikely model for transaction-intensive applications, so we will not for now examine these further.</p> <p>RDBMS usually implement row-level locking. This means that once a column of a row has an uncommitted state, any other transaction is prevented from changing the row. This has no ready RDF equivalent. RDF is usually implemented as a row-per-triple system and applying row-level locking to this does not give the semantic one expects of a relational row. </p> <p>I would argue that it is not essential to enforce transactional guarantees in units of rows. The guarantees must apply between data that is <i>read</i> and <i>written</i> by a transaction. It does not need to apply to columns that the transaction does not reference. To take the TPC-C example, the <i>new order</i> transaction updates the stock level and the <i>delivery</i> transaction updates the delivery count on the stock table. In practice, a <i>delivery</i> and a <i>new order</i> falling on the same row of stock will lock each other out, but nothing in the semantics of the workload mandates this.</p> <p>It does not seem <i>a priori</i> necessary to recreate the row as a unit of concurrency control in RDF. One could say that a multi-attribute whole (such as an address) ought to be atomic for concurrency control, but then applications updating addresses will most likely read and update all the fields together even if only the street name changes.</p> <h2>Pessimistic Vs. Optimistic Concurrency Control </h2> <p>We have so far spoken only in terms of row-level locking, which is to my <a class="auto-href" href="http://dbpedia.org/resource/Knowledge" id="link-id0x1f1230a0">knowledge</a> the most widely used model in RDBMS, and one we implement ourselves. Some databases (e.g., <a class="auto-href" href="http://dbpedia.org/resource/MonetDB" id="link-id0x173a5538">MonetDB</a> and <a class="auto-href" href="http://www.ingres.com/vectorwise/" id="link-id0x16feb008">VectorWise</a>) implement optimistic concurrency control. The general idea is that each transaction has a read and write set and when a transaction commits, any other transactions whose read or write set intersects with the write set of the committing transaction are marked un-committable. Once a transaction thus becomes un-committable, it may presumably continue reading indefinitely but may no longer commit its updates. Optimistic concurrency is generally coupled with multi-version semantics where the pre-image of a transaction is a clean committed state of the database as of a specific point in time, i.e., snapshot isolation. </p> <p>To implement <code>SERIALIZABLE</code> isolation, i.e., the guarantee that if a transaction twice performs a <code>COUNT</code> the result will be the same, one locks also the row that precedes the set of selected rows and marks each lock so as to prevent an insert to the right of the lock in key order. The same thing may be done in an optimistic setting.</p> <p> <a href="http://event.cwi.nl/SIGMOD-RWE/2010/22-7f15a1/paper.pdf" id="link-id0x1d5de810">Positional Handling of Updates in Column Stores</a> [Heman, Zukowski, <a class="auto-href" href="http://dbpedia.org/resource/National_Research_Institute_for_Mathematics_and_Computer_Science" id="link-id0x1df9c990">CWI</a> science library] discusses management of multiple consecutive snapshots in some detail. The paper does not go into the details of different levels of isolation but nothing there suggests that serializability could not be supported. There is some complexity in marking the space between ordered rows as non-insertable across multiple versions but this should be feasible enough. </p> <p>The issue of optimistic Vs. pessimistic concurrency does not seem to be affected by the differences between RDF and relational models. We note that an OLTP workload can be made to run with very few transaction aborts (deadlocks) by properly ordering operations when using a locking scheme. The same does not work with optimistic concurrency since updates happen immediately and transaction aborts occur whenever the writes of one intersect the reads or writes of another, regardless of the order in which these were made.</p> <p>Developers seldom understand transactions; therefore DBMS should, within the limits of the possible, optimize locking order for locking schemes. A simple example is locking in key order when doing an operation on a set of values. A more complex variant would consist of analyzing data dependencies in stored procedures and reordering updates so as to get the highest cardinality tables first. We note that this latter trick also benefits optimistic schemes.</p> <p>In RDF, the same principles apply but distinguishing cardinality of an updated set will have to rely on statistics of predicate cardinality. Such are anyhow needed for query <a class="auto-href" href="http://dbpedia.org/resource/Program_optimization" id="link-id0x1f51d5d0">optimization</a>.</p> <h2>Eventual Consistency </h2> <p>Web scale systems that need to maintain consistent state across multiple data centers sometimes use "eventual consistency" schemes. <a class="auto-href" href="http://dbpedia.org/resource/Two-phase_commit_protocol" id="link-id0x1e3ba5d8">Two-phase-commit</a> becomes very inefficient as latency increases, thus strict transactional semantics have prohibitive cost if the system is more distributed than a cluster with a fast interconnect.</p> <p>Eventual consistency schemes (<a href="http://dbpedia.org/page/Dynamo_(storage_system)" id="link-id0x1f9db8f8">Amazon Dynamo</a>, <a href="http://research.yahoo.com/project/212" id="link-id0x1da3db80">Yahoo! PNUTS</a>) maintain history <a class="auto-href" href="http://dbpedia.org/resource/Information" id="link-id0x8bf48e8">information</a> on the record which is the unit of concurrency control. The record is typically a non-first normal form chunk of related data that it makes sense to store together from the application's viewpoint. Application logic can then be applied to reconciling differing copies of the same logical record. </p> <p>Such a scheme seems <i>a priori</i> ill-suited for RDF, where the natural unit of concurrency control would seem to be the quad. We first note that only recently changed (i.e., <code>DELETEd + INSERTed</code> quads, as there is no <code>UPDATE</code>-in-place) need history information. This history information can be stored away from the quad itself, thus not disrupting compression. When detecting that one site has <code>INSERTed</code> a quad that another has <code>DELETEd</code> in the same general time period, application logic can still be applied for reading related quads in order to arrive at a decision on how to reconcile two databases that have diverged. The same can apply to conflicting values of properties that for the application should be single-valued. Comparing time-stamped transaction logs on quads is not fundamentally different from comparing record histories in Dynamo or PNUTS.</p> <p>As we overcome the data size penalties that have until recently been associated with RDF, RDF becomes even more interesting as a data model for large online systems such as social network platforms where frequent application changes lead to volatility of schema. Key value stores are currently found in such applications, but they generally do not provide the query flexibility at which RDF excels. </p> <h2>Conclusions </h2> <p>We have gone over basic aspects of the endlessly complex and variable topic of transactions, and drawn parallels as well as outlined two basic differences between relational and RDF systems: What used to be <code>REPEATABLE READ</code> becomes <code>SERIALIZABLE</code>; and row-level locking becomes locking at the level of a single attribute value. For the rest, we see that the optimistic and pessimistic modes of concurrency control, as well as guidelines for writing transaction procedures, remain much the same.</p> <p>Based on this overview, it should be possible to design an ACID test for describing the ACID behavior of benchmarked systems. We do not intend to make transaction support a qualification requirement for an RDF benchmark, but information on transaction support will still be valuable in comparing different systems.</p>
2015-08-20T17:31:00.946315-04:00
RDF and Transactions
http://www.openlinksw.com:443/blog/vdb/blog/?date=2011-03-22#1690
2011-03-22T22:52:56Z
<p>I will here talk about <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x249bc940">RDF</a> and transactions for developers in general. The next one talks about specifics and is for specialists.</p> <p>Transactions are certainly not the first thing that comes to mind when one hears "RDF". We have at times used a recruitment questionnaire where we ask applicants to define a transaction. Many vaguely remember that it is a unit of work, but usually not more than that. We sometimes get questions from users about why they get an error message that says "deadlock". "Deadlock" is what happens when multiple users concurrently update balances on multiple bank accounts in the wrong order. What does this have to do with RDF?</p> <p>There are in fact users who even use XA with a <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x22c8dbc8">Virtuoso</a>-based RDF application. <a class="auto-href" href="http://semanticweb.org/id/Franz_Inc" id="link-id0x27bd0c08">Franz</a> also has publicized their development of full <a class="auto-href" href="http://dbpedia.org/resource/ACID" id="link-id0x283985c8">ACID</a> capabilities for <a class="auto-href" href="http://semanticweb.org/id/AllegroGraph" id="link-id0x238ba438">AllegroGraph</a>. RDF is a database <a class="auto-href" href="http://dbpedia.org/resource/Database_schema" id="link-id0x2864fef8">schema</a> model, and transactions will inevitably become an issue in databases.</p> <p>At the same time, the developer population trained with <a class="auto-href" href="http://dbpedia.org/resource/MySQL" id="link-id0x284d2d80">MySQL</a> and <a class="auto-href" href="http://dbpedia.org/resource/PHP" id="link-id0x237230e8">PHP</a> is not particularly transaction-aware. Transactions have gone out of style, declares the No-<a class="auto-href" href="http://dbpedia.org/resource/SQL" id="link-id0x2920cc88">SQL</a> crowd. Well, it is not so much SQL they object to but ACID, i.e., transactional guarantees. We will talk more about this in the next post. The <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x283f0588">SPARQL</a> language and protocol do not go into transactions, except for expressing the wish that an <code>UPDATE</code> request to an end-point be atomic. But beware -- atomicity is a gateway drug, and soon one finds oneself on full ACID. </p> <p>If one says that a thing will either happen <i>in its entirety</i> or <i>not at all,</i> which is what (A) atomicity means, then the question arises of (I) isolation; that is, what happens if somebody else does something to the same <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x238280f8">data</a> at the same time? Then comes the question of whether a thing, once having happened, will stay that way; i.e., (D) durability. Finally, there is (<a class="auto-href" href="http://dbpedia.org/resource/C%2B%2B" id="link-id0x276714b8">C</a>) consistency, which means that the transaction's result must not contradict restrictions the database is supposed to enforce. RDF usually has no restrictions; thus consistency mostly means that the internal state of the DBMS must be consistent, e.g., different indices on triples/quads should contain the same data.</p> <p>There are, of course, database-like consistency criteria that one can express in RDF Schema and <a class="auto-href" href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x28625a90">OWL</a>, concerning data types, mandatory presence of properties, or restrictions on cardinality (i.e., one may only have one spouse at a time, and the like). </p> <p>If one indeed did enforce them all, then RDF would be very like the relational model -- with all the restrictions, but without the 40 years of work on <a class="auto-href" href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x249bf4f8">RDBMS</a> performance. For this reason, RDF use tends to involve data that is not structured enough to be a good fit for RDBMS.</p> <p>There is of course the OWL side, where consistency is important but is defined in such complex ways that they again are not a good fit for RDBMS. RDF could be seen to be split between the schema-last world and the <a class="auto-href" href="http://dbpedia.org/resource/Knowledge" id="link-id0x249504f8">knowledge</a> representation world. I will here focus on the schema-last side.</p> <p>Transactions are relevant in RDF in two cases: 1. If data is trickle loaded in small chunks, one likes to know that the chunks do not get lost or corrupted; 2. If the application has any semantics that reserve resources, then these operations need transactions. The latter is not so common with RDF but examples include read-write situations, like checking if a seat is available and then reserving it. Transactionality guarantees that the same seat does not get reserved twice.</p> <p>Web people argue with some justification that since the four cardinal virtues of database never existed on the web to begin with, applying strict ACID to web data is beside the point, like locking the stable after the horse has long since run away. This may be so; yet the systems used for processing data, whether that data is dirty or not, benefit from predictable operation under concurrency and from not losing data.</p> <p>Analytics workloads are not primarily about transactions, but still need to specify what happens with updates. Analyzing data from measurements may not have concurrent updates, but there the transaction issue is replaced by the question of making explicit how the data was acquired and what processing has been applied to it before storage.</p> <p>As mentioned before, the <a class="auto-href" href="http://lod2.eu/" id="link-id0x27d952d0">LOD2</a> project is at the crossroads of RDF and database. I construe its mission to be the making of RDF into a respectable database discipline. Database respectability in turn is as good as inconceivable without addressing the very bedrock on which this science was founded: transactions.</p> <p>As previously argued, we need well-defined and auditable benchmarks. This again brings up the topic of transactions. Once we embark on the database benchmark route, there is no way around this. <a class="auto-href" href="http://www.tpc.org/" id="link-id0x2359d2d0">TPC</a>-<a class="auto-href" href="http://dbpedia.org/resource/TPC-H" id="link-id0x28edb770">H</a> mandates that the system under test support transactions, and the audit involves a test for this. We can do no less.</p> <p>This has led me to more closely examine the issue of RDF and transactions, and whether there exist differences between transactions applied to RDF and to relational data. </p> <p>As concerns Virtuoso, our position has been that one can get full ACID in Virtuoso, whether in SQL or SPARQL, by using a connected client (e.g., <a class="auto-href" href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x23a55698">ODBC</a>, <a class="auto-href" href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x235cecf0">JDBC</a>, or the <a class="auto-href" href="http://jena.sourceforge.net/" id="link-id0x23213900">Jena</a> or <a class="auto-href" href="http://sourceforge.net/projects/sesame/" id="link-id0x277874d0">Sesame</a> frameworks), and setting the isolation options on the connection. Having taken this step, one then must take the next step, which consists of dealing with deadlocks; i.e., with concurrent utilization, it may happen that the database at any time notifies the client that the transaction got aborted and the client must retry.</p> <p>Web developers especially do not like this, because this is not what MySQL has taught them to expect. MySQL does have transactional back-ends like InnoDB, but often gets used without transactions.</p> <p>With the March 2011 Virtuoso releases, we have taken a closer look at transactions with RDF. It is more practical to reduce the possibility of errors than to require developers to pay attention. For this reason we have automated isolation settings for RDF, greatly reduced the incidence of deadlocks, and even incorporated automatic deadlock retries where applicable.</p> <p>If all users lock resources they need in the same order, there will be no deadlocks. This is what we do with RDF load in Virtuoso 7; thus any mix of concurrent <code>INSERTs</code> and <code>DELETEs</code>, if these are under a certain size (normally 10000 quads) are guaranteed never to fail due to locking. These could still fail due to running out of space, though. With previous versions, there always was a possibility of having an <code>INSERT</code> or <code>DELETE</code> fail because of deadlock with multiple users. Vectored <code>INSERT</code> and <code>DELETE</code> are sufficient for making web crawling or archive maintenance practically deadlock free, since there the primary transaction is the <code>INSERT</code> or <code>DELETE</code> of a small graph. </p> <p>Furthermore, since the <a class="auto-href" href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id0x23eadf50">SPARQL protocol</a> has no way of specifying transactions consisting of multiple client-server exchanges, the SPARQL end-point may deal with deadlocks by itself. If all else fails, it can simply execute requests one after the other, thus eliminating any possibility of locking. We note that many statements will be intrinsically free of deadlocks by virtue of always locking in key order, but this cannot be universally guaranteed with arbitrary size operations; thus concurrent operations might still sometimes deadlock. Anyway, vectored execution as introduced in Virtuoso 7, besides getting easily double-speed random access, also greatly reduces deadlocks by virtue of ordering operations.</p> <p>In the next post we will talk about what transactions mean with RDF and whether there is any difference with the relational model.</p>
2011-03-22T17:44:21-04:00
RDF and Transactions
http://www.openlinksw.com:443/weblog/oerling/?date=2011-03-22#1689
2011-03-22T22:52:56Z
<p>I will here talk about <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x235282b8">RDF</a> and transactions for developers in general. The next one talks about specifics and is for specialists.</p> <p>Transactions are certainly not the first thing that comes to mind when one hears "RDF". We have at times used a recruitment questionnaire where we ask applicants to define a transaction. Many vaguely remember that it is a unit of work, but usually not more than that. We sometimes get questions from users about why they get an error message that says "deadlock". "Deadlock" is what happens when multiple users concurrently update balances on multiple bank accounts in the wrong order. What does this have to do with RDF?</p> <p>There are in fact users who even use XA with a <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x235e5938">Virtuoso</a>-based RDF application. <a class="auto-href" href="http://semanticweb.org/id/Franz_Inc" id="link-id0x28c09308">Franz</a> also has publicized their development of full <a class="auto-href" href="http://dbpedia.org/resource/ACID" id="link-id0x2365f710">ACID</a> capabilities for <a class="auto-href" href="http://semanticweb.org/id/AllegroGraph" id="link-id0x22caecb0">AllegroGraph</a>. RDF is a database <a class="auto-href" href="http://dbpedia.org/resource/Database_schema" id="link-id0x235f1f70">schema</a> model, and transactions will inevitably become an issue in databases.</p> <p>At the same time, the developer population trained with <a class="auto-href" href="http://dbpedia.org/resource/MySQL" id="link-id0x240f6a90">MySQL</a> and <a class="auto-href" href="http://dbpedia.org/resource/PHP" id="link-id0x238cd088">PHP</a> is not particularly transaction-aware. Transactions have gone out of style, declares the No-<a class="auto-href" href="http://dbpedia.org/resource/SQL" id="link-id0x232d9068">SQL</a> crowd. Well, it is not so much SQL they object to but ACID, i.e., transactional guarantees. We will talk more about this in the next post. The <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x238c70a0">SPARQL</a> language and protocol do not go into transactions, except for expressing the wish that an <code>UPDATE</code> request to an end-point be atomic. But beware -- atomicity is a gateway drug, and soon one finds oneself on full ACID. </p> <p>If one says that a thing will either happen <i>in its entirety</i> or <i>not at all,</i> which is what (A) atomicity means, then the question arises of (I) isolation; that is, what happens if somebody else does something to the same <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x23eadf50">data</a> at the same time? Then comes the question of whether a thing, once having happened, will stay that way; i.e., (D) durability. Finally, there is (<a class="auto-href" href="http://dbpedia.org/resource/C%2B%2B" id="link-id0x23a1e280">C</a>) consistency, which means that the transaction's result must not contradict restrictions the database is supposed to enforce. RDF usually has no restrictions; thus consistency mostly means that the internal state of the DBMS must be consistent, e.g., different indices on triples/quads should contain the same data.</p> <p>There are, of course, database-like consistency criteria that one can express in RDF Schema and <a class="auto-href" href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x287b18e8">OWL</a>, concerning data types, mandatory presence of properties, or restrictions on cardinality (i.e., one may only have one spouse at a time, and the like). </p> <p>If one indeed did enforce them all, then RDF would be very like the relational model -- with all the restrictions, but without the 40 years of work on <a class="auto-href" href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x2450b488">RDBMS</a> performance. For this reason, RDF use tends to involve data that is not structured enough to be a good fit for RDBMS.</p> <p>There is of course the OWL side, where consistency is important but is defined in such complex ways that they again are not a good fit for RDBMS. RDF could be seen to be split between the schema-last world and the <a class="auto-href" href="http://dbpedia.org/resource/Knowledge" id="link-id0x2324ac40">knowledge</a> representation world. I will here focus on the schema-last side.</p> <p>Transactions are relevant in RDF in two cases: 1. If data is trickle loaded in small chunks, one likes to know that the chunks do not get lost or corrupted; 2. If the application has any semantics that reserve resources, then these operations need transactions. The latter is not so common with RDF but examples include read-write situations, like checking if a seat is available and then reserving it. Transactionality guarantees that the same seat does not get reserved twice.</p> <p>Web people argue with some justification that since the four cardinal virtues of database never existed on the web to begin with, applying strict ACID to web data is beside the point, like locking the stable after the horse has long since run away. This may be so; yet the systems used for processing data, whether that data is dirty or not, benefit from predictable operation under concurrency and from not losing data.</p> <p>Analytics workloads are not primarily about transactions, but still need to specify what happens with updates. Analyzing data from measurements may not have concurrent updates, but there the transaction issue is replaced by the question of making explicit how the data was acquired and what processing has been applied to it before storage.</p> <p>As mentioned before, the <a class="auto-href" href="http://lod2.eu/" id="link-id0x28ac0250">LOD2</a> project is at the crossroads of RDF and database. I construe its mission to be the making of RDF into a respectable database discipline. Database respectability in turn is as good as inconceivable without addressing the very bedrock on which this science was founded: transactions.</p> <p>As previously argued, we need well-defined and auditable benchmarks. This again brings up the topic of transactions. Once we embark on the database benchmark route, there is no way around this. <a class="auto-href" href="http://www.tpc.org/" id="link-id0x284d2d80">TPC</a>-<a class="auto-href" href="http://dbpedia.org/resource/TPC-H" id="link-id0x280dcd40">H</a> mandates that the system under test support transactions, and the audit involves a test for this. We can do no less.</p> <p>This has led me to more closely examine the issue of RDF and transactions, and whether there exist differences between transactions applied to RDF and to relational data. </p> <p>As concerns Virtuoso, our position has been that one can get full ACID in Virtuoso, whether in SQL or SPARQL, by using a connected client (e.g., <a class="auto-href" href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x235cecf0">ODBC</a>, <a class="auto-href" href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x27c4a0c0">JDBC</a>, or the <a class="auto-href" href="http://jena.sourceforge.net/" id="link-id0x283a89a8">Jena</a> or <a class="auto-href" href="http://sourceforge.net/projects/sesame/" id="link-id0x284b3490">Sesame</a> frameworks), and setting the isolation options on the connection. Having taken this step, one then must take the next step, which consists of dealing with deadlocks; i.e., with concurrent utilization, it may happen that the database at any time notifies the client that the transaction got aborted and the client must retry.</p> <p>Web developers especially do not like this, because this is not what MySQL has taught them to expect. MySQL does have transactional back-ends like InnoDB, but often gets used without transactions.</p> <p>With the March 2011 Virtuoso releases, we have taken a closer look at transactions with RDF. It is more practical to reduce the possibility of errors than to require developers to pay attention. For this reason we have automated isolation settings for RDF, greatly reduced the incidence of deadlocks, and even incorporated automatic deadlock retries where applicable.</p> <p>If all users lock resources they need in the same order, there will be no deadlocks. This is what we do with RDF load in Virtuoso 7; thus any mix of concurrent <code>INSERTs</code> and <code>DELETEs</code>, if these are under a certain size (normally 10000 quads) are guaranteed never to fail due to locking. These could still fail due to running out of space, though. With previous versions, there always was a possibility of having an <code>INSERT</code> or <code>DELETE</code> fail because of deadlock with multiple users. Vectored <code>INSERT</code> and <code>DELETE</code> are sufficient for making web crawling or archive maintenance practically deadlock free, since there the primary transaction is the <code>INSERT</code> or <code>DELETE</code> of a small graph. </p> <p>Furthermore, since the <a class="auto-href" href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id0x22ca4300">SPARQL protocol</a> has no way of specifying transactions consisting of multiple client-server exchanges, the SPARQL end-point may deal with deadlocks by itself. If all else fails, it can simply execute requests one after the other, thus eliminating any possibility of locking. We note that many statements will be intrinsically free of deadlocks by virtue of always locking in key order, but this cannot be universally guaranteed with arbitrary size operations; thus concurrent operations might still sometimes deadlock. Anyway, vectored execution as introduced in Virtuoso 7, besides getting easily double-speed random access, also greatly reduces deadlocks by virtue of ordering operations.</p> <p>In the next post we will talk about what transactions mean with RDF and whether there is any difference with the relational model.</p>
2011-03-22T17:44:14-04:00
Benchmarks, Redux (part 15): BSBM Test Driver Enhancements
http://www.openlinksw.com:443/blog/vdb/blog/?date=2011-03-22#1688
2011-03-22T22:32:28Z
<p>This article covers the changes we have made to the <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x2361bf18">BSBM</a> test driver during our series of experiments.</p> <ul> <li> <p> <b>Drill-down mode</b> - For queries that have a product type as parameter, the test driver will invoke the query multiple times with each time a random subtype of the product type of the previous invocation. The starting point of the drill-down is an a random type from a settable level in the hierarchy. The rationale for the drill-down mode is that depending on the parameter choice, there can be 1000x differences in query run time. Thus run times of consecutive query mixes will be incomparable unless we guarantee that each mix has a predictable number of queries with a product type from each level in the hierarchy.</p> </li> <li> <b>Permutation of query mix</b> - In the BI workload, the queries are run in a random order on each thread in multiuser mode. Doing exactly the same thing on many threads is not realistic for large queries. The <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x2834cec8">data</a> access patterns must be spread out in order to evaluate how bulk IO is organized with differing concurrent demands. The permutations are deterministic on consecutive runs and do not depend on the non-deterministic timing of concurrent activities. For queries with a drill-down, the individual executions that make up the drill-down are still consecutive.</li> <li> <p> <b>New metrics</b> - The BI Power is the geometric mean of query run times scaled to queries per hour and multiplied by the scale factor, where 100 Mt is considered the unit scale. The BI Throughput is the arithmetic mean of the run times scaled to QPH and adjusted to scale as with the Power metric. These are analogous to the <a class="auto-href" href="http://www.tpc.org/" id="link-id0x236c5158">TPC</a>-<a class="auto-href" href="http://dbpedia.org/resource/TPC-H" id="link-id0x28814950">H</a> Power and Throughput metrics. </p> <p>The <i>Power</i> is defined as</p> <blockquote>(scale_factor / 284826) * 3600 / ((t0 * t1 * ... * tn) ^(1 / n)) </blockquote> <p>The <i>Throughput</i> is defined as</p> <blockquote>(scale_factor / 284826) * 3600 / ((t0 + t2 + ... + tn) / n)</blockquote> <p>The magic number 284826 is the scale that generates approximately 100 million triples (100 Mt). We consider this "scale one." The reason for the multiplication is that scores at different scales should get similar numbers, otherwise 10x larger scale would result roughly in 10x lower throughput with the BI queries.</p> <p>We also show the percentage each query represents from the total time the test driver waits for responses. </p> </li> <li> <p> <b>Deadlock retry</b> - When running update mixes, it is possible that a transaction gets aborted by a deadlock. We have made a retry logic for this.</p> </li> <li> <p> <b>Cluster mode</b> - Cluster databases may have multiple interchangeable <a class="auto-href" href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x240f9008">HTTP</a> listeners. With this mode, one can specify multiple end-points so a multi-user workload can divide itself evenly over these.</p> </li> <li> <p> <b>Identifying matter</b> - A version number was added to test driver output. Use of the new switches is also indicated in the test driver output.</p> </li> <li> <p> <b>SUT <a class="auto-href" href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x249b7208">CPU</a></b> - In comparing results it is crucial to differentiate between in memory runs and IO bound runs. To make this easier, we have added an option to report server CPU times over the timed portion (excluding warm-ups). A pluggable self-script determines the CPU times for the system; thus clusters can be handled, too. The time is given as a sum of the time the server processes have aged during the run and as a percentage over the wall-clock time.</p> </li> </ul> <p>These changes will soon be available <a href="http://blogs.usnet.private:8893/RPC2" id="link-id0x1f9a57c0">as a diff</a> and <a href="http://blogs.usnet.private:8893/RPC2" id="link-id0x1f2fea08">as a source tree</a>. This version is labeled <b><code>BSBM Test Driver 1.1-opl</code></b>; the <b><code>-opl</code></b> signifies OpenLink additions. </p> <p>We invite FU Berlin to include these enhancements into their Source Forge repository of the BSBM test driver. There is more precise documentation of these options in the README file in the above distribution.</p> <p>The next planned upgrade of the test driver concerns adding support for "<a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x2865ac68">RDF</a>-H", the RDF adaptation of the industry standard TPC-H decision support benchmark for <a class="auto-href" href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x23597bb0">RDBMS</a>.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1db2be00">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1dfcc038">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x197c26d0">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x1d149cf0">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1ab69450">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1e67d688">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1dad87c8">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1cc73830">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1d6879a8">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x1dfae510">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1ef052a0">Benchmarks, Redux (part 11): The Substance of Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1dadddb0">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1e662ef0">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1df6fa70">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> Benchmarks, Redux (part 15): BSBM Test Driver Enhancements <i>(this post)</i> </li> </ul>
2011-03-22T17:04:43-04:00
Benchmarks, Redux (part 15): BSBM Test Driver Enhancements
http://www.openlinksw.com:443/weblog/oerling/?date=2011-03-22#1684
2011-03-22T22:32:28Z
<p>This article covers the changes we have made to the <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x283a2528">BSBM</a> test driver during our series of experiments.</p> <ul> <li> <p> <b>Drill-down mode</b> - For queries that have a product type as parameter, the test driver will invoke the query multiple times with each time a random subtype of the product type of the previous invocation. The starting point of the drill-down is an a random type from a settable level in the hierarchy. The rationale for the drill-down mode is that depending on the parameter choice, there can be 1000x differences in query run time. Thus run times of consecutive query mixes will be incomparable unless we guarantee that each mix has a predictable number of queries with a product type from each level in the hierarchy.</p> </li> <li> <b>Permutation of query mix</b> - In the BI workload, the queries are run in a random order on each thread in multiuser mode. Doing exactly the same thing on many threads is not realistic for large queries. The <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x23880860">data</a> access patterns must be spread out in order to evaluate how bulk IO is organized with differing concurrent demands. The permutations are deterministic on consecutive runs and do not depend on the non-deterministic timing of concurrent activities. For queries with a drill-down, the individual executions that make up the drill-down are still consecutive.</li> <li> <p> <b>New metrics</b> - The BI Power is the geometric mean of query run times scaled to queries per hour and multiplied by the scale factor, where 100 Mt is considered the unit scale. The BI Throughput is the arithmetic mean of the run times scaled to QPH and adjusted to scale as with the Power metric. These are analogous to the <a class="auto-href" href="http://www.tpc.org/" id="link-id0x28ccd3f8">TPC</a>-<a class="auto-href" href="http://dbpedia.org/resource/TPC-H" id="link-id0x29ad25c8">H</a> Power and Throughput metrics. </p> <p>The <i>Power</i> is defined as</p> <blockquote>(scale_factor / 284826) * 3600 / ((t0 * t1 * ... * tn) ^(1 / n)) </blockquote> <p>The <i>Throughput</i> is defined as</p> <blockquote>(scale_factor / 284826) * 3600 / ((t0 + t2 + ... + tn) / n)</blockquote> <p>The magic number 284826 is the scale that generates approximately 100 million triples (100 Mt). We consider this "scale one." The reason for the multiplication is that scores at different scales should get similar numbers, otherwise 10x larger scale would result roughly in 10x lower throughput with the BI queries.</p> <p>We also show the percentage each query represents from the total time the test driver waits for responses. </p> </li> <li> <p> <b>Deadlock retry</b> - When running update mixes, it is possible that a transaction gets aborted by a deadlock. We have made a retry logic for this.</p> </li> <li> <p> <b>Cluster mode</b> - Cluster databases may have multiple interchangeable <a class="auto-href" href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x236532c8">HTTP</a> listeners. With this mode, one can specify multiple end-points so a multi-user workload can divide itself evenly over these.</p> </li> <li> <p> <b>Identifying matter</b> - A version number was added to test driver output. Use of the new switches is also indicated in the test driver output.</p> </li> <li> <p> <b>SUT <a class="auto-href" href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x249c8f68">CPU</a></b> - In comparing results it is crucial to differentiate between in memory runs and IO bound runs. To make this easier, we have added an option to report server CPU times over the timed portion (excluding warm-ups). A pluggable self-script determines the CPU times for the system; thus clusters can be handled, too. The time is given as a sum of the time the server processes have aged during the run and as a percentage over the wall-clock time.</p> </li> </ul> <p>These changes will soon be available <a href="http://blogs.usnet.private:8893/RPC2" id="link-id0x1f9a57c0">as a diff</a> and <a href="http://blogs.usnet.private:8893/RPC2" id="link-id0x1f2fea08">as a source tree</a>. This version is labeled <b><code>BSBM Test Driver 1.1-opl</code></b>; the <b><code>-opl</code></b> signifies OpenLink additions. </p> <p>We invite FU Berlin to include these enhancements into their Source Forge repository of the BSBM test driver. There is more precise documentation of these options in the README file in the above distribution.</p> <p>The next planned upgrade of the test driver concerns adding support for "<a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x23de9eb8">RDF</a>-H", the RDF adaptation of the industry standard TPC-H decision support benchmark for <a class="auto-href" href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x22cca4e0">RDBMS</a>.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1db2be00">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1dfcc038">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x197c26d0">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x1d149cf0">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1ab69450">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1e67d688">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1dad87c8">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1cc73830">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1d6879a8">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x1dfae510">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1ef052a0">Benchmarks, Redux (part 11): The Substance of Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1681" id="link-id0x2aac14871988">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1682" id="link-id0x2aac0117d1a8">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1683" id="link-id0x2aac14dbfd58">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> Benchmarks, Redux (part 15): BSBM Test Driver Enhancements <i>(this post)</i> </li> </ul>
2014-08-21T10:42:54.780224-04:00
Benchmarks, Redux (part 14): BSBM BI Mix
http://www.openlinksw.com:443/blog/vdb/blog/?date=2011-03-22#1687
2011-03-22T22:31:32Z
<p>In this post, we look at how we run the <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x236dcda8">BSBM</a>-BI mix. We consider the 100 Mt and 1000 Mt scales with <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x284893c0">Virtuoso</a> 7 using the same hardware and software as in the previous posts. The changes to workload and metric are given in the previous post.</p> <p>Our intent here is to look at whether the metric works, and to see what results will look like in general. We are as much testing the benchmark as we are testing the system-under-test (SUT). The results shown here will likely not be comparable with future ones because we will most likely change the composition of the workload since it seems a bit out of balance. Anyway, for the sake of disclosure, we attach the query templates. The test driver we used will be made available soon, so the interested may still try a comparison with their systems. If you practice with this workload for the coming races, the effort will surely not be wasted.</p> <p>Once we have come up with a rules document, we will redo all that we have published so far by-the-book, and have it audited as part of the <a class="auto-href" href="http://lod2.eu/" id="link-id0x23724860">LOD2</a> service we plan for this (see previous posts in this series). This will introduce comparability; but before we get that far with the BI workload, the workload needs to evolve a bit.</p> <p>Below we show samples of test driver output; the whole output is <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/BenchmarksReduxSupportingFiles/br.tar.gz" id="link-id0x1b703ad8">downloadable</a>.</p> <p>100 Mt Single User</p> <blockquote> <code><pre> bsbm/testdriver -runs 1 -w 0 -idir /bs/1 -drill \ -ucf bsbm/usecases/businessIntelligence/<a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x2385eb48">sparql</a>.txt \ -dg <a class="auto-href" href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x22e2f508">http</a>://bsbm.org http://localhost:8604/sparql </pre> </code> </blockquote> <blockquote> <code><pre> 0: 43348.14ms, total: 43440ms Scale factor: 284826 Explore Endpoints: 1 Update Endpoints: 1 Drilldown: on Number of warmup runs: 0 Seed: 808080 Number of query mix runs (without warmups): 1 times min/max Querymix runtime: 43.3481s / 43.3481s Elapsed runtime: 43.348 seconds QMpH: 83.049 query mixes per hour CQET: 43.348 seconds average runtime of query mix CQET (geom.): 43.348 seconds geometric mean runtime of query mix AQET (geom.): 0.492 seconds geometric mean runtime of query Throughput: 1494.874 BSBM-BI throughput: qph*scale BI Power: 7309.820 BSBM-BI Power: qph*scale (geom) </pre> </code> </blockquote> <p>100 Mt 8 User </p> <blockquote> <code><pre> Thread 6: query mix 3: 195793.09ms, total: 196086.18ms Thread 8: query mix 0: 197843.84ms, total: 198010.50ms Thread 7: query mix 4: 201806.28ms, total: 201996.26ms Thread 2: query mix 5: 221983.93ms, total: 222105.96ms Thread 4: query mix 7: 225127.55ms, total: 225317.49ms Thread 3: query mix 6: 225860.49ms, total: 226050.17ms Thread 5: query mix 2: 230884.93ms, total: 231067.61ms Thread 1: query mix 1: 237836.61ms, total: 237959.11ms Benchmark run completed in 237.985427s Scale factor: 284826 Explore Endpoints: 1 Update Endpoints: 1 Drilldown: on Number of warmup runs: 0 Number of clients: 8 Seed: 808080 Number of query mix runs (without warmups): 8 times min/max Querymix runtime: 195.7931s / 237.8366s Total runtime (sum): 1737.137 seconds Elapsed runtime: 1737.137 seconds QMpH: 121.016 query mixes per hour CQET: 217.142 seconds average runtime of query mix CQET (geom.): 216.603 seconds geometric mean runtime of query mix AQET (geom.): 2.156 seconds geometric mean runtime of query Throughput: 2178.285 BSBM-BI throughput: qph*scale BI Power: 1669.745 BSBM-BI Power: qph*scale (geom) </pre> </code> </blockquote> <p>1000 Mt Single User</p> <blockquote> <code><pre> 0: 608707.03ms, total: 608768ms Scale factor: 2848260 Explore Endpoints: 1 Update Endpoints: 1 Drilldown: on Number of warmup runs: 0 Seed: 808080 Number of query mix runs (without warmups): 1 times min/max Querymix runtime: 608.7070s / 608.7070s Elapsed runtime: 608.707 seconds QMpH: 5.914 query mixes per hour CQET: 608.707 seconds average runtime of query mix CQET (geom.): 608.707 seconds geometric mean runtime of query mix AQET (geom.): 5.167 seconds geometric mean runtime of query Throughput: 1064.552 BSBM-BI throughput: qph*scale BI Power: 6967.325 BSBM-BI Power: qph*scale (geom) </pre> </code> </blockquote> <p>1000 Mt 8 User </p> <blockquote> <code><pre> bsbm/testdriver -runs 8 -mt 8 -w 0 -idir /bs/10 -drill \ -ucf bsbm/usecases/businessIntelligence/sparql.txt \ -dg http://bsbm.org http://localhost:8604/sparql </pre> </code> </blockquote> <blockquote> <code><pre> Thread 3: query mix 4: 2211275.25ms, total: 2211371.60ms Thread 4: query mix 0: 2212316.87ms, total: 2212417.99ms Thread 8: query mix 3: 2275942.63ms, total: 2276058.03ms Thread 5: query mix 5: 2441378.35ms, total: 2441448.66ms Thread 6: query mix 7: 2804001.05ms, total: 2804098.81ms Thread 2: query mix 2: 2808374.66ms, total: 2808473.71ms Thread 1: query mix 6: 2839407.12ms, total: 2839510.63ms Thread 7: query mix 1: 2889199.23ms, total: 2889263.17ms Benchmark run completed in 2889.302566s Scale factor: 2848260 Explore Endpoints: 1 Update Endpoints: 1 Drilldown: on Number of warmup runs: 0 Number of clients: 8 Seed: 808080 Number of query mix runs (without warmups): 8 times min/max Querymix runtime: 2211.2753s / 2889.1992s Total runtime (sum): 20481.895 seconds Elapsed runtime: 20481.895 seconds QMpH: 9.968 query mixes per hour CQET: 2560.237 seconds average runtime of query mix CQET (geom.): 2544.284 seconds geometric mean runtime of query mix AQET (geom.): 13.556 seconds geometric mean runtime of query Throughput: 1794.205 BSBM-BI throughput: qph*scale BI Power: 2655.678 BSBM-BI Power: qph*scale (geom) Metrics for Query: 1 Count: 8 times executed in whole run Time share 2.120884% of total execution time AQET: 54.299656 seconds (arithmetic mean) AQET(geom.): 34.607302 seconds (geometric mean) QPS: 0.13 Queries per second minQET/maxQET: 11.71547600s / 148.65379700s Metrics for Query: 2 Count: 8 times executed in whole run Time share 0.207382% of total execution time AQET: 5.309462 seconds (arithmetic mean) AQET(geom.): 2.737696 seconds (geometric mean) QPS: 1.34 Queries per second minQET/maxQET: 0.78729800s / 25.80948200s Metrics for Query: 3 Count: 8 times executed in whole run Time share 17.650472% of total execution time AQET: 451.893890 seconds (arithmetic mean) AQET(geom.): 410.481088 seconds (geometric mean) QPS: 0.02 Queries per second minQET/maxQET: 171.07262500s / 721.72939200s Metrics for Query: 5 Count: 32 times executed in whole run Time share 6.196565% of total execution time AQET: 39.661685 seconds (arithmetic mean) AQET(geom.): 6.849882 seconds (geometric mean) QPS: 0.18 Queries per second minQET/maxQET: 0.15696500s / 189.00906200s Metrics for Query: 6 Count: 8 times executed in whole run Time share 0.119916% of total execution time AQET: 3.070136 seconds (arithmetic mean) AQET(geom.): 2.056059 seconds (geometric mean) QPS: 2.31 Queries per second minQET/maxQET: 0.41524400s / 7.55655300s Metrics for Query: 7 Count: 40 times executed in whole run Time share 1.577963% of total execution time AQET: 8.079921 seconds (arithmetic mean) AQET(geom.): 1.342079 seconds (geometric mean) QPS: 0.88 Queries per second minQET/maxQET: 0.02205800s / 40.27761500s Metrics for Query: 8 Count: 40 times executed in whole run Time share 72.126818% of total execution time AQET: 369.323481 seconds (arithmetic mean) AQET(geom.): 114.431863 seconds (geometric mean) QPS: 0.02 Queries per second minQET/maxQET: 5.94377300s / 1824.57867400s </pre> </code> </blockquote> <p>The <a class="auto-href" href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x2809d998">CPU</a> for the multiuser runs stays above 1500% for the whole run. The CPU for the single user 100 Mt run is 630%; for the 1000 Mt run, this is 574%. This can be improved since the queries usually have a lot of <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x22cf75b8">data</a> to work on. But final <a class="auto-href" href="http://dbpedia.org/resource/Program_optimization" id="link-id0x238b94c8">optimization</a> is not our goal yet; we are just surveying the race track. The difference between a warm single user run and a cold single user run is about 15% with data on SSD; with data on disk, this would be more. The numbers shown are with warm <a class="auto-href" href="http://dbpedia.org/resource/Cache" id="link-id0x23ad8c08">cache</a>. The single-user and multi-user Throughput difference, 1064 single-user vs. 1794 multi-user, is about what one would expect from the CPU utilization.</p> <p>With these numbers, the CPU does not appear badly memory-bound, else the increase would be less; also core multi-threading seems to bring some benefit. If the single-user run was at 800%, the Throughput would be 1488. The speed in excess of this may be attributed to core multi-threading, although we must remember that not every query mix is exactly the same length, so the figure is not exact. Core multi-threading does not seem to hurt, at the very least. Comparison of the same numbers with the column store will be interesting since it misses the cache a lot less and accordingly has better SMP scaling. The <a class="auto-href" href="http://dbpedia.org/resource/Intel_Corporation" id="link-id0x23568308">Intel</a> Nehalem memory subsystem is really pretty good.</p> <p> </p> <p>For reference, we show a run with Virtuoso 6 at 100Mt. </p> <blockquote> <code><pre> 0: 424754.40ms, total: 424829ms Scale factor: 284826 Explore Endpoints: 1 Update Endpoints: 1 Drilldown: on Number of warmup runs: 0 Seed: 808080 Number of query mix runs (without warmups): 1 times min/max Querymix runtime: 424.7544s / 424.7544s Elapsed runtime: 424.754 seconds QMpH: 8.475 query mixes per hour CQET: 424.754 seconds average runtime of query mix CQET (geom.): 424.754 seconds geometric mean runtime of query mix AQET (geom.): 1.097 seconds geometric mean runtime of query Throughput: 152.559 BSBM-BI throughput: qph*scale BI Power: 3281.150 BSBM-BI Power: qph*scale (geom) </pre> </code> </blockquote> <p>and 8 user </p> <blockquote> <code><pre> Thread 5: query mix 3: 616997.86ms, total: 617042.83ms Thread 7: query mix 4: 625522.18ms, total: 625559.09ms Thread 3: query mix 7: 626247.62ms, total: 626304.96ms Thread 1: query mix 0: 629675.17ms, total: 629724.98ms Thread 4: query mix 6: 667633.36ms, total: 667670.07ms Thread 8: query mix 2: 674206.07ms, total: 674256.72ms Thread 6: query mix 5: 695020.21ms, total: 695052.29ms Thread 2: query mix 1: 701824.67ms, total: 701864.91ms Benchmark run completed in 701.909341s Scale factor: 284826 Explore Endpoints: 1 Update Endpoints: 1 Drilldown: on Number of warmup runs: 0 Number of clients: 8 Seed: 808080 Number of query mix runs (without warmups): 8 times min/max Querymix runtime: 616.9979s / 701.8247s Total runtime (sum): 5237.127 seconds Elapsed runtime: 5237.127 seconds QMpH: 41.031 query mixes per hour CQET: 654.641 seconds average runtime of query mix CQET (geom.): 653.873 seconds geometric mean runtime of query mix AQET (geom.): 2.557 seconds geometric mean runtime of query Throughput: 738.557 BSBM-BI throughput: qph*scale BI Power: 1408.133 BSBM-BI Power: qph*scale (geom) </pre> </code> </blockquote> <p>Having the numbers, let us look at the metric and its scaling. We take the geometric mean of the single-user Power and the multiuser Throughput.</p> <blockquote> <code><pre> 100 Mt: sqrt ( 7771 * 2178 ); = 4114 1000 Mt: sqrt ( 6967 * 1794 ); = 3535 </pre> </code> </blockquote> <p>Scaling seems to work; the results are in the same general ballpark. The real times for the 1000 Mt run are a bit over 10x the times for the 100Mt run, as expected. The relative percentages of the queries are about the same on both scales, with the drill-down in Q8 alone being 77% and 72% respectively. The Q8 drill-down starts at the root of the product hierarchy. If we made this start one level from the top, its share would drop. This seems reasonable.</p> <p>Conversely, Q2 is out of place, with far too little share of the time. It takes a product as a starting point and shows a list of products with common features, sorted by descending count of common features. This would more appropriately be applied to a leaf product category instead, measuring how many of the products in the category have the top 20 features found in this category, to name an example.</p> <p>Also there should be more queries.</p> <p>At present it appears that BSBM-BI is definitely runnable, but a cursory look suffices to show that the workload needs more development and variety. We remember that I dreamt up the business questions last fall without much analysis, and that these questions were subsequently translated to SPARQL by FU Berlin. So, on one hand, BSBM-BI is of crucial importance because it is the first attempt at doing a benchmark with long running queries in SPARQL. On the other hand, BSBM-BI is not very good as a benchmark; <a class="auto-href" href="http://www.tpc.org/" id="link-id0x23872a10">TPC</a>-<a class="auto-href" href="http://dbpedia.org/resource/TPC-H" id="link-id0x28487d98">H</a> is a lot better. This stands to reason, as TPC-H has had years and years of development and participation by many people.</p> <p>Benchmark queries are trick questions: For example, TPC-H Q18 cannot be done without changing an <code>IN</code> into a <code>JOIN</code> with the <code>IN</code> subquery in the outer loop and doing streaming aggregation. Q13 cannot be done without a well-optimized <code><a class="auto-href" href="http://dbpedia.org/resource/Hash_join" id="link-id0x24974830">HASH JOIN</a></code> which besides must be partitioned at the larger scales.</p> <p>Having such trick questions in an important benchmark eventually results in everybody doing the optimizations that the benchmark clearly calls for. Making benchmarks thus entails a responsibility ultimately to the end user, because an irrelevant benchmark might in the worst case send developers chasing things that are beside the point.</p> <p>In the following, we will look at what BSBM-BI requires from the database and how these requirements can be further developed and extended.</p> <p>BSBM-BI does not have any clear trick questions, at least not premeditatedly. BSBM-BI just requires a cost model that can guess the fanout of a <code>JOIN</code> and the cardinality of a <code>GROUP BY</code>; it is enough to distinguish smaller from greater; the guess does not otherwise have to be very good. Further, the queries are written in the benchmark text so that joining from left to right would work, so not even a cost-based optimizer is strictly needed. I did however have to add some cardinality statistics to get reasonable <code>JOIN</code> order since we always reorder the query regardless of the source formulation.</p> <p>BSBM-BI does have variable selectivity from the drill-downs; thus these may call for different <code>JOIN</code> orders for different parameter values. I have not looked into whether this really makes a difference, though.</p> <p>There are places in BSBM-BI where using a <code>HASH JOIN</code> makes sense. We do not use <code>HASH JOINs</code> with <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x23cbf908">RDF</a> because there is an index for everything and making a <code>HASH JOIN</code> in the wrong place can have a large up-front cost, so one is more robust against cost model errors if one does not do <code>HASH JOINs</code>. This said, a <code>HASH JOIN</code> in the right place is a lot better than an index lookup. With TPC-H Q13, our best <code>HASH JOIN</code> is over 2x better than the best <code>INDEX</code>-based <code>JOIN</code>, both being well tuned. For questions like "count the hairballs made in <a class="auto-href" href="http://dbpedia.org/resource/Germany" id="link-id0x249d3e28">Germany</a> reviewed by Japanese Hello Kitty fans," where two ends of a <code>JOIN</code> path are fairly selective doing the other as a <code>HASH JOIN</code> is good. This can, if the <code>JOIN</code> is always cardinality-reducing, even be merged inside an <code>INDEX</code> lookup. We have such capabilities since we have been for a while gearing up for the relational races, but are not using any of these with BSBM-BI, although they would be useful.</p> <p>Let us see the profile for a single user 100 Mt run.</p> <p>The database activity summary is --</p> <p> <code>select db_activity (0, 'http');</code> </p> <p> <code> 161.3M rnd 210.2M seq 0 same seg 104.5M same pg 45.08M same par 0 disk 0 spec disk 0B / 0 messages 2.393K fork</code> </p> <p>See the post "<a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1b1f3068">What Does BSBM Explore Measure</a>" for an explanation of the numbers. We see that there is more sequential access than random and the random has fair locality with over half on the same page as the previous and a lot of the rest falling under the same parent. Funnily enough, the explore mix has more locality. Running with a longer vector size would probably increase performance by getting better locality. There is an optimization that adjusts vector size on the fly if locality is not sufficient but this is not being used here. So we manually set vector size to 100000 instead of the default 10000. We get --</p> <p> <code> 172.4M rnd 220.8M seq 0 same seg 149.6M same pg 10.99M same par 21 disk 861 spec disk 0B / 0 messages 754 fork</code> </p> <p>The throughput goes from 1494 to 1779. We see more hits on the same page, as expected. We do not make this setting a default since it raises the cost for small queries; therefore the vector size must be self-adjusting -- besides, expecting a DBA to tune this is not reasonable. We will just have to correctly tune the self-adjust logic, and we have again clear gains.</p> <p>Let us now go back to the first run with vector size 10000.</p> <p>The top of the CPU <code>oprofile</code> is as follows:</p> <blockquote> <code><pre> 722309 15.4507 cmpf_iri64n_iri64n 434791 9.3005 cmpf_iri64n_iri64n_anyn_iri64n 294712 6.3041 itc_next_set 273488 5.8501 itc_vec_split_search 203970 4.3631 itc_dive_transit 199687 4.2714 itc_page_rcf_search 181614 3.8848 dc_itc_append_any 173043 3.7015 itc_bm_vec_row_check 146727 3.1386 cmpf_int64n 128224 2.7428 itc_vec_row_check 113515 2.4282 dk_alloc 97296 2.0812 page_wait_access 62523 1.3374 qst_vec_get_int64 59014 1.2623 itc_next_set_parent 53589 1.1463 sslr_qst_get 48003 1.0268 ds_add 46641 0.9977 dk_free_tree 44551 0.9530 kc_var_col 43650 0.9337 page_col_cmp_1 35297 0.7550 cmpf_iri64n_iri64n_anyn_gt_lt 34589 0.7399 dv_compare 25864 0.5532 cmpf_iri64n_anyn_iri64n_iri64n_lte 23088 0.4939 dk_free </pre> </code> </blockquote> <p>The top 10 are all index traversal, with the key compare for two leading IRI keys in the lead, corresponding to a lookup with <code>P</code> and <code>S</code> given. The one after that is with all parts given, corresponding to an existence test. The existence tests could probably be converted to <code>HASH JOIN</code> lookups to good advantage. Aggregation and arithmetic are absent. We should probably add a query like TPC-H Q1 that does nothing but these two. Considering the overall profile, <code>GROUP BY</code> seems to be around 3%. We should probably put in a query that makes a very large number of groups and could make use of streaming aggregation, i.e., take advantage of a situation where aggregation input comes already grouped by the grouping columns.</p> <p>A BI use case should offer no problem with including arithmetic, but there are not that many numbers in the BSBM set. Some code sections in the queries with conditional execution and costly tests inside <code>ANDs</code> and <code>ORs</code> would be good. TPC-H has such in Q21 and Q19. An <code>OR</code> with existences where there would be gain from good guesses of a subquery's selectivity would be appropriate. Also, there should be conditional expressions somewhere with a lot of data, like the <code>CASE-WHEN</code> in TPC-H Q12.</p> <p>We can make BSBM-BI more interesting by putting in the above. Also we will have to see where we can profit from <code>HASH JOIN</code>, both small and large. There should be such places in the workload already so this is a matter of just playing a bit more.</p> <p>This post amounts to a cheat sheet for the BSBM-BI runs a bit farther down the road. By then we should be operational with the column store and Virtuoso 7 Cluster, though, so not everything is yet on the table.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1fd1d4e0">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1d5b07d8">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1dfe6c48">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x197fce30">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1fbf4210">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1beeb1e0">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1d7e1818">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1dfc1730">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1ea819a8">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x1ec73da0">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1fbdce90">Benchmarks, Redux (part 11): The Substance of Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x19928618">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1f3d8710">Benchmarks, Redux (part 13): BSBM-BI Modifications </a> </li> <li> Benchmarks, Redux (part 14): BSBM-BI Mix <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1e627400">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2011-03-22T17:04:38-04:00
Benchmarks, Redux (part 14): BSBM BI Mix
http://www.openlinksw.com:443/weblog/oerling/?date=2011-03-22#1683
2011-03-22T22:31:32Z
<p>In this post, we look at how we run the <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x23be8d28">BSBM</a>-BI mix. We consider the 100 Mt and 1000 Mt scales with <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x23b69e40">Virtuoso</a> 7 using the same hardware and software as in the previous posts. The changes to workload and metric are given in the previous post.</p> <p>Our intent here is to look at whether the metric works, and to see what results will look like in general. We are as much testing the benchmark as we are testing the system-under-test (SUT). The results shown here will likely not be comparable with future ones because we will most likely change the composition of the workload since it seems a bit out of balance. Anyway, for the sake of disclosure, we attach the query templates. The test driver we used will be made available soon, so the interested may still try a comparison with their systems. If you practice with this workload for the coming races, the effort will surely not be wasted.</p> <p>Once we have come up with a rules document, we will redo all that we have published so far by-the-book, and have it audited as part of the <a class="auto-href" href="http://lod2.eu/" id="link-id0x23a74c40">LOD2</a> service we plan for this (see previous posts in this series). This will introduce comparability; but before we get that far with the BI workload, the workload needs to evolve a bit.</p> <p>Below we show samples of test driver output; the whole output is <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/BenchmarksReduxSupportingFiles/br.tar.gz" id="link-id0x1b703ad8">downloadable</a>.</p> <p>100 Mt Single User</p> <blockquote> <code><pre> bsbm/testdriver -runs 1 -w 0 -idir /bs/1 -drill \ -ucf bsbm/usecases/businessIntelligence/<a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x247b7e08">sparql</a>.txt \ -dg <a class="auto-href" href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x232a86b0">http</a>://bsbm.org http://localhost:8604/sparql </pre> </code> </blockquote> <blockquote> <code><pre> 0: 43348.14ms, total: 43440ms Scale factor: 284826 Explore Endpoints: 1 Update Endpoints: 1 Drilldown: on Number of warmup runs: 0 Seed: 808080 Number of query mix runs (without warmups): 1 times min/max Querymix runtime: 43.3481s / 43.3481s Elapsed runtime: 43.348 seconds QMpH: 83.049 query mixes per hour CQET: 43.348 seconds average runtime of query mix CQET (geom.): 43.348 seconds geometric mean runtime of query mix AQET (geom.): 0.492 seconds geometric mean runtime of query Throughput: 1494.874 BSBM-BI throughput: qph*scale BI Power: 7309.820 BSBM-BI Power: qph*scale (geom) </pre> </code> </blockquote> <p>100 Mt 8 User </p> <blockquote> <code><pre> Thread 6: query mix 3: 195793.09ms, total: 196086.18ms Thread 8: query mix 0: 197843.84ms, total: 198010.50ms Thread 7: query mix 4: 201806.28ms, total: 201996.26ms Thread 2: query mix 5: 221983.93ms, total: 222105.96ms Thread 4: query mix 7: 225127.55ms, total: 225317.49ms Thread 3: query mix 6: 225860.49ms, total: 226050.17ms Thread 5: query mix 2: 230884.93ms, total: 231067.61ms Thread 1: query mix 1: 237836.61ms, total: 237959.11ms Benchmark run completed in 237.985427s Scale factor: 284826 Explore Endpoints: 1 Update Endpoints: 1 Drilldown: on Number of warmup runs: 0 Number of clients: 8 Seed: 808080 Number of query mix runs (without warmups): 8 times min/max Querymix runtime: 195.7931s / 237.8366s Total runtime (sum): 1737.137 seconds Elapsed runtime: 1737.137 seconds QMpH: 121.016 query mixes per hour CQET: 217.142 seconds average runtime of query mix CQET (geom.): 216.603 seconds geometric mean runtime of query mix AQET (geom.): 2.156 seconds geometric mean runtime of query Throughput: 2178.285 BSBM-BI throughput: qph*scale BI Power: 1669.745 BSBM-BI Power: qph*scale (geom) </pre> </code> </blockquote> <p>1000 Mt Single User</p> <blockquote> <code><pre> 0: 608707.03ms, total: 608768ms Scale factor: 2848260 Explore Endpoints: 1 Update Endpoints: 1 Drilldown: on Number of warmup runs: 0 Seed: 808080 Number of query mix runs (without warmups): 1 times min/max Querymix runtime: 608.7070s / 608.7070s Elapsed runtime: 608.707 seconds QMpH: 5.914 query mixes per hour CQET: 608.707 seconds average runtime of query mix CQET (geom.): 608.707 seconds geometric mean runtime of query mix AQET (geom.): 5.167 seconds geometric mean runtime of query Throughput: 1064.552 BSBM-BI throughput: qph*scale BI Power: 6967.325 BSBM-BI Power: qph*scale (geom) </pre> </code> </blockquote> <p>1000 Mt 8 User </p> <blockquote> <code><pre> bsbm/testdriver -runs 8 -mt 8 -w 0 -idir /bs/10 -drill \ -ucf bsbm/usecases/businessIntelligence/sparql.txt \ -dg http://bsbm.org http://localhost:8604/sparql </pre> </code> </blockquote> <blockquote> <code><pre> Thread 3: query mix 4: 2211275.25ms, total: 2211371.60ms Thread 4: query mix 0: 2212316.87ms, total: 2212417.99ms Thread 8: query mix 3: 2275942.63ms, total: 2276058.03ms Thread 5: query mix 5: 2441378.35ms, total: 2441448.66ms Thread 6: query mix 7: 2804001.05ms, total: 2804098.81ms Thread 2: query mix 2: 2808374.66ms, total: 2808473.71ms Thread 1: query mix 6: 2839407.12ms, total: 2839510.63ms Thread 7: query mix 1: 2889199.23ms, total: 2889263.17ms Benchmark run completed in 2889.302566s Scale factor: 2848260 Explore Endpoints: 1 Update Endpoints: 1 Drilldown: on Number of warmup runs: 0 Number of clients: 8 Seed: 808080 Number of query mix runs (without warmups): 8 times min/max Querymix runtime: 2211.2753s / 2889.1992s Total runtime (sum): 20481.895 seconds Elapsed runtime: 20481.895 seconds QMpH: 9.968 query mixes per hour CQET: 2560.237 seconds average runtime of query mix CQET (geom.): 2544.284 seconds geometric mean runtime of query mix AQET (geom.): 13.556 seconds geometric mean runtime of query Throughput: 1794.205 BSBM-BI throughput: qph*scale BI Power: 2655.678 BSBM-BI Power: qph*scale (geom) Metrics for Query: 1 Count: 8 times executed in whole run Time share 2.120884% of total execution time AQET: 54.299656 seconds (arithmetic mean) AQET(geom.): 34.607302 seconds (geometric mean) QPS: 0.13 Queries per second minQET/maxQET: 11.71547600s / 148.65379700s Metrics for Query: 2 Count: 8 times executed in whole run Time share 0.207382% of total execution time AQET: 5.309462 seconds (arithmetic mean) AQET(geom.): 2.737696 seconds (geometric mean) QPS: 1.34 Queries per second minQET/maxQET: 0.78729800s / 25.80948200s Metrics for Query: 3 Count: 8 times executed in whole run Time share 17.650472% of total execution time AQET: 451.893890 seconds (arithmetic mean) AQET(geom.): 410.481088 seconds (geometric mean) QPS: 0.02 Queries per second minQET/maxQET: 171.07262500s / 721.72939200s Metrics for Query: 5 Count: 32 times executed in whole run Time share 6.196565% of total execution time AQET: 39.661685 seconds (arithmetic mean) AQET(geom.): 6.849882 seconds (geometric mean) QPS: 0.18 Queries per second minQET/maxQET: 0.15696500s / 189.00906200s Metrics for Query: 6 Count: 8 times executed in whole run Time share 0.119916% of total execution time AQET: 3.070136 seconds (arithmetic mean) AQET(geom.): 2.056059 seconds (geometric mean) QPS: 2.31 Queries per second minQET/maxQET: 0.41524400s / 7.55655300s Metrics for Query: 7 Count: 40 times executed in whole run Time share 1.577963% of total execution time AQET: 8.079921 seconds (arithmetic mean) AQET(geom.): 1.342079 seconds (geometric mean) QPS: 0.88 Queries per second minQET/maxQET: 0.02205800s / 40.27761500s Metrics for Query: 8 Count: 40 times executed in whole run Time share 72.126818% of total execution time AQET: 369.323481 seconds (arithmetic mean) AQET(geom.): 114.431863 seconds (geometric mean) QPS: 0.02 Queries per second minQET/maxQET: 5.94377300s / 1824.57867400s </pre> </code> </blockquote> <p>The <a class="auto-href" href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x249ce740">CPU</a> for the multiuser runs stays above 1500% for the whole run. The CPU for the single user 100 Mt run is 630%; for the 1000 Mt run, this is 574%. This can be improved since the queries usually have a lot of <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x2871b1f0">data</a> to work on. But final <a class="auto-href" href="http://dbpedia.org/resource/Program_optimization" id="link-id0x22c95b90">optimization</a> is not our goal yet; we are just surveying the race track. The difference between a warm single user run and a cold single user run is about 15% with data on SSD; with data on disk, this would be more. The numbers shown are with warm <a class="auto-href" href="http://dbpedia.org/resource/Cache" id="link-id0x22ca4300">cache</a>. The single-user and multi-user Throughput difference, 1064 single-user vs. 1794 multi-user, is about what one would expect from the CPU utilization.</p> <p>With these numbers, the CPU does not appear badly memory-bound, else the increase would be less; also core multi-threading seems to bring some benefit. If the single-user run was at 800%, the Throughput would be 1488. The speed in excess of this may be attributed to core multi-threading, although we must remember that not every query mix is exactly the same length, so the figure is not exact. Core multi-threading does not seem to hurt, at the very least. Comparison of the same numbers with the column store will be interesting since it misses the cache a lot less and accordingly has better SMP scaling. The <a class="auto-href" href="http://dbpedia.org/resource/Intel_Corporation" id="link-id0x28814950">Intel</a> Nehalem memory subsystem is really pretty good.</p> <p> </p> <p>For reference, we show a run with Virtuoso 6 at 100Mt. </p> <blockquote> <code><pre> 0: 424754.40ms, total: 424829ms Scale factor: 284826 Explore Endpoints: 1 Update Endpoints: 1 Drilldown: on Number of warmup runs: 0 Seed: 808080 Number of query mix runs (without warmups): 1 times min/max Querymix runtime: 424.7544s / 424.7544s Elapsed runtime: 424.754 seconds QMpH: 8.475 query mixes per hour CQET: 424.754 seconds average runtime of query mix CQET (geom.): 424.754 seconds geometric mean runtime of query mix AQET (geom.): 1.097 seconds geometric mean runtime of query Throughput: 152.559 BSBM-BI throughput: qph*scale BI Power: 3281.150 BSBM-BI Power: qph*scale (geom) </pre> </code> </blockquote> <p>and 8 user </p> <blockquote> <code><pre> Thread 5: query mix 3: 616997.86ms, total: 617042.83ms Thread 7: query mix 4: 625522.18ms, total: 625559.09ms Thread 3: query mix 7: 626247.62ms, total: 626304.96ms Thread 1: query mix 0: 629675.17ms, total: 629724.98ms Thread 4: query mix 6: 667633.36ms, total: 667670.07ms Thread 8: query mix 2: 674206.07ms, total: 674256.72ms Thread 6: query mix 5: 695020.21ms, total: 695052.29ms Thread 2: query mix 1: 701824.67ms, total: 701864.91ms Benchmark run completed in 701.909341s Scale factor: 284826 Explore Endpoints: 1 Update Endpoints: 1 Drilldown: on Number of warmup runs: 0 Number of clients: 8 Seed: 808080 Number of query mix runs (without warmups): 8 times min/max Querymix runtime: 616.9979s / 701.8247s Total runtime (sum): 5237.127 seconds Elapsed runtime: 5237.127 seconds QMpH: 41.031 query mixes per hour CQET: 654.641 seconds average runtime of query mix CQET (geom.): 653.873 seconds geometric mean runtime of query mix AQET (geom.): 2.557 seconds geometric mean runtime of query Throughput: 738.557 BSBM-BI throughput: qph*scale BI Power: 1408.133 BSBM-BI Power: qph*scale (geom) </pre> </code> </blockquote> <p>Having the numbers, let us look at the metric and its scaling. We take the geometric mean of the single-user Power and the multiuser Throughput.</p> <blockquote> <code><pre> 100 Mt: sqrt ( 7771 * 2178 ); = 4114 1000 Mt: sqrt ( 6967 * 1794 ); = 3535 </pre> </code> </blockquote> <p>Scaling seems to work; the results are in the same general ballpark. The real times for the 1000 Mt run are a bit over 10x the times for the 100Mt run, as expected. The relative percentages of the queries are about the same on both scales, with the drill-down in Q8 alone being 77% and 72% respectively. The Q8 drill-down starts at the root of the product hierarchy. If we made this start one level from the top, its share would drop. This seems reasonable.</p> <p>Conversely, Q2 is out of place, with far too little share of the time. It takes a product as a starting point and shows a list of products with common features, sorted by descending count of common features. This would more appropriately be applied to a leaf product category instead, measuring how many of the products in the category have the top 20 features found in this category, to name an example.</p> <p>Also there should be more queries.</p> <p>At present it appears that BSBM-BI is definitely runnable, but a cursory look suffices to show that the workload needs more development and variety. We remember that I dreamt up the business questions last fall without much analysis, and that these questions were subsequently translated to SPARQL by FU Berlin. So, on one hand, BSBM-BI is of crucial importance because it is the first attempt at doing a benchmark with long running queries in SPARQL. On the other hand, BSBM-BI is not very good as a benchmark; <a class="auto-href" href="http://www.tpc.org/" id="link-id0x23227ce0">TPC</a>-<a class="auto-href" href="http://dbpedia.org/resource/TPC-H" id="link-id0x279c6700">H</a> is a lot better. This stands to reason, as TPC-H has had years and years of development and participation by many people.</p> <p>Benchmark queries are trick questions: For example, TPC-H Q18 cannot be done without changing an <code>IN</code> into a <code>JOIN</code> with the <code>IN</code> subquery in the outer loop and doing streaming aggregation. Q13 cannot be done without a well-optimized <code><a class="auto-href" href="http://dbpedia.org/resource/Hash_join" id="link-id0x238cbf88">HASH JOIN</a></code> which besides must be partitioned at the larger scales.</p> <p>Having such trick questions in an important benchmark eventually results in everybody doing the optimizations that the benchmark clearly calls for. Making benchmarks thus entails a responsibility ultimately to the end user, because an irrelevant benchmark might in the worst case send developers chasing things that are beside the point.</p> <p>In the following, we will look at what BSBM-BI requires from the database and how these requirements can be further developed and extended.</p> <p>BSBM-BI does not have any clear trick questions, at least not premeditatedly. BSBM-BI just requires a cost model that can guess the fanout of a <code>JOIN</code> and the cardinality of a <code>GROUP BY</code>; it is enough to distinguish smaller from greater; the guess does not otherwise have to be very good. Further, the queries are written in the benchmark text so that joining from left to right would work, so not even a cost-based optimizer is strictly needed. I did however have to add some cardinality statistics to get reasonable <code>JOIN</code> order since we always reorder the query regardless of the source formulation.</p> <p>BSBM-BI does have variable selectivity from the drill-downs; thus these may call for different <code>JOIN</code> orders for different parameter values. I have not looked into whether this really makes a difference, though.</p> <p>There are places in BSBM-BI where using a <code>HASH JOIN</code> makes sense. We do not use <code>HASH JOINs</code> with <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x235d8d88">RDF</a> because there is an index for everything and making a <code>HASH JOIN</code> in the wrong place can have a large up-front cost, so one is more robust against cost model errors if one does not do <code>HASH JOINs</code>. This said, a <code>HASH JOIN</code> in the right place is a lot better than an index lookup. With TPC-H Q13, our best <code>HASH JOIN</code> is over 2x better than the best <code>INDEX</code>-based <code>JOIN</code>, both being well tuned. For questions like "count the hairballs made in <a class="auto-href" href="http://dbpedia.org/resource/Germany" id="link-id0x2358ae60">Germany</a> reviewed by Japanese Hello Kitty fans," where two ends of a <code>JOIN</code> path are fairly selective doing the other as a <code>HASH JOIN</code> is good. This can, if the <code>JOIN</code> is always cardinality-reducing, even be merged inside an <code>INDEX</code> lookup. We have such capabilities since we have been for a while gearing up for the relational races, but are not using any of these with BSBM-BI, although they would be useful.</p> <p>Let us see the profile for a single user 100 Mt run.</p> <p>The database activity summary is --</p> <p> <code>select db_activity (0, 'http');</code> </p> <p> <code> 161.3M rnd 210.2M seq 0 same seg 104.5M same pg 45.08M same par 0 disk 0 spec disk 0B / 0 messages 2.393K fork</code> </p> <p>See the post "<a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1b1f3068">What Does BSBM Explore Measure</a>" for an explanation of the numbers. We see that there is more sequential access than random and the random has fair locality with over half on the same page as the previous and a lot of the rest falling under the same parent. Funnily enough, the explore mix has more locality. Running with a longer vector size would probably increase performance by getting better locality. There is an optimization that adjusts vector size on the fly if locality is not sufficient but this is not being used here. So we manually set vector size to 100000 instead of the default 10000. We get --</p> <p> <code> 172.4M rnd 220.8M seq 0 same seg 149.6M same pg 10.99M same par 21 disk 861 spec disk 0B / 0 messages 754 fork</code> </p> <p>The throughput goes from 1494 to 1779. We see more hits on the same page, as expected. We do not make this setting a default since it raises the cost for small queries; therefore the vector size must be self-adjusting -- besides, expecting a DBA to tune this is not reasonable. We will just have to correctly tune the self-adjust logic, and we have again clear gains.</p> <p>Let us now go back to the first run with vector size 10000.</p> <p>The top of the CPU <code>oprofile</code> is as follows:</p> <blockquote> <code><pre> 722309 15.4507 cmpf_iri64n_iri64n 434791 9.3005 cmpf_iri64n_iri64n_anyn_iri64n 294712 6.3041 itc_next_set 273488 5.8501 itc_vec_split_search 203970 4.3631 itc_dive_transit 199687 4.2714 itc_page_rcf_search 181614 3.8848 dc_itc_append_any 173043 3.7015 itc_bm_vec_row_check 146727 3.1386 cmpf_int64n 128224 2.7428 itc_vec_row_check 113515 2.4282 dk_alloc 97296 2.0812 page_wait_access 62523 1.3374 qst_vec_get_int64 59014 1.2623 itc_next_set_parent 53589 1.1463 sslr_qst_get 48003 1.0268 ds_add 46641 0.9977 dk_free_tree 44551 0.9530 kc_var_col 43650 0.9337 page_col_cmp_1 35297 0.7550 cmpf_iri64n_iri64n_anyn_gt_lt 34589 0.7399 dv_compare 25864 0.5532 cmpf_iri64n_anyn_iri64n_iri64n_lte 23088 0.4939 dk_free </pre> </code> </blockquote> <p>The top 10 are all index traversal, with the key compare for two leading IRI keys in the lead, corresponding to a lookup with <code>P</code> and <code>S</code> given. The one after that is with all parts given, corresponding to an existence test. The existence tests could probably be converted to <code>HASH JOIN</code> lookups to good advantage. Aggregation and arithmetic are absent. We should probably add a query like TPC-H Q1 that does nothing but these two. Considering the overall profile, <code>GROUP BY</code> seems to be around 3%. We should probably put in a query that makes a very large number of groups and could make use of streaming aggregation, i.e., take advantage of a situation where aggregation input comes already grouped by the grouping columns.</p> <p>A BI use case should offer no problem with including arithmetic, but there are not that many numbers in the BSBM set. Some code sections in the queries with conditional execution and costly tests inside <code>ANDs</code> and <code>ORs</code> would be good. TPC-H has such in Q21 and Q19. An <code>OR</code> with existences where there would be gain from good guesses of a subquery's selectivity would be appropriate. Also, there should be conditional expressions somewhere with a lot of data, like the <code>CASE-WHEN</code> in TPC-H Q12.</p> <p>We can make BSBM-BI more interesting by putting in the above. Also we will have to see where we can profit from <code>HASH JOIN</code>, both small and large. There should be such places in the workload already so this is a matter of just playing a bit more.</p> <p>This post amounts to a cheat sheet for the BSBM-BI runs a bit farther down the road. By then we should be operational with the column store and Virtuoso 7 Cluster, though, so not everything is yet on the table.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1fd1d4e0">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1d5b07d8">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1dfe6c48">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x197fce30">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1fbf4210">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1beeb1e0">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1d7e1818">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1dfc1730">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1ea819a8">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x1ec73da0">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1fbdce90">Benchmarks, Redux (part 11): The Substance of Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1681" id="link-id0x2aac18f3a688">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1682" id="link-id0xb4102b8">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> Benchmarks, Redux (part 14): BSBM-BI Mix <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1684" id="link-id0x15b5b028">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2014-08-21T10:43:31.630110-04:00
Benchmarks, Redux (part 13): BSBM BI Modifications
http://www.openlinksw.com:443/blog/vdb/blog/?date=2011-03-22#1686
2011-03-22T22:30:44Z
<p>In this post we introduce changes to the <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x234e0ca0">BSBM</a> BI queries and metric. These changes are motivated by prevailing benchmark practice and by our experiences in optimizing for the BSBM BI workload.</p> <p>We will publish results according to the definitions given here and recommend that any interested parties do likewise. The rationales are given in the text.</p> <h3>Query Mix</h3> <p>We have removed Q4 from the mix because it is quadratic to the scale factor. The other queries are roughly <code>n * log (n)</code>. </p> <h3>Parameter Substitution </h3> <p>All queries that take a product type as parameter are run in flights of several query invocations where the product type goes from broader to more specific. The initial product type specifies either the root product type or an immediate subtype of this, and the last in the drill-down is a leaf type.</p> <p>The rationale for this is that the choice of product type may make several orders of magnitude difference in the run time of a query. In order to make consecutive query mixes roughly comparable in execution time, all mixes should have a predictable number of query invocations with product types of each level.</p> <h3>Query Order </h3> <p>In the BI mix, when running multiple concurrent clients, each query mix is submitted in a random order. Queries which do drill-downs always have the steps of the drill-down as consecutive in the session, but the query templates are permuted. This is done so as to make less likely that there were two concurrent queries accessing exactly the same <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x23be8d28">data</a>. In this way, scans cannot be trivially shared between queries -- but there are still opportunities for reuse of results and adapting execution to working set, e.g., starting with what is in memory.</p> <h3>Metrics </h3> <p>We use a <a class="auto-href" href="http://www.tpc.org/" id="link-id0x238c81a0">TPC</a>-<a class="auto-href" href="http://dbpedia.org/resource/TPC-H" id="link-id0x28c6bbd8">H</a>-like metric. This metric consists of a single-user part and a multi-user part, called respectively <i>Power</i> and <i>Throughput.</i> The <i>Power</i> metric is a geometric mean of query run-time. The <i>Throughput</i> is the total run-time divided by the number of queries completed. After taking the mean, the time is converted into queries-per-hour. This time is then multiplied by the scale factor divided by the scale factor for 100 Mt. In other words, we consider the 100 Mt data set as the unit scale.</p> <p>The <i>Power</i> is defined as</p> <blockquote>( scale_factor / 284826 ) * 3600 / ( ( t1 * t1 * ... * tn ) ^ ( 1 / n ) ) </blockquote> <p>The <i>Throughput</i> is defined as</p> <blockquote>( scale_factor / 284826 ) * 3600 / ( ( t1 + t2 + ... + tn ) / n ) </blockquote> <p>The magic number <b><code>284826</code></b> is the scale that generates approximately 100 million triples (100 Mt). We consider this scale "one". The reason for the multiplication is that scores at different scales should get similar numbers; otherwise 10x larger scale would result roughly in 10x lower throughput with the BI queries.</p> <p>The <i>Composite</i> metric is the geometric mean of the <i>Power</i> and <i>Throughput</i> metrics. A complete report shows both <i>Power</i> and <i>Throughput</i> metrics, as well as individual query times for all queries. The rationale for using a geometric mean is to give an equal importance to long and short queries. Halving the execution time of either a long query or a short query will have the same effect on the metric. This is good for encouraging research into all aspects of query processing. On the other hand, real-life users are more interested in halving the time of queries that take one hour than of queries that take one second; therefore, the throughput metric considers run times.</p> <p>Taking the geometric mean of the two metrics gives more weight to the lower of the two than an arithmetic mean, hence we pay more attention to the worse of the two.</p> <p>Single-user and multi-user metrics are separate because of the relative importance of intra-query parallelization in BI workloads: There may not be large numbers of concurrent users, yet queries are still complex, and it is important to have maximum parallelization. Therefore the metric rewards single-user performance.</p> <p>In the next post we will look at the use of this metric and the actual content of BSBM BI.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1b02d528">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1d65f740">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1a797860">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x1d3538e0">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1e566f60">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1dedffd8">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1eb11528">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1db46c38">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1c8174e8">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x1dfa9338">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1e6dd7b0">Benchmarks, Redux (part 11): The Substance of Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1d154bb0">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> Benchmarks, Redux (part 13): BSBM BI Modifications <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1f242ae0">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1ebf2f98">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2011-03-22T17:04:34.000003-04:00
Benchmarks, Redux (part 13): BSBM BI Modifications
http://www.openlinksw.com:443/weblog/oerling/?date=2011-03-22#1682
2011-03-22T22:30:44Z
<p>In this post we introduce changes to the <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x23cb6710">BSBM</a> BI queries and metric. These changes are motivated by prevailing benchmark practice and by our experiences in optimizing for the BSBM BI workload.</p> <p>We will publish results according to the definitions given here and recommend that any interested parties do likewise. The rationales are given in the text.</p> <h3>Query Mix</h3> <p>We have removed Q4 from the mix because it is quadratic to the scale factor. The other queries are roughly <code>n * log (n)</code>. </p> <h3>Parameter Substitution </h3> <p>All queries that take a product type as parameter are run in flights of several query invocations where the product type goes from broader to more specific. The initial product type specifies either the root product type or an immediate subtype of this, and the last in the drill-down is a leaf type.</p> <p>The rationale for this is that the choice of product type may make several orders of magnitude difference in the run time of a query. In order to make consecutive query mixes roughly comparable in execution time, all mixes should have a predictable number of query invocations with product types of each level.</p> <h3>Query Order </h3> <p>In the BI mix, when running multiple concurrent clients, each query mix is submitted in a random order. Queries which do drill-downs always have the steps of the drill-down as consecutive in the session, but the query templates are permuted. This is done so as to make less likely that there were two concurrent queries accessing exactly the same <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x244a3d88">data</a>. In this way, scans cannot be trivially shared between queries -- but there are still opportunities for reuse of results and adapting execution to working set, e.g., starting with what is in memory.</p> <h3>Metrics </h3> <p>We use a <a class="auto-href" href="http://www.tpc.org/" id="link-id0x23880db8">TPC</a>-<a class="auto-href" href="http://dbpedia.org/resource/TPC-H" id="link-id0x29201c58">H</a>-like metric. This metric consists of a single-user part and a multi-user part, called respectively <i>Power</i> and <i>Throughput.</i> The <i>Power</i> metric is a geometric mean of query run-time. The <i>Throughput</i> is the total run-time divided by the number of queries completed. After taking the mean, the time is converted into queries-per-hour. This time is then multiplied by the scale factor divided by the scale factor for 100 Mt. In other words, we consider the 100 Mt data set as the unit scale.</p> <p>The <i>Power</i> is defined as</p> <blockquote>( scale_factor / 284826 ) * 3600 / ( ( t1 * t1 * ... * tn ) ^ ( 1 / n ) ) </blockquote> <p>The <i>Throughput</i> is defined as</p> <blockquote>( scale_factor / 284826 ) * 3600 / ( ( t1 + t2 + ... + tn ) / n ) </blockquote> <p>The magic number <b><code>284826</code></b> is the scale that generates approximately 100 million triples (100 Mt). We consider this scale "one". The reason for the multiplication is that scores at different scales should get similar numbers; otherwise 10x larger scale would result roughly in 10x lower throughput with the BI queries.</p> <p>The <i>Composite</i> metric is the geometric mean of the <i>Power</i> and <i>Throughput</i> metrics. A complete report shows both <i>Power</i> and <i>Throughput</i> metrics, as well as individual query times for all queries. The rationale for using a geometric mean is to give an equal importance to long and short queries. Halving the execution time of either a long query or a short query will have the same effect on the metric. This is good for encouraging research into all aspects of query processing. On the other hand, real-life users are more interested in halving the time of queries that take one hour than of queries that take one second; therefore, the throughput metric considers run times.</p> <p>Taking the geometric mean of the two metrics gives more weight to the lower of the two than an arithmetic mean, hence we pay more attention to the worse of the two.</p> <p>Single-user and multi-user metrics are separate because of the relative importance of intra-query parallelization in BI workloads: There may not be large numbers of concurrent users, yet queries are still complex, and it is important to have maximum parallelization. Therefore the metric rewards single-user performance.</p> <p>In the next post we will look at the use of this metric and the actual content of BSBM BI.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1b02d528">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1d65f740">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1a797860">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x1d3538e0">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1e566f60">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1dedffd8">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1eb11528">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1db46c38">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1c8174e8">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x1dfa9338">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1e6dd7b0">Benchmarks, Redux (part 11): The Substance of Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1681" id="link-id0x2aac14880528">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> Benchmarks, Redux (part 13): BSBM BI Modifications <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1683" id="link-id0x2aac01988188">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1684" id="link-id0x2aac147d9c28">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2014-08-21T10:43:55.147818-04:00
Benchmarks, Redux (part 12): Our Own BSBM Results Report
http://www.openlinksw.com:443/blog/vdb/blog/?date=2011-03-22#1685
2011-03-22T22:29:56Z
<p> <i>This is a placeholder; it will be replaced with a complete report in the very near future.</i> </p>
2011-03-22T17:04:29.000008-04:00
Benchmarks, Redux (part 12): Our Own BSBM Results Report
http://www.openlinksw.com:443/weblog/oerling/?date=2011-03-22#1681
2011-03-22T22:29:56Z
<p> <i>This is a placeholder; it will be replaced with a complete report in the very near future.</i> </p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x2aac14130068">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x2aac145841f8">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x2aac18f850a8">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0xa55d678">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x2aac182567b8">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x18b86248">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x180538c8">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x14a8d558">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x103045f8">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x15504f48">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x19b87638">Benchmarks, Redux (part 11): The Substance of Benchmarks</a> </li> <li> Benchmarks, Redux (part 12): Our Own BSBM Results Report<i> (this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1682" id="link-id0x2aabf707c0b8">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1683" id="link-id0x17d856f8">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1684" id="link-id0x1b5dc328">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2014-08-21T10:44:46.103020-04:00
Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks
http://www.openlinksw.com:443/blog/vdb/blog/?date=2011-03-10#1680
2011-03-10T23:30:11Z
<p>Let us talk about what ought to be benchmarked in the context of <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x2a84d3c0">RDF</a>.</p> <p>A point that often gets brought up by RDF-ers when talking about benchmarks is that there already exist systems which perform very well at <a class="auto-href" href="http://www.tpc.org/" id="link-id0x2a9758e8">TPC</a>-<a class="auto-href" href="http://dbpedia.org/resource/TPC-H" id="link-id0x2a8fa2a0">H</a> and similar workloads, and therefore there is no need for RDF to go there. It is, as it were, somebody else's problem; besides, it is a solved one.</p> <p>On the other hand, being able to express what is generally expected of a query language might not be a core competence or a competitive edge, but it certainly is a checklist item.</p> <p> <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x29c75a30">BSBM</a> seems to be adopted as a de facto RDF benchmark, as there indeed is almost nothing else. But we should not lose sight of the fact that this is in fact a relational <a class="auto-href" href="http://dbpedia.org/resource/Database_schema" id="link-id0x2a0565b8">schema</a> and workload that has just been straightforwardly transformed to RDF. BSBM was made, after all, in part for measuring RDB to RDF mapping. Thus BSBM is no more RDF-ish than a trivially RDF-ized TPC-H would be. TPC-H is however a bit more difficult if also a better thought out benchmark than the BSBM BI Mix proposal. But I do not expect an RDF audience to have any enthusiasm for this as this is indeed a very tough race by now, and besides one in which RDB and <a class="auto-href" href="http://dbpedia.org/resource/SQL" id="link-id0x29c44d50">SQL</a> will keep some advantage. However, using this as a validation test is meaningful, as there exists a validation dataset and queries that we already have RDF-ized. We could publish these and call this "RDF-H". </p> <p>In the following I will outline what would constitute an RDF-friendly, scientifically interesting benchmark. The points are in part based on discussions with <a class="auto-href" href="http://nl.linkedin.com/in/peterboncz" id="link-id0x2ac282f0">Peter Boncz</a> of <a class="auto-href" href="http://dbpedia.org/resource/National_Research_Institute_for_Mathematics_and_Computer_Science" id="link-id0x2a1c9e10">CWI</a>.</p> <p>The <a class="auto-href" href="http://www.w3.org/wiki/Social_Network_Intelligence_BenchMark" id="link-id0x29e7d3d8">Social Network Intelligence Benchmark</a> (<a class="auto-href" href="http://www.w3.org/wiki/Social_Network_Intelligence_BenchMark" id="link-id0x2a70e3c0">SNIB</a>) takes the social web Facebook-style schema Ivan Mikhailov and I made last year under the name of Botnet BM. In <a class="auto-href" href="http://lod2.eu/" id="link-id0x2a9a70f0">LOD2</a>, CWI is presently working on this.</p> <p>The <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x2ad04408">data</a> includes <a class="auto-href" href="http://dbpedia.org/resource/DBpedia" id="link-id0x29d5eeb0">DBpedia</a> as a base component used for providing conversation topics, <a class="auto-href" href="http://dbpedia.org/resource/Information" id="link-id0x2ac97c40">information</a> about geographical locales of simulated users, etc. DBpedia is not very large, around 200M-300M triples, but it is diverse enough.</p> <p>The data will have correlations, e.g., people who talk about sports tend to know other people who talk about the same sport, and they are more likely to know people from their geographical area than from elsewhere. </p> <p>The bulk of the data consists of a rich history of interactions including messages to individuals and groups, linking to people, dropping links, joining and leaving groups, and so forth. The messages are tagged using real-world concepts from DBpedia, and there is correlation between tagging and textual content since both are generated from Dbpedia articles. Since there is such correlation, <a class="auto-href" href="http://dbpedia.org/resource/Natural_language_processing" id="link-id0x2ac359c0">NLP</a> techniques like <a class="auto-href" href="http://dbpedia.org/resource/Entity" id="link-id0x2a1c8ed0">entity</a> and relationship extraction can be used with the data even though this is not the primary thrust of SNIB.</p> <p>There is variation in frequency of online interaction, and this interaction consist of sessions. For example, one could analyze user behavior per time of day for online ad placement.</p> <p>The data probably should include propagating memes, fashions, and trends that travel on the social network. With this, one could query about their origin and speed of propagation.</p> <p>There should probably be cases of duplicate identities in the data, i.e., one real person using many online accounts to push an agenda. Resolving duplicate identities makes for nice queries.</p> <p>Ragged data with half-filled profiles and misspelled identifiers like person and place names are a natural part of the social web use case. The data generator should take this into account.</p> <ul> <li> <p>Distribution of popularity and activity should follow a power-law-like pattern; actual measures of popularity can be sampled from existing social networks even though large quantities of data cannot easily be extracted.</p> </li> <li> <p>The dataset should be predictably scalable. For the workload considered, the relative importance of the queries or other measured tasks should not change dramatically with the scale.</p> </li> </ul> <p>For example some queries are logarithmic to data size (e.g., find connections to a person), some are linear (e.g., find average online time of sports fans on Sundays), and some are quadratic or worse (e.g., find two extremists of the same ideology that are otherwise unrelated). Making a single metric from such parts may not be meaningful. Therefore, SNIB might be structured into different workloads.</p> <p>The first would be an online mix with typically short lookups and updates, around <code>O ( log ( n ) )</code>. </p> <p>The Business Intelligence Mix would be composed of queries around <code>OO ( n log ( n ) )</code>. Even so, with real data, choice of parameters will provide dramatic changes in query run-time. Therefore a run should be specified to have a predictable distribution of "hard" and "easy" parameter choices. In the BSBM BI mix modification, I did this by defining some to be drill downs from a more general to a more specific level of a hierarchy. This could be done here too in some cases; other cases would have to be defined with buckets of values. </p> <p>Both the real world and LOD2 are largely concerned with data integration. The SNIB workload can have aspects of this, for example, in resolving duplicate identities. These operations are more complex than typical database queries, as the attributes used for joining might not even match in the initial data.</p> <p>One characteristic of these is the production of sometimes large intermediate results that need to be materialized. Doing these operations in practice requires procedural control. Further, running algorithms like network analytics (e.g., Page rank, centrality, etc.) involves aggregation of intermediate results that is not very well expressible in a query language. Some basic graph operations like shortest path are expressible but then are not in unextended <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x29d26588">SPARQL</a> 1.1; as these would for example involve returning paths, which are explicitly excluded from the spec.</p> <p>These are however the areas where we need to go for a benchmark that is more than a repackaging of a relational BI workload.</p> <p>We find that such a workload will have procedural sections either in application code or stored procedures. Map-reduce is sometimes used for scaling these. As one would expect, many cluster databases have their own version of these control structures. Therefore some of the SNIB workload could even be implemented as map-reduce jobs alongside parallel database implementations. We might here touch base with the <a class="auto-href" href="http://www.larkc.eu/" id="link-id0x29b69640">LarKC</a> map-reduce work to see if it could be applied to SNIB workloads. </p> <p>We see a three-level structure emerging. There is an <i>Online</i> mix which is a bit like the BSBM <i>Explore</i> mix, and an <i>Analytics</i> mix which is on the same order of complexity as TPC-H. These may have a more-or-less fixed query formulation and test driver. Beyond these, yet working on the same data, we have a set of <i>Predefined Tasks</i> which the test sponsor may implement in a manner of their choice.</p> <p>We would finally get to the "raging conflict" between the "declarativists" and the "map reductionists." Last year's VLDB had a lot of map-reduce papers. I know of comparisons between <a class="auto-href" href="http://www.vertica.com/" id="link-id0x2a8c4510">Vertica</a> and map reduce for doing a fairly simple SQL query on a lot of data, but here we would be talking about much more complex jobs on more interesting (i.e., less uniform) data.</p> <p>We might even interest some of the cluster <a class="auto-href" href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x2995aaa8">RDBMS</a> players (<a class="auto-href" href="http://www.teradata.com/" id="link-id0x29c9af10">Teradata</a>, Vertica, <a class="auto-href" href="http://dbpedia.org/resource/Greenplum" id="link-id0x29c9af38">Greenplum</a>, <a class="auto-href" href="http://dbpedia.org/page/Oracle_Exadata" id="link-id0x29d48b78">Oracle Exadata</a>, <a class="auto-href" href="http://www.paraccel.com/" id="link-id0x29d48ba0">ParAccel</a>, and/or <a class="auto-href" href="http://www.asterdata.com/" id="link-id0x29bf8fb0">Aster Data</a>, to name a few) in running this workload using their map-reduce analogs.</p> <p>We see that as we get to topics beyond relational BI, we do not find ourselves in an RDF-only world but very much at a crossroads of many technologies, e.g., map-reduce and its database analogs, various custom built databases, graph libraries, data integration and cleaning tools, and so forth.</p> <p>There is not, nor ought there to be, a sheltered, RDF-only enclave. RDF will have to justify itself in a world of alternatives.</p> <p>This must be reflected in our benchmark development, so relational BI is not irrelevant; in fact, it is what everybody does. RDF cannot be a total failure at this, even if this were not RDF's claim to fame. The claim to fame comes after we pass this stage, which is what we intend to explore in SNIB.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1c9f7ab8">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1dd17b28">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1eb20620">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x1f8a5ae8">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1ac14a08">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1d1f8d58">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1ea83308">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1b548028">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1c3d9c58">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x1f5e6978">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1c082a28">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1ec73578">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1eb25d48">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1b261958">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2011-03-14T19:37:28-04:00
Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks
http://www.openlinksw.com:443/weblog/oerling/?date=2011-03-10#1678
2011-03-10T23:30:11Z
<p>Let us talk about what ought to be benchmarked in the context of <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x29979188">RDF</a>.</p> <p>A point that often gets brought up by RDF-ers when talking about benchmarks is that there already exist systems which perform very well at <a class="auto-href" href="http://www.tpc.org/" id="link-id0x2a7082d0">TPC</a>-<a class="auto-href" href="http://dbpedia.org/resource/TPC-H" id="link-id0x29997988">H</a> and similar workloads, and therefore there is no need for RDF to go there. It is, as it were, somebody else's problem; besides, it is a solved one.</p> <p>On the other hand, being able to express what is generally expected of a query language might not be a core competence or a competitive edge, but it certainly is a checklist item.</p> <p> <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x2b164128">BSBM</a> seems to be adopted as a de facto RDF benchmark, as there indeed is almost nothing else. But we should not lose sight of the fact that this is in fact a relational <a class="auto-href" href="http://dbpedia.org/resource/Database_schema" id="link-id0x2a84d3c0">schema</a> and workload that has just been straightforwardly transformed to RDF. BSBM was made, after all, in part for measuring RDB to RDF mapping. Thus BSBM is no more RDF-ish than a trivially RDF-ized TPC-H would be. TPC-H is however a bit more difficult if also a better thought out benchmark than the BSBM BI Mix proposal. But I do not expect an RDF audience to have any enthusiasm for this as this is indeed a very tough race by now, and besides one in which RDB and <a class="auto-href" href="http://dbpedia.org/resource/SQL" id="link-id0x2aba65b0">SQL</a> will keep some advantage. However, using this as a validation test is meaningful, as there exists a validation dataset and queries that we already have RDF-ized. We could publish these and call this "RDF-H". </p> <p>In the following I will outline what would constitute an RDF-friendly, scientifically interesting benchmark. The points are in part based on discussions with <a class="auto-href" href="http://nl.linkedin.com/in/peterboncz" id="link-id0x29c81a40">Peter Boncz</a> of <a class="auto-href" href="http://dbpedia.org/resource/National_Research_Institute_for_Mathematics_and_Computer_Science" id="link-id0x2a0c8190">CWI</a>.</p> <p>The <a class="auto-href" href="http://www.w3.org/wiki/Social_Network_Intelligence_BenchMark" id="link-id0x29ca15c0">Social Network Intelligence Benchmark</a> (<a class="auto-href" href="http://www.w3.org/wiki/Social_Network_Intelligence_BenchMark" id="link-id0x2990a6b8">SNIB</a>) takes the social web Facebook-style schema Ivan Mikhailov and I made last year under the name of Botnet BM. In <a class="auto-href" href="http://lod2.eu/" id="link-id0x2a2e5338">LOD2</a>, CWI is presently working on this.</p> <p>The <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x2a650cc0">data</a> includes <a class="auto-href" href="http://dbpedia.org/resource/DBpedia" id="link-id0x2a2e5808">DBpedia</a> as a base component used for providing conversation topics, <a class="auto-href" href="http://dbpedia.org/resource/Information" id="link-id0x29a19570">information</a> about geographical locales of simulated users, etc. DBpedia is not very large, around 200M-300M triples, but it is diverse enough.</p> <p>The data will have correlations, e.g., people who talk about sports tend to know other people who talk about the same sport, and they are more likely to know people from their geographical area than from elsewhere. </p> <p>The bulk of the data consists of a rich history of interactions including messages to individuals and groups, linking to people, dropping links, joining and leaving groups, and so forth. The messages are tagged using real-world concepts from DBpedia, and there is correlation between tagging and textual content since both are generated from Dbpedia articles. Since there is such correlation, <a class="auto-href" href="http://dbpedia.org/resource/Natural_language_processing" id="link-id0x29995600">NLP</a> techniques like <a class="auto-href" href="http://dbpedia.org/resource/Entity" id="link-id0x29910c58">entity</a> and relationship extraction can be used with the data even though this is not the primary thrust of SNIB.</p> <p>There is variation in frequency of online interaction, and this interaction consist of sessions. For example, one could analyze user behavior per time of day for online ad placement.</p> <p>The data probably should include propagating memes, fashions, and trends that travel on the social network. With this, one could query about their origin and speed of propagation.</p> <p>There should probably be cases of duplicate identities in the data, i.e., one real person using many online accounts to push an agenda. Resolving duplicate identities makes for nice queries.</p> <p>Ragged data with half-filled profiles and misspelled identifiers like person and place names are a natural part of the social web use case. The data generator should take this into account.</p> <ul> <li> <p>Distribution of popularity and activity should follow a power-law-like pattern; actual measures of popularity can be sampled from existing social networks even though large quantities of data cannot easily be extracted.</p> </li> <li> <p>The dataset should be predictably scalable. For the workload considered, the relative importance of the queries or other measured tasks should not change dramatically with the scale.</p> </li> </ul> <p>For example some queries are logarithmic to data size (e.g., find connections to a person), some are linear (e.g., find average online time of sports fans on Sundays), and some are quadratic or worse (e.g., find two extremists of the same ideology that are otherwise unrelated). Making a single metric from such parts may not be meaningful. Therefore, SNIB might be structured into different workloads.</p> <p>The first would be an online mix with typically short lookups and updates, around <code>O ( log ( n ) )</code>. </p> <p>The Business Intelligence Mix would be composed of queries around <code>OO ( n log ( n ) )</code>. Even so, with real data, choice of parameters will provide dramatic changes in query run-time. Therefore a run should be specified to have a predictable distribution of "hard" and "easy" parameter choices. In the BSBM BI mix modification, I did this by defining some to be drill downs from a more general to a more specific level of a hierarchy. This could be done here too in some cases; other cases would have to be defined with buckets of values. </p> <p>Both the real world and LOD2 are largely concerned with data integration. The SNIB workload can have aspects of this, for example, in resolving duplicate identities. These operations are more complex than typical database queries, as the attributes used for joining might not even match in the initial data.</p> <p>One characteristic of these is the production of sometimes large intermediate results that need to be materialized. Doing these operations in practice requires procedural control. Further, running algorithms like network analytics (e.g., Page rank, centrality, etc.) involves aggregation of intermediate results that is not very well expressible in a query language. Some basic graph operations like shortest path are expressible but then are not in unextended <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x2a628e60">SPARQL</a> 1.1; as these would for example involve returning paths, which are explicitly excluded from the spec.</p> <p>These are however the areas where we need to go for a benchmark that is more than a repackaging of a relational BI workload.</p> <p>We find that such a workload will have procedural sections either in application code or stored procedures. Map-reduce is sometimes used for scaling these. As one would expect, many cluster databases have their own version of these control structures. Therefore some of the SNIB workload could even be implemented as map-reduce jobs alongside parallel database implementations. We might here touch base with the <a class="auto-href" href="http://www.larkc.eu/" id="link-id0x29ab4860">LarKC</a> map-reduce work to see if it could be applied to SNIB workloads. </p> <p>We see a three-level structure emerging. There is an <i>Online</i> mix which is a bit like the BSBM <i>Explore</i> mix, and an <i>Analytics</i> mix which is on the same order of complexity as TPC-H. These may have a more-or-less fixed query formulation and test driver. Beyond these, yet working on the same data, we have a set of <i>Predefined Tasks</i> which the test sponsor may implement in a manner of their choice.</p> <p>We would finally get to the "raging conflict" between the "declarativists" and the "map reductionists." Last year's VLDB had a lot of map-reduce papers. I know of comparisons between <a class="auto-href" href="http://www.vertica.com/" id="link-id0x29bd5828">Vertica</a> and map reduce for doing a fairly simple SQL query on a lot of data, but here we would be talking about much more complex jobs on more interesting (i.e., less uniform) data.</p> <p>We might even interest some of the cluster <a class="auto-href" href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x29d49c18">RDBMS</a> players (<a class="auto-href" href="http://www.teradata.com/" id="link-id0x29d49c40">Teradata</a>, Vertica, <a class="auto-href" href="http://dbpedia.org/resource/Greenplum" id="link-id0x2bba2248">Greenplum</a>, <a class="auto-href" href="http://dbpedia.org/page/Oracle_Exadata" id="link-id0x2bba2270">Oracle Exadata</a>, <a class="auto-href" href="http://www.paraccel.com/" id="link-id0x2ac756d0">ParAccel</a>, and/or <a class="auto-href" href="http://www.asterdata.com/" id="link-id0x2ac756f8">Aster Data</a>, to name a few) in running this workload using their map-reduce analogs.</p> <p>We see that as we get to topics beyond relational BI, we do not find ourselves in an RDF-only world but very much at a crossroads of many technologies, e.g., map-reduce and its database analogs, various custom built databases, graph libraries, data integration and cleaning tools, and so forth.</p> <p>There is not, nor ought there to be, a sheltered, RDF-only enclave. RDF will have to justify itself in a world of alternatives.</p> <p>This must be reflected in our benchmark development, so relational BI is not irrelevant; in fact, it is what everybody does. RDF cannot be a total failure at this, even if this were not RDF's claim to fame. The claim to fame comes after we pass this stage, which is what we intend to explore in SNIB.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1c9f7ab8">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1dd17b28">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1eb20620">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x1f8a5ae8">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1ac14a08">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1d1f8d58">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1ea83308">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1b548028">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1c3d9c58">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x1f5e6978">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1681" id="link-id0x1c118338">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1682" id="link-id0x1b5e6bf8">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1683" id="link-id0x1a045c38">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1684" id="link-id0x612d978">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2014-08-21T10:45:05.266280-04:00
Benchmarks, Redux (part 10): LOD2 and the Benchmark Process
http://www.openlinksw.com:443/blog/vdb/blog/?date=2011-03-10#1679
2011-03-10T23:29:41Z
<p>I have in the previous posts generally argued for and demonstrated the usefulness of benchmarks.</p> <p>Here I will talk about how this could be organized in a way that is tractable, and takes vendor and end user interests into account. These are my views on the subject and do not represent a <a class="auto-href" href="http://lod2.eu/" id="link-id0x2acb0760">LOD2</a> members consensus, but have been discussed in the consortium. </p> <p>My colleague Ivan Mikhailov once proposed that the only way to get benchmarks run right is to package them as a single script that does everything, like instant noodles -- just add water! But even instant noodles can be abused: Cook too long, add too much water, maybe forget to light the stove, and complain that the result is unsatisfyingly hard and brittle, lacking the suppleness one has grown to expect from this delicacy. No, the answer lies at the other end of the culinary spectrum, in gourmet cooking. Let the best cooks show what they can do, and let them work at it; let those who in fact have capacity and motivation for creating <i>le chef d'oeuvre culinaire</i> ("the culinary masterpiece") create it. Even so, there are many value points along the dimensions of preparation time, cost, and esthetic layout, not to forget taste and nutritional values. Indeed, an intimate <a class="auto-href" href="http://dbpedia.org/resource/Knowledge" id="link-id0x2aca6a30">knowledge</a> <i>de la vie secrete du canard</i> ("the secret life of duck") is required in order to liberate the aroma that it might take flight and soar. In the previous, I have shed some light on how we prepare <i>le canard</i>, and if <i>le canard</i> be such then <i>la dinde</i> (turkey) might in some ways be analogous; who is to say?</p> <p>In other words, as a vendor, we want to have complete control over the benchmarking process, and have it take place in our environment at a time of our choice. In exchange for this, we are ready to document and observe possibly complicated rules, document how the runs are made, and let others monitor and repeat them on the equipment on which the results are obtained. This is the <a class="auto-href" href="http://www.tpc.org/" id="link-id0x2b847818">TPC</a> (Transaction Processing Performance Council) model.</p> <p>Another culture of doing benchmarks is the periodic challenge model used in TREC, the <a class="auto-href" href="http://challenge.semanticweb.org/" id="link-id0x2ac3a6f8">Billion Triples Challenge</a>, the Semantic Search Challenge and others. In this model, vendors prepare the benchmark submission and agree to joint publication.</p> <p>A third party performing benchmarks by itself is uncommon in databases. Licenses even often explicitly prohibit this, for understandable reasons.</p> <p>The LOD2 project has an outreach activity called Publink where we offer to help owners of <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x2aea5930">data</a> to publish it as <a class="auto-href" href="http://dbpedia.org/resource/Linked_Data" id="link-id0x2a790128">Linked Data</a>. Similarly, since FP 7s are supposed to offer a visible service to their communities, I proposed that LOD2 offer to serve a role in disseminating and auditing <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x29babb00">RDF</a> store benchmarks.</p> <p>One representative of an RDF store vendor I talked to, in relation to setting up a benchmark configuration of their product, told me that we could do this and that they would give some advice but that such an exercise was by its nature fundamentally flawed and could not possibly produce worthwhile results. The reason for this was that OpenLink engineers could not possibly learn enough about the other products nor unlearn enough of their own to make this a meaningful comparison.</p> <p>Isn't this the very truth? Let the chefs mix their own spices.</p> <p>This does not mean that there would not be comparability of results. If the benchmarks and processes are well defined, documented, and checked by a third party, these can be considered legitimate and not just one-off best-case results without further import.</p> <p>In order to stretch the envelope, which is very much a LOD2 goal, this benchmarking should be done on a variety of equipment -- whatever works best at the scale in question. Increasing the scale remains a stated objective. LOD2 even promised to run things with a trillion triples in another 3 years. </p> <p>Imagine that the unimpeachably impartial Berliners made house calls. Would this debase Justice to be a servant of mere show-off? Or would this on the contrary combine strict Justice with edifying Charity? Who indeed is in greater need of the light of objective evaluation than the vendor whose very nature makes a being of bias and prejudice?</p> <p>Even better, <a class="auto-href" href="http://dbpedia.org/resource/National_Research_Institute_for_Mathematics_and_Computer_Science" id="link-id0x2a21d108">CWI</a>, with its <a href="http://monetdb.cwi.nl/Development/Research/Articles/" id="link-id0x1d6479d0">stellar database pedigree</a>, agreed in principle to audit RDF benchmarks in LOD2. </p> <p>In this way one could get a stamp of approval for one's results regardless of when they were produced, and be free of the arbitrary schedule of third party benchmarking runs. On the relational side this is a process of some cost and complexity, but since the RDF side is still young and more on mutually friendly terms, the process can be somewhat lighter here. I did promise to draft some extra descriptions of process and result disclosure so that we could see how this goes.</p> <p>We could even do this unilaterally -- just publish <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x2a0d73d8">Virtuoso</a> results according to a predefined reporting and verification format. If others wished to publish by the same rules, LOD2 could use some of the benchmarking funds for auditing the proceedings. This could all take place over the <a class="auto-href" href="http://dbpedia.org/resource/.NET_Framework" id="link-id0x2a6b44a0">net</a>, so we are not talking about any huge cost or prohibitive amount of trouble. It would be in the FP7 spirit that LOD2 provide this service for free, naturally within reason.</p> <p>Then there is the matter of the <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x2a1722a8">BSBM</a> Business Intelligence (BI) mix. At present, it seems everybody has chosen to defer the matter to another round of BSBM runs in the summer. This seems to fit the pattern of a public challenge with a few months given for contenders to prepare their submissions. Here we certainly should look at bigger scales and more diverse hardware than in the Berlin runs published this time around. The BI workload is in fact fairly cluster friendly, with big joins and aggregations that parallelize well. There it would definitely make sense to reserve an actual cluster, and have all contenders set up their gear on it. If all have access to the run environment and to monitoring tools, we can be reasonably sure that things will be done in a transparent manner. </p> <p>(I will talk about the BI mix in more detail in <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1dfcc038">part 13</a> and <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1edaa388">part 14</a> of this series.)</p> <p>Once the BI mix has settled and there are a few interoperable implementations, likely in the summer, we could pass from the challenge model to a situation where vendors may publish results as they become available, with LOD2 offering its services for audit. </p> <p>Of course, this could be done even before then, but the content of the mix might not be settled. We likely need to check it on a few implementations first.</p> <p>For equipment, people can use their own, or LOD2 partners might on a case-by-case basis make some equipment available for running on the same hardware on which say the Virtuoso results were obtained. For example, FU Berlin could give people a login to get their recently published results fixed. Now this might or might not happen, so I will not hold my breath waiting for this but instead close with a proposal.</p> <p>As a unilateral diplomatic overture I put forth the following: If other vendors are interested in 1:1 comparison of their results with our publications, we can offer them a login to the same equipment. They can set up and tune their systems, and perform the runs. We will just watch. As an extra quid pro quo, they can try Virtuoso as configured for the results we have published, with the same data. Like this, both parties get to see the others' technology with proper tuning and installation. What, if anything, is reported about this activity is up to the owner of the technology being tested. We will publish a set of benchmark rules that can serve as a guideline for mutually comparable reporting, but we cannot force anybody to use these. This all will function as a catalyst for technological advance, all to the ultimate benefit of the end user. If you wish to take advantage of this offer, you may contact <a href="mailto:hwilliams@openlinksw.com?subject=Collaborative RDF Benchmark" id="link-id0x1c071100">Hugh Williams at OpenLink Software, and we will see how this can be arranged in practice.</a> </p> <p>The next post will talk about the <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x19933fd8">actual content of benchmarks</a>. The milestone after this will be when we publish the measurement and reporting protocols.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1c554800">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1ec159e8">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1dd5eb10">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x18f05940">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1ed5ef10">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1e9cb130">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1dfa79d8">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1eb6f478">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1de5a918">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> Benchmarks, Redux (part 10): LOD2 and the Benchmark Process <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1dae9060">Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1f45fa10">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1f49d2b8">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1e68e4c8">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1e353858">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2011-03-14T19:37:14.000001-04:00
Benchmarks, Redux (part 10): LOD2 and the Benchmark Process
http://www.openlinksw.com:443/weblog/oerling/?date=2011-03-10#1677
2011-03-10T23:29:41Z
<p>I have in the previous posts generally argued for and demonstrated the usefulness of benchmarks.</p> <p>Here I will talk about how this could be organized in a way that is tractable, and takes vendor and end user interests into account. These are my views on the subject and do not represent a <a class="auto-href" href="http://lod2.eu/" id="link-id0x1d4999d0">LOD2</a> members consensus, but have been discussed in the consortium. </p> <p>My colleague Ivan Mikhailov once proposed that the only way to get benchmarks run right is to package them as a single script that does everything, like instant noodles -- just add water! But even instant noodles can be abused: Cook too long, add too much water, maybe forget to light the stove, and complain that the result is unsatisfyingly hard and brittle, lacking the suppleness one has grown to expect from this delicacy. No, the answer lies at the other end of the culinary spectrum, in gourmet cooking. Let the best cooks show what they can do, and let them work at it; let those who in fact have capacity and motivation for creating <i>le chef d'oeuvre culinaire</i> ("the culinary masterpiece") create it. Even so, there are many value points along the dimensions of preparation time, cost, and esthetic layout, not to forget taste and nutritional values. Indeed, an intimate <a class="auto-href" href="http://dbpedia.org/resource/Knowledge" id="link-id0x1a63a168">knowledge</a> <i>de la vie secrete du canard</i> ("the secret life of duck") is required in order to liberate the aroma that it might take flight and soar. In the previous, I have shed some light on how we prepare <i>le canard</i>, and if <i>le canard</i> be such then <i>la dinde</i> (turkey) might in some ways be analogous; who is to say?</p> <p>In other words, as a vendor, we want to have complete control over the benchmarking process, and have it take place in our environment at a time of our choice. In exchange for this, we are ready to document and observe possibly complicated rules, document how the runs are made, and let others monitor and repeat them on the equipment on which the results are obtained. This is the <a class="auto-href" href="http://www.tpc.org/" id="link-id0x1d676280">TPC</a> (Transaction Processing Performance Council) model.</p> <p>Another culture of doing benchmarks is the periodic challenge model used in TREC, the <a class="auto-href" href="http://challenge.semanticweb.org/" id="link-id0x1c222020">Billion Triples Challenge</a>, the Semantic Search Challenge and others. In this model, vendors prepare the benchmark submission and agree to joint publication.</p> <p>A third party performing benchmarks by itself is uncommon in databases. Licenses even often explicitly prohibit this, for understandable reasons.</p> <p>The LOD2 project has an outreach activity called Publink where we offer to help owners of <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x112c3dc0">data</a> to publish it as <a class="auto-href" href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1d1b7078">Linked Data</a>. Similarly, since FP 7s are supposed to offer a visible service to their communities, I proposed that LOD2 offer to serve a role in disseminating and auditing <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x11c0ff08">RDF</a> store benchmarks.</p> <p>One representative of an RDF store vendor I talked to, in relation to setting up a benchmark configuration of their product, told me that we could do this and that they would give some advice but that such an exercise was by its nature fundamentally flawed and could not possibly produce worthwhile results. The reason for this was that OpenLink engineers could not possibly learn enough about the other products nor unlearn enough of their own to make this a meaningful comparison.</p> <p>Isn't this the very truth? Let the chefs mix their own spices.</p> <p>This does not mean that there would not be comparability of results. If the benchmarks and processes are well defined, documented, and checked by a third party, these can be considered legitimate and not just one-off best-case results without further import.</p> <p>In order to stretch the envelope, which is very much a LOD2 goal, this benchmarking should be done on a variety of equipment -- whatever works best at the scale in question. Increasing the scale remains a stated objective. LOD2 even promised to run things with a trillion triples in another 3 years. </p> <p>Imagine that the unimpeachably impartial Berliners made house calls. Would this debase Justice to be a servant of mere show-off? Or would this on the contrary combine strict Justice with edifying Charity? Who indeed is in greater need of the light of objective evaluation than the vendor whose very nature makes a being of bias and prejudice?</p> <p>Even better, <a class="auto-href" href="http://dbpedia.org/resource/National_Research_Institute_for_Mathematics_and_Computer_Science" id="link-id0x1c369958">CWI</a>, with its <a href="http://monetdb.cwi.nl/Development/Research/Articles/" id="link-id0x1d6479d0">stellar database pedigree</a>, agreed in principle to audit RDF benchmarks in LOD2. </p> <p>In this way one could get a stamp of approval for one's results regardless of when they were produced, and be free of the arbitrary schedule of third party benchmarking runs. On the relational side this is a process of some cost and complexity, but since the RDF side is still young and more on mutually friendly terms, the process can be somewhat lighter here. I did promise to draft some extra descriptions of process and result disclosure so that we could see how this goes.</p> <p>We could even do this unilaterally -- just publish <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1e0c0690">Virtuoso</a> results according to a predefined reporting and verification format. If others wished to publish by the same rules, LOD2 could use some of the benchmarking funds for auditing the proceedings. This could all take place over the <a class="auto-href" href="http://dbpedia.org/resource/.NET_Framework" id="link-id0x1ed4b9d8">net</a>, so we are not talking about any huge cost or prohibitive amount of trouble. It would be in the FP7 spirit that LOD2 provide this service for free, naturally within reason.</p> <p>Then there is the matter of the <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1ea75360">BSBM</a> Business Intelligence (BI) mix. At present, it seems everybody has chosen to defer the matter to another round of BSBM runs in the summer. This seems to fit the pattern of a public challenge with a few months given for contenders to prepare their submissions. Here we certainly should look at bigger scales and more diverse hardware than in the Berlin runs published this time around. The BI workload is in fact fairly cluster friendly, with big joins and aggregations that parallelize well. There it would definitely make sense to reserve an actual cluster, and have all contenders set up their gear on it. If all have access to the run environment and to monitoring tools, we can be reasonably sure that things will be done in a transparent manner. </p> <p>(I will talk about the BI mix in more detail in <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1dfcc038">part 13</a> and <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1edaa388">part 14</a> of this series.)</p> <p>Once the BI mix has settled and there are a few interoperable implementations, likely in the summer, we could pass from the challenge model to a situation where vendors may publish results as they become available, with LOD2 offering its services for audit. </p> <p>Of course, this could be done even before then, but the content of the mix might not be settled. We likely need to check it on a few implementations first.</p> <p>For equipment, people can use their own, or LOD2 partners might on a case-by-case basis make some equipment available for running on the same hardware on which say the Virtuoso results were obtained. For example, FU Berlin could give people a login to get their recently published results fixed. Now this might or might not happen, so I will not hold my breath waiting for this but instead close with a proposal.</p> <p>As a unilateral diplomatic overture I put forth the following: If other vendors are interested in 1:1 comparison of their results with our publications, we can offer them a login to the same equipment. They can set up and tune their systems, and perform the runs. We will just watch. As an extra quid pro quo, they can try Virtuoso as configured for the results we have published, with the same data. Like this, both parties get to see the others' technology with proper tuning and installation. What, if anything, is reported about this activity is up to the owner of the technology being tested. We will publish a set of benchmark rules that can serve as a guideline for mutually comparable reporting, but we cannot force anybody to use these. This all will function as a catalyst for technological advance, all to the ultimate benefit of the end user. If you wish to take advantage of this offer, you may contact <a href="mailto:hwilliams@openlinksw.com?subject=Collaborative RDF Benchmark" id="link-id0x1c071100">Hugh Williams at OpenLink Software, and we will see how this can be arranged in practice.</a> </p> <p>The next post will talk about the <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x19933fd8">actual content of benchmarks</a>. The milestone after this will be when we publish the measurement and reporting protocols.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1c554800">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1ec159e8">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1dd5eb10">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x18f05940">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1ed5ef10">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1e9cb130">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1dfa79d8">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1eb6f478">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1de5a918">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> Benchmarks, Redux (part 10): LOD2 and the Benchmark Process <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1dae9060">Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1681" id="link-id0xea8f8e8">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1682" id="link-id0x1ae34cd8">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1683" id="link-id0x19155c48">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1684" id="link-id0x1aa9bf78">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2014-08-21T10:45:14.858380-04:00
Benchmarks, Redux (part 9): BSBM With Cluster
http://www.openlinksw.com:443/blog/vdb/blog/?date=2011-03-09#1676
2011-03-09T22:54:50Z
<p>This post is dedicated to our brothers in horizontal partitioning (or sharding), <a class="auto-href" href="http://freebase.com/guid/9202a8c04000641f8000000005c908d6" id="link-id0x2a1e9010">Garlik</a> and <a class="auto-href" href="http://www.systap.com/bigdata.htm" id="link-id0x2acd5218">Bigdata</a>.</p> <p>At first sight, the <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x2bb33648">BSBM</a> <i>Explore</i> mix appears very cluster-unfriendly, as it contains short queries that access <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x2b8fffb8">data</a> at random. There is every opportunity for latency and few opportunities for parallelism.</p> <p>For this reason we had not even run the BSBM mix with <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x2a84b780">Virtuoso</a> Cluster. We were not surprised to learn that <a href="http://steveharris.tumblr.com/post/3453040647/bsbm-v3-post-mortem" id="link-id0x1c4ef8d8">Garlik hadn't run BSBM either</a>. We have understood from <a class="auto-href" href="http://www.systap.com/" id="link-id0x2ad3d050">Systap</a> that their Bigdata BSBM experiments were on a single-process configuration.</p> <p>But the 4Store results in the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/results/V6/index.html" id="link-id0x1f8090f8">recent Berlin report</a> were with a distributed setup, as 4Store always runs a multiprocess configuration, even on a single server, so it seemed interesting to us to compare how Virtuoso Cluster compares with Virtuoso Single with this workload. These tests were run on a different box than the recent BSBM tests, so those 4Store figures are not directly comparable.</p> <p>The setup here consists of 8 partitions, each managed by its own process, all running on the same box. Any of these processes can have its <a class="auto-href" href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x2ac28380">HTTP</a> and <a class="auto-href" href="http://dbpedia.org/resource/SQL" id="link-id0x2bba8720">SQL</a> listener and can provide the same service. Most access to data goes over the interconnect, except when the data is co-resident in the process which is coordinating the query. The interconnect is Unix domain sockets since all 8 processes are on the same box.</p> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="4" align="center">6 Cluster - Load Rates and Times</th> </tr> <tr> <th align="center">Scale</th> <th align="center">Rate <br /> (quads per second)</th> <th align="center">Load time <br /> (seconds)</th> <th align="center">Checkpoint time <br /> (seconds)</th> </tr> <tr> <th align="center">100 Mt</th> <td align="center"> 119,204 </td> <td align="center"> 749 </td> <td align="center"> 89 </td> </tr> <tr> <th align="center">200 Mt</th> <td align="center"> 121,607 </td> <td align="center"> 1486 </td> <td align="center"> 157 </td> </tr> <tr> <th align="center">1000 Mt</th> <td align="center"> 102,694 </td> <td align="center"> 8737 </td> <td align="center"> 979 </td> </tr> </table> <br /> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="4" align="center">6 Single - Load Rates and Times</th> </tr> <tr> <th align="center">Scale</th> <th align="center">Rate <br /> (quads per second)</th> <th align="center">Load time <br /> (seconds)</th> <th align="center">Checkpoint time <br /> (seconds)</th> </tr> <tr> <th align="center">100 Mt</th> <td align="center"> 74,713 </td> <td align="center"> 1192 </td> <td align="center"> 145 </td> </tr> </table> <p>The load times are systematically better than for 6 Single. This is also not bad compared to the 7 Single vectored load rates of 220 Kt/s or so. We note that loading is a cluster friendly operation, going at a steady 1400+% <a class="auto-href" href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x296b03b8">CPU</a> utilization with an aggregate message throughput of 40MB/s. 7 Single is faster because of vectoring at the index level, not because the clusters were hitting communication overheads. 6 Cluster is faster than 6 Single because scale-out in this case diminishes contention, even on a single box.</p> <p>Throughput is as follows:</p> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="3" align="center"> 6 Cluster - Throughput <br /> (QMpH, query mixes per hour) </th> </tr> <tr> <th align="center">Scale</th> <th align="center"> Single User </th> <th align="center"> 16 User </th> </tr> <tr> <th align="center">100 Mt</th> <td align="center"> 7318 </td> <td align="center"> 43120 </td> </tr> <tr> <th align="center">200 Mt</th> <td align="center"> 6222 </td> <td align="center"> 29981 </td> </tr> <tr> <th align="center">1000 Mt</th> <td align="center"> 2526 </td> <td align="center"> 11156 </td> </tr> </table> <br /> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="3" align="center"> 6 Single - Throughput <br /> (QMpH, query mixes per hour) </th> </tr> <tr> <th align="center">Scale</th> <th align="center"> Single User </th> <th align="center"> 16 User </th> </tr> <tr> <th align="center">100 Mt</th> <td align="center"> 7641 </td> <td align="center"> 29433 </td> </tr> <tr> <th align="center">200 Mt</th> <td align="center"> 6017 </td> <td align="center"> 13335 </td> </tr> <tr> <th align="center">1000 Mt</th> <td align="center"> 1770 </td> <td align="center"> 2487 </td> </tr> </table> <p>Below is a snapshot of status during the 6 Cluster 100 Mt run.</p> <blockquote> <code><pre> Cluster 8 nodes, 15 s. 25784 m/s 25682 KB/s 1160% cpu 0% read 740% clw threads 18r 0w 10i buffers 1133459 12 d 4 w 0 pfs cl 1: 10851 m/s 3911 KB/s 597% cpu 0% read 668% clw threads 17r 0w 10i buffers 143992 4 d 0 w 0 pfs cl 2: 2194 m/s 7959 KB/s 107% cpu 0% read 9% clw threads 1r 0w 0i buffers 143616 3 d 2 w 0 pfs cl 3: 2186 m/s 7818 KB/s 107% cpu 0% read 9% clw threads 0r 0w 0i buffers 140787 0 d 0 w 0 pfs cl 4: 2174 m/s 2804 KB/s 77% cpu 0% read 10% clw threads 0r 0w 0i buffers 140654 0 d 2 w 0 pfs cl 5: 2127 m/s 1612 KB/s 71% cpu 0% read 9% clw threads 0r 0w 0i buffers 140949 1 d 0 w 0 pfs cl 6: 2060 m/s 544 KB/s 66% cpu 0% read 10% clw threads 0r 0w 0i buffers 141295 2 d 0 w 0 pfs cl 7: 2072 m/s 517 KB/s 65% cpu 0% read 11% clw threads 0r 0w 0i buffers 141111 1 d 0 w 0 pfs cl 8: 2105 m/s 522 KB/s 66% cpu 0% read 10% clw threads 0r 0w 0i buffers 141055 1 d 0 w 0 pfs </pre> </code> </blockquote> <p>The main meters for cluster execution are the messages-per-second (m/s), the message volume (KB/s), and the total CPU% of the processes. </p> <p>We note that CPU utilization is highly uneven and messages are short, about 1K on the average, compared to about 100K during the load. CPU would be evenly divided between the nodes if each got a share of the HTTP requests. We changed the test driver to round-robin requests between multiple end points. The work does then get evenly divided, but the speed is not affected. Also, this does not improve the message sizes since the workload consists mostly of short lookups. However, with the processes spread over multiple servers, the round-robin would be essential for CPU and especially for interconnect throughput. </p> <p>Then we try 6 Cluster at 1000 Mt. For Single User, we get 1180 m/s, 6955 KB/s, and 173% cpu. For 16 User, this is 6573 m/s, 44366 KB/s, 1470% cpu.</p> <p>This is a lot better than the figures with 6 Single, due to lower contention on the index tree, as discussed in <i><a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1e9a0b58">A Benchmarking Story</a></i>. Also Single User throughput on 6 Cluster outperforms 6 Single, due to the natural parallelism of doing the Q5 joins in parallel in each partition. The larger the scale, the more weight this has in the metric. We see this also in the average message size, i.e., the KB/s throughput is almost double while the messages/s is a bit under a third.</p> <p>The small-scale 6 Cluster run is about even with the 6 Single figure. Looking at the details, we see that the qps for Q1 in 6 Cluster is half of that on 6 Single, whereas the qps for Q5 on 6 Cluster is about double that of the 6 Single. This is as one might expect; longer queries are favored, and single row lookups are penalized.</p> <p>Looking further at the 6 Cluster status we see the cluster wait (<code>clw</code>) to be 740%. For 16 Users, this means that about half of the execution real time is spent waiting for responses from other partitions. A high figure means uneven distribution between partitions; a low figure means even. This is as expected, since many queries are concerned with just one S and its related objects.</p> <p>We will update this section once 7 Cluster is ready. This will implement vectored execution and column store inside the cluster nodes.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1d7894d0">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1e434888">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1f6b5260">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x1dd29460">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1f0d78b8">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1f9a9670">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1c055370">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1dc06cd0">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> Benchmarks, Redux (part 9): BSBM With Cluster <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x18f04db0">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1ee729b8">Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1e2e76b8">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1d75ef48">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1ee518c0">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1d9244b0">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2011-03-14T19:36:58-04:00
Benchmarks, Redux (part 9): BSBM With Cluster
http://www.openlinksw.com:443/weblog/oerling/?date=2011-03-09#1675
2011-03-09T22:54:50Z
<p>This post is dedicated to our brothers in horizontal partitioning (or sharding), <a class="auto-href" href="http://freebase.com/guid/9202a8c04000641f8000000005c908d6" id="link-id0x113d92b0">Garlik</a> and <a class="auto-href" href="http://www.systap.com/bigdata.htm" id="link-id0x1cca0090">Bigdata</a>.</p> <p>At first sight, the <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1c1c1330">BSBM</a> <i>Explore</i> mix appears very cluster-unfriendly, as it contains short queries that access <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0xa2e1940">data</a> at random. There is every opportunity for latency and few opportunities for parallelism.</p> <p>For this reason we had not even run the BSBM mix with <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1e734de0">Virtuoso</a> Cluster. We were not surprised to learn that <a href="http://steveharris.tumblr.com/post/3453040647/bsbm-v3-post-mortem" id="link-id0x1c4ef8d8">Garlik hadn't run BSBM either</a>. We have understood from <a class="auto-href" href="http://www.systap.com/" id="link-id0x1c579da0">Systap</a> that their Bigdata BSBM experiments were on a single-process configuration.</p> <p>But the 4Store results in the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/results/V6/index.html" id="link-id0x1f8090f8">recent Berlin report</a> were with a distributed setup, as 4Store always runs a multiprocess configuration, even on a single server, so it seemed interesting to us to compare how Virtuoso Cluster compares with Virtuoso Single with this workload. These tests were run on a different box than the recent BSBM tests, so those 4Store figures are not directly comparable.</p> <p>The setup here consists of 8 partitions, each managed by its own process, all running on the same box. Any of these processes can have its <a class="auto-href" href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1bbcd560">HTTP</a> and <a class="auto-href" href="http://dbpedia.org/resource/SQL" id="link-id0x1ea554c0">SQL</a> listener and can provide the same service. Most access to data goes over the interconnect, except when the data is co-resident in the process which is coordinating the query. The interconnect is Unix domain sockets since all 8 processes are on the same box.</p> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="4" align="center">6 Cluster - Load Rates and Times</th> </tr> <tr> <th align="center">Scale</th> <th align="center">Rate <br /> (quads per second)</th> <th align="center">Load time <br /> (seconds)</th> <th align="center">Checkpoint time <br /> (seconds)</th> </tr> <tr> <th align="center">100 Mt</th> <td align="center"> 119,204 </td> <td align="center"> 749 </td> <td align="center"> 89 </td> </tr> <tr> <th align="center">200 Mt</th> <td align="center"> 121,607 </td> <td align="center"> 1486 </td> <td align="center"> 157 </td> </tr> <tr> <th align="center">1000 Mt</th> <td align="center"> 102,694 </td> <td align="center"> 8737 </td> <td align="center"> 979 </td> </tr> </table> <br /> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="4" align="center">6 Single - Load Rates and Times</th> </tr> <tr> <th align="center">Scale</th> <th align="center">Rate <br /> (quads per second)</th> <th align="center">Load time <br /> (seconds)</th> <th align="center">Checkpoint time <br /> (seconds)</th> </tr> <tr> <th align="center">100 Mt</th> <td align="center"> 74,713 </td> <td align="center"> 1192 </td> <td align="center"> 145 </td> </tr> </table> <p>The load times are systematically better than for 6 Single. This is also not bad compared to the 7 Single vectored load rates of 220 Kt/s or so. We note that loading is a cluster friendly operation, going at a steady 1400+% <a class="auto-href" href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x1c55c8f0">CPU</a> utilization with an aggregate message throughput of 40MB/s. 7 Single is faster because of vectoring at the index level, not because the clusters were hitting communication overheads. 6 Cluster is faster than 6 Single because scale-out in this case diminishes contention, even on a single box.</p> <p>Throughput is as follows:</p> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="3" align="center"> 6 Cluster - Throughput <br /> (QMpH, query mixes per hour) </th> </tr> <tr> <th align="center">Scale</th> <th align="center"> Single User </th> <th align="center"> 16 User </th> </tr> <tr> <th align="center">100 Mt</th> <td align="center"> 7318 </td> <td align="center"> 43120 </td> </tr> <tr> <th align="center">200 Mt</th> <td align="center"> 6222 </td> <td align="center"> 29981 </td> </tr> <tr> <th align="center">1000 Mt</th> <td align="center"> 2526 </td> <td align="center"> 11156 </td> </tr> </table> <br /> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="3" align="center"> 6 Single - Throughput <br /> (QMpH, query mixes per hour) </th> </tr> <tr> <th align="center">Scale</th> <th align="center"> Single User </th> <th align="center"> 16 User </th> </tr> <tr> <th align="center">100 Mt</th> <td align="center"> 7641 </td> <td align="center"> 29433 </td> </tr> <tr> <th align="center">200 Mt</th> <td align="center"> 6017 </td> <td align="center"> 13335 </td> </tr> <tr> <th align="center">1000 Mt</th> <td align="center"> 1770 </td> <td align="center"> 2487 </td> </tr> </table> <p>Below is a snapshot of status during the 6 Cluster 100 Mt run.</p> <blockquote> <code><pre> Cluster 8 nodes, 15 s. 25784 m/s 25682 KB/s 1160% cpu 0% read 740% clw threads 18r 0w 10i buffers 1133459 12 d 4 w 0 pfs cl 1: 10851 m/s 3911 KB/s 597% cpu 0% read 668% clw threads 17r 0w 10i buffers 143992 4 d 0 w 0 pfs cl 2: 2194 m/s 7959 KB/s 107% cpu 0% read 9% clw threads 1r 0w 0i buffers 143616 3 d 2 w 0 pfs cl 3: 2186 m/s 7818 KB/s 107% cpu 0% read 9% clw threads 0r 0w 0i buffers 140787 0 d 0 w 0 pfs cl 4: 2174 m/s 2804 KB/s 77% cpu 0% read 10% clw threads 0r 0w 0i buffers 140654 0 d 2 w 0 pfs cl 5: 2127 m/s 1612 KB/s 71% cpu 0% read 9% clw threads 0r 0w 0i buffers 140949 1 d 0 w 0 pfs cl 6: 2060 m/s 544 KB/s 66% cpu 0% read 10% clw threads 0r 0w 0i buffers 141295 2 d 0 w 0 pfs cl 7: 2072 m/s 517 KB/s 65% cpu 0% read 11% clw threads 0r 0w 0i buffers 141111 1 d 0 w 0 pfs cl 8: 2105 m/s 522 KB/s 66% cpu 0% read 10% clw threads 0r 0w 0i buffers 141055 1 d 0 w 0 pfs </pre> </code> </blockquote> <p>The main meters for cluster execution are the messages-per-second (m/s), the message volume (KB/s), and the total CPU% of the processes. </p> <p>We note that CPU utilization is highly uneven and messages are short, about 1K on the average, compared to about 100K during the load. CPU would be evenly divided between the nodes if each got a share of the HTTP requests. We changed the test driver to round-robin requests between multiple end points. The work does then get evenly divided, but the speed is not affected. Also, this does not improve the message sizes since the workload consists mostly of short lookups. However, with the processes spread over multiple servers, the round-robin would be essential for CPU and especially for interconnect throughput. </p> <p>Then we try 6 Cluster at 1000 Mt. For Single User, we get 1180 m/s, 6955 KB/s, and 173% cpu. For 16 User, this is 6573 m/s, 44366 KB/s, 1470% cpu.</p> <p>This is a lot better than the figures with 6 Single, due to lower contention on the index tree, as discussed in <i><a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1e9a0b58">A Benchmarking Story</a></i>. Also Single User throughput on 6 Cluster outperforms 6 Single, due to the natural parallelism of doing the Q5 joins in parallel in each partition. The larger the scale, the more weight this has in the metric. We see this also in the average message size, i.e., the KB/s throughput is almost double while the messages/s is a bit under a third.</p> <p>The small-scale 6 Cluster run is about even with the 6 Single figure. Looking at the details, we see that the qps for Q1 in 6 Cluster is half of that on 6 Single, whereas the qps for Q5 on 6 Cluster is about double that of the 6 Single. This is as one might expect; longer queries are favored, and single row lookups are penalized.</p> <p>Looking further at the 6 Cluster status we see the cluster wait (<code>clw</code>) to be 740%. For 16 Users, this means that about half of the execution real time is spent waiting for responses from other partitions. A high figure means uneven distribution between partitions; a low figure means even. This is as expected, since many queries are concerned with just one S and its related objects.</p> <p>We will update this section once 7 Cluster is ready. This will implement vectored execution and column store inside the cluster nodes.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1d7894d0">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1e434888">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1f6b5260">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x1dd29460">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1f0d78b8">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1f9a9670">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1c055370">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1dc06cd0">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> Benchmarks, Redux (part 9): BSBM With Cluster <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x18f04db0">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1ee729b8">Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1681" id="link-id0x2aac15e08658">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1682" id="link-id0x2aac15e083d8">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1683" id="link-id0x2aac15cfa158">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1684" id="link-id0x2aac15cfbd68">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2014-08-21T10:45:26.683242-04:00
Benchmarks, Redux (part 8): BSBM Explore and Update
http://www.openlinksw.com:443/blog/vdb/blog/?date=2011-03-09#1674
2011-03-09T17:32:47Z
<p>We will here look at the <i>Explore and Update</i> scenario of <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1c064218">BSBM</a>. This presents us with a novel problem as the specification does not address any aspect of <a class="auto-href" href="http://dbpedia.org/resource/ACID" id="link-id0x1c1852b0">ACID</a>.</p> <p>A transaction benchmark ought to have something to say about this. The <a class="auto-href" href="http://dbpedia.org/page/SPARUL" id="link-id0x1dbca228">SPARUL</a> (also known as <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x1eaa4fd0">SPARQL</a>/<a class="auto-href" href="http://dbpedia.org/page/SPARUL" id="link-id0x1dd12bb0">Update</a>) language does not say anything about transactionality, but I suppose it is in the spirit of the SPARUL protocol to promise atomicity and durability.</p> <p>We begin by running <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1c5f4830">Virtuoso</a> 7 Single, with Single User and 16 User, each at scales of 100 Mt, 200 Mt, and 1000 Mt. The transactionality is default, meaning <code>SERIALIZABLE</code> isolation between <code>INSERTs</code> and <code>DELETEs</code>, and <code>READ COMMITTED</code> isolation between <code>READ</code> and any <code>UPDATE</code> transaction. (Figures for Virtuoso 6 will also be presented here in the near future, as they are the currently shipping production versions.)</p> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="3" align="center"> Virtuoso 7 Single, Full ACID <br /> (QMpH, query mixes per hour) </th> </tr> <tr> <th align="center">Scale</th> <th align="center"> Single User </th> <th align="center"> 16 User </th> </tr> <tr> <th align="center">100 Mt</th> <td align="center"> 9,969 </td> <td align="center"> 65,537 </td> </tr> <tr> <th align="center">200 Mt</th> <td align="center"> 8,646 </td> <td align="center"> 40,527 </td> </tr> <tr> <th align="center">1000 Mt</th> <td align="center"> 5,512 </td> <td align="center"> 17,293 </td> </tr> </table> <br /> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="3" align="center"> Virtuoso 6 Cluster, Full ACID <br /> (QMpH, query mixes per hour) </th> </tr> <tr> <th align="center"> Scale </th> <th align="center"> Single User </th> <th align="center"> 16 User </th> </tr> <tr> <th align="center"> 100 Mt </th> <td align="center"> 5604.520 </td> <td align="center"> 34079.019 </td> </tr> <tr> <th align="center"> 1000 Mt </th> <td align="center"> 2866.616 </td> <td align="center"> 10028.325 </td> </tr> </table> <br /> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="3" align="center"> Virtuoso 6 Single, Full ACID <br /> (QMpH, query mixes per hour) </th> </tr> <tr> <th align="center">Scale</th> <th align="center"> Single User </th> <th align="center"> 16 User </th> </tr> <tr> <th align="center">100 Mt</th> <td align="center"> 7,152 </td> <td align="center"> 21,065 </td> </tr> <tr> <th align="center">200 Mt</th> <td align="center"> 5,862 </td> <td align="center"> 16,895 </td> </tr> <tr> <th align="center">1000 Mt</th> <td align="center"> 1,542 </td> <td align="center"> 4,548 </td> </tr> </table> <p>Each run is preceded by a warm-up of 500 or 300 mixes (the exact number is not material), resulting in a warm <a class="auto-href" href="http://dbpedia.org/resource/Cache" id="link-id0x1d4f13d8">cache</a>; see <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1f8ac510">previous post on read-ahead</a> for details. All runs do 1000 <i>Explore and Update</i> mixes. The initial database is in the state following the <i>Explore</i> only runs.</p> <p>The results are in line with the <i>Explore</i> results. There is a fair amount of variability between consecutive runs; the 16 User run at 1000 Mt varies between 14K and 19K QMpH depending on the measurement. The smaller runs exhibit less variability.</p> <p>In the following we will look at transactions and at how the definition of the workload and reporting could be made complete.</p> <p>Full ACID means serializable semantic of concurrent insert and delete of the same quad. Non-transactional means that on concurrent insert and delete of overlapping sets of quads the result is undefined. Further if one logged such "transactions," the replay would give serialization although the initial execution did not, hence further confusing the issue. Considering the hypothetical use case of an e-commerce information portal, there is little chance of deletes and inserts actually needing serialization. An insert-only workload does not need serializability because an insert cannot fail. If the <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x1ec05c10">data</a> already exists the insert does nothing, if the quad does not previously exist it is created. The same applies to deletes alone. If a delete and insert overlap, serialization would be needed but the semantics implicit in the use case make this improbable.</p> <p>Read-only transactions (i.e., the <i>Explore</i> mix in the <i>Explore and Update</i> scenario) will be run as <code>READ COMMITTED</code>. These do not see uncommitted data and never block for lock wait. The reads may not be repeatable.</p> <p>Our first point of call is to determine the cost of ACID. We run 1000 mixes of <i>Explore and Update</i> at 1000 Mt. The throughput is 19214 after a warm-up of 500 mixes. This is pretty good in comparison with the diverse read-only results at this scale.</p> <p>We look at the pertinent statistics:</p> <p> <code></code> </p> <pre> SELECT TOP 5 * FROM sys_l_stat ORDER BY waits DESC; </pre> <blockquote> <code><pre> KEY_TABLE INDEX_NAME LOCKS WAITS WAIT_PCT DEADLOCKS LOCK_ESC WAIT_MSECS =============== ============= ====== ===== ======== ========= ======== ========== DB.DBA.<a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x180837c8">RDF</a>_QUAD RDF_QUAD_POGS 179205 934 0 0 0 35164 DB.DBA.RDF_IRI RDF_IRI 20752 217 1 0 0 16445 DB.DBA.RDF_QUAD RDF_QUAD_SP 9244 3 0 0 0 235 </pre> </code> </blockquote> <p>We see 934 waits with a total duration of 35 seconds on the index with the most contention. The run was 187 seconds, real time. The lock wait time is not real time since this is the total elapsed wait time summed over all threads. The lock wait frequency is a little over one per query mix, meaning a little over one per five locking transactions. </p> <p>We note that we do not get deadlocks since all inserts and deletes are in ascending key order due to vectoring. This guarantees the absence of deadlocks for single insert transactions, as long as the transaction stays within the vector size. This is always the case since the inserts are a few hundred triples at the maximum. The waits concentrate on POGS, because this is a bitmap index where the locking resolution is less than a row, and the values do not correlate with insert order. The locking behavior could be better with the column store, where we would have row level locking also for this index. This is to be seen. The column store would otherwise tend to have higher cost per random insert.</p> <p>Considering these results it does not seem crucial to "drop ACID," though doing so would save <i>some</i> time. We will now run measurements for all scales with 16 Users and ACID. </p> <p>Let us now see what the benchmark writes:</p> <p> <code></code> </p> <pre> SELECT TOP 10 * FROM sys_d_stat ORDER BY n_dirty DESC; </pre> <blockquote> <code><pre> KEY_TABLE INDEX_NAME TOUCHES READS READ_PCT N_DIRTY N_BUFFERS =========================== ============================ ========= ======= ======== ======= ========= DB.DBA.RDF_QUAD RDF_QUAD_POGS 763846891 237436 0 58040 228606 DB.DBA.RDF_QUAD RDF_QUAD 213282706 1991836 0 30226 1940280 DB.DBA.RDF_OBJ RO_VAL 15474 17837 115 13438 17431 DB.DBA.RO_START RO_START 10573 11195 105 10228 11227 DB.DBA.RDF_IRI RDF_IRI 61902 125711 203 7705 121300 DB.DBA.RDF_OBJ RDF_OBJ 23809053 3205963 13 636 3072517 DB.DBA.RDF_IRI DB_DBA_RDF_IRI_UNQC_RI_ID 3237687 504486 15 340 488797 DB.DBA.RDF_QUAD RDF_QUAD_SP 89995 70446 78 99 68340 DB.DBA.RDF_QUAD RDF_QUAD_OP 19440 47541 244 66 45583 DB.DBA.VTLOG_DB_DBA_RDF_OBJ VTLOG_DB_DBA_RDF_OBJ 3014 1 0 11 11 DB.DBA.RDF_QUAD RDF_QUAD_GS 1261 801 63 10 751 DB.DBA.RDF_PREFIX RDF_PREFIX 14 168 1120 1 153 DB.DBA.RDF_PREFIX DB_DBA_RDF_PREFIX_UNQC_RP_ID 1807 200 11 1 200 </pre> </code> </blockquote> <p>The most dirty pages are on the <code>POGS</code> index, which is reasonable; values are spread out at random. After this we have the <code>PSOG</code> index, likely because of random deletes. New IRIs tend to get consecutive numbers and do not make many dirty pages. Literals come next, with the index from leading string or hash of the literal to id leading, as one would expect, again because of values being distributed at random. After this come IRIs. The distribution of updates is generally as one would expect.</p> <p align="center">* * *</p> <p>Going back to BSBM, at least the following aspects of the benchmark have to be further specified:</p> <ul> <li> <p> <b>Disclosure of ACID properties.</b> If the benchmark required full ACID many would not run this at all. Besides full ACID is not necessarily an absolute requirement based on the hypothetical usage scenario of the benchmark. However, when publishing numbers the guarantees that go with the numbers must be made explicit. This includes logging, checkpoint frequency or equivalent etc.</p> </li> <li> <p> <b>Steady state.</b> The working set of the <i>Update</i> mix is different from that of the <i>Explore</i> mixes. This touches more indices than <i>Explore</i>. The <i>Explore</i> warm-up is in part good but does not represent steady state.</p> </li> <li> <p> <b>Checkpoint and sustained throughput.</b> Benchmarks involving update generally have rules for checkpointing the state and for sustained throughput. In specific, the throughput of an update benchmark cannot rely on never flushing to persistent storage. Even bulk load must be timed with a checkpoint guaranteeing durability at the end. A steady update stream should be timed with a test interval of sufficient length involving a few checkpoints; for example, a minimum duration of 30 minutes with no less than 3 completed checkpoints in the interval with at least 9 minutes between the end of one and the start of the next. Not all DBMSs work with logs and checkpoints, but if an alternate scheme is used then this needs to be described.</p> </li> <li> <p> <b>Memory and warm-up issues.</b>We have seen the test data generator run out of memory when trying to generate update streams of meaningful length. Also the test driver should allow running updates in timed and non-timed mode (warm-up).</p> </li> </ul> <p>With an update benchmark, many more things need to be defined, and the set-up becomes more system specific, than with a read-only workload. We will address these shortcomings in the measurement rules proposal to come. Especially with update workloads, the vendors need to provide tuning expertise; however, this will not happen if the benchmark does not properly set the expectations. If benchmarks serve as a catalyst for clearly defining how things are to be set up, then they will have served the end user.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1de61db8">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1f9f96f8">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1f89eeb0">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x1ad83f30">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1de62178">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1b2ec018">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1ae6f028">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> Benchmarks, Redux (part 8): BSBM Explore and Update <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x132605c0">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x1a9871b0">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1baa20f8">Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1e25a840">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1b53db20">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1e7ce520">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1b18f400">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2011-03-15T17:18:32-04:00
Benchmarks, Redux (part 8): BSBM Explore and Update
http://www.openlinksw.com:443/weblog/oerling/?date=2011-03-09#1673
2011-03-09T17:32:47Z
<p>We will here look at the <i>Explore and Update</i> scenario of <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x2a621a00">BSBM</a>. This presents us with a novel problem as the specification does not address any aspect of <a class="auto-href" href="http://dbpedia.org/resource/ACID" id="link-id0x2a2d4310">ACID</a>.</p> <p>A transaction benchmark ought to have something to say about this. The <a class="auto-href" href="http://dbpedia.org/page/SPARUL" id="link-id0x27cf4478">SPARUL</a> (also known as <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x29bb7f80">SPARQL</a>/<a class="auto-href" href="http://dbpedia.org/page/SPARUL" id="link-id0x2978e570">Update</a>) language does not say anything about transactionality, but I suppose it is in the spirit of the SPARUL protocol to promise atomicity and durability.</p> <p>We begin by running <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x27a2b9f0">Virtuoso</a> 7 Single, with Single User and 16 User, each at scales of 100 Mt, 200 Mt, and 1000 Mt. The transactionality is default, meaning <code>SERIALIZABLE</code> isolation between <code>INSERTs</code> and <code>DELETEs</code>, and <code>READ COMMITTED</code> isolation between <code>READ</code> and any <code>UPDATE</code> transaction. (Figures for Virtuoso 6 will also be presented here in the near future, as they are the currently shipping production versions.)</p> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="3" align="center"> Virtuoso 7 Single, Full ACID <br /> (QMpH, query mixes per hour) </th> </tr> <tr> <th align="center">Scale</th> <th align="center"> Single User </th> <th align="center"> 16 User </th> </tr> <tr> <th align="center">100 Mt</th> <td align="center"> 9,969 </td> <td align="center"> 65,537 </td> </tr> <tr> <th align="center">200 Mt</th> <td align="center"> 8,646 </td> <td align="center"> 40,527 </td> </tr> <tr> <th align="center">1000 Mt</th> <td align="center"> 5,512 </td> <td align="center"> 17,293 </td> </tr> </table> <br /> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="3" align="center"> Virtuoso 6 Cluster, Full ACID <br /> (QMpH, query mixes per hour) </th> </tr> <tr> <th align="center"> Scale </th> <th align="center"> Single User </th> <th align="center"> 16 User </th> </tr> <tr> <th align="center"> 100 Mt </th> <td align="center"> 5604.520 </td> <td align="center"> 34079.019 </td> </tr> <tr> <th align="center"> 1000 Mt </th> <td align="center"> 2866.616 </td> <td align="center"> 10028.325 </td> </tr> </table> <br /> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="3" align="center"> Virtuoso 6 Single, Full ACID <br /> (QMpH, query mixes per hour) </th> </tr> <tr> <th align="center">Scale</th> <th align="center"> Single User </th> <th align="center"> 16 User </th> </tr> <tr> <th align="center">100 Mt</th> <td align="center"> 7,152 </td> <td align="center"> 21,065 </td> </tr> <tr> <th align="center">200 Mt</th> <td align="center"> 5,862 </td> <td align="center"> 16,895 </td> </tr> <tr> <th align="center">1000 Mt</th> <td align="center"> 1,542 </td> <td align="center"> 4,548 </td> </tr> </table> <p>Each run is preceded by a warm-up of 500 or 300 mixes (the exact number is not material), resulting in a warm <a class="auto-href" href="http://dbpedia.org/resource/Cache" id="link-id0x298139d0">cache</a>; see <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1f8ac510">previous post on read-ahead</a> for details. All runs do 1000 <i>Explore and Update</i> mixes. The initial database is in the state following the <i>Explore</i> only runs.</p> <p>The results are in line with the <i>Explore</i> results. There is a fair amount of variability between consecutive runs; the 16 User run at 1000 Mt varies between 14K and 19K QMpH depending on the measurement. The smaller runs exhibit less variability.</p> <p>In the following we will look at transactions and at how the definition of the workload and reporting could be made complete.</p> <p>Full ACID means serializable semantic of concurrent insert and delete of the same quad. Non-transactional means that on concurrent insert and delete of overlapping sets of quads the result is undefined. Further if one logged such "transactions," the replay would give serialization although the initial execution did not, hence further confusing the issue. Considering the hypothetical use case of an e-commerce information portal, there is little chance of deletes and inserts actually needing serialization. An insert-only workload does not need serializability because an insert cannot fail. If the <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x29a2f220">data</a> already exists the insert does nothing, if the quad does not previously exist it is created. The same applies to deletes alone. If a delete and insert overlap, serialization would be needed but the semantics implicit in the use case make this improbable.</p> <p>Read-only transactions (i.e., the <i>Explore</i> mix in the <i>Explore and Update</i> scenario) will be run as <code>READ COMMITTED</code>. These do not see uncommitted data and never block for lock wait. The reads may not be repeatable.</p> <p>Our first point of call is to determine the cost of ACID. We run 1000 mixes of <i>Explore and Update</i> at 1000 Mt. The throughput is 19214 after a warm-up of 500 mixes. This is pretty good in comparison with the diverse read-only results at this scale.</p> <p>We look at the pertinent statistics:</p> <p> <code></code> </p> <pre> SELECT TOP 5 * FROM sys_l_stat ORDER BY waits DESC; </pre> <blockquote> <code><pre> KEY_TABLE INDEX_NAME LOCKS WAITS WAIT_PCT DEADLOCKS LOCK_ESC WAIT_MSECS =============== ============= ====== ===== ======== ========= ======== ========== DB.DBA.<a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x299432f0">RDF</a>_QUAD RDF_QUAD_POGS 179205 934 0 0 0 35164 DB.DBA.RDF_IRI RDF_IRI 20752 217 1 0 0 16445 DB.DBA.RDF_QUAD RDF_QUAD_SP 9244 3 0 0 0 235 </pre> </code> </blockquote> <p>We see 934 waits with a total duration of 35 seconds on the index with the most contention. The run was 187 seconds, real time. The lock wait time is not real time since this is the total elapsed wait time summed over all threads. The lock wait frequency is a little over one per query mix, meaning a little over one per five locking transactions. </p> <p>We note that we do not get deadlocks since all inserts and deletes are in ascending key order due to vectoring. This guarantees the absence of deadlocks for single insert transactions, as long as the transaction stays within the vector size. This is always the case since the inserts are a few hundred triples at the maximum. The waits concentrate on POGS, because this is a bitmap index where the locking resolution is less than a row, and the values do not correlate with insert order. The locking behavior could be better with the column store, where we would have row level locking also for this index. This is to be seen. The column store would otherwise tend to have higher cost per random insert.</p> <p>Considering these results it does not seem crucial to "drop ACID," though doing so would save <i>some</i> time. We will now run measurements for all scales with 16 Users and ACID. </p> <p>Let us now see what the benchmark writes:</p> <p> <code></code> </p> <pre> SELECT TOP 10 * FROM sys_d_stat ORDER BY n_dirty DESC; </pre> <blockquote> <code><pre> KEY_TABLE INDEX_NAME TOUCHES READS READ_PCT N_DIRTY N_BUFFERS =========================== ============================ ========= ======= ======== ======= ========= DB.DBA.RDF_QUAD RDF_QUAD_POGS 763846891 237436 0 58040 228606 DB.DBA.RDF_QUAD RDF_QUAD 213282706 1991836 0 30226 1940280 DB.DBA.RDF_OBJ RO_VAL 15474 17837 115 13438 17431 DB.DBA.RO_START RO_START 10573 11195 105 10228 11227 DB.DBA.RDF_IRI RDF_IRI 61902 125711 203 7705 121300 DB.DBA.RDF_OBJ RDF_OBJ 23809053 3205963 13 636 3072517 DB.DBA.RDF_IRI DB_DBA_RDF_IRI_UNQC_RI_ID 3237687 504486 15 340 488797 DB.DBA.RDF_QUAD RDF_QUAD_SP 89995 70446 78 99 68340 DB.DBA.RDF_QUAD RDF_QUAD_OP 19440 47541 244 66 45583 DB.DBA.VTLOG_DB_DBA_RDF_OBJ VTLOG_DB_DBA_RDF_OBJ 3014 1 0 11 11 DB.DBA.RDF_QUAD RDF_QUAD_GS 1261 801 63 10 751 DB.DBA.RDF_PREFIX RDF_PREFIX 14 168 1120 1 153 DB.DBA.RDF_PREFIX DB_DBA_RDF_PREFIX_UNQC_RP_ID 1807 200 11 1 200 </pre> </code> </blockquote> <p>The most dirty pages are on the <code>POGS</code> index, which is reasonable; values are spread out at random. After this we have the <code>PSOG</code> index, likely because of random deletes. New IRIs tend to get consecutive numbers and do not make many dirty pages. Literals come next, with the index from leading string or hash of the literal to id leading, as one would expect, again because of values being distributed at random. After this come IRIs. The distribution of updates is generally as one would expect.</p> <p align="center">* * *</p> <p>Going back to BSBM, at least the following aspects of the benchmark have to be further specified:</p> <ul> <li> <p> <b>Disclosure of ACID properties.</b> If the benchmark required full ACID many would not run this at all. Besides full ACID is not necessarily an absolute requirement based on the hypothetical usage scenario of the benchmark. However, when publishing numbers the guarantees that go with the numbers must be made explicit. This includes logging, checkpoint frequency or equivalent etc.</p> </li> <li> <p> <b>Steady state.</b> The working set of the <i>Update</i> mix is different from that of the <i>Explore</i> mixes. This touches more indices than <i>Explore</i>. The <i>Explore</i> warm-up is in part good but does not represent steady state.</p> </li> <li> <p> <b>Checkpoint and sustained throughput.</b> Benchmarks involving update generally have rules for checkpointing the state and for sustained throughput. In specific, the throughput of an update benchmark cannot rely on never flushing to persistent storage. Even bulk load must be timed with a checkpoint guaranteeing durability at the end. A steady update stream should be timed with a test interval of sufficient length involving a few checkpoints; for example, a minimum duration of 30 minutes with no less than 3 completed checkpoints in the interval with at least 9 minutes between the end of one and the start of the next. Not all DBMSs work with logs and checkpoints, but if an alternate scheme is used then this needs to be described.</p> </li> <li> <p> <b>Memory and warm-up issues.</b>We have seen the test data generator run out of memory when trying to generate update streams of meaningful length. Also the test driver should allow running updates in timed and non-timed mode (warm-up).</p> </li> </ul> <p>With an update benchmark, many more things need to be defined, and the set-up becomes more system specific, than with a read-only workload. We will address these shortcomings in the measurement rules proposal to come. Especially with update workloads, the vendors need to provide tuning expertise; however, this will not happen if the benchmark does not properly set the expectations. If benchmarks serve as a catalyst for clearly defining how things are to be set up, then they will have served the end user.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1de61db8">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1f9f96f8">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1f89eeb0">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x1ad83f30">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1de62178">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1b2ec018">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1ae6f028">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> Benchmarks, Redux (part 8): BSBM Explore and Update <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x132605c0">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x1a9871b0">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1baa20f8">Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1681" id="link-id0x2aabe7f1a968">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1682" id="link-id0x2aabe628feb8">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1683" id="link-id0x2aabe7e4d5d8">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1684" id="link-id0x2aac0214de48">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2014-08-21T10:45:37.814559-04:00
Benchmarks, Redux (part 7): What Does BSBM Explore Measure?
http://www.openlinksw.com:443/blog/vdb/blog/?date=2011-03-07#1672
2011-03-07T23:39:22Z
<p>We will here analyze what the <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1db49f28">BSBM</a> Explore workload does. This is necessary in order to compare benchmark results at different scales. Historically, BSBM had a Query 6 whose share of the metric approached 100% as scale increased. The present mix does not have this query, but different queries still have different relative importance at different scales.</p> <p>We will here look at database-running statistics for BSBM at different scales. Finally, we look at <a class="auto-href" href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x1f150460">CPU</a> profiles.</p> <p>But first, let us see what BSBM reads in general. The system is in steady state after around 1500 query mixes; after this the working set does not shift much. After several thousand query mixes, we have:</p> <p> <code>SELECT TOP 10 * FROM sys_d_stat ORDER BY reads DESC;</code> </p> <blockquote> <code><pre> KEY_TABLE INDEX_NAME TOUCHES READS READ_PCT N_DIRTY N_BUFFERS ================= ============================ ========== ======= ======== ======= ========= DB.DBA.<a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1ddb0b50">RDF</a>_OBJ RDF_OBJ 114105938 3302150 2 0 3171275 DB.DBA.RDF_QUAD RDF_QUAD 977426773 2041156 0 0 1970712 DB.DBA.RDF_IRI DB_DBA_RDF_IRI_UNQC_RI_ID 8250414 509239 6 15 491631 DB.DBA.RDF_QUAD RDF_QUAD_POGS 3677233812 183860 0 0 175386 DB.DBA.RDF_IRI RDF_IRI 32 99710 302151 5 95353 DB.DBA.RDF_QUAD RDF_QUAD_OP 30597 51593 168 0 48941 DB.DBA.RDF_QUAD RDF_QUAD_SP 265474 47210 17 0 46078 DB.DBA.RDF_PREFIX DB_DBA_RDF_PREFIX_UNQC_RP_ID 6020 212 3 0 212 DB.DBA.RDF_PREFIX RDF_PREFIX 0 167 16700 0 157 </pre> </code> </blockquote> <p>The first column is the table, then the index, then the number of times a row was found. The fourth number is the count of disk pages read. The last number is the count of 8K buffer pool pages in use for caching pages of the index in question. Note that the index is clustered, i.e., there is no table <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x1d4f9808">data</a> structure separate from the index. Most of the reads are for strings or RDF literals. After this comes the <code>PSOG</code> index for getting a property value given the subject. After this, but much lower, we have lookups of IRI strings given the ID. The index from object value to subject is used the most but the number of pages is small; only a few properties seem to be concerned. The rest is minimal in comparison.</p> <p>Now let us reset the counts and see what the steady state I/O profile is.</p> <p> <code>SELECT key_stat (key_table, name_part (key_name, 2), 'reset') FROM sys_keys WHERE key_migrate_to IS NULL;</code> </p> <p> <code>SELECT TOP 10 * FROM sys_d_stat ORDER BY reads DESC;</code> </p> <blockquote> <code><pre> KEY_TABLE INDEX_NAME TOUCHES READS READ_PCT N_DIRTY N_BUFFERS ================= ============================ ========== ======= ======== ======= ========= DB.DBA.RDF_OBJ RDF_OBJ 30155789 79659 0 0 3191391 DB.DBA.RDF_QUAD RDF_QUAD 259008064 8904 0 0 1948707 DB.DBA.RDF_QUAD RDF_QUAD_SP 68002 7730 11 0 53360 DB.DBA.RDF_IRI RDF_IRI 12 5415 41653 6 98804 DB.DBA.RDF_QUAD RDF_QUAD_POGS 975147136 1597 0 0 173459 DB.DBA.RDF_IRI DB_DBA_RDF_IRI_UNQC_RI_ID 2213525 1286 0 17 485093 DB.DBA.RDF_QUAD RDF_QUAD_OP 7999 904 11 0 48568 DB.DBA.RDF_PREFIX DB_DBA_RDF_PREFIX_UNQC_RP_ID 1494 1 0 0 213 </pre> </code> </blockquote> <p>Literal strings dominate. The <code>SP</code> index is used only for situations where the <code>P</code> is not specified, i.e., the <code>DESCRIBE</code> query. Based on this, I/O seems to be attributable mostly to this. The first <code>RDF_IRI</code> represents translations from string to IRI id; the second represents translations from IRI id to string. The touch count for the first <code>RDF_IRI</code> is not properly recorded, hence the miss % is out of line. We see <code>SP</code> missing the <a class="auto-href" href="http://dbpedia.org/resource/Cache" id="link-id0x17d2e670">cache</a> the most since its use is infrequent in the mix.</p> <p>We will next look at query processing statistics. For this we introduce a new meter.</p> <p>The <code>db_activity</code> <a class="auto-href" href="http://dbpedia.org/resource/SQL" id="link-id0x1d4915b8">SQL</a> function provides a session-by-session cumulative statistic of activity. The fields are: </p> <ul> <li> <b><code>rnd</code> </b> - Count of <i>random index lookups</i>. Each first row of a select or insert counts as one, regardless of whether something was found.</li> <li> <b><code>seq</code> </b> - Count of <i>sequential rows</i>. Every move to next row on a cursor counts as 1, regardless of whether conditions match.</li> <li> <b><code>same seg</code> </b> - For column store only; counts how many times the next row in a vectored join using an index falls in the <i>same segment</i> as the previous random access. A segment is the stretch of rows between entries in the sparse top level index on the column projection.</li> <li> <b><code>same pg</code> </b> - Counts how many times a vectored index join finds the next match on the <i>same page</i> as the previous one.</li> <li> <b><code>same par</code> </b> - Counts how many times the next lookup in a vectored index join falls on a different page than the previous but still under the <i>same parent</i>.</li> <li> <b><code>disk</code> </b> - Counts how many <i>disk reads</i> were made, including any speculative reads initiated.</li> <li> <b><code>spec disk</code> </b> - Counts <i>speculative disk reads</i>.</li> <li> <b><code>messages</code> </b> - Counts <i>cluster interconnect messages</i> </li> <li> <b><code>B (KB, MB, GB)</code> </b> - is the <i>total length</i> of the cluster interconnect messages.</li> <li> <b><code>fork</code> </b> - Counts how many times a <i>thread was forked (started)</i> for query parallelization.</li> </ul> <p>The numbers are given with 4 significant digits and a scale suffix. G is 10^9 (1,000,000,000); M is 10^6 (1,000,000), K is 10^3 (1,000).</p> <p>We run 2000 query mixes with 16 Users. The special <code><a class="auto-href" href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1bf7f318">http</a></code> account keeps a cumulative account of all activity on web server threads.</p> <blockquote> <p> <code>SELECT db_activity (2, 'http');</code> </p> <p> <code>1.674G rnd 3.223G seq 0 same seg 1.286G same pg 314.8M same par 6.186M disk 6.461M spec disk 0B / 0 messages 298.6K fork</code> </p> </blockquote> <p>We see that random access dominates. The <code>seq</code> number is about twice the <code>rnd</code> number, meaning that the average random lookup gets two rows. Getting a row at random obviously takes more time than getting the next row. Since the index used is row-wise, the <code>same seg</code> is 0; the <code>same pg</code> indicates that 77% of the random accesses fall on the same page as the previous random access; most of the remaining random accesses fall under the same parent as the previous one.</p> <p>There are more speculative reads than disk reads which is an artifact of counting some concurrently speculated reads twice. This does indicate that speculative reads dominate. This is because a large part of the run was in the warm-up state with aggressive speculative reading. We reset the counts and run another 2000 mixes.</p> <p>Now let us look at the same reading after 2000 mixes, 16 user at 100Mt.</p> <blockquote> <p> <code>234.3M rnd 420.5M seq 0 same seg 188.8M same pg 29.09M same par 808.9K disk 919.9K spec disk 0B / 0 messages 76K fork</code> </p> </blockquote> <p>We note that the ratios between the random and sequential and same page/parent counts are about the same. The sequential number looks to be even a bit smaller in proportion. The count of random accesses for the 100Mt run is 14% of the count for the 1000Mt run. The count of query parallelization threads is also much lower since it is worthwhile to schedule a new thread only if there are at least a few thousand operations to perform on it. The precise criterion for making a thread is that according to the cost model guess, the thread must have at least 5ms worth of work.</p> <p>We note that the 100 Mt throughput is a little over three-times that of the 1000 Mt throughput, as reported before. We might justifiably ask why the 100 Mt run is not seven-times faster instead, for this much less work. </p> <p>We note that for one-off random access, it makes no real difference whether the tree has 100 M or 1000 M rows; this translates to roughly 27 vs 30 comparisons, so the depth of the tree is not a factor <i>per se</i>. Besides, vectoring makes the tree often look only one or two levels deep, so the total row count matters even less there.</p> <p>To elucidate this last question, we look at the CPU profiles. We take an <a href="http://oprofile.sourceforge.net/about/" id="link-id0x1efb3360">oprofile</a> of 100 Single User mixes at both scales.</p> For 100 Mt: <blockquote> <code><pre> 61161 10.1723 cmpf_iri64n_iri64n_anyn_gt_lt 31321 5.2093 box_equal 19027 3.1646 sqlo_parse_tree_has_node 15905 2.6453 dk_alloc 15647 2.6024 itc_next_set_neq 12702 2.1126 itc_vec_split_search 12487 2.0768 itc_dive_transit 11450 1.9044 itc_bm_vec_row_check 10646 1.7706 itc_page_rcf_search 9223 1.5340 id_hash_get 9215 1.5326 gen_qsort 8867 1.4748 sqlo_key_part_best 8807 1.4648 itc_param_cmp 8062 1.3409 cmpf_iri64n_iri64n 6820 1.1343 sqlo_in_list 6005 0.9987 dc_iri_id_cmp 5905 0.9821 dk_free_tree 5801 0.9648 box_hash 5509 0.9163 dks_esc_write 5444 0.9054 sql_tree_hash_1 </pre> </code> </blockquote> For 1000 Mt <blockquote> <code><pre> 754331 31.4149 cmpf_iri64n_iri64n_anyn_gt_lt 146165 6.0872 itc_vec_split_search 144795 6.0301 itc_next_set_neq 131671 5.4836 itc_dive_transit 110870 4.6173 itc_page_rcf_search 66780 2.7811 gen_qsort 66434 2.7667 itc_param_cmp 58450 2.4342 itc_bm_vec_row_check 55213 2.2994 dk_alloc 47793 1.9904 cmpf_iri64n_iri64n 44277 1.8440 dc_iri_id_cmp 39489 1.6446 cmpf_int64n 36880 1.5359 dc_append_bytes 36601 1.5243 dv_compare 31286 1.3029 dc_any_value_prefetch 25457 1.0602 itc_next_set 20852 0.8684 box_equal 19895 0.8285 dk_free_tree 19698 0.8203 itc_page_insert_search 19367 0.8066 dc_copy </pre> </code> </blockquote> <p>The top function in both is the compare for an equality of two leading IRIs and a range for the trailing any. This corresponds to the range check in Q5. At the larger scale this is three times more important. At the smaller scale, the share of query <a class="auto-href" href="http://dbpedia.org/resource/Program_optimization" id="link-id0x1bf8ca38">optimization</a> is about 6.5 times greater. The top function in this category is <code>box_equal</code> with 5.2% vs 0.87%. The remaining SQL compiler functions are all in proportion to this, totaling 14.3% of the 100 Mt top-20 profile.</p> <p>From this sample it appears ten times more scale is seven times more database operations. This is not taken into account in the metric. Query compilation is significant at the small end, and no longer significant at 1000 Mt. From these numbers, we could say that <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1be12350">Virtuoso</a> is about two times more efficient in terms of database operation throughput at 1000 Mt than at 100 Mt.</p> <p>We may conclude that different BSBM scales measure different things. The <a class="auto-href" href="http://www.tpc.org/" id="link-id0x17eb98a0">TPC</a> workloads are relatively better in that they have a balance between metric components that stay relatively constant across a large range of scales.</p> <p>This is not necessarily something that should be fixed in the BSBM Explore mix. We must however take these factors better into account in developing the BI mix.</p> <p>Let us also remember that BSBM Explore is a relational workload. Future posts in this series will outline how we propose to make RDF-friendlier benchmarks. </p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1a9bcff8">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1d3e5470">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1de94770">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x1ea66470">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1f1118d8">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1d1c0cd8">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> Benchmarks, Redux (part 7): What Does BSBM Explore Measure? <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1aaf4180">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1a957610">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x127e75c8">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1c9400f0">Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1d2c1d68">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1ea1fb40">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1c073a10">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1c5541e8">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2011-03-14T17:57:20-04:00
Benchmarks, Redux (part 7): What Does BSBM Explore Measure?
http://www.openlinksw.com:443/weblog/oerling/?date=2011-03-07#1671
2011-03-07T23:39:22Z
<p>We will here analyze what the <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1d1a4e30">BSBM</a> Explore workload does. This is necessary in order to compare benchmark results at different scales. Historically, BSBM had a Query 6 whose share of the metric approached 100% as scale increased. The present mix does not have this query, but different queries still have different relative importance at different scales.</p> <p>We will here look at database-running statistics for BSBM at different scales. Finally, we look at <a class="auto-href" href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x1c1df888">CPU</a> profiles.</p> <p>But first, let us see what BSBM reads in general. The system is in steady state after around 1500 query mixes; after this the working set does not shift much. After several thousand query mixes, we have:</p> <p> <code>SELECT TOP 10 * FROM sys_d_stat ORDER BY reads DESC;</code> </p> <blockquote> <code><pre> KEY_TABLE INDEX_NAME TOUCHES READS READ_PCT N_DIRTY N_BUFFERS ================= ============================ ========== ======= ======== ======= ========= DB.DBA.<a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1c585f20">RDF</a>_OBJ RDF_OBJ 114105938 3302150 2 0 3171275 DB.DBA.RDF_QUAD RDF_QUAD 977426773 2041156 0 0 1970712 DB.DBA.RDF_IRI DB_DBA_RDF_IRI_UNQC_RI_ID 8250414 509239 6 15 491631 DB.DBA.RDF_QUAD RDF_QUAD_POGS 3677233812 183860 0 0 175386 DB.DBA.RDF_IRI RDF_IRI 32 99710 302151 5 95353 DB.DBA.RDF_QUAD RDF_QUAD_OP 30597 51593 168 0 48941 DB.DBA.RDF_QUAD RDF_QUAD_SP 265474 47210 17 0 46078 DB.DBA.RDF_PREFIX DB_DBA_RDF_PREFIX_UNQC_RP_ID 6020 212 3 0 212 DB.DBA.RDF_PREFIX RDF_PREFIX 0 167 16700 0 157 </pre> </code> </blockquote> <p>The first column is the table, then the index, then the number of times a row was found. The fourth number is the count of disk pages read. The last number is the count of 8K buffer pool pages in use for caching pages of the index in question. Note that the index is clustered, i.e., there is no table <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x1ac97430">data</a> structure separate from the index. Most of the reads are for strings or RDF literals. After this comes the <code>PSOG</code> index for getting a property value given the subject. After this, but much lower, we have lookups of IRI strings given the ID. The index from object value to subject is used the most but the number of pages is small; only a few properties seem to be concerned. The rest is minimal in comparison.</p> <p>Now let us reset the counts and see what the steady state I/O profile is.</p> <p> <code>SELECT key_stat (key_table, name_part (key_name, 2), 'reset') FROM sys_keys WHERE key_migrate_to IS NULL;</code> </p> <p> <code>SELECT TOP 10 * FROM sys_d_stat ORDER BY reads DESC;</code> </p> <blockquote> <code><pre> KEY_TABLE INDEX_NAME TOUCHES READS READ_PCT N_DIRTY N_BUFFERS ================= ============================ ========== ======= ======== ======= ========= DB.DBA.RDF_OBJ RDF_OBJ 30155789 79659 0 0 3191391 DB.DBA.RDF_QUAD RDF_QUAD 259008064 8904 0 0 1948707 DB.DBA.RDF_QUAD RDF_QUAD_SP 68002 7730 11 0 53360 DB.DBA.RDF_IRI RDF_IRI 12 5415 41653 6 98804 DB.DBA.RDF_QUAD RDF_QUAD_POGS 975147136 1597 0 0 173459 DB.DBA.RDF_IRI DB_DBA_RDF_IRI_UNQC_RI_ID 2213525 1286 0 17 485093 DB.DBA.RDF_QUAD RDF_QUAD_OP 7999 904 11 0 48568 DB.DBA.RDF_PREFIX DB_DBA_RDF_PREFIX_UNQC_RP_ID 1494 1 0 0 213 </pre> </code> </blockquote> <p>Literal strings dominate. The <code>SP</code> index is used only for situations where the <code>P</code> is not specified, i.e., the <code>DESCRIBE</code> query. Based on this, I/O seems to be attributable mostly to this. The first <code>RDF_IRI</code> represents translations from string to IRI id; the second represents translations from IRI id to string. The touch count for the first <code>RDF_IRI</code> is not properly recorded, hence the miss % is out of line. We see <code>SP</code> missing the <a class="auto-href" href="http://dbpedia.org/resource/Cache" id="link-id0x11b80700">cache</a> the most since its use is infrequent in the mix.</p> <p>We will next look at query processing statistics. For this we introduce a new meter.</p> <p>The <code>db_activity</code> <a class="auto-href" href="http://dbpedia.org/resource/SQL" id="link-id0x1ddd4bf0">SQL</a> function provides a session-by-session cumulative statistic of activity. The fields are: </p> <ul> <li> <b><code>rnd</code> </b> - Count of <i>random index lookups</i>. Each first row of a select or insert counts as one, regardless of whether something was found.</li> <li> <b><code>seq</code> </b> - Count of <i>sequential rows</i>. Every move to next row on a cursor counts as 1, regardless of whether conditions match.</li> <li> <b><code>same seg</code> </b> - For column store only; counts how many times the next row in a vectored join using an index falls in the <i>same segment</i> as the previous random access. A segment is the stretch of rows between entries in the sparse top level index on the column projection.</li> <li> <b><code>same pg</code> </b> - Counts how many times a vectored index join finds the next match on the <i>same page</i> as the previous one.</li> <li> <b><code>same par</code> </b> - Counts how many times the next lookup in a vectored index join falls on a different page than the previous but still under the <i>same parent</i>.</li> <li> <b><code>disk</code> </b> - Counts how many <i>disk reads</i> were made, including any speculative reads initiated.</li> <li> <b><code>spec disk</code> </b> - Counts <i>speculative disk reads</i>.</li> <li> <b><code>messages</code> </b> - Counts <i>cluster interconnect messages</i> </li> <li> <b><code>B (KB, MB, GB)</code> </b> - is the <i>total length</i> of the cluster interconnect messages.</li> <li> <b><code>fork</code> </b> - Counts how many times a <i>thread was forked (started)</i> for query parallelization.</li> </ul> <p>The numbers are given with 4 significant digits and a scale suffix. G is 10^9 (1,000,000,000); M is 10^6 (1,000,000), K is 10^3 (1,000).</p> <p>We run 2000 query mixes with 16 Users. The special <code><a class="auto-href" href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1ebdef18">http</a></code> account keeps a cumulative account of all activity on web server threads.</p> <blockquote> <p> <code>SELECT db_activity (2, 'http');</code> </p> <p> <code>1.674G rnd 3.223G seq 0 same seg 1.286G same pg 314.8M same par 6.186M disk 6.461M spec disk 0B / 0 messages 298.6K fork</code> </p> </blockquote> <p>We see that random access dominates. The <code>seq</code> number is about twice the <code>rnd</code> number, meaning that the average random lookup gets two rows. Getting a row at random obviously takes more time than getting the next row. Since the index used is row-wise, the <code>same seg</code> is 0; the <code>same pg</code> indicates that 77% of the random accesses fall on the same page as the previous random access; most of the remaining random accesses fall under the same parent as the previous one.</p> <p>There are more speculative reads than disk reads which is an artifact of counting some concurrently speculated reads twice. This does indicate that speculative reads dominate. This is because a large part of the run was in the warm-up state with aggressive speculative reading. We reset the counts and run another 2000 mixes.</p> <p>Now let us look at the same reading after 2000 mixes, 16 user at 100Mt.</p> <blockquote> <p> <code>234.3M rnd 420.5M seq 0 same seg 188.8M same pg 29.09M same par 808.9K disk 919.9K spec disk 0B / 0 messages 76K fork</code> </p> </blockquote> <p>We note that the ratios between the random and sequential and same page/parent counts are about the same. The sequential number looks to be even a bit smaller in proportion. The count of random accesses for the 100Mt run is 14% of the count for the 1000Mt run. The count of query parallelization threads is also much lower since it is worthwhile to schedule a new thread only if there are at least a few thousand operations to perform on it. The precise criterion for making a thread is that according to the cost model guess, the thread must have at least 5ms worth of work.</p> <p>We note that the 100 Mt throughput is a little over three-times that of the 1000 Mt throughput, as reported before. We might justifiably ask why the 100 Mt run is not seven-times faster instead, for this much less work. </p> <p>We note that for one-off random access, it makes no real difference whether the tree has 100 M or 1000 M rows; this translates to roughly 27 vs 30 comparisons, so the depth of the tree is not a factor <i>per se</i>. Besides, vectoring makes the tree often look only one or two levels deep, so the total row count matters even less there.</p> <p>To elucidate this last question, we look at the CPU profiles. We take an <a href="http://oprofile.sourceforge.net/about/" id="link-id0x1efb3360">oprofile</a> of 100 Single User mixes at both scales.</p> For 100 Mt: <blockquote> <code><pre> 61161 10.1723 cmpf_iri64n_iri64n_anyn_gt_lt 31321 5.2093 box_equal 19027 3.1646 sqlo_parse_tree_has_node 15905 2.6453 dk_alloc 15647 2.6024 itc_next_set_neq 12702 2.1126 itc_vec_split_search 12487 2.0768 itc_dive_transit 11450 1.9044 itc_bm_vec_row_check 10646 1.7706 itc_page_rcf_search 9223 1.5340 id_hash_get 9215 1.5326 gen_qsort 8867 1.4748 sqlo_key_part_best 8807 1.4648 itc_param_cmp 8062 1.3409 cmpf_iri64n_iri64n 6820 1.1343 sqlo_in_list 6005 0.9987 dc_iri_id_cmp 5905 0.9821 dk_free_tree 5801 0.9648 box_hash 5509 0.9163 dks_esc_write 5444 0.9054 sql_tree_hash_1 </pre> </code> </blockquote> For 1000 Mt <blockquote> <code><pre> 754331 31.4149 cmpf_iri64n_iri64n_anyn_gt_lt 146165 6.0872 itc_vec_split_search 144795 6.0301 itc_next_set_neq 131671 5.4836 itc_dive_transit 110870 4.6173 itc_page_rcf_search 66780 2.7811 gen_qsort 66434 2.7667 itc_param_cmp 58450 2.4342 itc_bm_vec_row_check 55213 2.2994 dk_alloc 47793 1.9904 cmpf_iri64n_iri64n 44277 1.8440 dc_iri_id_cmp 39489 1.6446 cmpf_int64n 36880 1.5359 dc_append_bytes 36601 1.5243 dv_compare 31286 1.3029 dc_any_value_prefetch 25457 1.0602 itc_next_set 20852 0.8684 box_equal 19895 0.8285 dk_free_tree 19698 0.8203 itc_page_insert_search 19367 0.8066 dc_copy </pre> </code> </blockquote> <p>The top function in both is the compare for an equality of two leading IRIs and a range for the trailing any. This corresponds to the range check in Q5. At the larger scale this is three times more important. At the smaller scale, the share of query <a class="auto-href" href="http://dbpedia.org/resource/Program_optimization" id="link-id0x1ea77ab8">optimization</a> is about 6.5 times greater. The top function in this category is <code>box_equal</code> with 5.2% vs 0.87%. The remaining SQL compiler functions are all in proportion to this, totaling 14.3% of the 100 Mt top-20 profile.</p> <p>From this sample it appears ten times more scale is seven times more database operations. This is not taken into account in the metric. Query compilation is significant at the small end, and no longer significant at 1000 Mt. From these numbers, we could say that <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1c819148">Virtuoso</a> is about two times more efficient in terms of database operation throughput at 1000 Mt than at 100 Mt.</p> <p>We may conclude that different BSBM scales measure different things. The <a class="auto-href" href="http://www.tpc.org/" id="link-id0x1f0ffb40">TPC</a> workloads are relatively better in that they have a balance between metric components that stay relatively constant across a large range of scales.</p> <p>This is not necessarily something that should be fixed in the BSBM Explore mix. We must however take these factors better into account in developing the BI mix.</p> <p>Let us also remember that BSBM Explore is a relational workload. Future posts in this series will outline how we propose to make RDF-friendlier benchmarks. </p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1a9bcff8">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1d3e5470">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1de94770">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x1ea66470">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1f1118d8">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1d1c0cd8">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> Benchmarks, Redux (part 7): What Does BSBM Explore Measure? <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1aaf4180">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1a957610">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x127e75c8">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1c9400f0">Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1681" id="link-id0x2aab822515c8">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1682" id="link-id0x2aabf7235508">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1683" id="link-id0x2aabdcfe7418">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1684" id="link-id0x2aab716d53d8">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2014-08-21T10:45:43.982705-04:00
Benchmarks, Redux (part 6): BSBM and I/O, continued
http://www.openlinksw.com:443/blog/vdb/blog/?date=2011-03-07#1670
2011-03-07T22:36:24Z
<p>In the words of Jim Gray, disks have become tapes. By this he means that a disk is really only good for sequential access. For this reason, the SSD extent read ahead was incomparably better. We note that in the experiment, every page in the general area of the database the experiment touched would in time be touched, and that the whole working set would end up in memory. Therefore no speculative read would be wasted. Therefore it stands to reason to read whole extents.</p> <p>So I changed the default behavior to use a very long window for triggering read-ahead as long as the buffer pool was not full. After the initial filling of the buffer pool, the read ahead would require more temporal locality before kicking in. </p> <p>Still, the scheme was not really good since the rest of the extent would go for background-read and the triggering read would be done right then, leading to extra seeks. Well, this is good for latency but bad for throughput. So I changed this too, going to an "elevator only" scheme where reads that triggered read-ahead would go with the read-ahead batch. Reads that did not trigger read-ahead would still be done right in place, thus favoring latency but breaking any sequentiality with its attendant 10+ ms penalty.</p> <p>We keep in mind that the test we target is <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x17c88010">BSBM</a> warm-up time, which is purely a throughput business. One could have timeouts and could penalize queries that sacrificed too much latency to throughput.</p> <p>We note that even for this very simple metric, just reading the allocated database pages from start to end is not good since a large number of pages in fact never get read during a run.</p> <p>We further note that the vectored read-ahead without any speculation will be useful as-is for cases with few threads and striping, since at least one thread's random I/Os get to go to multiple threads. The benefit is less in multiuser situations where disks are randomly busy anyhow. </p> <p>In the previous I/O experiments, we saw that with vectored read ahead and no speculation, there were around 50 pages waiting for I/O at all times. With an easily-triggered extent read-ahead, there were around 4000 pages waiting. The more pages are waiting for I/O, the greater the benefit from the elevator algorithm of servicing I/O in order of file offset. </p> <p>In <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1c51fae0">Virtuoso</a> 5 we had a trick that would, if the buffer pool was not full, speculatively read every uncached sibling of every index tree node it visited. This filled the <a class="auto-href" href="http://dbpedia.org/resource/Cache" id="link-id0x1d6a0cf0">cache</a> quite fast, but was useless after the cache was full. The extent read ahead first implemented in 6 was less aggressive, but would continue working with full cache and did in fact help with shifts in the working set.</p> <p>The next logical step is to combine the vector and extent read-ahead modes. We see what pages we will be getting, then take the distinct extents; if we have been to this extent within the time window, we just add all the uncached allocated pages of the extent to the batch.</p> <p>With this setting, especially at the start of the run, we get large read-ahead batches and maintain I/O queues of 5000 to 20000 pages. The SSD starting time drops to about 120 seconds from cold start to reach 1200% <a class="auto-href" href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x1d295448">CPU</a>. We see transfer rates of up to 150 MB/s per SSD. With HDDs, we see transfer rates around 14 MB/s per drive, mostly reading chunks of an average of seventy-one (71) 8K pages.</p> <p>The BSBM workload does not offer better possibilities for <a class="auto-href" href="http://dbpedia.org/resource/Program_optimization" id="link-id0x1aca8b40">optimization</a>, short of pre-reading the whole database, which is not practical at large scales. </p> <h2>Some Details</h2> <p>First we start from cold disk, with and without mandatory read of the whole extent on the touch.</p> <p>Without any speculation but with vectored read-ahead, here are the times for the first 11 query mixes:</p> <blockquote> <code><pre> 0: 151560.82 ms, total: 151718 ms 1: 179589.08 ms, total: 179648 ms 2: 71974.49 ms, total: 72017 ms 3: 102701.73 ms, total: 102729 ms 4: 58834.41 ms, total: 58856 ms 5: 65926.34 ms, total: 65944 ms 6: 68244.69 ms, total: 68274 ms 7: 39197.15 ms, total: 39215 ms 8: 45654.93 ms, total: 45674 ms 9: 34850.30 ms, total: 34878 ms 10: 100061.30 ms, total: 100079 ms </pre> </code> </blockquote> <p>The average CPU during this time was 5%. The best read throughput was 2.5 MB/s; the average was 1.35 MB/s. The average disk read was 16 ms. </p> <p>With vectored read-ahead and full extents only, i.e., max speculation:</p> <blockquote> <code><pre> 0: 178854.23 ms, total: 179034 ms 1: 110826.68 ms, total: 110887 ms 2: 19896.11 ms, total: 19941 ms 3: 36724.43 ms, total: 36753 ms 4: 21253.70 ms, total: 21285 ms 5: 18417.73 ms, total: 18439 ms 6: 21668.92 ms, total: 21690 ms 7: 12236.49 ms, total: 12267 ms 8: 14922.74 ms, total: 14945 ms 9: 11502.96 ms, total: 11523 ms 10: 15762.34 ms, total: 15792 ms ... 90: 1747.62 ms, total: 1761 ms 91: 1701.01 ms, total: 1714 ms 92: 1300.62 ms, total: 1318 ms 93: 1873.15 ms, total: 1886 ms 94: 1508.24 ms, total: 1524 ms 95: 1748.15 ms, total: 1761 ms 96: 2076.92 ms, total: 2090 ms 97: 2199.38 ms, total: 2212 ms 98: 2305.75 ms, total: 2319 ms 99: 1771.91 ms, total: 1784 ms Scale factor: 2848260 Number of warmup runs: 0 Seed: 808080 Number of query mix runs (without warmups): 100 times min/max Querymix runtime: 1.3006s / 178.8542s Elapsed runtime: 872.993 seconds QMpH: 412.374 query mixes per hour </pre> </code> </blockquote> <p>The peak throughput is 91 MB/s, with average around 50 MB/s; CPU average around 50%.</p> <p>We note that the latency of the first query mix is hardly greater than in the non-speculative run, but starting from mix 3 the speed is clearly better. </p> <p>Then the same with cold SSDs. First with no speculation:</p> <blockquote> <code><pre> 0: 5177.68 ms, total: 5302 ms 1: 2570.16 ms, total: 2614 ms 2: 1353.06 ms, total: 1391 ms 3: 1957.63 ms, total: 1978 ms 4: 1371.13 ms, total: 1386 ms 5: 1765.55 ms, total: 1781 ms 6: 1658.23 ms, total: 1673 ms 7: 1273.87 ms, total: 1289 ms 8: 1355.19 ms, total: 1380 ms 9: 1152.78 ms, total: 1167 ms 10: 1787.91 ms, total: 1802 ms ... 90: 1116.25 ms, total: 1128 ms 91: 989.50 ms, total: 1001 ms 92: 833.24 ms, total: 844 ms 93: 1137.83 ms, total: 1150 ms 94: 969.47 ms, total: 982 ms 95: 1138.04 ms, total: 1149 ms 96: 1155.98 ms, total: 1168 ms 97: 1178.15 ms, total: 1193 ms 98: 1120.18 ms, total: 1132 ms 99: 1013.16 ms, total: 1025 ms Scale factor: 2848260 Number of warmup runs: 0 Seed: 808080 Number of query mix runs (without warmups): 100 times min/max Querymix runtime: 0.8201s / 5.1777s Elapsed runtime: 127.555 seconds QMpH: 2822.321 query mixes per hour </pre> </code> </blockquote> <p>The peak I/O is 45 MB/s, with average 28.3 MB/s; CPU average is 168%.</p> <p>Now, SSDs with max speculation.</p> <blockquote> <code><pre> 0: 44670.34 ms, total: 44809 ms 1: 18490.44 ms, total: 18548 ms 2: 7306.12 ms, total: 7353 ms 3: 9452.66 ms, total: 9485 ms 4: 5648.56 ms, total: 5668 ms 5: 5493.21 ms, total: 5511 ms 6: 5951.48 ms, total: 5970 ms 7: 3815.59 ms, total: 3834 ms 8: 4560.71 ms, total: 4579 ms 9: 3523.74 ms, total: 3543 ms 10: 4724.04 ms, total: 4741 ms ... 90: 673.53 ms, total: 685 ms 91: 534.62 ms, total: 545 ms 92: 730.81 ms, total: 742 ms 93: 1358.14 ms, total: 1370 ms 94: 1098.64 ms, total: 1110 ms 95: 1232.20 ms, total: 1243 ms 96: 1259.57 ms, total: 1273 ms 97: 1298.95 ms, total: 1310 ms 98: 1156.01 ms, total: 1166 ms 99: 1025.45 ms, total: 1034 ms Scale factor: 2848260 Number of warmup runs: 0 Seed: 808080 Number of query mix runs (without warmups): 100 times min/max Querymix runtime: 0.4725s / 44.6703s Elapsed runtime: 269.323 seconds QMpH: 1336.683 query mixes per hour </pre> </code> </blockquote> <p>The peak I/O is 339 MB/s, with average 192 MB/s; average CPU is 121%.</p> <p>The above was measured with the read-ahead thread doing single-page reads. We repeated the test with merging reads with small differences. The max IO was 353 MB/s, and average 173 MB/s; average CPU 113%.</p> <p>We see that the start latency is quite a bit longer than without speculation and the CPU % is lower due to higher latency of individual I/O. The I/O rate is fair. We would expect more throughput however. </p> <p>We find that a supposedly better use of the API, doing single requests of up to 100 pages instead of consecutive requests of 1 page, does not make a lot of difference. The peak I/O is a bit higher; overall throughput is a bit lower.</p> <p>We will have to retry these experiments with a better controller. We have at no point seen anything like the 50K 4KB random I/Os promised for the SSDs by the manufacturer. We know for a fact that the controller gives about 700 MB/s sequential read with <code>cat file /dev/null</code> and two drives busy. With 4 drives busy, this does not get better. The best 30 second stretch we saw in a multiuser BSBM warm-up was 590 MB/s, which is consistent with the <code>cat</code> to <code>/dev/null</code> figure. We will later test with 8 SSDs with better controllers. </p> <p>Note that the average I/O and CPU are averages over 30 second measurement windows; thus for short running tests, there is some error from the window during which the activity ended. </p> <p>Let us now see if we can make a BSBM instance warm up from disk in a reasonable time. We run 16 users with max speculation. We note that after reading 7,500,000 buffers we are not entirely free of disk. The max speculation read-ahead filled the cache in 17 minutes, with an average of 58 MB/s. After the cache is filled, the system shifts to a more conservative policy on extent read-ahead; one which in fact never gets triggered with the BSBM <i>Explore</i> in steady state. The vectored read-ahead is kept on since this by itself does not read pages that are not needed. However, the vectored read-ahead does not run either, because the <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x1c9bca60">data</a> that is accessed in larger batches is already in memory. Thus there remains a trickle of an average 0.49 MB/s from disk. This keeps CPU around 350%. With SSDs, the trickle is about 1.5 MB/s and CPU is around 1300% in steady state. Thus SSDs give approximately triple the throughput in a situation where there is a tiny amount of continuous random disk access. The disk access in question is 80% for retrieving <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1c05e280">RDF</a> literal strings, presumably on behalf of the <code>DESCRIBE</code> query in the mix. This query touches things no other query touches and does so one subject at a time, in a way that can neither be anticipated nor optimized.</p> <p>The Virtuoso 7 column store will deal with this better because it is more space efficient overall. If we apply stream-compression to literals, these will go in under half the space, while quads will go in maybe one-quarter the space. Thus 3000 Mt all from memory should be possible with 72 GB RAM. 1000 Mt row-wise did fit in in 72 GB RAM except for the random literals accessed by the the <code>DESCRIBE</code>. This alone drops throughput to under a third of the memory-only throughput if using HDDs. SSDs, on the other hand, can largely neutralize this effect.</p> <h2>Conclusions</h2> <p>We have looked at basics of I/O. SSDs have been found to be a readily available solution to I/O bottlenecks without need for reconfiguration or complex I/O policies. We have been able to get a decent read rate under conditions of server warm-up or shift of working set even with HDDs.</p> <p>More advanced I/O matters will be covered with the column store. We note that the techniques discussed here apply identically to rows and columns.</p> <p>As concerns BSBM, it seems appropriate to include a warm-up time. In practice, this means that the store just must eagerly pre-read. This is not hard to do and can be quite useful.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1b4342b0">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1d3e7388">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x153c7ba8">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x1da11d98">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1d25d630">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs</a> </li> <li> Benchmarks, Redux (part 6): BSBM and I/O, continued <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1f1f5ee8">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1cd44938">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1d51f848">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x13d333c0">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1e77a5e8">Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1ea1fb40">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1e7786c8">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1f8a37f8">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1c69e018">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2011-03-14T17:57:06.000001-04:00
Benchmarks, Redux (part 6): BSBM and I/O, continued
http://www.openlinksw.com:443/weblog/oerling/?date=2011-03-07#1669
2011-03-07T22:36:24Z
<p>In the words of Jim Gray, disks have become tapes. By this he means that a disk is really only good for sequential access. For this reason, the SSD extent read ahead was incomparably better. We note that in the experiment, every page in the general area of the database the experiment touched would in time be touched, and that the whole working set would end up in memory. Therefore no speculative read would be wasted. Therefore it stands to reason to read whole extents.</p> <p>So I changed the default behavior to use a very long window for triggering read-ahead as long as the buffer pool was not full. After the initial filling of the buffer pool, the read ahead would require more temporal locality before kicking in. </p> <p>Still, the scheme was not really good since the rest of the extent would go for background-read and the triggering read would be done right then, leading to extra seeks. Well, this is good for latency but bad for throughput. So I changed this too, going to an "elevator only" scheme where reads that triggered read-ahead would go with the read-ahead batch. Reads that did not trigger read-ahead would still be done right in place, thus favoring latency but breaking any sequentiality with its attendant 10+ ms penalty.</p> <p>We keep in mind that the test we target is <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1beac870">BSBM</a> warm-up time, which is purely a throughput business. One could have timeouts and could penalize queries that sacrificed too much latency to throughput.</p> <p>We note that even for this very simple metric, just reading the allocated database pages from start to end is not good since a large number of pages in fact never get read during a run.</p> <p>We further note that the vectored read-ahead without any speculation will be useful as-is for cases with few threads and striping, since at least one thread's random I/Os get to go to multiple threads. The benefit is less in multiuser situations where disks are randomly busy anyhow. </p> <p>In the previous I/O experiments, we saw that with vectored read ahead and no speculation, there were around 50 pages waiting for I/O at all times. With an easily-triggered extent read-ahead, there were around 4000 pages waiting. The more pages are waiting for I/O, the greater the benefit from the elevator algorithm of servicing I/O in order of file offset. </p> <p>In <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x117fd820">Virtuoso</a> 5 we had a trick that would, if the buffer pool was not full, speculatively read every uncached sibling of every index tree node it visited. This filled the <a class="auto-href" href="http://dbpedia.org/resource/Cache" id="link-id0x1de04030">cache</a> quite fast, but was useless after the cache was full. The extent read ahead first implemented in 6 was less aggressive, but would continue working with full cache and did in fact help with shifts in the working set.</p> <p>The next logical step is to combine the vector and extent read-ahead modes. We see what pages we will be getting, then take the distinct extents; if we have been to this extent within the time window, we just add all the uncached allocated pages of the extent to the batch.</p> <p>With this setting, especially at the start of the run, we get large read-ahead batches and maintain I/O queues of 5000 to 20000 pages. The SSD starting time drops to about 120 seconds from cold start to reach 1200% <a class="auto-href" href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x180fa748">CPU</a>. We see transfer rates of up to 150 MB/s per SSD. With HDDs, we see transfer rates around 14 MB/s per drive, mostly reading chunks of an average of seventy-one (71) 8K pages.</p> <p>The BSBM workload does not offer better possibilities for <a class="auto-href" href="http://dbpedia.org/resource/Program_optimization" id="link-id0x1e0f7078">optimization</a>, short of pre-reading the whole database, which is not practical at large scales. </p> <h2>Some Details</h2> <p>First we start from cold disk, with and without mandatory read of the whole extent on the touch.</p> <p>Without any speculation but with vectored read-ahead, here are the times for the first 11 query mixes:</p> <blockquote> <code><pre> 0: 151560.82 ms, total: 151718 ms 1: 179589.08 ms, total: 179648 ms 2: 71974.49 ms, total: 72017 ms 3: 102701.73 ms, total: 102729 ms 4: 58834.41 ms, total: 58856 ms 5: 65926.34 ms, total: 65944 ms 6: 68244.69 ms, total: 68274 ms 7: 39197.15 ms, total: 39215 ms 8: 45654.93 ms, total: 45674 ms 9: 34850.30 ms, total: 34878 ms 10: 100061.30 ms, total: 100079 ms </pre> </code> </blockquote> <p>The average CPU during this time was 5%. The best read throughput was 2.5 MB/s; the average was 1.35 MB/s. The average disk read was 16 ms. </p> <p>With vectored read-ahead and full extents only, i.e., max speculation:</p> <blockquote> <code><pre> 0: 178854.23 ms, total: 179034 ms 1: 110826.68 ms, total: 110887 ms 2: 19896.11 ms, total: 19941 ms 3: 36724.43 ms, total: 36753 ms 4: 21253.70 ms, total: 21285 ms 5: 18417.73 ms, total: 18439 ms 6: 21668.92 ms, total: 21690 ms 7: 12236.49 ms, total: 12267 ms 8: 14922.74 ms, total: 14945 ms 9: 11502.96 ms, total: 11523 ms 10: 15762.34 ms, total: 15792 ms ... 90: 1747.62 ms, total: 1761 ms 91: 1701.01 ms, total: 1714 ms 92: 1300.62 ms, total: 1318 ms 93: 1873.15 ms, total: 1886 ms 94: 1508.24 ms, total: 1524 ms 95: 1748.15 ms, total: 1761 ms 96: 2076.92 ms, total: 2090 ms 97: 2199.38 ms, total: 2212 ms 98: 2305.75 ms, total: 2319 ms 99: 1771.91 ms, total: 1784 ms Scale factor: 2848260 Number of warmup runs: 0 Seed: 808080 Number of query mix runs (without warmups): 100 times min/max Querymix runtime: 1.3006s / 178.8542s Elapsed runtime: 872.993 seconds QMpH: 412.374 query mixes per hour </pre> </code> </blockquote> <p>The peak throughput is 91 MB/s, with average around 50 MB/s; CPU average around 50%.</p> <p>We note that the latency of the first query mix is hardly greater than in the non-speculative run, but starting from mix 3 the speed is clearly better. </p> <p>Then the same with cold SSDs. First with no speculation:</p> <blockquote> <code><pre> 0: 5177.68 ms, total: 5302 ms 1: 2570.16 ms, total: 2614 ms 2: 1353.06 ms, total: 1391 ms 3: 1957.63 ms, total: 1978 ms 4: 1371.13 ms, total: 1386 ms 5: 1765.55 ms, total: 1781 ms 6: 1658.23 ms, total: 1673 ms 7: 1273.87 ms, total: 1289 ms 8: 1355.19 ms, total: 1380 ms 9: 1152.78 ms, total: 1167 ms 10: 1787.91 ms, total: 1802 ms ... 90: 1116.25 ms, total: 1128 ms 91: 989.50 ms, total: 1001 ms 92: 833.24 ms, total: 844 ms 93: 1137.83 ms, total: 1150 ms 94: 969.47 ms, total: 982 ms 95: 1138.04 ms, total: 1149 ms 96: 1155.98 ms, total: 1168 ms 97: 1178.15 ms, total: 1193 ms 98: 1120.18 ms, total: 1132 ms 99: 1013.16 ms, total: 1025 ms Scale factor: 2848260 Number of warmup runs: 0 Seed: 808080 Number of query mix runs (without warmups): 100 times min/max Querymix runtime: 0.8201s / 5.1777s Elapsed runtime: 127.555 seconds QMpH: 2822.321 query mixes per hour </pre> </code> </blockquote> <p>The peak I/O is 45 MB/s, with average 28.3 MB/s; CPU average is 168%.</p> <p>Now, SSDs with max speculation.</p> <blockquote> <code><pre> 0: 44670.34 ms, total: 44809 ms 1: 18490.44 ms, total: 18548 ms 2: 7306.12 ms, total: 7353 ms 3: 9452.66 ms, total: 9485 ms 4: 5648.56 ms, total: 5668 ms 5: 5493.21 ms, total: 5511 ms 6: 5951.48 ms, total: 5970 ms 7: 3815.59 ms, total: 3834 ms 8: 4560.71 ms, total: 4579 ms 9: 3523.74 ms, total: 3543 ms 10: 4724.04 ms, total: 4741 ms ... 90: 673.53 ms, total: 685 ms 91: 534.62 ms, total: 545 ms 92: 730.81 ms, total: 742 ms 93: 1358.14 ms, total: 1370 ms 94: 1098.64 ms, total: 1110 ms 95: 1232.20 ms, total: 1243 ms 96: 1259.57 ms, total: 1273 ms 97: 1298.95 ms, total: 1310 ms 98: 1156.01 ms, total: 1166 ms 99: 1025.45 ms, total: 1034 ms Scale factor: 2848260 Number of warmup runs: 0 Seed: 808080 Number of query mix runs (without warmups): 100 times min/max Querymix runtime: 0.4725s / 44.6703s Elapsed runtime: 269.323 seconds QMpH: 1336.683 query mixes per hour </pre> </code> </blockquote> <p>The peak I/O is 339 MB/s, with average 192 MB/s; average CPU is 121%.</p> <p>The above was measured with the read-ahead thread doing single-page reads. We repeated the test with merging reads with small differences. The max IO was 353 MB/s, and average 173 MB/s; average CPU 113%.</p> <p>We see that the start latency is quite a bit longer than without speculation and the CPU % is lower due to higher latency of individual I/O. The I/O rate is fair. We would expect more throughput however. </p> <p>We find that a supposedly better use of the API, doing single requests of up to 100 pages instead of consecutive requests of 1 page, does not make a lot of difference. The peak I/O is a bit higher; overall throughput is a bit lower.</p> <p>We will have to retry these experiments with a better controller. We have at no point seen anything like the 50K 4KB random I/Os promised for the SSDs by the manufacturer. We know for a fact that the controller gives about 700 MB/s sequential read with <code>cat file /dev/null</code> and two drives busy. With 4 drives busy, this does not get better. The best 30 second stretch we saw in a multiuser BSBM warm-up was 590 MB/s, which is consistent with the <code>cat</code> to <code>/dev/null</code> figure. We will later test with 8 SSDs with better controllers. </p> <p>Note that the average I/O and CPU are averages over 30 second measurement windows; thus for short running tests, there is some error from the window during which the activity ended. </p> <p>Let us now see if we can make a BSBM instance warm up from disk in a reasonable time. We run 16 users with max speculation. We note that after reading 7,500,000 buffers we are not entirely free of disk. The max speculation read-ahead filled the cache in 17 minutes, with an average of 58 MB/s. After the cache is filled, the system shifts to a more conservative policy on extent read-ahead; one which in fact never gets triggered with the BSBM <i>Explore</i> in steady state. The vectored read-ahead is kept on since this by itself does not read pages that are not needed. However, the vectored read-ahead does not run either, because the <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x1d7674e8">data</a> that is accessed in larger batches is already in memory. Thus there remains a trickle of an average 0.49 MB/s from disk. This keeps CPU around 350%. With SSDs, the trickle is about 1.5 MB/s and CPU is around 1300% in steady state. Thus SSDs give approximately triple the throughput in a situation where there is a tiny amount of continuous random disk access. The disk access in question is 80% for retrieving <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1c4adda0">RDF</a> literal strings, presumably on behalf of the <code>DESCRIBE</code> query in the mix. This query touches things no other query touches and does so one subject at a time, in a way that can neither be anticipated nor optimized.</p> <p>The Virtuoso 7 column store will deal with this better because it is more space efficient overall. If we apply stream-compression to literals, these will go in under half the space, while quads will go in maybe one-quarter the space. Thus 3000 Mt all from memory should be possible with 72 GB RAM. 1000 Mt row-wise did fit in in 72 GB RAM except for the random literals accessed by the the <code>DESCRIBE</code>. This alone drops throughput to under a third of the memory-only throughput if using HDDs. SSDs, on the other hand, can largely neutralize this effect.</p> <h2>Conclusions</h2> <p>We have looked at basics of I/O. SSDs have been found to be a readily available solution to I/O bottlenecks without need for reconfiguration or complex I/O policies. We have been able to get a decent read rate under conditions of server warm-up or shift of working set even with HDDs.</p> <p>More advanced I/O matters will be covered with the column store. We note that the techniques discussed here apply identically to rows and columns.</p> <p>As concerns BSBM, it seems appropriate to include a warm-up time. In practice, this means that the store just must eagerly pre-read. This is not hard to do and can be quite useful.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1b4342b0">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1d3e7388">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x153c7ba8">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x1da11d98">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1d25d630">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs</a> </li> <li> Benchmarks, Redux (part 6): BSBM and I/O, continued <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1f1f5ee8">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1cd44938">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1d51f848">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x13d333c0">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1e77a5e8">Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1681" id="link-id0x2aabddff1f28">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1682" id="link-id0x2aabf6b82c78">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1683" id="link-id0x2aabf4cb7f28">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1684" id="link-id0x2aabdfe82538">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2014-08-21T10:45:52.436400-04:00
Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs
http://www.openlinksw.com:443/blog/vdb/blog/?date=2011-03-07#1668
2011-03-07T19:17:36Z
<p>In the context of database benchmarks we cannot ignore I/O, as pretty much has been done so far by <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1ea17348">BSBM</a>. </p> <p>There are two approaches:</p> <ol> <li> <p>run twice or otherwise make sure one runs from memory and forget about I/O, or</p> </li> <li> <p>make rules and metrics for warm-up.</p> </li> </ol> <p>We will see if the second is possible with BSBM.</p> <p>From this starting point, we look at various ways of scheduling I/O in <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x125c4f90">Virtuoso</a> using a 1000 Mt BSBM database on sets of each of HDDs (hard disk devices) and SSDs (solid-state storage devices). We will see that SSDs in this specific application can make a significant difference. </p> <p>In this test we have the same 4 stripes of a 1000 Mt BSBM database on each of two storage arrays.</p> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="9" align="center">Storage Arrays</th> </tr> <tr> <th align="center"> Type </th> <th align="center"> Quantity </th> <th align="center"> Maker </th> <th align="center"> Size </th> <th align="center"> Speed </th> <th align="center"> Interface speed </th> <th align="center"> Controller </th> <th align="center"> Drive <a class="auto-href" href="http://dbpedia.org/resource/Cache" id="link-id0x1cab1358">Cache</a> </th> <th align="center"> RAID </th> </tr> <tr> <td align="center"> SSD </td> <td align="center"> 4 </td> <td align="center"> Crucial </td> <td align="center"> 128 GB </td> <td align="center"> N/A </td> <td align="center"> 6Gbit SATA </td> <td align="center"> RocketRaid 640 </td> <td align="center"> 128 MB </td> <td align="center"> None </td> </tr> <tr> <td align="center"> HDD </td> <td align="center"> 4 </td> <td align="center"> Samsung </td> <td align="center"> 1000 GB </td> <td align="center"> 7200 RPM </td> <td align="center"> 3Gbit SATA </td> <td align="center"> <a class="auto-href" href="http://dbpedia.org/resource/Intel_Corporation" id="link-id0x1ab6edd8">Intel</a> ICH on Supermicro motherboard </td> <td align="center"> 16 MB </td> <td align="center"> None </td> </tr> </table> <p>We make sure that the files are not in OS cache by filling it with other big files, reading a total of 120 GB off SSDs with <code>`cat file > /dev/null`</code>. </p> <p>The configuration files are as in the report on the 1000 Mt run. We note as significant that we have a few file descriptors for each stripe, and that read-ahead for each is handled by its own thread.</p> <p>Two different read-ahead schemes are used: </p> <ul> <li> <p>With 6 Single, if a 2MB extent gets a second read within a given time after the first, the whole extent is scheduled for background read.</p> </li> <li> <p>With 7 Single, as an index search is vectored, we know a large number of values to fetch at one time and these values are sorted into an ascending sequence. Therefore, by looking at a node in an index tree, we can determine which sub-trees will be accessed and schedule these for read-ahead, skipping any that will not be accessed.</p> </li> </ul> <p>In either model, a sequential scan touching more than a couple of consecutive index leaf pages triggers a read-ahead, to the end of the scanned range or to the next 3000 index leaves, whichever comes first. However, there are no sequential scans of significant size in BSBM.</p> <p>There are a few different possibilities for the physical I/O: </p> <ol> <li> <p>Using a separate read system call for each page. There may be several open file descriptors on a file so that many such calls can proceed concurrently on different threads; the OS will order the operations.</p> </li> <li> <p>A thread finds it needs a page and reads it.</p> </li> <li> <p>Using Unix asynchronous I/O, <code>aio.h</code>, with the <code>aio_*</code> and <code>lio_listio</code> functions.</p> </li> <li> <p>Using single-read system calls for adjacent pages. In this way, the drive sees longer requests and should give better throughput. If there are short gaps in the sequence, the gaps are also read, wasting bandwidth but saving on latency.</p> </li> </ol> <p>The two latter apply only to bulk I/O that are scheduled on background threads, one per independently-addressable device (HDD, SSD, or RAID-set). These bulk-reads operate on an elevator model, keeping a sorted queue of things to read or write and moving through this queue from start to end. At any time, the queue may get more work from other threads.</p> <p>There is a further choice when seeing single-page random requests. They can either go to the elevator or they can be done in place. Taking the elevator is presumably good for throughput but bad for latency. In general, the elevator should have a notion of fairness; these matters are discussed in the <a href="http://www.cwi.nl/" id="link-id0x1f62abb8">CWI collaborative scan paper</a>. Here we do not have long queries, so we do not have to talk about elevator policies or scan sharing; there are no scans. We may touch on these questions later with the column store, the BSBM BI mix, and <a class="auto-href" href="http://www.tpc.org/" id="link-id0x1bfb17c0">TPC</a>-<a class="auto-href" href="http://dbpedia.org/resource/TPC-H" id="link-id0x1e76bfc8">H</a>.</p> <p>While we may know principles, I/O has always given us surprises; the only way to optimize this is to measure.</p> <p>The metric we try to optimize here is the time it takes for a multiuser BSBM run starting from cold cache to get to 1200% <a class="auto-href" href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x1d7b1d10">CPU</a>. When running from memory, the CPU is around 1350% for the system in question. </p> <p>This depends on getting I/O throughput, which in turn depends on having a lot of speculative reading since the workload itself does not give any long stretches to read. </p> <p>The test driver is set at 16 clients, and the run continues for 2000 query mixes or until target throughput is reached. Target throughput is deemed reached after the first 20 second stretch with CPU at 1200% or higher.</p> <p>The meter is a stored procedure that records the CPU time, count of reads, cumulative elapsed time spent waiting for I/O, and other metrics. The code for this procedure (for 7 Single; this file will not work on Virtuoso 6 or earlier) is <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/BenchmarksReduxSupportingFiles/ldmeter.sql" id="link-id0x1b5adb08">available here</a>. </p> <p>The database space allocation gives each index a number of 2MB segments, each with 256 8K pages. When a page splits, the new page is allocated from the same extent if possible, or from a specific second extent which is designated as the overflow extent of this extent. This scheme provides for a sort of pseudo-locality within extents over random insert order. Thus there is a chance that pre-reading an extent will get key values in the same range a the ones on the page being requested in the first place. At least the pre-read pages will be from the same index tree. There are insertion orders that do not create good locality with this allocation scheme, though. In order to generally improve locality, one could shuffle pages of an all-dirty subtree before writing this out so as to have physical order match key order. We will look at some tricks in this vein with the column store.</p> <p>For the sake of simplicity we only run 7 Single with the 1000 Mt scale.</p> <p>The first experiment was with SSDs and the vectored read-ahead. The target throughput was reached after 280 seconds. </p> <p>The next test was with HDDs and extent read-ahead. One hour into the experiment, the CPU was about 70% after processing around 1000 query mixes. It might have been hours before HDD reads became rare enough for hitting 1200% CPU. The test was not worth continuing.</p> <p>The result with HDDs and vectored read-ahead would be worse since vectored read-ahead leads to smaller read-ahead batches and to less contiguous read patterns. The individual read times here, are over twice the individual read times with per-extent read-ahead. The fact that vectored read-ahead does not read potentially unneeded pages makes no difference. Hence this test is also not worth running to completion.</p> <p>There are other possibilities for improving HDD I/O. If only 2MB read requests are made, a transfer will be about 20 ms at a sequential transfer speed of 50 MB/s. Then seeking to the next 2MB extent will be a few ms, most often less than 20, so the HDD should give at least half the nominal throughput.</p> <p>We note that, when reading sequential 8K pages inside a single 2MB (256 page) extent, the seek latency is not 0 as one would expect but an extreme 5 ms. One would think that the drive would buffer a whole track, and a track would hold a large number of 2MB sections, but apparently this is not so. </p> <p>Therefore, now if we have a sequential read pattern that is more dense than 1 page out of 10, we read all the pages and just keep the ones we want.</p> <p>So now we set the read-ahead to merge reads that fall within 10 pages. This wastes bandwidth, but supposedly saves on latency. We will see. </p> <p>So we try, and we find that read-ahead does not account for most pages since it does not get triggered. Thus, we change the triggering condition to be the 2nd read to fall in the extent within 20 seconds of the first.</p> <p>The HDDs were in all cases 700% busy for 4 HDDs. But with the new setting we get longer requests, most often full extents, which gets a per-HDD transfer rate of about 5 MB/s. With the looser condition for starting read-ahead, 89% of all pages were read in a read-ahead batch. We see the I/O throughput decrease during the run because there are more single-page reads that do not trigger extent read-ahead. So HDDs have 1.7 concurrent operations pending, but the batch size drops, dropping the throughput.</p> <p> </p> <p>Thus with the best settings, the test with 2000 query mixes finishes in 46 minutes, and the CPU utilization is steadily increasing, hitting 392% for the last minute. In comparison, with SSDs and our worst read-ahead setting we got 1200% CPU in under 5 minutes from cold start. The I/O system can be further tuned; for example, by only reading full extents as long as the buffer pool is not full. In the next post we will measure some more. </p> <p> </p> <h3>BSBM Note </h3> <p>We look at query times with semi-warm cache, with CPU around 400%. We note that Q8-Q12 are especially bad. Q5 runs at about half speed. Q12 runs at under 1/10th speed. The relatively slowest queries appear to be single-instance lookups. Nothing short of the most aggressive speculative reading can help there. Neither query nor workload has any exploitable pattern. Therefore if an I/O component is to be included in a BSBM metric, the only way to score in this is to use speculative read to the maximum.</p> <p>Some of the queries take consecutive property values of a single instance. One could parallelize this pipeline, but this would be a one-off and would make sense only when reading from storage (whether HDD, SSD, or otherwise). Multithreading for single rows is not worth the overhead.</p> <p>A metric for BSBM warm-up is not interesting for database science, but may still be of practical interest in the specific case of <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1d6371d8">RDF</a> stores. Specially reading large chunks at startup time is good, so putting a section in BSBM that would force one to implement this would be a service to most end users. Measuring and reporting such I/O performance would favor space efficiency in general. Space efficiency is generally a good thing, especially at larger scales, so we can put an optional section in the report for warm-up. This is also good for comparing HDDs and SSDs, and for testing read-ahead, which is still something a database is expected to do. Implementors have it easy; just speculatively read everything.</p> <p>Looking at the BSBM fictional use case, anybody running such a portal would do this from RAM only, so it makes sense to define the primary metric as running from warm cache, in practice 100% from memory.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1ecb2af0">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x19d05678">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1d542328">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x13947e08">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1a7f6b30">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1d67dd40">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1ebcee68">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1a855ba0">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x1b081e70">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1d7a7940">Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1d7e2cd0">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1e375338">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1d199728">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1e808818">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2011-03-14T17:56:52-04:00
Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs
http://www.openlinksw.com:443/weblog/oerling/?date=2011-03-07#1667
2011-03-07T19:17:36Z
<p>In the context of database benchmarks we cannot ignore I/O, as pretty much has been done so far by <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x2a0452f8">BSBM</a>. </p> <p>There are two approaches:</p> <ol> <li> <p>run twice or otherwise make sure one runs from memory and forget about I/O, or</p> </li> <li> <p>make rules and metrics for warm-up.</p> </li> </ol> <p>We will see if the second is possible with BSBM.</p> <p>From this starting point, we look at various ways of scheduling I/O in <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x2a9fdb88">Virtuoso</a> using a 1000 Mt BSBM database on sets of each of HDDs (hard disk devices) and SSDs (solid-state storage devices). We will see that SSDs in this specific application can make a significant difference. </p> <p>In this test we have the same 4 stripes of a 1000 Mt BSBM database on each of two storage arrays.</p> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="9" align="center">Storage Arrays</th> </tr> <tr> <th align="center"> Type </th> <th align="center"> Quantity </th> <th align="center"> Maker </th> <th align="center"> Size </th> <th align="center"> Speed </th> <th align="center"> Interface speed </th> <th align="center"> Controller </th> <th align="center"> Drive <a class="auto-href" href="http://dbpedia.org/resource/Cache" id="link-id0x2ad20cd0">Cache</a> </th> <th align="center"> RAID </th> </tr> <tr> <td align="center"> SSD </td> <td align="center"> 4 </td> <td align="center"> Crucial </td> <td align="center"> 128 GB </td> <td align="center"> N/A </td> <td align="center"> 6Gbit SATA </td> <td align="center"> RocketRaid 640 </td> <td align="center"> 128 MB </td> <td align="center"> None </td> </tr> <tr> <td align="center"> HDD </td> <td align="center"> 4 </td> <td align="center"> Samsung </td> <td align="center"> 1000 GB </td> <td align="center"> 7200 RPM </td> <td align="center"> 3Gbit SATA </td> <td align="center"> <a class="auto-href" href="http://dbpedia.org/resource/Intel_Corporation" id="link-id0x2a8cfca0">Intel</a> ICH on Supermicro motherboard </td> <td align="center"> 16 MB </td> <td align="center"> None </td> </tr> </table> <p>We make sure that the files are not in OS cache by filling it with other big files, reading a total of 120 GB off SSDs with <code>`cat file > /dev/null`</code>. </p> <p>The configuration files are as in the report on the 1000 Mt run. We note as significant that we have a few file descriptors for each stripe, and that read-ahead for each is handled by its own thread.</p> <p>Two different read-ahead schemes are used: </p> <ul> <li> <p>With 6 Single, if a 2MB extent gets a second read within a given time after the first, the whole extent is scheduled for background read.</p> </li> <li> <p>With 7 Single, as an index search is vectored, we know a large number of values to fetch at one time and these values are sorted into an ascending sequence. Therefore, by looking at a node in an index tree, we can determine which sub-trees will be accessed and schedule these for read-ahead, skipping any that will not be accessed.</p> </li> </ul> <p>In either model, a sequential scan touching more than a couple of consecutive index leaf pages triggers a read-ahead, to the end of the scanned range or to the next 3000 index leaves, whichever comes first. However, there are no sequential scans of significant size in BSBM.</p> <p>There are a few different possibilities for the physical I/O: </p> <ol> <li> <p>Using a separate read system call for each page. There may be several open file descriptors on a file so that many such calls can proceed concurrently on different threads; the OS will order the operations.</p> </li> <li> <p>A thread finds it needs a page and reads it.</p> </li> <li> <p>Using Unix asynchronous I/O, <code>aio.h</code>, with the <code>aio_*</code> and <code>lio_listio</code> functions.</p> </li> <li> <p>Using single-read system calls for adjacent pages. In this way, the drive sees longer requests and should give better throughput. If there are short gaps in the sequence, the gaps are also read, wasting bandwidth but saving on latency.</p> </li> </ol> <p>The two latter apply only to bulk I/O that are scheduled on background threads, one per independently-addressable device (HDD, SSD, or RAID-set). These bulk-reads operate on an elevator model, keeping a sorted queue of things to read or write and moving through this queue from start to end. At any time, the queue may get more work from other threads.</p> <p>There is a further choice when seeing single-page random requests. They can either go to the elevator or they can be done in place. Taking the elevator is presumably good for throughput but bad for latency. In general, the elevator should have a notion of fairness; these matters are discussed in the <a href="http://www.cwi.nl/" id="link-id0x1f62abb8">CWI collaborative scan paper</a>. Here we do not have long queries, so we do not have to talk about elevator policies or scan sharing; there are no scans. We may touch on these questions later with the column store, the BSBM BI mix, and <a class="auto-href" href="http://www.tpc.org/" id="link-id0x2a9067d0">TPC</a>-<a class="auto-href" href="http://dbpedia.org/resource/TPC-H" id="link-id0x2a8874f0">H</a>.</p> <p>While we may know principles, I/O has always given us surprises; the only way to optimize this is to measure.</p> <p>The metric we try to optimize here is the time it takes for a multiuser BSBM run starting from cold cache to get to 1200% <a class="auto-href" href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x2997a660">CPU</a>. When running from memory, the CPU is around 1350% for the system in question. </p> <p>This depends on getting I/O throughput, which in turn depends on having a lot of speculative reading since the workload itself does not give any long stretches to read. </p> <p>The test driver is set at 16 clients, and the run continues for 2000 query mixes or until target throughput is reached. Target throughput is deemed reached after the first 20 second stretch with CPU at 1200% or higher.</p> <p>The meter is a stored procedure that records the CPU time, count of reads, cumulative elapsed time spent waiting for I/O, and other metrics. The code for this procedure (for 7 Single; this file will not work on Virtuoso 6 or earlier) is <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/BenchmarksReduxSupportingFiles/ldmeter.sql" id="link-id0x1b5adb08">available here</a>. </p> <p>The database space allocation gives each index a number of 2MB segments, each with 256 8K pages. When a page splits, the new page is allocated from the same extent if possible, or from a specific second extent which is designated as the overflow extent of this extent. This scheme provides for a sort of pseudo-locality within extents over random insert order. Thus there is a chance that pre-reading an extent will get key values in the same range a the ones on the page being requested in the first place. At least the pre-read pages will be from the same index tree. There are insertion orders that do not create good locality with this allocation scheme, though. In order to generally improve locality, one could shuffle pages of an all-dirty subtree before writing this out so as to have physical order match key order. We will look at some tricks in this vein with the column store.</p> <p>For the sake of simplicity we only run 7 Single with the 1000 Mt scale.</p> <p>The first experiment was with SSDs and the vectored read-ahead. The target throughput was reached after 280 seconds. </p> <p>The next test was with HDDs and extent read-ahead. One hour into the experiment, the CPU was about 70% after processing around 1000 query mixes. It might have been hours before HDD reads became rare enough for hitting 1200% CPU. The test was not worth continuing.</p> <p>The result with HDDs and vectored read-ahead would be worse since vectored read-ahead leads to smaller read-ahead batches and to less contiguous read patterns. The individual read times here, are over twice the individual read times with per-extent read-ahead. The fact that vectored read-ahead does not read potentially unneeded pages makes no difference. Hence this test is also not worth running to completion.</p> <p>There are other possibilities for improving HDD I/O. If only 2MB read requests are made, a transfer will be about 20 ms at a sequential transfer speed of 50 MB/s. Then seeking to the next 2MB extent will be a few ms, most often less than 20, so the HDD should give at least half the nominal throughput.</p> <p>We note that, when reading sequential 8K pages inside a single 2MB (256 page) extent, the seek latency is not 0 as one would expect but an extreme 5 ms. One would think that the drive would buffer a whole track, and a track would hold a large number of 2MB sections, but apparently this is not so. </p> <p>Therefore, now if we have a sequential read pattern that is more dense than 1 page out of 10, we read all the pages and just keep the ones we want.</p> <p>So now we set the read-ahead to merge reads that fall within 10 pages. This wastes bandwidth, but supposedly saves on latency. We will see. </p> <p>So we try, and we find that read-ahead does not account for most pages since it does not get triggered. Thus, we change the triggering condition to be the 2nd read to fall in the extent within 20 seconds of the first.</p> <p>The HDDs were in all cases 700% busy for 4 HDDs. But with the new setting we get longer requests, most often full extents, which gets a per-HDD transfer rate of about 5 MB/s. With the looser condition for starting read-ahead, 89% of all pages were read in a read-ahead batch. We see the I/O throughput decrease during the run because there are more single-page reads that do not trigger extent read-ahead. So HDDs have 1.7 concurrent operations pending, but the batch size drops, dropping the throughput.</p> <p> </p> <p>Thus with the best settings, the test with 2000 query mixes finishes in 46 minutes, and the CPU utilization is steadily increasing, hitting 392% for the last minute. In comparison, with SSDs and our worst read-ahead setting we got 1200% CPU in under 5 minutes from cold start. The I/O system can be further tuned; for example, by only reading full extents as long as the buffer pool is not full. In the next post we will measure some more. </p> <p> </p> <h3>BSBM Note </h3> <p>We look at query times with semi-warm cache, with CPU around 400%. We note that Q8-Q12 are especially bad. Q5 runs at about half speed. Q12 runs at under 1/10th speed. The relatively slowest queries appear to be single-instance lookups. Nothing short of the most aggressive speculative reading can help there. Neither query nor workload has any exploitable pattern. Therefore if an I/O component is to be included in a BSBM metric, the only way to score in this is to use speculative read to the maximum.</p> <p>Some of the queries take consecutive property values of a single instance. One could parallelize this pipeline, but this would be a one-off and would make sense only when reading from storage (whether HDD, SSD, or otherwise). Multithreading for single rows is not worth the overhead.</p> <p>A metric for BSBM warm-up is not interesting for database science, but may still be of practical interest in the specific case of <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x29987c90">RDF</a> stores. Specially reading large chunks at startup time is good, so putting a section in BSBM that would force one to implement this would be a service to most end users. Measuring and reporting such I/O performance would favor space efficiency in general. Space efficiency is generally a good thing, especially at larger scales, so we can put an optional section in the report for warm-up. This is also good for comparing HDDs and SSDs, and for testing read-ahead, which is still something a database is expected to do. Implementors have it easy; just speculatively read everything.</p> <p>Looking at the BSBM fictional use case, anybody running such a portal would do this from RAM only, so it makes sense to define the primary metric as running from warm cache, in practice 100% from memory.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1ecb2af0">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x19d05678">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1d542328">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x13947e08">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1a7f6b30">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1d67dd40">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1ebcee68">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1a855ba0">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x1b081e70">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1d7a7940">Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1681" id="link-id0x2aac00eabec8">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1682" id="link-id0x2aac021b2c78">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1683" id="link-id0x2aabe483e2b8">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1684" id="link-id0x2aac1400d3c8">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2014-08-21T10:45:58.995101-04:00
Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire
http://www.openlinksw.com:443/blog/vdb/blog/?date=2011-03-04#1666
2011-03-04T20:28:28Z
<p>Below is a questionnaire I sent to the <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x17f62428">BSBM</a> participants in order to get tuning instructions for the runs we were planning. I have filled in the answers for <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1d48ed28">Virtuoso</a>, here. This can be a checklist for pretty much any <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1e11b228">RDF</a> database tuning.</p> <ol> <li> <p> <b>Threading - What settings should be used (e.g., for query parallelization, I/O parallelization [e.g., prefetch, flush of dirty], thread pools [e,.g. web server], any other thread related)? We will run with 8 and 32 cores, so if there are settings controlling number of read/write (R/W) locks or mutexes or such for serializing diverse things, these should be set accordingly to minimize contention.</b> </p> <p>The following three settings are all <a href="http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ini_Parameters" id="link-id0x1ed4fe10">in the <code>[Parameters]</code> section of the <code>virtuoso.ini</code> file</a>. </p> <ul> <li> <p> <b><code>AsyncQueueMaxThreads</code> </b> controls the size of a pool of extra threads that can be used for query parallelization. This should be set to either <b>1.5 * the number of cores</b> or <b>1.5 * the number of core threads</b>; see which works better.</p> </li> <li> <p> <b><code>ThreadsPerQuery</code> </b> is the maximum number of threads a single query will take. This should be set to either <b>the number of cores</b> or <b>the number of core threads</b>; see which works better. </p> </li> <li> <p> <b><code>IndexTreeMaps</code> </b> is the number of mutexes over which control for buffering an index tree is split. This can generally be left at default (<b>256</b> in normal operation; valid settings are powers of 2 from 2 to 1024), but setting to <b>64, 128, or 512</b> may be beneficial.</p> <p>A low number will lead to frequent contention; upwards of 64 will have little contention. We have sometimes seen a multiuser workload go 10% faster when setting this to 64 (down from 256), which seems counter-intuitive. This may be a <a class="auto-href" href="http://dbpedia.org/resource/Cache" id="link-id0x1e12b618">cache</a> artifact.</p> </li> </ul> <p></p> <p> <a href="http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ini_HTTPServer" id="link-id0x1f8960a0">In the <code>[HTTPServer]</code> section of the <code>virtuoso.ini</code> file</a>, the <b><code>ServerThreads</code></b> setting is the number of web server threads, i.e., the maximum number of concurrent <a class="auto-href" href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id0x17e4d690">SPARQL protocol</a> requests. Having a value larger than the number of concurrent clients is OK; for large numbers of concurrent clients a lower value may be better, which will result in requests waiting for a thread to be available.</p> <p>Note — The <code>[HTTPServer] ServerThreads</code> are taken from the total pool made available by the <code>[Parameters] ServerThreads</code>. Thus, the <code>[Parameters] ServerThreads</code> should always be at least as large as (and is best set greater than) the <code>[HTTPServer] ServerThreads</code>, and if using the closed-source Commercial Version, <code>[Parameters] ServerThreads</code> cannot exceed the licensed thread count. </p> </li> <li> <p> <b>File layout - Are there settings for striping over multiple devices? Settings for other file access parallelism? Settings for SSDs (e.g., SSD based cache of hot set of larger db files on disk)? The target config is for 4 independent disks and 4 independent SSDs. If you depend on RAID, are there settings for this? If you need RAID to be set up, please provide the settings/script for doing this with 4 SSDs on Linux (RH and Debian). This will be software RAID, as we find the hardware RAID to be much worse than an independent disk setup on the system in question.</b> </p> <p>It is best to stripe database files over all available disks, and to not use RAID. If RAID is desired, then stripe database files across many RAID sets. Use the <code>segment</code> declaration in the <code>virtuoso.ini</code> file. It is very important to give each independently seekable device its own I/O queue thread. See the documentation on the <a class="auto-href" href="http://www.tpc.org/" id="link-id0x1e9a6bc0">TPC</a>-<a class="auto-href" href="http://dbpedia.org/resource/C%2B%2B" id="link-id0x1ebdf210">C</a> sample for examples. </p> <p> <a href="http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ini_Parameters" id="link-id0x1f893f48">in the <code>[Parameters]</code> section of the <code>virtuoso.ini</code> file</a>, set <code>FDsPerFile</code> to be <code> (the number of concurrent threads * 1.5) ÷ the number of distinct database files</code>.</p> <p>There are no SSD specific settings.</p> </li> <li> <p> <b>Loading - How many parallel streams work best? We are looking for non-transactional bulk load, with no inference materialization. For partitioned cluster settings, do we divide the load streams over server processes? </b> </p> <p>Use one stream per core (not per core thread). In the case of a cluster, divide load streams evenly across all processes. The total number of streams on a cluster can equal the total number of cores; adjust up or down depending on what is observed.</p> <p>Use the built-in bulk load facility, i.e., </p> <blockquote> <code>ld_dir ('<source-filename-or-directory>', '<file name pattern>', '<destination graph iri>');</code> </blockquote> <p>For example,</p> <blockquote> <code><a class="auto-href" href="http://dbpedia.org/resource/SQL" id="link-id0x1dc52c58">SQL</a>> ld_dir ('/path/to/files', '*.n3', '<a class="auto-href" href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1e76bfc8">http</a>://<a class="auto-href" href="http://dbpedia.org/resource/DBpedia" id="link-id0x1e9a6ad8">dbpedia</a>.org');</code> </blockquote> <p>Then do a <code>rdf_loader_run ()</code> on enough connections. For example, you can use the shell command </p> <blockquote> <code>isql rdf_loader_run () &</code> </blockquote> <p>to start one in a background isql process. When starting background load commands from the shell, you can use the shell <code>wait</code> command to wait for completion. If starting from isql, use the <code>wait_for_children;</code> command (see <a href="http://docs.openlinksw.com/virtuoso/isql.html" id="link-id0x1ae0f230">isql documentation</a> for details). </p> <p>See the <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1d635820">BSBM disclosure report</a> for an example load script.</p> </li> <li> <p> <b>What command should be used after non-transactional bulk load, to ensure a consistent persistent state on disk, like a log checkpoint or similar? Load and checkpoint will be timed separately, load being <a class="auto-href" href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x1e6f1000">CPU</a>-bound and checkpoint being I/O-bound. No roll-forward log or similar is required; the load does not have to recover if it fails before the checkpoint.</b> </p> <p>Execute </p> <blockquote> <code> CHECKPOINT;</code> </blockquote> <p>through a SQL client, e.g., <code>isql</code>. This is not a <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x1c2401d8">SPARQL</a> statement and cannot be executed over the SPARQL protocol.</p> </li> <li> <p> <b>What settings should be used for trickle load of small triple sets into a pre-existing graph? This should be as transactional as supported; at least there should be a roll forward log, unlike the case for the bulk load.</b> </p> <p>No special settings are needed for load testing; defaults will produce transactional behavior with a roll forward log. Default transaction isolation is <b><code>REPEATABLE READ</code></b>, but this may be altered via SQL session settings or at Virtuoso server start-up through <a href="http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ini_Parameters" id="link-id0x1a791b80">the <code>[Parameters]</code> section of the <code>virtuoso.ini</code> file</a>, with</p> <blockquote> <b><code><a href="http://wikis.openlinksw.com/dataspace/owiki/wiki/VirtuosoWikiWeb/ChangeVirtuosoSDefaultTransactionIsolationLevel" id="link-id0x1e5536b8">DefaultIsolation</a> = 4</code> </b> </blockquote> <p> Transaction isolation cannot be set over the SPARQL protocol.</p> <p> NOTE: When testing full CRUD operations, other isolation settings may be preferable, due to <a class="auto-href" href="http://dbpedia.org/resource/ACID" id="link-id0x1ce6a310">ACID</a> considerations. See answer #12, below, and detailed discussion in part 8 of this series, <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1b7eb5f0">BSBM <i>Explore and Update</i></a>.</p> </li> <li> <p> <b>What settings control allocation of memory for database caching? We will be running mostly from memory, so we need to make sure that there is enough memory configured. </b> </p> <p> <a href="http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ini_Parameters" id="link-id0x1acd8fe8">In the <code>[Parameters]</code> section of the <code>virtuoso.ini</code> file</a>, <b><code>NumberOfBuffers</code></b> controls the amount of RAM used by Virtuoso to cache database files. One buffer caches an 8KB database page. In practice, count 10KB of memory per page. If "swappiness" on Linux is low (e.g., 2), two-thirds or more of physical memory can be used for database buffers. If swapping occurs, decrease the setting.</p> </li> <li> <p> <b>What command gives status on memory allocation (e.g., number of buffers, number of dirty buffers, etc.) so that we can verify that things are indeed in server memory and not, for example, being served from OS disk cache. If the cached format is different from the disk layout (e.g., decompression after disk read), is there a command for space statistics for database cache? </b> </p> <p>In an <code>isql</code> session, execute </p> <blockquote> <code>STATUS ( ? ? );</code> </blockquote> <p>The second result paragraph gives counts of total, used, and dirty buffers. If used buffers is steady and less than total, and if the disk read count on the line below does not increase, the system is running from memory. The cached format is the same as the disk based format.</p> </li> <li> <p> <b>What command gives <a class="auto-href" href="http://dbpedia.org/resource/Information" id="link-id0x1c185f28">information</a> on disk allocation for different things? We are looking for the total size of allocated database pages for quads (including table, indices, anything else associated with quads) and dictionaries for literals, IRI names, etc. If there is a text index on literals, what command gives space stats for this? We count used pages, excluding any preallocated unused pages or other gaps. There is one number for quads and another for the dictionaries or other such structures, optionally a third for text index.</b> </p> <p>Execute on an <code>isql</code> session: </p> <blockquote> <code><pre> CHECKPOINT; SELECT TOP 20 * FROM sys_index_space_stats ORDER BY iss_pages DESC; </pre> </code> </blockquote> <p>The <code>iss_pages</code> column is the total pages for each index, including blob pages. Pages are 8KB. Only used pages are reported, gaps and unused pages are not counted. The rows pertaining to <code>RDF_QUAD</code> are for quads; <code>RDF_IRI</code>, <code>RDF_PREFIX</code>, <code>RO_START</code>, <code>RDF_OBJ</code> are for dictionaries; <code>RDF_OBJ_RO_FLAGS_WORDS</code> and <code>VTLOG_DB_DBA_RDF_OBJ</code> are for text index. </p> </li> <li> <p> <b>If there is a choice between triples and quads, we will run with quads. How do we ascertain that the run is with quads? How do we find out the index scheme? Should be use an alternate index scheme? Most of the <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x1c573780">data</a> will be in a single big graph.</b> </p> <p>The default scheme uses quads. The default index layout is <code>PSOG</code>, <code>POGS</code>, <code>GS</code>, <code>SP</code>, <code>OP</code>. To see the current index scheme, use an <code>isql</code> session to execute</p> <blockquote> <code>STATISTICS DB.DBA.RDF_QUAD;</code> </blockquote> </li> <li> <p> <b>For partitioned cluster settings, are there partitioning-related settings to control even distribution of data between partitions? For example, is there a way to set partitioning by <code>S</code> or <code>O</code> depending on which is first in key order for each index? </b> </p> <p>The default partitioning settings are good, i.e., partitioning is on <code>O</code> or <code>S</code>, whichever is first in key order.</p> </li> <li> <p> <b>For partitioned clusters, are there settings to control message batching or similar? What are the statistics available for checking interconnect operation, e.g. message counts, latencies, total aggregate throughput of interconnect?</b> </p> <p> <a href="http://docs.openlinksw.com/virtuoso/clusteroperation.html#clusteroperationgeneralclusterinifields" id="link-id0x1ec6dff0">In the <code>[Cluster]</code> section of the <code>cluster.ini</code> file</a>, <b><code>ReqBatchSize</code></b> is the number of query states dispatched between cluster nodes per message round trip. This may be incremented from the default of <code>10000</code> to <code>50000</code> or so if this is seen to be useful. </p> <p>To change this on the fly, the following can be issued through an <code>isql</code> session:</p> <blockquote> <code>cl_exec ( ' __dbf_set (''cl_request_batch_size'', 50000) ' ); </code> </blockquote> <p>The commands below may be executed through an <code>isql</code> session to get a summary of CPU and message traffic for the whole cluster or process-by-process, respectively. The documentation <a href="http://docs.openlinksw.com/virtuoso/clusteroperation.html#clusteroperationadminstdispl" id="link-id0x1dfccec0">details the fields</a>. </p> <blockquote> <pre> <code>STATUS ('cluster') ;; whole cluster</code> <br /> <code>STATUS ('cluster_d') ;; process-by-process</code> </pre></blockquote> </li> <li> <p> <b>Other settings - Are there settings for limiting query planning, when appropriate? For example, the BSBM <i>Explore</i> mix has a large component of unnecessary query optimizer time, since the queries themselves access almost no data. Any other relevant settings?</b> </p> <ul> <li> <p>For BSBM, needless query <a class="auto-href" href="http://dbpedia.org/resource/Program_optimization" id="link-id0x1f0ffab8">optimization</a> should be capped at Virtuoso server start-up through the <code>[Parameters]</code> section of the <code>virtuoso.ini</code>, with</p> <blockquote> <b><code>StopCompilerWhenXOverRun = 1</code> </b> </blockquote> </li> <li> <p>When testing full CRUD operations (not simply CREATE, i.e., load, as discussed in #5, above), it is essential to make queries run with transaction isolation of <code>READ COMMITTED</code>, to remove most lock contention. Transaction isolation cannot be adjusted via SPARQL. This can be changed through SQL session settings, or at Virtuoso server start-up <a href="http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ini_Parameters" id="link-id0x1f3a43c8">through the <code>[Parameters]</code> section of the <code>virtuoso.ini</code> file</a>, with</p> <blockquote> <b><code><a href="http://wikis.openlinksw.com/dataspace/owiki/wiki/VirtuosoWikiWeb/ChangeVirtuosoSDefaultTransactionIsolationLevel" id="link-id0x1a5a51e0">DefaultIsolation</a> = 2</code> </b> </blockquote> </li> </ul> </li> </ol> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1d6e5428">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1c3ea770">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1efeca30">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1bda5158">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1ec74808">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1ea253a0">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1b02d528">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1ae81fc0">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x197515c0">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1a78db90">Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1d32ae10">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1e8fcc18">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1ae95050">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1dbf3158">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2011-03-14T17:56:38.000002-04:00
Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire
http://www.openlinksw.com:443/weblog/oerling/?date=2011-03-04#1665
2011-03-04T20:28:28Z
<p>Below is a questionnaire I sent to the <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0xa2f6798">BSBM</a> participants in order to get tuning instructions for the runs we were planning. I have filled in the answers for <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x195c2070">Virtuoso</a>, here. This can be a checklist for pretty much any <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1e1c9bb0">RDF</a> database tuning.</p> <ol> <li> <p> <b>Threading - What settings should be used (e.g., for query parallelization, I/O parallelization [e.g., prefetch, flush of dirty], thread pools [e,.g. web server], any other thread related)? We will run with 8 and 32 cores, so if there are settings controlling number of read/write (R/W) locks or mutexes or such for serializing diverse things, these should be set accordingly to minimize contention.</b> </p> <p>The following three settings are all <a href="http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ini_Parameters" id="link-id0x1ed4fe10">in the <code>[Parameters]</code> section of the <code>virtuoso.ini</code> file</a>. </p> <ul> <li> <p> <b><code>AsyncQueueMaxThreads</code> </b> controls the size of a pool of extra threads that can be used for query parallelization. This should be set to either <b>1.5 * the number of cores</b> or <b>1.5 * the number of core threads</b>; see which works better.</p> </li> <li> <p> <b><code>ThreadsPerQuery</code> </b> is the maximum number of threads a single query will take. This should be set to either <b>the number of cores</b> or <b>the number of core threads</b>; see which works better. </p> </li> <li> <p> <b><code>IndexTreeMaps</code> </b> is the number of mutexes over which control for buffering an index tree is split. This can generally be left at default (<b>256</b> in normal operation; valid settings are powers of 2 from 2 to 1024), but setting to <b>64, 128, or 512</b> may be beneficial.</p> <p>A low number will lead to frequent contention; upwards of 64 will have little contention. We have sometimes seen a multiuser workload go 10% faster when setting this to 64 (down from 256), which seems counter-intuitive. This may be a <a class="auto-href" href="http://dbpedia.org/resource/Cache" id="link-id0x1262b4b0">cache</a> artifact.</p> </li> </ul> <p></p> <p> <a href="http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ini_HTTPServer" id="link-id0x1f8960a0">In the <code>[HTTPServer]</code> section of the <code>virtuoso.ini</code> file</a>, the <b><code>ServerThreads</code></b> setting is the number of web server threads, i.e., the maximum number of concurrent <a class="auto-href" href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id0x17c1bef0">SPARQL protocol</a> requests. Having a value larger than the number of concurrent clients is OK; for large numbers of concurrent clients a lower value may be better, which will result in requests waiting for a thread to be available.</p> <p>Note — The <code>[HTTPServer] ServerThreads</code> are taken from the total pool made available by the <code>[Parameters] ServerThreads</code>. Thus, the <code>[Parameters] ServerThreads</code> should always be at least as large as (and is best set greater than) the <code>[HTTPServer] ServerThreads</code>, and if using the closed-source Commercial Version, <code>[Parameters] ServerThreads</code> cannot exceed the licensed thread count. </p> </li> <li> <p> <b>File layout - Are there settings for striping over multiple devices? Settings for other file access parallelism? Settings for SSDs (e.g., SSD based cache of hot set of larger db files on disk)? The target config is for 4 independent disks and 4 independent SSDs. If you depend on RAID, are there settings for this? If you need RAID to be set up, please provide the settings/script for doing this with 4 SSDs on Linux (RH and Debian). This will be software RAID, as we find the hardware RAID to be much worse than an independent disk setup on the system in question.</b> </p> <p>It is best to stripe database files over all available disks, and to not use RAID. If RAID is desired, then stripe database files across many RAID sets. Use the <code>segment</code> declaration in the <code>virtuoso.ini</code> file. It is very important to give each independently seekable device its own I/O queue thread. See the documentation on the <a class="auto-href" href="http://www.tpc.org/" id="link-id0x1e0deb38">TPC</a>-<a class="auto-href" href="http://dbpedia.org/resource/C%2B%2B" id="link-id0x1ddc2bf0">C</a> sample for examples. </p> <p> <a href="http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ini_Parameters" id="link-id0x1f893f48">in the <code>[Parameters]</code> section of the <code>virtuoso.ini</code> file</a>, set <code>FDsPerFile</code> to be <code> (the number of concurrent threads * 1.5) ÷ the number of distinct database files</code>.</p> <p>There are no SSD specific settings.</p> </li> <li> <p> <b>Loading - How many parallel streams work best? We are looking for non-transactional bulk load, with no inference materialization. For partitioned cluster settings, do we divide the load streams over server processes? </b> </p> <p>Use one stream per core (not per core thread). In the case of a cluster, divide load streams evenly across all processes. The total number of streams on a cluster can equal the total number of cores; adjust up or down depending on what is observed.</p> <p>Use the built-in bulk load facility, i.e., </p> <blockquote> <code>ld_dir ('<source-filename-or-directory>', '<file name pattern>', '<destination graph iri>');</code> </blockquote> <p>For example,</p> <blockquote> <code><a class="auto-href" href="http://dbpedia.org/resource/SQL" id="link-id0x17c854c0">SQL</a>> ld_dir ('/path/to/files', '*.n3', '<a class="auto-href" href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1f10c3d8">http</a>://<a class="auto-href" href="http://dbpedia.org/resource/DBpedia" id="link-id0x1c6378a0">dbpedia</a>.org');</code> </blockquote> <p>Then do a <code>rdf_loader_run ()</code> on enough connections. For example, you can use the shell command </p> <blockquote> <code>isql rdf_loader_run () &</code> </blockquote> <p>to start one in a background isql process. When starting background load commands from the shell, you can use the shell <code>wait</code> command to wait for completion. If starting from isql, use the <code>wait_for_children;</code> command (see <a href="http://docs.openlinksw.com/virtuoso/isql.html" id="link-id0x1ae0f230">isql documentation</a> for details). </p> <p>See the <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1d635820">BSBM disclosure report</a> for an example load script.</p> </li> <li> <p> <b>What command should be used after non-transactional bulk load, to ensure a consistent persistent state on disk, like a log checkpoint or similar? Load and checkpoint will be timed separately, load being <a class="auto-href" href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x1c522378">CPU</a>-bound and checkpoint being I/O-bound. No roll-forward log or similar is required; the load does not have to recover if it fails before the checkpoint.</b> </p> <p>Execute </p> <blockquote> <code> CHECKPOINT;</code> </blockquote> <p>through a SQL client, e.g., <code>isql</code>. This is not a <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x1c1e95b0">SPARQL</a> statement and cannot be executed over the SPARQL protocol.</p> </li> <li> <p> <b>What settings should be used for trickle load of small triple sets into a pre-existing graph? This should be as transactional as supported; at least there should be a roll forward log, unlike the case for the bulk load.</b> </p> <p>No special settings are needed for load testing; defaults will produce transactional behavior with a roll forward log. Default transaction isolation is <b><code>REPEATABLE READ</code></b>, but this may be altered via SQL session settings or at Virtuoso server start-up through <a href="http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ini_Parameters" id="link-id0x1a791b80">the <code>[Parameters]</code> section of the <code>virtuoso.ini</code> file</a>, with</p> <blockquote> <b><code><a href="http://wikis.openlinksw.com/dataspace/owiki/wiki/VirtuosoWikiWeb/ChangeVirtuosoSDefaultTransactionIsolationLevel" id="link-id0x1e5536b8">DefaultIsolation</a> = 4</code> </b> </blockquote> <p> Transaction isolation cannot be set over the SPARQL protocol.</p> <p> NOTE: When testing full CRUD operations, other isolation settings may be preferable, due to <a class="auto-href" href="http://dbpedia.org/resource/ACID" id="link-id0x1c592f70">ACID</a> considerations. See answer #12, below, and detailed discussion in part 8 of this series, <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1b7eb5f0">BSBM <i>Explore and Update</i></a>.</p> </li> <li> <p> <b>What settings control allocation of memory for database caching? We will be running mostly from memory, so we need to make sure that there is enough memory configured. </b> </p> <p> <a href="http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ini_Parameters" id="link-id0x1acd8fe8">In the <code>[Parameters]</code> section of the <code>virtuoso.ini</code> file</a>, <b><code>NumberOfBuffers</code></b> controls the amount of RAM used by Virtuoso to cache database files. One buffer caches an 8KB database page. In practice, count 10KB of memory per page. If "swappiness" on Linux is low (e.g., 2), two-thirds or more of physical memory can be used for database buffers. If swapping occurs, decrease the setting.</p> </li> <li> <p> <b>What command gives status on memory allocation (e.g., number of buffers, number of dirty buffers, etc.) so that we can verify that things are indeed in server memory and not, for example, being served from OS disk cache. If the cached format is different from the disk layout (e.g., decompression after disk read), is there a command for space statistics for database cache? </b> </p> <p>In an <code>isql</code> session, execute </p> <blockquote> <code>STATUS ( ? ? );</code> </blockquote> <p>The second result paragraph gives counts of total, used, and dirty buffers. If used buffers is steady and less than total, and if the disk read count on the line below does not increase, the system is running from memory. The cached format is the same as the disk based format.</p> </li> <li> <p> <b>What command gives <a class="auto-href" href="http://dbpedia.org/resource/Information" id="link-id0x11bf3008">information</a> on disk allocation for different things? We are looking for the total size of allocated database pages for quads (including table, indices, anything else associated with quads) and dictionaries for literals, IRI names, etc. If there is a text index on literals, what command gives space stats for this? We count used pages, excluding any preallocated unused pages or other gaps. There is one number for quads and another for the dictionaries or other such structures, optionally a third for text index.</b> </p> <p>Execute on an <code>isql</code> session: </p> <blockquote> <code><pre> CHECKPOINT; SELECT TOP 20 * FROM sys_index_space_stats ORDER BY iss_pages DESC; </pre> </code> </blockquote> <p>The <code>iss_pages</code> column is the total pages for each index, including blob pages. Pages are 8KB. Only used pages are reported, gaps and unused pages are not counted. The rows pertaining to <code>RDF_QUAD</code> are for quads; <code>RDF_IRI</code>, <code>RDF_PREFIX</code>, <code>RO_START</code>, <code>RDF_OBJ</code> are for dictionaries; <code>RDF_OBJ_RO_FLAGS_WORDS</code> and <code>VTLOG_DB_DBA_RDF_OBJ</code> are for text index. </p> </li> <li> <p> <b>If there is a choice between triples and quads, we will run with quads. How do we ascertain that the run is with quads? How do we find out the index scheme? Should be use an alternate index scheme? Most of the <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x17eb98f8">data</a> will be in a single big graph.</b> </p> <p>The default scheme uses quads. The default index layout is <code>PSOG</code>, <code>POGS</code>, <code>GS</code>, <code>SP</code>, <code>OP</code>. To see the current index scheme, use an <code>isql</code> session to execute</p> <blockquote> <code>STATISTICS DB.DBA.RDF_QUAD;</code> </blockquote> </li> <li> <p> <b>For partitioned cluster settings, are there partitioning-related settings to control even distribution of data between partitions? For example, is there a way to set partitioning by <code>S</code> or <code>O</code> depending on which is first in key order for each index? </b> </p> <p>The default partitioning settings are good, i.e., partitioning is on <code>O</code> or <code>S</code>, whichever is first in key order.</p> </li> <li> <p> <b>For partitioned clusters, are there settings to control message batching or similar? What are the statistics available for checking interconnect operation, e.g. message counts, latencies, total aggregate throughput of interconnect?</b> </p> <p> <a href="http://docs.openlinksw.com/virtuoso/clusteroperation.html#clusteroperationgeneralclusterinifields" id="link-id0x1ec6dff0">In the <code>[Cluster]</code> section of the <code>cluster.ini</code> file</a>, <b><code>ReqBatchSize</code></b> is the number of query states dispatched between cluster nodes per message round trip. This may be incremented from the default of <code>10000</code> to <code>50000</code> or so if this is seen to be useful. </p> <p>To change this on the fly, the following can be issued through an <code>isql</code> session:</p> <blockquote> <code>cl_exec ( ' __dbf_set (''cl_request_batch_size'', 50000) ' ); </code> </blockquote> <p>The commands below may be executed through an <code>isql</code> session to get a summary of CPU and message traffic for the whole cluster or process-by-process, respectively. The documentation <a href="http://docs.openlinksw.com/virtuoso/clusteroperation.html#clusteroperationadminstdispl" id="link-id0x1dfccec0">details the fields</a>. </p> <blockquote> <pre> <code>STATUS ('cluster') ;; whole cluster</code> <br /> <code>STATUS ('cluster_d') ;; process-by-process</code> </pre></blockquote> </li> <li> <p> <b>Other settings - Are there settings for limiting query planning, when appropriate? For example, the BSBM <i>Explore</i> mix has a large component of unnecessary query optimizer time, since the queries themselves access almost no data. Any other relevant settings?</b> </p> <ul> <li> <p>For BSBM, needless query <a class="auto-href" href="http://dbpedia.org/resource/Program_optimization" id="link-id0x11be47b8">optimization</a> should be capped at Virtuoso server start-up through the <code>[Parameters]</code> section of the <code>virtuoso.ini</code>, with</p> <blockquote> <b><code>StopCompilerWhenXOverRun = 1</code> </b> </blockquote> </li> <li> <p>When testing full CRUD operations (not simply CREATE, i.e., load, as discussed in #5, above), it is essential to make queries run with transaction isolation of <code>READ COMMITTED</code>, to remove most lock contention. Transaction isolation cannot be adjusted via SPARQL. This can be changed through SQL session settings, or at Virtuoso server start-up <a href="http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ini_Parameters" id="link-id0x1f3a43c8">through the <code>[Parameters]</code> section of the <code>virtuoso.ini</code> file</a>, with</p> <blockquote> <b><code><a href="http://wikis.openlinksw.com/dataspace/owiki/wiki/VirtuosoWikiWeb/ChangeVirtuosoSDefaultTransactionIsolationLevel" id="link-id0x1a5a51e0">DefaultIsolation</a> = 2</code> </b> </blockquote> </li> </ul> </li> </ol> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1d6e5428">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1c3ea770">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1efeca30">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1bda5158">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1ec74808">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1ea253a0">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1b02d528">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1ae81fc0">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x197515c0">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1a78db90">Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1681" id="link-id0x2aabe7f48408">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1682" id="link-id0x2aac0351e488">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1683" id="link-id0x2aac1428d5e8">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1684" id="link-id0x2aac00a80338">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2014-08-21T10:46:06.565138-04:00
Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore
http://www.openlinksw.com:443/blog/vdb/blog/?date=2011-03-02#1664
2011-03-02T23:23:16Z
<p>In this post I will summarize the figures for <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1dcf58f8">BSBM</a> Load and <i>Explore</i> mixes at 100 Mt, 200 Mt, and 1000 Mt. (1 Mt = 1 Megatriple, or one million triples.) The measurements were made on a 72GB 2xXeon 5520 with 4 SSDs. The exact specifications and configurations are in the raw reports to follow.</p> <p>The load time in <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/results/V6/index.html" id="link-id0x1f3716d8">the recent Berlin report</a> was measured with <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/results/V6/index.html#resultsExplore" id="link-id0x1dd37f80">the wrong function</a>, and so far as we can tell, without multiple threads. The intermediate cut of <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1ddb0c90">Virtuoso</a> they tested also <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/results/V6/index.html#resultsExploreAndUpdate" id="link-id0x1e5fcf40"> had broken</a> <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x1e1d2b70">SPARQL</a>/<a class="auto-href" href="http://dbpedia.org/page/SPARUL" id="link-id0x1bfb00c0">Update</a> (also known as <a class="auto-href" href="http://dbpedia.org/page/SPARUL" id="link-id0x1e0d5fd8">SPARUL</a>) features. We have fixed this since, and give <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/BenchmarksReduxSupportingFiles/results.zip" id="link-id0x1edf36b0">here the right numbers</a>.</p> <p>In the course of the discussion to follow, we talk about 3 different kinds of Virtuoso:</p> <ul> <li> <p> <i>6 Single</i> is the generally available single server configuration of Virtuoso. Whether this is open source or not does not make a difference.</p> </li> <li> <p> <i>6 Cluster</i> is the generally available commercial only cluster-capable Virtuoso.</p> </li> <li> <p> <i>7 Single</i> is the next generation single server Virtuoso, about to be released as a preview.</p> </li> </ul> <p>To understand the numbers, we must explain how these differ from each other in execution:</p> <ul> <li> <p> <i>6 Single</i> has one thread-per-query, and operates on one state of the query at a time.</p> </li> <li> <p> <i>6 Cluster</i> has one thread-per-query-per-process, and between processes it operates on batches of some tens-of-thousands of simultaneous query states. Within each node, these batches run through the execution pipeline one state at a time. Aggregation is distributed, and the query optimizer is generally smart about shipping colocated functions together.</p> </li> <li> <p> <i>7 Single</i> has multiple threads-per-query and in all situations operates on batches of 10,000 or more simultaneous query states. This means, for example, that index lookups get large numbers of parameters which then are sorted to get an ascending search pattern which benefits from locality, so the <code>n * log(n)</code> index access for the batch becomes more like linear if the <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x1ceca188">data</a> accessed has any locality. Furthermore, if there are many operands to an operator, these can be split on multiple threads. Also, scans of consecutive rows can be split before the scan on multiple threads, each doing a range of the scan. These features are called <i>vectored execution</i> and <i>query parallelization</i>. These techniques will also be applied to the cluster variant in due time.</p> </li> </ul> <p>The version 6 and 7 variants discussed here use the same physical storage layout with row-wise <a class="auto-href" href="http://dbpedia.org/resource/Data_compression" id="link-id0x1e521fa0">key compression</a>. Additionally, there exists a column-wise storage option in 7 that can fit 4x the number of quads in the same space. This column store option is not used here because it still has some problems with random order inserts.</p> <p> We will first consider loading. Below are the load times and rates for 7 at each scale.</p> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="4" align="center">7 Single</th> </tr> <tr> <th align="center">Scale</th> <th align="center">Rate <br /> (quads per second)</th> <th align="center">Load time <br /> (seconds)</th> <th align="center">Checkpoint time <br /> (seconds)</th> </tr> <tr> <th align="center">100 Mt</th> <td align="center"> 261,366 </td> <td align="center"> 301 </td> <td align="center"> 82 </td> </tr> <tr> <th align="center">200 Mt</th> <td align="center"> 216,000 </td> <td align="center"> 802 </td> <td align="center"> 123 </td> </tr> <tr> <th align="center">1000 Mt</th> <td align="center"> 130,378 </td> <td align="center"> 6641 </td> <td align="center"> 1012 </td> </tr> </table> <p>In each case the load was made on 8 concurrent streams, each reading a file from a pool of 80 files for the two smaller scales and 360 files for the larger scale.</p> <p>We also loaded the smallest data set with 6 Single using the same load script. </p> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="4" align="center">6 Single</th> </tr> <tr> <th align="center">Scale</th> <th align="center">Rate <br /> (quads per second)</th> <th align="center">Load time <br /> (seconds)</th> <th align="center">Checkpoint time <br /> (seconds)</th> </tr> <tr> <th align="center">100 Mt</th> <td align="center"> 74,713 </td> <td align="center"> 1192 </td> <td align="center"> 145 </td> </tr> </table> <p> <a class="auto-href" href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x1132ad18">CPU</a> time with 6 Single was 8047 seconds. We compare this to 4453 seconds of CPU for the same load on 7 Single. The CPU% during the run was on either side of 700% for 6 Single and 1300% for 7 Single. Note that high percentages involve core threads, not real cores. </p> <p>The difference is mostly attributable to vectoring and the introduction of a non-transactional insert. The 6 Single inserts transactionally but makes very frequent commits and writes no log, resulting in <i>de facto</i> non-transactional behavior but still there is a lock and commit cycle. Inserts in <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1c750368">RDF</a> load usually exhibit locality on all SPOG. Sorting by value gives ascending insert order and eliminates much of the lookup time for deciding where the next row will go. Contention on page read-write locks is less because the engine stays longer on a page, inserting multiple values in one go, instead of re-acquiring the read-write lock and possible transaction locks for each row.</p> <p>Furthermore, for single stream loading the non-transactional mode can serve one thread doing the parsing with many threads doing the inserting; hence, in practice the speed is bounded by the parsing speed. In multi-stream load this parallelization also happens but is less significant, as adding threads past the count of core threads is not useful. Writes are all in-place, and no delta-merge mechanism is involved. For transactional inserts, the uncommitted rows are not visible to read-committed readers, which do not block. Repeatable and serializable readers would block before an uncommitted insert.</p> <p>Now for the run (larger numbers indicate more queries executed, and are therefore better):</p> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="3" align="center"> 6 Single Throughput <br /> (QMpH, query mixes per hour) </th> </tr> <tr> <th align="center">Scale</th> <th align="center"> Single User </th> <th align="center"> 16 User </th> </tr> <tr> <th align="center">100 Mt</th> <td align="center"> 7641 </td> <td align="center"> 29433 </td> </tr> <tr> <th align="center">200 Mt</th> <td align="center"> 6017 </td> <td align="center"> 13335 </td> </tr> <tr> <th align="center">1000 Mt</th> <td align="center"> 1770 </td> <td align="center"> 2487 </td> </tr> </table> <br /> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="3" align="center"> 7 Single Throughput <br /> (QMpH, query mixes per hour) </th> </tr> <tr> <th align="center">Scale</th> <th align="center"> Single User </th> <th align="center"> 16 User </th> </tr> <tr> <th align="center">100 Mt</th> <td align="center"> 11742 </td> <td align="center"> 72278 </td> </tr> <tr> <th align="center">200 Mt</th> <td align="center"> 10225 </td> <td align="center"> 60951 </td> </tr> <tr> <th align="center">1000 Mt</th> <td align="center"> 6262 </td> <td align="center"> 24672 </td> </tr> </table> <p>The 100 Mt and 200 Mt runs are entirely in memory; the 1000 Mt run is mostly in memory, with about a 1.6 MB/s trickle from SSD in steady state. Accordingly, the 1000 Mt run is longer, with 2000 query mixes in the timed period, preceded by a warm-up of 2000 mixes with a different seed. For the memory-only scales, we run 500 mixes twice, and take the timing of the second run.</p> <p>Looking at single user speeds, 6 Single and 7 Single are closest at the small end and drift farther apart at the larger scales. This comes from the increased opportunity to parallelize Q5, since this works on more data and is relatively more important as the scale gets larger. The 100 Mt run of 7 Single has about 130% CPU, and the 1000 Mt run has about 270%. This also explains why adding clients gives a larger boost at the smaller scale. </p> <p>Now let us look at the relative effects of parallelizing and vectoring in 7 Single. We run 50 mixes of Single User <i>Explore</i>: 6132 QMpH with both parallelizing and vectoring on; 2805 QMpH with execution limited to a single thread. Then we set the vector size to 1, meaning that the query pipeline runs one row at a time. This gets us 1319 QMpH which is a bit worse than 6 Single. This is to be expected since there is some overhead to running vectored with single-element vectors. Q5 on 7 Single with vectoring and a single thread runs at 1.9 qps; with single-element vectors, at 0.8 qps. The 6 Single engine runs Q5 at 1.13 qps.</p> <p>The 100 Mt scale 7 Single gains the most from adding clients; the 1000 Mt 6 Single gains the least. The reason for the latter is covered in detail in <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1b9ed390">A Benchmarking Story</a>. We note that while vectoring is primarily geared to better single-thread speed and better <a class="auto-href" href="http://dbpedia.org/resource/Cache" id="link-id0x1ddc2f48">cache</a> hit rates, it delivers a huge multithreaded benefit by eliminating the mutex contention at the index tree top which stops 6 Single dead at 1000 Mt.</p> <p>In conclusion, we see that even with a workload of short queries and little opportunity for parallelism, we get substantial benefits from query parallelization and vectoring. When moving to more complex workloads, the benefits become more pronounced. For a single user complex query load, we can get 7x speed-up from parallelism (8 core), plus up to 3x from vectoring. These numbers do not take into account the benefits of the column store; those will be analyzed separately a bit later.</p> <p>The full run details will be supplied at the end of this <a class="auto-href" href="http://dbpedia.org/resource/Blog" id="link-id0x1e9f6960">blog</a> series.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1d0bb988">Benchmarks, Redux (part 1): On RDF Benchmarks </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x155fc700">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li>Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x1d96e218">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1d7a5170">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1def9ca0">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1a7a7800">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1e9c6c68">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1e80c208">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x1dafd290">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1f34f7f8">Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1df24f50">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1f4b19c8">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1de90cf8">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1ebefbe8">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2011-03-14T17:16:56-04:00
Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore
http://www.openlinksw.com:443/weblog/oerling/?date=2011-03-02#1663
2011-03-02T23:23:16Z
<p>In this post I will summarize the figures for <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1edb1dd0">BSBM</a> Load and <i>Explore</i> mixes at 100 Mt, 200 Mt, and 1000 Mt. (1 Mt = 1 Megatriple, or one million triples.) The measurements were made on a 72GB 2xXeon 5520 with 4 SSDs. The exact specifications and configurations are in the raw reports to follow.</p> <p>The load time in <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/results/V6/index.html" id="link-id0x1f3716d8">the recent Berlin report</a> was measured with <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/results/V6/index.html#resultsExplore" id="link-id0x1dd37f80">the wrong function</a>, and so far as we can tell, without multiple threads. The intermediate cut of <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1c1c7798">Virtuoso</a> they tested also <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/results/V6/index.html#resultsExploreAndUpdate" id="link-id0x1e5fcf40"> had broken</a> <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x1bfa40b8">SPARQL</a>/<a class="auto-href" href="http://dbpedia.org/page/SPARUL" id="link-id0x1c1e1320">Update</a> (also known as <a class="auto-href" href="http://dbpedia.org/page/SPARUL" id="link-id0x1ddc87d8">SPARUL</a>) features. We have fixed this since, and give <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/BenchmarksReduxSupportingFiles/results.zip" id="link-id0x1edf36b0">here the right numbers</a>.</p> <p>In the course of the discussion to follow, we talk about 3 different kinds of Virtuoso:</p> <ul> <li> <p> <i>6 Single</i> is the generally available single server configuration of Virtuoso. Whether this is open source or not does not make a difference.</p> </li> <li> <p> <i>6 Cluster</i> is the generally available commercial only cluster-capable Virtuoso.</p> </li> <li> <p> <i>7 Single</i> is the next generation single server Virtuoso, about to be released as a preview.</p> </li> </ul> <p>To understand the numbers, we must explain how these differ from each other in execution:</p> <ul> <li> <p> <i>6 Single</i> has one thread-per-query, and operates on one state of the query at a time.</p> </li> <li> <p> <i>6 Cluster</i> has one thread-per-query-per-process, and between processes it operates on batches of some tens-of-thousands of simultaneous query states. Within each node, these batches run through the execution pipeline one state at a time. Aggregation is distributed, and the query optimizer is generally smart about shipping colocated functions together.</p> </li> <li> <p> <i>7 Single</i> has multiple threads-per-query and in all situations operates on batches of 10,000 or more simultaneous query states. This means, for example, that index lookups get large numbers of parameters which then are sorted to get an ascending search pattern which benefits from locality, so the <code>n * log(n)</code> index access for the batch becomes more like linear if the <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x1ea197c8">data</a> accessed has any locality. Furthermore, if there are many operands to an operator, these can be split on multiple threads. Also, scans of consecutive rows can be split before the scan on multiple threads, each doing a range of the scan. These features are called <i>vectored execution</i> and <i>query parallelization</i>. These techniques will also be applied to the cluster variant in due time.</p> </li> </ul> <p>The version 6 and 7 variants discussed here use the same physical storage layout with row-wise <a class="auto-href" href="http://dbpedia.org/resource/Data_compression" id="link-id0x1bd035c0">key compression</a>. Additionally, there exists a column-wise storage option in 7 that can fit 4x the number of quads in the same space. This column store option is not used here because it still has some problems with random order inserts.</p> <p> We will first consider loading. Below are the load times and rates for 7 at each scale.</p> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="4" align="center">7 Single</th> </tr> <tr> <th align="center">Scale</th> <th align="center">Rate <br /> (quads per second)</th> <th align="center">Load time <br /> (seconds)</th> <th align="center">Checkpoint time <br /> (seconds)</th> </tr> <tr> <th align="center">100 Mt</th> <td align="center"> 261,366 </td> <td align="center"> 301 </td> <td align="center"> 82 </td> </tr> <tr> <th align="center">200 Mt</th> <td align="center"> 216,000 </td> <td align="center"> 802 </td> <td align="center"> 123 </td> </tr> <tr> <th align="center">1000 Mt</th> <td align="center"> 130,378 </td> <td align="center"> 6641 </td> <td align="center"> 1012 </td> </tr> </table> <p>In each case the load was made on 8 concurrent streams, each reading a file from a pool of 80 files for the two smaller scales and 360 files for the larger scale.</p> <p>We also loaded the smallest data set with 6 Single using the same load script. </p> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="4" align="center">6 Single</th> </tr> <tr> <th align="center">Scale</th> <th align="center">Rate <br /> (quads per second)</th> <th align="center">Load time <br /> (seconds)</th> <th align="center">Checkpoint time <br /> (seconds)</th> </tr> <tr> <th align="center">100 Mt</th> <td align="center"> 74,713 </td> <td align="center"> 1192 </td> <td align="center"> 145 </td> </tr> </table> <p> <a class="auto-href" href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x1c0b96c0">CPU</a> time with 6 Single was 8047 seconds. We compare this to 4453 seconds of CPU for the same load on 7 Single. The CPU% during the run was on either side of 700% for 6 Single and 1300% for 7 Single. Note that high percentages involve core threads, not real cores. </p> <p>The difference is mostly attributable to vectoring and the introduction of a non-transactional insert. The 6 Single inserts transactionally but makes very frequent commits and writes no log, resulting in <i>de facto</i> non-transactional behavior but still there is a lock and commit cycle. Inserts in <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1ddef3e8">RDF</a> load usually exhibit locality on all SPOG. Sorting by value gives ascending insert order and eliminates much of the lookup time for deciding where the next row will go. Contention on page read-write locks is less because the engine stays longer on a page, inserting multiple values in one go, instead of re-acquiring the read-write lock and possible transaction locks for each row.</p> <p>Furthermore, for single stream loading the non-transactional mode can serve one thread doing the parsing with many threads doing the inserting; hence, in practice the speed is bounded by the parsing speed. In multi-stream load this parallelization also happens but is less significant, as adding threads past the count of core threads is not useful. Writes are all in-place, and no delta-merge mechanism is involved. For transactional inserts, the uncommitted rows are not visible to read-committed readers, which do not block. Repeatable and serializable readers would block before an uncommitted insert.</p> <p>Now for the run (larger numbers indicate more queries executed, and are therefore better):</p> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="3" align="center"> 6 Single Throughput <br /> (QMpH, query mixes per hour) </th> </tr> <tr> <th align="center">Scale</th> <th align="center"> Single User </th> <th align="center"> 16 User </th> </tr> <tr> <th align="center">100 Mt</th> <td align="center"> 7641 </td> <td align="center"> 29433 </td> </tr> <tr> <th align="center">200 Mt</th> <td align="center"> 6017 </td> <td align="center"> 13335 </td> </tr> <tr> <th align="center">1000 Mt</th> <td align="center"> 1770 </td> <td align="center"> 2487 </td> </tr> </table> <br /> <table border="1" cellspacing="2" cellpadding="2" align="center" width="90%"> <tr> <th colspan="3" align="center"> 7 Single Throughput <br /> (QMpH, query mixes per hour) </th> </tr> <tr> <th align="center">Scale</th> <th align="center"> Single User </th> <th align="center"> 16 User </th> </tr> <tr> <th align="center">100 Mt</th> <td align="center"> 11742 </td> <td align="center"> 72278 </td> </tr> <tr> <th align="center">200 Mt</th> <td align="center"> 10225 </td> <td align="center"> 60951 </td> </tr> <tr> <th align="center">1000 Mt</th> <td align="center"> 6262 </td> <td align="center"> 24672 </td> </tr> </table> <p>The 100 Mt and 200 Mt runs are entirely in memory; the 1000 Mt run is mostly in memory, with about a 1.6 MB/s trickle from SSD in steady state. Accordingly, the 1000 Mt run is longer, with 2000 query mixes in the timed period, preceded by a warm-up of 2000 mixes with a different seed. For the memory-only scales, we run 500 mixes twice, and take the timing of the second run.</p> <p>Looking at single user speeds, 6 Single and 7 Single are closest at the small end and drift farther apart at the larger scales. This comes from the increased opportunity to parallelize Q5, since this works on more data and is relatively more important as the scale gets larger. The 100 Mt run of 7 Single has about 130% CPU, and the 1000 Mt run has about 270%. This also explains why adding clients gives a larger boost at the smaller scale. </p> <p>Now let us look at the relative effects of parallelizing and vectoring in 7 Single. We run 50 mixes of Single User <i>Explore</i>: 6132 QMpH with both parallelizing and vectoring on; 2805 QMpH with execution limited to a single thread. Then we set the vector size to 1, meaning that the query pipeline runs one row at a time. This gets us 1319 QMpH which is a bit worse than 6 Single. This is to be expected since there is some overhead to running vectored with single-element vectors. Q5 on 7 Single with vectoring and a single thread runs at 1.9 qps; with single-element vectors, at 0.8 qps. The 6 Single engine runs Q5 at 1.13 qps.</p> <p>The 100 Mt scale 7 Single gains the most from adding clients; the 1000 Mt 6 Single gains the least. The reason for the latter is covered in detail in <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1b9ed390">A Benchmarking Story</a>. We note that while vectoring is primarily geared to better single-thread speed and better <a class="auto-href" href="http://dbpedia.org/resource/Cache" id="link-id0x1ddc2f78">cache</a> hit rates, it delivers a huge multithreaded benefit by eliminating the mutex contention at the index tree top which stops 6 Single dead at 1000 Mt.</p> <p>In conclusion, we see that even with a workload of short queries and little opportunity for parallelism, we get substantial benefits from query parallelization and vectoring. When moving to more complex workloads, the benefits become more pronounced. For a single user complex query load, we can get 7x speed-up from parallelism (8 core), plus up to 3x from vectoring. These numbers do not take into account the benefits of the column store; those will be analyzed separately a bit later.</p> <p>The full run details will be supplied at the end of this <a class="auto-href" href="http://dbpedia.org/resource/Blog" id="link-id0x1e9f69f0">blog</a> series.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1d0bb988">Benchmarks, Redux (part 1): On RDF Benchmarks </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x155fc700">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li>Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x1d96e218">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1d7a5170">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1def9ca0">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1a7a7800">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1e9c6c68">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1e80c208">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x1dafd290">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1f34f7f8">Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1681" id="link-id0x2aac193e9828">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1682" id="link-id0x2aabdc4b7a98">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1683" id="link-id0x2aabf5018ad8">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1684" id="link-id0x2aabdf3496b8">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2014-08-21T10:46:13.751710-04:00
Data Spaces
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2011-03-01#1662
2011-03-01T23:49:26Z
<p>There is increasing coalescence around the idea that HTTP-based <a class="auto-href" href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1e93cbd0">Linked Data</a> adds a tangible dimension to the <a class="auto-href" href="http://dbpedia.org/resource/World_Wide_Web" id="link-id0x1dfdde10">World Wide Web</a> (<a href="http://dbpedia.org/resource/World_Wide_Web">Web</a>). This <i><a href="http://dbpedia.org/resource/Data">Data</a> Dimension</i> grants end-users, power-users, integrators, and developers the ability to experience the Web not solely as a <i><a class="auto-href" href="http://dbpedia.org/resource/Information" id="link-id0x19d02b00">Information</a> Space</i> or <i>Document Space,</i> but now also as a <i><a class="auto-href" href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id0x1ac33378">Data Space</a>.</i> </p> <p>Here is a simple What and Why guide covering the essence of Data Spaces.</p> <h2>What is a Data Space?</h2> <p>A Data Space is a point of presence on a network, where every <i>Data Object</i> (item or <a class="auto-href" href="http://dbpedia.org/resource/Entity" id="link-id0x1d55f910">entity</a>) is given a <i>Name</i> (e.g., a <a class="auto-href" href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x1736ea28">URI</a>) by which it may be Referenced or Identified. </p> <p>In a Data Space, every <i>Representation</i> of those Data Objects (i.e., every <i>Object Representation</i>) has an <i>Address</i> (e.g., a <a class="auto-href" href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id0x1f17f5a8">URL</a>) from which it may be Retrieved (or "gotten").</p> <p>In a Data Space, every Object Representation is a time variant (that is, it changes over time), streamable, and format-agnostic <i>Resource.</i> </p> <p>An Object Representation is simply a Description of that Object. It takes the form of a graph, pictorially constructed from sets of 3 elements which are themselves named <i>Subject,</i> <i>Predicate,</i> and <i>Object</i> (or <i>SPO</i>); or <i>Entity,</i> <i>Attribute,</i> and <i>Value</i> (or <i>EAV</i>). Each <a class="auto-href" href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id0x1dedcfe0">Entity</a>+Attribute+Value or Subject+Predicate+Object set (or <i>triple</i>), is one datum, one piece of data, one persisted observation about a given Subject or Entity.</p> <p>The underlying Schema that defines and constrains the construction of Object Representations is based on Logic, specifically <i>First-Order Logic</i>. Each Object Representation is a collection of persisted observations (<i>Data</i>) about a given Subject, which aid observers in materializing their perception (<i>Information</i>), and ultimately comprehension (<i><a class="auto-href" href="http://dbpedia.org/resource/Knowledge" id="link-id0x1a4c7bf8">Knowledge</a></i>), of that Subject.</p> <h2>Why are Data Spaces important?</h2> <p>In the real-world -- which is networked by nature -- data is heterogeneously (or "differently") shaped, and disparately located. </p> <p>Data has been increasing at an alarming rate since the advent of computing; the interWeb simply provides <a class="auto-href" href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id0x1ad97358">context</a> that makes this reality more palpable and more exploitable, and in the process virtuously ups the ante through increasingly exponential growth rates.</p> <p>We can't stop data heterogeneity; it is endemic to the nature of its producers -- humans and/or human-directed machines. What we can do, though, is create a powerful Conceptual-level "bus" or "interface" for data integration, based on <i>Data Description oriented Logic</i> rather than Data Representation oriented Formats. Basically, it's possible for us to use a <i><a href="http://en.wikipedia.org/wiki/First-order_predicate_logic" id="link-id0x1a481248">Common Logic</a></i> as the basis for expressing and blending SPO- or EAV-based Object Representations in a variety of Formats (or "dialects").</p> <p>The roadmap boils down to:</p> <ol> <li> <p>Assigning unambiguous Object Names to:</p> <ul> <li> <p>Every record (or, in table terms, every row); </p> </li> <li> <p>Every record attribute (or, in table terms, every field or column);</p> </li> <li> <p>Every record relationship (that is, every relationship between one record and another);</p> </li> <li> <p>Every record container (e.g., every table or view in a relational database, every named graph, every spreadsheet, every text file, etc.);</p> </li> </ul> </li> <li> <p>Making each Object Name resolve to an Address through which Create, Read, Update, and Delete ("CRUD") operations can be performed against (can <i>access</i>) the associated Object Representation graph.</p> </li> </ol>
2011-03-01T17:26:15-05:00
Benchmarks, Redux (part 2): A Benchmarking Story
http://www.openlinksw.com:443/blog/vdb/blog/?date=2011-02-28#1661
2011-02-28T21:12:28Z
<blockquote> <i>Caeterum censeo, benchmarks are for vendors...</i> </blockquote> <p>This is an edifying story about benchmarks and how databases work. I will show how one detail makes a 5+x difference, and how one really must understand how things work in order to make sense of benchmarks.</p> <p>We begin right after the publication of the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/results/V6/index.html" id="link-id0x1df843f8">recent Berlin report</a>. This report gives us OK performance for queries and very bad performance for loading. Trickle updates were not measurable. This comes as a consequence of testing intermediate software cuts and having incomplete instructions for operating them. I will cover the whole <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1d0b6ea0">BSBM</a> matter and the general benchmarking question in forthcoming posts; for now, let's talk about specifics.</p> <p>In the course of the discussion to follow, we talk about 3 different kinds of <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1e09ee88">Virtuoso</a>:</p> <ul> <li> <p> <i>6 Single</i> is the generally available single-instance-server configuration of Virtuoso. Whether this is open source or not does not make a difference.</p> </li> <li> <p> <i>6 Cluster</i> is the generally available, commercial-only, cluster-capable Virtuoso.</p> </li> <li> <p> <i>7 Single</i> is the next-generation single-instance-server Virtuoso, about to be released as a preview.</p> </li> </ul> <p>We began by running the various parts of BSBM at different scales with different Virtuoso variants. In so doing, we noticed that the BSBM <i>Explore</i> mix at one scale got better throughput as we added more clients, approximately as one would expect based on <a class="auto-href" href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x1c1b4860">CPU</a> usage and number of cores, while at another scale this was not so.</p> <p>At the 1-billion-triple scale (1000 Mt; 1 Mt = 1 Megatriple, or one million triples) we saw CPU going from 200% with 1 client to 1400% with 16 clients but throughput increased by less than 20%. </p> <p>When we ran the same scale with our shared-nothing 6 Cluster, running 8 processes on the same box, throughput increased normally with the client count. We have not previously tried BSBM with 6 Cluster simply because there is little to gain and a lot to lose by distributing this workload. But here we got a multiuser throughput with 6 Cluster that is easily 3 times that of the single server, even with a cluster-unfriendly workload. </p> <p> See, sometimes scaling out even within a shared memory multiprocessor pays! Still, what we saw was rather anomalous.</p> <p>Over the years we have looked at performance any number of times and have a lot of built-in meters. For cases of high CPU with no throughput, the prime suspect is contention on critical sections. Quite right, when building with the mutex meter enabled, counting how many times each mutex is acquired and how many times this results in a wait, we found a mutex which gets acquired 600M times in the run, of which an insane 450M result in a wait. One can count a microsecond of real time each time a mutex wait results in the kernel switching tasks. The run took 500 s or so, of which 450 s of real time were attributable to the overhead of waiting for this one mutex.</p> <p>Waiting for a mutex is a real train wreck. We have tried spinning a few times before it, which the OS does anyhow, but this does not help. Using spin locks is good only if waits are extremely rare; with any frequency of waiting, even for very short waits, a mutex is still a lot better.</p> <p>Now, the mutex in question happens to serialize the buffer <a class="auto-href" href="http://dbpedia.org/resource/Cache" id="link-id0x1e542088">cache</a> for one specific page of <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x17c853d8">data</a>, one level down from the root of the index for <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1d64a658">RDF</a> PSOG. By the luck of the draw, the Ps falling on that page are commonly accessed Ps pertaining to product features. In order to get any product feature value, one must pass via this page. At the smaller scale, the different properties web their different ways based on the index root.</p> <p>One might here ask why the problem is one level down from the root and not in the root. The index root is already handled specially, so the read-write locks for buffers usually apply only for the first level down. One might also ask why have a mutex in the first place. Well, unless one is read-only and all in memory, there simply must be a way to say that a buffer must not get written to by one thread while another is reading it. Same for cache replacement. Some in-memory people fork a whole copy of the database process to do a large query and so can forget about serialization. But one must have long queries for this and have all in memory. One can make writes less frequent by keeping deltas, but this does not remove the need to merge the deltas at some point, which cannot happen without serializing this with the readers.</p> <p>Most of the time the offending mutex is acquired for getting a property of a product in Q5, the one that looks for products with similar values of a numeric property. We retrieve this property for a number of products in one go, due to vectoring. Vectoring is supposed to save us from constantly hitting the index tree top when getting the next match. So how come there is contention in the index tree top? As it happens, the vectored index lookup checks for locality only when all search conditions on key parts are equalities. Here however there is equality on P and S and a range on O; hence, the lookup starts from the index root every time.</p> <p>So I changed this. The effect was Q5 getting over twice as fast, with the single user throughput at 1000 Mt going from 2000 to 5200 QMpH (Query Mixes per Hour) and the 16-user throughput going from 3800 to over 21000 QMpH. The previously "good" throughput of 40K QMpH at 100 Mt went to 66K QMpH. </p> <p>Vectoring can make a real difference. The throughputs for the same workload on 6 Single, without vectoring, thus unavoidably hitting the page with the crazy contention, are 1770 QMpH single user and 2487 QMpH with 16 users. The 6 Cluster throughput, avoiding the contention but without the increased locality from vectoring and with the increased latency of going out-of-process for most of the data, was about 11.5K QMpH with 16 users. Each partition had a page getting the hits but since the partitioning was on S and S was about-evenly distributed, each partition got 1/8 of the load; thus waiting on the mutex did not become a killer issue. </p> <p>We see how detailed analysis of benchmarks can lead to almost an order of magnitude improvements in a short time. This analysis is however both difficult and tedious. It is not readily delegable; one needs real <a class="auto-href" href="http://dbpedia.org/resource/Knowledge" id="link-id0x1e7249d0">knowledge</a> of how things work and of how they ought to work in order to get anywhere with this. Experience tends to show that a competitive situation is needed in order to motivate one to go to the trouble. Unless something really sticks out in an obvious manner, one is most likely not going to look deep enough. Of course, this is seen in applications too but application <a class="auto-href" href="http://dbpedia.org/resource/Program_optimization" id="link-id0x1d429e80">optimization</a> tends to stop at a point where the application is usable. Also stored procedures and specially-tweaked queries will usually help. In most application scenarios, we are not simultaneously looking at multiple different implementations, except maybe at the start of development but then this falls under benchmarking and evaluation.</p> <p>So, the usefulness of benchmarks is again confirmed. There is likely great unexplored space for improvement as we move to more interesting and diverse scenarios.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1f619550">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li>Benchmarks, Redux (part 2): A Benchmarking Story <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1caa7cd8">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x1d8b7648">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1f2a6ba8">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x17b425f0">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1a7f6b30">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1ee5ec98">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1b7c5af8">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x1dad7588">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1c5520a0">Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1eb19bf8">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1eb2c398">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1fb6a118">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1f160580">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2011-03-14T17:16:40-04:00
Benchmarks, Redux (part 2): A Benchmarking Story
http://www.openlinksw.com:443/weblog/oerling/?date=2011-02-28#1660
2011-02-28T21:12:28Z
<blockquote> <i>Caeterum censeo, benchmarks are for vendors...</i> </blockquote> <p>This is an edifying story about benchmarks and how databases work. I will show how one detail makes a 5+x difference, and how one really must understand how things work in order to make sense of benchmarks.</p> <p>We begin right after the publication of the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/results/V6/index.html" id="link-id0x1df843f8">recent Berlin report</a>. This report gives us OK performance for queries and very bad performance for loading. Trickle updates were not measurable. This comes as a consequence of testing intermediate software cuts and having incomplete instructions for operating them. I will cover the whole <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1da5e160">BSBM</a> matter and the general benchmarking question in forthcoming posts; for now, let's talk about specifics.</p> <p>In the course of the discussion to follow, we talk about 3 different kinds of <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1c1e12f0">Virtuoso</a>:</p> <ul> <li> <p> <i>6 Single</i> is the generally available single-instance-server configuration of Virtuoso. Whether this is open source or not does not make a difference.</p> </li> <li> <p> <i>6 Cluster</i> is the generally available, commercial-only, cluster-capable Virtuoso.</p> </li> <li> <p> <i>7 Single</i> is the next-generation single-instance-server Virtuoso, about to be released as a preview.</p> </li> </ul> <p>We began by running the various parts of BSBM at different scales with different Virtuoso variants. In so doing, we noticed that the BSBM <i>Explore</i> mix at one scale got better throughput as we added more clients, approximately as one would expect based on <a class="auto-href" href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x1ddb5200">CPU</a> usage and number of cores, while at another scale this was not so.</p> <p>At the 1-billion-triple scale (1000 Mt; 1 Mt = 1 Megatriple, or one million triples) we saw CPU going from 200% with 1 client to 1400% with 16 clients but throughput increased by less than 20%. </p> <p>When we ran the same scale with our shared-nothing 6 Cluster, running 8 processes on the same box, throughput increased normally with the client count. We have not previously tried BSBM with 6 Cluster simply because there is little to gain and a lot to lose by distributing this workload. But here we got a multiuser throughput with 6 Cluster that is easily 3 times that of the single server, even with a cluster-unfriendly workload. </p> <p> See, sometimes scaling out even within a shared memory multiprocessor pays! Still, what we saw was rather anomalous.</p> <p>Over the years we have looked at performance any number of times and have a lot of built-in meters. For cases of high CPU with no throughput, the prime suspect is contention on critical sections. Quite right, when building with the mutex meter enabled, counting how many times each mutex is acquired and how many times this results in a wait, we found a mutex which gets acquired 600M times in the run, of which an insane 450M result in a wait. One can count a microsecond of real time each time a mutex wait results in the kernel switching tasks. The run took 500 s or so, of which 450 s of real time were attributable to the overhead of waiting for this one mutex.</p> <p>Waiting for a mutex is a real train wreck. We have tried spinning a few times before it, which the OS does anyhow, but this does not help. Using spin locks is good only if waits are extremely rare; with any frequency of waiting, even for very short waits, a mutex is still a lot better.</p> <p>Now, the mutex in question happens to serialize the buffer <a class="auto-href" href="http://dbpedia.org/resource/Cache" id="link-id0x1c1c29d8">cache</a> for one specific page of <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x1c1c2a30">data</a>, one level down from the root of the index for <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1e543ce0">RDF</a> PSOG. By the luck of the draw, the Ps falling on that page are commonly accessed Ps pertaining to product features. In order to get any product feature value, one must pass via this page. At the smaller scale, the different properties web their different ways based on the index root.</p> <p>One might here ask why the problem is one level down from the root and not in the root. The index root is already handled specially, so the read-write locks for buffers usually apply only for the first level down. One might also ask why have a mutex in the first place. Well, unless one is read-only and all in memory, there simply must be a way to say that a buffer must not get written to by one thread while another is reading it. Same for cache replacement. Some in-memory people fork a whole copy of the database process to do a large query and so can forget about serialization. But one must have long queries for this and have all in memory. One can make writes less frequent by keeping deltas, but this does not remove the need to merge the deltas at some point, which cannot happen without serializing this with the readers.</p> <p>Most of the time the offending mutex is acquired for getting a property of a product in Q5, the one that looks for products with similar values of a numeric property. We retrieve this property for a number of products in one go, due to vectoring. Vectoring is supposed to save us from constantly hitting the index tree top when getting the next match. So how come there is contention in the index tree top? As it happens, the vectored index lookup checks for locality only when all search conditions on key parts are equalities. Here however there is equality on P and S and a range on O; hence, the lookup starts from the index root every time.</p> <p>So I changed this. The effect was Q5 getting over twice as fast, with the single user throughput at 1000 Mt going from 2000 to 5200 QMpH (Query Mixes per Hour) and the 16-user throughput going from 3800 to over 21000 QMpH. The previously "good" throughput of 40K QMpH at 100 Mt went to 66K QMpH. </p> <p>Vectoring can make a real difference. The throughputs for the same workload on 6 Single, without vectoring, thus unavoidably hitting the page with the crazy contention, are 1770 QMpH single user and 2487 QMpH with 16 users. The 6 Cluster throughput, avoiding the contention but without the increased locality from vectoring and with the increased latency of going out-of-process for most of the data, was about 11.5K QMpH with 16 users. Each partition had a page getting the hits but since the partitioning was on S and S was about-evenly distributed, each partition got 1/8 of the load; thus waiting on the mutex did not become a killer issue. </p> <p>We see how detailed analysis of benchmarks can lead to almost an order of magnitude improvements in a short time. This analysis is however both difficult and tedious. It is not readily delegable; one needs real <a class="auto-href" href="http://dbpedia.org/resource/Knowledge" id="link-id0x1c1e2578">knowledge</a> of how things work and of how they ought to work in order to get anywhere with this. Experience tends to show that a competitive situation is needed in order to motivate one to go to the trouble. Unless something really sticks out in an obvious manner, one is most likely not going to look deep enough. Of course, this is seen in applications too but application <a class="auto-href" href="http://dbpedia.org/resource/Program_optimization" id="link-id0x1c59f4d8">optimization</a> tends to stop at a point where the application is usable. Also stored procedures and specially-tweaked queries will usually help. In most application scenarios, we are not simultaneously looking at multiple different implementations, except maybe at the start of development but then this falls under benchmarking and evaluation.</p> <p>So, the usefulness of benchmarks is again confirmed. There is likely great unexplored space for improvement as we move to more interesting and diverse scenarios.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1658" id="link-id0x1f619550">Benchmarks, Redux (part 1): On RDF Benchmarks</a> </li> <li>Benchmarks, Redux (part 2): A Benchmarking Story <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1caa7cd8">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x1d8b7648">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1f2a6ba8">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x17b425f0">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x1a7f6b30">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1ee5ec98">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1b7c5af8">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x1dad7588">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1c5520a0">Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1681" id="link-id0x199d1098">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1682" id="link-id0x19dd7508">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1683" id="link-id0x1742dac8">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1684" id="link-id0x163b7178">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2014-08-21T10:46:20.966390-04:00
Benchmarks, Redux (part 1): On RDF Benchmarks
http://www.openlinksw.com:443/blog/vdb/blog/?date=2011-02-28#1659
2011-02-28T20:20:22Z
<p>This post introduces a series on <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1e724ae0">RDF</a> benchmarking. In these posts I will cover the following:</p> <ul> <li> <p>Correct misleading <a class="auto-href" href="http://dbpedia.org/resource/Information" id="link-id0x1e325480">information</a> about us in the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/results/V6/index.html" id="link-id0x1ded41d0">recent Berlin report</a>: The load rate is off-the wall and the update mix is missing. We supply the right numbers and explain how to load things so that one gets decent performance.</p> </li> <li> <p>Discuss configuration options for <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1e0a2548">Virtuoso</a>.</p> </li> <li> <p>Tell a story about multithreading and its perils and how vectoring and scale-out can save us.</p> </li> <li> <p>Analyze the run time behavior of Virtuoso 6 Single, 6 Cluster, and 7 Single.</p> </li> <li> <p>Look at the benefits of SSDs (solid-state storage devices) over HDDs (hard disk devices; spinning platters), and I/O matters in general.</p> </li> <li> <p>Talk in general about modalities of benchmark running, and how to reconcile vendors doing what they know best with the air of legitimacy of a third party. Whether to do things a la <a class="auto-href" href="http://www.tpc.org/" id="link-id0x1e0ef4f0">TPC</a> or a la TREC? We will hopefully try a bit of both, at least so I have proposed to our partners in <a class="auto-href" href="http://lod2.eu/" id="link-id0x1e54d3d8">LOD2</a>, the EU FP7 that also funded the recent Berlin report.</p> </li> <li> <p>Outline the desiderata for an RDF benchmark that is not just an RDF-ized relational workload, the Social Intelligence Benchmark.</p> </li> <li> <p>Talk about <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1e730bc8">BSBM</a> in specific. What does it measure?</p> </li> <li> <p>Discuss some experiments with the BI use case of BSBM.</p> </li> <li> <p>Document how the results mentioned here were obtained and suggest practices for benchmark running and disclosure.</p> </li> </ul> <p>The background is that the LOD2 FP7 project is supposed to deliver a report about the state of the art and benchmark laboratory by March 1. The Berlin report is a part thereof. In the project proposal we talk about an ongoing benchmarking activity and about having up-to-date installations of the relevant RDF stores and <a class="auto-href" href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1c1551e0">RDBMS</a>.</p> <p>Since this is taxpayer money for supposedly the common good, I see no reason why such a useful thing should be restricted to the project participants. On the other hand, running a display window of stuff for benchmarking, when in at least in some cases licenses prohibit unauthorized publishing of benchmark results might be seen to conflict with the spirit of the license if not its letter. We will see.</p> <p>For now, my take is that we want to run benchmarks of all interesting software, inviting the vendors to tell us how to do that if they will, and maybe even letting them perform those runs themselves. Then we promise not to disclose results without the vendor's permission. Access to the installations is limited to whoever operates the equipment. Configuration files and detailed hardware specs and such on the other hand will be made public. If a run is published, it will be with permission and in a format that includes full information for replicating the experiment.</p> <p>In the LOD2 proposal we also in so many words say that we will stretch the limits of the state of the art. This stretching is surely not limited to the project's own products but should also include the general benchmarking aspect. I will say with confidence that running single server benchmarks at a max 200 Mtriples of <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x11327f10">data</a> is not stretching anything.</p> <p>So to ameliorate this situation, I thought to run the same at 10x the scale on a couple of large boxes we have access to. 1 and 2 billion triples are still comfortably single server scales. Then we could go for example to Giovanni's cluster at <a class="auto-href" href="http://dbpedia.org/resource/Digital_Enterprise_Research_Institute" id="link-id0x1bfaffa0">DERI</a> and do 10 and 20 billion triples, this should fly reasonably on 8 or 16 nodes of the DERI gear. Or we might talk to SEALS who by now should have their own lab. Even Amazon <a class="auto-href" href="http://aws.amazon.com/ec2/" id="link-id0x1bfafef8">EC2</a> might be an option, although not the preferred one.</p> <p>So I asked everybody about config instructions, which produced a certain amount of dismay as I might be said to be biased and to be skirting the edges of conflict of interest. The inquiry was not altogether negative though since <a class="auto-href" href="http://dbpedia.org/resource/Ontotext" id="link-id0x1eccc1e0">Ontotext</a> and <a class="auto-href" href="http://freebase.com/guid/9202a8c04000641f8000000005c908d6" id="link-id0x1eccc208">Garlik</a> provided some information. We will look into these this and next week. We will not publish any information without asking first.</p> <p>In this series of posts I will only talk about <a class="auto-href" href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id0x1bfa4030">OpenLink Software</a>.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li>Benchmarks, Redux (part 1): On RDF Benchmarks <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1b668d10">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1b3a0c08">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x1f9f1740">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1ad929f8">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1db437c0">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x17138c38">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1c0e74f8">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1f297d10">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x1e4994b8">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1ebea6d0">Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1d5c86c0">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1efec0e0">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1a9941f8">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=" id="link-id0x1ea26de8">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2011-03-14T17:16:34.000002-04:00
Benchmarks, Redux (part 1): On RDF Benchmarks
http://www.openlinksw.com:443/weblog/oerling/?date=2011-02-28#1658
2011-02-28T20:20:22Z
<p>This post introduces a series on <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1ea861a8">RDF</a> benchmarking. In these posts I will cover the following:</p> <ul> <li> <p>Correct misleading <a class="auto-href" href="http://dbpedia.org/resource/Information" id="link-id0x17f5faa8">information</a> about us in the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/results/V6/index.html" id="link-id0x1ded41d0">recent Berlin report</a>: The load rate is off-the wall and the update mix is missing. We supply the right numbers and explain how to load things so that one gets decent performance.</p> </li> <li> <p>Discuss configuration options for <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1ac63820">Virtuoso</a>.</p> </li> <li> <p>Tell a story about multithreading and its perils and how vectoring and scale-out can save us.</p> </li> <li> <p>Analyze the run time behavior of Virtuoso 6 Single, 6 Cluster, and 7 Single.</p> </li> <li> <p>Look at the benefits of SSDs (solid-state storage devices) over HDDs (hard disk devices; spinning platters), and I/O matters in general.</p> </li> <li> <p>Talk in general about modalities of benchmark running, and how to reconcile vendors doing what they know best with the air of legitimacy of a third party. Whether to do things a la <a class="auto-href" href="http://www.tpc.org/" id="link-id0x19c0a7f0">TPC</a> or a la TREC? We will hopefully try a bit of both, at least so I have proposed to our partners in <a class="auto-href" href="http://lod2.eu/" id="link-id0xa2e6170">LOD2</a>, the EU FP7 that also funded the recent Berlin report.</p> </li> <li> <p>Outline the desiderata for an RDF benchmark that is not just an RDF-ized relational workload, the Social Intelligence Benchmark.</p> </li> <li> <p>Talk about <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x17c81c20">BSBM</a> in specific. What does it measure?</p> </li> <li> <p>Discuss some experiments with the BI use case of BSBM.</p> </li> <li> <p>Document how the results mentioned here were obtained and suggest practices for benchmark running and disclosure.</p> </li> </ul> <p>The background is that the LOD2 FP7 project is supposed to deliver a report about the state of the art and benchmark laboratory by March 1. The Berlin report is a part thereof. In the project proposal we talk about an ongoing benchmarking activity and about having up-to-date installations of the relevant RDF stores and <a class="auto-href" href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x17f2a230">RDBMS</a>.</p> <p>Since this is taxpayer money for supposedly the common good, I see no reason why such a useful thing should be restricted to the project participants. On the other hand, running a display window of stuff for benchmarking, when in at least in some cases licenses prohibit unauthorized publishing of benchmark results might be seen to conflict with the spirit of the license if not its letter. We will see.</p> <p>For now, my take is that we want to run benchmarks of all interesting software, inviting the vendors to tell us how to do that if they will, and maybe even letting them perform those runs themselves. Then we promise not to disclose results without the vendor's permission. Access to the installations is limited to whoever operates the equipment. Configuration files and detailed hardware specs and such on the other hand will be made public. If a run is published, it will be with permission and in a format that includes full information for replicating the experiment.</p> <p>In the LOD2 proposal we also in so many words say that we will stretch the limits of the state of the art. This stretching is surely not limited to the project's own products but should also include the general benchmarking aspect. I will say with confidence that running single server benchmarks at a max 200 Mtriples of <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x1d7499d8">data</a> is not stretching anything.</p> <p>So to ameliorate this situation, I thought to run the same at 10x the scale on a couple of large boxes we have access to. 1 and 2 billion triples are still comfortably single server scales. Then we could go for example to Giovanni's cluster at <a class="auto-href" href="http://dbpedia.org/resource/Digital_Enterprise_Research_Institute" id="link-id0x17c46a88">DERI</a> and do 10 and 20 billion triples, this should fly reasonably on 8 or 16 nodes of the DERI gear. Or we might talk to SEALS who by now should have their own lab. Even Amazon <a class="auto-href" href="http://aws.amazon.com/ec2/" id="link-id0x8bed290">EC2</a> might be an option, although not the preferred one.</p> <p>So I asked everybody about config instructions, which produced a certain amount of dismay as I might be said to be biased and to be skirting the edges of conflict of interest. The inquiry was not altogether negative though since <a class="auto-href" href="http://dbpedia.org/resource/Ontotext" id="link-id0x1d269998">Ontotext</a> and <a class="auto-href" href="http://freebase.com/guid/9202a8c04000641f8000000005c908d6" id="link-id0x1bbc0a48">Garlik</a> provided some information. We will look into these this and next week. We will not publish any information without asking first.</p> <p>In this series of posts I will only talk about <a class="auto-href" href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id0x1ea6d948">OpenLink Software</a>.</p> <h3> <i>Benchmarks, Redux</i> Series</h3> <ul> <li>Benchmarks, Redux (part 1): On RDF Benchmarks <i>(this post)</i> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1660" id="link-id0x1b668d10">Benchmarks, Redux (part 2): A Benchmarking Story</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1663" id="link-id0x1b3a0c08">Benchmarks, Redux (part 3): Virtuoso 7 vs 6 on BSBM Load and Explore</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1665" id="link-id0x1f9f1740">Benchmarks, Redux (part 4): Benchmark Tuning Questionnaire</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1667" id="link-id0x1ad929f8">Benchmarks, Redux (part 5): BSBM and I/O; HDDs and SSDs </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1669" id="link-id0x1db437c0">Benchmarks, Redux (part 6): BSBM and I/O, continued</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1671" id="link-id0x17138c38">Benchmarks, Redux (part 7): What Does BSBM Explore Measure?</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1673" id="link-id0x1c0e74f8">Benchmarks, Redux (part 8): BSBM Explore and Update </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1675" id="link-id0x1f297d10">Benchmarks, Redux (part 9): BSBM With Cluster</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1677" id="link-id0x1e4994b8">Benchmarks, Redux (part 10): LOD2 and the Benchmark Process</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1678" id="link-id0x1ebea6d0">Benchmarks, Redux (part 11): On the Substance of RDF Benchmarks</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1681" id="link-id0xde5f8d8">Benchmarks, Redux (part 12): Our Own BSBM Results Report</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1682" id="link-id0x2aac0096f988">Benchmarks, Redux (part 13): BSBM BI Modifications </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1683" id="link-id0x1b7a1e98">Benchmarks, Redux (part 14): BSBM BI Mix </a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1684" id="link-id0x19f13b68">Benchmarks, Redux (part 15): BSBM Test Driver Enhancements </a> </li> </ul>
2014-08-21T10:46:27.527016-04:00
New Preconfigured Virtuoso AMI for Amazon EC2 Cloud comprised of Linked Data from BBC & DBpedia
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2011-02-18#1657
2011-02-19T01:20:30Z
<h2>What?</h2> <p>Introducing a new preloaded and preconfigured <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1bbe32d8">Virtuoso</a> (Cluster Edition) AMI for the Amazon EC2 Cloud that hosts combined Linked Datasets from: </p> <ul> <li> <a href="http://dbpedia.org/About" id="link-id0x1d21e780">DBpedia 3.6</a> </li> <li> <a href="http://www.bbc.co.uk/programmes" id="link-id0x1e1e0b10">BBC Programmes</a> </li> <li> <a href="http://www.bbc.co.uk/music" id="link-id0x1db12bd0">BBC Music</a> </li> <li> <a href="http://www.bbc.co.uk/nature/" id="link-id0x1bd46450">BBC Nature</a> </li> <li> <a href="http://www.bbc.co.uk/food/recipes/" id="link-id0x1d1b2468">BBC Food Recipes</a> </li> </ul> <h2>Why?</h2> <p> Predictably instantiate a powerful database with high quality <a href="http://dbpedia.org/resource/Data">data</a> and cross links within minutes, for personal or service specific use. </p> <h2>How?</h2> <p>Simply follow the instructions in our <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtAWSBBCMusicProgNatureFoodAndDBpedia36" id="link-id0x1d4f3210">Amazon EC2 guide for the BBC + DBpedia 3.6 Linked Dataset</a> guide.</p> <p>Your installation steps are as follows:</p> <ol> <li> Instantiate a Virtuoso EC2 AMI </li> <li> Mount the Amazon Elastic Block Storage (EBS) snapshot that hosts the preloaded Virtuoso Database. </li> </ol> <h2>Related</h2> <ul> <li> <a href="http://www.slideshare.net/reduxd/beyond-the-polar-bear" id="link-id0x1b384af0">BBC Linked Data Spaces Presentation</a> </li> <li> <a href="http://kidehen-images.s3.amazonaws.com/bbc_music_solo_artists_snapshot.png" id="link-id0x1a7a5ae0">BBC Music Linked Dataset Snapshot</a> -- PivotViewer Page Screenshot </li> <li> <a href="http://kidehen-images.s3.amazonaws.com/bbc_programmes_snapshot_sorted_by_genre.png" id="link-id0x1c2022a8">BBC Programmes Linked Dataset Snapshot</a> -- -- PivotViewer Page Screenshot </li> <li> <a href="http://kidehen-images.s3.amazonaws.com/bbc_nature_snapshot_sorted_by_adaptation.png" id="link-id0x1e138ac0">BBC Nature Linked Dataset Snapshot</a> -- PivotViewer Page Screenshot </li> <li> <a href="http://kidehen-images.s3.amazonaws.com/bbc_recipes_snapshot.png" id="link-id0x1b795100">BBC Food Recipes Snapshot </a> -- PivotViewer Page Screenshot </li> <li> <a href="http://www.delicious.com/kidehen/bbc_linkeddata" id="link-id0x1a581cf8">My Del.icio.us bookmark collection re. BBC Linked Data Demos</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/dav/wiki/Main/VirtAWSDBpediaBBC" id="link-id0x1dc0cc08">Amazon EC2 Snapshots for DBpedia 3.6 + BBC combo</a> -- delivers the BBC and DBpedia dataset combo via a mountable Elastic Block Storage (EBS) device usable with an Amazon Machine Image (AMI) </li> <li> <a href="http://www.openlinksw.com/dataspace/dav/wiki/Main/VirtAWSDBpedia351C" id="link-id0x1de33b50">Amazon EC2 Snapshots for DBpedia 3.6 & 3.5</a> </li> <li> <a href="http://virtuoso.openlinksw.com/download/" id="link-id0x1c3e27c8">Virtuoso Commercial Edition Download Page</a> </li> <li> <a href="http://docs.openlinksw.com/virtuoso/clusterstcnf.html" id="link-id0x1d0ff170">Virtuoso Cluster Edition Guide</a> </li> </ul>
2011-03-29T09:52:17.000001-04:00
DBpedia + BBC (combined) Linked Data Space Installation Guide
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2011-02-17#1656
2011-02-17T22:15:41Z
<h2>What? </h2> <p> The <i><a class="auto-href" href="http://dbpedia.org/resource/DBpedia" id="link-id0x1c489cc8">DBpedia</a> + <a class="auto-href" href="http://dbpedia.org/resource/BBC" id="link-id0x1bf12698">BBC</a> Combo Linked Dataset </i> is a preconfigured <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1b16cbb0">Virtuoso</a> Cluster (4 Virtuoso Cluster Nodes, each comprised of one Virtuoso Instance; initial deployment is to a single Cluster Host, but license may be converted for physically distributed deployment), available via the Amazon EC2 Cloud, preloaded with the following datasets: </p> <ul> <li> <a href="http://dbpedia.org/About" id="link-id0x1d21e780">DBpedia 3.6</a> </li> <li> <a href="http://www.bbc.co.uk/programmes" id="link-id0x1e1e0b10">BBC Programmes</a> </li> <li> <a href="http://www.bbc.co.uk/music" id="link-id0x1db12bd0">BBC Music</a> </li> <li> <a href="http://www.bbc.co.uk/nature/" id="link-id0x1bd46450">BBC Nature</a> </li> <li> <a href="http://www.bbc.co.uk/food/recipes/" id="link-id0x1d1b2468">BBC Food Recipes</a> </li> </ul> <h2>Why?</h2> <p>The BBC has been publishing <a class="auto-href" href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1b15eb60">Linked Data</a> from its <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> <a class="auto-href" href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id0x1c4c38a8">Data Space</a> for a number of years. In line with best practices for injecting Linked Data into the <a class="auto-href" href="http://dbpedia.org/resource/World_Wide_Web" id="link-id0x1e5acda0">World Wide Web</a> (Web), the BBC datasets are interlinked with other datasets such as DBpedia and MusicBrainz. </p> <p>Typical follow-your-nose exploration using a Web Browser (or even via sophisticated <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x1d21e728">SPARQL</a> query crawls) isn't always practical once you get past the initial euphoria that comes from comprehending the Linked Data concept. As your queries get more complex, the overhead of remote sub-queries increases its impact, until query results take so long to return that you simply give up.</p> <p>Thus, maximizing the effects of the BBC's efforts requires Linked Data that shares locality in a Web-accessible Data Space — i.e., where all Linked Data sets have been loaded into the same data store or warehouse. This holds true even when leveraging SPARQL-FED style virtualization — there's always a need to localize data as part of any marginally-decent locality-aware cost-optimization algorithm.</p> <p>This DBpedia + BBC dataset, exposed via a preloaded and preconfigured Virtuoso Cluster, delivers a practical point of presence on the Web for immediate and cost-effective exploitation of Linked Data at the individual and/or service specific levels.</p> <h2>How?</h2> To work through this guide, you'll need to start with 90 GB of free disk space. (Only 41 GB will be consumed after you delete the installer archives, but starting with 90+ GB ensures enough work space for the installation.) <h3>Install Virtuoso</h3> <ol> <li> <p> <a href="http://virtuoso.openlinksw.com/download/" id="link-id0x1af0d230">Download Virtuoso installer archive(s)</a>. You must deploy the Personal or Enterprise Edition; the Open Source Edition does not support Shared-Nothing Cluster Deployment.</p> </li> <li> <p> <a href="http://virtuoso.openlinksw.com/pricing/" id="link-id0x1e089f40">Obtain a Virtuoso Cluster license</a>.</p> </li> <li> <p> <a href="http://wikis.openlinksw.com/dataspace/owiki/wiki/VirtuosoWikiWeb/VirtuosoInstallDocs" id="link-id0x1e86d060">Install Virtuoso</a>.</p> </li> <li> <p>Set key environment variables and start the OpenLink License Manager, using command (this may vary depending on your shell and install directory): </p> <blockquote> <code>. /opt/virtuoso/virtuoso-enterprise.sh</code> </blockquote> </li> <li> <p> <i>Optional:</i> To keep the default single-server configuration file and demo database intact, set the <code>VIRTUOSO_HOME</code> environment variable to a different directory, e.g., </p> <blockquote> <code>export VIRTUOSO_HOME=/opt/virtuoso/cluster-home/</code> </blockquote> <p> <i><b>Note:</b> You will have to adjust this setting every time you shift between this cluster setup and your single-server setup. Either may be made your environment's default through the <code>virtuoso-enterprise.sh</code> and related scripts.</i> </p> </li> <li> <p> <a href="http://docs.openlinksw.com/virtuoso/clusterstcnf.html" id="link-id0x1e184dc0">Set up your cluster</a> by running the <code>mkcluster.sh</code> script. Note that initial deployment of the <i>DBpedia + BBC Combo</i> requires a 4 node cluster, which is the default for this script.</p> </li> <li> <p>Start the Virtuoso Cluster with this command:</p> <blockquote> <code>virtuoso-start.sh</code> </blockquote> </li> <li> <p>Stop the Virtuoso Cluster with this command:</p> <blockquote> <code>virtuoso-stop.sh</code> </blockquote> </li> </ol> <h3>Using the DBpedia + BBC Combo dataset</h3> <ol> <li> <p>Navigate to your installation directory.</p> </li> <li> <p>Download the combo dataset installer script — <code><a href="https://s3.amazonaws.com/bbc-dbpedia-36-usa/bbc-dbpedia-install.sh" id="link-id0x195d7940">bbc-dbpedia-install.sh</a></code>.</p> </li> <li> <p>For best results, set the downloaded script to fully executable using this command:</p> <blockquote> <code>chmod 755 bbc-dbpedia-install.sh </code> </blockquote> </li> <li> <p>Shut down any Virtuoso instances that may be currently running.</p> </li> <li> <p> <i>Optional:</i> As above, if you have decided to keep the default single-server configuration file and demo database intact, set the <code>VIRTUOSO_HOME</code> environment variable appropriately, e.g., </p> <blockquote> <code>export VIRTUOSO_HOME=/opt/virtuoso/cluster-home/</code> </blockquote> </li> <li> <p>Run the combo dataset installer script with this command:</p> <blockquote> <code>sh bbc-dbpedia-install.sh</code> </blockquote> </li> </ol> <h3>Verify installation</h3> <p>The combo dataset typically deploys to EC2 virtual machines in under 90 minutes; your time will vary depending on your network connection speed, machine speed, and other variables.</p> <p>Once the script completes, perform the following steps:</p> <ol> <li> <p>Verify that the Virtuoso Conductor (HTTP-based Admin UI) is in place via:</p> <blockquote> <code>http://localhost:[port]/conductor</code> </blockquote> </li> <li> <p>Verify that the Virtuoso SPARQL endpoint is in place via:</p> <blockquote> <code>http://localhost:[port]/sparql</code> </blockquote> </li> <li> <p>Verify that the Precision Search & Find UI is in place via:</p> <blockquote> <code>http://localhost:[port]/fct</code> </blockquote> </li> <li> <p>Verify that the Virtuoso hosted PivotViewer is in place via:</p> <blockquote> <code>http://localhost:[port]/PivotViewer</code> </blockquote> </li> </ol> <h2>Related</h2> <ul> <li> <a href="http://www.slideshare.net/reduxd/beyond-the-polar-bear" id="link-id0x1bd43bf0">BBC Linked Data Spaces Presentation</a> </li> <li> <a href="http://kidehen-images.s3.amazonaws.com/bbc_music_solo_artists_snapshot.png" id="link-id0x1a7a5ae0">BBC Music Linked Dataset Snapshot</a> -- PivotViewer Page Screenshot </li> <li> <a href="http://kidehen-images.s3.amazonaws.com/bbc_programmes_snapshot_sorted_by_genre.png" id="link-id0x1c2022a8">BBC Programmes Linked Dataset Snapshot</a> -- -- PivotViewer Page Screenshot </li> <li> <a href="http://kidehen-images.s3.amazonaws.com/bbc_nature_snapshot_sorted_by_adaptation.png" id="link-id0x1e138ac0">BBC Nature Linked Dataset Snapshot</a> -- PivotViewer Page Screenshot </li> <li> <a href="http://kidehen-images.s3.amazonaws.com/bbc_recipes_snapshot.png" id="link-id0x1b795100">BBC Food Recipes Snapshot </a> -- PivotViewer Page Screenshot </li> <li> <a href="http://www.delicious.com/kidehen/bbc_linkeddata" id="link-id0x1c0ffcc8">My Del.icio.us bookmark collection re. BBC Linked Data Demos</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/dav/wiki/Main/VirtAWSDBpediaBBC" id="link-id0x1dc0cc08">Amazon EC2 Snapshots for DBpedia 3.6 + BBC combo</a> -- delivers the BBC and DBpedia dataset combo via a mountable Elasti<a class="auto-href" href="http://dbpedia.org/resource/C%2B%2B" id="link-id0x1c2ad728">c</a> Block Storage (EBS) device usable with an Amazon Machine Image (AMI) </li> <li> <a href="http://www.openlinksw.com/dataspace/dav/wiki/Main/VirtAWSDBpedia351C" id="link-id0x1de33b50">Amazon EC2 Snapshots for DBpedia 3.6 & 3.5</a> </li> <li> <a href="http://virtuoso.openlinksw.com/download/" id="link-id0x1c3e27c8">Virtuoso Commercial Edition Download Page</a> </li> <li> <a href="http://docs.openlinksw.com/virtuoso/clusterstcnf.html" id="link-id0x1d0ff170">Virtuoso Cluster Edition Guide</a> </li> </ul>
2011-03-29T10:09:45.000001-04:00
SPARQL Guide for the Perl Developer
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2011-01-25#1655
2011-01-25T16:05:17Z
<h3>What?</h3> <p>A simple guide usable by any <a class="auto-href" href="http://dbpedia.org/resource/Perl" id="link-id0x1bdcab80">Perl</a> developer seeking to exploit <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x17b447e8">SPARQL</a> without hassles.</p> <h3>Why?</h3> <p>SPARQL is a powerful query language, results serialization format, and an HTTP based <a href="http://dbpedia.org/resource/Data">data</a> access protocol from the W3C. It provides a mechanism for accessing and integrating data across <a href="http://en.wikipedia.org/wiki/Deductive_database" id="link-id0x1cc76540">Deductive Database Systems</a> (colloquially referred to as triple or quad stores in <a class="auto-href" href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1d944d78">Semantic Web</a> and <a class="auto-href" href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1c7a87c8">Linked Data</a> circles) -- database systems (or data spaces) that manage proposition oriented records in 3-tuple (triples) or 4-tuple (quads) form. </p> <h3>How?</h3> <p>SPARQL queries are actually HTTP payloads (typically). Thus, using a RESTful client-server interaction pattern, you can dispatch calls to a SPARQL compliant data server and receive a payload for local processing.</p> <h4>Steps:</h4> <ol> <li>Determine which SPARQL endpoint you want to access e.g. <a href="http://dbpedia.org/sparql" id="link-id0x1d476520">DBpedia</a> or a local <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1bcfe140">Virtuoso</a> instance (typically: http://localhost:8890/sparql). </li> <li>If using Virtuoso, and you want to populate its quad store using SPARQL, assign "<a href="http://docs.openlinksw.com/virtuoso/rdfsparql.html#rdfsupportedprotocolendpointuri" id="link-id0x1c7630b8">SPARQL_SPONGE</a>" privileges to user "SPARQL" (this is basic control, more sophisticated WebID based ACLs are available for controlling SPARQL access).</li> </ol> <h4>Script:</h4> <pre> # # Demonstrating use of a single query to populate a # Virtuoso Quad Store via Perl. # # # HTTP <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id0x1d6465e8">URL</a> is constructed accordingly with CSV query results format as the default via mime type. # use CGI qw/:standard/; use LWP::UserAgent; use Data::Dumper; use Text::CSV_XS; sub sparqlQuery(@args) { my $query=shift; my $baseURL=shift; my $format=shift; %params=( "default-graph" => "", "should-sponge" => "soft", "query" => $query, "debug" => "on", "timeout" => "", "format" => $format, "save" => "display", "fname" => "" ); @fragments=(); foreach $k (keys %params) { $fragment="$k=".CGI::escape($params{$k}); push(@fragments,$fragment); } $query=join("&", @fragments); $sparqlURL="${baseURL}?$query"; my $ua = LWP::UserAgent->new; $ua->agent("MyApp/0.1 "); my $req = HTTP::Request->new(GET => $sparqlURL); my $res = $ua->request($req); $str=$res->content; $csv = Text::CSV_XS->new(); foreach $line ( split(/^/, $str) ) { $csv->parse($line); @bits=$csv->fields(); push(@rows, [ @bits ] ); } return \@rows; } # Setting Data Source Name (DSN) $dsn="http://dbpedia.org/resource/DBpedia"; # Virtuoso pragmas for instructing SPARQL engine to perform an HTTP GET using the IRI in # FROM clause as Data Source URL en route to DBMS # record Inserts. $query="DEFINE get:soft \"replace\"\n # Generic (non Virtuoso specific SPARQL # Note: this will not add records to the # DBMS SELECT DISTINCT * FROM <$dsn> WHERE {?s ?p ?o}"; $data=sparqlQuery($query, "http://localhost:8890/sparql/", "text/csv"); print "Retrieved data:\n"; print Dumper($data); </pre> <h4>Output</h4> <pre> Retrieved data: $VAR1 = [ [ 's', 'p', 'o' ], [ 'http://dbpedia.org/resource/DBpedia', 'http://www.w3.org/1999/02/22-rdf-syntax-ns#type', 'http://www.w3.org/2002/07/owl#Thing' ], [ 'http://dbpedia.org/resource/DBpedia', 'http://www.w3.org/1999/02/22-rdf-syntax-ns#type', 'http://dbpedia.org/ontology/Work' ], [ 'http://dbpedia.org/resource/DBpedia', 'http://www.w3.org/1999/02/22-rdf-syntax-ns#type', 'http://dbpedia.org/class/yago/Software106566077' ], ... </pre> <h3>Conclusion</h3> <p> CSV was chosen over XML (re. output format) since this is about a "no-brainer installation and utilization" guide for a Perl developer that already knows how to use Perl for HTTP based data access within HTML. SPARQL just provides an added bonus to URL dexterity (delivered via <a class="auto-href" href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x1d29da98">URI</a> abstraction) with regards to constructing Data Source Names or Addresses.</p> <h3>Related</h3> <ul> <li> <a href="http://cpansearch.perl.org/src/TOBYINK/RDF-Query-Client-0.103/README" id="link-id0x1c279130">RDF::Query::Client Guide</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1653" id="link-id0x1cf307f0">SPARQL Guide for the Perl Developer</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1652" id="link-id0x1b0ffb28">SPARQL Guide for the PHP Developer</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1651" id="link-id0x1a8c5ae0">SPARQL Guide for the Python Developer</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1648" id="link-id0x1b86ad28">SPARQL Guide for the Ruby Developer</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1646" id="link-id0x1c7af188">Simple Guide for using SPARQL with Virtuoso</a> </li> <li> <a href="http://www.delicious.com/kidehen/sparql_tutorial" id="link-id0x1ac1ba48">General SPARQL Tutorial Collection</a> </li> <li> <a href="http://www.delicious.com/kidehen/virtuoso_sparql_tutorial" id="link-id0x1c7be660">Virtuoso Specific SPARQL Tutorial Collection</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1567" id="link-id0x1c52b438">The URI, URL, and Linked Data Meme's Generic HTTP URI</a>. </li> </ul>
2011-01-26T18:11:13-05:00
Virtuoso + DBpedia 3.6 Installation Guide (Update 1)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2011-01-24#1654
2011-01-25T01:08:55Z
<h3>What is <a class="auto-href" href="http://dbpedia.org/resource/DBpedia" id="link-id0x1d8b5df0">DBpedia</a>?</h3> <p> DBpedia is a community effort to provide a contemporary deductive database derived from Wikipedia content. Project contributions can be partitioned as follows: </p> <ol> <li> Ontology Construction and Maintenance </li> <li> Dataset Generation via Wikipedia Content Extraction & Transformation </li> <li> Live Database Maintenance & Administration -- includes actual <a class="auto-href" href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1ba81190">Linked Data</a> loading and publishing, provision of <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x1d8af808">SPARQL</a> endpoint, and traditional DBA activity </li> <li> Internationalization. </li> </ol> <h3>Why is DBpedia important?</h3> <p> Comprising the nucleus of the Linked Open <a href="http://dbpedia.org/resource/Data">Data</a> effort, DBpedia also serves as a fulcrum for the burgeoning <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> of Linked Data by delivering a dense and highly-interlinked lookup database. In its most basic form, DBpedia is a great source of strong and resolvable identifiers for People, Places, Organizations, Subject Matter, and many other data items of interest. Naturally, it provides a fantastic starting point for comprehending the fundamental concepts underlying <a class="auto-href" href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id0x1a8cc3d0">TimBL</a>'s initial <a href="http://blogs.usnet.private:8893/www.w3.org/DesignIssues/LinkedData.html" id="link-id0x1cbbaf50">Linked Data</a> meme. </p> <h3>How do I use DBpedia?</h3> <p> Depending on your particular requirements, whether personal or service-specific, DBpedia offers the following: </p> <ul> <li> Datasets that can be loaded on your deductive database (also known as triple or quad stores) platform of choice </li> <li> Live browsable HTML+<a class="auto-href" href="http://dbpedia.org/resource/RDFa" id="link-id0x1d6b2148">RDFa</a> based <a class="auto-href" href="http://dbpedia.org/resource/Entity" id="link-id0x1d766a98">entity</a> description pages </li> <li> A wide variety of data formats for importing entity description data into a broad range of existing applications and services </li> <li> A SPARQL endpoint allowing ad-hoc querying over HTTP using the SPARQL query language, and delivering results serialized in a variety of formats </li> <li> A broad variety of tools covering query by example, faceted browsing, <a class="auto-href" href="http://dbpedia.org/resource/Full_text_search" id="link-id0x1b330ff8">full text search</a>, entity name lookups, etc. </li> </ul> <h3>What is the DBpedia 3.6 + <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1d705780">Virtuoso</a> Cluster Edition Combo?</h3> <p> <a class="auto-href" href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id0x1c894338">OpenLink Software</a> has preloaded the DBpedia 3.6 datasets into a preconfigured Virtuoso Cluster Edition database, and made the package available for easy installation.</p> <h3>Why is the DBpedia+Virtuoso package important?</h3> <p> The DBpedia+Virtuoso package provides a cost-effective option for personal or service-specific incarnations of DBpedia. </p> <p>For instance, you may have a service that isn't best-served by competing with the rest of the world for ad-hoc query time and resources on the live instance, which itself operates under various restrictions which enable this ad-hoc query service to be provided at Web Scale.</p> <p>Now you can easily commission your own instance and quickly exploit DBpedia and Virtuoso's database feature set to the max, powered by your own hardware and network infrastructure. </p> <h3>How do I use the DBpedia+Virtuoso package?</h3> <p>Pre-requisites are simply:</p> <ol> <li> <a href="http://wikis.openlinksw.com/dataspace/owiki/wiki/VirtuosoWikiWeb/VirtuosoInstallConfig" id="link-id0x19e3e450">Functional Virtuoso Cluster Edition installation</a>. </li> <li> <a href="http://virtuoso.openlinksw.com/pricing/" id="link-id0x1b703ad8">Virtuoso Cluster Edition License</a>. </li> <li>90 GB of free disk space -- you ultimately only need 43 gigs, but this our recommended free disk space size pre installation completion.</li> </ol> <p> To install the Virtuoso Cluster Edition simply perform the following steps: </p> <ol> <li> <a href="http://virtuoso.openlinksw.com/download/" id="link-id0x17b41648">Download Software</a>. </li> <li> Run installer </li> <li> <p>Set key environment variables and start the OpenLink License Manager, using command (this may vary depending on your shell): </p> <blockquote> <code>. /opt/virtuoso/virtuoso-enterprise.sh</code> </blockquote> </li> <li> Run the <code>mkcluster.sh</code> script which defaults to a 4 node cluster </li> <li> Set <code>VIRTUOSO_HOME</code> environment variable -- if you want to start cluster databases distinct from single server databases via distinct root directory for database files (one that isn't adjacent to single-server database directories) </li> <li> Start Virtuoso Cluster Edition instances using command: <blockquote> <code>virtuoso-start.sh</code> </blockquote> </li> <li> Stop Virtuoso Cluster Edition instances using command: <blockquote> <code>virtuoso-stop.sh</code> </blockquote> </li> </ol> <p>To install your personal or service specific edition of DBpedia simply perform the following steps:</p> <ol> <li> Navigate to your installation directory </li> <li> Download Installer script (<code><a href="https://s3.amazonaws.com/dbpedia-36-usa/dbpedia-install.sh" id="link-id0x1da0c978">dbpedia-install.sh</a></code>) </li> <li> Set execution mode on script using command: <blockquote> <code>chmod 755 dbpedia-install.sh </code> </blockquote> </li> <li> Shutdown any Virtuoso instances that may be currently running </li> <li> Set your <code>VIRTUOSO_HOME</code> environment variable, e.g., to the current directory, via command (this may vary depending on your shell): <blockquote> <code>export VIRTUOSO_HOME=`pwd`</code> </blockquote> </li> <li> Run script using command: <blockquote> <code>sh dbpedia-install.sh</code> </blockquote> </li> </ol> <p> Once the installation completes (approximately 1 hour and 30 minutes from start time), perform the following steps: </p> <ol> <li> Verify that the Virtuoso Conductor (HTML based Admin UI) is in place via: <blockquote> <code>http://localhost:[port]/conductor</code> </blockquote> </li> <li> Verify that the Precision Search & Find UI is in place via: <blockquote> <code>http://localhost:[port]/fct</code> </blockquote> </li> <li>Verify that DBpedia's Green Entity Description Pages are in place via: <blockquote> <code>http://localhost:[port]/resource/DBpedia</code> </blockquote> </li> </ol> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/dav/wiki/Main/VirtAWSDBpedia351C" id="link-id0x1d819b90">Amazon EC2 Snapshots for DBpedia 3.6 & 3.5</a> </li> <li> <a href="http://virtuoso.openlinksw.com/download/" id="link-id0x1c3e27c8">Virtuoso Commercial Edition Download Page</a> </li> <li> <a href="http://docs.openlinksw.com/virtuoso/clusterstcnf.html" id="link-id0x1d0ff170">Virtuoso Cluster Edition Guide</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1594" id="link-id0x1c891cf8">What is the DBpedia Project?</a> </li> </ul>
2011-01-25T14:46:26-05:00
SPARQL Guide for the Javascript Developer
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2011-01-21#1653
2011-01-21T19:59:49Z
<h3>What?</h3> <p>A simple guide usable by any Javascript developer seeking to exploit <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x17b447e8">SPARQL</a> without hassles.</p> <h3>Why?</h3> <p>SPARQL is a powerful query language, results serialization format, and an HTTP based <a href="http://dbpedia.org/resource/Data">data</a> access protocol from the W3C. It provides a mechanism for accessing and integrating data across <a href="http://en.wikipedia.org/wiki/Deductive_database" id="link-id0x1cc76540">Deductive Database Systems</a> (colloquially referred to as triple or quad stores in <a class="auto-href" href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1d944d78">Semantic Web</a> and <a class="auto-href" href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1c7a87c8">Linked Data</a> circles) -- database systems (or data spaces) that manage proposition oriented records in 3-tuple (triples) or 4-tuple (quads) form. </p> <h3>How?</h3> <p>SPARQL queries are actually HTTP payloads (typically). Thus, using a RESTful client-server interaction pattern, you can dispatch calls to a SPARQL compliant data server and receive a payload for local processing.</p> <h4>Steps:</h4> <ol> <li>Determine which SPARQL endpoint you want to access e.g. <a href="http://dbpedia.org/sparql" id="link-id0x1d476520">DBpedia</a> or a local <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1bcfe140">Virtuoso</a> instance (typically: http://localhost:8890/sparql). </li> <li>If using Virtuoso, and you want to populate its quad store using SPARQL, assign "<a href="http://docs.openlinksw.com/virtuoso/rdfsparql.html#rdfsupportedprotocolendpointuri" id="link-id0x1c7630b8">SPARQL_SPONGE</a>" privileges to user "SPARQL" (this is basic control, more sophisticated WebID based ACLs are available for controlling SPARQL access).</li> </ol> <h4>Script:</h4> <pre> /* Demonstrating use of a single query to populate a # Virtuoso Quad Store via Javascript. */ /* HTTP <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id0x1bc27a18">URL</a> is constructed accordingly with JSON query results format as the default via mime type. */ function sparqlQuery(query, baseURL, format) { if(!format) format="application/json"; var params={ "default-graph": "", "should-sponge": "soft", "query": query, "debug": "on", "timeout": "", "format": format, "save": "display", "fname": "" }; var querypart=""; for(var k in params) { querypart+=k+"="+encodeURIComponent(params[k])+"&"; } var queryURL=baseURL + '?' + querypart; if (window.XMLHttpRequest) { xmlhttp=new XMLHttpRequest(); } else { xmlhttp=new ActiveXObject("Microsoft.XMLHTTP"); } xmlhttp.open("GET",queryURL,false); xmlhttp.send(); return JSON.parse(xmlhttp.responseText); } /* setting Data Source Name (DSN) */ var dsn="http://dbpedia.org/resource/DBpedia"; /* Virtuoso pragma "DEFINE get:soft "replace" instructs Virtuoso SPARQL engine to perform an HTTP GET using the IRI in FROM clause as Data Source URL with regards to DBMS record inserts */ var query="DEFINE get:soft \"replace\"\nSELECT DISTINCT * FROM <"+dsn+"> WHERE {?s ?p ?o}"; var data=sparqlQuery(query, "/sparql/"); </pre> <h4>Output</h4> <p> Place the snippet above into the <script/> section of an HTML document to see the <a href="http://twitpic.com/3s2vs3/full" id="link-id0x1cff2288">query result</a>. </p> <h3>Conclusion</h3> <p> JSON was chosen over XML (re. output format) since this is about a "no-brainer installation and utilization" guide for a Javascript developer that already knows how to use Javascript for HTTP based data access within HTML. SPARQL just provides an added bonus to URL dexterity (delivered via <a class="auto-href" href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x1d29da98">URI</a> abstraction) with regards to constructing Data Source Names or Addresses.</p> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1652" id="link-id0x1b0ffb28">SPARQL Guide for the PHP Developer</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1651" id="link-id0x1a8c5ae0">SPARQL Guide for the Python Developer</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1648" id="link-id0x1b86ad28">SPARQL Guide for the Ruby Developer</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1646" id="link-id0x1c7af188">Simple Guide for using SPARQL with Virtuoso</a> </li> <li> <a href="http://www.delicious.com/kidehen/sparql_tutorial" id="link-id0x1ac1ba48">General SPARQL Tutorial Collection</a> </li> <li> <a href="http://www.delicious.com/kidehen/virtuoso_sparql_tutorial" id="link-id0x1c7be660">Virtuoso Specific SPARQL Tutorial Collection</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1567" id="link-id0x1c52b438">The URI, URL, and Linked Data Meme's Generic HTTP URI</a>. </li> </ul>
2011-01-26T18:10:28-05:00
SPARQL Guide for the PHP Developer
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2011-01-20#1652
2011-01-20T21:25:49Z
<h3>What?</h3> <p>A simple guide usable by any <a class="auto-href" href="http://dbpedia.org/resource/PHP_programming_language" id="link-id0x1bdca7b8">PHP</a> developer seeking to exploit <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x1c894338">SPARQL</a> without hassles.</p> <h3>Why?</h3> <p>SPARQL is a powerful query language, results serialization format, and an HTTP based <a href="http://dbpedia.org/resource/Data">data</a> access protocol from the W3C. It provides a mechanism for accessing and integrating data across <a href="http://en.wikipedia.org/wiki/Deductive_database" id="link-id0x1c319af0">Deductive Database Systems</a> (colloquially referred to as triple or quad stores in <a class="auto-href" href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1d944d78">Semantic Web</a> and <a class="auto-href" href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1c7a87c8">Linked Data</a> circles) -- database systems (or data spaces) that manage proposition oriented records in 3-tuple (triples) or 4-tuple (quads) form. </p> <h3>How?</h3> <p>SPARQL queries are actually HTTP payloads (typically). Thus, using a RESTful client-server interaction pattern, you can dispatch calls to a SPARQL compliant data server and receive a payload for local processing e.g. local object binding re. PHP.</p> <h4>Steps:</h4> <ol> <li> From your command line execute: aptitude search '^PHP26', to verify PHP is in place </li> <li>Determine which SPARQL endpoint you want to access e.g. <a href="http://dbpedia.org/sparql" id="link-id0x1d476520">DBpedia</a> or a local <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1bcfe140">Virtuoso</a> instance (typically: http://localhost:8890/sparql). </li> <li>If using Virtuoso, and you want to populate its quad store using SPARQL, assign "<a href="http://docs.openlinksw.com/virtuoso/rdfsparql.html#rdfsupportedprotocolendpointuri" id="link-id0x1c7630b8">SPARQL_SPONGE</a>" privileges to user "SPARQL" (this is basic control, more sophisticated WebID based ACLs are available for controlling SPARQL access).</li> </ol> <h4>Script:</h4> <pre> #!/usr/bin/env php <?php # # Demonstrating use of a single query to populate a # Virtuoso Quad Store via PHP. # # HTTP <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id0x1ce1d6d8">URL</a> is constructed accordingly with JSON query results format in mind. function sparqlQuery($query, $baseURL, $format="application/json") { $params=array( "default-graph" => "", "should-sponge" => "soft", "query" => $query, "debug" => "on", "timeout" => "", "format" => $format, "save" => "display", "fname" => "" ); $querypart="?"; foreach($params as $name => $value) { $querypart=$querypart . $name . '=' . urlencode($value) . "&"; } $sparqlURL=$baseURL . $querypart; return json_decode(file_get_contents($sparqlURL)); }; # Setting Data Source Name (DSN) $dsn="http://dbpedia.org/resource/DBpedia"; #Virtuoso pragmas for instructing SPARQL engine to perform an HTTP GET #using the IRI in FROM clause as Data Source URL $query="DEFINE get:soft \"replace\" SELECT DISTINCT * FROM <$dsn> WHERE {?s ?p ?o}"; $data=sparqlQuery($query, "http://localhost:8890/sparql/"); print "Retrieved data:\n" . json_encode($data); ?> </pre> <h4>Output</h4> <pre> Retrieved data: {"head": {"link":[],"vars":["s","p","o"]}, "results": {"distinct":false,"ordered":true, "bindings":[ {"s": {"type":"<a class="auto-href" href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x1ca44a98">uri</a>","value":"http:\/\/dbpedia.org\/resource\/DBpedia"},"p": {"type":"uri","value":"http:\/\/www.w3.org\/1999\/02\/22-rdf-syntax-ns#type"},"o": {"type":"uri","value":"http:\/\/www.w3.org\/2002\/07\/owl#Thing"}}, {"s": {"type":"uri","value":"http:\/\/dbpedia.org\/resource\/DBpedia"},"p": {"type":"uri","value":"http:\/\/www.w3.org\/1999\/02\/22-rdf-syntax-ns#type"},"o": {"type":"uri","value":"http:\/\/dbpedia.org\/ontology\/Work"}}, {"s": {"type":"uri","value":"http:\/\/dbpedia.org\/resource\/DBpedia"},"p": {"type":"uri","value":"http:\/\/www.w3.org\/1999\/02\/22-rdf-syntax-ns#type"},"o": {"type":"uri","value":"http:\/\/dbpedia.org\/class\/yago\/Software106566077"}}, ... </pre> <h3>Conclusion</h3> <p> JSON was chosen over XML (re. output format) since this is about a "no-brainer installation and utilization" guide for a PHP developer that already knows how to use PHP for HTTP based data access. SPARQL just provides an added bonus to URL dexterity (delivered via URI abstraction) with regards to constructing Data Source Names or Addresses.</p> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1651" id="link-id0x1a8c5ae0">SPARQL Guide for the Python Developer</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1648" id="link-id0x1b86ad28">SPARQL Guide for the Ruby Developer</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1646" id="link-id0x1c7af188">Simple Guide for using SPARQL with Virtuoso</a> </li> <li> <a href="http://www.delicious.com/kidehen/sparql_tutorial" id="link-id0x1ac1ba48">General SPARQL Tutorial Collection</a> </li> <li> <a href="http://www.delicious.com/kidehen/virtuoso_sparql_tutorial" id="link-id0x1c7be660">Virtuoso Specific SPARQL Tutorial Collection</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1567" id="link-id0x1c52b438">The URI, URL, and Linked Data Meme's Generic HTTP URI</a>. </li> </ul>
2011-01-25T10:36:58-05:00
SPARQL Guide for Python Developer
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2011-01-19#1651
2011-01-19T17:13:30Z
<h3>What?</h3> <p>A simple guide usable by any <a class="auto-href" href="http://dbpedia.org/resource/Python_programming_language" id="link-id0x1bdca7b8">Python</a> developer seeking to exploit <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x1c894338">SPARQL</a> without hassles.</p> <h3>Why?</h3> <p>SPARQL is a powerful query language, results serialization format, and an HTTP based <a href="http://dbpedia.org/resource/Data">data</a> access protocol from the W3C. It provides a mechanism for accessing and integrating data across <a href="http://en.wikipedia.org/wiki/Deductive_database" id="link-id0x1c319af0">Deductive Database Systems</a> (colloquially referred to as triple or quad stores in <a class="auto-href" href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1d944d78">Semantic Web</a> and <a class="auto-href" href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1c7a87c8">Linked Data</a> circles) -- database systems (or data spaces) that manage proposition oriented records in 3-tuple (triples) or 4-tuple (quads) form. </p> <h3>How?</h3> <p>SPARQL queries are actually HTTP payloads (typically). Thus, using a RESTful client-server interaction pattern, you can dispatch calls to a SPARQL compliant data server and receive a payload for local processing e.g. local object binding re. Python.</p> <h4>Steps:</h4> <ol> <li> From your command line execute: aptitude search '^python26', to verify Python is in place </li> <li>Determine which SPARQL endpoint you want to access e.g. <a href="http://dbpedia.org/sparql" id="link-id0x1d476520">DBpedia</a> or a local <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1bcfe140">Virtuoso</a> instance (typically: http://localhost:8890/sparql). </li> <li>If using Virtuoso, and you want to populate its quad store using SPARQL, assign "<a href="http://docs.openlinksw.com/virtuoso/rdfsparql.html#rdfsupportedprotocolendpointuri" id="link-id0x1c7630b8">SPARQL_SPONGE</a>" privileges to user "SPARQL" (this is basic control, more sophisticated WebID based ACLs are available for controlling SPARQL access).</li> </ol> <h4>Script:</h4> <pre> #!/usr/bin/env python # # Demonstrating use of a single query to populate a # Virtuoso Quad Store via Python. # import urllib, json # HTTP <a class="auto-href" href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id0x1bd91cf0">URL</a> is constructed accordingly with JSON query results format in mind. def sparqlQuery(query, baseURL, format="application/json"): params={ "default-graph": "", "should-sponge": "soft", "query": query, "debug": "on", "timeout": "", "format": format, "save": "display", "fname": "" } querypart=urllib.urlencode(params) response = urllib.urlopen(baseURL,querypart).read() return json.loads(response) # Setting Data Source Name (DSN) dsn="http://dbpedia.org/resource/DBpedia" # Virtuoso pragmas for instructing SPARQL engine to perform an HTTP GET # using the IRI in FROM clause as Data Source URL query="""DEFINE get:soft "replace" SELECT DISTINCT * FROM <%s> WHERE {?s ?p ?o}""" % dsn data=sparqlQuery(query, "http://localhost:8890/sparql/") print "Retrieved data:\n" + json.dumps(data, sort_keys=True, indent=4) # # End </pre> <h4>Output</h4> <pre> Retrieved data: { "head": { "link": [], "vars": [ "s", "p", "o" ] }, "results": { "bindings": [ { "o": { "type": "<a class="auto-href" href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x1b1470b8">uri</a>", "value": "http://www.w3.org/2002/07/owl#Thing" }, "p": { "type": "uri", "value": "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" }, "s": { "type": "uri", "value": "http://dbpedia.org/resource/DBpedia" } }, ... </pre> <h3>Conclusion</h3> <p> JSON was chosen over XML (re. output format) since this is about a "no-brainer installation and utilization" guide for a Python developer that already knows how to use Python for HTTP based data access. SPARQL just provides an added bonus to URL dexterity (delivered via URI abstraction) with regards to constructing Data Source Names or Addresses.</p> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1648" id="link-id0x1c9e26b0">SPARQL Guide for the Ruby Developer</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1646" id="link-id0x1c7af188">Simple Guide for using SPARQL with Virtuoso</a> </li> <li> <a href="http://www.delicious.com/kidehen/sparql_tutorial" id="link-id0x1ac1ba48">General SPARQL Tutorial Collection</a> </li> <li> <a href="http://www.delicious.com/kidehen/virtuoso_sparql_tutorial" id="link-id0x1c7be660">Virtuoso Specific SPARQL Tutorial Collection</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1567" id="link-id0x1c52b438">The URI, URL, and Linked Data Meme's Generic HTTP URI</a>. </li> </ul>
2011-01-25T10:35:46-05:00
Virtuoso Directions for 2011
http://www.openlinksw.com:443/blog/vdb/blog/?date=2011-01-19#1650
2011-01-19T16:29:37Z
<p> <a href="http://www.openlinksw.com/weblog/oerling/?id=1603" id="link-id0x1d584720">At the start of 2010, I wrote</a> that 2010 would be the year when <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x2007b778">RDF</a> became performance- and cost-competitive with relational technology for <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x7f5bf68">data</a> warehousing and analytics. More specifically, RDF would shine where data was heterogenous and/or where there was a high frequency of <a class="auto-href" href="http://dbpedia.org/resource/Database_schema" id="link-id0x1ffa18b0">schema</a> change.</p> <p>I will now discuss what we have done towards this end in 2010 and how you will gain by this in 2011.</p> <p>At the start of 2010, we had internally demonstrated 4x space efficiency gains from column-wise compression and 3x loop join speed gains from vectored execution. To recap, <i>column-wise compression</i> means a column-wise storage layout where values of consecutive rows of a single column are consecutive in memory/disk and are compressed in a manner that benefits from the homogenous data type and possible sort order of the column. <i>Vectored execution</i> means passing large numbers of query variable bindings between query operators and possibly sorting inputs to joins for improving locality. Furthermore, always operating on large sets of values gives extra opportunities for parallelism, from instruction level to threads to scale out.</p> <p>So, during 2010, we integrated these technologies into <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1fdf3f90">Virtuoso</a>, for relational- and graph-based applications alike. Further, even if we say that RDF will be close to relational speed in Virtuoso, the point is moot if Virtuoso's relational speed is not up there with the best of analytics-oriented <a class="auto-href" href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x7bf0d40">RDBMS</a>. RDF performance does rest on the basis of general-purpose database performance; what is sauce for the goose is sauce for the gander. So we reimplemented <code><a class="auto-href" href="http://dbpedia.org/resource/Hash_join" id="link-id0x7815c60">HASH JOIN</a></code> and <code>GROUP BY</code>, and fine-tuned many of the tricks required by <a class="auto-href" href="http://www.tpc.org/" id="link-id0x213d6de8">TPC</a>-<a class="auto-href" href="http://dbpedia.org/resource/TPC-H" id="link-id0x1fd92690">H. TPC-H</a> is not the sole final destination, but it is a step on the way and a valuable checklist for what a database ought to do.</p> <p>At the Semdata workshop of <a class="auto-href" href="http://www.vldb2010.org/" id="link-id0x21178a50">VLDB 2010</a> <a href="http://www.openlinksw.com/weblog/oerling/?id=1632" id="link-id0x1de8fee8">we presented some results</a> of our column store applied to RDF and relational tasks. As noted in the paper, the implementation did demonstrate significant gains over the previous row-wise architecture but was not yet well optimized, so not ready to be compared with the best of the relational analytics world. A good part of the fall of 2010 went into optimizing the column store and completing functionality such as transaction support with columns.</p> <p>A lot of this work is not specifically RDF oriented, but all of this work is constantly informed by the specific requirements of RDF. For example, the general idea of vectored execution is to eliminate overheads and optimize <a class="auto-href" href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x7ae0d58">CPU</a> <a class="auto-href" href="http://dbpedia.org/resource/Cache" id="link-id0x7bb7150">cache</a> and other locality by doing single query operations on arrays of operands so that the whole batch runs more or less in CPU cache. Are the gains not lost if data is typed at run time, as in RDF? In fact, the cost of run-time-typing turns out to be small, since data in practice tends to be of homogenous type and with locality of reference in values. Virtuoso's column store implementation resembles in broad outline other column stores like <a class="auto-href" href="http://www.vertica.com/" id="link-id0x7f61080">Vertica</a> or <a class="auto-href" href="http://www.ingres.com/vectorwise/" id="link-id0x2154ce38">VectorWise</a>, the main difference being the built-in support for run-time heterogenous types.</p> <p>The <a class="auto-href" href="http://lod2.eu/" id="link-id0x755e668">LOD2</a> EU FP 7 project <a href="http://www.openlinksw.com/weblog/oerling/?id=1630" id="link-id0x1d8eaf28">started in September 2010</a>. In this project OpenLink and the celebrated heroes of the column store, <a class="auto-href" href="http://dbpedia.org/resource/National_Research_Institute_for_Mathematics_and_Computer_Science" id="link-id0x1feba470">CWI</a> of <a class="auto-href" href="http://dbpedia.org/resource/MonetDB" id="link-id0x223bbe70">MonetDB</a> and VectorWise fame, represent the database side.</p> <p>The first database task of LOD2 is making a survey of the state of the art and a round of benchmarking of RDF stores. The <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x20f50c20">Berlin SPARQL Benchmark</a> (<a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x780c430">BSBM</a>) has accordingly evolved to include a business intelligence section and an update stream. Initial results from running these will become available in February/March, 2011. The specifics of this process merit another post; let it for now be said that benchmarking is making progress. In the end, it is our conviction that we need a situation where vendors may publish results as and when they are available and where there exists a well defined process for documenting and checking results.</p> <p>LOD2 will continue by <i>linking the universe,</i> as I half-facetiously put it on a presentation slide. This means alignment of anything from schema to instance identifiers, with and without supervision, and always with provenance, summarization, visualization, and so forth. In fact, putting it this way, this gets to sound like the old chimera of generating applications from data or allowing users to derive actionable intelligence from data of which they do not even know the structure. No, we are not that unrealistic. But we are moving toward more ad-hoc discovery and faster time to answer. And since we provide an infrastructure element under all this, we want to do away with the "RDF tax," by which we mean any significant extra cost of RDF compared to an alternate technology. To put it another way, you ought to pay for unpredictable heterogeneity or complex inference only when you actually use them, not as a fixed up-front overhead.</p> <p>So much for promises. When will you see something? It is safe to say that we cannot very well publish benchmarks of systems that are not generally available in some form. This places an initial technology preview cut of Virtuoso 7 with vectored execution somewhere in January or early February. The column store feature will be built in, but more than likely the row-wise compressed RDF format of Virtuoso 6 will still be the default. Version 6 and 7 databases will be interchangeable unless column-store structures are used.</p> <p>For now, our priority is to release the substantial gains that have already been accomplished.</p> <p>After an initial preview cut, we will return to the agenda of making sure Virtuoso is up there with the best in relational analytics, and that the equivalent workload with an RDF data model runs as close as possible to relational performance. As a first step this means taking TPC-H as is, and then converting the data and queries to the trivially equivalent RDF and <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x25716618">SPARQL</a> and seeing how it goes. In <a href="http://www.openlinksw.com/weblog/oerling/?id=1627" id="link-id0x1af60d40">the September paper</a> we dabbled a little with the data at a small scale but now we must run the full set of queries at 100GB and 300GB scales, which come to about 14 billion and 42 billion triples, respectively. A well done analysis of the issues encountered, covering similarities and dissimilarities of the implementation of the workload as <a class="auto-href" href="http://dbpedia.org/resource/SQL" id="link-id0x223b0a88">SQL</a> and SPARQL, should make a good VLDB paper.</p> <p>Database performance is an entirely open-ended quest and the bag of potentially applicable tricks is as good as infinite. Having said this, it seems that the scales comfortably reached in the TPC benchmarks are more than adequate for pretty much anything one is likely to encounter in real world applications involving comparable workloads. Businesses getting over 6 million new order transactions per minute (the high score of TPC-<a class="auto-href" href="http://dbpedia.org/resource/C%2B%2B" id="link-id0x1f72a180">C</a>) or analyzing a warehouse of 60 billion orders shipped to 6 billion customers over 7 years (10000GB or 10TB TPC-H) are not very common if they exist at all.</p> <p>The real world frontier has moved on. Scaling up the TPC workloads remains a generally useful exercise that continues to contribute to the state of the art but the applications requiring this advance are changing.</p> <p>Someone once said that for a new technology to become mainstream, it needs to solve a new class of problem. Yes, while it is a preparatory step to run TPC-H translated to SPARQL without dying of overheads, there is little point in doing this in production since SQL is anyway likely better and already known, proven, and deployed.</p> <p>The new class of problem, as LOD2 sees it, is the matter of web-wide cross-organizational data integration. Web-wide does not necessarily mean crawling the whole web, but does tend to mean running into significant heterogeneity of sources, both in terms of modeling and in terms of usage of more-or-less standard data models. Around this topic we hear two messages. The database people say that inference beyond what you can express in SQL views is theoretically nice but practically not needed; on the other side, we hear that the inference now being standardized in efforts like <a class="auto-href" href="http://dbpedia.org/resource/Rule_Interchange_Format" id="link-id0x22b3ad68">RIF</a> and <a class="auto-href" href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x22b3ad90">OWL</a> is not expressive enough for the real world. As one expert put it, <i>if enterprise data integration in the 1980s was between a few databases, today it is more like between 1000 databases,</i> which makes this matter similar to searching the web. How can one know in such a situation that the data being aggregated is in fact meaningfully aggregate-able?</p> <p>Add to this the prevalence of unstructured data in the world and the need to mine it for actionable intelligence. Think of combining data from CRM, worldwide media coverage of own and competitive brands, and in-house emails for assessing organizational response to events on the market.</p> <p>These are the actual use cases for which we need RDF at relational DW performance and scale. This is not limited to RDF and OWL profiles, since we fully believe that inference needs are more diverse. The reason why this is RDF and not SQL plus some extension of <a class="auto-href" href="http://dbpedia.org/resource/Datalog" id="link-id0x7ee5130">Datalog</a>, is the widespread adoption of RDF and <a class="auto-href" href="http://dbpedia.org/resource/Linked_Data" id="link-id0x2111f968">linked data</a> as a data publishing format, with all the schema-last and <a class="auto-href" href="http://dbpedia.org/resource/Open_world_assumption" id="link-id0x2111f990">open world</a> aspects that have been there from the start.</p> <p>Stay tuned for more news later this month!</p> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1603" id="link-id0x1de6b370">Linked Data and Virtuoso in 2010</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1510" id="link-id0x1b031180">Linked Data & The Year 2009</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1286" id="link-id0x1a582d10">Retrospective and Outlook for 2008</a> </li> </ul>
2011-01-20T12:54:42.000002-05:00
Virtuoso Directions for 2011
http://www.openlinksw.com:443/weblog/oerling/?date=2011-01-19#1649
2011-01-19T16:29:37Z
<p> <a href="http://www.openlinksw.com/weblog/oerling/?id=1603" id="link-id0x1d584720">At the start of 2010, I wrote</a> that 2010 would be the year when <a class="auto-href" href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x29d4eeb8">RDF</a> became performance- and cost-competitive with relational technology for <a class="auto-href" href="http://dbpedia.org/resource/Data" id="link-id0x1d92cc18">data</a> warehousing and analytics. More specifically, RDF would shine where data was heterogenous and/or where there was a high frequency of <a class="auto-href" href="http://dbpedia.org/resource/Database_schema" id="link-id0x1ccf1d80">schema</a> change.</p> <p>I will now discuss what we have done towards this end in 2010 and how you will gain by this in 2011.</p> <p>At the start of 2010, we had internally demonstrated 4x space efficiency gains from column-wise compression and 3x loop join speed gains from vectored execution. To recap, <i>column-wise compression</i> means a column-wise storage layout where values of consecutive rows of a single column are consecutive in memory/disk and are compressed in a manner that benefits from the homogenous data type and possible sort order of the column. <i>Vectored execution</i> means passing large numbers of query variable bindings between query operators and possibly sorting inputs to joins for improving locality. Furthermore, always operating on large sets of values gives extra opportunities for parallelism, from instruction level to threads to scale out.</p> <p>So, during 2010, we integrated these technologies into <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1c294c00">Virtuoso</a>, for relational- and graph-based applications alike. Further, even if we say that RDF will be close to relational speed in Virtuoso, the point is moot if Virtuoso's relational speed is not up there with the best of analytics-oriented <a class="auto-href" href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1d7bc8d0">RDBMS</a>. RDF performance does rest on the basis of general-purpose database performance; what is sauce for the goose is sauce for the gander. So we reimplemented <code><a class="auto-href" href="http://dbpedia.org/resource/Hash_join" id="link-id0x29d25c10">HASH JOIN</a></code> and <code>GROUP BY</code>, and fine-tuned many of the tricks required by <a class="auto-href" href="http://www.tpc.org/" id="link-id0x8ce58b8">TPC</a>-<a class="auto-href" href="http://dbpedia.org/resource/TPC-H" id="link-id0x1d610298">H. TPC-H</a> is not the sole final destination, but it is a step on the way and a valuable checklist for what a database ought to do.</p> <p>At the Semdata workshop of <a class="auto-href" href="http://www.vldb2010.org/" id="link-id0x1950c050">VLDB 2010</a> <a href="http://www.openlinksw.com/weblog/oerling/?id=1632" id="link-id0x1de8fee8">we presented some results</a> of our column store applied to RDF and relational tasks. As noted in the paper, the implementation did demonstrate significant gains over the previous row-wise architecture but was not yet well optimized, so not ready to be compared with the best of the relational analytics world. A good part of the fall of 2010 went into optimizing the column store and completing functionality such as transaction support with columns.</p> <p>A lot of this work is not specifically RDF oriented, but all of this work is constantly informed by the specific requirements of RDF. For example, the general idea of vectored execution is to eliminate overheads and optimize <a class="auto-href" href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x1d75a9c0">CPU</a> <a class="auto-href" href="http://dbpedia.org/resource/Cache" id="link-id0x1ce80608">cache</a> and other locality by doing single query operations on arrays of operands so that the whole batch runs more or less in CPU cache. Are the gains not lost if data is typed at run time, as in RDF? In fact, the cost of run-time-typing turns out to be small, since data in practice tends to be of homogenous type and with locality of reference in values. Virtuoso's column store implementation resembles in broad outline other column stores like <a class="auto-href" href="http://www.vertica.com/" id="link-id0x1b303538">Vertica</a> or <a class="auto-href" href="http://www.ingres.com/vectorwise/" id="link-id0x279f6968">VectorWise</a>, the main difference being the built-in support for run-time heterogenous types.</p> <p>The <a class="auto-href" href="http://lod2.eu/" id="link-id0x29d48f00">LOD2</a> EU FP 7 project <a href="http://www.openlinksw.com/weblog/oerling/?id=1630" id="link-id0x1d8eaf28">started in September 2010</a>. In this project OpenLink and the celebrated heroes of the column store, <a class="auto-href" href="http://dbpedia.org/resource/National_Research_Institute_for_Mathematics_and_Computer_Science" id="link-id0x19ec66f0">CWI</a> of <a class="auto-href" href="http://dbpedia.org/resource/MonetDB" id="link-id0x1cda7178">MonetDB</a> and VectorWise fame, represent the database side.</p> <p>The first database task of LOD2 is making a survey of the state of the art and a round of benchmarking of RDF stores. The <a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1e144608">Berlin SPARQL Benchmark</a> (<a class="auto-href" href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x139cd920">BSBM</a>) has accordingly evolved to include a business intelligence section and an update stream. Initial results from running these will become available in February/March, 2011. The specifics of this process merit another post; let it for now be said that benchmarking is making progress. In the end, it is our conviction that we need a situation where vendors may publish results as and when they are available and where there exists a well defined process for documenting and checking results.</p> <p>LOD2 will continue by <i>linking the universe,</i> as I half-facetiously put it on a presentation slide. This means alignment of anything from schema to instance identifiers, with and without supervision, and always with provenance, summarization, visualization, and so forth. In fact, putting it this way, this gets to sound like the old chimera of generating applications from data or allowing users to derive actionable intelligence from data of which they do not even know the structure. No, we are not that unrealistic. But we are moving toward more ad-hoc discovery and faster time to answer. And since we provide an infrastructure element under all this, we want to do away with the "RDF tax," by which we mean any significant extra cost of RDF compared to an alternate technology. To put it another way, you ought to pay for unpredictable heterogeneity or complex inference only when you actually use them, not as a fixed up-front overhead.</p> <p>So much for promises. When will you see something? It is safe to say that we cannot very well publish benchmarks of systems that are not generally available in some form. This places an initial technology preview cut of Virtuoso 7 with vectored execution somewhere in January or early February. The column store feature will be built in, but more than likely the row-wise compressed RDF format of Virtuoso 6 will still be the default. Version 6 and 7 databases will be interchangeable unless column-store structures are used.</p> <p>For now, our priority is to release the substantial gains that have already been accomplished.</p> <p>After an initial preview cut, we will return to the agenda of making sure Virtuoso is up there with the best in relational analytics, and that the equivalent workload with an RDF data model runs as close as possible to relational performance. As a first step this means taking TPC-H as is, and then converting the data and queries to the trivially equivalent RDF and <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x29b26b50">SPARQL</a> and seeing how it goes. In <a href="http://www.openlinksw.com/weblog/oerling/?id=1627" id="link-id0x1af60d40">the September paper</a> we dabbled a little with the data at a small scale but now we must run the full set of queries at 100GB and 300GB scales, which come to about 14 billion and 42 billion triples, respectively. A well done analysis of the issues encountered, covering similarities and dissimilarities of the implementation of the workload as <a class="auto-href" href="http://dbpedia.org/resource/SQL" id="link-id0x1ce99a98">SQL</a> and SPARQL, should make a good VLDB paper.</p> <p>Database performance is an entirely open-ended quest and the bag of potentially applicable tricks is as good as infinite. Having said this, it seems that the scales comfortably reached in the TPC benchmarks are more than adequate for pretty much anything one is likely to encounter in real world applications involving comparable workloads. Businesses getting over 6 million new order transactions per minute (the high score of TPC-<a class="auto-href" href="http://dbpedia.org/resource/C%2B%2B" id="link-id0x1ce764f8">C</a>) or analyzing a warehouse of 60 billion orders shipped to 6 billion customers over 7 years (10000GB or 10TB TPC-H) are not very common if they exist at all.</p> <p>The real world frontier has moved on. Scaling up the TPC workloads remains a generally useful exercise that continues to contribute to the state of the art but the applications requiring this advance are changing.</p> <p>Someone once said that for a new technology to become mainstream, it needs to solve a new class of problem. Yes, while it is a preparatory step to run TPC-H translated to SPARQL without dying of overheads, there is little point in doing this in production since SQL is anyway likely better and already known, proven, and deployed.</p> <p>The new class of problem, as LOD2 sees it, is the matter of web-wide cross-organizational data integration. Web-wide does not necessarily mean crawling the whole web, but does tend to mean running into significant heterogeneity of sources, both in terms of modeling and in terms of usage of more-or-less standard data models. Around this topic we hear two messages. The database people say that inference beyond what you can express in SQL views is theoretically nice but practically not needed; on the other side, we hear that the inference now being standardized in efforts like <a class="auto-href" href="http://dbpedia.org/resource/Rule_Interchange_Format" id="link-id0x1cb916b0">RIF</a> and <a class="auto-href" href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x29dd4a60">OWL</a> is not expressive enough for the real world. As one expert put it, <i>if enterprise data integration in the 1980s was between a few databases, today it is more like between 1000 databases,</i> which makes this matter similar to searching the web. How can one know in such a situation that the data being aggregated is in fact meaningfully aggregate-able?</p> <p>Add to this the prevalence of unstructured data in the world and the need to mine it for actionable intelligence. Think of combining data from CRM, worldwide media coverage of own and competitive brands, and in-house emails for assessing organizational response to events on the market.</p> <p>These are the actual use cases for which we need RDF at relational DW performance and scale. This is not limited to RDF and OWL profiles, since we fully believe that inference needs are more diverse. The reason why this is RDF and not SQL plus some extension of <a class="auto-href" href="http://dbpedia.org/resource/Datalog" id="link-id0x1cde9dc8">Datalog</a>, is the widespread adoption of RDF and <a class="auto-href" href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1d029c80">linked data</a> as a data publishing format, with all the schema-last and <a class="auto-href" href="http://dbpedia.org/resource/Open_world_assumption" id="link-id0x1d81f5b0">open world</a> aspects that have been there from the start.</p> <p>Stay tuned for more news later this month!</p> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1603" id="link-id0x1de6b370">Linked Data and Virtuoso in 2010</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1510" id="link-id0x1b031180">Linked Data & The Year 2009</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1286" id="link-id0x1a582d10">Retrospective and Outlook for 2008</a> </li> </ul>
2011-01-20T12:54:30.000002-05:00
SPARQL for the Ruby Developer
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2011-01-18#1648
2011-01-18T19:48:34Z
<h3>What?</h3> <p>A simple guide usable by any <a class="auto-href" href="http://dbpedia.org/resource/Ruby_programming_language" id="link-id0x1bb88908">Ruby</a> developer seeking to exploit <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x1ae67500">SPARQL</a> without hassles.</p> <h3>Why?</h3> <p>SPARQL is a powerful query language, results serialization format, and an HTTP based <a href="http://dbpedia.org/resource/Data">data</a> access protocol from the W3C. It provides a mechanism for accessing and integrating data across <a href="http://en.wikipedia.org/wiki/Deductive_database" id="link-id0x1bc61d88">Deductive Database Systems</a> (colloquially referred to as triple or quad stores in <a class="auto-href" href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1cc11420">Semantic Web</a> and <a class="auto-href" href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1b2e7780">Linked Data</a> circles) -- database systems (or data spaces) that manage proposition oriented records in 3-tuple (triples) or 4-tuple (quads) form. </p> <h3>How?</h3> <p>SPARQL queries are actually HTTP payloads (typically). Thus, using a RESTful client-server interaction pattern, you can dispatch calls to a SPARQL compliant data server and receive a payload for local processing e.g. local object binding re. Ruby. </p> <h4>Steps:</h4> <ol> <li> From your command line execute: aptitude search '^ruby', to verify Ruby is in place </li> <li>Determine which SPARQL endpoint you want to access e.g. <a href="http://dbpedia.org/sparql" id="link-id0x1d476520">DBpedia</a> or a local <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1bcfe140">Virtuoso</a> instance (typically: http://localhost:8890/sparql). </li> <li>If using Virtuoso, and you want to populate its quad store using SPARQL, assign "<a href="http://docs.openlinksw.com/virtuoso/rdfsparql.html#rdfsupportedprotocolendpointuri" id="link-id0x1c7630b8">SPARQL_SPONGE</a>" privileges to user "SPARQL" (this is basic control, more sophisticated WebID based ACLs are available for controlling SPARQL access).</li> </ol> <h4>Script:</h4> <pre> #!/usr/bin/env ruby # # Demonstrating use of a single query to populate a # Virtuoso Quad Store. # require 'net/http' require 'cgi' require 'csv' # # We opt for CSV based output since handling this format is straightforward in Ruby, by default. # HTTP <a class="auto-href" href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id0x1acee348">URL</a> is constructed accordingly with CSV as query results format in mind. def sparqlQuery(query, baseURL, format="text/csv") params={ "default-graph" => "", "should-sponge" => "soft", "query" => query, "debug" => "on", "timeout" => "", "format" => format, "save" => "display", "fname" => "" } querypart="" params.each { |k,v| querypart+="#{k}=#{CGI.escape(v)}&" } sparqlURL=baseURL+"?#{querypart}" response = Net::HTTP.get_response(<a class="auto-href" href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x1d24dfd8">URI</a>.parse(sparqlURL)) return CSV::parse(response.body) end # Setting Data Source Name (DSN) dsn="http://dbpedia.org/resource/DBpedia" #Virtuoso pragmas for instructing SPARQL engine to perform an HTTP GET #using the IRI in FROM clause as Data Source URL query="DEFINE get:soft \"replace\" SELECT DISTINCT * FROM <#{dsn}> WHERE {?s ?p ?o} " #Assume use of local installation of Virtuoso #otherwise you can change URL to that of a public endpoint #for example DBpedia: http://dbpedia.org/sparql data=sparqlQuery(query, "http://localhost:8890/sparql/") puts "Got data:" p data # # End </pre><h4>Output</h4> <pre> Got data: [["s", "p", "o"], ["http://dbpedia.org/resource/DBpedia", "http://www.w3.org/1999/02/22-rdf-syntax-ns#type", "http://www.w3.org/2002/07/owl#Thing"], ["http://dbpedia.org/resource/DBpedia", "http://www.w3.org/1999/02/22-rdf-syntax-ns#type", "http://dbpedia.org/ontology/Work"], ["http://dbpedia.org/resource/DBpedia", "http://www.w3.org/1999/02/22-rdf-syntax-ns#type", "http://dbpedia.org/class/yago/Software106566077"], ... </pre> <h3>Conclusion</h3> <p> <a href="http://en.wikipedia.org/wiki/Comma-separated_values" id="link-id0x1cac8420">CSV</a> was chosen over XML (re. output format) since this is about a "no-brainer installation and utilization" guide for a Ruby developer that already knows how to use Ruby for HTTP based data access. SPARQL just provides an added bonus to URL dexterity (delivered via URI abstraction) with regards to constructing Data Source Names or Addresses.</p> <h3>Related</h3> <ul> <li> <a href="http://www.taxonconcept.org/how-to/ruby-code-examples/how-do-i-use-ruby-to-query-a-sparql-endpoint.html" id="link-id0x1aa83678">SPARQL and Ruby SPARQL Client Library Example</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1646" id="link-id0x1c7af188">Simple Guide for using SPARQL with Virtuoso</a> </li> <li> <a href="http://www.delicious.com/kidehen/sparql_tutorial" id="link-id0x1ac1ba48">General SPARQL Tutorial Collection</a> </li> <li> <a href="http://www.delicious.com/kidehen/virtuoso_sparql_tutorial" id="link-id0x1c7be660">Virtuoso Specific SPARQL Tutorial Collection</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1567" id="link-id0x1c52b438">The URI, URL, and Linked Data Meme's Generic HTTP URI</a>. </li> </ul>
2011-01-25T10:17:12.000002-05:00
Simple Virtuoso Installation & Utilization Guide for SPARQL Users (Update 5)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2011-01-16#1647
2011-01-16T07:06:21Z
<h3>What is <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x1ab60ac0">SPARQL</a>?</h3> <p>A declarative query language from the W3C for querying structured propositional <a href="http://dbpedia.org/resource/Data">data</a> (in the form of 3-<a href="http://en.wikipedia.org/wiki/Tuple" id="link-id0x1b1e0010">tuple</a> [triples] or 4-tuple [quads] records) stored in a <a href="http://en.wikipedia.org/wiki/Deductive_database" id="link-id0x1cf8af98">deductive database</a> (colloquially referred to as triple or quad stores in <a class="auto-href" href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1caf5050">Semantic Web</a> and <a class="auto-href" href="http://dbpedia.org/resource/Linked_Data" id="link-id0x19d781b8">Linked Data</a> parlance).</p> <p>SPARQL is inherently platform independent. Like <a class="auto-href" href="http://dbpedia.org/resource/SQL" id="link-id0x1b879140">SQL</a>, the query language and the backend database engine are distinct. Database clients capture SPARQL queries which are then passed on to compliant backend databases.</p> <h3>Why is it important?</h3> <p>Like SQL for relational databases, it provides a powerful mechanism for accessing and joining data across one or more data partitions (named graphs identified by IRIs). The aforementioned capability also enables the construction of sophisticated Views, Reports (HTML or those produced in native form by desktop productivity tools), and data streams for other services.</p> <p>Unlike SQL, SPARQL includes result serialization formats and an HTTP based wire protocol. Thus, the ubiquity and sophistication of HTTP is integral to SPARQL i.e., client side applications (user agents) only need to be able to perform an HTTP GET against a <a class="auto-href" href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id0x1ba287e8">URL</a> en route to exploiting the power of SPARQL.</p> <h3>How do I use it, generally?</h3> <ol> <li>Locate a SPARQL endpoint (<a href="http://dbpedia.org/sparql" id="link-id0x1d7436b0">DBpedia</a>, <a href="http://lod.openlinksw.com/sparql" id="link-id0x1bf20690">LOD Cloud Cache</a>, <a href="http://semantic.data.gov" id="link-id0x1a8ebc28">Data.Gov</a>, <a href="http://linkeddata.uriburner.com/sparql" id="link-id0x1be93070">URIBurner</a>, <a href="http://www.delicious.com/kidehen/sparql_endpoint" id="link-id0x1cce9b40">others</a>), or;</li> <li>Install a SPARQL compliant database server (quad or triple store) on your desktop, workgroup server, data center, or cloud (e.g., <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtuosoEC2AMI" id="link-id0x1cd697a0">Amazon EC2 AMI</a>)</li> <li>Start the database server</li> <li>Execute SPARQL Queries via the <a href="http://lod.openlinksw.com/sparql" id="link-id0x1b99d790">SPARQL endpoint.</a> </li> </ol> <h3>How do I use SPARQL with <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1c9adc80">Virtuoso</a>?</h3> <p>What follows is a very simple guide for using SPARQL against your own instance of Virtuoso:</p> <ol> <li>Software Download and Installation</li> <li>Data Loading from Data Sources exposed at Network Addresses (e.g. HTTP URLs) using very simple methods</li> <li>Actual SPARQL query execution via SPARQL endpoint.</li> </ol> <h3>Installation Steps</h3> <ol> <li> Download <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSDownload" id="link-id0x1b795100">Virtuoso Open Source</a> or <a href="http://download.openlinksw.com/virtwiz/virtuoso.php" id="link-id0x1cce46f0">Virtuoso Commercial</a> Editions </li> <li> Run installer (if using Commercial edition of Windows Open Source Edition, otherwise follow build guide) </li> <li> Follow post-installation guide and verify installation by typing in the command: virtuoso -? (if this fails check you've followed installation and setup steps, then verify environment variables have been set) </li> <li> Start the Virtuoso server using the command: virtuoso-start.sh </li> <li> Verify you have a connection to the Virtuoso Server via the command: isql localhost (assuming you're using default DB settings) or the command: isql localhost:1112 (assuming demo database) or goto your browser and type in: http://<virtuoso-server-host-name>:[port]/conductor (e.g. http://localhost:8889/conductor for default DB or http://localhost:8890/conductor if using Demo DB) </li> <li> Go to SPARQL endpoint which is typically -- http://<virtuoso-server-host-name>:[port]/sparql </li> <li> Run a quick sample query (since the database always has system data in place): select distinct * where {?s ?p ?o} limit 50 .</li> </ol> <h3>Troubleshooting</h3> <ol> <li>Ensure environment settings are set and functional -- if using Mac OS X or Windows, so you don't have to worry about this, just start and stop your Virtuoso server using native OS services applets</li> <li>If using the Open Source Edition, follow the <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSMake#Getting%20Started" id="link-id0x1bfa7548">getting started guide</a> -- it covers PATH and startup directory location re. starting and stopping Virtuoso servers.</li> <li>Sponging (HTTP GETs against external Data Sources) within SPARQL queries is disabled by default. You can enable this feature by assigning "<a href="http://docs.openlinksw.com/virtuoso/rdfsparql.html#rdfsupportedprotocolendpointuri" id="link-id0x1d566270">SPARQL_SPONGE</a>" privileges to user "SPARQL". Note, more sophisticated security exists via <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtAuthPolicyFOAFSSL" id="link-id0x1a3c9eb8">WebID based ACLs</a>. </li> </ol> <h3>Data Loading Steps</h3> <ol> <li> Identify an RDF based structured data source of interest -- a file that contains 3-tuple / triples available at an address on a public or private HTTP based network </li> <li>Determine the Address (URL) of the RDF data source</li> <li>Go to your Virtuoso SPARQL endpoint and type in the following SPARQL query: DEFINE GET:SOFT "replace" SELECT DISTINCT * FROM <RDFDataSourceURL> WHERE {?s ?p ?o} </li> <li> All the triples in the RDF resource (data source accessed via URL) will be loaded into the Virtuoso Quad Store (using RDF Data Source URL as the internal quad store Named Graph IRI) as part of the SPARQL query processing pipeline. </li> </ol> <p> Note: the data source URL doesn't even have to be RDF based -- which is where the Virtuoso <a class="auto-href" href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id0x1d1a0978">Sponger</a> Middleware comes into play (download and install the <a href="http://s3.amazonaws.com/opldownload/uda/vad-packages/6.1/virtuoso/rdf_mappers_dav.vad" id="link-id0x1d0e1530">VAD installer package</a> first) since it delivers the following features to Virtuoso's SPARQL engine: </p> <ol> <li> Transformation of data from non RDF data sources (file content, hypermedia resources, <a href="http://dbpedia.org/resource/World_Wide_Web">web</a> services output etc..) into RDF based 3-tuples (triples)</li> <li> Cache Invalidation Scheme Construction -- thus, subsequent queries (without the define get:soft "replace" pragma will not be required bar when you forcefully want to override cache).</li> <li> If you have very large data sources like DBpedia etc. from CKAN, simply use our <a href="http://www.openlinksw.com/dataspace/dav/wiki/Main/VirtBulkRDFLoader" id="link-id0x1d19b4b0">bulk loader</a> . </li> </ol> <h3>SPARQL Endpoint Discovery</h3> <p>Public SPARQL endpoints are emerging at an ever increasing rate. Thus, we've setup up a DNS lookup service that provides access to a large number of SPARQL endpoints. Of course, this doesn't cover all existing endpoints, so if our endpoint is missing please ping <a class="auto-href" href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id0x1d634848">me</a>.</p> <p>Here are a collection of commands for using DNS-SD to discover SPARQL endpoints:</p> <ol> <li>dns-sd -B _sparql._tcp sparql.openlinksw.com -- browse for services instances</li> <li>dns-sd -Z _sparql._tcp sparql.openlinksw.com -- output results in Zone File format</li> <li></li> </ol> <h3>Related</h3> <ol> <li> <a href="http://www.ensta.fr/~diam/ruby/online/ruby-doc-stdlib/libdoc/net/http/rdoc/index.html" id="link-id0x1b156610">Using HTTP from Ruby</a> -- you can just make <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSSparqlProtocol" id="link-id0x1d024d60">SPARQL Protocol URLs</a> re. SPARQL</li> <li> <a href="http://sparql.rubyforge.org/client/" id="link-id0x1cd43a48">Using SPARQL Endpoints via Ruby</a> -- Ruby example using DBpedia endpoint</li> <li> <a href="http://wikis.openlinksw.com/dataspace/owiki/wiki/OATWikiWeb/InteractiveSparqlQueryBuilder" id="link-id0x1b9d2190">Interactive SPARQL Query By Example (QBE) tool</a> -- provides a graphical user interface (as is common in SQL realm re. query building against <a class="auto-href" href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1bfffb70">RDBMS</a> engines) that works with any SPARQL endpoint </li> <li> <a href="http://www.openlinksw.com/dataspace/dav/wiki/Main/VirtRDFInsert" id="link-id0x1ab63de0">Other methods of loading RDF data into Virtuoso</a> </li> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtSponger" id="link-id0x1ca248e0">Virtuoso Sponger</a> -- architecture and how it turns a wide variety of non RDF data sources into SPARQL accessible data </li> <li> <a href="http://ode.openlinksw.com/example.html" id="link-id0x1be34758">Using OpenLink Data Explorer</a> (ODE) to populate Virtuoso -- locate a resource of interest; click on a bookmarklet or use <a class="auto-href" href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id0x1ca84af0">context</a> menus (if using ODE extensions for Firefox, Safari, or Chrome); and you'll have SPARQL accessible data automatically inserted into your Virtuoso instance. </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1295" id="link-id0x1c9060f0">W3C's SPARQLing Data Access Ingenuity</a> -- an older generic SPARQL introduction post </li> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtSPARQLRef" id="link-id0x1cf1e298">Collection of SPARQL Query Examples </a>-- GoodRelations (Product Offers), <a class="auto-href" href="http://dbpedia.org/resource/Friend_of_a_friend" id="link-id0x1c0445d0">FOAF</a> (Profiles), <a class="auto-href" href="http://dbpedia.org/resource/SIOC" id="link-id0x1b785e48">SIOC</a> (Data Spaces -- <a href="http://ods.openlinksw.com/dataspace/dav/wiki/ODS/ODSAtomOWLRefExampleBlog" id="link-id0x1b6c9f78">Blogs</a>, <a href="http://ods.openlinksw.com/dataspace/dav/wiki/ODS/ODSAtomOWLRefExampleWiki" id="link-id0x1c188280">Wikis</a>, <a href="http://ods.openlinksw.com/dataspace/dav/wiki/ODS/ODSAtomOWLRefExampleBookmarks" id="link-id0x1a9a8f98">Bookmarks</a>, <a href="http://ods.openlinksw.com/dataspace/dav/wiki/ODS/ODSAtomOWLRefExampleFeeds" id="link-id0x1720c658">Feed Collections</a>, <a href="http://ods.openlinksw.com/dataspace/dav/wiki/ODS/ODSAtomOWLRefExampleGallery" id="link-id0x1cdba348">Photo Galleries</a>, <a href="http://ods.openlinksw.com/dataspace/dav/wiki/ODS/ODSAtomOWLRefExampleBriefcase" id="link-id0x1c8f1148">Briefcase/DropBox</a>, <a href="http://ods.openlinksw.com/dataspace/dav/wiki/ODS/ODSAtomOWLRefExampleAddressbook" id="link-id0x1b5eb7e0">AddressBook</a>, <a href="http://ods.openlinksw.com/dataspace/dav/wiki/ODS/ODSAtomOWLRefExampleCalendar" id="link-id0x1c575120">Calendars</a>, <a href="http://ods.openlinksw.com/dataspace/dav/wiki/ODS/ODSAtomOWLRefExampleDiscussions" id="link-id0x1c73be98">Discussion Forums</a>) </li> <li> <a href="http://lod.openlinksw.com/demo_queries/" id="link-id0x1b08aa00">Collection of Live SPARQL Queries against LOD Cloud Cache</a> -- simple and advanced queries. </li> </ol>
2011-01-19T10:43:35-05:00
Rough draft poem: Document, what art thou?
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-11-11#1646
2010-11-11T18:44:36Z
<em>I am the <a href="http://dbpedia.org/resource/Data">Data</a> Container, Disseminator, and Canvas.<br /> I came to be when the cognitive skills of mankind deemed oral history inadequate.<br /> I am transcendent, I take many forms, but my core purpose is constant - Container, Disseminator, and Canvas.<br /> I am dexterous, so I can be blank, partitioned horizontally, horizontally and vertically, and if you get moi excited and I'll show you fractals.<br /> I am accessible in a number of ways, across a plethora of media.<br /> I am loose, so you can access my content too.<br /> I am loose in a cool way, so you can refer to moi independent of my content.<br /> I am cool in a loose way, so you can refer to my content independent of moi.<br /> I am even cool and loose enough to let you figure out stuff from my content including how its totally distinct from moi.<br /> <strong>But...</strong> <br /> I am possessive about my coolness, so all Containment, Dissemination, and Canvas requirements must first call upon moi, wherever I might be.<br /> <strong>So...</strong> <br /> If you postulate about my demise or irrelevance, across any medium, I will punish you with confusion!<br /> <strong>Remember...</strong> <br /> I just told you who I am. <br /> <strong>Lesson to be learned..</strong> <br /> When something tells you what it is, and it is as powerful as I, best you believe it.<br /> BTW -- I am Okay with HTTP response code 200 OK :-) </em>
2010-11-12T18:08:25-05:00
7 Things Brought to You by HTTP-based Hypermedia
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-11-08#1644
2010-11-08T21:43:28Z
<p>There are some very powerful benefits that accrue from the use of <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1b498648">HTTP</a> based <a href="http://dbpedia.org/resource/Hypermedia" id="link-id0x1be1e208">Hypermedia</a>. 7 that come to mind immediately include: </p> <ol> <li>Structured & Platform Independent Enterprise <a class="auto-href" href="http://dbpedia.org/resource/Federated_database_system" id="link-id0x1ab5d6c8">Data Virtualization</a> -- concrete conceptual level access and provisioning of abstract domain entities such as Customers, Orders, Employees, Products, Countries, Competitors etc.</li> <li>Distributed Application State (<a href="http://dbpedia.org/resource/Representational_State_Transfer" id="link-id0x1a8a0e38">REST</a>) -- application state transitions via links</li> <li> Structured Data Representation (<a class="auto-href" href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1acf1aa0">Linked Data</a>) -- whole data data representation via links </li> <li> Structured Identity (<a href="http://esw.w3.org/WebID" id="link-id0x1a484548">WebID</a>) -- verifiable distributed identity </li> <li> Structured Profiles (<a class="auto-href" href="http://dbpedia.org/resource/Friend_of_a_friend" id="link-id0xa00bca8">FOAF</a>) -- platform independent profiles for people and organizations </li> <li> Articulation of Structured Value Propositions (<a href="http://www.heppnetz.de/projects/goodrelations/" id="link-id0x1a4793d0">GoodRelations</a>) -- Product & Service Offers, Business Entities, Locations, Business Hours, etc. </li> <li> Structured Collaboration Spaces (<a href="http://rdfs.org/sioc/spec/" id="link-id0x1afb8b40">SIOC</a>) -- Blogs, Wikis, File Sharing, Discussion Forums, Aggregated Feeds, Statuses, Photo Galleries, Polls etc.</li> </ol>
2010-11-08T15:29:43-05:00
6 Things That Must Remain Distinct re. Data
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-11-03#1643
2010-11-03T17:02:32Z
<p>Conflation is the tech industry's equivalent of macroeconomic inflation. Whenever it rears it head, we lose value courtesy of diminishing productivity.</p> <p>Looking retrospectively at any technology failure -- enterprises or industry at large -- you will eventually discover -- at the core -- messy conflation of at least one of the following:</p> <ol> <li> <a href="http://dbpedia.org/resource/Data">Data</a> Model (Semantics) </li> <li> Data Object (<a class="auto-href" href="http://dbpedia.org/resource/Entity" id="link-id0x138a4c88">Entity</a>) Names (Identifiers) </li> <li> Data Representation Syntax (Markup) </li> <li> Data Access Protocol </li> <li> Data Presentation Syntax (Markup) </li> <li> Data Presentation Media. </li> </ol> <p>The <a class="auto-href" href="http://dbpedia.org/resource/Internet" id="link-id0x1b4a9918">Internet</a> & <a class="auto-href" href="http://dbpedia.org/resource/World_Wide_Web" id="link-id0x1a8f8700">World Wide Web</a> (InterWeb) are massive successes because their respective architectural cores embody the critical separation outlined above.</p> <p>The <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> of <a class="auto-href" href="http://dbpedia.org/resource/Linked_Data" id="link-id0x156246e0">Linked Data</a> is going to become a global reality, and massive success, because it leverages inherently sound architecture -- bar conflationary distractions of RDF. :-)</p>
2010-11-04T11:01:39.000002-04:00
Virtuoso Linked Data Deployment 3-Step
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-10-29#1641
2010-10-29T22:54:32Z
<p>Injecting <a class="auto-href" href="http://dbpedia.org/resource/Linked_Data" id="link-id0x17012e18">Linked Data</a> into the Web has been a major pain point for those who seek personal, service, or organization-specific variants of <a class="auto-href" href="http://dbpedia.org/resource/DBpedia" id="link-id0x196518a8">DBpedia</a>. Basically, the sequence goes something like this: </p> <ol> <li> You encounter DBpedia or the <a class="auto-href" href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1b26d008">LOD</a> Cloud Pictorial.</li> <li> You look around (typically following your nose from link to link). </li> <li> You attempt to publish your own stuff. </li> <li> You get stuck. </li> </ol> <p>The problems typically take the following form:</p> <ol> <li> Functionality confusion about the complementary Name and Address functionality of a single <a class="auto-href" href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0xa108a00">URI</a> abstraction </li> <li> Terminology confusion due to conflation and over-loading of terms such as Resource, <a class="auto-href" href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id0x1b3d08f8">URL</a>, Representation, Document, etc. </li> <li> Inability to find robust tools with which to generate Linked Data from existing data sources such as relational databases, CSV files, XML, Web Services, etc. </li> </ol> <p>To start addressing these problems, here is a simple guide for generating and publishing Linked Data using <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1a7841e0">Virtuoso</a>.</p> <h3>Step 1 - RDF Data Generation</h3> <p>Existing RDF data can be added to the Virtuoso RDF Quad Store via a variety of built-in data loader utilities.</p> <p>Many options allow you to easily and quickly generate RDF data from other data sources:</p> <ul> <li> Install the Sponger Bookmarklet for the <a href="http://uriburner.com" id="link-id0x1aa50800">URIBurner service</a>. Bind this to your own <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x1a4255e0">SPARQL</a>-compliant backend RDF database (in this scenario, your local Virtuoso instance), and then Sponge some HTTP-accessible resources. </li> <li> Convert relational DBMS data to RDF using the Virtuoso RDF Views Wizard. </li> <li> Starting with CSV files, you can <ul> <li>Place them at an HTTP-accessible location, and use the Virtuoso <a class="auto-href" href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id0x16f7ba58">Sponger</a> to convert them to RDF or; </li> <li> Use the CVS import feature to import their content into Virtuoso's relational data engine; then use the built-in RDF Views Wizard as with other <a class="auto-href" href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1982ea80">RDBMS</a> data. </li> </ul> </li> <li> Starting from XML files, you can <ul> <li> Use Virtuoso's inbuilt XSLT-Processor for manual XML to RDF/XML transformation or;</li> <li>Leverage the Sponger Cartridge for <a class="auto-href" href="http://dbpedia.org/resource/GRDDL" id="link-id0x1b350968">GRDDL</a>, if there is a transformation service associated with your XML data source, or;</li> <li>Let the Sponger analyze the XML data source and make a best-effort transformation to RDF.</li> </ul> </li> </ul> <h3>Step 2 - Linked Data Deployment</h3> <p> Install the <a href="http://download.openlinksw.com/packages/6.2/virtuoso/fct_dav.vad" id="link-id0x19845ad0">Faceted Browser VAD package (<code>fct_dav.vad</code>)</a> which delivers the following:</p> <ol> <li> Faceted Browser Engine UI</li> <li> Dynamic Hypermedia Resource Generator <ul> <li>delivers descriptor resources for every <a class="auto-href" href="http://dbpedia.org/resource/Entity" id="link-id0x1b3a69f0">entity</a> (data object) in the Native or Virtual Quad Stores</li> <li>supports a broad array of output formats, including HTML+<a class="auto-href" href="http://dbpedia.org/resource/RDFa" id="link-id0x1a92d2f8">RDFa</a>, RDF/XML, N3/Turtle, NTriples, RDF-JSON, OData+Atom, and OData+JSON. </li> </ul> </li> </ol> <h3>Step 3 - Linked Data Consumption & Exploitation</h3> <p> Three simple steps allow you, your enterprise, and your customers to consume and exploit your newly deployed Linked Data -- </p> <ol> <li> Load a page like this in your browser: <code>http://<cname>[:<port>]/describe/?uri=<entity-uri></code> <ul> <li> <code><cname>[:<port>]</code> gets replaced by the host and port of your Virtuoso instance</li> <li> <code><entity-uri></code> gets replaced by the URI you want to see described -- for instance, the URI of one of the resources you let the Sponger handle. </li> </ul> </li> <li> Follow the links presented in the descriptor page. </li> <li>If you ever see a blank page with a hyperlink subject name in the About: section at the top of the page, simply add the parameter "&sp=1" to the URL in the browser's Address box, and hit [ENTER]. This will result in an "on the fly" resource retrieval, transformation, and descriptor page generation.</li> <li> Use the navigator controls to page up and down the data associated with the "in scope" resource descriptor. </li> </ol> <h3>Related</h3> <ul> <li> <a href="http://linkeddata.uriburner.com/describe/?url=http%3A%2F%2Flinkeddata.uriburner.com%2Fabout%2Fid%2Fentity%2Fhttp%2Fwww.amazon.com%2Fo%2FASIN%2F006251587X" id="link-id0x1a8aeaf8">Sample Descriptor Page</a> (what you see post completion of the steps in this post) </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1639" id="link-id0x1af66f38">What is Linked Data, really?</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1613" id="link-id0x1acdbc58">Painless Linked Data Generation via URIBurner</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/dav/wiki/Main/VirtRDFInsert" id="link-id0x1abe3b18">How To Load RDF Data Into Virtuoso</a> (various methods)</li> <li> <a href="http://www.openlinksw.com/dataspace/dav/wiki/Main/VirtBulkRDFLoader" id="link-id0x1a441ff0">Virtuoso Bulk Loader Script for RDF</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/dav/wiki/Main/VirtCsvFileBulkLoader" id="link-id0x190382e8">Bulk Loader Script for CSV</a> </li> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtRdb2RDFViewsGeneration#OneClickLinkedDataGenerationAndDemployment" id="link-id0x1ac9c9c0">Wizard based generation of RDF based Linked Data from ODBC accessible Relational Databases </a> </li> </ul>
2010-11-02T11:57:47.000001-04:00
Virtuoso Linked Data Deployment In 3 Simple Steps
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-10-29#1642
2010-10-29T22:54:32Z
<p>Injecting <a class="auto-href" href="http://dbpedia.org/resource/Linked_Data" id="link-id0x17012e18">Linked Data</a> into the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> has been a major pain point for those who seek personal, service, or organization-specific variants of <a class="auto-href" href="http://dbpedia.org/resource/DBpedia" id="link-id0x196518a8">DBpedia</a>. Basically, the sequence goes something like this: </p> <ol> <li> You encounter DBpedia or the <a class="auto-href" href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1b26d008">LOD</a> Cloud Pictorial.</li> <li> You look around (typically following your nose from link to link). </li> <li> You attempt to publish your own stuff. </li> <li> You get stuck. </li> </ol> <p>The problems typically take the following form:</p> <ol> <li> Functionality confusion about the complementary Name and Address functionality of a single <a class="auto-href" href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0xa108a00">URI</a> abstraction </li> <li> Terminology confusion due to conflation and over-loading of terms such as Resource, <a class="auto-href" href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id0x1b3d08f8">URL</a>, Representation, Document, etc. </li> <li> Inability to find robust tools with which to generate Linked Data from existing <a href="http://dbpedia.org/resource/Data">data</a> sources such as relational databases, CSV files, XML, Web Services, etc. </li> </ol> <p>To start addressing these problems, here is a simple guide for generating and publishing Linked Data using <a class="auto-href" href="http://virtuoso.openlinksw.com" id="link-id0x1a7841e0">Virtuoso</a>.</p> <h3>Step 1 - RDF Data Generation</h3> <p>Existing RDF data can be added to the Virtuoso RDF Quad Store via a variety of built-in data loader utilities.</p> <p>Many options allow you to easily and quickly generate RDF data from other data sources:</p> <ul> <li> Install the Sponger Bookmarklet for the <a href="http://uriburner.com" id="link-id0x1aa50800">URIBurner service</a>. Bind this to your own <a class="auto-href" href="http://dbpedia.org/resource/SPARQL" id="link-id0x1a4255e0">SPARQL</a>-compliant backend RDF database (in this scenario, your local Virtuoso instance), and then Sponge some HTTP-accessible resources. </li> <li> Convert relational DBMS data to RDF using the Virtuoso RDF Views Wizard. </li> <li> Starting with CSV files, you can <ul> <li>Place them at an HTTP-accessible location, and use the Virtuoso <a class="auto-href" href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id0x16f7ba58">Sponger</a> to convert them to RDF or; </li> <li> Use the CVS import feature to import their content into Virtuoso's relational data engine; then use the built-in RDF Views Wizard as with other <a class="auto-href" href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1982ea80">RDBMS</a> data. </li> </ul> </li> <li> Starting from XML files, you can <ul> <li> Use Virtuoso's inbuilt XSLT-Processor for manual XML to RDF/XML transformation or;</li> <li>Leverage the Sponger Cartridge for <a class="auto-href" href="http://dbpedia.org/resource/GRDDL" id="link-id0x1b350968">GRDDL</a>, if there is a transformation service associated with your XML data source, or;</li> <li>Let the Sponger analyze the XML data source and make a best-effort transformation to RDF.</li> </ul> </li> </ul> <h3>Step 2 - Linked Data Deployment</h3> <p> Install the <a href="http://download.openlinksw.com/packages/6.2/virtuoso/fct_dav.vad" id="link-id0x19845ad0">Faceted Browser VAD package (<code>fct_dav.vad</code>)</a> which delivers the following:</p> <ol> <li> Faceted Browser Engine UI</li> <li> Dynamic Hypermedia Resource Generator <ul> <li>delivers descriptor resources for every <a class="auto-href" href="http://dbpedia.org/resource/Entity" id="link-id0x1b3a69f0">entity</a> (data object) in the Native or Virtual Quad Stores</li> <li>supports a broad array of output formats, including HTML+<a class="auto-href" href="http://dbpedia.org/resource/RDFa" id="link-id0x1a92d2f8">RDFa</a>, RDF/XML, N3/Turtle, NTriples, RDF-JSON, OData+Atom, and OData+JSON. </li> </ul> </li> </ol> <h3>Step 3 - Linked Data Consumption & Exploitation</h3> <p> Three simple steps allow you, your enterprise, and your customers to consume and exploit your newly deployed Linked Data -- </p> <ol> <li> Load a page like this in your browser: <code>http://<cname>[:<port>]/describe/?uri=<entity-uri></code> <ul> <li> <code><cname>[:<port>]</code> gets replaced by the host and port of your Virtuoso instance</li> <li> <code><entity-uri></code> gets replaced by the URI you want to see described -- for instance, the URI of one of the resources you let the Sponger handle. </li> </ul> </li> <li> Follow the links presented in the descriptor page. </li> <li>If you ever see a blank page with a hyperlink subject name in the About: section at the top of the page, simply add the parameter "&sp=1" to the URL in the browser's Address box, and hit [ENTER]. This will result in an "on the fly" resource retrieval, transformation, and descriptor page generation.</li> <li> Use the navigator controls to page up and down the data associated with the "in scope" resource descriptor. </li> </ol> <h3>Related</h3> <ul> <li> <a href="http://linkeddata.uriburner.com/describe/?url=http%3A%2F%2Flinkeddata.uriburner.com%2Fabout%2Fid%2Fentity%2Fhttp%2Fwww.amazon.com%2Fo%2FASIN%2F006251587X" id="link-id0x1a8aeaf8">Sample Descriptor Page</a> (what you see post completion of the steps in this post) </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1639" id="link-id0x1af66f38">What is Linked Data, really?</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1613" id="link-id0x1acdbc58">Painless Linked Data Generation via URIBurner</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/dav/wiki/Main/VirtRDFInsert" id="link-id0x1abe3b18">How To Load RDF Data Into Virtuoso</a> (various methods)</li> <li> <a href="http://www.openlinksw.com/dataspace/dav/wiki/Main/VirtBulkRDFLoader" id="link-id0x1a441ff0">Virtuoso Bulk Loader Script for RDF</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/dav/wiki/Main/VirtCsvFileBulkLoader" id="link-id0x190382e8">Bulk Loader Script for CSV</a> </li> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtRdb2RDFViewsGeneration#OneClickLinkedDataGenerationAndDemployment" id="link-id0x1ac9c9c0">Wizard based generation of RDF based Linked Data from ODBC accessible Relational Databases </a> </li> </ul>
2010-11-02T11:55:31.000005-04:00
Business Of Linked Data: Data Quality Factors
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-10-25#1640
2010-10-25T17:50:23Z
<p>Via my "<a class="auto-href" href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id0x1a39dd88">context</a> lenses" (i.e., my subjective view of the world) a unit of <a href="http://dbpedia.org/resource/Data">Data</a> (or Datum) is like a cube of sugar, each side representing a value factor along the following dimensions:</p> <ol> <li> Identity -- via Resolvable URIs based Names for everything </li> <li> Data Representation Format Dexterity -- e.g., HTTP based Content Negotiation which loosens the coupling between Data Model Semantics and actual Data Representation (Syntax/Markup) </li> <li> Platform Agnostic Data Access -- e.g. via ubiquitous HTTP </li> <li> Change Sensitivity -- data warehouses are like real-world warehouses, goods rot and perish overtime </li> <li>Provenance -- data about the data (metadata) that helps establish "Who", "What", "When", "Where", and at least approximate or guesstimate "Why" </li> <li> Data Mesh Navigability -- delivered via inference rules.</li> </ol> <p>The quality of service factors above nullify many of the typical concerns associated data driven business models, such as:</p> <ul> <li> Wholesale Imports (crawls) - where your data is crawled and/or imported wholesale into a new <a class="auto-href" href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id0x1b1aaa98">data space</a> with zero attribution to the source </li> <li> Lossy Attribution -- attribution is delivered in literal form which doesn't deliver branding fidelity across many value chain layers or entire life cycle of a given data item </li> <li> Service Provisioning -- effectively build any business model if you can align services with unambiguously identifiable consumers with actual data items or across entire data spaces. </li> </ul>
2010-10-25T17:09:02.000013-04:00
What is Linked Data, really?
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-10-14#1645
2010-10-14T23:10:26Z
<p> <b> <i><a class="auto-href" href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1e81beb0">Linked Data</a> </i> </b> is simply <i><a href="http://dbpedia.org/resource/Hypermedia" id="link-id0x1d9d5e30">hypermedia</a>-based structured <a href="http://dbpedia.org/resource/Data">data</a>.</i> </p> <p>Linked Data offers everyone a <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a>-scale, Enterprise-grade mechanism for platform-independent creation, curation, access, and integration of data.</p> <p>The fundamental steps to creating Linked Data are as follows:</p> <ol> <li> <p>Choose a <i>Name Reference Mechanism</i> — i.e., URIs.</p> </li> <li> <p>Choose a <i>Data Model</i> with which to Structure your Data — minimally, you need a model which clearly distinguishes</p> <ol type="a"> <li> <i>Subjects</i> (also known as <i>Entities</i>)</li> <li> <i>Subject Attributes</i> (also known as <i><a class="auto-href" href="http://dbpedia.org/resource/Entity" id="link-id0x171a1808">Entity</a> Attributes</i>), and</li> <li> <i>Attribute Values</i> (also known as <i>Subject Attribute Values</i> or <i>Entity Attribute Values</i>).</li> </ol> </li> <li> <p>Choose one or more <i>Data Representation Syntaxes</i> (also called <i>Markup Languages</i> or <i>Data Formats</i>) to use when creating <i>Resources</i> with <i>Content</i> based on your chosen <i>Data Model.</i> Some Syntaxes in common use today are HTML+<a class="auto-href" href="http://dbpedia.org/resource/RDFa" id="link-id0x1a95cc58">RDFa</a>, <a href="http://www.w3.org/DesignIssues/Notation3" id="link-id0x1f596330">N3</a>, <a href="http://www.w3.org/TeamSubmission/turtle/" id="link-id0x16fdca68">Turtle</a>, <a href="http://www.w3.org/TR/REC-rdf-syntax/" id="link-id0x1d7cf0c0">RDF/XML</a>, <a href="http://sw.nokia.com/trix/TriX.html" id="link-id0x19690b60">TriX</a>, <a href="http://dbpedia.org/resource/Extensible_Resource_Descriptor" id="link-id0x1bb46968">XRDS</a>, <a href="http://code.google.com/apis/gdata/index.html" id="link-id0x18f63f20">GData</a>, <a href="http://odata.org" id="link-id0x19aee1e0">OData</a>, <a href="http://opengraphprotocol.org/" id="link-id0x1a43eb78">OpenGraph</a>, and many others.</p> </li> <li> <p>Choose a <i><a class="auto-href" href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x19aa3900">URI</a> Scheme</i> that facilitates binding <i>Referenced Names</i> to the <i>Resources</i> which will carry your <i>Content</i> -- your <i>Structured Data.</i> </p> </li> <li> <p>Create <i>Structured Data</i> by using your chosen <i>Name Reference Mechanism,</i> your chosen <i>Data Model,</i> and your chosen <i>Data Representation Syntax,</i> as follows:</p> <ol type="a"> <li>Identify <i>Subject(s)</i> using <i>Resolvable URI(s).</i> </li> <li>Identify <i>Subject Attribute(s)</i> using <i>Resolvable URI(s).</i> </li> <li>Assign <i>Attribute Values</i> to <i>Subject Attributes.</i> These <i>Values</i> may be either <i>Literals</i> (e.g., STRINGs, BLOBs) or <i>Resolvable URIs.</i> </li> </ol> </li> </ol> <p>You can create Linked Data (hypermedia-based data representations) Resources from or for many things. Examples include: personal profiles, calendars, address books, blogs, photo albums; there are many, many more.</p> <h3>Related</h3> <ol> <li> <a href="http://socialmedia.net/linked-data-introduction" id="link-id0x1bb13d50">Linked Data an Introduction</a> -- simple introduction to Linked Data and its virtues</li> <li> <a href="http://gigaom.com/2010/10/11/jeff-jonas-big-data/" id="link-id0xa00d7e8">How Data Makes Corporations Dumb</a> -- Jeff Jonas (IBM) interview</li> <li> <a href="http://www.amundsen.com/hypermedia/" id="link-id0x18f64958">Hypermedia Types</a> -- evolving <a class="auto-href" href="http://dbpedia.org/resource/Information" id="link-id0x1903b880">information</a> portal covering different aspects of Hypermedia resource types</li> <li> <a href="http://linkeddata.uriburner.com" id="link-id0x18af0cf8">URIBurner </a>-- service that generates Linked Data from a plethora of heterogeneous data sources</li> <li> <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id0x1929eea0">Linked Data Meme</a> -- <a class="auto-href" href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id0x1e8127c8">TimbL</a> design issues note about Linked Data</li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1624" id="link-id0x18a5b768">Data 3.0 Manifesto</a> -- note about format agnostic Linked Data</li> <li> <a href="http://dbpedia.org/About" id="link-id0x19ae9338">DBpedia</a> -- large Linked Data Hub</li> <li> <a href="http://richard.cyganiak.de/2007/10/lod/" id="link-id0x14d677f8">Linked Open Data Cloud</a> -- collection of Linked Data Spaces</li> <li> <a href="http://linkedopencommerce.com" id="link-id0x17c6dbf8">Linked Open Commerce Cloud </a>-- commerce (clicks & mortar and/or clicks & clicks) oriented <a class="auto-href" href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id0x13959308">Linked Data Space</a> </li> <li> <a href="http://lod.openlinksw.com" id="link-id0x18ccb9e8">LOD Cloud Cache </a>-- massive Linked Data Space hosting most of the LOD Cloud Datasets</li> <li> <a href="http://lod2.eu" id="link-id0x1a472c20">LOD2 Initiative</a> -- EU Co-Funded Project to develop global <a class="auto-href" href="http://dbpedia.org/resource/Knowledge" id="link-id0x1c0ae7d0">knowledge</a> space from LOD</li>. </ol>
2010-11-09T13:53:01-05:00
What is Linked Data, really?
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-10-14#1639
2010-10-14T21:54:31Z
<p> <b> <i><a class="auto-href" href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1e81beb0">Linked Data</a> </i> </b> is simply <i><a href="http://dbpedia.org/resource/Hypermedia" id="link-id0x1d9d5e30">hypermedia</a>-based structured <a href="http://dbpedia.org/resource/Data">data</a>.</i> </p> <p>Linked Data offers everyone a <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a>-scale, Enterprise-grade mechanism for platform-independent creation, curation, access, and integration of data.</p> <p>The fundamental steps to creating Linked Data are as follows:</p> <ol> <li> <p>Choose a <i>Name Reference Mechanism</i> — i.e., URIs.</p> </li> <li> <p>Choose a <i>Data Model</i> with which to Structure your Data — minimally, you need a model which clearly distinguishes</p> <ol type="a"> <li> <i>Subjects</i> (also known as <i>Entities</i>)</li> <li> <i>Subject Attributes</i> (also known as <i><a class="auto-href" href="http://dbpedia.org/resource/Entity" id="link-id0x171a1808">Entity</a> Attributes</i>), and</li> <li> <i>Attribute Values</i> (also known as <i>Subject Attribute Values</i> or <i>Entity Attribute Values</i>).</li> </ol> </li> <li> <p>Choose one or more <i>Data Representation Syntaxes</i> (also called <i>Markup Languages</i> or <i>Data Formats</i>) to use when creating <i>Resources</i> with <i>Content</i> based on your chosen <i>Data Model.</i> Some Syntaxes in common use today are HTML+<a class="auto-href" href="http://dbpedia.org/resource/RDFa" id="link-id0x1a95cc58">RDFa</a>, <a href="http://www.w3.org/DesignIssues/Notation3" id="link-id0x1f596330">N3</a>, <a href="http://www.w3.org/TeamSubmission/turtle/" id="link-id0x16fdca68">Turtle</a>, <a href="http://www.w3.org/TR/REC-rdf-syntax/" id="link-id0x1d7cf0c0">RDF/XML</a>, <a href="http://sw.nokia.com/trix/TriX.html" id="link-id0x19690b60">TriX</a>, <a href="http://dbpedia.org/resource/Extensible_Resource_Descriptor" id="link-id0x1bb46968">XRDS</a>, <a href="http://code.google.com/apis/gdata/index.html" id="link-id0x18f63f20">GData</a>, and <a href="http://odata.org" id="link-id0x19aee1e0">OData</a>; there are many others.</p> </li> <li> <p>Choose a <i><a class="auto-href" href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x19aa3900">URI</a> Scheme</i> that facilitates binding <i>Referenced Names</i> to the <i>Resources</i> which will carry your <i>Content</i> -- your <i>Structured Data.</i> </p> </li> <li> <p>Create <i>Structured Data</i> by using your chosen <i>Name Reference Mechanism,</i> your chosen <i>Data Model,</i> and your chosen <i>Data Representation Syntax,</i> as follows:</p> <ol type="a"> <li>Identify <i>Subject(s)</i> using <i>Resolvable URI(s).</i> </li> <li>Identify <i>Subject Attribute(s)</i> using <i>Resolvable URI(s).</i> </li> <li>Assign <i>Attribute Values</i> to <i>Subject Attributes.</i> These <i>Values</i> may be either <i>Literals</i> (e.g., STRINGs, BLOBs) or <i>Resolvable URIs.</i> </li> </ol> </li> </ol> <p>You can create Linked Data (hypermedia-based data representations) Resources from or for many things. Examples include: personal profiles, calendars, address books, blogs, photo albums; there are many, many more.</p> <h3>Related</h3> <ol> <li> <a href="http://www.amundsen.com/hypermedia/" id="link-id0x18f64958">Hypermedia Types</a> -- evolving <a class="auto-href" href="http://dbpedia.org/resource/Information" id="link-id0x1903b880">information</a> portal covering different aspects of Hypermedia resource types</li> <li> <a href="http://linkeddata.uriburner.com" id="link-id0x18af0cf8">URIBurner </a>-- service that generates Linked Data from a plethora of heterogeneous data sources</li> <li> <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id0x1929eea0">Linked Data Meme</a> -- <a class="auto-href" href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id0x1e8127c8">TimbL</a> design issues note about Linked Data</li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1624" id="link-id0x18a5b768">Data 3.0 Manifesto</a> -- note about format agnostic Linked Data</li> <li> <a href="http://dbpedia.org/About" id="link-id0x19ae9338">DBpedia</a> -- large Linked Data Hub</li> <li> <a href="http://richard.cyganiak.de/2007/10/lod/" id="link-id0x14d677f8">Linked Open Data Cloud</a> -- collection of Linked Data Spaces</li> <li> <a href="http://linkedopencommerce.com" id="link-id0x17c6dbf8">Linked Open Commerce Cloud </a>-- commerce (clicks & mortar and/or clicks & clicks) oriented <a class="auto-href" href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id0x13959308">Linked Data Space</a> </li> <li> <a href="http://lod.openlinksw.com" id="link-id0x18ccb9e8">LOD Cloud Cache </a>-- massive Linked Data Space hosting most of the LOD Cloud Datasets</li> <li> <a href="http://lod2.eu" id="link-id0x1a472c20">LOD2 Initiative</a> -- EU Co-Funded Project to develop global <a class="auto-href" href="http://dbpedia.org/resource/Knowledge" id="link-id0x1c0ae7d0">knowledge</a> space from LOD</li>. </ol>
2011-02-15T17:28:06.000002-05:00
Virtuoso 6.2 brings New Features!
http://www.openlinksw.com:443/blog/vdb/blog/?date=2010-09-22#1638
2010-09-22T21:08:24Z
<p>Virtuoso 6.2 introduces a major number of enhancements to areas including...</p> <ul> <li> Linked Data Deployment </li> <li> Linked Data Middleware</li> <li> Data Virtualization</li> <li> Dynamic Data Exchange & Data Replication </li> <li> Security</li> </ul> <p> </p> <h3> <a name="LinkedDataDeployment" id="LinkedDataDeployment"></a> Linked Data Deployment</h3> <table class="data" border="1" cellspacing="2" cellpadding="2"> <tr> <th id="0" width="15%">Feature</th> <th id="1" width="42%">Description</th> <th id="2">Benefit</th> </tr> <tr> <td align="center"> <b>Automatic Deployment</b> </td> <td> Linked Data Pages are now automatically published for every Virtuoso Data Object; users need only load their data into the RDF Quad Store. </td> <td> Handcrafted URL-Rewrite Rules are no longer necessary. </td> </tr> <tr> <td align="center"> <b>HTTP Metadata Enhancements</b> </td> <td> HTTP <code>Link:</code> header is used to transfer vital metadata (e.g., relationships between a Descriptor Resource and its Subject) from HTTP Servers to User Agents. </td> <td> Enables HTTP-oriented tools to work with such relationships and other metadata. </td> </tr> <tr> <td align="center"> <b>HTML Metadata Embedding</b> </td> <td> HTML resource <code><head /></code> and <code><link /></code> elements and their <code>@rel</code> attributes are used to transfer vital metadata (e.g., relationships between a Descriptor Resource and its Subject) from HTTP Servers to User Agents. </td> <td> Enables HTML-oriented tools to work with such relationships and other metadata. </td> </tr> <tr> <td align="center"> <b>Hammer Stack Auto-Discovery Patterns</b> </td> <td> HTML resource <code><head /></code> section and <code><link /></code> elements, the HTTP <code>Link:</code> header, and XRD-based <code>"host-meta"</code> resources collectively provide structured metadata about Virtuoso hosts, associated Linked Data Spaces, and specific Data Items (Entities). </td> <td> Enables humans and machines to easily distinguish between Descriptor Resources and their Subjects, irrespective of URI scheme. </td> </tr> </table> <h3> <a name="LinkedDataMiddleware" id="LinkedDataMiddleware"></a> Linked Data Middleware</h3> <table class="data" border="1" cellspacing="2" cellpadding="2"> <tr> <th id="3" width="15%">Feature</th> <th id="4" width="42%">Description</th> <th id="5">Benefit</th> </tr> <tr> <td align="center"> <b>New Sponger Cartridges</b> </td> <td> New cartridges (data access and transformation drivers) for Twitter, Facebook, Amazon, eBay, <nop></nop>LinkedIn, and others. </td> <td> Enable users and user agents to deal with the Sponged data spaces as though they were named graphs in a quad store, or tables in an RDBMS. </td> </tr> <tr> <td align="center"> <b>New Descriptor Pages</b> </td> <td> HTML-based descriptor pages are automatically generated. </td> <td> Descriptor subjects, and the constellation of navigable attribute-and-value pairs that constitute their descriptive representation, are clearly identified. </td> </tr> <tr> <td align="center"> <b>Automatic Subject Identifier Generation</b> </td> <td> De-referenceable data object identifiers are automatically created. </td> <td> Removes tedium and risk of error associated with nuance-laced manual construction of identifiers. </td> </tr> <tr> <td align="center"> <b>Support for OData, JSON, RDFa</b> </td> <td> Additional data representation and serialization formats associated with Linked Data. </td> <td> Increases flexibility and interoperability. </td> </tr> </table> <h3> <a name="DataVirtualization" id="DataVirtualization"></a> Data Virtualization</h3> <table class="data" border="1" cellspacing="2" cellpadding="2"> <tr> <th id="6" width="15%">Feature</th> <th id="7" width="42%">Description</th> <th id="8">Benefit</th> </tr> <tr> <td align="center"> <b>Materialized RDF Views</b> </td> <td> RDF Views over ODBC/JDBC Data Sources can now (optionally) keep the Quad Store in sync with the RDBMS data source. </td> <td> Enables high-performance Faceted Browsing while remaining sensitive to changes in the RDBMS data sources. </td> </tr> <tr> <td align="center"> <b>CSV-to-RDF Transformation</b> </td> <td> Wizard-based generation of RDF Linked Data from CSV files. </td> <td> Speeds deployment of data which may only exist in CSV form as Linked Data. </td> </tr> <tr> <td align="center"> <b>Transparent Data Access Binding</b> </td> <td> SPASQL (SPARQL Query Language integrated into SQL) is usable over ODBC, JDBC, ADO.NET, OLEDB, or XMLA connections. </td> <td> Enables Desktop Productivity Tools to transparently work with any blend of RDBMS and RDF data sources. </td> </tr> </table> <h3> <a name="DynamicDataExchangeDataReplication" id="DynamicDataExchangeDataReplication"></a> Dynamic Data Exchange & Data Replication</h3> <table class="data" border="1" cellspacing="2" cellpadding="2"> <tr> <th id="9" width="15%">Feature</th> <th id="10" width="42%">Description</th> <th id="11">Benefit</th> </tr> <tr> <td align="center"> <b>Quad Store to Quad Store Replication</b> </td> <td> High-fidelity graph-data replication between one or more database instances. </td> <td> Enables a wide variety of deployment topologies. </td> </tr> <tr> <td align="center"> <b>Delta Engine</b> </td> <td> Automated generation of deltas at the named-graph-level, matches transactional replication offered by the Virtuoso SQL engine. </td> <td> Brings RDF replication on par with SQL replication. </td> </tr> <tr> <td align="center"> <b><nop></nop>PubSubHubbub Support</b> </td> <td> Deep integration within Quad Store as an optional mechanism for shipping deltas. </td> <td> Enables push-based data replication across a variety of topologies. </td> </tr> </table> <h3> <a name="Security" id="Security"></a> Security</h3> <table class="data" border="1" cellspacing="2" cellpadding="2"> <tr> <th id="12" width="15%">Feature</th> <th id="13" width="42%">Description</th> <th id="14">Benefit</th> </tr> <tr> <td align="center"> <b><nop></nop>WebID support at the DBMS core</b> </td> <td> Use <nop></nop>WebID protocol for low-level ACL-based protection of database objects (RDF or Relational) and Web Services. </td> <td> Enables application of sophisticated security and data access policies to Web Services (e.g., SPARQL endpoint) and actual DBMS objects. </td> </tr> <tr> <td align="center"> <b>Webfinger</b> </td> <td> Supports using <code>mailto:</code> and <code>acct:</code> URIs in the context of <nop></nop>WebID and other mechanisms, when domain holders have published necessary XRDS resources. </td> <td> Enables more intuitive identification of people and organizations. </td> </tr> <tr> <td align="center"> <b>Fingerpoint</b> </td> <td> Similar to Webfinger but does not require XRDS resources; instea,d it works directly with SPARQL endpoints exposed using auto-discovery patterns in the <code><head /></code> section of HTML documents. </td> <td> Enables more intuitive identification of people and organizations. </td> </tr> </table> <p> </p>
2015-08-21T14:43:46.770292-04:00
The Business of Semantically Linked Data ("SemData")
http://www.openlinksw.com:443/blog/vdb/blog/?date=2010-09-22#1637
2010-09-22T18:20:56Z
<p>I had the opportunity the other day to converse about the semantic technology business proposition in terms of business development. My interlocutor was a business development consultant who had little prior <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x245787f0">knowledge</a> of this technology but a background in business development inside a large diversified enterprise.</p> <p>I will here recap some of the points discussed, since these can be of broader interest.</p> <b><i>Why is there no single dominant vendor?</i></b> <p>The field is young. We can take the relational database industry as a historical precedent. From the inception of the relational database around 1970, it took 15 years for the relational model to become mainstream. "Mainstream" here does not mean dominant in installed base, but does mean something that one tends to include as a component in new systems. The figure of 15 years might repeat with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x25147d40">RDF</a>, from around 1990 for the first beginnings to 2015 for routine inclusion in new systems, where applicable.</p> <p>This does not necessarily mean that the RDF graph <a href="http://dbpedia.org/resource/Data" id="link-id0x25325290">data</a> model (or more properly, <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id0x23dc23b0">EAV</a>+CR; <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id0x25d7c238">Entity</a>-Attribute-Value + Classes and Relationships) will take the place of the <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x268cd248">RDBMS</a> as the preferred data backbone. This could mean that RDF model serialization formats will be supported as data exchange mechanisms, and that systems will integrate data extracted by semantic technology from unstructured sources. Some degree of EAV storage is likely to be common, but on-line transactional data is guaranteed to stay pure relational, as EAV is suboptimal for OLTP. Analytics will see EAV alongside relational especially in applications where in-house data is being combined with large numbers of outside structured sources or with other open sources such as <a href="http://dbpedia.org/resource/Information" id="link-id0x23ce37d8">information</a> extracted from the web.</p> <p>EAV offerings will become integrated by major DBMS vendors, as is already the case with <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x26a17030">Oracle</a>. Specialized vendors will exist alongside these, just as is the case with relational databases.</p> <p> <b><i>Can there be a positive reinforcement cycle (e.g., building cars creates a need for road construction, and better roads drive demand for more cars)? Or is this an up-front infrastructure investment that governments make for some future payoff or because of science-funding policies?</i> </b> </p> <p>The Document Web did not start as a government infrastructure initiative. The infrastructure was already built, albeit first originating with the US defense establishment. The <a href="http://dbpedia.org/resource/Internet" id="link-id0x2551cf60">Internet</a> became ubiquitous through the adoption of the Web. The general public's adoption of the Web was bootstrapped by all major business and media adopting the Web. They did not adopt the web because they particularly liked it, as it was essentially a threat to the position of media and to the market dominance of big players who could afford massive advertising in this same media. Adopting the web became necessary because of the prohibitive opportunity cost of <i>not</i> adopting it.</p> <p>A similar process may take place with open data. For example, in E-commerce, vendors do not necessarily welcome easy-and-automatic machine-based comparison of their offerings against those of their competitors. Publishing data will however be necessary in order to be listed at all. Also, in social networks, we have the identity portability movement which strives to open the big social network silos. Data exchange via RDF serializations, as already supported in many places, is the natural enabling technology for this.</p> <p> <b><i>Will the web of structured data parallel the development of web 2.0?</i> </b> </p> <p>Web 2.0 was about the blogosphere, exposure of web site service APIs, creation of affiliate programs, and so forth. If the Document Web was like a universal printing press, where anybody could publish at will, Web 2.0 was a newspaper, bringing the democratization of journalism, creating the blogger, the citizen journalist. The Data Web will create the Citizen Analyst, the Mini Media Mogul (e.g., social-network-driven coops comprised of citizen journalists, analysts, and other content providers such as video and audio producers and publishers). As the blogosphere became an alternative news source to the big media, the web of data may create an ecosystem of alternative data products. Analytics is no longer a government or big business only proposition.</p> <p> <b><i>Is there a specifically semantic market or business model, or will semantic technology be exploited under established business models and merged as a component technology into existing offerings?</i> </b> </p> <p>We have seen a migration from capital expenses to operating expenses in the IT sector in general, as exemplified by cloud computing's Platform as a Service (PaaS) and Software as a Service (SaaS). It is reasonable to anticipate that this trend will continue to Data as a Service (DaaS). <a href="http://dbpedia.org/resource/Microsoft" id="link-id0x25382248">Microsoft</a> Odata and Dallas are early examples of this and go towards legitimizing the data as service concept. DaaS is not related to semantic technology <i>per se</i>, but since this will involve integration of data, RDF serializations will be attractive, especially given the takeoff of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x24d9ac10">linked data</a> in general. The data models in Odata are also much like RDF, as both stem from EAV+CR, which makes for easy translation and a degree of inherent interoperability.</p> <p>The integration of semantic technology into existing web properties and business applications will manifest to the end user as increased serendipity. The systems will be able to provide more relevant and better contextualized data for the user's situation. This applies equally to the consumer and business user cases.</p> <p>Identity virtualization in the forms of WebID and Webfinger — making first-class de-referenceable identifiers of <code>mailto:</code> and <code>acct:</code> schemes — is emerging as a new way to open social network and Web 2.0 data silos.</p> <p>On the software production side, especially as concerns data integration, the increased <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x252fd428">schema</a>- and inference-flexibility of EAV will lead to a quicker time to answer in many situations. The more complex the task or the more diverse the data, the higher the potential payoff. Data in <a href="http://dbpedia.org/resource/Cyberspace" id="link-id0x28ca9510">cyberspace</a> is mirroring the complexity and diversity of the real world, where heterogeneity and disparity are simply facts of life, and such flexibility is becoming an inescapable necessity.</p>
2010-09-22T13:44:18-04:00
The Business of Semantically Linked Data ("SemData")
http://www.openlinksw.com:443/weblog/oerling/?date=2010-09-22#1636
2010-09-22T18:20:56Z
<p>I had the opportunity the other day to converse about the semantic technology business proposition in terms of business development. My interlocutor was a business development consultant who had little prior <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x23fd79c0">knowledge</a> of this technology but a background in business development inside a large diversified enterprise.</p> <p>I will here recap some of the points discussed, since these can be of broader interest.</p> <b><i>Why is there no single dominant vendor?</i></b> <p>The field is young. We can take the relational database industry as a historical precedent. From the inception of the relational database around 1970, it took 15 years for the relational model to become mainstream. "Mainstream" here does not mean dominant in installed base, but does mean something that one tends to include as a component in new systems. The figure of 15 years might repeat with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x2682f278">RDF</a>, from around 1990 for the first beginnings to 2015 for routine inclusion in new systems, where applicable.</p> <p>This does not necessarily mean that the RDF graph <a href="http://dbpedia.org/resource/Data" id="link-id0x26d601e0">data</a> model (or more properly, <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id0x2689c570">EAV</a>+CR; <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id0x2512f268">Entity</a>-Attribute-Value + Classes and Relationships) will take the place of the <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x24423890">RDBMS</a> as the preferred data backbone. This could mean that RDF model serialization formats will be supported as data exchange mechanisms, and that systems will integrate data extracted by semantic technology from unstructured sources. Some degree of EAV storage is likely to be common, but on-line transactional data is guaranteed to stay pure relational, as EAV is suboptimal for OLTP. Analytics will see EAV alongside relational especially in applications where in-house data is being combined with large numbers of outside structured sources or with other open sources such as <a href="http://dbpedia.org/resource/Information" id="link-id0x25134320">information</a> extracted from the web.</p> <p>EAV offerings will become integrated by major DBMS vendors, as is already the case with <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x25759780">Oracle</a>. Specialized vendors will exist alongside these, just as is the case with relational databases.</p> <p> <b><i>Can there be a positive reinforcement cycle (e.g., building cars creates a need for road construction, and better roads drive demand for more cars)? Or is this an up-front infrastructure investment that governments make for some future payoff or because of science-funding policies?</i> </b> </p> <p>The Document Web did not start as a government infrastructure initiative. The infrastructure was already built, albeit first originating with the US defense establishment. The <a href="http://dbpedia.org/resource/Internet" id="link-id0x24613448">Internet</a> became ubiquitous through the adoption of the Web. The general public's adoption of the Web was bootstrapped by all major business and media adopting the Web. They did not adopt the web because they particularly liked it, as it was essentially a threat to the position of media and to the market dominance of big players who could afford massive advertising in this same media. Adopting the web became necessary because of the prohibitive opportunity cost of <i>not</i> adopting it.</p> <p>A similar process may take place with open data. For example, in E-commerce, vendors do not necessarily welcome easy-and-automatic machine-based comparison of their offerings against those of their competitors. Publishing data will however be necessary in order to be listed at all. Also, in social networks, we have the identity portability movement which strives to open the big social network silos. Data exchange via RDF serializations, as already supported in many places, is the natural enabling technology for this.</p> <p> <b><i>Will the web of structured data parallel the development of web 2.0?</i> </b> </p> <p>Web 2.0 was about the blogosphere, exposure of web site service APIs, creation of affiliate programs, and so forth. If the Document Web was like a universal printing press, where anybody could publish at will, Web 2.0 was a newspaper, bringing the democratization of journalism, creating the blogger, the citizen journalist. The Data Web will create the Citizen Analyst, the Mini Media Mogul (e.g., social-network-driven coops comprised of citizen journalists, analysts, and other content providers such as video and audio producers and publishers). As the blogosphere became an alternative news source to the big media, the web of data may create an ecosystem of alternative data products. Analytics is no longer a government or big business only proposition.</p> <p> <b><i>Is there a specifically semantic market or business model, or will semantic technology be exploited under established business models and merged as a component technology into existing offerings?</i> </b> </p> <p>We have seen a migration from capital expenses to operating expenses in the IT sector in general, as exemplified by cloud computing's Platform as a Service (PaaS) and Software as a Service (SaaS). It is reasonable to anticipate that this trend will continue to Data as a Service (DaaS). <a href="http://dbpedia.org/resource/Microsoft" id="link-id0x25d44398">Microsoft</a> Odata and Dallas are early examples of this and go towards legitimizing the data as service concept. DaaS is not related to semantic technology <i>per se</i>, but since this will involve integration of data, RDF serializations will be attractive, especially given the takeoff of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x2578b788">linked data</a> in general. The data models in Odata are also much like RDF, as both stem from EAV+CR, which makes for easy translation and a degree of inherent interoperability.</p> <p>The integration of semantic technology into existing web properties and business applications will manifest to the end user as increased serendipity. The systems will be able to provide more relevant and better contextualized data for the user's situation. This applies equally to the consumer and business user cases.</p> <p>Identity virtualization in the forms of WebID and Webfinger — making first-class de-referenceable identifiers of <code>mailto:</code> and <code>acct:</code> schemes — is emerging as a new way to open social network and Web 2.0 data silos.</p> <p>On the software production side, especially as concerns data integration, the increased <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x243877e8">schema</a>- and inference-flexibility of EAV will lead to a quicker time to answer in many situations. The more complex the task or the more diverse the data, the higher the potential payoff. Data in <a href="http://dbpedia.org/resource/Cyberspace" id="link-id0x2428fd30">cyberspace</a> is mirroring the complexity and diversity of the real world, where heterogeneity and disparity are simply facts of life, and such flexibility is becoming an inescapable necessity.</p>
2010-09-22T13:44:12.000002-04:00
VLDB Semdata Workshop
http://www.openlinksw.com:443/blog/vdb/blog/?date=2010-09-21#1635
2010-09-21T21:14:14Z
<p>I will begin by extending my thanks to the organizers, in specific <a href="http://members.deri.at/~retok" id="link-id0x236ebfd0">Reto Krummenacher</a> of <a href="http://www.sti-innsbruck.at/" id="link-id0x2371aca8">STI</a> and Atanas Kiryakov of <a href="http://dbpedia.org/resource/Ontotext" id="link-id0x22e24190">Ontotext</a> for inviting me to give a position paper at the workshop. Indeed, it is the builders of bridges, the pontifs (pontifex) amongst us who shall be remembered by history. The idea of organizing a semantic <a href="http://dbpedia.org/resource/Data" id="link-id0x23781ba8">data</a> management workshop at VLDB is a laudable attempt at rapprochement between two communities to the advantage of all concerned.</p> <p> <a href="http://semanticweb.org/id/Franz_Inc" id="link-id0x22e09fa8">Franz</a>, Ontotext, and OpenLink were the vendors present at the workshop. To summarize very briefly, <a href="http://data.semanticweb.org/person/jans-aasman" id="link-id0x2380e7c8">Jans Aasman</a> of Franz talked about the telco call center automation solution by Amdocs, where the <a href="http://semanticweb.org/id/AllegroGraph" id="link-id0x237c9408">AllegroGraph</a> <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x236f96a8">RDF</a> store is integrated. On the technical side, AllegroGraph has Javascript as a stored procedure language, which is certainly a good idea. Naso of Ontotext talked about the BBC FIFA World Cup site. The technical proposition was that materialization is good and data partitioning is not needed; a set of replicated read-only copies is good enough.</p> <p>I talked about making RDF cost competitive with relational for data integration and BI. The crux is space efficiency and column store techniques.</p> <p>One question that came up was that maybe RDF could approach relational in some things, but what about string literals being stored in a separate table? Or <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x22ff2c78">URI</a> strings being stored in a separate table?</p> <p>The answer is that if one accesses a lot of these literals the access will be local and fairly efficient. If one accesses just a few, it does not matter. For user-facing reports, there is no point in returning a million strings that the user will not read anyhow. But then it turned out that there in fact exist reports in bioinformatics where there are 100,000 strings. Now taking the worst abuse of <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x236e43f8">SPARQL</a>, a regexp over all literals in a property of a given class. With a column store this is a scan of the column; with RDF, a three table join. The join is about 10x slower than the column scan. Quite OK, considering that a full text index is the likely solution for such workloads anyway. Besides, a sensible relational <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x22e31050">schema</a> will also not use strings for foreign keys, and will therefore incur a similar burden from fetching the strings before returning the result.</p> <p>Another question was about whether the attitude was one of confrontation between RDF and relational and whether it would not be better to join forces. Well, as said in my talk, sauce for the goose is sauce for the gander and generally speaking relational techniques apply equally to RDF. There are a few RDB tricks that have no RDF equivalent, like clustering a fact table on dimension values, e.g., sales ordered by country, manufacturer, month. But by and large, column-store techniques apply. The execution engine can be essentially identical, just needing a couple of extra data types and some run-time typing and in some cases producing nulls instead of errors. Query <a href="http://dbpedia.org/resource/Program_optimization" id="link-id0x237d76e0">optimization</a> is much the same, except that RDB stats are not applicable as such; one needs to sample the data in the cost model. All in all, these adaptations to a RDB are not so large, even though they do require changes to source code.</p> <p>Another question was about combining data models, e.g., relational (rows and columns), RDF (graph), <a href="http://dbpedia.org/resource/XML" id="link-id0x23845418">XML</a> (tree), and full text. Here I would say that it is a fault of our messaging that we do not constantly repeat the necessity of this combining, as we take it for granted. Most RDF stores have a full text index on literal values. OWLIM and a <a href="http://dbpedia.org/resource/National_Research_Institute_for_Mathematics_and_Computer_Science" id="link-id0x22feefa0">CWI</a> prototype even have it for URIs. XML is a valid data type for an RDF literal, even though this does not get used very much. So doing SPARQL to select the values, and then doing <a href="http://dbpedia.org/resource/XPath" id="link-id0x235b5890">XPath</a> and XSLT on the values, is entirely possible, at least in <a href="http://virtuoso.openlinksw.com" id="link-id0x237f6428">Virtuoso</a> which has an XPath/XSLT engine built in. Same for invoking SPARQL from an XSLT sheet. Colocating a native <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x238265a8">RDBMS</a> with local and federated <a href="http://dbpedia.org/resource/SQL" id="link-id0x236f7bc8">SQL</a> is what Virtuoso has always done. One can, for example, map tables in heterogenous remote RDBs into tables in Virtuoso, then map these into RDF, and run SPARQL queries that get translated into SQL against the original tables, thereby getting SPARQL access without any materialization. Alongside this, one can ETL relational data into RDF via the same declarative mapping.</p> <p>Further, there are RDF extensions for geospatial queries in Virtuoso and AllegroGraph, and soon also in others.</p> <p>With all this cross-model operation, RDF is definitely not a closed island. We'll have to repeat this more.</p> <p>Of the academic papers, the SpiderStore (<a href="http://dbis-informatik.uibk.ac.at/5-1-Publications.html" id="link-id0x19ecd3f0">paper</a> is not yet available at time of writing, but should be soon) and <a href="http://www.few.vu.nl/~jui200/webpie.html" id="link-id0x1d60a498">Webpie</a> that should be specially noted.</p> <p>Let us talk about SpiderStore first.</p> <h2>SpiderStore</h2> <p>The SpiderStore from the University of Innsbruck is a main-memory-only system that has a record for each distinct IRI. The IRI record has one array of pointers to all IRI records that are objects where the referencing record is the subject, and a similar array of pointers to all records where the referencing record is the object. Both sets of pointers are clustered based on the predicate labeling the edge.</p> <p>According to the authors (Robert Binna, Wolfgang Gassler, Eva Zangerle, Dominic Pacher, and Günther Specht), a distinct IRI is 5 pointers and each triple is 3 pointers. This would make about 4 pointers per triple, i.e., 32 bytes with 64-bit pointers.</p> <p>This is not particularly memory efficient, since one must count unused space after growing the lists, fragmentation, etc., which will make the space consumption closer to 40 bytes per triple, plus should one add a graph to the mix one would need another pointer per distinct predicate, adding another 1-4 bytes per triple. Supporting non-IRI types in the object position is not a problem, as long as all distinct values have a chunk of memory to them with a type <a href="http://dbpedia.org/resource/Tag" id="link-id0x236fe4d0">tag</a>.</p> <p>We get a few times better memory efficiency with column compressed quads, plus we are not limited to main memory.</p> <p>But SpiderStore has a point. Making the traversal of an edge in the graph into a pointer dereference is not such a bad deal, especially if the data set is not that big. Furthermore, compiling the queries into <a href="http://dbpedia.org/resource/C%2B%2B" id="link-id0x235a2228">C</a> procedures playing with the pointers alone would give performance to match or exceed any hard coded graph traversal library and would not be very difficult. Supporting multithreaded updates would spoil much of the gain but allowing single threaded updates and forking read-only copies for reading would be fine.</p> <p>SpiderStore as such is not attractive for what we intend to do, this being aggregating RDF quads in volumes far exceeding main memory and scaling to clusters. We note that SpiderStore hits problems with distributed memory, since SpiderStore executes depth first, which is manifestly impossible if significant latencies are involved. In other words, if there can be latency, one must amortize by having a lot of other possible work available. Running with long vectors of values is one way, as in <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x236e14a0">MonetDB</a> or Virtuoso Cluster. The other way is to have a massively multithreaded platform which favors code with few instructions but little memory locality. SpiderStore could be a good fit for massive multithreading, specially if queries were compiled to C, dramatically cutting down on the count of instructions to execute.</p> <p>We too could adopt some ideas from SpiderStore. Namely, if running vectored, one just in passing, without extra overhead, generates an array of links to the next IRI, a bit like the array that SpiderStore has for each predicate for the incoming and outgoing edges of a given IRI. Of course, here these would be persistent IDs and not pointers, but a hash from one to the other takes almost no time. So, while SpiderStore alone may not be what we are after for data warehousing, Spiderizing parts of the working set would not be so bad. This is especially so since the Spiderizable data structure almost gets made as a by-product of query evaluation.</p> <p>If an algorithm made several passes over a relatively small subgraph of the whole database, Spiderizing it would accelerate things. The memory overhead could have a fixed cap so as not to ruin the working set if locality happened not to hold.</p> <p>Running a SpiderStore-like execution model on vectors instead of single values would likely do no harm and might even result in better <a href="http://dbpedia.org/resource/Cache" id="link-id0x237eb508">cache</a> behavior. The exception is in the event of completely unpredictable patterns of connections which may only be amortized by massive multithreading.</p> <h2>Webpie</h2> <p>Webpie from <a href="http://www.vu.nl/" id="link-id0x23811bf8">VU Amsterdam</a> and the <a href="http://www.larkc.eu/" id="link-id0x22ff8fe8">LarKC</a> EU FP 7 project is, as it were, the opposite of SpiderStore. This is a map-reduce-based RDFS and <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x238482a0">OWL</a> Horst inference engine which is all about breadth-first passes over the data in a map-reduce framework with intermediate disk-based storage.</p> <p>Webpie is not however a database. After the inference result has been materialized, it must be loaded into a SPARQL engine in order to evaluate a query against the result.</p> <p>The execution plan of Webpie is made from the ontology whose consequences must be materialized. The steps are sorted and run until a fixed point is reached for each. This is similar to running SPARQL <code>INSERT … SELECT</code> statements until no new inserts are produced. The only requirement is that the <code>INSERT</code> statement should report whether new inserts were actually made. This is easy to do. In this way, a comparison between map-reduce plus memory-based joining and a parallel RDF database could be made.</p> <p>We have suggested such an experiment to the LarKC people. We will see.</p>
2010-09-21T16:22:18-04:00
VLDB Semdata Workshop
http://www.openlinksw.com:443/weblog/oerling/?date=2010-09-21#1632
2010-09-21T21:14:14Z
<p>I will begin by extending my thanks to the organizers, in specific <a href="http://members.deri.at/~retok" id="link-id0x23923550">Reto Krummenacher</a> of <a href="http://www.sti-innsbruck.at/" id="link-id0x22ff3fb0">STI</a> and Atanas Kiryakov of <a href="http://dbpedia.org/resource/Ontotext" id="link-id0x236e43f8">Ontotext</a> for inviting me to give a position paper at the workshop. Indeed, it is the builders of bridges, the pontifs (pontifex) amongst us who shall be remembered by history. The idea of organizing a semantic <a href="http://dbpedia.org/resource/Data" id="link-id0x23707ad0">data</a> management workshop at VLDB is a laudable attempt at rapprochement between two communities to the advantage of all concerned.</p> <p> <a href="http://semanticweb.org/id/Franz_Inc" id="link-id0x237b5010">Franz</a>, Ontotext, and OpenLink were the vendors present at the workshop. To summarize very briefly, <a href="http://data.semanticweb.org/person/jans-aasman" id="link-id0x22e6d8e0">Jans Aasman</a> of Franz talked about the telco call center automation solution by Amdocs, where the <a href="http://semanticweb.org/id/AllegroGraph" id="link-id0x237e95a0">AllegroGraph</a> <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x23908828">RDF</a> store is integrated. On the technical side, AllegroGraph has Javascript as a stored procedure language, which is certainly a good idea. Naso of Ontotext talked about the BBC FIFA World Cup site. The technical proposition was that materialization is good and data partitioning is not needed; a set of replicated read-only copies is good enough.</p> <p>I talked about making RDF cost competitive with relational for data integration and BI. The crux is space efficiency and column store techniques.</p> <p>One question that came up was that maybe RDF could approach relational in some things, but what about string literals being stored in a separate table? Or <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x23845418">URI</a> strings being stored in a separate table?</p> <p>The answer is that if one accesses a lot of these literals the access will be local and fairly efficient. If one accesses just a few, it does not matter. For user-facing reports, there is no point in returning a million strings that the user will not read anyhow. But then it turned out that there in fact exist reports in bioinformatics where there are 100,000 strings. Now taking the worst abuse of <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x237f6320">SPARQL</a>, a regexp over all literals in a property of a given class. With a column store this is a scan of the column; with RDF, a three table join. The join is about 10x slower than the column scan. Quite OK, considering that a full text index is the likely solution for such workloads anyway. Besides, a sensible relational <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x2380cba0">schema</a> will also not use strings for foreign keys, and will therefore incur a similar burden from fetching the strings before returning the result.</p> <p>Another question was about whether the attitude was one of confrontation between RDF and relational and whether it would not be better to join forces. Well, as said in my talk, sauce for the goose is sauce for the gander and generally speaking relational techniques apply equally to RDF. There are a few RDB tricks that have no RDF equivalent, like clustering a fact table on dimension values, e.g., sales ordered by country, manufacturer, month. But by and large, column-store techniques apply. The execution engine can be essentially identical, just needing a couple of extra data types and some run-time typing and in some cases producing nulls instead of errors. Query <a href="http://dbpedia.org/resource/Program_optimization" id="link-id0x23918db8">optimization</a> is much the same, except that RDB stats are not applicable as such; one needs to sample the data in the cost model. All in all, these adaptations to a RDB are not so large, even though they do require changes to source code.</p> <p>Another question was about combining data models, e.g., relational (rows and columns), RDF (graph), <a href="http://dbpedia.org/resource/XML" id="link-id0x23915530">XML</a> (tree), and full text. Here I would say that it is a fault of our messaging that we do not constantly repeat the necessity of this combining, as we take it for granted. Most RDF stores have a full text index on literal values. OWLIM and a <a href="http://dbpedia.org/resource/National_Research_Institute_for_Mathematics_and_Computer_Science" id="link-id0x2382cc70">CWI</a> prototype even have it for URIs. XML is a valid data type for an RDF literal, even though this does not get used very much. So doing SPARQL to select the values, and then doing <a href="http://dbpedia.org/resource/XPath" id="link-id0x235a57b0">XPath</a> and XSLT on the values, is entirely possible, at least in <a href="http://virtuoso.openlinksw.com" id="link-id0x22ff2c78">Virtuoso</a> which has an XPath/XSLT engine built in. Same for invoking SPARQL from an XSLT sheet. Colocating a native <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x237cbde8">RDBMS</a> with local and federated <a href="http://dbpedia.org/resource/SQL" id="link-id0x22e63b28">SQL</a> is what Virtuoso has always done. One can, for example, map tables in heterogenous remote RDBs into tables in Virtuoso, then map these into RDF, and run SPARQL queries that get translated into SQL against the original tables, thereby getting SPARQL access without any materialization. Alongside this, one can ETL relational data into RDF via the same declarative mapping.</p> <p>Further, there are RDF extensions for geospatial queries in Virtuoso and AllegroGraph, and soon also in others.</p> <p>With all this cross-model operation, RDF is definitely not a closed island. We'll have to repeat this more.</p> <p>Of the academic papers, the SpiderStore (<a href="http://dbis-informatik.uibk.ac.at/5-1-Publications.html" id="link-id0x19ecd3f0">paper</a> is not yet available at time of writing, but should be soon) and <a href="http://www.few.vu.nl/~jui200/webpie.html" id="link-id0x1d60a498">Webpie</a> that should be specially noted.</p> <p>Let us talk about SpiderStore first.</p> <h2>SpiderStore</h2> <p>The SpiderStore from the University of Innsbruck is a main-memory-only system that has a record for each distinct IRI. The IRI record has one array of pointers to all IRI records that are objects where the referencing record is the subject, and a similar array of pointers to all records where the referencing record is the object. Both sets of pointers are clustered based on the predicate labeling the edge.</p> <p>According to the authors (Robert Binna, Wolfgang Gassler, Eva Zangerle, Dominic Pacher, and Günther Specht), a distinct IRI is 5 pointers and each triple is 3 pointers. This would make about 4 pointers per triple, i.e., 32 bytes with 64-bit pointers.</p> <p>This is not particularly memory efficient, since one must count unused space after growing the lists, fragmentation, etc., which will make the space consumption closer to 40 bytes per triple, plus should one add a graph to the mix one would need another pointer per distinct predicate, adding another 1-4 bytes per triple. Supporting non-IRI types in the object position is not a problem, as long as all distinct values have a chunk of memory to them with a type <a href="http://dbpedia.org/resource/Tag" id="link-id0x22f9d318">tag</a>.</p> <p>We get a few times better memory efficiency with column compressed quads, plus we are not limited to main memory.</p> <p>But SpiderStore has a point. Making the traversal of an edge in the graph into a pointer dereference is not such a bad deal, especially if the data set is not that big. Furthermore, compiling the queries into <a href="http://dbpedia.org/resource/C%2B%2B" id="link-id0x2370e520">C</a> procedures playing with the pointers alone would give performance to match or exceed any hard coded graph traversal library and would not be very difficult. Supporting multithreaded updates would spoil much of the gain but allowing single threaded updates and forking read-only copies for reading would be fine.</p> <p>SpiderStore as such is not attractive for what we intend to do, this being aggregating RDF quads in volumes far exceeding main memory and scaling to clusters. We note that SpiderStore hits problems with distributed memory, since SpiderStore executes depth first, which is manifestly impossible if significant latencies are involved. In other words, if there can be latency, one must amortize by having a lot of other possible work available. Running with long vectors of values is one way, as in <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x23919348">MonetDB</a> or Virtuoso Cluster. The other way is to have a massively multithreaded platform which favors code with few instructions but little memory locality. SpiderStore could be a good fit for massive multithreading, specially if queries were compiled to C, dramatically cutting down on the count of instructions to execute.</p> <p>We too could adopt some ideas from SpiderStore. Namely, if running vectored, one just in passing, without extra overhead, generates an array of links to the next IRI, a bit like the array that SpiderStore has for each predicate for the incoming and outgoing edges of a given IRI. Of course, here these would be persistent IDs and not pointers, but a hash from one to the other takes almost no time. So, while SpiderStore alone may not be what we are after for data warehousing, Spiderizing parts of the working set would not be so bad. This is especially so since the Spiderizable data structure almost gets made as a by-product of query evaluation.</p> <p>If an algorithm made several passes over a relatively small subgraph of the whole database, Spiderizing it would accelerate things. The memory overhead could have a fixed cap so as not to ruin the working set if locality happened not to hold.</p> <p>Running a SpiderStore-like execution model on vectors instead of single values would likely do no harm and might even result in better <a href="http://dbpedia.org/resource/Cache" id="link-id0x235a1f00">cache</a> behavior. The exception is in the event of completely unpredictable patterns of connections which may only be amortized by massive multithreading.</p> <h2>Webpie</h2> <p>Webpie from <a href="http://www.vu.nl/" id="link-id0x236fc430">VU Amsterdam</a> and the <a href="http://www.larkc.eu/" id="link-id0x22e53810">LarKC</a> EU FP 7 project is, as it were, the opposite of SpiderStore. This is a map-reduce-based RDFS and <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x235f4960">OWL</a> Horst inference engine which is all about breadth-first passes over the data in a map-reduce framework with intermediate disk-based storage.</p> <p>Webpie is not however a database. After the inference result has been materialized, it must be loaded into a SPARQL engine in order to evaluate a query against the result.</p> <p>The execution plan of Webpie is made from the ontology whose consequences must be materialized. The steps are sorted and run until a fixed point is reached for each. This is similar to running SPARQL <code>INSERT … SELECT</code> statements until no new inserts are produced. The only requirement is that the <code>INSERT</code> statement should report whether new inserts were actually made. This is easy to do. In this way, a comparison between map-reduce plus memory-based joining and a parallel RDF database could be made.</p> <p>We have suggested such an experiment to the LarKC people. We will see.</p>
2010-09-21T16:22:09.000004-04:00
Suggested Extensions to the BSBM
http://www.openlinksw.com:443/blog/vdb/blog/?date=2010-09-21#1634
2010-09-21T21:13:39Z
<p>Below is a list of possible extensions to the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x236e9d38">Berlin SPARQL Benchmark</a>. Our previous critique of <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x237b63c0">BSBM</a> consists of:</p> <ol> <li> <p>The queries touch very little <a href="http://dbpedia.org/resource/Data" id="link-id0x23845418">data</a>, to the point where compilation is a large fraction of execution time. This is not representative of the data integration/analytics orientation of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x237a80a0">RDF</a>. </p> </li> <li> <p>Most queries are logarithmic to scale factor, but some are linear. The linear ones come to dominate the metric at larger scales.</p> </li> <li> <p>An update stream would make the workload more realistic.</p> </li> </ol> <p>We could rectify this all with almost no changes to the data generator or test driver by adding one or two more metrics.</p> <p>So I am publishing the below as a starting point for discussion.</p> <h2>BSBM Analytics Mix</h2> <p>Below is a set of business questions that can be answered with the BSBM data set. These are more complex and touch a greater percentage of the data than the initial mix. Their evaluation is between linear and <i>n * log(n)</i> to the data size. The <a href="http://www.tpc.org/" id="link-id0x2381e420">TPC</a>-<a href="http://dbpedia.org/resource/TPC-H" id="link-id0x2380e7c8">H</a> rules can be used for a power (single user) and a throughput (multi-user, where each submits queries from the mix with different parameters and in different order). The TPC-H score formula and executive summary formats are directly applicable.</p> <p>This can be a separate metric from the "restricted" BSBM score. Restricted means "without a full scan with regexp" which will dominate the whole metric at larger scales.</p> <p>Vendor specific variations in syntax will occur, hence these are allowed but disclosure of specific query text should accompany results. Hints for <code>JOIN</code> order and the like are not allowed; queries must be declarative. We note that both <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x2380dc10">SPARQL</a> and <a href="http://dbpedia.org/resource/SQL" id="link-id0x237c98e8">SQL</a> implementations of the queries are possible.</p> <p>The queries are ordered so that the first ones fill the <a href="http://dbpedia.org/resource/Cache" id="link-id0x236f8170">cache</a>. Running the analytics mix immediately after backup post initial load is allowed, resulting in semi-warm cache. Steady-state rules will be defined later, seeing the characteristics of the actual workload.</p> <ol> <li> <p>For each country, list the top 10 product categories, ordered by the count of reviews from the country.</p> </li> <li> <p>Product with the most reviews during its first month on the market</p> </li> <li> <p>10 products most similar to X, with similarity score based on the count of features in common</p> </li> <li> <p>Top 10 reviewers of category X</p> </li> <li> <p>Product with largest increase in reviews in month X compared to month X-minus-1.</p> </li> <li> <p>Product of category X with largest change in mean price in the last month </p> </li> <li> <p>Most active American reviewer of Japanese cameras last year</p> </li> <li> <p>Correlation of price and average review</p> </li> <li> <p>Features with greatest impact on price — for features occurring in category X, find the top 10 features where the mean price with the feature is most above the mean price without the feature</p> </li> <li> <p>Country with greatest popularity of products in category X — reviews of category X from country Y divided by total reviews</p> </li> <li> <p>Leading product of category X by country, mentioning mean price in each country and number of offers, sort by number of offers</p> </li> <li> <p>Fans of manufacturer — find top reviewers who score manufacturer above their mean score</p> </li> <li> <p>Products sold only in country X</p> </li> </ol> <h2>BSBM IR</h2> <p>Since RDF stores often implement a full text index, and since a full scan with regexp matching would never be used in an online E-commerce portal, it is meaningful to extend the benchmark to have some full text queries.</p> <p>For the SPARQL implementation, text indexing should be enabled for all string-valued literals even though only some of them will be queried in the workload.</p> <ul> <li> <p>Q6 from the original mix, now allowing use of text index.</p> </li> <li> <p>Reviews of products of category X where the review contains the names of 1 to 3 product features that occur in said category of products; e.g., MP3 players with support for mp4 and ogg.</p> </li> <li> <p>ibid but now specifying review author. The intent is that structured criteria are here more selective than text.</p> </li> <li> <p>Difference in the frequency of use of "awesome", "super", and "suck(s)" by American vs. European vs. Asian review authors.</p> </li> </ul> <h2>Changes to Test Driver</h2> <p>For full text queries, the search terms have to be selected according to a realistic distribution. <a href="http://dbpedia.org/resource/Digital_Enterprise_Research_Institute" id="link-id0x2383bd48">DERI</a> has offered to provide a definition and possibly an implementation for this.</p> <p>The parameter distribution for the analytics queries will be defined when developing the queries; the intent is that one run will touch 90% of the values in the properties mentioned in the queries.</p> <p>The result report will have to be adapted to provide a TPC-H executive summary-style report and appropriate metrics.</p> <h2>Changes to Data Generation</h2> <p>For supporting the IR mix, reviews should, in addition to random text, contain the following:</p> <ul> <li> <p>For each feature in the product concerned, add the label of said feature to 60% of the reviews.</p> </li> <li> <p>Add the names of review author, product, product category, and manufacturer.</p> </li> <li> <p>The review score should be expressed in the text by adjectives (e.g., awesome, super, good, dismal, bad, sucky). Every 20th word can be an adjective from the list correlating with the score in 80% of uses of the word and random in 20%. For 90% of adjectives, pick the adjectives from lists of idiomatic expressions corresponding to the country of the reviewer. In 10% of cases, use a random list of idioms.</p> </li> <li> <p>Skew the review scores so that comparatively expensive products have a smaller chance for a bad review.</p> </li> </ul> <h2>Update Stream</h2> <p>During the benchmark run:</p> <ul> <li> <p>1% of products are added;</p> </li> <li> <p>3% of initial offers are deleted and 3% are added; and </p> </li> <li> <p>5% of reviews are added.</p> </li> </ul> <p>Updates may be divided into transactions and run in series or in parallel in a manner specified by the test sponsor. The code for loading the update stream is vendor specific but must be disclosed.</p> <p>The initial bulk load does not have to be transactional in any way.</p> <p>Loading the update stream must be transactional, guaranteeing that all <a href="http://dbpedia.org/resource/Information" id="link-id0x236f20f0">information</a> pertaining to a product or an offer constitutes a transaction. Multiple offers or products may be combined in a transaction. Queries should run at least in <code>READ COMMITTED</code> isolation, so that half-inserted products or offers are not seen.</p> <p>Full text indices do not have to be updated transactionally; the update can lag up to 2 minutes behind the insertion of the literal being indexed.</p> <p>The test data generator generates the update stream together with the initial data. The update stream is a set of files containing Turtle-serialized data for the updates, with all triples belonging to a transaction in consecutive order. The possible transaction boundaries are marked with a comment distinguishable from the text. The test sponsor may implement a special load program if desired. The files must be loaded in sequence but a single file may be loaded on any number of parallel threads.</p> <p>The data generator should generate multiple files for the initial dump in order to facilitate parallel loading.</p> <p>The same update stream can be used during all tests, starting each run from a backup containing only the initial state. In the original run, the update stream is applied starting at the measurement interval, after the SUT is in steady state.</p>
2010-09-21T16:22:15-04:00
Suggested Extensions to the BSBM
http://www.openlinksw.com:443/weblog/oerling/?date=2010-09-21#1631
2010-09-21T21:13:39Z
<p>Below is a list of possible extensions to the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x236ebfd0">Berlin SPARQL Benchmark</a>. Our previous critique of <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x237c3af0">BSBM</a> consists of:</p> <ol> <li> <p>The queries touch very little <a href="http://dbpedia.org/resource/Data" id="link-id0x22e140a8">data</a>, to the point where compilation is a large fraction of execution time. This is not representative of the data integration/analytics orientation of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x237ccb50">RDF</a>. </p> </li> <li> <p>Most queries are logarithmic to scale factor, but some are linear. The linear ones come to dominate the metric at larger scales.</p> </li> <li> <p>An update stream would make the workload more realistic.</p> </li> </ol> <p>We could rectify this all with almost no changes to the data generator or test driver by adding one or two more metrics.</p> <p>So I am publishing the below as a starting point for discussion.</p> <h2>BSBM Analytics Mix</h2> <p>Below is a set of business questions that can be answered with the BSBM data set. These are more complex and touch a greater percentage of the data than the initial mix. Their evaluation is between linear and <i>n * log(n)</i> to the data size. The <a href="http://www.tpc.org/" id="link-id0x237c81a8">TPC</a>-<a href="http://dbpedia.org/resource/TPC-H" id="link-id0x2382baf0">H</a> rules can be used for a power (single user) and a throughput (multi-user, where each submits queries from the mix with different parameters and in different order). The TPC-H score formula and executive summary formats are directly applicable.</p> <p>This can be a separate metric from the "restricted" BSBM score. Restricted means "without a full scan with regexp" which will dominate the whole metric at larger scales.</p> <p>Vendor specific variations in syntax will occur, hence these are allowed but disclosure of specific query text should accompany results. Hints for <code>JOIN</code> order and the like are not allowed; queries must be declarative. We note that both <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x235c9048">SPARQL</a> and <a href="http://dbpedia.org/resource/SQL" id="link-id0x239194b0">SQL</a> implementations of the queries are possible.</p> <p>The queries are ordered so that the first ones fill the <a href="http://dbpedia.org/resource/Cache" id="link-id0x23845240">cache</a>. Running the analytics mix immediately after backup post initial load is allowed, resulting in semi-warm cache. Steady-state rules will be defined later, seeing the characteristics of the actual workload.</p> <ol> <li> <p>For each country, list the top 10 product categories, ordered by the count of reviews from the country.</p> </li> <li> <p>Product with the most reviews during its first month on the market</p> </li> <li> <p>10 products most similar to X, with similarity score based on the count of features in common</p> </li> <li> <p>Top 10 reviewers of category X</p> </li> <li> <p>Product with largest increase in reviews in month X compared to month X-minus-1.</p> </li> <li> <p>Product of category X with largest change in mean price in the last month </p> </li> <li> <p>Most active American reviewer of Japanese cameras last year</p> </li> <li> <p>Correlation of price and average review</p> </li> <li> <p>Features with greatest impact on price — for features occurring in category X, find the top 10 features where the mean price with the feature is most above the mean price without the feature</p> </li> <li> <p>Country with greatest popularity of products in category X — reviews of category X from country Y divided by total reviews</p> </li> <li> <p>Leading product of category X by country, mentioning mean price in each country and number of offers, sort by number of offers</p> </li> <li> <p>Fans of manufacturer — find top reviewers who score manufacturer above their mean score</p> </li> <li> <p>Products sold only in country X</p> </li> </ol> <h2>BSBM IR</h2> <p>Since RDF stores often implement a full text index, and since a full scan with regexp matching would never be used in an online E-commerce portal, it is meaningful to extend the benchmark to have some full text queries.</p> <p>For the SPARQL implementation, text indexing should be enabled for all string-valued literals even though only some of them will be queried in the workload.</p> <ul> <li> <p>Q6 from the original mix, now allowing use of text index.</p> </li> <li> <p>Reviews of products of category X where the review contains the names of 1 to 3 product features that occur in said category of products; e.g., MP3 players with support for mp4 and ogg.</p> </li> <li> <p>ibid but now specifying review author. The intent is that structured criteria are here more selective than text.</p> </li> <li> <p>Difference in the frequency of use of "awesome", "super", and "suck(s)" by American vs. European vs. Asian review authors.</p> </li> </ul> <h2>Changes to Test Driver</h2> <p>For full text queries, the search terms have to be selected according to a realistic distribution. <a href="http://dbpedia.org/resource/Digital_Enterprise_Research_Institute" id="link-id0x2391c0a0">DERI</a> has offered to provide a definition and possibly an implementation for this.</p> <p>The parameter distribution for the analytics queries will be defined when developing the queries; the intent is that one run will touch 90% of the values in the properties mentioned in the queries.</p> <p>The result report will have to be adapted to provide a TPC-H executive summary-style report and appropriate metrics.</p> <h2>Changes to Data Generation</h2> <p>For supporting the IR mix, reviews should, in addition to random text, contain the following:</p> <ul> <li> <p>For each feature in the product concerned, add the label of said feature to 60% of the reviews.</p> </li> <li> <p>Add the names of review author, product, product category, and manufacturer.</p> </li> <li> <p>The review score should be expressed in the text by adjectives (e.g., awesome, super, good, dismal, bad, sucky). Every 20th word can be an adjective from the list correlating with the score in 80% of uses of the word and random in 20%. For 90% of adjectives, pick the adjectives from lists of idiomatic expressions corresponding to the country of the reviewer. In 10% of cases, use a random list of idioms.</p> </li> <li> <p>Skew the review scores so that comparatively expensive products have a smaller chance for a bad review.</p> </li> </ul> <h2>Update Stream</h2> <p>During the benchmark run:</p> <ul> <li> <p>1% of products are added;</p> </li> <li> <p>3% of initial offers are deleted and 3% are added; and </p> </li> <li> <p>5% of reviews are added.</p> </li> </ul> <p>Updates may be divided into transactions and run in series or in parallel in a manner specified by the test sponsor. The code for loading the update stream is vendor specific but must be disclosed.</p> <p>The initial bulk load does not have to be transactional in any way.</p> <p>Loading the update stream must be transactional, guaranteeing that all <a href="http://dbpedia.org/resource/Information" id="link-id0x237aa408">information</a> pertaining to a product or an offer constitutes a transaction. Multiple offers or products may be combined in a transaction. Queries should run at least in <code>READ COMMITTED</code> isolation, so that half-inserted products or offers are not seen.</p> <p>Full text indices do not have to be updated transactionally; the update can lag up to 2 minutes behind the insertion of the literal being indexed.</p> <p>The test data generator generates the update stream together with the initial data. The update stream is a set of files containing Turtle-serialized data for the updates, with all triples belonging to a transaction in consecutive order. The possible transaction boundaries are marked with a comment distinguishable from the text. The test sponsor may implement a special load program if desired. The files must be loaded in sequence but a single file may be loaded on any number of parallel threads.</p> <p>The data generator should generate multiple files for the initial dump in order to facilitate parallel loading.</p> <p>The same update stream can be used during all tests, starting each run from a backup containing only the initial state. In the original run, the update stream is applied starting at the measurement interval, after the SUT is in steady state.</p>
2010-09-21T16:22:06-04:00
LOD2 Kick Off
http://www.openlinksw.com:443/blog/vdb/blog/?date=2010-09-21#1633
2010-09-21T21:13:03Z
<p>The <a href="http://lod2.eu/" id="link-id0x22e06810">LOD2</a> <a href="http://lod2.eu/BlogPost/9-press-release-lod2-project-launch.html" id="link-id0x18c0c770">kick off meeting</a> was held in Leipzig on Sept 6-8. I will here talk about OpenLink plans as concerns LOD2; hence this is not to be taken as representative of the whole project. I will first discuss the immediate and conclude with the long term.</p> <p>As concerns OpenLink specifically, we have two short term activities, namely publishing the initial LOD2 repository in December and publishing a set of RDB and <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x22f9ba70">RDF</a> benchmarks in February.</p> <p>The LOD2 repository is a fusion of the OpenLink <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x2378d288">LOD</a> <a href="http://lod.openlinksw.com/" id="link-id0x23908828">Cloud</a> <a href="http://dbpedia.org/resource/Cache" id="link-id0x2378e6c8">Cache</a> (which includes <a href="http://dbpedia.org/resource/Data" id="link-id0x237d7d20">data</a> from <a href="http://uriburner.com/" id="link-id0x237c9408">URIBurner</a> and <a href="http://www.pingthesemanticweb.com/" id="link-id0x235b03b0">PingTheSemanticWeb</a>) and <a href="http://sindice.com/" id="link-id0x22e24190">Sindice</a>, both hosted at <a href="http://dbpedia.org/resource/Digital_Enterprise_Research_Institute" id="link-id0x237b80f8">DERI</a>. The value-add compared to Sindice or the <a href="http://virtuoso.openlinksw.com" id="link-id0x237b63c0">Virtuoso</a>-based LOD Cloud Cache alone is the merger of the timeliness and ping-ping crawling of Sindice with the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x237f7568">SPARQL</a> of Virtuoso.</p> <p>Further down the road, after we migrate the system to the Virtuoso column store, we will also see gains in performance, primarily due to much better working set, as data is many times more compact than with the present row-wise <a href="http://dbpedia.org/resource/Data_compression" id="link-id0x235b0c38">key compression</a>.</p> <p>Still further, but before next September, we will have dynamic repartitioning; the time of availability is set as this is part of the LOD2 project roadmap. The operational need for this is pushed back somewhat by the compression gains from column-wise storage.</p> <p>As for benchmarks, I just compiled <a href="http://www.openlinksw.com/weblogs/oerling/" id="link-id0x1c29e720">a draft of suggested extensions to the BSBM</a> (<a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x22e31050">Berlin SPARQL Benchmark</a>). I talked about this with <a href="http://nl.linkedin.com/in/peterboncz" id="link-id0x237c90b0">Peter Boncz</a> and <a href="http://data.semanticweb.org/person/christian-bizer" id="link-id0x23813340">Chris Bizer</a>, to the effect that some extensions of BSBM could be done but that the time was a bit short for making a RDF-specific benchmark. We do recall that BSBM is fully feasible with a relational <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x236f7ef8">schema</a> and that RDF offers no fundamental edge for the workload.</p> <p>There was a graph benchmark talk at the <a href="http://www.tpc.org/" id="link-id0x236f8170">TPC</a> workshop at <a href="http://www.vldb2010.org/" id="link-id0x235c6b90">VLDB 2010</a>. There too, the authors were suggesting a social network use case for benchmarking anything from RDF stores to graph libraries. The presentation did not include any specification of test data, so it may be that some cooperation is possible there. The need for such a benchmark is well acknowledged. The final form of this is not yet set but LOD2 will in time publish results from such.</p> <p>We did informally talk about a process for publishing with our colleagues from <a href="http://semanticweb.org/id/Franz_Inc" id="link-id0x23781d28">Franz</a> and <a href="http://dbpedia.org/resource/Ontotext" id="link-id0x23782740">Ontotext</a> at VLDB 2010. The idea is that vendors tune their own systems and do the runs and that the others check on this, preferably all using the same hardware.</p> <p>Now, the LOD2 benchmarks will also include relational-to-RDF comparisons, for example TPC-<a href="http://dbpedia.org/resource/TPC-H" id="link-id0x235a3568">H</a> in <a href="http://dbpedia.org/resource/SQL" id="link-id0x22e67370">SQL</a> and SPARQL. The SQL will be Virtuoso, <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x22e70db0">MonetDB</a>, and possibly <a href="http://www.ingres.com/vectorwise/" id="link-id0x2378f750">VectorWise</a> and others, depending on what legal restrictions apply at the time. This will give an RDF-to-SQL comparison of TPC-H at least on Virtuoso, later also on MonetDB, depending on the schedule for a MonetDB SPARQL front-end.</p> <p>In the immediate term, this of course focuses our efforts on productizing the Virtuoso column store extension and the optimizations that go with it.</p> <p>LOD2 is however about much more than database benchmarks. Over the longer term, we plan to apply suitable parts of the ground-breaking database research done at <a href="http://dbpedia.org/resource/National_Research_Institute_for_Mathematics_and_Computer_Science" id="link-id0x23911830">CWI</a> to RDF use cases.</p> <p>This involves anything from adaptive indexing, to reuse and caching of intermediate results, to adaptive execution. This is however more than just mapping column store concepts to RDF. New challenges are posed by running on clusters and dealing with more expressive queries than just SQL, in specific queries with Datalog-like rules and recursion.</p> <p>LOD2 is principally about integration and alignment, from the schema to the instance level. This involves complex batch processing, close to the data, on large volumes of data. Map-reduce is not the be-all-end-all of this. Of course, a parallel database like Virtuoso, <a href="http://dbpedia.org/resource/Greenplum" id="link-id0x22feb520">Greenplum</a>, or <a href="http://www.vertica.com/" id="link-id0x237f7428">Vertica</a> can do map-reduce style operations under control of the SQL engine. After all, the SQL engine needs to do map-reduce and a lot more to provide good throughput for parallel, distributed SQL. Something like the <a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id0x235c2e28">Berkeley Orders Of Magnitude</a> (<a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id0x2380e7c8">BOOM</a>) distributed Datalog implementation (Overlog, Deadalus, BLOOM) could be a parallel computation framework that would subsume any map-reduce-style functionality under a more elegant declarative framework while still leaving control of execution to the developer for the cases where this is needed.</p> <p>From our viewpoint, the project's gains include:</p> <ul> <li> <p>Significant narrowing of the RDB to RDF performance gap. RDF will be an option for large scale warehousing, cutting down on time to integration by providing greater schema flexibility.</p> </li> <li> <p>Ready to use toolbox for data integration, including schema alignment and resolution of coreference.</p> </li> <li> <p>Data discovery, summarization and visualization</p> </li> </ul> <p>Integrating this into a relatively unified stack of tools is possible, since these all cluster around the task of linking the universe with RDF and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x236e14a0">linked data</a>. In this respect the integration of results may be stronger than often seen in European large scale integrating projects.</p> <p>The use cases fit the development profile well: </p> <ul> <li> <p> <a href="http://dbpedia.org/resource/Wolters_Kluwer" id="link-id0x23820568">Wolters Kluwer</a> will develop an application for integrating resources around law, from the actual laws to court cases to media coverage. The content is modeled in a fine grained legal ontology.</p> </li> <li> <p> <a href="http://dbpedia.org/resource/Exalead" id="link-id0x22e50ba0">Exalead</a> will implement the linked data enterprise, addressing enterprise search and any typical enterprise data integration plus generating added value from open sources.</p> </li> <li> <p>The Open <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x236fb248">Knowledge</a> Foundation will create a portal of all government published data for easy access by citizens.</p> </li> </ul> <p>In all these cases, the integration requirements of schema alignment, resolution of identity, <a href="http://dbpedia.org/resource/Information" id="link-id0x2381ebb0">information</a> extraction, and efficient storage and retrieval play a significant role. The end user interfaces will be task-specific but developer interfaces around integration tools and query formulation may be quite generic and suited for generic RDF application development.</p>
2010-09-21T16:22:12-04:00
LOD2 Kick Off
http://www.openlinksw.com:443/weblog/oerling/?date=2010-09-21#1630
2010-09-21T21:13:03Z
<p>The <a href="http://lod2.eu/" id="link-id0x236f6368">LOD2</a> <a href="http://lod2.eu/BlogPost/9-press-release-lod2-project-launch.html" id="link-id0x18c0c770">kick off meeting</a> was held in Leipzig on Sept 6-8. I will here talk about OpenLink plans as concerns LOD2; hence this is not to be taken as representative of the whole project. I will first discuss the immediate and conclude with the long term.</p> <p>As concerns OpenLink specifically, we have two short term activities, namely publishing the initial LOD2 repository in December and publishing a set of RDB and <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x22e46c78">RDF</a> benchmarks in February.</p> <p>The LOD2 repository is a fusion of the OpenLink <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x235aaf00">LOD</a> <a href="http://lod.openlinksw.com/" id="link-id0x237a3470">Cloud</a> <a href="http://dbpedia.org/resource/Cache" id="link-id0x237d6380">Cache</a> (which includes <a href="http://dbpedia.org/resource/Data" id="link-id0x236d3830">data</a> from <a href="http://uriburner.com/" id="link-id0x22ff6f88">URIBurner</a> and <a href="http://www.pingthesemanticweb.com/" id="link-id0x235a3be8">PingTheSemanticWeb</a>) and <a href="http://sindice.com/" id="link-id0x23783d68">Sindice</a>, both hosted at <a href="http://dbpedia.org/resource/Digital_Enterprise_Research_Institute" id="link-id0x22e48ff8">DERI</a>. The value-add compared to Sindice or the <a href="http://virtuoso.openlinksw.com" id="link-id0x23904730">Virtuoso</a>-based LOD Cloud Cache alone is the merger of the timeliness and ping-ping crawling of Sindice with the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x2382c800">SPARQL</a> of Virtuoso.</p> <p>Further down the road, after we migrate the system to the Virtuoso column store, we will also see gains in performance, primarily due to much better working set, as data is many times more compact than with the present row-wise <a href="http://dbpedia.org/resource/Data_compression" id="link-id0x236e64d0">key compression</a>.</p> <p>Still further, but before next September, we will have dynamic repartitioning; the time of availability is set as this is part of the LOD2 project roadmap. The operational need for this is pushed back somewhat by the compression gains from column-wise storage.</p> <p>As for benchmarks, I just compiled <a href="http://www.openlinksw.com/weblogs/oerling/" id="link-id0x1c29e720">a draft of suggested extensions to the BSBM</a> (<a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x2391d000">Berlin SPARQL Benchmark</a>). I talked about this with <a href="http://nl.linkedin.com/in/peterboncz" id="link-id0x22e4efe8">Peter Boncz</a> and <a href="http://data.semanticweb.org/person/christian-bizer" id="link-id0x23910980">Chris Bizer</a>, to the effect that some extensions of BSBM could be done but that the time was a bit short for making a RDF-specific benchmark. We do recall that BSBM is fully feasible with a relational <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x236fd5c8">schema</a> and that RDF offers no fundamental edge for the workload.</p> <p>There was a graph benchmark talk at the <a href="http://www.tpc.org/" id="link-id0x236eaa88">TPC</a> workshop at <a href="http://www.vldb2010.org/" id="link-id0x2391e818">VLDB 2010</a>. There too, the authors were suggesting a social network use case for benchmarking anything from RDF stores to graph libraries. The presentation did not include any specification of test data, so it may be that some cooperation is possible there. The need for such a benchmark is well acknowledged. The final form of this is not yet set but LOD2 will in time publish results from such.</p> <p>We did informally talk about a process for publishing with our colleagues from <a href="http://semanticweb.org/id/Franz_Inc" id="link-id0x235a4c40">Franz</a> and <a href="http://dbpedia.org/resource/Ontotext" id="link-id0x236ec978">Ontotext</a> at VLDB 2010. The idea is that vendors tune their own systems and do the runs and that the others check on this, preferably all using the same hardware.</p> <p>Now, the LOD2 benchmarks will also include relational-to-RDF comparisons, for example TPC-<a href="http://dbpedia.org/resource/TPC-H" id="link-id0x23819e80">H</a> in <a href="http://dbpedia.org/resource/SQL" id="link-id0x2382b890">SQL</a> and SPARQL. The SQL will be Virtuoso, <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x2382b8b8">MonetDB</a>, and possibly <a href="http://www.ingres.com/vectorwise/" id="link-id0x237c2b10">VectorWise</a> and others, depending on what legal restrictions apply at the time. This will give an RDF-to-SQL comparison of TPC-H at least on Virtuoso, later also on MonetDB, depending on the schedule for a MonetDB SPARQL front-end.</p> <p>In the immediate term, this of course focuses our efforts on productizing the Virtuoso column store extension and the optimizations that go with it.</p> <p>LOD2 is however about much more than database benchmarks. Over the longer term, we plan to apply suitable parts of the ground-breaking database research done at <a href="http://dbpedia.org/resource/National_Research_Institute_for_Mathematics_and_Computer_Science" id="link-id0x2381d9a0">CWI</a> to RDF use cases.</p> <p>This involves anything from adaptive indexing, to reuse and caching of intermediate results, to adaptive execution. This is however more than just mapping column store concepts to RDF. New challenges are posed by running on clusters and dealing with more expressive queries than just SQL, in specific queries with Datalog-like rules and recursion.</p> <p>LOD2 is principally about integration and alignment, from the schema to the instance level. This involves complex batch processing, close to the data, on large volumes of data. Map-reduce is not the be-all-end-all of this. Of course, a parallel database like Virtuoso, <a href="http://dbpedia.org/resource/Greenplum" id="link-id0x22e1f068">Greenplum</a>, or <a href="http://www.vertica.com/" id="link-id0x23905c58">Vertica</a> can do map-reduce style operations under control of the SQL engine. After all, the SQL engine needs to do map-reduce and a lot more to provide good throughput for parallel, distributed SQL. Something like the <a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id0x23905c80">Berkeley Orders Of Magnitude</a> (<a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id0x234bc1f8">BOOM</a>) distributed Datalog implementation (Overlog, Deadalus, BLOOM) could be a parallel computation framework that would subsume any map-reduce-style functionality under a more elegant declarative framework while still leaving control of execution to the developer for the cases where this is needed.</p> <p>From our viewpoint, the project's gains include:</p> <ul> <li> <p>Significant narrowing of the RDB to RDF performance gap. RDF will be an option for large scale warehousing, cutting down on time to integration by providing greater schema flexibility.</p> </li> <li> <p>Ready to use toolbox for data integration, including schema alignment and resolution of coreference.</p> </li> <li> <p>Data discovery, summarization and visualization</p> </li> </ul> <p>Integrating this into a relatively unified stack of tools is possible, since these all cluster around the task of linking the universe with RDF and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x2390bdf8">linked data</a>. In this respect the integration of results may be stronger than often seen in European large scale integrating projects.</p> <p>The use cases fit the development profile well: </p> <ul> <li> <p> <a href="http://dbpedia.org/resource/Wolters_Kluwer" id="link-id0x237a3420">Wolters Kluwer</a> will develop an application for integrating resources around law, from the actual laws to court cases to media coverage. The content is modeled in a fine grained legal ontology.</p> </li> <li> <p> <a href="http://dbpedia.org/resource/Exalead" id="link-id0x235c5d70">Exalead</a> will implement the linked data enterprise, addressing enterprise search and any typical enterprise data integration plus generating added value from open sources.</p> </li> <li> <p>The Open <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x235f8138">Knowledge</a> Foundation will create a portal of all government published data for easy access by citizens.</p> </li> </ul> <p>In all these cases, the integration requirements of schema alignment, resolution of identity, <a href="http://dbpedia.org/resource/Information" id="link-id0x22fef2e8">information</a> extraction, and efficient storage and retrieval play a significant role. The end user interfaces will be task-specific but developer interfaces around integration tools and query formulation may be quite generic and suited for generic RDF application development.</p>
2010-09-21T16:22:02-04:00
Perseus, Andromeda, and RDF
http://www.openlinksw.com:443/blog/vdb/blog/?date=2010-09-13#1629
2010-09-13T22:10:12Z
<p>It has been several months since <a href="http://www.openlinksw.com/weblog/oerling/?id=1622" id="link-id0x1c86c3a0">my last</a> <a href="http://dbpedia.org/resource/Blog" id="link-id0x15fa418">blog</a> post. In this day and age of the attention economy, what gives me the insolence so to neglect my duty to mindshare?</p> <p>Well, Perseus wasn't blogging or checking his email either, when he went to fetch the Gorgon's head. As Joseph Campbell puts it, the hero breaks into a world separate from the ordinary in order to bring back a blessing which will revitalize the community.</p> <p>Thus, I deliberately withdrew from the public conversation, in faith that it would take care of itself and that I would still not be altogether forgotten. As it happens, I was confirmed in this when recently invited to submit a talk for the <a href="http://semdata.org/events/2010/vldb" id="link-id0x1c319338">Semdata workshop</a> at <a href="http://www.vldb2010.org" id="link-id0x1b334640">VLDB 2010</a>.</p> <p>Great deeds are not only personal accomplishments but also play a role in a broader context. The quest may appear remote and difficult to execute but its outcome can be quite tangible: Andromeda needed no elaborate sales pitch to convince her of the advantages of not being eaten by the sea serpent.</p> <p>Thus right after <a href="http://www.openlinksw.com/weblog/oerling/?id=1614" id="link-id0x15fd6968">the meeting in Sofia last March</a>, I followed the vertical treasure map into the realm of first principles. As Perseus received advice from Athena, so was I informed by the Platonic ideas of locality and concurrency.</p> <p>The great quests have an outer and inner aspect. Likewise here, bringing the ideas to physical reality gave me a great deal of material on cognitive function itself. For human and computer alike, it appears that the main reason why anything at all works is <a href="http://dbpedia.org/resource/Cache" id="link-id0x140f1dc0">cache</a>. Locality and parallelism again. Maybe I will say something more about memory, attention, interface, and paradigm some other time. On the other hand, such material is bound to be unpopular even if valid.</p> <p>By now, you may ask yourself what I am talking about.</p> <p>We remember that Andromeda's fix was due to her mother, Cassiopeia, having claimed greater beauty than the daughters of the sea-god Poseidon. To transpose the archetype into the present, it is like Tim B-L saying that OWLs (by the way sacred to Athena) are more semantic than Codd's brainchild. Yet the relational community sees <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1d241648">RDF</a> as something not quite serious. A matter of scale(s) — just think of the sea serpent.</p> <p>So, I am talking about what I alluded to in the <a href="http://www.openlinksw.com/weblog/oerling/?id=1603" id="link-id0x19182b88">2010 New Year's statement on this blog</a>: RDF as a viable alternative to relational for big <a href="http://dbpedia.org/resource/Data" id="link-id0x1b896350">data</a>. This means that RDF is no longer a specialty niche where, due to the hopeless task of bringing everything into a relational model, the fact of everything taking several times both the time and space is tolerated because there is no real alternative.</p> <p>The value proposition is that for any current RDF user, the present assets will go four times farther than before with the next release of <a href="http://virtuoso.openlinksw.com" id="link-id0x6a223b0">Virtuoso</a>. For a prospective RDF user, the cost of keeping an ETLed RDF integration warehouse is now in the same ballpark as the relational cost, except that <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x15f8ed8">schema</a> is now flexible, and the time to integrate and answer is accordingly shorter. For users of analytics-oriented <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x8bc44d8">RDBMS</a>, the next Virtuoso is a full cluster-capable <a href="http://dbpedia.org/resource/SQL" id="link-id0x127faf40">SQL</a> column store. Its merits compared to others in this space will be published later with benchmarks like <a href="http://www.tpc.org/" id="link-id0x6af7ae0">TPC</a>-<a href="http://dbpedia.org/resource/TPC-H" id="link-id0x1d46f230">H</a>. As an extra bonus for such users, Virtuoso brings SQL federation and a growth path to RDF, should this become interesting.</p> <p>This is accomplished by introducing a new column-wise compressed-storage engine with corresponding changes to query execution. The general principles are explained in <a href="http://cs-www.cs.yale.edu/homes/dna/papers/abadiphd.pdf" id="link-id0x1d259a88">Daniel Abadi's famous Ph.D. thesis</a>. The compression is tuned by the data itself, without user intervention. Further, our implementation remains capable of run-time-typing, thus the column-store advantages to RDF are obtained without going to a task-specific schema. But since data types, even if determined at run-time, are still in practice repetitive, the advantages of running on homogenous vectors are not lost.</p> <p>When storing an RDF extraction of TPC-H data, we get a storage usage of 6.3 bytes per quad. If you do not care about queries where the predicate is unspecified, the storage requirement drops to 4.7 bytes per quad. Whether storing the data as RDF quads or as Vertica-style multicolumn projections, the working set is about the same. Since having enough of the data in memory is the <i>sine qua non</i> prerequisite of flexible querying, the point is made. QED.</p> <p>In Virtuoso also, relational remains a bit faster but a penalty of 1.3x or so for RDF is quite tolerable, considering that <i>a priori</i> schema is no longer needed.</p> <p>This means that we are coming into an age where the warehouse becomes an <i>ad hoc</i> asset, to be filled with RDF, without the need to develop an <i>a priori</i> universal schema for all data one may ever wish to integrate, now or in the future. The data can be stored as RDF and projected from there into any form that may be needed at any time, whether the target format is more RDF or a task-specific relational schema.</p> <p>Availability is planned for late 2010, first as a Virtuoso Open Source preview.</p>
2010-09-13T17:35:04-04:00
Perseus, Andromeda, and RDF
http://www.openlinksw.com:443/weblog/oerling/?date=2010-09-13#1627
2010-09-13T22:10:12Z
<p>It has been several months since <a href="http://www.openlinksw.com/weblog/oerling/?id=1622" id="link-id0x1c86c3a0">my last</a> <a href="http://dbpedia.org/resource/Blog" id="link-id0x7816808">blog</a> post. In this day and age of the attention economy, what gives me the insolence so to neglect my duty to mindshare?</p> <p>Well, Perseus wasn't blogging or checking his email either, when he went to fetch the Gorgon's head. As Joseph Campbell puts it, the hero breaks into a world separate from the ordinary in order to bring back a blessing which will revitalize the community.</p> <p>Thus, I deliberately withdrew from the public conversation, in faith that it would take care of itself and that I would still not be altogether forgotten. As it happens, I was confirmed in this when recently invited to submit a talk for the <a href="http://semdata.org/events/2010/vldb" id="link-id0x1c319338">Semdata workshop</a> at <a href="http://www.vldb2010.org" id="link-id0x1b334640">VLDB 2010</a>.</p> <p>Great deeds are not only personal accomplishments but also play a role in a broader context. The quest may appear remote and difficult to execute but its outcome can be quite tangible: Andromeda needed no elaborate sales pitch to convince her of the advantages of not being eaten by the sea serpent.</p> <p>Thus right after <a href="http://www.openlinksw.com/weblog/oerling/?id=1614" id="link-id0x15fd6968">the meeting in Sofia last March</a>, I followed the vertical treasure map into the realm of first principles. As Perseus received advice from Athena, so was I informed by the Platonic ideas of locality and concurrency.</p> <p>The great quests have an outer and inner aspect. Likewise here, bringing the ideas to physical reality gave me a great deal of material on cognitive function itself. For human and computer alike, it appears that the main reason why anything at all works is <a href="http://dbpedia.org/resource/Cache" id="link-id0x6177050">cache</a>. Locality and parallelism again. Maybe I will say something more about memory, attention, interface, and paradigm some other time. On the other hand, such material is bound to be unpopular even if valid.</p> <p>By now, you may ask yourself what I am talking about.</p> <p>We remember that Andromeda's fix was due to her mother, Cassiopeia, having claimed greater beauty than the daughters of the sea-god Poseidon. To transpose the archetype into the present, it is like Tim B-L saying that OWLs (by the way sacred to Athena) are more semantic than Codd's brainchild. Yet the relational community sees <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x15ffc78">RDF</a> as something not quite serious. A matter of scale(s) — just think of the sea serpent.</p> <p>So, I am talking about what I alluded to in the <a href="http://www.openlinksw.com/weblog/oerling/?id=1603" id="link-id0x19182b88">2010 New Year's statement on this blog</a>: RDF as a viable alternative to relational for big <a href="http://dbpedia.org/resource/Data" id="link-id0x141372d0">data</a>. This means that RDF is no longer a specialty niche where, due to the hopeless task of bringing everything into a relational model, the fact of everything taking several times both the time and space is tolerated because there is no real alternative.</p> <p>The value proposition is that for any current RDF user, the present assets will go four times farther than before with the next release of <a href="http://virtuoso.openlinksw.com" id="link-id0x2b7a148">Virtuoso</a>. For a prospective RDF user, the cost of keeping an ETLed RDF integration warehouse is now in the same ballpark as the relational cost, except that <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x23357490">schema</a> is now flexible, and the time to integrate and answer is accordingly shorter. For users of analytics-oriented <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x12f83a0">RDBMS</a>, the next Virtuoso is a full cluster-capable <a href="http://dbpedia.org/resource/SQL" id="link-id0x6aede08">SQL</a> column store. Its merits compared to others in this space will be published later with benchmarks like <a href="http://www.tpc.org/" id="link-id0x1d1e860">TPC</a>-<a href="http://dbpedia.org/resource/TPC-H" id="link-id0x12774b8">H</a>. As an extra bonus for such users, Virtuoso brings SQL federation and a growth path to RDF, should this become interesting.</p> <p>This is accomplished by introducing a new column-wise compressed-storage engine with corresponding changes to query execution. The general principles are explained in <a href="http://cs-www.cs.yale.edu/homes/dna/papers/abadiphd.pdf" id="link-id0x1d259a88">Daniel Abadi's famous Ph.D. thesis</a>. The compression is tuned by the data itself, without user intervention. Further, our implementation remains capable of run-time-typing, thus the column-store advantages to RDF are obtained without going to a task-specific schema. But since data types, even if determined at run-time, are still in practice repetitive, the advantages of running on homogenous vectors are not lost.</p> <p>When storing an RDF extraction of TPC-H data, we get a storage usage of 6.3 bytes per quad. If you do not care about queries where the predicate is unspecified, the storage requirement drops to 4.7 bytes per quad. Whether storing the data as RDF quads or as Vertica-style multicolumn projections, the working set is about the same. Since having enough of the data in memory is the <i>sine qua non</i> prerequisite of flexible querying, the point is made. QED.</p> <p>In Virtuoso also, relational remains a bit faster but a penalty of 1.3x or so for RDF is quite tolerable, considering that <i>a priori</i> schema is no longer needed.</p> <p>This means that we are coming into an age where the warehouse becomes an <i>ad hoc</i> asset, to be filled with RDF, without the need to develop an <i>a priori</i> universal schema for all data one may ever wish to integrate, now or in the future. The data can be stored as RDF and projected from there into any form that may be needed at any time, whether the target format is more RDF or a task-specific relational schema.</p> <p>Availability is planned for late 2010, first as a Virtuoso Open Source preview.</p>
2010-09-13T17:34:58.000014-04:00
VLDB Semdata Workshop - The New Frontier of Semdata
http://www.openlinksw.com:443/blog/vdb/blog/?date=2010-09-13#1628
2010-09-13T22:09:24Z
<p>This is a revised version of the talk I will be giving at the <a href="http://semdata.org/events/2010/vldb" id="link-id0x1d137fe0">Semdata workshop</a> at <a href="http://www.vldb2010.org/" id="link-id0x2533b280">VLDB 2010</a>.</p> <p> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtDirectionsChallengesSemdata" id="link-id0x1cff6678">The paper</a> shows how we store <a href="http://www.tpc.org/" id="link-id0x244a65c0">TPC</a>-<a href="http://dbpedia.org/resource/TPC-H" id="link-id0x25136af8">H</a> <a href="http://dbpedia.org/resource/Data" id="link-id0x259a6460">data</a> as <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x268767b0">RDF</a> with relational-level efficiency and how we query both RDF and relational versions in comparable time. We also compare row-wise and column-wise storage formats as implemented in <a href="http://virtuoso.openlinksw.com" id="link-id0x2596dbc8">Virtuoso</a>.</p> <p>A question that has come up a few times during the Semdata initiative is how semantic data will avoid the fate of other would-be database revolutions like OODBMS and deductive databases.</p> <p>The need and opportunity are driven by the explosion of data in quantity and diversity of structure. The competition consists of analytics <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x2681bb10">RDBMS</a>, point solutions done with map-reduce or the like, and lastly in some cases from key-value stores with relaxed <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x2493ca50">schema</a> but limited querying.</p> <p>The benefits of RDF are the ever expanding volume of data published in it, reuse of vocabulary, and well-defined semantics. The downside is efficiency. This is not so much a matter of absolute scalability — you can run an RDF database on a cluster — but a question of relative cost as opposed to alternatives.</p> <p>The baseline is that for relational-style queries, one should get relational performance or close enough. We outline in the paper how RDF reduces to a run-time-typed relational column-store, and gets all the compression and locality advantages traditionally associated with such. After memory is no longer the differentiator, the rest is engineering. So much for the scalability barrier to adoption.</p> <p>I do not need to talk here about the benefits of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x245f72e8">linked data</a> and more or less <i>ad hoc</i> integration <i>per se</i>. But again, to make these practical, there are logistics to resolve: How to keep data up to date? How to distribute it incrementally? How to monetize freshness? We propose some solutions for these, looking at diverse-RDF replication and RDB-to-RDF replication in Virtuoso.</p> <p>But to realize the ultimate promise of RDF/Linked Data/Semdata, however we call it, we must look farther into the landscape of what is being done with big data. Here we are no longer so much running against the RDBMS, but against map-reduce and key-value stores.</p> <p>Given the psychology of geekdom, the charm of map-reduce is understandable: One controls what is going on, can work in the usual languages, can run on big iron without being picked to pieces by the endless concurrency and timing and order-of-events issues one gets when programming a cluster. Tough for the best, and unworkable for the rest.</p> <p>The key-value store has some of the same appeal, as it is the DBMS laid bare, so to say, made understandable, without the again intractably-complex questions of fancy query planning and distributed <a href="http://dbpedia.org/resource/ACID" id="link-id0x25c9a008">ACID</a> transactions. The psychological rewards of the sense of control are there, never mind the complex query; one can always hard code a point solution for the business question, if really must — maybe even in map-reduce.</p> <p>Besides, for some things that go beyond <a href="http://dbpedia.org/resource/SQL" id="link-id0x25149078">SQL</a> (for example, with graph structures), there really isn't a good solution.</p> <p>Now, enter <a href="http://www.vertica.com/" id="link-id0x268ecb90">Vertica</a>, <a href="http://dbpedia.org/resource/Greenplum" id="link-id0x25954eb8">Greenplum</a>, <a href="http://www.ingres.com/vectorwise/" id="link-id0x28cac500">VectorWise</a> (a <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x28c239f8">MonetDB</a> project derivative from <a href="http://dbpedia.org/resource/Ingres" id="link-id0x24a2f498">Ingres</a>) and Virtuoso, maybe others, who all propose some combination of SQL- and explicit map-reduce-style control structures. This is nice but better is possible.</p> <p>Here we find the next frontier of Semdata. Take <a href="http://dbpedia.org/resource/Joseph_M._Hellerstein" id="link-id0x257db7c0">Joe Hellerstein</a> et al's work on <a href="http://www.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-90.html" id="link-id0x1c64ba98">declarative logic for the data centric data center</a>.</p> <p>We have heard it many times — when the data is big, the logic must go to it. We can take declarative, location-conscious rules, <i>à la</i> <a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id0x29affc18">BOOM</a> and BLOOM, and combine these with the declarative query, well-defined semantics, parallel-database capability of the leading RDF stores. Merge this with locality compression and throughput from the best analytics DBMS.</p> <p>Here we have a data infrastructure that subsumes map-reduce as a special case of arbitrary distributed-parallel control flow, can send the processing to the data, and has flexible queries and schema-last capability.</p> <p>Further, since RDF more or less reduces to relational columns, the techniques of caching and reuse and materialized joins and demand-driven indexing, <i>à la</i> MonetDB, are applicable with minimal if any adaptation.</p> <p>Such a hybrid database-fusion frontier is relevant because it addresses heterogenous, large-scale data, with operations that are not easy to reduce to SQL, still without loss of the advantages of SQL. Apply this to anything from enhancing the business intelligence process by faster integration, including integration with <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x268168c8">linked open data</a> to the map-reduce bulk processing of today. Do it with strong semantics and inference close to the data.</p> <p>In short, RDF stays relevant by tackling real issues, with scale second to none, and decisive advantages in time-to-integrate and expressive power.</p> <p>Last week I was at the <a href="http://lod2.eu/" id="link-id0x29e23be8">LOD2</a> <a href="http://lod2.eu/BlogPost/9-press-release-lod2-project-launch.html" id="link-id0x1aec1c10">kick off</a> and a <a href="http://www.larkc.eu/" id="link-id0x245f1168">LarKC</a> meeting. The capabilities envisioned in this and the following post mirror our commitments to the EU co-funded LOD2 project. This week is VLDB and the Semdata workshop. I will talk more about how these trends are taking shape within the Virtuoso product development roadmap in future posts.</p>
2010-09-21T10:52:19.000004-04:00
VLDB Semdata Workshop - The New Frontier of Semdata
http://www.openlinksw.com:443/weblog/oerling/?date=2010-09-13#1626
2010-09-13T22:09:24Z
<p>This is a revised version of the talk I will be giving at the <a href="http://semdata.org/events/2010/vldb" id="link-id0x1d137fe0">Semdata workshop</a> at <a href="http://www.vldb2010.org/" id="link-id0x24e5c4d0">VLDB 2010</a>.</p> <p> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtDirectionsChallengesSemdata" id="link-id0x1cff6678">The paper</a> shows how we store <a href="http://www.tpc.org/" id="link-id0x25037270">TPC</a>-<a href="http://dbpedia.org/resource/TPC-H" id="link-id0x28c239f8">H</a> <a href="http://dbpedia.org/resource/Data" id="link-id0x242a5378">data</a> as <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x25900d40">RDF</a> with relational-level efficiency and how we query both RDF and relational versions in comparable time. We also compare row-wise and column-wise storage formats as implemented in <a href="http://virtuoso.openlinksw.com" id="link-id0x25904ab8">Virtuoso</a>.</p> <p>A question that has come up a few times during the Semdata initiative is how semantic data will avoid the fate of other would-be database revolutions like OODBMS and deductive databases.</p> <p>The need and opportunity are driven by the explosion of data in quantity and diversity of structure. The competition consists of analytics <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x24c7c6b8">RDBMS</a>, point solutions done with map-reduce or the like, and lastly in some cases from key-value stores with relaxed <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x23ce37d8">schema</a> but limited querying.</p> <p>The benefits of RDF are the ever expanding volume of data published in it, reuse of vocabulary, and well-defined semantics. The downside is efficiency. This is not so much a matter of absolute scalability — you can run an RDF database on a cluster — but a question of relative cost as opposed to alternatives.</p> <p>The baseline is that for relational-style queries, one should get relational performance or close enough. We outline in the paper how RDF reduces to a run-time-typed relational column-store, and gets all the compression and locality advantages traditionally associated with such. After memory is no longer the differentiator, the rest is engineering. So much for the scalability barrier to adoption.</p> <p>I do not need to talk here about the benefits of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x25345270">linked data</a> and more or less <i>ad hoc</i> integration <i>per se</i>. But again, to make these practical, there are logistics to resolve: How to keep data up to date? How to distribute it incrementally? How to monetize freshness? We propose some solutions for these, looking at diverse-RDF replication and RDB-to-RDF replication in Virtuoso.</p> <p>But to realize the ultimate promise of RDF/Linked Data/Semdata, however we call it, we must look farther into the landscape of what is being done with big data. Here we are no longer so much running against the RDBMS, but against map-reduce and key-value stores.</p> <p>Given the psychology of geekdom, the charm of map-reduce is understandable: One controls what is going on, can work in the usual languages, can run on big iron without being picked to pieces by the endless concurrency and timing and order-of-events issues one gets when programming a cluster. Tough for the best, and unworkable for the rest.</p> <p>The key-value store has some of the same appeal, as it is the DBMS laid bare, so to say, made understandable, without the again intractably-complex questions of fancy query planning and distributed <a href="http://dbpedia.org/resource/ACID" id="link-id0x25365e30">ACID</a> transactions. The psychological rewards of the sense of control are there, never mind the complex query; one can always hard code a point solution for the business question, if really must — maybe even in map-reduce.</p> <p>Besides, for some things that go beyond <a href="http://dbpedia.org/resource/SQL" id="link-id0x2533b280">SQL</a> (for example, with graph structures), there really isn't a good solution.</p> <p>Now, enter <a href="http://www.vertica.com/" id="link-id0x253fe1d8">Vertica</a>, <a href="http://dbpedia.org/resource/Greenplum" id="link-id0x257102a0">Greenplum</a>, <a href="http://www.ingres.com/vectorwise/" id="link-id0x248ce120">VectorWise</a> (a <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x2596a8a8">MonetDB</a> project derivative from <a href="http://dbpedia.org/resource/Ingres" id="link-id0x2502a170">Ingres</a>) and Virtuoso, maybe others, who all propose some combination of SQL- and explicit map-reduce-style control structures. This is nice but better is possible.</p> <p>Here we find the next frontier of Semdata. Take <a href="http://dbpedia.org/resource/Joseph_M._Hellerstein" id="link-id0x24d3dca8">Joe Hellerstein</a> et al's work on <a href="http://www.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-90.html" id="link-id0x1c64ba98">declarative logic for the data centric data center</a>.</p> <p>We have heard it many times — when the data is big, the logic must go to it. We can take declarative, location-conscious rules, <i>à la</i> <a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id0x257d2988">BOOM</a> and BLOOM, and combine these with the declarative query, well-defined semantics, parallel-database capability of the leading RDF stores. Merge this with locality compression and throughput from the best analytics DBMS.</p> <p>Here we have a data infrastructure that subsumes map-reduce as a special case of arbitrary distributed-parallel control flow, can send the processing to the data, and has flexible queries and schema-last capability.</p> <p>Further, since RDF more or less reduces to relational columns, the techniques of caching and reuse and materialized joins and demand-driven indexing, <i>à la</i> MonetDB, are applicable with minimal if any adaptation.</p> <p>Such a hybrid database-fusion frontier is relevant because it addresses heterogenous, large-scale data, with operations that are not easy to reduce to SQL, still without loss of the advantages of SQL. Apply this to anything from enhancing the business intelligence process by faster integration, including integration with <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x25455a40">linked open data</a> to the map-reduce bulk processing of today. Do it with strong semantics and inference close to the data.</p> <p>In short, RDF stays relevant by tackling real issues, with scale second to none, and decisive advantages in time-to-integrate and expressive power.</p> <p>Last week I was at the <a href="http://lod2.eu/" id="link-id0x2438c3e0">LOD2</a> <a href="http://lod2.eu/BlogPost/9-press-release-lod2-project-launch.html" id="link-id0x1aec1c10">kick off</a> and a <a href="http://www.larkc.eu/" id="link-id0x2836b780">LarKC</a> meeting. The capabilities envisioned in this and the following post mirror our commitments to the EU co-funded LOD2 project. This week is VLDB and the Semdata workshop. I will talk more about how these trends are taking shape within the Virtuoso product development roadmap in future posts.</p>
2010-09-21T10:52:12.000038-04:00
Solving Real Problems by Leveraging Linked Data: Unambiguous & Verifiable Identity for HTTP Networks
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-07-11#1625
2010-07-12T03:25:03Z
<h3>Problem: Unambiguous Verifiable Network Identity.</h3> <p>How Does <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1b1ad1d0">Linked Data</a> Address This Problem? It provides critical infrastructure for the WebID Protocol that enables an innovative tweak of SSL/TLS. </p> <p>What about OpenID? The WebID Protocol embraces and extends OpenID (<strong><em>in an open and positive way</em></strong>) via the WebID + OpenID Hybrid variant of the protocol -- basic effect is that OpenID calls are re-routed to the WebID aspect which simply removes Username and Password Authentication from the authentication challenge interaction pattern.</p> <h3>WebID Components</h3> <ol> <li> X.509 Certificate and Private Key Generator </li> <li> Structured Profile Document (e.g. a <a href="http://dbpedia.org/resource/Friend_of_a_friend" id="link-id0x1a301338">FOAF</a> based Profile) published to an HTTP Network (e.g. <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id0x1ba50e10">World Wide Web</a>) and accessible at an Address (<a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id0x12ded2e0">URL</a>) </li> <li> An Agent Identifier aka. WebID (an HTTP Name Reference re. <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x1b197378">URI</a> variant) that's the Subject of a Structured Profile Document (actually a Descriptor Resource)</li> <li> Mechanism for persisting Public Key data from X.509 Certificate to Structured Profile Document and associating it with Subject WebID (e.g. SPARUL or other HTTP based methods) </li> <li> Mechanism for de-referencing Public Key data associated with a WebID (from its Structured Profile Document) for comparison against Public Key data following successful standard SSL/TLS protocol handshake (e.g. via <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x16d26ec8">SPARQL</a> Query). </li> </ol> <h3>Demo</h3> <ul> <li> <a href="http://www.youtube.com/watch?v=mjgXsjd8PDE" id="link-id0x1b9cc4d8">WebID + OpenID Hybrid Protocol Demo using ODS, Stackoverflow.com, and identi.ca.</a> - YouTube Screencast Demo Part 1 using Firefox</li> <li> <a href="http://www.youtube.com/watch?v=eXoxUo7Py4M " id="link-id0x1a2db140">WebID + OpenID Hybrid Protocol Demo using ODS, Stackoverflow.com, and identi.ca.</a> - YouTube Screencast Demo Part 2 using Safari </li> </ul> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=webid&type=text&output=html" id="link-id0x1bc37a58">Prior Posts about WebIDs</a> </li> <li> <a href="http://payswarm.com/webid/drafts/ED-webid-20100711/" id="link-id0x1a0eecb8">Draft WebID Spec</a> </li> </ul>
2010-07-12T07:22:02.000018-04:00
Data 3.0 (a Manifesto for Platform Agnostic Structured Data) Update 5
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-04-16#1624
2010-04-16T21:09:05Z
<p>After a long period of trying to demystify and unravel the wonders of standards compliant structured <a href="http://dbpedia.org/resource/Data">data</a> access, combined with protocols (e.g., HTTP) that separate: </p> <ol> <li>Identity,</li> <li>Access,</li> <li>Storage,</li> <li>Representation, and</li> <li>Presentation.</li> </ol> <p>I ended up with what I can best describe as the Data 3.0 Manifesto. A manifesto for standards complaint access to structured data object (or <a href="http://dbpedia.org/resource/Entity" id="link-id0x1a0bc238">entity</a>) descriptors.</p> <h3>Some Related Work</h3> <p> <a href="http://blogs.msdn.com/alexj/" id="link-id0x1a3c5b70">Alex James</a> (Program Manager <a href="http://blogs.msdn.com/efdesign/" id="link-id0x1a3c5bd8">Entity Frameworks</a> at Microsoft), put together something quite similar to this via his Base4 <a href="http://dbpedia.org/resource/Blog" id="link-id0x13c374c8">blog</a> (around the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> 2.0 bootstrap time), sadly -- quoting Alex -- that post has gone where discontinued blogs and their host platforms go (deep deep irony here). </p> <p>It's also important to note that this manifesto is also a variant of the <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id0x1a29f338">TimBL</a>'s <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id0x1a4e8580">Linked Data Design Issues</a> <a href="http://dbpedia.org/resource/Meme" id="link-id0x199efc30">meme</a> re. Linked Data, but totally decoupled from RDF (data representation formats aspect) and <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x199efc58">SPARQL</a> which -- in my world view -- remain implementation details.</p> <h3>Data 3.0 manifesto</h3> <ul> <li>An "Entity" is the "Referent" of an "Identifier."</li> <li>An "Identifier" SHOULD provide a global, unambiguous, and unchanging (though it MAY be opaque!) "Name" for its "Referent".</li> <li>A "Referent" MAY have many "Identifiers" (Names), but each "Identifier" MUST have only one "Referent".</li> <li>Structured Entity Descriptions SHOULD be based on the <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id0x1a2a15c0">Entity-Attribute-Value (EAV) Data Model</a>, and SHOULD therefore take the form of one or more 3-tuples (triples), each comprised of: <ul> <li>an "Identifier" that names an "Entity" (i.e., Entity Name),</li> <li>an "Identifier" that names an "Attribute" (i.e., Attribute Name), and</li> <li>an "Attribute Value", which may be an "Identifier" or a "Literal".</li> </ul> </li> <li>Structured Descriptions SHOULD be CARRIED by "Descriptor Documents" (i.e., purpose specific documents where Entity Identifiers, Attribute Identifiers, and Attribute Values are clearly discernible by the document's intended consumers, e.g., humans or machines).</li> <li>Structured Descriptor Documents can contain (carry) several Structured Entity Descriptions</li> <li>Stuctured Descriptor Documents SHOULD be network accessible via network addresses (e.g., HTTP URLs when dealing with HTTP-based Networks).</li> <li>An Identifier SHOULD resolve (de-reference) to a Structured Representation of the Referent's Structured Description.</li> </ul> <h3>Related</h3> <ul> <li> <a href="http://twitpic.com/1g02q8/full" id="link-id0x1a3d1428">Referent, Identifier, and Descriptor/Sense (The Data Perception Trinity)</a> illustration</li> <li> <a href="http://twitpic.com/1g03vo/full" id="link-id0x1a353a20">Referent, Identifier, and Descriptor/Sense Trinity</a> (as exploited in <a href="http://esw.w3.org/Foaf%2Bssl" id="link-id0x135ed828">FOAF+SSL</a> based Secure WebIDs) illustration</li> <li> <a href="http://www.slideshare.net/kidehen/understanding-linked-data-via-eav-model-based-structured-descriptions" id="link-id0x1961ae30">Demystifying Linked Data via EAV Model based Structured Descriptions</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1388" id="link-id0x1a28db38">What do people have against URIs and URLs?</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1567" id="link-id0x1a4cedc8">The URI, URL, and Linked Data Meme's Generic HTTP URI</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1543" id="link-id0x19ac04c8">Simple Explanation of RDF and Linked Data Dynamics</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1547" id="link-id0x13c24748">Linked Data and Identity</a> </li> <li> <a href="http://esw.w3.org/Foaf%2Bssl/FAQ" id="link-id0x199ef720">FOAF+SSL FAQ</a> </li> <li> <a href="http://lists.w3.org/Archives/Public/public-lod/2010Apr/0278.html" id="link-id0x1a361640">LOD Community Thread</a> (showing evolution of this manifesto based on feedback from members such as <a href="http://richard.cyganiak.de/foaf.rdf#cygri" id="link-id0x1a361668">Richard Cyganiak</a>).</li> <li> <a href="http://code.google.com/apis/base/starting-out.html#terms" id="link-id0x18e0b578">Googlebase Data API Docs</a> </li> <li> <a href="http://code.google.com/apis/gdata/docs/2.0/basics.html" id="link-id0x199c77b0">Google Data Protocol</a> (GData)</li> <li> <a href="http://odata.org" id="link-id0x19d1e578">Microsoft's OData Protocol</a> </li> <li> <a href="http://www.youtube.com/watch?v=6pmWojisM_E" id="link-id0x1a40a998">Magic of De-referencable Names and actual Data via Binky Video</a> </li> <li> <a href="http://www.slideshare.net/jyri/building-sites-around-social-objects-web-20-expo-sf-2009" id="link-id0x19ad7e70">Social Objects Presentation</a> (aka. Social Linked Data Objects) - by <a href="http://www.slideshare.net/jyri" id="link-id0x19e71700">Jyri Engeström</a> </li> <li> <a href="http://en.wikipedia.org/wiki/Reference_%28computer_science%29" id="link-id0x199c6178">What's a Reference?</a> </li> </ul>
2010-05-25T17:10:28.000001-04:00
Transactional High Availability in Virtuoso Cluster Edition
http://www.openlinksw.com:443/blog/vdb/blog/?date=2010-04-14#1623
2010-04-14T22:21:52Z
<h2>Introduction</h2> <p>This post discusses the technical specifics of how we accomplish smooth transactional operation in a database server cluster under different failure conditions. (<a href="http://www.openlinksw.com/weblog/oerling/?id=1621" id="link-id0x198e8e68">A higher-level short version</a> was posted last week.) The reader is expected to be familiar with the basics of <a href="http://dbpedia.org/resource/Distributed_transaction" id="link-id0x25088028">distributed transactions</a>.</p> <p>Someone on a cloud computing discussion list called <a href="http://dbpedia.org/resource/Two-phase_commit_protocol" id="link-id0x21addd50">two-phase commit</a> (<a href="http://dbpedia.org/resource/Two-phase_commit_protocol" id="link-id0x1eb6bc90">2PC</a>) the "anti-availability protocol." There is indeed a certain anti-<a href="http://dbpedia.org/resource/SQL" id="link-id0x28e1dbd0">SQL</a> and anti-2PC sentiment out there, with key-value stores and "eventual consistency" being talked about a lot. Indeed, if we are talking about wide-area replication over high-latency connections, then 2PC with synchronously-sharp transaction boundaries over all copies is not really workable.</p> <p>For multi-site operations, a level of <i>eventual</i> consistency is indeed quite unavoidable. Exactly what the requirements are depends on the application, so I will focus here on operations inside one site.</p> <p>The key-value store culture seems to focus on workloads where a record is relatively self-contained. The record can be quite long, with repeating fields, different selections of fields in consecutive records, and so forth. Such a record would typically be split over many tables of a relational <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x216479f8">schema</a>. In the <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x672d740">RDF</a> world, such a record would be split even wider, with the <a href="http://dbpedia.org/resource/Information" id="link-id0x72c8ec0">information</a> needed to reconstitute the full record almost invariably split over many servers. This comes from the mapping between the text of URIs and their internal IDs being partitioned in one way, and the many indices on the RDF quads each in yet another way.</p> <p>So it comes to pass that in the <a href="http://dbpedia.org/resource/Data" id="link-id0x216c6280">data</a> models we are most interested in, the application-level <a href="http://dbpedia.org/resource/Entity" id="link-id0x224444c0">entity</a> (<i>e.g.,</i> a user account in a social network) is not a contiguous unit with a single global identifier. The social network user account, that the key-value store would consider a unit of replication mastering and eventual consistency, will be in RDF or SQL a set of maybe hundreds of tuples, each with more than one index, nearly invariably spanning multiple nodes of the database cluster.</p> <p>So, before we can talk about wide-area replication and eventual consistency with application-level semantics, we need a database that can run on a fair-sized cluster and have cast-iron consistency within its bounds. If such a cluster is to be large and is to operate continuously, it must have some form of redundancy to cover for hardware failures, software upgrades, reboots, etc., without interruption of service.</p> <p>This is the point of the design space we are tackling here.</p> <h2>Non Fault-Tolerant Operation</h2> <p>There are two basic modes of operation we cover: bulk load, and online transactions.</p> <p>In the case of bulk load, we start with a consistent image of the database; load data; and finish by making another consistent image. If there is a failure during load, we lose the whole load, and restart from the initial consistent image. This is quite simple and is not properly transactional. It is quicker for filling a warehouse but is not to be used for anything else. In the remainder, we will only talk about online transactions.</p> <p>When all cluster nodes are online, operation is relatively simple. Each entry of each index belongs to a partition that is determined by the values of one or more partitioning columns of said index. There are no tables separate from indices; the relational row is on the index leaf of its primary key. Secondary indices reference the row by including the primary key. Blobs are in the same partition as the row which contains the blob. Each partition is then stored on a "cluster node." In non fault-tolerant operations, each such cluster node is a single process with exclusive access to its own permanent storage, consisting of database files and logs; <i>i.e.,</i> each node is a single server instance. It does not matter if the storage is local or on a SAN, the cluster node is still the only one accessing it.</p> <p>When things are not fault tolerant, transactions work as follows:</p> <p>When there are updates, two-phase commit is used to guarantee a consistent result. Each transaction is coordinated by one cluster node, which issues the updates in parallel to all cluster nodes concerned. Sending two update messages instead of one does not significantly impact latency. The coordinator of each transaction is the primary authority for the transaction's outcome. If the coordinator of the transaction dies between the phases of the commit, the transaction branches stay in the prepared state until the coordinator is recovered and can be asked again about the outcome of the transaction. Likewise, if a non-coordinating cluster node with a transaction branch dies between the phases, it will do a roll-forward and ask the coordinator for the outcome of the transaction.</p> <p>If cluster nodes occasionally crash and then recover relatively quickly, without ever losing transaction logs or database files, this is resilient enough. Everything is symmetrical; there are no cluster nodes with special functions, except for one master node that has the added task of resolving distributed deadlocks.</p> <p>I suppose our anti-SQL person called 2PC "anti-availability" because in the above situation we have the following problems: if any one cluster node is offline, it is quite likely that no transaction can be committed. This is so unless the data is partitioned on a key with application semantics, and all data touched by a transaction usually stays within a single partition. Then operations could proceed on most of the data while one cluster node was recovering. But, especially with RDF, this is never the case, since keys are partitioned in ways that have nothing to do with application semantics. Further, if one uses XA or <a href="http://dbpedia.org/resource/Microsoft" id="link-id0x785bc50">Microsoft</a> DTC with the monitor on a single box, this box can become a bottleneck and/or a single point of failure. (Among other considerations, this is why <a href="http://virtuoso.openlinksw.com" id="link-id0x72a1ea8">Virtuoso</a> does not rely on any such monitor.) Further, if a cluster node dies never to be heard of again, leaving prepared but uncommitted transaction branches, the rest of the system has no way of telling what to do with them, again unless relying on a monitor that is itself liable to fail.</p> <p>If transactions have a real world counterpart, it is possible, at least in theory, to check the outcome against the real world state: One can ask a customer if an order was actually placed or a shipment delivered. But when a transaction has to do with internal identifiers of things, for example whether <b><code>mailto://plaidskirt@hotdate.com</code></b> has internal ID <b><code>0xacebabe</code></b>, such a check against external reality is not possible.</p> <h2>Fault-Tolerant Operation</h2> <p>In a fault tolerant setting, we introduce the following extra elements: Cluster nodes are comprised of "quorums" of mutually-mirroring server instances. Each such quorum holds a partition of the data. Such a quorum typically consists of two server instances, but may have three for extra safety. If all server instances in the quorum are offline, then the cluster node is offline, and the cluster is not fully operational. If at least one server instance in a quorum is online, then the cluster node is online, and the cluster is operational and can process new transactions.</p> <p>We designate one cluster node (<i>i.e.,</i> one quorum of 2 or 3 server instances) to act as a master node, and we set an order of precedence among its member instances. In addition to arbitrating distributed deadlocks, the master instance on duty will handle reports of server instance failures, and answer questions about any transactions left hanging in prepared state by a dead transaction coordinator. If the master on duty fails, the next master in line will either notice this itself in the line of normal business or get a complaint from another server instance about not being able to contact the previous master.</p> <p>There is no global heartbeat messaging <i>per se,</i> but since connections between server instances are reused long-term, a dropped connection will be noticed and the master on duty will be notified. If all masters are unavailable, that entire quorum (<i>i.e.,</i> the master node) is offline and thus (as with any entire node going offline) most operations will fail anyway, unless by chance they do not hit any data managed by that failed quorum.</p> <p>When it receives a notice of unavailability, the master instance on duty tries to contact the unavailable server instance and if it fails, it will notify all remaining instances that that server instance is removed from the cluster. The effect is that the remaining server instances will stop attempting to access the failed instance. Updates to the partitions managed by the failed server instance are no longer sent to it, which results in updates to this data succeeding, as they are made against the other server instances in that quorum. Updates to the data of the failed server instance <i>will</i> fail in the window of time between the actual failure and the removal, which is typically well under a second. The removal of a failed server instance is delegated to a central authority in order not to have everybody get in each other's way when trying to effect the removal.</p> <p>If the failed server instance left prepared uncommitted transactions behind, the server instances having such branches will in due order contact the transaction coordinator to ask what should be done. This is a normal procedure for dealing with possibly dropped commit or rollback messages. When they discover that the coordinator has been removed, the master on duty will be contacted instead. Each prepare message of a transaction lists all the server instances participating in the transaction; thus the master can check whether each has received the prepare. If all have the prepare and none has an abort, the transaction is committed. The dead coordinator may not know this or may indeed not have the transaction logged, since it sends the prepares before logging its own prepare. The recovery will handle this though. We note that of the remaining branches, there is at least one copy of the branch with the failed server instance, or else we would have a whole quorum failed. In cases where there are branches participating in an unresolved transaction where all the quorum members have failed, the system cannot decide the outcome, and will periodically retry until at least one member of the failed quorum becomes available.</p> <p>The most complex part of the protocol is the recovery of a failed server instance. The recovery starts with a normal roll forward from the local transaction log. After this, the server instance will contact the master on duty to ask for its state. Typically, the master will reply that the recovering server instance had been removed and is out of date. When this is established, the recovering server instance will contact a live member of its quorum and ask for sync. The failed server instance has an approximate timestamp of its last received transaction. It knows this from the roll forward, where time markers are interspersed now and then between transaction records. The live partner then sends its transaction log(s) covering the time from a few seconds before the last transaction of the failed partner up to the present. A few transactions may get rolled forward twice but this does no harm, since these records have absolute values and no deltas and the second insert of a key is simply ignored. When the sender of the log reaches its last committed log entry, it asks the recovering server instance to confirm successful replay of the log so far. Having the confirmation, the sender will abort all unprepared transactions affecting it and will not accept any new ones until the sync is completed. If new transactions were committed between sending the last of the log and killing the uncommitted new transactions, these too are shipped to the recovering server instance in their committed or prepared state. When these are also confirmed replayed, the recovering server instance is in exact sync up to the transaction. The sender then notifies the rest of the cluster that the sync is complete and that the recovered server instance will be included in any updates of its slice of the data. The time between freeze and re-enable of transactions is the time to replay what came in between the first sync and finishing the freeze. Typically nothing came in, so the time is in milliseconds. If an application got its transaction killed in this maneuver, it will be seen as a deadlock.</p> <p>If the recovering server instance received transactions in prepared state, it will ask about their outcome as a part of the periodic sweep through pending transactions. One of these transactions could have been one originally prepared by itself, where the prepares had gone out before it had time to log the transaction. Thus, this eventuality too is covered and has a consistent outcome. Failures can interrupt the recovery process. The recovering server instance will have logged as far as it got, and will pick up from this point onward. Real time clocks on the host nodes of the cluster will have to be in approximate sync, within a margin of a minute or so. This is not a problem in a closely connected network.</p> <p>For simultaneous failure of a entire quorum of server instances (<i>i.e.,</i> a set of mutually-mirroring partners; a cluster node), the rule is that the last one to fail must be the first to come back up. In order to have uninterrupted service across arbitrary double failures, one must store things in triplicate; statistically, however, most double failures will not hit cluster nodes of the same group.</p> <p>The protocol for recovery of failed server instances of the master quorum (<i>i.e.,</i> the master cluster node) is identical, except that a recovering master will have to ask the other master(s) which one is more up to date. If the recovering master has a log entry of having excluded all other masters in its quorum from the cluster, it can come back online without asking anybody. If there is no such entry, it must ask the other master(s). If all had failed at the exact same instant, none has an entry of the other(s) being excluded and all will know that they are in the same state since any update to one would also have been sent to the other(s).</p> <h2>Failure of Storage Media</h2> <p>When a server instance fails, its permanent storage may or may not survive. Especially with mirrored disks, storage most often survives a failure. However, the survival of the database does not depend on any single server instance retaining any permanent storage over failure. If storage is left in place, as in the case of an OS reboot or replacing a faulty memory chip, rejoining the cluster is done based on the existing copy of the database on the server instance. if there is no existing copy, a copy can be taken from any surviving member of the same quorum. This consists of the following steps: First, a log checkpoint is forced on the surviving instance. Normally log checkpoints are done at regular intervals, independently on each server instance. The log checkpoint writes a consistent state of the database to permanent storage. The disk pages forming this consistent image will not be written to until the next log checkpoint. Therefore copying the database file is safe and consistent as long as a log checkpoint does not take place between the start and end of copy. Thus checkpoints are disabled right after the initial checkpoint. The copy can take a relatively long time; consider 20s per gigabyte on a 1GbE network a good day. At the end of copy, checkpoints are re-enabled on the surviving cluster node. The recovering database starts without a log, sees the timestamp of the checkpoint in the database, and asks for transactions from just before this time up to present. The recovery then proceeds as outlined above.</p> <h2>Network Failures</h2> <p>The CAP theorem states that Consistency, Availability, and Partition-tolerance do not mix. "Partition" here means the split of a network.</p> <p>It is trivially true that if the network splits so that on both sides there is a copy of each partition of the data, both sides will think themselves the live copy left online after the other died, and each will thus continue to accumulate updates. Such an event is not very probable within one site where all machines are redundantly connected to two independent switches. Most servers have dual 1GbE on the motherboard, and both ports should be used for cluster interconnect for best performance, with each attached to an independent switch. Both switches would have to fail in such a way as to split their respective network for a single-site network split to happen. Of course, the likelihood of a network split in multi-site situations is higher.</p> <p>One way of guarding against network splits is to require that at least one partition of the data have all copies online. Additionally, the master on duty can request each cluster node or server instance it expects to be online to connect to every other node or instance, and to report which they could reach. If the reports differ, there is a network problem. This procedure can be performed using both interfaces or only the first or second interface of each server to determine if one of the switches selectively blocks some paths. These simple sanity checks protect against arbitrary network errors. Using TCP for inter-cluster-node communication in principle protects against random message loss, but the Virtuoso cluster protocols do not rely on this. Instead, there are protocols for retry of any transaction messages and for using keep-alive messages on any long-running functions sent across the cluster. Failure to get a keep-alive message within a certain period will abort a query even if the network connections look OK. </p> <h2>Backups, and Recovery from Loss of Entire Site</h2> <p>For a constantly-operating distributed system, it is hard to define what exactly constitutes a consistent snapshot. The checkpointed state on each cluster node is consistent as far as this cluster node is concerned (<i>i.e.,</i> it contains no uncommitted data), but the checkpointed states on all the cluster nodes are not from exactly the same moment in time. The complete state of a cluster is the checkpoint state of each cluster node plus the current transaction log of each. If the logs were shipped in real time to off-site storage, a consistent image could be reconstructed from them. Since such shipping cannot be synchronous due to latency considerations, some transactions could be received only in part in the event of a failure of the off-site link. Such partial transactions can however be detected at reconstruction time because each record contains the list of all participants of the transaction. If some piece is found missing, the whole can be discarded. In this way integrity is guaranteed but it is possible that a few milliseconds worth of transactions get lost. In these cases, the online client will almost certainly fail to get the final success message and will recheck the status after recovery.</p> <p>For business continuity purposes, a live feed of transactions can be constantly streamed off-site, for example to a cloud infrastructure provider. One low-cost virtual machine on the cloud will typically be enough for receiving the feed. In the event of long-term loss of the whole site, replacement servers can be procured on the cloud; thus, capital is not tied up in an aging inventory of spare servers. The cloud-based substitute can be maintained for the time it takes to rebuild an owned infrastructure, which is still at present more economical than a cloud-only solution.</p> <p>Switching a cluster from an owned site to the cloud could be accomplished in a few hours. The prerequisite of this is that there are reasonably recent snapshots of the database files, so that replay of logs does not take too long. The bulk of the time taken by such a switch would be in transferring the database snapshots from S3 or similar to the newly provisioned machines, formatting the newly provisioned virtual disks, etc.</p> <p>Rehearsing such a maneuver beforehand is quite necessary for predictable execution. We do not presently have a productized set of tools for such a switch, but can advise any interested parties on implementing and testing such a disaster recovery scheme.</p> <h2>Conclusions</h2> <p>In conclusion, we have shown how we can have strong transactional guarantees in a database cluster without single points of failure or performance penalties when compared with a non fault-tolerant cluster. Operator intervention is not required for anything short of hardware failure. Recovery procedures are simple, at most consisting of installing software and copying database files from a surviving cluster node. Unless permanent storage is lost in the failure, not even this is required. Real-time off-site log shipment can easily be added to these procedures to protect against site-wide failures.</p> <p>Future work may be directed toward concurrent operation of geographically-distributed data centers with eventual consistency. Such a setting would allow for migration between sites in the event of whole-site failures, and for reconciliation between inconsistent histories of different halves of a temporarily split network. Such schemes are likely to require application-level logic for reconciliation and cannot consist of an out-of-the-box DBMS alone. All techniques discussed here are application-agnostic and will work equally well for Graph Model (<i>e.g.,</i> RDF) and Relational Model (<i>e.g.,</i> SQL) workloads.</p> <h3> <a href="http://dbpedia.org/resource/Glossary" id="link-id0x24f4e378">Glossary</a> </h3> <ul> <li> <b>Virtuoso Cluster (VC)</b> -- a collection of Virtuoso Cluster Nodes on one or more machines, working in parallel as part of a Virtuoso Cluster.</li> <li> <b>Virtuoso Cluster Node (VCN)</b> -- a Virtuoso Server Instance (Non Fault-Tolerant Operations), or a Quorum of Server Instances (Fault Tolerant Operations), which is a member of a collection of Virtuoso Cluster Nodes working in parallel as part of a Virtuoso Cluster.</li> <li> <b>Virtuoso Host Cluster (VHC)</b> -- a collection of machines, each hosting one or more Virtuoso Server Instances, making up a Virtuoso Cluster.</li> <li> <b>Virtuoso Host Cluster Node (VHCN)</b> -- a machine hosting one or more Virtuoso Server Instances that are members of a Virtuoso Cluster.</li> <li> <b>Virtuoso Server Instance (VSI)</b> -- a single Virtuoso process with exclusive access to its own permanent storage, consisting of database files and logs. May comprise an entire Virtuoso Cluster Node (Non Fault-Tolerant Operations), or be one member of a quorum which comprises a Virtuoso Cluster Node (Fault Tolerant Operations).</li> </ul> <h3>Also see</h3> <ul> <li> <a href="http://www.gbcacm.org/sites/www.gbcacm.org/files/slides/SpecialRelativity[1]_0.pdf" id="link-id0x16cb22d8">Special Relativity and the Problem of Database Scalability (PDF)</a>, by James Starkey of <a href="http://www.nimbusdb.com/" id="link-id0x18f30d58">NimbusDB, Inc.</a> </li> </ul>
2010-04-14T19:13:00-04:00
Transactional High Availability in Virtuoso Cluster Edition
http://www.openlinksw.com:443/weblog/oerling/?date=2010-04-14#1622
2010-04-14T22:21:52Z
<h2>Introduction</h2> <p>This post discusses the technical specifics of how we accomplish smooth transactional operation in a database server cluster under different failure conditions. (<a href="http://www.openlinksw.com/weblog/oerling/?id=1621" id="link-id0x198e8e68">A higher-level short version</a> was posted last week.) The reader is expected to be familiar with the basics of <a href="http://dbpedia.org/resource/Distributed_transaction" id="link-id0x6674ec8">distributed transactions</a>.</p> <p>Someone on a cloud computing discussion list called <a href="http://dbpedia.org/resource/Two-phase_commit_protocol" id="link-id0x6ab2480">two-phase commit</a> (<a href="http://dbpedia.org/resource/Two-phase_commit_protocol" id="link-id0x66e9568">2PC</a>) the "anti-availability protocol." There is indeed a certain anti-<a href="http://dbpedia.org/resource/SQL" id="link-id0x2238b330">SQL</a> and anti-2PC sentiment out there, with key-value stores and "eventual consistency" being talked about a lot. Indeed, if we are talking about wide-area replication over high-latency connections, then 2PC with synchronously-sharp transaction boundaries over all copies is not really workable.</p> <p>For multi-site operations, a level of <i>eventual</i> consistency is indeed quite unavoidable. Exactly what the requirements are depends on the application, so I will focus here on operations inside one site.</p> <p>The key-value store culture seems to focus on workloads where a record is relatively self-contained. The record can be quite long, with repeating fields, different selections of fields in consecutive records, and so forth. Such a record would typically be split over many tables of a relational <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x1ae05058">schema</a>. In the <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x6956f38">RDF</a> world, such a record would be split even wider, with the <a href="http://dbpedia.org/resource/Information" id="link-id0x21ae7cc8">information</a> needed to reconstitute the full record almost invariably split over many servers. This comes from the mapping between the text of URIs and their internal IDs being partitioned in one way, and the many indices on the RDF quads each in yet another way.</p> <p>So it comes to pass that in the <a href="http://dbpedia.org/resource/Data" id="link-id0x2bd28d30">data</a> models we are most interested in, the application-level <a href="http://dbpedia.org/resource/Entity" id="link-id0x214d55c0">entity</a> (<i>e.g.,</i> a user account in a social network) is not a contiguous unit with a single global identifier. The social network user account, that the key-value store would consider a unit of replication mastering and eventual consistency, will be in RDF or SQL a set of maybe hundreds of tuples, each with more than one index, nearly invariably spanning multiple nodes of the database cluster.</p> <p>So, before we can talk about wide-area replication and eventual consistency with application-level semantics, we need a database that can run on a fair-sized cluster and have cast-iron consistency within its bounds. If such a cluster is to be large and is to operate continuously, it must have some form of redundancy to cover for hardware failures, software upgrades, reboots, etc., without interruption of service.</p> <p>This is the point of the design space we are tackling here.</p> <h2>Non Fault-Tolerant Operation</h2> <p>There are two basic modes of operation we cover: bulk load, and online transactions.</p> <p>In the case of bulk load, we start with a consistent image of the database; load data; and finish by making another consistent image. If there is a failure during load, we lose the whole load, and restart from the initial consistent image. This is quite simple and is not properly transactional. It is quicker for filling a warehouse but is not to be used for anything else. In the remainder, we will only talk about online transactions.</p> <p>When all cluster nodes are online, operation is relatively simple. Each entry of each index belongs to a partition that is determined by the values of one or more partitioning columns of said index. There are no tables separate from indices; the relational row is on the index leaf of its primary key. Secondary indices reference the row by including the primary key. Blobs are in the same partition as the row which contains the blob. Each partition is then stored on a "cluster node." In non fault-tolerant operations, each such cluster node is a single process with exclusive access to its own permanent storage, consisting of database files and logs; <i>i.e.,</i> each node is a single server instance. It does not matter if the storage is local or on a SAN, the cluster node is still the only one accessing it.</p> <p>When things are not fault tolerant, transactions work as follows:</p> <p>When there are updates, two-phase commit is used to guarantee a consistent result. Each transaction is coordinated by one cluster node, which issues the updates in parallel to all cluster nodes concerned. Sending two update messages instead of one does not significantly impact latency. The coordinator of each transaction is the primary authority for the transaction's outcome. If the coordinator of the transaction dies between the phases of the commit, the transaction branches stay in the prepared state until the coordinator is recovered and can be asked again about the outcome of the transaction. Likewise, if a non-coordinating cluster node with a transaction branch dies between the phases, it will do a roll-forward and ask the coordinator for the outcome of the transaction.</p> <p>If cluster nodes occasionally crash and then recover relatively quickly, without ever losing transaction logs or database files, this is resilient enough. Everything is symmetrical; there are no cluster nodes with special functions, except for one master node that has the added task of resolving distributed deadlocks.</p> <p>I suppose our anti-SQL person called 2PC "anti-availability" because in the above situation we have the following problems: if any one cluster node is offline, it is quite likely that no transaction can be committed. This is so unless the data is partitioned on a key with application semantics, and all data touched by a transaction usually stays within a single partition. Then operations could proceed on most of the data while one cluster node was recovering. But, especially with RDF, this is never the case, since keys are partitioned in ways that have nothing to do with application semantics. Further, if one uses XA or <a href="http://dbpedia.org/resource/Microsoft" id="link-id0x16f60a98">Microsoft</a> DTC with the monitor on a single box, this box can become a bottleneck and/or a single point of failure. (Among other considerations, this is why <a href="http://virtuoso.openlinksw.com" id="link-id0x21c1aaa0">Virtuoso</a> does not rely on any such monitor.) Further, if a cluster node dies never to be heard of again, leaving prepared but uncommitted transaction branches, the rest of the system has no way of telling what to do with them, again unless relying on a monitor that is itself liable to fail.</p> <p>If transactions have a real world counterpart, it is possible, at least in theory, to check the outcome against the real world state: One can ask a customer if an order was actually placed or a shipment delivered. But when a transaction has to do with internal identifiers of things, for example whether <b><code>mailto://plaidskirt@hotdate.com</code></b> has internal ID <b><code>0xacebabe</code></b>, such a check against external reality is not possible.</p> <h2>Fault-Tolerant Operation</h2> <p>In a fault tolerant setting, we introduce the following extra elements: Cluster nodes are comprised of "quorums" of mutually-mirroring server instances. Each such quorum holds a partition of the data. Such a quorum typically consists of two server instances, but may have three for extra safety. If all server instances in the quorum are offline, then the cluster node is offline, and the cluster is not fully operational. If at least one server instance in a quorum is online, then the cluster node is online, and the cluster is operational and can process new transactions.</p> <p>We designate one cluster node (<i>i.e.,</i> one quorum of 2 or 3 server instances) to act as a master node, and we set an order of precedence among its member instances. In addition to arbitrating distributed deadlocks, the master instance on duty will handle reports of server instance failures, and answer questions about any transactions left hanging in prepared state by a dead transaction coordinator. If the master on duty fails, the next master in line will either notice this itself in the line of normal business or get a complaint from another server instance about not being able to contact the previous master.</p> <p>There is no global heartbeat messaging <i>per se,</i> but since connections between server instances are reused long-term, a dropped connection will be noticed and the master on duty will be notified. If all masters are unavailable, that entire quorum (<i>i.e.,</i> the master node) is offline and thus (as with any entire node going offline) most operations will fail anyway, unless by chance they do not hit any data managed by that failed quorum.</p> <p>When it receives a notice of unavailability, the master instance on duty tries to contact the unavailable server instance and if it fails, it will notify all remaining instances that that server instance is removed from the cluster. The effect is that the remaining server instances will stop attempting to access the failed instance. Updates to the partitions managed by the failed server instance are no longer sent to it, which results in updates to this data succeeding, as they are made against the other server instances in that quorum. Updates to the data of the failed server instance <i>will</i> fail in the window of time between the actual failure and the removal, which is typically well under a second. The removal of a failed server instance is delegated to a central authority in order not to have everybody get in each other's way when trying to effect the removal.</p> <p>If the failed server instance left prepared uncommitted transactions behind, the server instances having such branches will in due order contact the transaction coordinator to ask what should be done. This is a normal procedure for dealing with possibly dropped commit or rollback messages. When they discover that the coordinator has been removed, the master on duty will be contacted instead. Each prepare message of a transaction lists all the server instances participating in the transaction; thus the master can check whether each has received the prepare. If all have the prepare and none has an abort, the transaction is committed. The dead coordinator may not know this or may indeed not have the transaction logged, since it sends the prepares before logging its own prepare. The recovery will handle this though. We note that of the remaining branches, there is at least one copy of the branch with the failed server instance, or else we would have a whole quorum failed. In cases where there are branches participating in an unresolved transaction where all the quorum members have failed, the system cannot decide the outcome, and will periodically retry until at least one member of the failed quorum becomes available.</p> <p>The most complex part of the protocol is the recovery of a failed server instance. The recovery starts with a normal roll forward from the local transaction log. After this, the server instance will contact the master on duty to ask for its state. Typically, the master will reply that the recovering server instance had been removed and is out of date. When this is established, the recovering server instance will contact a live member of its quorum and ask for sync. The failed server instance has an approximate timestamp of its last received transaction. It knows this from the roll forward, where time markers are interspersed now and then between transaction records. The live partner then sends its transaction log(s) covering the time from a few seconds before the last transaction of the failed partner up to the present. A few transactions may get rolled forward twice but this does no harm, since these records have absolute values and no deltas and the second insert of a key is simply ignored. When the sender of the log reaches its last committed log entry, it asks the recovering server instance to confirm successful replay of the log so far. Having the confirmation, the sender will abort all unprepared transactions affecting it and will not accept any new ones until the sync is completed. If new transactions were committed between sending the last of the log and killing the uncommitted new transactions, these too are shipped to the recovering server instance in their committed or prepared state. When these are also confirmed replayed, the recovering server instance is in exact sync up to the transaction. The sender then notifies the rest of the cluster that the sync is complete and that the recovered server instance will be included in any updates of its slice of the data. The time between freeze and re-enable of transactions is the time to replay what came in between the first sync and finishing the freeze. Typically nothing came in, so the time is in milliseconds. If an application got its transaction killed in this maneuver, it will be seen as a deadlock.</p> <p>If the recovering server instance received transactions in prepared state, it will ask about their outcome as a part of the periodic sweep through pending transactions. One of these transactions could have been one originally prepared by itself, where the prepares had gone out before it had time to log the transaction. Thus, this eventuality too is covered and has a consistent outcome. Failures can interrupt the recovery process. The recovering server instance will have logged as far as it got, and will pick up from this point onward. Real time clocks on the host nodes of the cluster will have to be in approximate sync, within a margin of a minute or so. This is not a problem in a closely connected network.</p> <p>For simultaneous failure of a entire quorum of server instances (<i>i.e.,</i> a set of mutually-mirroring partners; a cluster node), the rule is that the last one to fail must be the first to come back up. In order to have uninterrupted service across arbitrary double failures, one must store things in triplicate; statistically, however, most double failures will not hit cluster nodes of the same group.</p> <p>The protocol for recovery of failed server instances of the master quorum (<i>i.e.,</i> the master cluster node) is identical, except that a recovering master will have to ask the other master(s) which one is more up to date. If the recovering master has a log entry of having excluded all other masters in its quorum from the cluster, it can come back online without asking anybody. If there is no such entry, it must ask the other master(s). If all had failed at the exact same instant, none has an entry of the other(s) being excluded and all will know that they are in the same state since any update to one would also have been sent to the other(s).</p> <h2>Failure of Storage Media</h2> <p>When a server instance fails, its permanent storage may or may not survive. Especially with mirrored disks, storage most often survives a failure. However, the survival of the database does not depend on any single server instance retaining any permanent storage over failure. If storage is left in place, as in the case of an OS reboot or replacing a faulty memory chip, rejoining the cluster is done based on the existing copy of the database on the server instance. if there is no existing copy, a copy can be taken from any surviving member of the same quorum. This consists of the following steps: First, a log checkpoint is forced on the surviving instance. Normally log checkpoints are done at regular intervals, independently on each server instance. The log checkpoint writes a consistent state of the database to permanent storage. The disk pages forming this consistent image will not be written to until the next log checkpoint. Therefore copying the database file is safe and consistent as long as a log checkpoint does not take place between the start and end of copy. Thus checkpoints are disabled right after the initial checkpoint. The copy can take a relatively long time; consider 20s per gigabyte on a 1GbE network a good day. At the end of copy, checkpoints are re-enabled on the surviving cluster node. The recovering database starts without a log, sees the timestamp of the checkpoint in the database, and asks for transactions from just before this time up to present. The recovery then proceeds as outlined above.</p> <h2>Network Failures</h2> <p>The CAP theorem states that Consistency, Availability, and Partition-tolerance do not mix. "Partition" here means the split of a network.</p> <p>It is trivially true that if the network splits so that on both sides there is a copy of each partition of the data, both sides will think themselves the live copy left online after the other died, and each will thus continue to accumulate updates. Such an event is not very probable within one site where all machines are redundantly connected to two independent switches. Most servers have dual 1GbE on the motherboard, and both ports should be used for cluster interconnect for best performance, with each attached to an independent switch. Both switches would have to fail in such a way as to split their respective network for a single-site network split to happen. Of course, the likelihood of a network split in multi-site situations is higher.</p> <p>One way of guarding against network splits is to require that at least one partition of the data have all copies online. Additionally, the master on duty can request each cluster node or server instance it expects to be online to connect to every other node or instance, and to report which they could reach. If the reports differ, there is a network problem. This procedure can be performed using both interfaces or only the first or second interface of each server to determine if one of the switches selectively blocks some paths. These simple sanity checks protect against arbitrary network errors. Using TCP for inter-cluster-node communication in principle protects against random message loss, but the Virtuoso cluster protocols do not rely on this. Instead, there are protocols for retry of any transaction messages and for using keep-alive messages on any long-running functions sent across the cluster. Failure to get a keep-alive message within a certain period will abort a query even if the network connections look OK. </p> <h2>Backups, and Recovery from Loss of Entire Site</h2> <p>For a constantly-operating distributed system, it is hard to define what exactly constitutes a consistent snapshot. The checkpointed state on each cluster node is consistent as far as this cluster node is concerned (<i>i.e.,</i> it contains no uncommitted data), but the checkpointed states on all the cluster nodes are not from exactly the same moment in time. The complete state of a cluster is the checkpoint state of each cluster node plus the current transaction log of each. If the logs were shipped in real time to off-site storage, a consistent image could be reconstructed from them. Since such shipping cannot be synchronous due to latency considerations, some transactions could be received only in part in the event of a failure of the off-site link. Such partial transactions can however be detected at reconstruction time because each record contains the list of all participants of the transaction. If some piece is found missing, the whole can be discarded. In this way integrity is guaranteed but it is possible that a few milliseconds worth of transactions get lost. In these cases, the online client will almost certainly fail to get the final success message and will recheck the status after recovery.</p> <p>For business continuity purposes, a live feed of transactions can be constantly streamed off-site, for example to a cloud infrastructure provider. One low-cost virtual machine on the cloud will typically be enough for receiving the feed. In the event of long-term loss of the whole site, replacement servers can be procured on the cloud; thus, capital is not tied up in an aging inventory of spare servers. The cloud-based substitute can be maintained for the time it takes to rebuild an owned infrastructure, which is still at present more economical than a cloud-only solution.</p> <p>Switching a cluster from an owned site to the cloud could be accomplished in a few hours. The prerequisite of this is that there are reasonably recent snapshots of the database files, so that replay of logs does not take too long. The bulk of the time taken by such a switch would be in transferring the database snapshots from S3 or similar to the newly provisioned machines, formatting the newly provisioned virtual disks, etc.</p> <p>Rehearsing such a maneuver beforehand is quite necessary for predictable execution. We do not presently have a productized set of tools for such a switch, but can advise any interested parties on implementing and testing such a disaster recovery scheme.</p> <h2>Conclusions</h2> <p>In conclusion, we have shown how we can have strong transactional guarantees in a database cluster without single points of failure or performance penalties when compared with a non fault-tolerant cluster. Operator intervention is not required for anything short of hardware failure. Recovery procedures are simple, at most consisting of installing software and copying database files from a surviving cluster node. Unless permanent storage is lost in the failure, not even this is required. Real-time off-site log shipment can easily be added to these procedures to protect against site-wide failures.</p> <p>Future work may be directed toward concurrent operation of geographically-distributed data centers with eventual consistency. Such a setting would allow for migration between sites in the event of whole-site failures, and for reconciliation between inconsistent histories of different halves of a temporarily split network. Such schemes are likely to require application-level logic for reconciliation and cannot consist of an out-of-the-box DBMS alone. All techniques discussed here are application-agnostic and will work equally well for Graph Model (<i>e.g.,</i> RDF) and Relational Model (<i>e.g.,</i> SQL) workloads.</p> <h3> <a href="http://dbpedia.org/resource/Glossary" id="link-id0x28eef990">Glossary</a> </h3> <ul> <li> <b>Virtuoso Cluster (VC)</b> -- a collection of Virtuoso Cluster Nodes on one or more machines, working in parallel as part of a Virtuoso Cluster.</li> <li> <b>Virtuoso Cluster Node (VCN)</b> -- a Virtuoso Server Instance (Non Fault-Tolerant Operations), or a Quorum of Server Instances (Fault Tolerant Operations), which is a member of a collection of Virtuoso Cluster Nodes working in parallel as part of a Virtuoso Cluster.</li> <li> <b>Virtuoso Host Cluster (VHC)</b> -- a collection of machines, each hosting one or more Virtuoso Server Instances, making up a Virtuoso Cluster.</li> <li> <b>Virtuoso Host Cluster Node (VHCN)</b> -- a machine hosting one or more Virtuoso Server Instances that are members of a Virtuoso Cluster.</li> <li> <b>Virtuoso Server Instance (VSI)</b> -- a single Virtuoso process with exclusive access to its own permanent storage, consisting of database files and logs. May comprise an entire Virtuoso Cluster Node (Non Fault-Tolerant Operations), or be one member of a quorum which comprises a Virtuoso Cluster Node (Fault Tolerant Operations).</li> </ul> <h3>Also see</h3> <ul> <li> <a href="http://www.gbcacm.org/sites/www.gbcacm.org/files/slides/SpecialRelativity[1]_0.pdf" id="link-id0x16cb22d8">Special Relativity and the Problem of Database Scalability (PDF)</a>, by James Starkey of <a href="http://www.nimbusdb.com/" id="link-id0x18f30d58">NimbusDB, Inc.</a> </li> </ul>
2010-04-14T19:12:36-04:00
Fault Tolerance in Virtuoso Cluster Edition (Short Version)
http://www.openlinksw.com:443/blog/vdb/blog/?date=2010-04-07#1621
2010-04-07T16:40:02Z
<p>We have for some time had the option of storing <a href="http://dbpedia.org/resource/Data" id="link-id0x28eb2178">data</a> in a cluster in multiple copies, in the Commercial Edition of <a href="http://virtuoso.openlinksw.com" id="link-id0x25178ed0">Virtuoso</a>. (This feature is not in and is not planned to be added to the Open Source Edition.)</p> <p>Based on some feedback from the field, we decided to make this feature more user friendly. The gist of the matter is that failure and recovery processes have been automated so that neither application developer nor operating personnel needs any <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x21fea428">knowledge</a> of how things actually work.</p> <p>So I will here make a few high level statements about what we offer for fault tolerance. I will follow up with technical specifics in another post.</p> <p>Three types of individuals need to know about fault tolerance:</p> <ul> <li>Executives: What does it cost? Will it really eliminate downtime?</li> <li>System Administrators: Is it hard to configure? What do I do when I get an alert?</li> <li>Application Developers/Programmers: Will I need to write extra code? Can old applications get fault tolerance with no changes?</li> </ul> <p>I will explain the matter to each of these three groups:</p> <h2>Executives</h2> <p>The value gained is elimination of downtime. The cost is in purchasing twice (or thrice) the hardware and software licenses. In reality, the cost is less since you get the whole money's worth of read throughput and half the money's worth of write throughput. Since most applications are about reading, this is a good deal. You do not end up paying for unused capacity.</p> <p>Server instances are grouped in "quorums" of two or, for extra safety, three; as long as one member of each quorum is available, the system keeps running and nobody sees a difference, except maybe for slower response. This does not protect against widespread power outage or the building burning down; the scope is limited to hardware and software failures at one site.</p> <p>The most basic site-wide disaster recovery plan consists of constantly streaming updates off-site. Using an off-site backup plus update stream, one can reconstitute the failed data center on a cloud provider in a few hours. Details will vary; please <a href="http://www.openlinksw.com/contact/" id="link-id0x2bdb0db8">contact us</a> for specifics.</p> <p>Running multiple sites in parallel is also possible but specifics will depend on the application. Again, please contact us if you have a specific case in mind.</p> <h2> System Administrators</h2> <p>To configure, divide your server instances into quorums of 2 or 3, according to which will be mirrors of each other, with each quorum member on a different host from the others in its quorum. These things are declared in a configuration file. Table definitions do not have to be altered for fault tolerance. It is enough for tables and indices to specify partitioning. Use two switches, and two NICs per machine, and connect one of each server's network cables to each switch, to cover switch failures.</p> <p>When things break, as long as there is at least one server instance up from each quorum, things will continue to work. Reboots and the like are handled without operator intervention; if there is a broken host, then remove it and put a spare in its place. If the disks are OK, put the old disks in the replacement host and start. If the disks are gone, then copy the database files from the live copy. Finally start the replacement database, and the system will do the rest. The system is online in read-write mode during all this time, including during copying.</p> <p>Having mirrored disks in individual hosts is optional since data will anyhow be in two copies. Mirrored disks will shorten the vulnerability window of running a partition on a single server instance since this will for the most part eliminate the need to copy many (hundreds) of GB of database files when recovering a failed instance.</p> <h2> Application Developers/Programmers</h2> <p>An application can connect to any server instance in the cluster and have access to the same data, with full <a href="http://dbpedia.org/resource/ACID" id="link-id0x6451870">ACID</a> properties.</p> <p>There are two types of errors that can occur in any database application: The database server instance may be offline or otherwise unreachable; and a transaction may be aborted due to a deadlock.</p> <p>For the missing server instance, the application should try to reconnect. An <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x28e859b8">ODBC</a>/<a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x28e11940">JDBC</a> connect string can specify a list of alternate server instances; thus as long as the application is written to try to reconnect as best practices dictate, there is no new code needed.</p> <p>For the deadlock, the application is supposed to retry the transaction. Sometimes when a server instance drops out or rejoins a running cluster, some transactions will have to be retried. To the application, these conditions look like a deadlock. If the application handles deadlocks (<a href="http://dbpedia.org/resource/SQL" id="link-id0x2bda4e40">SQL</a> State 40001) as best practices dictate, there is no change needed.</p> <h2>Conclusion</h2> <p>In summary...</p> <ul> <li>Limited extra cost for fault tolerance; no equipment sitting idle.</li> <li>Easy operation: Replace servers when they fail; the cluster does the rest.</li> <li>No changes needed to most applications.</li> <li>No proprietary SQL APIs or special fault tolerance logic needed in applications.</li> <li>Fully transactional programming model.</li> </ul> <p>All the above applies to both the Graph Model (<a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x22606f10">RDF</a>) and Relational (SQL) sides of Virtuoso. These features will be in the commercial release of Virtuoso to be publicly available in the next 2-3 weeks. Please <a href="http://www.openlinksw.com/contact/" id="link-id0x24f35648">contact OpenLink Software</a> Sales for details of availability or for getting advance evaluation copies.</p> <h3> <a href="http://dbpedia.org/resource/Glossary" id="link-id0x6648890">Glossary</a> </h3> <ul> <li> <b>Virtuoso Cluster (VC)</b> -- a collection of Virtuoso Cluster Nodes on one or more machines, working in parallel as part of a Virtuoso Cluster.</li> <li> <b>Virtuoso Cluster Node (VCN)</b> -- a Virtuoso Server Instance (Non Fault-Tolerant Operations), or a Quorum of Server Instances (Fault Tolerant Operations), which is a member of a collection of Virtuoso Cluster Nodes working in parallel as part of a Virtuoso Cluster.</li> <li> <b>Virtuoso Host Cluster (VHC)</b> -- a collection of machines, each hosting one or more Virtuoso Server Instances, making up a Virtuoso Cluster.</li> <li> <b>Virtuoso Host Cluster Node (VHCN)</b> -- a machine hosting one or more Virtuoso Server Instances that are members of a Virtuoso Cluster.</li> <li> <b>Virtuoso Server Instance (VSI)</b> -- a single Virtuoso process with exclusive access to its own permanent storage, consisting of database files and logs. May comprise an entire Virtuoso Cluster Node (Non Fault-Tolerant Operations), or be one member of a quorum which comprises a Virtuoso Cluster Node (Fault Tolerant Operations).</li> </ul> <h3>Also see</h3> <ul> <li> <a href="http://www.gbcacm.org/sites/www.gbcacm.org/files/slides/SpecialRelativity[1]_0.pdf" id="link-id0x1320f1e8">Special Relativity and the Problem of Database Scalability (PDF)</a>, by James Starkey of <a href="http://www.nimbusdb.com/" id="link-id0x1320f2b0">NimbusDB, Inc.</a> </li> </ul>
2010-04-14T19:12:47.000003-04:00
Fault Tolerance in Virtuoso Cluster Edition (Short Version)
http://www.openlinksw.com:443/weblog/oerling/?date=2010-04-07#1620
2010-04-07T16:40:02Z
<p>We have for some time had the option of storing <a href="http://dbpedia.org/resource/Data" id="link-id0x7278bc0">data</a> in a cluster in multiple copies, in the Commercial Edition of <a href="http://virtuoso.openlinksw.com" id="link-id0x16da4d00">Virtuoso</a>. (This feature is not in and is not planned to be added to the Open Source Edition.)</p> <p>Based on some feedback from the field, we decided to make this feature more user friendly. The gist of the matter is that failure and recovery processes have been automated so that neither application developer nor operating personnel needs any <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x22447f98">knowledge</a> of how things actually work.</p> <p>So I will here make a few high level statements about what we offer for fault tolerance. I will follow up with technical specifics in another post.</p> <p>Three types of individuals need to know about fault tolerance:</p> <ul> <li>Executives: What does it cost? Will it really eliminate downtime?</li> <li>System Administrators: Is it hard to configure? What do I do when I get an alert?</li> <li>Application Developers/Programmers: Will I need to write extra code? Can old applications get fault tolerance with no changes?</li> </ul> <p>I will explain the matter to each of these three groups:</p> <h2>Executives</h2> <p>The value gained is elimination of downtime. The cost is in purchasing twice (or thrice) the hardware and software licenses. In reality, the cost is less since you get the whole money's worth of read throughput and half the money's worth of write throughput. Since most applications are about reading, this is a good deal. You do not end up paying for unused capacity.</p> <p>Server instances are grouped in "quorums" of two or, for extra safety, three; as long as one member of each quorum is available, the system keeps running and nobody sees a difference, except maybe for slower response. This does not protect against widespread power outage or the building burning down; the scope is limited to hardware and software failures at one site.</p> <p>The most basic site-wide disaster recovery plan consists of constantly streaming updates off-site. Using an off-site backup plus update stream, one can reconstitute the failed data center on a cloud provider in a few hours. Details will vary; please <a href="http://www.openlinksw.com/contact/" id="link-id0x227c91d0">contact us</a> for specifics.</p> <p>Running multiple sites in parallel is also possible but specifics will depend on the application. Again, please contact us if you have a specific case in mind.</p> <h2> System Administrators</h2> <p>To configure, divide your server instances into quorums of 2 or 3, according to which will be mirrors of each other, with each quorum member on a different host from the others in its quorum. These things are declared in a configuration file. Table definitions do not have to be altered for fault tolerance. It is enough for tables and indices to specify partitioning. Use two switches, and two NICs per machine, and connect one of each server's network cables to each switch, to cover switch failures.</p> <p>When things break, as long as there is at least one server instance up from each quorum, things will continue to work. Reboots and the like are handled without operator intervention; if there is a broken host, then remove it and put a spare in its place. If the disks are OK, put the old disks in the replacement host and start. If the disks are gone, then copy the database files from the live copy. Finally start the replacement database, and the system will do the rest. The system is online in read-write mode during all this time, including during copying.</p> <p>Having mirrored disks in individual hosts is optional since data will anyhow be in two copies. Mirrored disks will shorten the vulnerability window of running a partition on a single server instance since this will for the most part eliminate the need to copy many (hundreds) of GB of database files when recovering a failed instance.</p> <h2> Application Developers/Programmers</h2> <p>An application can connect to any server instance in the cluster and have access to the same data, with full <a href="http://dbpedia.org/resource/ACID" id="link-id0x6af3708">ACID</a> properties.</p> <p>There are two types of errors that can occur in any database application: The database server instance may be offline or otherwise unreachable; and a transaction may be aborted due to a deadlock.</p> <p>For the missing server instance, the application should try to reconnect. An <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x80112c8">ODBC</a>/<a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x66956d8">JDBC</a> connect string can specify a list of alternate server instances; thus as long as the application is written to try to reconnect as best practices dictate, there is no new code needed.</p> <p>For the deadlock, the application is supposed to retry the transaction. Sometimes when a server instance drops out or rejoins a running cluster, some transactions will have to be retried. To the application, these conditions look like a deadlock. If the application handles deadlocks (<a href="http://dbpedia.org/resource/SQL" id="link-id0x6a34cc0">SQL</a> State 40001) as best practices dictate, there is no change needed.</p> <h2>Conclusion</h2> <p>In summary...</p> <ul> <li>Limited extra cost for fault tolerance; no equipment sitting idle.</li> <li>Easy operation: Replace servers when they fail; the cluster does the rest.</li> <li>No changes needed to most applications.</li> <li>No proprietary SQL APIs or special fault tolerance logic needed in applications.</li> <li>Fully transactional programming model.</li> </ul> <p>All the above applies to both the Graph Model (<a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x20533288">RDF</a>) and Relational (SQL) sides of Virtuoso. These features will be in the commercial release of Virtuoso to be publicly available in the next 2-3 weeks. Please <a href="http://www.openlinksw.com/contact/" id="link-id0x6809510">contact OpenLink Software</a> Sales for details of availability or for getting advance evaluation copies.</p> <h3> <a href="http://dbpedia.org/resource/Glossary" id="link-id0x24faaae8">Glossary</a> </h3> <ul> <li> <b>Virtuoso Cluster (VC)</b> -- a collection of Virtuoso Cluster Nodes on one or more machines, working in parallel as part of a Virtuoso Cluster.</li> <li> <b>Virtuoso Cluster Node (VCN)</b> -- a Virtuoso Server Instance (Non Fault-Tolerant Operations), or a Quorum of Server Instances (Fault Tolerant Operations), which is a member of a collection of Virtuoso Cluster Nodes working in parallel as part of a Virtuoso Cluster.</li> <li> <b>Virtuoso Host Cluster (VHC)</b> -- a collection of machines, each hosting one or more Virtuoso Server Instances, making up a Virtuoso Cluster.</li> <li> <b>Virtuoso Host Cluster Node (VHCN)</b> -- a machine hosting one or more Virtuoso Server Instances that are members of a Virtuoso Cluster.</li> <li> <b>Virtuoso Server Instance (VSI)</b> -- a single Virtuoso process with exclusive access to its own permanent storage, consisting of database files and logs. May comprise an entire Virtuoso Cluster Node (Non Fault-Tolerant Operations), or be one member of a quorum which comprises a Virtuoso Cluster Node (Fault Tolerant Operations).</li> </ul> <h3>Also see</h3> <ul> <li> <a href="http://www.gbcacm.org/sites/www.gbcacm.org/files/slides/SpecialRelativity[1]_0.pdf" id="link-id0x1320f1e8">Special Relativity and the Problem of Database Scalability (PDF)</a>, by James Starkey of <a href="http://www.nimbusdb.com/" id="link-id0x1320f2b0">NimbusDB, Inc.</a> </li> </ul>
2010-04-14T19:12:23-04:00
"The Acquired, The Innate, and the Semantic" or "Teaching Sem Tech"
http://www.openlinksw.com:443/blog/vdb/blog/?date=2010-04-05#1619
2010-04-05T15:21:19Z
<p>I was recently asked to write a section for a policy document touching the intersection of database and semantics, as a follow up to the meeting in Sofia I <a href="http://www.openlinksw.com/weblog/oerling/?id=1614" id="link-id0x19c4f938">blogged about earlier</a>. I will write about technology, but this same document also touches the matter of education and computer science curricula. Since the matter came up, I will share a few thoughts on the latter topic.</p> <p>I have over the years trained a few truly excellent engineers and managed a heterogeneous lot of people. These days, since what we are doing is in fact quite difficult and the world is not totally without competition, I find that I must stick to core competence, which is hardcore tech and leave management to those who have time for it.</p> <p>When younger, I thought that I could, through sheer personal charisma, transfer either technical skills, sound judgment, or drive and ambition to people I was working with. Well, to the extent I believed this, my own judgment was not sound. Transferring anything at all is difficult and chancy. I must here think of a fantasy novel where a wizard said that, "working such magic that makes things do what they already want to do is easy." There is a grain of truth in that.</p> <p>In order to build or manage organizations, we must work, as the wizard put it, <i>with</i> nature, not against it. There are also counter-examples, for example my wife's grandmother had decided to transform a regular willow into a weeping one by tying down the branches. Such "magic," needless to say, takes constant maintenance; else the spell breaks.</p> <p>To operate efficiently, either in business or education, we need to steer away from such endeavors. This is a valuable lesson, but now consider teaching this to somebody. Those who would most benefit from this wisdom are the least receptive to it. So again, we are reminded to stay away from the fantasy of being able to transfer some understanding we think to have and to have this take root. It will if it will and if it does not, it will take constant follow up, like the would-be weeping willow.</p> <p>Now, in more specific terms, what can we realistically expect to teach about computer science?</p> <p>Complexity of algorithms would be the first thing. Understanding the relative throughputs and latencies of the memory hierarchy (i.e., <a href="http://dbpedia.org/resource/Cache" id="link-id0x13fcc8b8">cache</a>, memory, local network, disk, wide area network) is the second. Understanding the difference of synchronous and asynchronous and the cost of synchronization (i.e., anything from waiting for a mutex to waiting for a network message) is the third.</p> <p>Understanding how a database works would be immensely helpful for almost any application development task but this is probably asking too much.</p> <p>Then there is the question of engineering. Where do we put interfaces and what should these interfaces expose? Well, they certainly should expose multiple instances of whatever it is they expose, since passing through an interface takes time.</p> <p>I tried once to tell the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x72d7490">SPARQL</a> committee that parameterized queries and array parameters are a self-evident truism on the database side. This is an example of an interface that exposes multiple instances of what it exposes. But the committee decided not to standardize these. There is something in the "semanticist" mind that is irrationally antagonistic to what is self-evident for databasers. This is further an example of ignoring precept 2 above, the point about the throughputs and latencies in the memory hierarchy. Nature is a better and more patient teacher than I; the point will become clear of itself in due time, no worry.</p> <p>Interfaces seem to be overvalued in education. This is tricky because we should not teach that interfaces are bad either. Nature has islands of tightly intertwined processes, separated by fairly narrow interfaces. People are taught to think in block diagrams, so they probably project this also where it does not apply, thereby missing some connections and porosity of interfaces.</p> <p> <a href="http://www.larkc.eu/" id="link-id0x1c5591f0">LarKC</a> (EU FP7 Large <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x15fae798">Knowledge</a> Collider project) is an exercise in interfaces. The lessons so far are that coupling needs to be tight, and that the roles of the components are not always as neatly separable as the block diagram suggests.</p> <p>Recognizing the points where interfaces are naturally narrow is very difficult. Teaching this in a curriculum is likely impossible. This is not to say that the matter should not be mentioned and examples of over-"paradigmatism" given. The geek mind likes to latch on to a paradigm (e.g., object orientation), and then they try to put it everywhere. It is safe to say that taking block diagrams too naively or too seriously makes for poor performance and needless code. In some cases, block diagrams can serve as tactical disinformation; i.e., you give lip service to the values of structure, <a href="http://dbpedia.org/resource/Information" id="link-id0x6f03e90">information</a> hiding, and reuse, which one is not allowed to challenge, ever, and at the same time you do not disclose the competitive edge, which is pretty much always a breach of these same principles.</p> <p>I was once at a <a href="http://dbpedia.org/resource/Data" id="link-id0x1d524ce0">data</a> integration workshop in the US where some very qualified people talked about the process of science. They had this delightfully American metaphor for it:</p> <blockquote> <i>The edge is created in the "Wild West" — there are no standards or hard-and-fast rules, and paradigmatism for paradigmatism's sake is a laughing matter with the cowboys in the fringe where new ground is broken. Then there is the OK Corral, where the cowboys shoot it out to see who prevails. Then there is Dodge City, where the lawman already reigns, and compliance, standards, and paradigms are not to be trifled with, lest one get the tar-and-feather treatment and be "driven out o'Dodge."</i> </blockquote> <p>So, if reality is like this, what attitude should the curriculum have towards it? Do we make innovators or followers? Well, as said before, they are not made. Or if they are made, they are not at least made in the university but much before that. I never made any of either, in spite of trying, but did meet many of both kinds. The education system needs to recognize individual differences, even though this is against the trend of turning out a standardized product. Enforced mediocrity makes mediocrity. The world has an amazing tolerance for mediocrity, it is true. But the edge is not created with this, if edge is what we are after.</p> <p>But let us move to specifics of semantic technology. What are the core precepts, the equivalent of the complexity/memory/synchronization triangle of general purpose CS basics? Let us not forget that, especially in semantic technology, when we have complex operations, lots of data, and almost always multiple distributed data sources, forgetting the laws of physics carries an especially high penalty.</p> <ul> <li> <p> <b>Know when to ontologize, when to folksonomize.</b> The history of standards has examples of "stacks of Babel," sky-high and all-encompassing, which just result in non-communication and non-adoption. Lighter weight, community driven, <a href="http://dbpedia.org/resource/Tag" id="link-id0x1dbd9018">tag</a> folksonomy, VoCamp-style approaches can be better. But this is a judgment call, entirely contextual, having to do with the maturity of the domain of discourse, etc.</p> </li> <li> <p> <b>Answer only questions that are actually asked.</b> This precept is two-pronged. The literal interpretation is not to do inferential closure for its own sake, materializing all implied facts of the knowledge base.</p> <p>The broader interpretation is to take real-world problems. Expanding RDFS semantics with map-reduce and proving how many iterations this will take is a thing one can do but real-world problems will be more complex and less neat.</p> </li> <li> <p> <b>Deal with ambiguity.</b> Data on which semantic technologies will be applied will be dirty, with errors from machine processing of natural language to erroneous human annotations. The knowledge bases will not be contradiction free. Michael Witbrock of CYC said many good things about this in Sofia; he would have something to say about a curriculum, no doubt.</p> </li> </ul> <p>Here we see that semantic technology is a younger discipline than computer science. We can outline some desirable skills and directions to follow but the idea of core precepts is not as well formed.</p> <p>So we can approach the question from the angle of needed skills more than of precepts of science. What should the certified semantician be able to do?</p> <ul> <li> <p> <b>Data integration.</b> Given heterogenous relational schemas talking about the same entities, the semantician should find existing ontologies for the domain, possibly extend these, and then map the relational data to them. After the mapping is conceptually done, the semantician must know what combination of ETL and on-the-fly mapping fits the situation. This does mean that the semantician indeed must understand databases, which I above classified as an almost unreachable ideal. But there is no getting around this. Data is increasingly what makes the world go round. From this it follows that everybody must increasingly publish, consume, and refine, i.e., integrate. The anti-database attitude of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x2038d520">semantic web</a> community simply has to go.</p> </li> <li> <p> <b>Design and implement workflows for content extraction,</b> e.g., <a href="http://dbpedia.org/resource/Natural_language_processing" id="link-id0x713cdc0">NLP</a> or information extraction from images. This also means familiarity with NLP, desirably to the point of being able to tune the extraction rule sets of various NLP frameworks.</p> </li> <li> <p> <b>Design SOA workflows.</b> The semantician should be able to extract and represent the semantics of business transactions and the data involved therein.</p> </li> <li> <p> <b>Lightweight knowledge engineering.</b> The experience of building expert systems from the early days of AI is not the best possible, but with semantics attached to data, some sort of rules seem about inevitable. The rule systems will merge into the DBMS in time. Some ability to work with these, short of making expert systems, will be desirable.</p> </li> <li> <p> <b>Understand information quality</b> in the sense of trust, provenance, errors in the information, etc. If the world is run based on data analytics, then one must know what the data in the warehouse means, what accidental and deliberate errors it contains, etc.</p> </li> </ul> <p>Of course, most of these tasks take place at some sort of organizational crossroads or interface. This means that the semantician must have some project management skills; must be capable of effectively communicating with different publics and simply getting the job done, always in the face of organizational inertia and often in the face of active resistance from people who view the semantician as some kind of intruder on their turf.</p> <p>Now, this is a tall order. The semantician will have to be reasonably versatile technically, reasonably clever, and a self-starter on top. The self-starter aspect is the hardest.</p> <p>The semanticists I have met are more of the scholar than the IT consultant profile. I say <i>semanticist</i> for the semantic web research people and <i>semantician</i> for the practitioner we are trying to define.</p> <p>We could start by taking people who already do data integration projects and educating them in some semantic technology. We are here talking about a different breed than the one that by nature gravitates to description logics and AI. Projecting semanticist interests or attributes on this public is a source of bias and error.</p> <p>If we talk about a university curriculum, the part that cannot be taught is the leadership and self-starter aspect, or whatever makes a good IT consultant. Thus the semantic technology studies must be profiled so as to attract people with this profile. As quoted before, the dream job for each era is a scarce skill that makes value from something that is plentiful in the environment. At this moment and for a few moments to come, this is the data geek, or maybe even semantician profile, if we take data geek past statistics and traditional business intelligence skills.</p> <p>The semantic tech community, especially the academic branch of it, needs to reinvent itself in order to rise to this occasion. The flavor of the dream job curriculum will be away from the theoretical computer science towards the hands-on of database, large systems performance, and the practicalities of getting data intensive projects delivered.</p> <p> <b>Related</b> </p> <ul> <li> <a href="http://virtuoso.openlinksw.com/presentations/Linked_Data_Virtualization/Linked_Data_Virtualization.html" id="link-id0x199aca78">Linked Data Driven Data Virtualization for Web-scale Integration (presentation)</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1603" id="link-id0x13297a70">Linked Data and Virtuoso in 2010</a> </li> <li> <a href="http://www.openlinksw.com/blog/~kidehen/?id=1595" id="link-id0x1a3d0bd0">Getting The Linked Data Value Pyramid Layers Right</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1572" id="link-id0x1802b170">Provenance and Reification in Virtuoso</a> </li> <li> <a href="http://www.openlinksw.com/blog/~kidehen/?id=1519" id="link-id0x19af4220">The Time for RDBMS Primacy Downgrade is Nigh!</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1375" id="link-id0x1a07a378">Aspects of RDF to RDF Mapping</a> </li> </ul>
2010-05-05T13:49:57-04:00
"The Acquired, The Innate, and the Semantic" or "Teaching Sem Tech"
http://www.openlinksw.com:443/weblog/oerling/?date=2010-04-05#1618
2010-04-05T15:21:19Z
<p>I was recently asked to write a section for a policy document touching the intersection of database and semantics, as a follow up to the meeting in Sofia I <a href="http://www.openlinksw.com/weblog/oerling/?id=1614" id="link-id0x19c4f938">blogged about earlier</a>. I will write about technology, but this same document also touches the matter of education and computer science curricula. Since the matter came up, I will share a few thoughts on the latter topic.</p> <p>I have over the years trained a few truly excellent engineers and managed a heterogeneous lot of people. These days, since what we are doing is in fact quite difficult and the world is not totally without competition, I find that I must stick to core competence, which is hardcore tech and leave management to those who have time for it.</p> <p>When younger, I thought that I could, through sheer personal charisma, transfer either technical skills, sound judgment, or drive and ambition to people I was working with. Well, to the extent I believed this, my own judgment was not sound. Transferring anything at all is difficult and chancy. I must here think of a fantasy novel where a wizard said that, "working such magic that makes things do what they already want to do is easy." There is a grain of truth in that.</p> <p>In order to build or manage organizations, we must work, as the wizard put it, <i>with</i> nature, not against it. There are also counter-examples, for example my wife's grandmother had decided to transform a regular willow into a weeping one by tying down the branches. Such "magic," needless to say, takes constant maintenance; else the spell breaks.</p> <p>To operate efficiently, either in business or education, we need to steer away from such endeavors. This is a valuable lesson, but now consider teaching this to somebody. Those who would most benefit from this wisdom are the least receptive to it. So again, we are reminded to stay away from the fantasy of being able to transfer some understanding we think to have and to have this take root. It will if it will and if it does not, it will take constant follow up, like the would-be weeping willow.</p> <p>Now, in more specific terms, what can we realistically expect to teach about computer science?</p> <p>Complexity of algorithms would be the first thing. Understanding the relative throughputs and latencies of the memory hierarchy (i.e., <a href="http://dbpedia.org/resource/Cache" id="link-id0x15761008">cache</a>, memory, local network, disk, wide area network) is the second. Understanding the difference of synchronous and asynchronous and the cost of synchronization (i.e., anything from waiting for a mutex to waiting for a network message) is the third.</p> <p>Understanding how a database works would be immensely helpful for almost any application development task but this is probably asking too much.</p> <p>Then there is the question of engineering. Where do we put interfaces and what should these interfaces expose? Well, they certainly should expose multiple instances of whatever it is they expose, since passing through an interface takes time.</p> <p>I tried once to tell the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x5ca2ea8">SPARQL</a> committee that parameterized queries and array parameters are a self-evident truism on the database side. This is an example of an interface that exposes multiple instances of what it exposes. But the committee decided not to standardize these. There is something in the "semanticist" mind that is irrationally antagonistic to what is self-evident for databasers. This is further an example of ignoring precept 2 above, the point about the throughputs and latencies in the memory hierarchy. Nature is a better and more patient teacher than I; the point will become clear of itself in due time, no worry.</p> <p>Interfaces seem to be overvalued in education. This is tricky because we should not teach that interfaces are bad either. Nature has islands of tightly intertwined processes, separated by fairly narrow interfaces. People are taught to think in block diagrams, so they probably project this also where it does not apply, thereby missing some connections and porosity of interfaces.</p> <p> <a href="http://www.larkc.eu/" id="link-id0x7f63780">LarKC</a> (EU FP7 Large <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x4f50c00">Knowledge</a> Collider project) is an exercise in interfaces. The lessons so far are that coupling needs to be tight, and that the roles of the components are not always as neatly separable as the block diagram suggests.</p> <p>Recognizing the points where interfaces are naturally narrow is very difficult. Teaching this in a curriculum is likely impossible. This is not to say that the matter should not be mentioned and examples of over-"paradigmatism" given. The geek mind likes to latch on to a paradigm (e.g., object orientation), and then they try to put it everywhere. It is safe to say that taking block diagrams too naively or too seriously makes for poor performance and needless code. In some cases, block diagrams can serve as tactical disinformation; i.e., you give lip service to the values of structure, <a href="http://dbpedia.org/resource/Information" id="link-id0x1f00c378">information</a> hiding, and reuse, which one is not allowed to challenge, ever, and at the same time you do not disclose the competitive edge, which is pretty much always a breach of these same principles.</p> <p>I was once at a <a href="http://dbpedia.org/resource/Data" id="link-id0x2021f598">data</a> integration workshop in the US where some very qualified people talked about the process of science. They had this delightfully American metaphor for it:</p> <blockquote> <i>The edge is created in the "Wild West" — there are no standards or hard-and-fast rules, and paradigmatism for paradigmatism's sake is a laughing matter with the cowboys in the fringe where new ground is broken. Then there is the OK Corral, where the cowboys shoot it out to see who prevails. Then there is Dodge City, where the lawman already reigns, and compliance, standards, and paradigms are not to be trifled with, lest one get the tar-and-feather treatment and be "driven out o'Dodge."</i> </blockquote> <p>So, if reality is like this, what attitude should the curriculum have towards it? Do we make innovators or followers? Well, as said before, they are not made. Or if they are made, they are not at least made in the university but much before that. I never made any of either, in spite of trying, but did meet many of both kinds. The education system needs to recognize individual differences, even though this is against the trend of turning out a standardized product. Enforced mediocrity makes mediocrity. The world has an amazing tolerance for mediocrity, it is true. But the edge is not created with this, if edge is what we are after.</p> <p>But let us move to specifics of semantic technology. What are the core precepts, the equivalent of the complexity/memory/synchronization triangle of general purpose CS basics? Let us not forget that, especially in semantic technology, when we have complex operations, lots of data, and almost always multiple distributed data sources, forgetting the laws of physics carries an especially high penalty.</p> <ul> <li> <p> <b>Know when to ontologize, when to folksonomize.</b> The history of standards has examples of "stacks of Babel," sky-high and all-encompassing, which just result in non-communication and non-adoption. Lighter weight, community driven, <a href="http://dbpedia.org/resource/Tag" id="link-id0x6b5d288">tag</a> folksonomy, VoCamp-style approaches can be better. But this is a judgment call, entirely contextual, having to do with the maturity of the domain of discourse, etc.</p> </li> <li> <p> <b>Answer only questions that are actually asked.</b> This precept is two-pronged. The literal interpretation is not to do inferential closure for its own sake, materializing all implied facts of the knowledge base.</p> <p>The broader interpretation is to take real-world problems. Expanding RDFS semantics with map-reduce and proving how many iterations this will take is a thing one can do but real-world problems will be more complex and less neat.</p> </li> <li> <p> <b>Deal with ambiguity.</b> Data on which semantic technologies will be applied will be dirty, with errors from machine processing of natural language to erroneous human annotations. The knowledge bases will not be contradiction free. Michael Witbrock of CYC said many good things about this in Sofia; he would have something to say about a curriculum, no doubt.</p> </li> </ul> <p>Here we see that semantic technology is a younger discipline than computer science. We can outline some desirable skills and directions to follow but the idea of core precepts is not as well formed.</p> <p>So we can approach the question from the angle of needed skills more than of precepts of science. What should the certified semantician be able to do?</p> <ul> <li> <p> <b>Data integration.</b> Given heterogenous relational schemas talking about the same entities, the semantician should find existing ontologies for the domain, possibly extend these, and then map the relational data to them. After the mapping is conceptually done, the semantician must know what combination of ETL and on-the-fly mapping fits the situation. This does mean that the semantician indeed must understand databases, which I above classified as an almost unreachable ideal. But there is no getting around this. Data is increasingly what makes the world go round. From this it follows that everybody must increasingly publish, consume, and refine, i.e., integrate. The anti-database attitude of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1c5f5398">semantic web</a> community simply has to go.</p> </li> <li> <p> <b>Design and implement workflows for content extraction,</b> e.g., <a href="http://dbpedia.org/resource/Natural_language_processing" id="link-id0x16443fe0">NLP</a> or information extraction from images. This also means familiarity with NLP, desirably to the point of being able to tune the extraction rule sets of various NLP frameworks.</p> </li> <li> <p> <b>Design SOA workflows.</b> The semantician should be able to extract and represent the semantics of business transactions and the data involved therein.</p> </li> <li> <p> <b>Lightweight knowledge engineering.</b> The experience of building expert systems from the early days of AI is not the best possible, but with semantics attached to data, some sort of rules seem about inevitable. The rule systems will merge into the DBMS in time. Some ability to work with these, short of making expert systems, will be desirable.</p> </li> <li> <p> <b>Understand information quality</b> in the sense of trust, provenance, errors in the information, etc. If the world is run based on data analytics, then one must know what the data in the warehouse means, what accidental and deliberate errors it contains, etc.</p> </li> </ul> <p>Of course, most of these tasks take place at some sort of organizational crossroads or interface. This means that the semantician must have some project management skills; must be capable of effectively communicating with different publics and simply getting the job done, always in the face of organizational inertia and often in the face of active resistance from people who view the semantician as some kind of intruder on their turf.</p> <p>Now, this is a tall order. The semantician will have to be reasonably versatile technically, reasonably clever, and a self-starter on top. The self-starter aspect is the hardest.</p> <p>The semanticists I have met are more of the scholar than the IT consultant profile. I say <i>semanticist</i> for the semantic web research people and <i>semantician</i> for the practitioner we are trying to define.</p> <p>We could start by taking people who already do data integration projects and educating them in some semantic technology. We are here talking about a different breed than the one that by nature gravitates to description logics and AI. Projecting semanticist interests or attributes on this public is a source of bias and error.</p> <p>If we talk about a university curriculum, the part that cannot be taught is the leadership and self-starter aspect, or whatever makes a good IT consultant. Thus the semantic technology studies must be profiled so as to attract people with this profile. As quoted before, the dream job for each era is a scarce skill that makes value from something that is plentiful in the environment. At this moment and for a few moments to come, this is the data geek, or maybe even semantician profile, if we take data geek past statistics and traditional business intelligence skills.</p> <p>The semantic tech community, especially the academic branch of it, needs to reinvent itself in order to rise to this occasion. The flavor of the dream job curriculum will be away from the theoretical computer science towards the hands-on of database, large systems performance, and the practicalities of getting data intensive projects delivered.</p> <p> <b>Related</b> </p> <ul> <li> <a href="http://virtuoso.openlinksw.com/presentations/Linked_Data_Virtualization/Linked_Data_Virtualization.html" id="link-id0x199aca78">Linked Data Driven Data Virtualization for Web-scale Integration (presentation)</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1603" id="link-id0x13297a70">Linked Data and Virtuoso in 2010</a> </li> <li> <a href="http://www.openlinksw.com/blog/~kidehen/?id=1595" id="link-id0x1a3d0bd0">Getting The Linked Data Value Pyramid Layers Right</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1572" id="link-id0x1802b170">Provenance and Reification in Virtuoso</a> </li> <li> <a href="http://www.openlinksw.com/blog/~kidehen/?id=1519" id="link-id0x19af4220">The Time for RDBMS Primacy Downgrade is Nigh!</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1375" id="link-id0x1a07a378">Aspects of RDF to RDF Mapping</a> </li> </ul>
2010-05-05T13:49:44-04:00
Upcoming RDF Loader in Unclustered Virtuoso loads Uniprot at 279 Ktriples/s!
http://www.openlinksw.com:443/blog/vdb/blog/?date=2010-04-02#1617
2010-04-02T14:15:01Z
<p>We recently heard that <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x21414c58">Oracle</a> 11G loaded <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x28281e50">RDF</a> faster than we did. Now, we never thought the speed of loading a database was as important as the speed of query results, but since this is the <b><i>sole</i></b> area where they have reportedly been tested as faster, we decided it was time loading was addressed. Indeed, without Oracle to challenge us on query performance, we would not be half as good as we are. So, spurred on by the Oracular influence, we did something about our RDF loading.</p> <p>Performance, I have said before, is a matter of locality and parallelism. So we applied both to the otherwise quite boring exercise of loading RDF. The recipe is this: Take a large set of triples; resolve the IRIs and literals into their IDs; then insert each index of the triple table on its own thread. All the lookups and inserts are first sorted in key order to get the locality. Running the indices in parallel gets the parallelism. Then run the parser on its own thread, fetching chunks of consecutive triples and queueing them for a pool of loader threads. Then run several parsers concurrently on different files so as to make sure there is work enough at all times. Do not make many more process threads than available <a href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x30f3b20">CPU</a> threads, since they would just get in each other's way.</p> <p>The whole process is non-transactional, starting from a checkpoint and ending with a checkpoint.</p> <p>The test system was a dual-Xeon 5520 with 72G RAM. The <a href="http://virtuoso.openlinksw.com" id="link-id0x3256138">Virtuoso</a> was a single server; no cluster capability was used.</p> <p>We loaded English <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x33b3e58">Dbpedia</a>, 179M triples, in 15 minutes, for a rate of 198 Kt/s. Uniprot with 1.33 G triples loaded in 79 minutes, for 279 Kt/s.</p> <p>The source files were the Dbpedia 3.4 English files and the <a href="http://www.bio2rdf.org/" id="link-id0x28266c20">Bio2RDF</a> copy of Uniprot, both in Turtle syntax. The uniref, uniparc and uniprot files from the Bio2RDF set were sliced into smaller chunks so as to have more files to load in parallel; the taxonomy file was as such; and no other Bio2RDF files were loaded. Both experiments ran with 8 load streams, 1 per core. The CPU utilization was mostly between 1400% and 1500%, 14-15 of 16 CPU threads busy. Top load speed for a measurement window of 2 minutes was 383 Kt/s.</p> <p>The index scheme for RDF quads was the default Virtuoso 6 configuration of 5 indices — GS, SP, OP, PSOG, and POGS. (We call this "3+2" indexing, because there are 3 partial and 2 full indices, delivering massive performance benefits over most other index schemes.) IRIs and literals reside in their own tables, each indexed from string to ID and vice versa. A full-text index on literals was not used.</p> <p>Compared to previous performance, we have more than tripled our best single server multi-stream load speed, and multiplied our single stream load speed by a factor of 8. Some further gains may be reached by adjusting thread counts and matching vector sizes to CPU <a href="http://dbpedia.org/resource/Cache" id="link-id0x20403130">cache</a>.</p> <p>This will be available in a forthcoming release; this is not for download yet. Now that you know this, you may guess what we are doing with queries. More on this another time.</p>
2010-04-02T12:59:15-04:00
Upcoming RDF Loader in Unclustered Virtuoso loads Uniprot at 279 Ktriples/s!
http://www.openlinksw.com:443/weblog/oerling/?date=2010-04-02#1616
2010-04-02T14:15:01Z
<p>We recently heard that <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x20585a18">Oracle</a> 11G loaded <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1fdae3e0">RDF</a> faster than we did. Now, we never thought the speed of loading a database was as important as the speed of query results, but since this is the <b><i>sole</i></b> area where they have reportedly been tested as faster, we decided it was time loading was addressed. Indeed, without Oracle to challenge us on query performance, we would not be half as good as we are. So, spurred on by the Oracular influence, we did something about our RDF loading.</p> <p>Performance, I have said before, is a matter of locality and parallelism. So we applied both to the otherwise quite boring exercise of loading RDF. The recipe is this: Take a large set of triples; resolve the IRIs and literals into their IDs; then insert each index of the triple table on its own thread. All the lookups and inserts are first sorted in key order to get the locality. Running the indices in parallel gets the parallelism. Then run the parser on its own thread, fetching chunks of consecutive triples and queueing them for a pool of loader threads. Then run several parsers concurrently on different files so as to make sure there is work enough at all times. Do not make many more process threads than available <a href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x200f9570">CPU</a> threads, since they would just get in each other's way.</p> <p>The whole process is non-transactional, starting from a checkpoint and ending with a checkpoint.</p> <p>The test system was a dual-Xeon 5520 with 72G RAM. The <a href="http://virtuoso.openlinksw.com" id="link-id0x21415250">Virtuoso</a> was a single server; no cluster capability was used.</p> <p>We loaded English <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x24604630">Dbpedia</a>, 179M triples, in 15 minutes, for a rate of 198 Kt/s. Uniprot with 1.33 G triples loaded in 79 minutes, for 279 Kt/s.</p> <p>The source files were the Dbpedia 3.4 English files and the <a href="http://www.bio2rdf.org/" id="link-id0x20366360">Bio2RDF</a> copy of Uniprot, both in Turtle syntax. The uniref, uniparc and uniprot files from the Bio2RDF set were sliced into smaller chunks so as to have more files to load in parallel; the taxonomy file was as such; and no other Bio2RDF files were loaded. Both experiments ran with 8 load streams, 1 per core. The CPU utilization was mostly between 1400% and 1500%, 14-15 of 16 CPU threads busy. Top load speed for a measurement window of 2 minutes was 383 Kt/s.</p> <p>The index scheme for RDF quads was the default Virtuoso 6 configuration of 5 indices — GS, SP, OP, PSOG, and POGS. (We call this "3+2" indexing, because there are 3 partial and 2 full indices, delivering massive performance benefits over most other index schemes.) IRIs and literals reside in their own tables, each indexed from string to ID and vice versa. A full-text index on literals was not used.</p> <p>Compared to previous performance, we have more than tripled our best single server multi-stream load speed, and multiplied our single stream load speed by a factor of 8. Some further gains may be reached by adjusting thread counts and matching vector sizes to CPU <a href="http://dbpedia.org/resource/Cache" id="link-id0x201033c8">cache</a>.</p> <p>This will be available in a forthcoming release; this is not for download yet. Now that you know this, you may guess what we are doing with queries. More on this another time.</p>
2015-08-20T17:30:31.198816-04:00
SemData@Sofia Roundtable write-up
http://www.openlinksw.com:443/blog/vdb/blog/?date=2010-03-15#1615
2010-03-15T14:46:57Z
<p>There was last week an <a href="http://www.semdata.org/" id="link-id11a83cf98">invitation-based roundtable</a> about semantic <a href="http://dbpedia.org/resource/Data" id="link-id0x1d37f598">data</a> management in <a href="http://www.dbpedia.org/resource/Sofia" id="link-id0x1ba4a208">Sofia, Bulgaria</a>.</p> <p>Lots of smart people together. The meeting was hosted by <a href="http://dbpedia.org/resource/Ontotext" id="link-id0x1cfc83f8">Ontotext</a> and chaired by <a href="http://www.dbpedia.org/resource/Dieter_Fensel" id="link-id0x1dc6e0d0">Dieter Fensel</a>. On the database side we had Ontotext, <a href="http://www.systap.com/" id="link-id0x1cda77f0">SYSTAP</a> (<a href="http://www.systap.com/bigdata.htm" id="link-id0x1dba6a30">Bigdata</a>), <a href="http://dbpedia.org/resource/National_Research_Institute_for_Mathematics_and_Computer_Science" id="link-id0x1d8e1d88">CWI</a> (<a href="http://dbpedia.org/resource/MonetDB" id="link-id0x1d8cbcf0">MonetDB</a>), <a href="http://www.dbpedia.org/resource/Karlsruhe_Institute_of_Technology" id="link-id0x1e204cb0">Karlsruhe Institute of Technology</a> (YARS2/<a href="http://swse.deri.ie/" id="link-id0x1e653bf0">SWSE</a>). <a href="http://www.larkc.eu/" id="link-id0x1e6a4408">LarKC</a> was well represented, being our hosts, with STI, Ontotext, CYC, and <a href="http://www.vu.nl/" id="link-id0x1c8a6090">VU Amsterdam</a>. Notable absences were <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x1e5ab690">Oracle</a>, <a href="http://freebase.com/guid/9202a8c04000641f8000000005c908d6" id="link-id0x1f5e5ff0">Garlik</a>, <a href="http://semanticweb.org/id/Franz_Inc" id="link-id0x1d9c08f0">Franz</a>, and <a href="http://www.talis.com/" id="link-id0x1d338b30">Talis</a>.</p> <p>Now of semantic data management... What is the difference between a relational database and a semantic repository, a triple/quad store, a whatever-you-call-them?</p> <p>I had last fall a meeting at CWI with Martin Kersten, Peter Boncz and Lefteris Sidirourgos from CWI, and Frank van Harmelen and Spiros Kotoulas of VU Amsterdam, to start a dialogue between semanticists and databasers. Here we were with many more people trying to discover what the case might be. What are the differences?</p> <p>Michael <a href="http://dbpedia.org/resource/Michael_Stonebraker" id="link-id0x1da55730">Stonebraker</a> and Martin Kersten have basically said that what is sauce for the goose is sauce for the gander, and that there is no real difference between relational DB and <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1d828310">RDF</a> storage, except maybe for a little tuning in some data structures or parameters. Semantic repository implementors on the other hand say that when they tried putting triples inside an RDB it worked so poorly that they did everything from scratch. (It is a geekly penchant to do things from scratch, but then this is not always unjustified.)</p> <p> <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id0x1cf1e620">OpenLink Software</a> and <a href="http://virtuoso.openlinksw.com" id="link-id0x1cfbc1d8">Virtuoso</a> are in agreement with both sides, contradictory as this might sound. We took our <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1e1f6a20">RDBMS</a> and added data types and structures and cost model alterations to an existing platform. Oracle did the same. MonetDB considers doing this and time will tell the extent of their RDF-oriented alterations. Right now the estimate is that this will be small and not in the kernel.</p> <p>I would say with confidence that without source code access to the RDB, RDF will not be particularly convenient or efficient to accommodate. With source access, we found that what serves RDB also serves RDF. For example, execution engine and data compression considerations are the same, with minimal tweaks for RDF's run time typing needs.</p> <p>So now we are founding a platform for continuing this discussion. There will be workshops and calls for papers and the beginnings of a research community.</p> <p>After the initial meeting at CWI, I tried to figure what the difference was between the databaser and semanticist minds. Really, the things are close but there is still a disconnect. Database is about big sets and semantics is about individuals, maybe. The databaser discovers that the operation on each member of the set is not always the same, and the semanticist discovers that the operation on each member of the set is often the same.</p> <p>So the semanticist says that big joins take time. The databaser tells the semanticist not to repeat what's been obvious for 40 years and for which there is anything from partitioned hashes to merges to various vectored execution models. Not to mention columns.</p> <p>Spiros of VU Amsterdam/LarKC says that map-reduce materializes inferential closure really fast. Lefteris of CWI says that while he is not a semantic person, he does not understand what the point of all this materializing is, nobody is asking the question, right? So why answer? I say that computing inferential closure is a semanticist tradition; this is just what they do. Atanas Kiryakov of Ontotext says that this is not just a tradition whose start and justification is in the forgotten mists of history, but actually a clear and present need; just look at all the joining you would need.</p> <p>Michael Witbrock of CYC says that it is not about forward or backward inference on toy rule sets, but that both will be needed and on massively bigger rule sets at that. Further, there can be machine learning to direct the inference, doing the meta-reasoning merged with the reasoning itself.</p> <p>I say that there is nothing wrong with materialization if it is guided by need, in the vein of memo-ization or cracking or recycling as is done in MonetDB. Do the work when it is needed, and do not do it again.</p> <p>Brian Thompson of Systap/Bigdata asks whether it is not a contradiction in terms to both want pluggability and merging inference into the data, like LarKC would be doing. I say that this is difficult but not impossible and that when you run joins in a cluster database, as you decide based on the data where the next join step will be, so it will be with inference. Right there, between join steps, integrated with whatever data partitioning logic you have, for partitioning you <i>will</i> have, data being bigger and bigger. And if you have reuse of intermediates and demand driven indexing <i>à la</i> MonetDB, this too integrates and applies to inference results.</p> <p>So then, LarKC and CYC, can you picture a pluggable inference interface at this level of granularity? So far, I have received some more detail as to the needs of inference and database integration, essentially validating our previous intuitions and plans.</p> <p>Aside talking of inference, we have the more immediate issue of creating an industry out of the semantic data management offerings of today.</p> <p>What do we need for this? We need close-to-parity with relational — doing your warehouse in RDF with the attendant agility thereof can't cost 10x more to deploy than the equivalent relational solution.</p> <p>We also want to tell the key-value, anti-<a href="http://dbpedia.org/resource/SQL" id="link-id0x172e8c80">SQL</a> people, who throw away transactions and queries, that there is a better way. And for this, we need to improve our gig just a little bit. Then you have the union of some level of <a href="http://dbpedia.org/resource/ACID" id="link-id0x1e0de2e8">ACID</a>, at least consistent read, availability, complex query, large scale.</p> <p>And to do this, we need a benchmark. It needs a differentiation of online queries and browsing and analytics, graph algorithms and such. We are getting there. We will soon propose a social web benchmark for RDF which has both online and analytical aspects, a data generator, a test driver, and so on, with a <a href="http://www.tpc.org/" id="link-id0x1e3cb130">TPC</a>-style set of rules. If there is agreement on this, we will all get a few times faster. At this point, RDF will be a lot more competitive with mainstream and we will cross another qualitative threshold. </p>
2010-03-22T12:34:40.000010-04:00
SemData@Sofia Roundtable write-up
http://www.openlinksw.com:443/weblog/oerling/?date=2010-03-15#1614
2010-03-15T14:46:57Z
<p>There was last week an <a href="http://www.semdata.org/" id="link-id11a83cf98">invitation-based roundtable</a> about semantic <a href="http://dbpedia.org/resource/Data" id="link-id0x1d5ae638">data</a> management in <a href="http://www.dbpedia.org/resource/Sofia" id="link-id0x1c147340">Sofia, Bulgaria</a>.</p> <p>Lots of smart people together. The meeting was hosted by <a href="http://dbpedia.org/resource/Ontotext" id="link-id0x1c77a6e8">Ontotext</a> and chaired by <a href="http://www.dbpedia.org/resource/Dieter_Fensel" id="link-id0x1e64f350">Dieter Fensel</a>. On the database side we had Ontotext, <a href="http://www.systap.com/" id="link-id0x1cc261c8">SYSTAP</a> (<a href="http://www.systap.com/bigdata.htm" id="link-id0x1dad5348">Bigdata</a>), <a href="http://dbpedia.org/resource/National_Research_Institute_for_Mathematics_and_Computer_Science" id="link-id0x1d3b68f8">CWI</a> (<a href="http://dbpedia.org/resource/MonetDB" id="link-id0x1dba4028">MonetDB</a>), <a href="http://www.dbpedia.org/resource/Karlsruhe_Institute_of_Technology" id="link-id0x1a01f668">Karlsruhe Institute of Technology</a> (YARS2/<a href="http://swse.deri.ie/" id="link-id0x1ceeed50">SWSE</a>). <a href="http://www.larkc.eu/" id="link-id0x1e650c98">LarKC</a> was well represented, being our hosts, with STI, Ontotext, CYC, and <a href="http://www.vu.nl/" id="link-id0x1ca044f0">VU Amsterdam</a>. Notable absences were <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x1b9e5418">Oracle</a>, <a href="http://freebase.com/guid/9202a8c04000641f8000000005c908d6" id="link-id0x1e55adc8">Garlik</a>, <a href="http://semanticweb.org/id/Franz_Inc" id="link-id0x1cf1d4b8">Franz</a>, and <a href="http://www.talis.com/" id="link-id0x1cbb8740">Talis</a>.</p> <p>Now of semantic data management... What is the difference between a relational database and a semantic repository, a triple/quad store, a whatever-you-call-them?</p> <p>I had last fall a meeting at CWI with Martin Kersten, Peter Boncz and Lefteris Sidirourgos from CWI, and Frank van Harmelen and Spiros Kotoulas of VU Amsterdam, to start a dialogue between semanticists and databasers. Here we were with many more people trying to discover what the case might be. What are the differences?</p> <p>Michael <a href="http://dbpedia.org/resource/Michael_Stonebraker" id="link-id0x1e7b3080">Stonebraker</a> and Martin Kersten have basically said that what is sauce for the goose is sauce for the gander, and that there is no real difference between relational DB and <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1dba61f8">RDF</a> storage, except maybe for a little tuning in some data structures or parameters. Semantic repository implementors on the other hand say that when they tried putting triples inside an RDB it worked so poorly that they did everything from scratch. (It is a geekly penchant to do things from scratch, but then this is not always unjustified.)</p> <p> <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id0x1cf45d00">OpenLink Software</a> and <a href="http://virtuoso.openlinksw.com" id="link-id0x1d8b3ac0">Virtuoso</a> are in agreement with both sides, contradictory as this might sound. We took our <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1d51e110">RDBMS</a> and added data types and structures and cost model alterations to an existing platform. Oracle did the same. MonetDB considers doing this and time will tell the extent of their RDF-oriented alterations. Right now the estimate is that this will be small and not in the kernel.</p> <p>I would say with confidence that without source code access to the RDB, RDF will not be particularly convenient or efficient to accommodate. With source access, we found that what serves RDB also serves RDF. For example, execution engine and data compression considerations are the same, with minimal tweaks for RDF's run time typing needs.</p> <p>So now we are founding a platform for continuing this discussion. There will be workshops and calls for papers and the beginnings of a research community.</p> <p>After the initial meeting at CWI, I tried to figure what the difference was between the databaser and semanticist minds. Really, the things are close but there is still a disconnect. Database is about big sets and semantics is about individuals, maybe. The databaser discovers that the operation on each member of the set is not always the same, and the semanticist discovers that the operation on each member of the set is often the same.</p> <p>So the semanticist says that big joins take time. The databaser tells the semanticist not to repeat what's been obvious for 40 years and for which there is anything from partitioned hashes to merges to various vectored execution models. Not to mention columns.</p> <p>Spiros of VU Amsterdam/LarKC says that map-reduce materializes inferential closure really fast. Lefteris of CWI says that while he is not a semantic person, he does not understand what the point of all this materializing is, nobody is asking the question, right? So why answer? I say that computing inferential closure is a semanticist tradition; this is just what they do. Atanas Kiryakov of Ontotext says that this is not just a tradition whose start and justification is in the forgotten mists of history, but actually a clear and present need; just look at all the joining you would need.</p> <p>Michael Witbrock of CYC says that it is not about forward or backward inference on toy rule sets, but that both will be needed and on massively bigger rule sets at that. Further, there can be machine learning to direct the inference, doing the meta-reasoning merged with the reasoning itself.</p> <p>I say that there is nothing wrong with materialization if it is guided by need, in the vein of memo-ization or cracking or recycling as is done in MonetDB. Do the work when it is needed, and do not do it again.</p> <p>Brian Thompson of Systap/Bigdata asks whether it is not a contradiction in terms to both want pluggability and merging inference into the data, like LarKC would be doing. I say that this is difficult but not impossible and that when you run joins in a cluster database, as you decide based on the data where the next join step will be, so it will be with inference. Right there, between join steps, integrated with whatever data partitioning logic you have, for partitioning you <i>will</i> have, data being bigger and bigger. And if you have reuse of intermediates and demand driven indexing <i>à la</i> MonetDB, this too integrates and applies to inference results.</p> <p>So then, LarKC and CYC, can you picture a pluggable inference interface at this level of granularity? So far, I have received some more detail as to the needs of inference and database integration, essentially validating our previous intuitions and plans.</p> <p>Aside talking of inference, we have the more immediate issue of creating an industry out of the semantic data management offerings of today.</p> <p>What do we need for this? We need close-to-parity with relational — doing your warehouse in RDF with the attendant agility thereof can't cost 10x more to deploy than the equivalent relational solution.</p> <p>We also want to tell the key-value, anti-<a href="http://dbpedia.org/resource/SQL" id="link-id0x1cbeaf70">SQL</a> people, who throw away transactions and queries, that there is a better way. And for this, we need to improve our gig just a little bit. Then you have the union of some level of <a href="http://dbpedia.org/resource/ACID" id="link-id0x1e11fbd0">ACID</a>, at least consistent read, availability, complex query, large scale.</p> <p>And to do this, we need a benchmark. It needs a differentiation of online queries and browsing and analytics, graph algorithms and such. We are getting there. We will soon propose a social web benchmark for RDF which has both online and analytical aspects, a data generator, a test driver, and so on, with a <a href="http://www.tpc.org/" id="link-id0x1a950de0">TPC</a>-style set of rules. If there is agreement on this, we will all get a few times faster. At this point, RDF will be a lot more competitive with mainstream and we will cross another qualitative threshold. </p>
2015-08-20T17:29:59.615038-04:00
URIBurner: Painless Generation & Exploitation of Linked Data (Update 1 - Demo Links Added)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-03-10#1613
2010-03-10T17:52:03Z
<h2>What is URIBurner? </h2> <p>A service from <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id11a8a2768">OpenLink Software</a>, available at: <a href="http://uriburner.com" id="link-id11ace9988">http://uriburner.com</a>, that enables anyone to generate structured descriptions -on the fly- for resources that are already published to HTTP based networks. These descriptions exist as hypermedia resource representations where links are used to identify: </p> <ul> <li> the <a href="http://dbpedia.org/resource/Entity" id="link-id11ae10768">entity</a> (<a href="http://dbpedia.org/resource/Data">data</a> object or datum) being described,</li> <li>each of its attributes, and</li> <li>each of its attributes values (optionally).</li> </ul> <p>The hypermedia resource representation outlined above is what is commonly known as an <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id121aec368">Entity</a>-Attribute-Value (EAV) Graph. The use of generic HTTP scheme based Identifiers is what distinguishes this type of hypermedia resource from others.</p> <h2>Why is it Important?</h2> <p> The virtues (dual pronged serendipitous discovery) of publishing HTTP based <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11f5f53e8">Linked Data</a> across public (<a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id11b14e1f8">World Wide Web</a>) or private (Intranets and/or Extranets) is rapidly becoming clearer to everyone. That said, the nuance laced nature of Linked Data publishing presents significant challenges to most. Thus, for Linked Data to really blossom the process of publishing needs to be simplified i.e., "just click and go" (for human interaction) or REST-ful orchestration of HTTP CRUD (Create, Read, Update, Delete) operations between Client Applications and Linked Data Servers.</p> <h2>How Do I Use It?</h2> <p> In similar vane to the role played by FeedBurner with regards to Atom and RSS feed generation, during the early stages of the Blogosphere, it enables anyone to publish Linked Data bearing hypermedia resources on an HTTP network. Thus, its usage covers two profiles: Content Publisher and Content Consumer. </p> <h3> </h3> <h3>Content Publisher </h3> <h3> </h3> <p>The steps that follow cover all you need to do:</p> <ul> <li>place a <link /> <a href="http://dbpedia.org/resource/Tag" id="link-id11a62f908">tag</a> within your HTTP based hypermedia resource (e.g. within section for HTML )</li> <li>use a <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id11e7e5228">URL</a> via the @href attribute value to identify the location of the structured description of your resource, in this case it takes the form: http://linkeddata.uriburner.com/about/id/{scheme-or-protocol}/{your-hostname-or-authority}/{your-local-resource}</li> <li>for human visibility you may consider adding associating a button (as you do with Atom and RSS) with the URL above.</li> </ul> <p> That's it! The discoverability (SDQ) of your content has just multiplied significantly, its structured description is now part of the Linked Data Cloud with a reference back to your site (which is now a bona fide HTTP based Linked Data <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id120a6e5c8">Space</a>).</p> <h4>Examples</h4> <p> <strong>HTML+<a href="http://dbpedia.org/resource/RDFa" id="link-id11ae8fdc8">RDFa</a> based representation of a structured resource description:</strong> </p> <blockquote> <link rel="describedby" title="Resource Description (HTML)"type="text/html" href="http://linkeddata.uriburner.com/about/id/http/example.org/xyz.html"/></blockquote> <p> <strong>JSON based representation of a structured resource description:</strong> </p> <blockquote><link rel="describedby" title="Resource Description (JSON)" type="application/json" href="http://linkeddata.uriburner.com/about/id/http/example.org/xyz.html"/></blockquote> <p> <strong>N3 based representation of a structured resource description:</strong> </p> <blockquote><link rel="describedby" title="Resource Description (N3)" type="text/n3" href="http://linkeddata.uriburner.com/about/id/http/example.org/xyz.html"/></blockquote> <p> <strong>RDF/XML based representations of a structured resource description</strong>: </p> <blockquote><link rel="describedby" title="Resource Description (RDF/XML)" type="application/rdf+xml" href="http://linkeddata.uriburner.com/about/id/http/example.org/xyz.html"/></blockquote> <h3>Content Consumer</h3> <p>As an end-user, obtaining a structured description of any resource published to an HTTP network boils down to the following steps:</p> <ol> <li>go to: http://uriburner.com</li> <li>drag the Page Metadata Bookmarklet link to your Browser's toolbar</li> <li>whenever you encounter a resource of interest (e.g. an HTML page) simply click on the Bookmarklet</li> <li>you will be presented with an HTML representation of a structured resource description (i.e., identifier of the entity being described, its attributes, and its attribute values will be clearly presented).</li> </ol> <h3>Examples</h3> <ul> <li> <a href="http://linkeddata.uriburner.com/about/id/entity/http/www.amazon.com/o/ASIN/1591842778" id="link-id11ba54a48">Description of a Book culled from an Amazon web page</a> </li> <li> <a href="http://linkeddata.uriburner.com/about/id/entity/http/www.bestbuy.com/site/Flip+Video+-+UltraHD+Camcorder+-+Black/Chrome/9281984.p?id=1218073822126&skuId=9281984" id="link-id11f621848">Description of a product offering culled from a BestBuy web page</a> </li> <li> <a href="http://linkeddata.uriburner.com/about/id/entity/http/reviews.cnet.com/digital-cameras/canon-eos-5d-mark/4505-6501_7-33280763.html?tag=tpr" id="link-id115f27e08">Description of a product (a camera) culled from a CNET web page</a> </li> <li> <a href="http://linkeddata.uriburner.com/about/id/entity/http/cgi.ebay.com/23PORT-Canon-SLR-EOS-5D-Mark-II-Body-Only-New_W0QQitemZ140367785136QQcategoryZ31388QQcmdZViewItem#Offer" id="link-id120b4b258">Description of the same CNET product as an Offer on eBay</a> (exposed by the description above via seeAlso property value).</li> </ul> <p>If you are a developer, you can simply perform an HTTP operation request (from your development environment of choice) using any of the URL patterns presented below:</p> <a id="HTML:"> </a><strong>HTML: </strong> <ul> <li> <tt>curl -I -H "Accept: text/html" http://linkeddata.uriburner.com/about/id/{scheme}/{authority}/{local-path} </tt> </li> </ul> <h4> <a id="JSON:"> </a>JSON:</h4> <ul> <li> <tt>curl -I -H "Accept: application/json" http://linkeddata.uriburner.com/about/id/{scheme}/{authority}/{local-path} </tt> </li> <li> <tt>curl http://linkeddata.uriburner.com/about/data/json/{scheme}/{authority}/{local-path}</tt> </li> </ul> <h4> <a id="Notation_3_N3:"> </a>Notation 3 (N3):</h4> <ul> <li> <tt>curl -I -H "Accept: text/n3" http://linkeddata.uriburner.com/about/id/{scheme}/{authority}/{local-path} </tt> </li> <li> <tt>curl http://linkeddata.uriburner.com/about/data/n3/{scheme}/{authority}/{local-path}</tt> </li> </ul> <ul> <li> <tt>curl -I -H "Accept: text/turtle" http://linkeddata.uriburner.com/about/id/{scheme}/{authority}/{local-path}</tt> </li> <li> <tt>curl http://linkeddata.uriburner.com/about/data/ttl/{scheme}/{authority}/{local-path} </tt> </li> </ul> <h4> <a id="RDFXML:"> </a>RDF/XML:</h4> <ul> <li> <tt>curl -I -H "Accept: application/rdf+xml" http://linkeddata.uriburner.com/about/id/{scheme}/{authority}/{local-path} </tt> </li> <li> <tt>curl http://linkeddata.uriburner.com/about/data/xml/{scheme}/{authority}/{local-path} </tt> </li> </ul> <h2>Conclusion</h2> <p>URIBurner is a "deceptively simple" solution for cost-effective exploitation of HTTP based Linked Data meshes. It doesn't require any programming or customization en route to immediately realizing its virtues. </p> <p> If you like what URIBurner offers, but prefer to leverage its capabilities within your domain -- such that resource description URLs reside in your domain, all you have to do is perform the following steps:</p> <ol> <li> <a href="http://download.openlinksw.com/virtwiz/" id="link-id1158f8658">download a copy of Virtuoso</a> (for local desktop, workgroup, or data center installation) or</li> <li>instantiate <a href="http://www.openlinksw.com/dataspace/dav/wiki/Main/VirtInstallationEC2" id="link-id11e03e558">Virtuoso via the Amazon EC2 Cloud</a> </li> <li>enable the Sponger Middleware component via the RDF Mapper VAD package (which includes <a href="http://www.openlinksw.com/dataspace/dav/wiki/Main/VirtSpongerCartridgeSupportedDataSources" id="link-id1205ffe78">cartridges for over 30 different resources types</a>)</li> </ol> <p>When you install your own URIBurner instances, you also have the ability to perform customizations that increase resource description fidelity in line with your specific needs. All you need to do is develop a custom extractor cartridge and/or meta cartridge. </p> <h2>Related:</h2> <ul> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtSponger" id="link-id120582118"> Virtuoso Sponger Middleware</a> -- (technology behind <a href="http://uriburner.com" id="link-id11b634448">URIBurner Service</a>)</li> <li> <a href="http://virtuoso.openlinksw.com/screencasts/virtuoso-rdf-middleware3.swf" id="link-id12082e958">Animation demonstrating how the Virtuoso Sponger works</a>.</li> </ul>
2010-03-11T10:16:34.000003-05:00
Meshups Demonstrating How SPARQL-GEO Enhances Linked Data Exploitation (Update 2)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-03-06#1612
2010-03-06T22:43:49Z
<p>Deceptively simple demonstrations of how <a href="http://virtuoso.openlinksw.com" id="link-id11dfe45b8">Virtuoso</a>'s <a href="http://dbpedia.org/resource/SPARQL" id="link-id11a3d8968">SPARQL</a>-GEO extensions to SPARQL lay critical foundation for Geo Spatial solutions that seek to leverage the burgeoning <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11ae855b8">Linked Data</a>. </p> <h3>Setup <a href="http://dbpedia.org/resource/Information" id="link-id120a6f478">Information</a> </h3> <p>SPARQL Endpoint: <a href="http://lod.openlinksw.com/sparql" id="link-id120401958">Linked Open Data Cache</a> (8.5 Billion+ Quad Store which includes data from Geonames and the <a href="http://dl-learner.org/Projects/LinkedGeoData" id="link-id11b8f31d8">Linked GeoData Project</a> Data Sets) .</p> <h3>Live Linked Data Meshup Links:</h3> <ul> <li> <a href="http://bit.ly/cyJjwo" id="link-id120396168">LinkedGeoData things within 2km ORDER BY Dist LIMIT 10 </a>(Use from <strong>iPhone</strong> only since its an iPhone oriented Linked Data driven application)</li> <li> <a href="http://uriburner.com/isparql/view/?query=PREFIX%20foaf%3A%20%3Chttp%3A%2F%2Fxmlns.com%2Ffoaf%2F0.1%2F%3E%0APREFIX%20lgv%3A%20%3Chttp%3A%2F%2Flinkedgeodata.org%2Fvocabulary%23%3E%0Aconstruct%20%7B%3Fthing%20a%20%3Ftype%3B%0A%20geo%3Ageometry%20%3Fgeo%3B%0A%20foaf%3Aname%20%3Fname%7D%0Awhere%20%7B%0A%3Fthing%20geo%3Ageometry%20%3Fgeo%20.%0A%3Fthing%20lgv%3Aname%20%3Fname%20.%0A%3Fthing%20a%20%3Ftype.%0AFILTER%20%28bif%3Ast_intersects%20%28%3Fgeo%2C%20bif%3Ast_point%20%28-0.128056%2C%2051.508057%29%2C%202%29%29%0A%7D%0ALIMIT%20100&endpoint=http://lod.openlinksw.com/sparql&resultview=map&maxrows=50" id="link-id1209a6f38">LinkedGeoData things within 2km of Trafalgar Square</a> | <a href="http://uriburner.com/isparql/view/?query=PREFIX%20foaf%3A%20%3Chttp%3A%2F%2Fxmlns.com%2Ffoaf%2F0.1%2F%3E%0APREFIX%20lgv%3A%20%3Chttp%3A%2F%2Flinkedgeodata.org%2Fvocabulary%23%3E%0Aconstruct%20%7B%0A%20%3Fthing%20rdfs%3Atype%20%3Ftype%3B%0A%20geo%3Ageometry%20%3Fgeo%3B%0A%20foaf%3Aname%20%3Fname%7D%0Awhere%20%7B%0A%20%3Fthing%20geo%3Ageometry%20%3Fgeo%20.%0A%20%3Fthing%20lgv%3Aname%20%3Fname%20.%0A%20%3Fthing%20a%20%3Ftype.%0AFILTER%20(bif%3Ast_intersects%20(%3Fgeo%2C%20bif%3Ast_point%20(-0.128056%2C%2051.508057)%2C%202))%0A%7D%0Aorder%20by%20asc%20(bif%3Ast_distance%20(%3Fgeo%2C%20bif%3Ast_point%20(-0.128056%2C%2051.508057)))%0ALIMIT%20100&endpoint=http://lod.openlinksw.com/sparql&resultview=map&maxrows=50" id="link-id11ebb07f8">ORDER By Distance - closest first</a> | <a href="http://uriburner.com/isparql/view/?query=PREFIX%20foaf%3A%20%3Chttp%3A%2F%2Fxmlns.com%2Ffoaf%2F0.1%2F%3E%0APREFIX%20lgv%3A%20%3Chttp%3A%2F%2Flinkedgeodata.org%2Fvocabulary%23%3E%0Aconstruct%20%7B%0A%20%3Fthing%20rdfs%3Atype%20%3Ftype%3B%0A%20geo%3Ageometry%20%3Fgeo%3B%0A%20foaf%3Aname%20%3Fname%7D%0Awhere%20%7B%0A%20%3Fthing%20geo%3Ageometry%20%3Fgeo%20.%0A%20%3Fthing%20lgv%3Aname%20%3Fname%20.%0A%20%3Fthing%20a%20%3Ftype.%0AFILTER%20(bif%3Ast_intersects%20(%3Fgeo%2C%20bif%3Ast_point%20(-0.128056%2C%2051.508057)%2C%202))%0A%7D%0Aorder%20by%20desc%20(bif%3Ast_distance%20(%3Fgeo%2C%20bif%3Ast_point%20(-0.128056%2C%2051.508057)))%0ALIMIT%20100&endpoint=http://lod.openlinksw.com/sparql&resultview=map&maxrows=50" id="link-id1207a27e8">ORDER By Distance - most distant first</a> .</li> </ul> <h3>Related</h3> <ul> <li> <a href="http://delicious.com/kidehen/linked_data_demo" id="link-id11ac9a2a8">Collection of Live Linked Data Demos</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1587" id="link-id11aca1d68">Virtuoso's SPARQL-GEO Extensions</a> </li> </ul>
2010-03-24T11:44:24.000002-04:00
Revisiting HTTP based Linked Data (Update 1 - Demo Video Links Added)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-03-04#1611
2010-03-04T15:16:14Z
<p>Motivation for this post arose from a series of Twitter exchanges between <a href="http://ouseful.wordpress.com/about/#this" id="link-id115699ae8">Tony Hirst</a> and I, in relation to his <a href="http://dbpedia.org/resource/Blog" id="link-id11a0cbc08">blog</a> post titled: <a href="http://ouseful.wordpress.com/2010/03/03/so-what-is-it-about-linked-data-that-makes-it-linked-data%e2%84%a2/" id="link-id1158f8ce8">So What Is It About Linked Data that Makes it Linked Data™ ?</a> </p> <p>At the end of the marathon session, it was clear to <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id11557da58">me</a> that a blog post was required for future reference, at the very least :-)</p> <h3>What is <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11a7ee3a8">Linked Data</a>?</h3> <p>"<a href="http://dbpedia.org/resource/Reference_(computer_science)" id="link-id11a682338">Data Access by Reference</a>" mechanism for Data Objects (or Entities) on HTTP networks. It enables you to Identify a Data Object and Access its structured Data Representation via a single Generic HTTP scheme based Identifier (HTTP <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id125037288">URI</a>). Data Object representation formats may vary; but in all cases, they are <a href="http://dbpedia.org/resource/Hypermedia" id="link-id115548f78">hypermedia</a> oriented, fully structured, and negotiable within the <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id11c955888">context</a> of a client-server message exchange.</p> <h3>Why is it Important?</h3> <p> <a href="http://dbpedia.org/resource/Information" id="link-id125154778">Information</a> makes the world tick!</p> <p>Information doesn't exist without data to contextualize.</p> <p>Information is inaccessible without a projection (presentation) medium. </p> <p>All information (without exception, when produced by humans) is subjective. Thus, to truly maximize the innate heterogeneity of collective human intelligence, loose coupling of our information and associated data sources is imperative.</p> <h3>How is Linked Data Delivered?</h3> <p>Linked Data is exposed to HTTP networks (e.g. <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id125321238">World Wide Web</a>) via hypermedia resources bearing structured representations of data object descriptions. Remember, you have a single Identifier abstraction (generic HTTP URI) that embodies: Data Object Name and Data Representation Location (aka <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id1249a7a88">URL</a>).</p> <h3>How are Linked Data Object Representations Structured?</h3> <p>A structured representation of data exists when an <a href="http://dbpedia.org/resource/Entity" id="link-id1250630d8">Entity</a> (Datum), its Attributes, and its Attribute Values are clearly discernible. In the case of a Linked Data Object, structured descriptions take the form of a hypermedia based <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id126ed7608">Entity</a>-Attribute-Value (EAV) graph pictorial -- where each Entity, its Attributes, and its Attribute Values (optionally) are identified using Generic HTTP URIs. </p> <p>Examples of structured data representation formats (content types) associated with Linked Data Objects include:</p> <ul> <li>text/html</li> <li>text/turtle</li> <li>text/n3</li> <li>application/json</li> <li>application/rdf+xml</li> <li>Others </li> </ul> <h3>How Do I Create Linked Data oriented Hypermedia Resources?</h3> <p>You markup resources by expressing distinct entity-attribute-value statements (basically these a 3-tuple records) using a variety of notations:</p> <ul> <li>(X)HTML+<a href="http://dbpedia.org/resource/RDFa" id="link-id1252975b8">RDFa</a>,</li> <li> <a href="http://n2.talis.com/wiki/RDF_JSON_Specification" id="link-id115015458">JSON</a>,</li> <li> <a href="http://www.dajobe.org/2004/01/turtle/" id="link-id116458478">Turtle</a>,</li> <li> <a href="http://www.w3.org/DesignIssues/Notation3" id="link-id11a62f9f8">N3</a>,</li> <li> <a href="http://swdev.nokia.com/trix/trix.html" id="link-id11a8f56b8">TriX</a>,</li> <li> <a href="http://www4.wiwiss.fu-berlin.de/bizer/TriG/" id="link-id117156978">TriG</a>,</li> <li> <a href="http://www.w3.org/TR/REC-rdf-syntax/" id="link-id126f52a58">RDF/XML</a>, and</li> <li>Others (for instance you can use Atom data format extensions to model EAV graph as per OData initiative from Microsoft).</li> </ul> <p>You can achieve this task using any of the following approaches:</p> <ul> <li>Notepad</li> <li>WYSIWYG Editor </li> <li>Transformation of Database Records via Middleware</li> <li>Transformation of XML based <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> Services output via Middleware</li> <li>Transformation of other Hypermedia Resources via Middleware</li> <li>Transformation of non Hypermedia Resources via Middleware</li> <li>Use a platform that delivers all of the above.</li> </ul> <h3>Practical Examples of Linked Data Objects Enable</h3> <ul> <li>Describe Who You Are, What You Offer, and What You Need via your structured profile, then leave your HTTP network to perform the REST (serendipitous discovery of relevant things)</li> <li>Identify (via map overlay) all items of interest based on a 2km+ radious of my current location (this could include vendor offerings or services sought by existing or future customers)</li> <li>Share the latest and greatest family photos with family members *only* without forcing them to signup for Yet Another Web 2.0 service or Social Network</li> <li>No repetitive signup and username and password based login sequences per Web 2.0 or Mobile Application combo</li> <li>Going beyond imprecise Keyword Search to the new frontier of Precision Find - Example, Find Data Objects associated with the keywords: Tiger, while enabling the seeker disambiguate across the "Who", "What", "Where", "When" dimensions (with negation capability)</li> <li>Determine how two Data Objects are Connected - person to person, person to subject matter etc. (LinkedIn outside the walled garden)</li> <li>Use any resource address (e.g <a href="http://dbpedia.org/resource/Blog" id="link-id124fd8118">blog</a> or bookmark URL) as the conduit into a Data Object mesh that exposes all associated Entities and their social network relationships</li> <li>Apply patterns (social dimensions) above to traditional enterprise data sources in combination (optionally) with external data without compromising security etc.</li> </ul> <h3>How Do <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id124fd0d98">OpenLink Software</a> Products Enable Linked Data Exploitation?</h3> <p>Our data access middleware heritage (which spans 16+ years) has enabled us to assemble a rich portfolio of coherently integrated products that enable cost-effective evaluation and utilization of Linked Data, without writing a single line of code, or exposing you to the hidden, but extensive admin and configuration costs. Post installation, the benefits of Linked Data simply materialize (along the lines described above).</p> <p>Our main Linked Data oriented products include:</p> <ul> <li> <a href="http://ode.openlinksw.com" id="link-id125058d68">OpenLink Data Explorer</a> -- visualizes Linked Data or Linked Data transformed "on the fly" from hypermedia and non hypermedia data sources </li> <li> <a href="http://uriburner.com" id="link-id1251db6a8">URIBurner</a> -- a "deceptively simple" solution that enables the generation of Linked Data "on the fly" from a broad collection of data sources and resource types</li> <li> <a href="http://ods.openlinksw.com/wiki/ODS/" id="link-id1252caae8">OpenLink Data Spaces</a> -- a platform for enterprises and individuals that enhances distributed collaboration via Linked Data driven virtualization of data across its native and/or 3rd party content manager for: Blogs, Wikis, Shared Bookmarks, Discussion Forums, Social Networks etc</li> <li> <a href="http://virtuoso.openlinksw.com/overview/index.htm" id="link-id124809b58">OpenLink Virtuoso</a> -- a secure and high-performance native hybrid data server (Relational, RDF-Graph, Document models) that includes in-built Linked Data transformation middleware (aka. Sponger). </li> </ul> <h3>Related</h3> <ul> <li> <a href="http://www.ietf.org/rfc/rfc2616.txt" id="link-id125306d78">Hypertext Transfer Protocol 1.1 RFC</a> </li> <li> <a href="http://www.odata.org/docs/%5BMC-APDSU%5D.htm#_Toc246716495" id="link-id11c948e98">Open Data Protocol Glossary</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1543" id="link-id126fae278">Simple Explanation of RDF and Linked Data Dynamics</a> </li> <li> <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=linked%20data%0D%0A&type=text&output=html" id="link-id1252e0018">Collection of post from the past about Linked Data</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1584" id="link-id124fefea8">Are We There Yet Re. Web++?</a> -- includes link to <a href="http://itc.conversationsnetwork.org/shows/detail4233.html" id="link-id125188078">podcast conversation with Jon Udell</a> </li> <li> <a href="http://www.ted.com/talks/gary_flake_is_pivot_a_turning_point_for_web_exploration.html" id="link-id11a501c28">Web of Linked Data Pivoting Demo from TED</a> -- by Microsoft's Gary Flake </li> <li> <a href="http://www.youtube.com/watch?v=G29DBIEcIuQ" id="link-id1204fff18">Microsoft Pivot atop Virtuoso Quad Store's Faceted Browser Engine</a>-- My Demonstration of EAV model transcending data representation variations (i.e., RDF's EAV data model data served up in Microsoft CXML data representation format). </li> </ul>
2010-03-08T09:59:37.000010-05:00
Linked Data & Socially Enhanced Collaboration (Enterprise or Individual) -- Update 1
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-03-02#1610
2010-03-02T20:47:54Z
<p>Socially enhanced enterprise and invididual collaboration is becoming a focal point for a variety of solutions that offer erswhile distinct content managment features across the realms of Blogging, Wikis, Shared Bookmarks, Discussion Forums etc.. as part of an integrated platform suite. Recently, <a href="http://www.socialtext.com/" id="link-id112be850">Socialtext</a> has caught my attention courtesy of its nice <a href="http://www.socialtext.com/products/socialnetworking.php" id="link-id145d9850">features and benefits page</a> . In addition, I've also found the <a href="http://mike2.openmethodology.org/" id="link-id14103cc8">Mike 2.0 portal</a> immensely interesting and valuable, for those with an enterprise collaboration bent.</p> <p>Anyway, Socialtext and Mike 2.0 (they aren't identical and juxtaposition isn't seeking to imply this) provide nice demonstrations of socially enhanced collaboration for individuals and/or enterprises is all about:</p> <ol> <li>Identifying Yourself</li> <li>Identifying Others (key contributors, peers, collaborators)</li> <li>Serendipitous Discovery of key contributors, peers, and collaborators</li> <li>Serendipitous Discovery by key contributors, peers, and collaborators</li> <li>Develop and sustain relationships via socially enhanced professional network hybrid</li> <li>Utilize your new "trusted network" (which you've personally indexed) when seeking help or propagating a <a href="http://dbpedia.org/resource/Meme" id="link-id13ad00d0">meme</a>.</li> </ol> <p>As is typically the case in this emerging realm, the critical issue of discrete "identifiers" (record keys in sense) for <a href="http://dbpedia.org/resource/Data">data</a> items, data containers, and data creators (individuals and groups) is overlooked albeit unintentionally. </p> <h3>How HTTP based <a href="http://dbpedia.org/resource/Linked_Data" id="link-id112e1ba8">Linked Data</a> Addresses the Identifier Issue</h3> <p>Rather than using platform constrained identifiers such as: </p> <ul> <li>email address (a "mailto" scheme identifier), </li> <li>a dbms user account, </li> <li>application specific account, or</li> <li>OpenID.</li> </ul> <p>It enables you to leverage the platform independence of HTTP scheme Identifiers (Generic URIs) such that Identifiers for: </p> <ol> <li>You, </li> <li>Your Peers, </li> <li>Your Groups, and </li> <li>Your Activity Generated Data, </li> </ol> <p>simply become conduits into a mesh of <a href="http://virtuoso.openlinksw.com/presentations/Creating_Deploying_Exploiting_Linked_Data2/images/My_Data_Spaces.png" id="link-id13fe1168">HTTP -- referencable and accessible -- Linked Data Objects</a> endowed with High SDQ (Serendipitious Discovery Quotient). For example my <a href="http://kingsley.idehen.name/dataspace/person/kidehen#this" id="link-id13bdcc80">Personal WebID </a>is all anyone needs to know if they want to explore:</p> <ol> <li>My Profile (which includes references to data objects associated with my interests, social-network, calendar, bookmarks etc.)</li> <li>Data generated by my activities across various data spaces (via data objects associated with my online accounts e.g. <a href="http://linkeddata.uriburner.com/about/id/entity/http/feeds.delicious.com/v2/rss/kidehen?count=15" id="link-id141cce38">Del.icio.us</a>, <a href="http://linkeddata.uriburner.com/about/id/entity/http/twitter.com/kidehen" id="link-id11802ce8">Twitter</a>, <a href="http://linkeddata.uriburner.com/about/id/entity/http/www.last.fm/user/kidehen" id="link-id118bf470">Last.FM</a>)</li> <li> <a href="http://uriburner.com/fct/rdfdesc/usage.vsp?g=http%3A%2F%2Fkingsley.idehen.name%2Fdataspace%2Fperson%2Fkidehen%23this&tp=4" id="link-id13c0f528">Linked Data Meshups via URIBurner</a> (or any other <a href="http://virtuoso.openlinksw.com" id="link-id11334f00">Virtuoso</a> instance) that provide an extend view of my profile</li> </ol> <h3>How <a href="http://dbpedia.org/resource/Friend_of_a_friend" id="link-id14324eb0">FOAF</a>+SSL adds Socially aware Security </h3> <p>Even when you reach a point of equilibrium where: your daily activities trigger orchestratestration of CRUD (Create, Read, Update, Delete) operations against Linked Data Objects within your socially enhanced collaboration network, you still have to deal with the thorny issues of security, that includes the following: </p> <ol> <li>Single Sign On, </li> <li>Authentication, and </li> <li>Data Access Policies.</li> </ol> <p>FOAF+SSL, an application of HTTP based Linked Data, enables you to enhance your Personal HTTP scheme based Identifer (or WebID) via the following steps (peformed by a FOAF+SSL compliant platform):</p> <ol> <li>Imprint WebID within a self-signed x.509 based public key (certificate) associated with your private key (generated by FOAF+SSL platform or manually via OpenSSL)</li> <li>Store public key components (modulous and exponent) into your FOAF based profile document which references your Personal HTTP Identifier as its primary topic</li> <li>Leverage HTTP <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id141f8b30">URL</a> component of WebID for making public key components (modulous and exponent) available for x.509 certificate based authentication challenges posed by systems secured by FOAF+SSL (directly) or OpenID (indirectly via FOAF+SSL to OpenID proxy services).</li> </ol> <p>Contrary to conventional experiences with all things PKI (Public Key Infrastructure) related, FOAF+SSL compliant platforms typically handle the PKI issues as part of the protocol implementation; thereby protecting you from any administrative tedium without compromising security.</p> <h3>Conclusions</h3> <p>Understanding how new technology innovations address long standing problems, or understanding how new solutions inadvertently fail to address old problems, provides time tested mechanisms for product selection and value proposition comprehension that ultimately save scarce resources such as time and money. </p> <p>If you want to understand real world problem solution #1 with regards to HTTP based Linked Data look no further than the issues of secure, socially aware, and platform independent identifiers for data objects, that build bridges across erstwhile data silos.</p> <p>If you want to cost-effectively experience what I've outlined in this post, take a look at <a href="http://ods.openlinksw.com/wiki/ODS/" id="link-id13c21220">OpenLink Data Spaces</a> (<a href="http://ods.openlinksw.com/wiki/ODS/" id="link-id1422cdd8">ODS</a>) which is a distributed collaboration engine (enterprise of individual) built around the <a href="http://virtuoso.openlinksw.com" id="link-id14211c98">Virtuoso</a> database engines. It simply enhances existing collaboration tools via the following capabilities:</p> <p>Addition of Social Dimensions via HTTP based <a href="http://dbpedia.org/resource/Identity_(object-oriented_programming)" id="link-id116ecd88">Data Object Identifiers</a> for all Data Items (if missing)</p> <ol> <li>Ability to integrate across a myriad of Data Source Types rather than a select few across RDBM Engines, LDAP, <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> Services, and various HTTP accessible Resources (Hypermedia or Non Hypermedia content types)</li> <li>Addition of FOAF+SSL based authentication</li> <li>Addition of FOAF+SSL based Access Control Lists (ACLs) for policy based data access.</li> </ol> <h3>Related:</h3> <ul> <li> <a href="http://ods.openlinksw.com/dataspace/dav/wiki/ODS/GetAPersonalURIIn5MinutesOrLess" id="link-id117b2610">Get Yourself A WebID in 5 Minutes or Less</a> via OpenLink Data Spaces (an application layer built atop Virtuoso)</li> <li> <a href="http://ods.openlinksw.com/dataspace/dav/wiki/ODS/ODSBriefcaseFOAFSSL" id="link-id140311a0">How To Share Resources Securely Using FOAF+SSL</a> </li> <li> <a href="http://www.youtube.com/watch?v=CRbdeNMPCug" id="link-id11ad5448">FOAF+SSL & WebID Demonstration</a> </li> <li> <a href="http://www.slideshare.net/kidehen/linked-data-spaces-data-portability-access" id="link-id141f43a8">OpenLink Data Spaces & Data Portability</a>.</li> </ul>
2010-03-03T19:50:37-05:00
OpenLink Virtuoso - Product Value Proposition Overiew
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-02-26#1609
2010-02-26T19:12:32Z
<h2>Situation Analysis</h2> <p>Since the beginning of the modern IT era, each period of innovation has inadvertently introduced its fair share of <a href="http://dbpedia.org/resource/Data">Data</a> Silos. The driving force behind this anomaly remains an overemphasis on the role of applications when selecting problem solutions. Unfortunately, most solution selecting decision makers remain oblivious to the fact that most applications are architecturally monolithic; i.e., they fail to separate the following five layers that are critical to all solutions: </p> <ol> <li>Data Unit (Datum or Data Object) Identity,</li> <li>Data Storage/Persistence,</li> <li>Data Access,</li> <li>Data Representation, and</li> <li>Data Presentation/Visualization. </li> </ol> <p>The rise of the <a href="http://dbpedia.org/resource/Internet" id="link-id13fe21b0">Internet</a>, and its exponentially-growing user-friendly enclave known as the <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id1233c608">World Wide Web</a>, is bringing the intrinsic costs of the monolithic application architecture anomaly to bear -- in manners unanticipated by many. For example, the emergence of network-oriented solutions across the realms of Enterprise 2.0-based Collaboration and <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> 2.0-based Software-as-a-Service (SaaS), combined with the overarching influence of Social Media, are producing more heterogeneously-structured and disparately-located data sources than people can effectively process.</p> <p>As is often the case, a variety of problem and product monikers have emerged for the data access and integration challenges outlined above. Contemporary examples include Enterprise <a href="http://dbpedia.org/resource/Information" id="link-id13f7e458">Information</a> Integration, Master Data Management, and <a href="http://dbpedia.org/resource/Federated_database_system" id="link-id13f57da0">Data Virtualization</a>. Labeling aside, the fundamental issues of the unresolved Data Integration challenge boil down to the following:</p> <ul> <li>Data Model Heterogeneity</li> <li>Data Quality (Cleanliness)</li> <li>Semantic Variance across Contexts (e.g., weights and measures).</li> </ul> <p>Effectively solving today's data integration challenges requires a move away from monolithic application architecture to loosely-coupled, network-centric application architectures. Basically, we need a ubiquitous network-centric application protocol that lends itself to loosely-coupled across-the-wire orchestration of data interactions. In short, this will be what revitalizes the art of application development and deployment.</p> <p>The World Wide Web is built around a network application protocol called HTTP. This protocol intrinsically separates the five layers listed earlier, thereby enabling:</p> <ul> <li>Use of Generic HTTP URIs as Data Object (<a href="http://dbpedia.org/resource/Entity" id="link-id113b7318">Entity</a>) Identifiers;</li> <li>Identifier Co-reference, such that multiple <a href="http://dbpedia.org/resource/Identity_(object-oriented_programming)" id="link-id117151d8">Data Object Identifiers</a> may reference the same Data Object;</li> <li>Use of the <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id13fa4fa0">Entity</a>-Attribute-Value Model to describe Data Objects using real world modeling friendly conceptual graphs;</li> <li>Use of HTTP URLs to Identify Locations of Resources that bear (host) Data Object Descriptions (Representations);</li> <li>Data Access mechanism for retrieving Data Object Representations from persistent or transient storage locations.</li> </ul> <h2>What is <a href="http://virtuoso.openlinksw.com" id="link-id116af950">Virtuoso</a>?</h2> <p>A uniquely designed to address today's escalating Data Access and Integration challenges without compromising performance, security, or platform independence. At its core lies an unrivaled commitment to industry standards combined with unique technology innovation that transcends erstwhile distinct realms such as: </p> <ul> <li>Data Management (<a href="http://blogs.usnet.private:8893/main/rdbms-engine.html" id="link-id11943dc0">Relational</a>, <a href="http://blogs.usnet.private:8893/main/rdf-quad-store.html" id="link-id12312240">RDF Graph</a>, or Document), </li> <li> <a href="http://blogs.usnet.private:8893/main/middleware.htm" id="link-id115d71c0">Data Access Middleware</a>, </li> <li> <a href="http://blogs.usnet.private:8893/main/web-application-server.html" id="link-id142ca788">Web Application & Services Deployment</a>, </li> <li> <a href="http://blogs.usnet.private:8893/main/linked-data.html" id="link-id112b92c0">Linked Data Deployment</a>, and </li> <li>Messaging. </li> </ul> <p>When Virtuoso is installed and running, HTTP-based Data Objects are automatically created as a by-product of its powerful data virtualization, transcending data sources and data representation formats. The benefits of such power extend across profiles such as:</p> <ul> <li> <a href="http://blogs.usnet.private:8893/information-and-knowledge-worker-benefits" id="link-id118df198">Information & Knowledge Workers</a>, </li> <li> <a href="http://blogs.usnet.private:8893/systems-integrator-benefits" id="link-id1429d178">Systems Integrators & Architects</a>, </li> <li> <a href="http://blogs.usnet.private:8893/distributed-collaboration-benefits" id="link-id142fa2a0">Distributed Collaboration & Social Media</a>, </li> <li> <a href="http://blogs.usnet.private:8893/cloud-computing-benefits" id="link-id11aee6b0">Cloud Computing</a>, and </li> <li> <a href="http://blogs.usnet.private:8893/application-developer-benefits" id="link-id142440b8">Application Development</a>. </li> </ul> <h2>Product Benefits Summary</h2> <ul> <li> <b>Enterprise Agility</b> — Virtuoso lets you mix-&-match best-of-class combinations of Operating Systems, Programming Environments, Database Engines and Data-Access Middleware when building or tweaking your IS infrastructure, without the typical impedance of vendor-lock-in.</li> <li> <b>Data Model Dexterity</b> — By supporting multiple protocols and data models in a single product, Virtuoso protects you against costly vulnerabilities such as: perennial acquisition and accumulation of expensive data model specific DBMS products that still operate on the fundamental principle of: proprietary technology lock-in, at a time when heterogeneity continues to intrinsically define the information technology landscape.</li> <li> <b>Cost-effectiveness</b> — By providing a single point of access (and single-sign-on, SSO) to a plethora of Web 2.0-style social networks, Web Services, and Content Management Systems, and by using Data Object Identifiers as units of Data Virtualization that become the focal points of all data access, Virtuoso lowers the cost to exploit emerging frontiers such as socially-enhanced enterprise collaboration.</li> <li> <b>Speed of Exploitation</b> — Virtuoso provides the ability to rapidly assemble 360-degree conceptual views of data, across internal line-of-business application (CRM, ERP, ECM, HR, etc.) data and/or external data sources, whether these are unstructured, semi-structured, or fully structured.</li> </ul> <p>Bottom line, Virtuoso delivers unrivaled flexibility and scalability, without compromising performance or security.</p> <h2>Related</h2> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com's BLOG [127]/1567" id="link-id13ee6840">HTTP URI Abstraction and Linked Data</a> </li> <li> <a href="http://blog.jonudell.net/2009/09/09/talking-with-kingsley-idehen-about-mastering-your-own-search-index/" id="link-id1428b698">Be The Master of Your Own Search Index</a> </li> <li> <a href="http://walkingoncoals.blogspot.com/2009/12/whos-data-is-it-part-1.html" id="link-id117db508">Who's Data Is It?</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1482" id="link-id13f64d90">MDM & Linked Data</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1453" id="link-id118861d8">What is Linked Data Oriented RDF-zation?</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1444" id="link-id11820d70">Semantic Web: Travails to Harmony Illustrated</a> </li> </ul> <p> </p>
2010-02-27T12:46:36-05:00
Re-introducing the Virtuoso Virtual Database Engine
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-02-17#1608
2010-02-17T21:38:01Z
<p>In recent times a lot of the commentary and focus re. <a href="http://virtuoso.openlinksw.com" id="link-id16a22f48">Virtuoso</a> has centered on the RDF Quad Store and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id112d82a0">Linked Data</a>. What sometimes gets overlooked is the sophisticated <a href="http://dbpedia.org/resource/Virtual_Database" id="link-id6493cc8">Virtual Database</a> Engine that provides the foundation for all of Virtuoso's <a href="http://dbpedia.org/resource/Data">data</a> integration capabilities.</p> <p>In this post I provide a brief re-introduction to this essential aspect of Virtuoso.</p> <h3>What is it?</h3> <p>This component of Virtuoso is known as the Virtual Database Engine (VDBMS). It provides transparent high-performance and secure access to disparate data sources that are external to Virtuoso. It enables federated access and integration of data hosted by any <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id13c26008">ODBC</a>- or <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id166604c0">JDBC</a>-accessible <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id139dfdb8">RDBMS</a>, RDF Store, XML database, or Document (Free Text)-oriented Content Management System. In addition, it facilitates integration with <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> Services (SOAP-based SOA RPCs or REST-fully accessible Web Resources). </p> <h3>Why is it important?</h3> <p>In the most basic sense, you shouldn't need to upgrade your existing database engine version simply because your current DBMS and Data Access Driver combo isn't compatible with ODBC-compliant desktop tools such as Microsoft Access, Crystal Reports, BusinessObjects, Impromptu, or other of ODBC, JDBC, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id13c7ceb8">ADO</a>.NET, or OLE DB-compliant applications. Simply place Virtuoso in front of your so-called "legacy database," and let it deliver the compliance levels sought by these tools</p> <p>In addition, it's important to note that today's enterprise, through application evolution, company mergers, or acquisitions, is often faced with disparately-structured data residing in any number of line-of-business-oriented data silos. Compounding the problem is the exponential growth of user-generated data via new social media-oriented collaboration tools and platforms. For companies to cost-effectively harness the opportunities accorded by the increasing intersection between line-of-business applications and social media, virtualization of data silos must be achieved, and this virtualization must be delivered in a manner that doesn't prohibitively compromise performance or completely undermine security at either the enterprise or personal level. Again, this is what you get by simply installing Virtuoso.</p> <h3>How do I use it?</h3> <p>The VDBMS may be used in a variety of ways, depending on the data access and integration task at hand. Examples include: </p> <h4>Relational Database Federation</h4> <p>You can make a single ODBC, JDBC, ADO.NET, OLE DB, or XMLA connection to multiple ODBC- or JDBC-accessible RDBMS data sources, concurrently, with the ability to perform intelligent distributed joins against externally-hosted database tables. For instance, you can join internal human resources data against internal sales and external stock market data, even when the HR team uses <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id16706720">Oracle</a>, the Sales team uses <a href="http://dbpedia.org/resource/IBM_Informix" id="link-ide5a15c8">Informix</a>, and the Stock Market figures come from <a href="http://dbpedia.org/resource/Ingres" id="link-id13c0e138">Ingres</a>!</p> <h4>Conceptual Level Data Access using the RDF Model</h4> <p>You can construct RDF Model-based Conceptual Views atop Relational Data Sources. This is about generating HTTP-based <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id115150f8">Entity</a>-Attribute-Value (E-A-V) graphs using data culled "on the fly" from native or external data sources (Relational Tables/Views, XML-based Web Services, or User Defined Types).</p> <p>You can also derive RDF Model-based Conceptual Views from Web Resource transformations "on the fly" -- the Virtuoso <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id1675db50">Sponger</a> (RDFizing middleware component) enables you to generate RDF Model Linked Data via a RESTful Web Service or within the process pipeline of the <a href="http://dbpedia.org/resource/SPARQL" id="link-id166b8d90">SPARQL</a> query engine (i.e., you simply use the <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id167d00c8">URL</a> of a Web Resource in the FROM clause of a SPARQL query).</p> <p>It's important to note that Views take the form of HTTP links that serve as both Data Source Names and Data Source Addresses. This enables you to query and explore relationships across entities (i.e., People, Places, and other Real World Things) via HTTP clients (e.g., Web Browsers) or directly via SPARQL Query Language constructs transmitted over HTTP.</p> <h4>Conceptual Level Data Access using ADO.NET <a href="http://dbpedia.org/resource/Entity" id="link-id13c6bb60">Entity</a> <a href="http://dbpedia.org/resource/ADO.NET_Entity_Framework" id="link-id16ad3f68">Frameworks</a> </h4> <p>As an alternative to RDF, Virtuoso can expose ADO.NET Entity Frameworks-based Conceptual Views over Relational Data Sources. It achieves this by generating Entity Relationship graphs via its native ADO.NET Provider, exposing all externally attached ODBC- and JDBC-accessible data sources. In addition, the ADO.NET Provider supports direct access to Virtuoso's native RDF database engine, eliminating the need for resource intensive Entity Frameworks model transformations.</p> <h3>Related</h3> <ul> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtLinkRemoteTables" id="link-id1183acd8">Attaching ODBC or JDBC accessible Relational Tables to Virtuoso</a> </li> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtRdb2RDFViewsGeneration#One-Click%20Linked%20Data%20Generation%20&%20Deployment" id="link-id113f2fd8">Using an HTML based Wizard to Generate RDF based Linked Views over Relational Tables</a> </li> <li> <a href="http://www.youtube.com/watch?v=bj7AbJ0ZYCk&feature=channel" id="link-id16ad4480">Screencast Demonstrating Wizard based generation of RDF based Linked Data Views Part 1</a> </li> <li> <a href="http://www.youtube.com/watch?v=yXNlcISS0aY&feature=channel" id="link-id114eb720">Screencast Demonstrating Wizard based generation of RDF based Linked Data Views Part 1</a> </li> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtSponger" id="link-id116e5810">Generating RDF based Linked Data from non RDF based Web Resources via the Sponger</a> </li> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtAdoNet35Provider" id="link-id16706118">Building ADO.NET based Entity Frameworks Views over Relational Data</a> </li> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtSilverlightSPARQLExample" id="link-id139c1278">Building Silverlight Rich Internat Applicaitons using ADO.NET, Entity Frameworks, and RDF based Linked Data</a>.</li> </ul>
2010-02-17T16:46:53-05:00
Compare & Contrast: SQL Server's Linked Server vs Virtuoso's Virtual Database Layer
http://www.openlinksw.com:443/blog/vdb/blog/?date=2010-02-12#1607
2010-02-12T21:44:10Z
<h2> <a href="http://dbpedia.org/resource/Microsoft" id="link-id166785f0">Microsoft</a> <a href="http://dbpedia.org/resource/SQL" id="link-id169b6bb8">SQL</a> <a href="http://dbpedia.org/resource/Microsoft_SQL_Server" id="link-id163b8350">Server</a>'s Linked Server Promise</h2> <p>The ability to use distributed queries -- i.e., to issue SQL queries against any OLE-DB-accessible back end -- via Linked Servers.</p> <p>The promise fails to materialize, primarily because while there are several ways of issuing such distributed queries, none of them work with all <a href="http://dbpedia.org/resource/Data" id="link-id1675e128">data</a> access providers, and even for those that do, results received via different methods may differ.</p> <p>Compounding the issue, there are specific configuration options which must be set correctly, often differing from defaults, to permit such things as "ad-hoc distributed queries".</p> <p>Common tools that are typically used with such Linked Servers include SSIS and DTS. Such generic tools typically rely on four-part naming for their queries, expecting SQL Server to properly rewrite remotely executed queries for the DBMS engine which ultimately executes them.</p> <p>The most common cause of failure is that when SQL Server rewrites a query, it typically does so using SQL-92 syntax, regardless of the back-end's abilities, and using the Transact-SQL dialect for implementation-specific query syntaxes, regardless of the back-end's dialect. This leads to problems especially when the Linked Server is an older variant which doesn't support SQL-92 (e.g., Progress 8.x or earlier, <a href="http://dbpedia.org/resource/IBM_Informix" id="link-id167f6fa0">Informix</a> 7 or earlier), or which SQL dialect differs substantially from Transact-SQL (e.g., Informix, Progress, <a href="http://dbpedia.org/resource/MySQL" id="link-id166c7848">MySQL</a>, etc.).</p> <h3>Basic Four-Part Naming</h3> <blockquote> <code>SELECT * <br /> FROM linked_server.[catalog].[<a href="http://dbpedia.org/resource/Database_schema" id="link-id163c3f78">schema</a>].object</code> </blockquote> <p>Four-part naming presumes that you have pre-defined a Linked Server, and executes the query on SQL Server. SQL Server decides what if any sub- or partial-queries to execute on the linked server, tends not to use appropriate syntax for these, and usually does not take advantage of linked server or provider features.</p> <h3>OpenQuery</h3> <blockquote> <code>SELECT * <br /> FROM OPENQUERY ( linked_server , 'query' )</code> </blockquote> <p>OpenQuery also presumes that you have pre-defined a Linked Server, but executes the query as a "pass-through", handing it directly to the remote provider. Features of the remote server and the data access provider may be taken advantage of, but only if the query author knows about them.</p> <h4>From the product docs:</h4> <blockquote> <p> <i>SQL Server's Linked Server extension executes the specified pass-through query on the specified linked server. This server is an OLE DB data source. <code>OPENQUERY</code> can be referenced in the <code>FROM</code> clause of a query as if it were a table name. <code>OPENQUERY</code> can also be referenced as the target table of an <code>INSERT</code>, <code>UPDATE</code>, or <code>DELETE</code> statement. This is subject to the capabilities of the OLE DB provider. Although the query may return multiple result sets, <code>OPENQUERY</code> returns only the first one.</i> </p> <p> <i>...</i> </p> <p> <i><code>OPENQUERY</code> does not accept variables for its arguments. <code>OPENQUERY</code> cannot be used to execute extended stored procedures on a linked server. However, an extended stored procedure can be executed on a linked server by using a four-part name. </i> </p> </blockquote> <h3>OpenRowset</h3> <blockquote> <code>SELECT * <br /> FROM OPENROWSET <br /> ( 'provider_name' , <br /> 'datasource' ; 'user_id' ; 'password', <br /> { [ catalog. ] [ schema. ] object | 'query' }<br /> )</code> </blockquote> <p> <code>OpenRowset</code> does not require a pre-defined Linked Server, but does require the user to know what data access providers are available on the SQL Server host, and how to manually construct a valid connection string for the chosen provider. It does permit both "pass-through" and "local execution" queries, which can lead to confusion when the results differ (as they regularly will).</p> <h4>More from product docs:</h4> <blockquote> <p> <i>Includes all connection <a href="http://dbpedia.org/resource/Information" id="link-id163ab840">information</a> that is required to access remote data from an OLE DB data source. This method is an alternative to accessing tables in a linked server and is a one-time, ad hoc method of connecting and accessing remote data by using OLE DB. For more frequent references to OLE DB data sources, use linked servers instead. For more information, see Linking Servers. The <code>OPENROWSET</code> function can be referenced in the <code>FROM</code> clause of a query as if it were a table name. The <code>OPENROWSET</code> function can also be referenced as the target table of an <code>INSERT</code>, <code>UPDATE</code>, or <code>DELETE</code> statement, subject to the capabilities of the OLE DB provider. Although the query might return multiple result sets, <code>OPENROWSET</code> returns only the first one.</i> </p> <p> <i>OPENROWSET also supports bulk operations through a built-in <code>BULK</code> provider that enables data from a file to be read and returned as a rowset.</i> </p> <p> <i>...</i> </p> <p> <i><code>OPENROWSET</code> can be used to access remote data from OLE DB data sources only when the <code>DisallowAdhocAccess</code> registry option is explicitly set to <code>0</code> for the specified provider, and the <code>Ad Hoc Distributed Queries</code> advanced configuration option is enabled. When these options are not set, the default behavior does not allow for ad hoc access. When accessing remote OLE DB data sources, the login identity of trusted connections is not automatically delegated from the server on which the client is connected to the server that is being queried. Authentication delegation must be configured. For more information, see Configuring Linked Servers for Delegation.</i> </p> <p> <i>Catalog and schema names are required if the OLE DB provider supports multiple catalogs and schemas in the specified data source. Values for catalog and schema can be omitted when the OLE DB provider does not support them. If the provider supports only schema names, a two-part name of the form <code>schema.object</code> must be specified. If the provider supports only catalog names, a three-part name of the form <code>catalog.schema.object</code> must be specified. Three-part names must be specified for pass-through queries that use the SQL Server Native Client OLE DB provider. For more information, see Transact-SQL Syntax Conventions (Transact-SQL). <code>OPENROWSET</code> does not accept variables for its arguments.</i> </p> </blockquote> <h3>OpenDataSource</h3> <blockquote> <code>SELECT * <br /> FROM OPENDATASOURCE<br /> ( 'provider_name',<br /> 'provider_specific_datasource_specification'<br /> ).[catalog].[schema].object</code> </blockquote> <p>As with basic four-part naming, <code>OpenDataSource</code> executes the query on SQL Server. SQL Server decides what if any sub-queries to execute on the linked server, tends not to use appropriate syntax for these, and usually does not take advantage of linked server or provider features.</p> <h4>Additional doc excerpts</h4> <blockquote> <p> <i>Provides ad hoc connection information as part of a four-part object name without using a linked server name.</i> </p> <p> <i>...</i> </p> <p> <i><code>OPENDATASOURCE</code> can be used to access remote data from OLE DB data sources only when the <code>DisallowAdhocAccess</code> registry option is explicitly set to <code>0</code> for the specified provider, and the <code>Ad Hoc Distributed Queries</code> advanced configuration option is enabled. When these options are not set, the default behavior does not allow for ad hoc access.</i> </p> <p> <i>The <code>OPENDATASOURCE</code> function can be used in the same Transact-SQL syntax locations as a linked-server name. Therefore, <code>OPENDATASOURCE</code> can be used as the first part of a four-part name that refers to a table or view name in a <code>SELECT</code>, <code>INSERT</code>, <code>UPDATE</code>, or <code>DELETE</code> statement, or to a remote stored procedure in an <code>EXECUTE</code> statement. When executing remote stored procedures, <code>OPENDATASOURCE</code> should refer to another instance of SQL Server. <code>OPENDATASOURCE</code> does not accept variables for its arguments.</i> </p> <p> <i>Like the <code>OPENROWSET</code> function, <code>OPENDATASOURCE</code> should only reference OLE DB data sources that are accessed infrequently. Define a linked server for any data sources accessed more than several times. Neither <code>OPENDATASOURCE</code> nor <code>OPENROWSET</code> provide all the functionality of linked-server definitions, such as security management and the ability to query catalog information. All connection information, including passwords, must be provided every time that <code>OPENDATASOURCE</code> is called.</i> </p> </blockquote> <h2> <a href="http://virtuoso.openlinksw.com" id="link-id122c66b8">Virtuoso</a>'s <a href="http://dbpedia.org/resource/Virtual_Database" id="link-id167af7d8">Virtual Database</a> Promise & Deliverables</h2> <p>The ability to link objects (tables, views, stored procedures) from any <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id1394ab90">ODBC</a>-accessible data source. This includes any <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id11c38748">JDBC</a>-accessible data source, through the OpenLink ODBC Driver for JDBC Data Sources.</p> <p>There are no limitations on the data types which can be queried or read, nor must the target DBMS have primary keys set on linked tables or views.</p> <p>All linked objects may be used in single-site or distributed queries, and the user need not know anything about the actual data structure, including whether the objects being queried are remote or local to Virtuoso -- all objects are made to appear as part of a Virtuoso-local schema.</p>
2010-02-17T11:21:26-05:00
Compare & Contrast: Oracle Heterogeneous Services (HSODBC, DG4ODBC) vs Virtuoso's Virtual Database Layer
http://www.openlinksw.com:443/blog/vdb/blog/?date=2010-02-12#1606
2010-02-12T21:43:51Z
<h3> <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id12349be8">Oracle</a> Gateway Promise</h3> <p>Ability to use distributed queries over a generic connectivity gateway (HSODBC, DG4ODBC) -- i.e., to issue <a href="http://dbpedia.org/resource/SQL" id="link-id167e5760">SQL</a> queries against any <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id13c6bfa0">ODBC</a>- or OLE-DB-accessible linked back end.</p> <h3>Reality</h3> <p>Promise fails to materialize for several reasons. Immediate limitations include:</p> <ul> <li>All tables locked by a <code>FOR UPDATE</code> clause and all tables with <code>LONG</code> columns selected by the query must be located in the same external database.</li> <li>Distributed queries cannot select user-defined types or object <code>REF</code> datatypes on remote tables.</li> </ul> <p>In addition to the above, which apply to database-specific heterogeneous environments, the database-agnostic generic connectivity components have the following limitations:</p> <ul> <li>A table including a <code>BLOB</code> column must have a separate column that serves as a primary key.</li> <li> <code>BLOB</code> and <code>CLOB</code> <a href="http://dbpedia.org/resource/Data" id="link-id163e07f0">data</a> cannot be read by passthrough queries.</li> <li>Updates or deletes that include unsupported functions within a <code>WHERE</code> clause are not allowed.</li> <li>Generic Connectivity does not support stored procedures.</li> <li>Generic Connectivity agents cannot participate in distributed transactions; they support single-site transactions only.</li> <li>Generic Connectivity does not support multithreaded agents.</li> <li>Updating <code>LONG</code> columns with bind variables is not supported.</li> <li>Generic Connectivity does not support <code>ROWID</code>s.</li> </ul> <p>Compounding the issue, the HSODBC and DG4ODBC generic connectivity agents perform many of their functions by brute-force methods. Rather than interrogating the data access provider (whether ODBC or OLE DB) or DBMS to which they are connected, to learn their capabilities, many things are done by using the lowest possible function.</p> <p>For instance, when a <code>SELECT COUNT (*) FROM table@link</code> is issued through Oracle SQL, the target DBMS doesn't simply perform a <code>SELECT COUNT (*) FROM table</code>. Rather, it performs a <code>SELECT * FROM table</code> which is used to inventory all columns in the table, and then performs and fully retrieves <code>SELECT field FROM table</code> into an internal temporary table, where it does the <code>COUNT (*)</code> itself, locally. Testing has confirmed this process to be the case despite Oracle documentation stating that target data sources must support <code>COUNT (*)</code> (among other functions).</p> <h3> <a href="http://virtuoso.openlinksw.com" id="link-id16814bd8">Virtuoso</a>'s <a href="http://dbpedia.org/resource/Virtual_Database" id="link-id1185b9d0">Virtual Database</a> Comparison</h3> <p>The Virtuoso <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id1666f658">Universal Server</a> will link/attach objects (tables, views, stored procedures) from any ODBC-accessible data source. This includes any <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id1668aec8">JDBC</a>-accessible data source, through the OpenLink ODBC Driver for JDBC Data Sources.</p> <p>There are no limitations on the data types which can be queried or read, nor must the target DBMS have primary keys set on linked tables or views.</p> <p>All linked objects may be used in single-site or distributed queries, and the user need not know anything about the actual data structure, including whether the objects being queried are remote or local to Virtuoso -- all objects are made to appear as part of a Virtuoso-local <a href="http://dbpedia.org/resource/Database_schema" id="link-id1628c438">schema</a>.</p>
2010-02-17T11:21:22.000001-05:00
Why Do I Need To Pay For ODBC, JDBC, ADO.NET, OLE-DB Drivers?
http://www.openlinksw.com:443/weblog/uda/135/?date=2010-02-05#1605
2010-02-05T06:03:53Z
<p>Payment is a function of pain alleviation (opportunity cost) monetization. </p> <p> This post is about highlighting the real pains associated with the $0.00 misconception associated with Data Access Drivers: <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id13bcea18">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id137def30">JDBC</a>, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id14da8ce8">ADO</a>.NET, <a href="http://dbpedia.org/resource/OLE_DB" id="link-id113ee4f0">OLE-DB</a> etc.</p> <p>In the most basic sense, there are some fundament aspects of data access that are complex to implement and rarely implemented (if at all) by free drivers, the list includes:</p> <ol> <li>Escape Syntaxes for Dates and Functions</li> <li>Metadata Calls which enable smarter ODBC compliant applications (this feature is typically missing on Driver Side and abused on the Client side i.e., making clients DBMS specific by testing for specific DBMS names)</li> <li>Scrollable Cursors, this is how you deal with change sensitivity, and most drivers actually fake support and get away with it due to shortage of applications to test proper cursor types (Static, Forward-Only, Key-Set, Dynamic, and Mixed models).</li> </ol> <p>Okay, so we're done with actual driver sophistication re. implementation of critical features. Let's Up the ante by veering into the area of security. At the most basic level, It's extremely important to understand that all data access driver types provide read-write access to your databases; thus, it's imperative that data access drivers address the following:</p> <ol> <li> Read-Only or Read-Write Access scoped to specific Users</li> <li> Ditto applied to specific User Groups </li> <li> Ditto applied to Database Names </li> <li> Ditto applied to specific ODBC compliant applications </li> <li> Ditto applied to specific ODBC host operating systems </li> <li> Ditto applied to specific IP addresses or Ranges on your Network </li> <li> Any combination of items 1-6 as part of a configurable data access rules/policy system. </li> </ol> <p>Once you're done with security, you then have the thorny issue of data access and data flow management. In a nutshell, your driver needs to be able to handle:</p> <ol> <li> Protection against cartesian product network flooding (e.g., user clicks on Customer Table via an ODBC compliant application without comprehension of back-end implications)</li> <li> Enabling or Disabling of key DBMS engine data access optimization features (e.g. DBMS specific extensions exposed via Environment Variables of <a href="http://dbpedia.org/resource/SQL" id="link-id113cc808">SQL</a> commands based settings) </li> <li> Conditional Connection Pooling across User, User Groups, Applications, Host Operating System, IP Address dimensions.</li> </ol> <p>Once you've dealt with Security and Data Flow, you then have to address the enforcement of these settings across a myriad of ODBC compliant host, which is where Zeroconfig and centralized data access administration comes into play i.e., configure once (locally) and enforce globally.</p> <p>When <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id1161faf0">OpenLink Software</a> entered the ODBC Driver Market segment in 1992, the issues above where the fundamental basis of our Multi-Tier Drivers. Thus, although we distinguished ourselves via performance, stability, and specification adherence, our fundamental engineering focus has always been skewed towards security and configurability, alongside high-performance and scalability.</p> <p>As we close 2009, the security issues that pervade Native DBMS Drives, ODBC, JDBC, ADO.NET, OLE-DB etc. Drivers have only increased, courtesy of ubiquitous computing, sadly though, there remains a fundamental illusion that Data Access Drivers simply connect you to DBMS back-ends, and since you can get these drivers at $0.00 from most DBMS vendors they can't be that important. </p> <p>I hope that this post brings some clarity to a very serious security and general configuration management issues associated with Data Access Drivers. Free ODBC Drivers offer nothing, when it comes to the real issues of Open Data Access. If they did, they wouldn't be worth $0.00!</p> <p> <strong>Note:</strong> wondering if this has anything to do with <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1188f968">Linked Data</a> (my current data access focal point)? Well, remember, the <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id14d6d788">Linked Data meme</a> is fundamentally about <a href="http://dbpedia.org/resource/Representational_State_Transfer" id="link-id114051e8">REST</a> based Open Data Access & Integration via HTTP; thus, what applies to Relational Model databases naturally applies to their more granular Graph Model relatives. Basically, data access security never goes away, it just gets more granular, complex, and ultimately, mercurial.</p> <h3>Related</h3> <ul> <li> <a href="http://uda.openlinksw.com/res/oplfeatures/index.html" id="link-id1388e920">OpenLink Universal Data Access Drivers Overview</a> - clickable diagram exposing features and benefits</li> <li> <a href="http://uda.openlinksw.com/odbc/mt/" id="link-id118ab728">OpenLink Multi-Tier ODBC Drivers</a> </li> <li> <a href="http://uda.openlinksw.com/jdbc/mt/" id="link-id13dea018">OpenLink Multi-Tier JDBC Drivers</a> </li> <li> <a href="http://uda.openlinksw.com/dotnet/mt/" id="link-id13b4aec8">OpenLink Multi-Tier ADO.NET Providers</a> </li> <li> <a href="http://www.openlinksw.com/info/docs/odbcwhp/mtarc.htm" id="link-id137d74d0">Multi-Tier Drivers Overview</a> (1993 White Paper excerpt)</li> <li> <a href="http://www.openlinksw.com/info/docs/odbcwhp/tableof.htm#Table%20of%20Contents" id="link-id13b8fb28">OpenLink ODBC White Paper</a> (actual 1993 White Paper)</li> <li> <a href="http://virtuoso.openlinksw.com" id="link-id14702040">OpenLink Virtuoso</a> - which provides a Virtual Conceptual Model (via HTTP, RDF, based <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11b01080">Linked Data</a>) Layer above ODBC or JDBC accessible Data Sources </li> <li> <a href="http://www.dba-oracle.com/t_auditing4_horror_stories_inside_jobs.htm" id="link-id13e64618">Oracle Security Auditing Horror Stories</a> -- Social Dimensions of Security compounded by Value (literal username and password) Based User Identity</li> <li> <a href="http://searchoracle.techtarget.com/news/2240015728/Database-security-strategies-need-to-grow-up-in-2010?" id="link-id116c25c8">Database Security Strategies Need to Grow Up in 2010</a> </li> <li> <a href="http://www.dataflux.com/dfblog/?p=1458" id="link-id13e92180">First Law of Data Quality</a> </li> <li> <a href="http://walkingoncoals.blogspot.com/2009/12/whos-data-is-it-part-1.html" id="link-id13fa4948">Who's Data is it? Part 1</a> </li> <li> <a href="http://walkingoncoals.blogspot.com/2009/12/whos-data-is-it-part-2.html" id="link-id13f00b08">Who's Data is it? Part 2</a> </li> </ul>
2015-07-17T13:21:13.919963-04:00
The Business Of Linked Data (BOLD) Discussion Space
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-01-31#1596
2010-01-31T22:48:36Z
<p>I've created a new discussion space that's squarely focused on the business development and marketing aspects of "HTTP based <a href="http://dbpedia.org/resource/Linked_Data" id="link-id129e32d8">Linked Data" (Linked Data</a>). As its name indicates, It's a BOLD attempt to fill a VoiD. :-)</p> <h3>Background</h3> <p>A few months ago, <a href="http://blog.aldobucchi.com/#this" id="link-id1110eb30">Aldo Bucchi</a> posted a message to the <a href="http://esw.w3.org/topic/SweoIG/TaskForces/CommunityProjects/LinkingOpenData" id="link-id111d08a0">LOD</a> <a href="http://lists.w3.org/Archives/Public/public-lod/" id="link-id118b3778">mailing list</a> seeking a discussion space for more business and marketing oriented topic, in relation to Linked Data. At the time, my assumption was that the existing LOD mailing list served that purpose absolutely fine, but in due course I came to realize that Aldo's request had a much lager foundation than I initially suspected.</p> <h3>Historic Oversight</h3> <p>Linked Data, like its umbrella <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id16ceb618">Semantic Web</a> Project, has suffered from an inadvertent oversight on the parts of many of its enthusiasts (myself included): 100% of the discussion spaces are created by, geared towards, or dominated by researchers (from Academia primarily) and/or developers. Thus, at the very least, we've been operating in an echo chamber that only feed the existing void between the core community and those who are more interested in discussing business and marketing related topics.</p> <p>The new discussion space seeks to cover the following:</p> <ol> <li> Brainstorming Value Proposition Articulation</li> <li>War Story Exchanges</li> <li>Case Studies and Use-cases</li> <li>Market Research & Positioning (for instance Linked Data is killer technology that redefines <a href="http://dbpedia.org/resource/Data" id="link-id0x1d491e90">Data</a> Integration, but none of the major research firms currently make that connection)</li>. </ol> <p>How Do I Join The Conversation? Simply sign up on the Google hosted <a href="http://groups.google.com/group/business-of-linked-data-bold" id="link-id129e4d08">BOLD mailing list</a>, introduce yourself (ideally), and then start conversing! :-)</p>
2010-01-31T17:48:48-05:00
Getting The Linked Data Value Pyramid Layers Right (Update #2)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-01-31#1595
2010-01-31T22:46:47Z
<p> One of the real problems that pervades all routes to <a href="http://dbpedia.org/resource/Linked_Data" id="link-id13539328">Linked Data</a> value prop. incomprehension stems from the layering of its value pyramid; especially when communicating with -initially detached- end-users. </p> <p> <strong>Note to <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id0x1c85f498">Web</a> Programmers:</strong> Linked Data is about <a href="http://dbpedia.org/resource/Data" id="link-id0x1c85f650">Data</a> (Wine) and not about Code (Fish). Thus, it isn't a "programmer only zone", far from it. More than anything else, its inherently inclusive and spreads its participation net widely across: Data Architects, Data Integrators, Power Users, <a href="http://dbpedia.org/resource/Knowledge" id="link-id13600d98">Knowledge</a> Workers, <a href="http://dbpedia.org/resource/Information" id="link-id149f8230">Information</a> Workers, Data Analysts, etc.. Basically, everyone that can "click on a link" is invited to this particular party; remember, it is about "Linked Data" not "Linked Code", after all. :-) </p> <h3>Problematic Value Pyramid Layering</h3> <p> Here is an example of a Linked Data value pyramid that I am stumbling across --with some frequency-- these days (note: 1 being the pyramid apex):</p> <ol> <li> <a href="http://dbpedia.org/resource/SPARQL" id="link-id10e85538">SPARQL</a> Queries</li> <li> <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id1495b578">RDF</a> Data Stores</li> <li> RDF Data Sets </li> <li> <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id158e4be0">HTTP</a> scheme URIs</li> </ol> <p> Basically, Linked Data deployment (assigning de-referencable HTTP URIs to DBMS records, their attributes, and attribute values [optionally] ) is occurring last. Even worse, this happens in the <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id626d988">context</a> of Linked Open Data oriented endeavors, resulting in nothing but confusion or inadvertent perpetuation of the overarching pragmatically challenged "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id111774b8">Semantic Web</a>" stereotype. </p> <p> As you can imagine, hitting SPARQL as your introduction to Linked Data is akin to hitting <a href="http://dbpedia.org/resource/SQL" id="link-id151f9938">SQL</a> as your introduction to Relational Database Technology, neither is an elevator-style value prop. relay mechanism. </p> <p> In the relational realm, killer demos always started with desktop productivity tools (spreadsheets, report-writers, SQL QBE tools etc.) accessing, relational data sources en route to unveiling the "Productivity" and "Agility" value prop. that such binding delivered i.e., the desktop application (clients) and the databases (servers) are distinct, but operating in a mutually beneficial manner to all, courtesy of a data access standards such as <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id1519aac0">ODBC</a> (Open Database Connectivity). </p> <p> In the Linked Data realm, learning to embrace and extend best practices from the relational dbms realm remains a challenge, a lot of this has to do with hangovers from a misguided perception that RDF databases will somehow completely replace <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id110dec88">RDBMS</a> engines, rather than compliment them. Thus, you have a counter productive variant of NIH (Not Invented Here) in play, taking us to the dreaded realm of: Break the Pot and You Own It (exemplified by the 11+ year Semantic Web Project comprehension and appreciation odyssey). </p> <p> From my vantage point, here is how I believe the <a href="http://virtuoso.openlinksw.com/presentations/Creating_Deploying_Exploiting_Linked_Data2/images/URI_Data_Source_SemWeb.png" id="link-id1592f528">Linked Data value pyramid should be layered</a>, especially when communicating the essential value prop.: </p> <ol> <li> HTTP URLs -- LINKs to documents (Reports) that users already appreciate, across the public Web and/or Intranets </li> <li> HTTP URIs -- typically not visually distinguishable from the URLs, so use the Data exposed by de-referencing a <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id11209ce8">URL</a> to show how each Data Item (<a href="http://dbpedia.org/resource/Entity" id="link-id1449b558">Entity</a> or Object) is uniquely identified by a Generic HTTP <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id112065f8">URI</a>, and how clicking on the said URIs leads to more structured metadata bearing documents available in a variety of data representation formats, thereby enabling flexible data presentation (e.g., smarter HTML pages) </li> <li> SPARQL -- when a user appreciates the data representation and presentation dexterity of a Generic HTTP URI, they will be more inclined to drill down an additional layer to unravel how HTTP URIs mechanically deliver such flexibility </li> <li> RDF Data Stores -- at this stage the user is now interested data sources behind the Generic HTTP URIs, courtesy of natural desire to tweak the data presented in the report; thus, you now have an engaged user ready to absorb the "How Generic HTTP URIs Pull This Off" message </li> <li>RDF Data Sets -- while attempting to make or tweak HTTP URIs, users become curious about the actual data loaded into the RDF Data Store, which is where data sets used to create powerful Lookup Data Spaces (e.g., <a href="http://dbpedia.org/resource/DBpedia" id="link-id110675c0">DBpedia</a>) come into play such as those from the <a href="http://www4.wiwiss.fu-berlin.de/bizer/pub/lod-datasets_2009-07-14.html" id="link-id11127ff8">LOD</a> constellation as exemplified by <a href="http://wiki.dbpedia.org/Datasets" id="link-id14a2fad8">DBpedia (extractions from Wikipedia)</a>.</li> </ol> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1565" id="link-id149c7048">Exploring the Linked Data Value Proposition</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1543" id="link-id14998f98">Simple Explanation of Linked Data & RDF Dynamics</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1546" id="link-id114fbd58">What is the Linked Data Meme About?</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1547" id="link-id1447ada0">Linked Data & Data Item Identifiers (Identity)</a> </li> </ul>
2010-01-31T17:47:04-05:00
What is the DBpedia Project? (Updated)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-01-31#1594
2010-01-31T22:45:55Z
<p>The recent <a href="http://en.wikipedia.org/wiki/Talk:DBpedia" id="link-id1120a260">Wikipedia imbroglio</a> centered around <a href="http://dbpedia.org/resource/DBpedia" id="link-id14a5e588">DBpedia</a> is the fundamental driver for this particular <a href="http://dbpedia.org/resource/Blog" id="link-id113ddc10">blog</a> post. At time of writing this blog post, the <a href="http://en.wikipedia.org/wiki/DBpedia" id="link-id158edec0">DBpedia project definition in Wikipedia</a> remains unsatisfactory due to the following shortcomings:</p> <ol> <li>inaccurate and incomplete definition of the Project's What, Why, Who, Where, When, and How</li> <li>inaccurate reflection of project essence, by skewing focus towards <a href="http://dbpedia.org/resource/Data" id="link-id0x1bc892d0">data</a> extraction and data set dump production, which is at best a quarter of the project.</li> </ol> <p>Here are some insights on DBpedia, from the perspective of someone intimately involved with the other three-quarters of the project.</p> <h3>What is DBpedia?</h3> <p>A live <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id0x1c0c0cc0">Web</a> accessible RDF model database (Quad Store) derived from Wikipedia content snapshots, taken periodically. The RDF database underlies a <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11ba0ad0">Linked Data</a> <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id1183c978">Space</a> comprised of: HTML (and most recently HTML+<a href="http://dbpedia.org/resource/RDFa" id="link-id602eab8">RDFa</a>) based data browser pages and a <a href="http://dbpedia.org/resource/SPARQL" id="link-id11af5400">SPARQL</a> endpoint.</p> <p>Note: <a href="http://blog.dbpedia.org/2009/11/11/dbpedia-34-released/" id="link-id110b8248">DBpedia 3.4</a> now exists in snapshot (warehouse) and <a href="http://dbpedia-live.openlinksw.com/stats/" id="link-id6473258">Live Editions</a> (currently being hot-staged). This post is about the snapshot (warehouse) edition, I'll drop a different post about the DBpedia Live Edition where a new Delta-Engine covers both extraction and database record replacement, in realtime.</p> <h3>When was it Created?</h3> <p>As an idea under the moniker "DBpedia" it was conceptualized in late 2006 by researchers at University of Leipzig (lead by Soren Auer) and Freie University, Berlin (lead by <a href="http://www.wiwiss.fu-berlin.de/en/institute/pwo/bizer/" id="link-id14982c78">Chris Bizer</a>). The first public instance of DBpedia (as described above) was released in February 2007. The official DBpedia coming out party occurred at <a href="http://www2007.org/" id="link-id1497c788">WWW2007</a>, Banff, during the <a href="http://esw.w3.org/topic/SweoIG/TaskForces/CommunityProjects/LinkingOpenData/BanffGathering" id="link-id1448b9e8">inaugural Linked Data gathering</a>, where it showcased the virtues and immense potential of <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id152257e0">TimBL</a>'s <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id111759a8">Linked Data meme</a>.</p> <h3>Who's Behind It?</h3> <p> <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id110e70f8">OpenLink Software</a> (developers of OpenLink <a href="http://virtuoso.openlinksw.com" id="link-id14462f60">Virtuoso</a> and providers of Web Hosting infrastructure), University of Leipzig, and Freie Univerity, Berlin. In addition, there is a burgeoning community of collaborators and contributors responsible DBpedia based applications, cross-linked data sets, ontologies (<a href="http://dbpedia.org/resource/Cyc" id="link-id11244aa0">OpenCyc</a>, <a href="http://www.ontologyportal.org/" id="link-id110e4a40">SUMO</a>, <a href="http://umbel.org/about/" id="link-id11109e48">UMBEL</a>, and <a href="http://www.mpi-inf.mpg.de/yago-naga/yago/" id="link-id10fb4218">YAGO</a>) and other utilities. Finally, DBpedia wouldn't be possible without the global content contribution and curation efforts of Wikipedians, a point typically overlooked (albeit inadvertently).</p> <h3>How is it Constructed?</h3> <p>The steps are as follows:</p> <ol> <li> RDF data set dump preparation via Wikipedia content extraction and transformation to RDF model data, using the N3 data representation format - Java and <a href="http://dbpedia.org/resource/PHP" id="link-id111c93b8">PHP</a> extraction code produced and maintained by the teams at Leipzig and Berlin </li> <li> Deployment of Linked Data that enables Data browsing and exploration using any HTTP aware user agent (e.g. basic Web Browsers) - handled by OpenLink Virtuoso (handled by Berlin via the Pubby Linked Data Server during the early months of the DBpedia project) </li> <li> SPARQL compliant Quad Store, enabling direct access to database records via SPARQL (Query language, REST or SOAP Web Service, plus a variety of query results serialization formats) - OpenLink Virtuoso since first public release of DBpedia </li> </ol> <p> In a nutshell, there are four distinct and vital components to DBpedia. Thus, DBpedia doesn't exist if all the project offered was a collection of RDF data dumps. Likewise, it doesn't exist if you have a SPARQL compliant Quad Store without loaded data sets, and of course it doesn't exist if you have a fully loaded SPARQL compliant Quad Store is up to the cocktail of challenges presented by live Web accessibility.</p> <h3>Why is it Important?</h3> <p> It remains a live exemplar for any individual or organization seeking to publishing or exploit HTTP based Linked Data on the <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id118e6388">World Wide Web</a>. Its existence continues to stimulate growth in both density and quality of the burgeoning Web of Linked Data.</p> <h3>How Do I Use it?</h3> <p> In the most basic sense, simply browse the HTML pages en route to discovery erstwhile relationships that exist across <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id112def88">named entities</a> and <a href="http://dbpedia.org/resource/Topic" id="link-id1591c5f8">subject matter concepts</a> / headings. Beyond that, simply look at DBpedia as a master lookup table in a Web hosted <a href="http://dbpedia.org/resource/federated_database_system" id="link-id11762618">distributed database</a> setup; enabling you to mesh your local domain specific details with DBpedia records via structured relations (triples or 3-tuples records) comprised of HTTP URIs from both realms e.g., owl:sameAs relations.</p> <h3>What Can I Use it For?</h3> <p> Expanding on the Master-Details point above, you can use its rich <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id1170c000">URI</a> corpus to alleviate tedium associated with activities such as: </p> <ol> <li>List maintenance - e.g., Countries, States, Companies, Units of Measurement, Subject Headings etc.</li> <li>Tagging - as a compliment to existing practices</li> <li>Analytical Research - you're only a LINK (URI) away from erstwhile difficult to attain research data spread across a broad range of topics</li> <li>Closed Vocabulary Construction - rather than commence the futile quest of building your own closed vocabulary, simply leverage Wikipedia's human curated vocabulary as our common base. </li> </ol> <h3>Related</h3> <ul> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtAWSDBpedia34S" id="link-id14a2e698">Pre-loaded and Pre-configured instances of DBpedia 3.4</a> - via publicly shared <a href="http://aws.amazon.com/ebs/" id="link-id1147fcf0">Amazon Elastic Block Storage</a> Snapshots</li> <li> <a href="http://docs.openlinksw.com/virtuoso/rdfperformancetuning.html#rdfperfgeneraldbpedia" id="link-id149ab528">Virtuoso & DBpedia Tunning Guide</a> </li> <li> <a href="http://dowhatimean.net/2009/11/whats-in-a-name-and-the-linked-data-police" id="link-id110cba10">What's In a Name & The Linked Data Police</a>. </li> </ul>
2010-01-31T17:46:10.000002-05:00
Getting The Linked Data Value Pyramid Layers Right (Update #2)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-01-31#1593
2010-01-31T22:44:04Z
<p> One of the real problems that pervades all routes to <a href="http://dbpedia.org/resource/Linked_Data" id="link-id13539328">Linked Data</a> value prop. incomprehension stems from the layering of its value pyramid; especially when communicating with -initially detached- end-users. </p> <p> <strong>Note to <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> Programmers:</strong> Linked Data is about <a href="http://dbpedia.org/resource/Data">Data</a> (Wine) and not about Code (Fish). Thus, it isn't a "programmer only zone", far from it. More than anything else, its inherently inclusive and spreads its participation net widely across: Data Architects, Data Integrators, Power Users, <a href="http://dbpedia.org/resource/Knowledge" id="link-id13600d98">Knowledge</a> Workers, <a href="http://dbpedia.org/resource/Information" id="link-id149f8230">Information</a> Workers, Data Analysts, etc.. Basically, everyone that can "click on a link" is invited to this particular party; remember, it is about "Linked Data" not "Linked Code", after all. :-) </p> <h3>Problematic Value Pyramid Layering</h3> <p> Here is an example of a Linked Data value pyramid that I am stumbling across --with some frequency-- these days (note: 1 being the pyramid apex):</p> <ol> <li> <a href="http://dbpedia.org/resource/SPARQL" id="link-id10e85538">SPARQL</a> Queries</li> <li> <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id1495b578">RDF</a> Data Stores</li> <li> RDF Data Sets </li> <li> <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id158e4be0">HTTP</a> scheme URIs</li> </ol> <p> Basically, Linked Data deployment (assigning de-referencable HTTP URIs to DBMS records, their attributes, and attribute values [optionally] ) is occurring last. Even worse, this happens in the <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id626d988">context</a> of Linked Open Data oriented endeavors, resulting in nothing but confusion or inadvertent perpetuation of the overarching pragmatically challenged "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id111774b8">Semantic Web</a>" stereotype. </p> <p> As you can imagine, hitting SPARQL as your introduction to Linked Data is akin to hitting <a href="http://dbpedia.org/resource/SQL" id="link-id151f9938">SQL</a> as your introduction to Relational Database Technology, neither is an elevator-style value prop. relay mechanism. </p> <p> In the relational realm, killer demos always started with desktop productivity tools (spreadsheets, report-writers, SQL QBE tools etc.) accessing, relational data sources en route to unveiling the "Productivity" and "Agility" value prop. that such binding delivered i.e., the desktop application (clients) and the databases (servers) are distinct, but operating in a mutually beneficial manner to all, courtesy of a data access standards such as <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id1519aac0">ODBC</a> (Open Database Connectivity). </p> <p> In the Linked Data realm, learning to embrace and extend best practices from the relational dbms realm remains a challenge, a lot of this has to do with hangovers from a misguided perception that RDF databases will somehow completely replace <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id110dec88">RDBMS</a> engines, rather than compliment them. Thus, you have a counter productive variant of NIH (Not Invented Here) in play, taking us to the dreaded realm of: Break the Pot and You Own It (exemplified by the 11+ year Semantic Web Project comprehension and appreciation odyssey). </p> <p> From my vantage point, here is how I believe the <a href="http://virtuoso.openlinksw.com/presentations/Creating_Deploying_Exploiting_Linked_Data2/images/URI_Data_Source_SemWeb.png" id="link-id1592f528">Linked Data value pyramid should be layered</a>, especially when communicating the essential value prop.: </p> <ol> <li> HTTP URLs -- LINKs to documents (Reports) that users already appreciate, across the public Web and/or Intranets </li> <li> HTTP URIs -- typically not visually distinguishable from the URLs, so use the Data exposed by de-referencing a <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id11209ce8">URL</a> to show how each Data Item (<a href="http://dbpedia.org/resource/Entity" id="link-id1449b558">Entity</a> or Object) is uniquely identified by a Generic HTTP <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id112065f8">URI</a>, and how clicking on the said URIs leads to more structured metadata bearing documents available in a variety of data representation formats, thereby enabling flexible data presentation (e.g., smarter HTML pages) </li> <li> SPARQL -- when a user appreciates the data representation and presentation dexterity of a Generic HTTP URI, they will be more inclined to drill down an additional layer to unravel how HTTP URIs mechanically deliver such flexibility </li> <li> RDF Data Stores -- at this stage the user is now interested data sources behind the Generic HTTP URIs, courtesy of natural desire to tweak the data presented in the report; thus, you now have an engaged user ready to absorb the "How Generic HTTP URIs Pull This Off" message </li> <li>RDF Data Sets -- while attempting to make or tweak HTTP URIs, users become curious about the actual data loaded into the RDF Data Store, which is where data sets used to create powerful Lookup Data Spaces (e.g., <a href="http://dbpedia.org/resource/DBpedia" id="link-id110675c0">DBpedia</a>) come into play such as those from the <a href="http://www4.wiwiss.fu-berlin.de/bizer/pub/lod-datasets_2009-07-14.html" id="link-id11127ff8">LOD</a> constellation as exemplified by <a href="http://wiki.dbpedia.org/Datasets" id="link-id14a2fad8">DBpedia (extractions from Wikipedia)</a>.</li> </ol> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1565" id="link-id149c7048">Exploring the Linked Data Value Proposition</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1543" id="link-id14998f98">Simple Explanation of Linked Data & RDF Dynamics</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1546" id="link-id114fbd58">What is the Linked Data Meme About?</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1547" id="link-id1447ada0">Linked Data & Data Item Identifiers (Identity)</a> </li> </ul>
2010-02-01T09:02:14.000004-05:00
What is the DBpedia Project? (Updated)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-01-31#1592
2010-01-31T22:43:08Z
<p> The recent <a href="http://en.wikipedia.org/wiki/Talk:DBpedia" id="link-id1120a260">Wikipedia imbroglio</a> centered around <a href="http://dbpedia.org/resource/DBpedia" id="link-id14a5e588">DBpedia</a> is the fundamental driver for this particular <a href="http://dbpedia.org/resource/Blog" id="link-id113ddc10">blog</a> post. At time of writing this blog post, the <a href="http://en.wikipedia.org/wiki/DBpedia" id="link-id158edec0">DBpedia project definition in Wikipedia</a> remains unsatisfactory due to the following shortcomings:</p> <ol> <li> inaccurate and incomplete definition of the Project's What, Why, Who, Where, When, and How</li> <li> inaccurate reflection of project essence, by skewing focus towards <a href="http://dbpedia.org/resource/Data">data</a> extraction and data set dump production, which is at best a quarter of the project.</li> </ol> <p> Here are some insights on DBpedia, from the perspective of someone intimately involved with the other three-quarters of the project.</p> <h3> What is DBpedia?</h3> <p> A live <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> accessible RDF model database (Quad Store) derived from Wikipedia content snapshots, taken periodically. The RDF database underlies a <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11ba0ad0">Linked Data</a> <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id1183c978">Space</a> comprised of: HTML (and most recently HTML+<a href="http://dbpedia.org/resource/RDFa" id="link-id602eab8">RDFa</a>) based data browser pages and a <a href="http://dbpedia.org/resource/SPARQL" id="link-id11af5400">SPARQL</a> endpoint.</p> <p> Note: <a href="http://blog.dbpedia.org/2009/11/11/dbpedia-34-released/" id="link-id110b8248">DBpedia 3.4</a> now exists in snapshot (warehouse) and <a href="http://dbpedia-live.openlinksw.com/stats/" id="link-id6473258">Live Editions</a> (currently being hot-staged). This post is about the snapshot (warehouse) edition, I'll drop a different post about the DBpedia Live Edition where a new Delta-Engine covers both extraction and database record replacement, in realtime.</p> <h3> When was it Created?</h3> <p> As an idea under the moniker "DBpedia" it was conceptualized in late 2006 by researchers at University of Leipzig (lead by Soren Auer) and Freie University, Berlin (lead by <a href="http://www.wiwiss.fu-berlin.de/en/institute/pwo/bizer/" id="link-id14982c78">Chris Bizer</a>). The first public instance of DBpedia (as described above) was released in February 2007. The official DBpedia coming out party occurred at <a href="http://www2007.org/" id="link-id1497c788">WWW2007</a>, Banff, during the <a href="http://esw.w3.org/topic/SweoIG/TaskForces/CommunityProjects/LinkingOpenData/BanffGathering" id="link-id1448b9e8">inaugural Linked Data gathering</a>, where it showcased the virtues and immense potential of <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id152257e0">TimBL</a>'s <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id111759a8">Linked Data meme</a>.</p> <h3> Who's Behind It?</h3> <p> <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id110e70f8">OpenLink Software</a> (developers of OpenLink <a href="http://virtuoso.openlinksw.com" id="link-id14462f60">Virtuoso</a> and providers of Web Hosting infrastructure), University of Leipzig, and Freie Univerity, Berlin. In addition, there is a burgeoning community of collaborators and contributors responsible DBpedia based applications, cross-linked data sets, ontologies (<a href="http://dbpedia.org/resource/Cyc" id="link-id11244aa0">OpenCyc</a>, <a href="http://www.ontologyportal.org/" id="link-id110e4a40">SUMO</a>, <a href="http://umbel.org/about/" id="link-id11109e48">UMBEL</a>, and <a href="http://www.mpi-inf.mpg.de/yago-naga/yago/" id="link-id10fb4218">YAGO</a>) and other utilities. Finally, DBpedia wouldn't be possible without the global content contribution and curation efforts of Wikipedians, a point typically overlooked (albeit inadvertently).</p> <h3> How is it Constructed?</h3> <p> The steps are as follows:</p> <ol> <li> RDF data set dump preparation via Wikipedia content extraction and transformation to RDF model data, using the N3 data representation format - Java and <a href="http://dbpedia.org/resource/PHP" id="link-id111c93b8">PHP</a> extraction code produced and maintained by the teams at Leipzig and Berlin</li> <li> Deployment of Linked Data that enables Data browsing and exploration using any HTTP aware user agent (e.g. basic Web Browsers) - handled by OpenLink Virtuoso (handled by Berlin via the Pubby Linked Data Server during the early months of the DBpedia project)</li> <li> SPARQL compliant Quad Store, enabling direct access to database records via SPARQL (Query language, REST or SOAP Web Service, plus a variety of query results serialization formats) - OpenLink Virtuoso since first public release of DBpedia</li> </ol> <p> In a nutshell, there are four distinct and vital components to DBpedia. Thus, DBpedia doesn't exist if all the project offered was a collection of RDF data dumps. Likewise, it doesn't exist without a fully populated SPARQL compliant Quad Store. Last but not least, it doesn't exist if you have a fully loaded SPARQL compliant Quad Store isn't up to the cocktail of challenges (query load and complexity) presented by live Web database accessibility.</p> <h3> Why is it Important?</h3> <p> It remains a live exemplar for any individual or organization seeking to publishing or exploit HTTP based Linked Data on the <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id118e6388">World Wide Web</a>. Its existence continues to stimulate growth in both density and quality of the burgeoning Web of Linked Data.</p> <h3> How Do I Use it?</h3> <p> In the most basic sense, simply browse the HTML based resource decriptor pages en route to discovering erstwhile undiscovered relationships that exist across <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id112def88">named entities</a> and <a href="http://dbpedia.org/resource/Topic" id="link-id1591c5f8">subject matter concepts</a> / headings. Beyond that, simply look at DBpedia as a master lookup table in a Web hosted <a href="http://dbpedia.org/resource/federated_database_system" id="link-id11762618">distributed database</a> setup; enabling you to mesh your local domain specific details with DBpedia records via structured relations (triples or 3-tuples records), comprised of HTTP URIs from both realms e.g., via owl:sameAs relations.</p> <h3> What Can I Use it For?</h3> <p> Expanding on the Master-Details point above, you can use its rich <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id1170c000">URI</a> corpus to alleviate tedium associated with activities such as:</p> <ol> <li> List maintenance - e.g., Countries, States, Companies, Units of Measurement, Subject Headings etc.</li> <li> Tagging - as a compliment to existing practices</li> <li> Analytical Research - you're only a LINK (URI) away from erstwhile difficult to attain research data spread across a broad range of topics</li> <li> Closed Vocabulary Construction - rather than commence the futile quest of building your own closed vocabulary, simply leverage Wikipedia's human curated vocabulary as our common base.</li> </ol> <h3> Related</h3> <ul> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtAWSDBpedia34S" id="link-id14a2e698">Pre-loaded and Pre-configured instances of DBpedia 3.4</a> - via publicly shared <a href="http://aws.amazon.com/ebs/" id="link-id1147fcf0">Amazon Elastic Block Storage</a> Snapshots</li> <li> <a href="http://docs.openlinksw.com/virtuoso/rdfperformancetuning.html#rdfperfgeneraldbpedia" id="link-id149ab528">Virtuoso & DBpedia Tunning Guide</a> </li> <li> <a href="http://dowhatimean.net/2009/11/whats-in-a-name-and-the-linked-data-police" id="link-id110cba10">What's In a Name & The Linked Data Police</a>.</li> </ul>
2010-09-15T18:10:51.000002-04:00
5 Very Important Things to Note about HTTP based Linked Data
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-01-31#1591
2010-01-31T22:31:35Z
<ol> <li> It isn't <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id115dfd68">World Wide Web</a> Specific (HTTP != World Wide Web)</li> <li> It isn't Open <a href="http://dbpedia.org/resource/Data">Data</a> Specific </li> <li> It isn't about "Free" (Beer or Speech) </li> <li> It isn't about Markup (so don't expect to grok it via "markup first" approach) </li> <li>It's about <a href="http://dbpedia.org/resource/Linked_Data" id="link-id13a6aa98">Hyperdata</a> - the use of HTTP and REST to deliver a powerful platform agnostic mechanism for Data Reference, Access, and Integration.</li> </ol> <p> When trying to understand HTTP based <a href="http://dbpedia.org/resource/Linked_Data" id="link-id18aa1490">Linked Data</a>, especially if you're well versed in DBMS technology use (User, Power User, Architect, Analyst, DBA, or Programmer) think: <br /> </p> <ul> <li> Open Database Connectivity (<a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id1428fba0">ODBC</a>) without operating system, data model, or wire-protocol specificity or lock-in potential </li> <li> Java Database Connectivity (<a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id18d3c2a8">JDBC</a>) without programming language specificity </li> <li> <a href="http://dbpedia.org/resource/ADO.NET" id="link-id125725b8">ADO</a>.NET without .NET runtime specificity and .NET bound language specificity </li> <li> OLE-DB without Windows operating system & programming language specificity </li> <li> XMLA without XML format specificity - with Tabular and Multidimensional results formats expressible in a variety of data representation formats. </li> <li>All of the above scoped to the Record rather than Container level, with Generic HTTP scheme URIs associated with each Record, Field, and Field value (optionally) </li> </ul> <p>Remember the need for Data Access & Integration technology is the by product of the following realities:</p> <ol> <li> Human curated data is ultimately dirty, because: <ul> <li>our thick thumbs, inattention, distractions, and general discomfort with typing, make typos prevalent</li> <li>database engines exist for a variety of data models - Graph, Relational, Hierarchical;</li> <li>within databases you have different record container/partition names e.g. Table Names;</li> <li>within a database record container you have records that are really aspects of the same thing (different keys exist in a plethora of operational / line of business systems that expose aspects of the same <a href="http://dbpedia.org/resource/Entity" id="link-id13378338">entity</a> e.g., customer data that spans Accounts, CRM, ERP application databases);</li> <li>different field names (one database has "EMP" while another has "Employee") for the same record</li>.</ul> </li> <li>Units of measurement is driven by locale, the UK office wants to see sales in Pounds Sterling while the French office prefers Euros etc.</li> <li>All of the above is subject to <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id17e46398">context</a> halos which can be quite granular re. sensitivity e.g. staff travel between locations that alter locales and their roles; basically, profiles matters a lot.</li> </ol> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1364" id="link-id128f0fe8">ODBC and WODBC (Web Open Database Connectivity) Comparison</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1364" id="link-id1367cd18">Creating, Deploying, and Exploiting Linked Data Presentation</a> </li> <li> <a href="http://www.odata.org/" id="link-id122ab708">Open Data Protocol Project</a> </li> </ul>
2010-02-01T09:00:56-05:00
5 Game Changing Things about the OpenLink Virtuoso + AWS Cloud Combo
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-01-31#1590
2010-01-31T22:29:34Z
<p> Here are 5 powerful benefits you can immediately derive from the combination of <a href="http://virtuoso.openlinksw.com" id="link-id17eb8988">Virtuoso</a> and Amazon's AWS services (specifically the EC2 and EBS components): <br /> </p> <ol> <li> Acquire your own personal or service specific <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id1423e520">data space</a> in the Cloud. Think DBase, Paradox, FoxPRO, Access of yore, but with the power of <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id136c6290">Oracle</a>, <a href="http://dbpedia.org/resource/IBM_Informix" id="link-id11b269b8">Informix</a>, <a href="http://dbpedia.org/resource/Microsoft_SQL_Server" id="link-id138084b8">Microsoft SQL Server</a> etc.. using a Conceptual, as opposed to solely Logical, model based DBMS (i.e., a Hybrid DBMS Engine for: <a href="http://dbpedia.org/resource/SQL" id="link-id132a7938">SQL</a>, RDF, XML, and Full Text) </li> <li> Ability to share and control access to your resources using innovations like <a href="http://dbpedia.org/resource/Friend_of_a_friend" id="link-id17ee9d28">FOAF</a>+SSL, OpenID, and OAuth, all from one place </li> <li> Construction of personal or organization based FOAF profiles in a matter of minutes; by simply creating a basic DBMS (or <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id14784ae0">ODS</a> application layer) account; and then using this profile to create strong links (references) to all your Data silos (esp. those from the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> 2.0 realm) </li> <li> Load data sets from the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id17e6ac98">LOD</a> cloud or Sponge existing Web resources (i.e., on the fly data transformation to RDF model based <a href="http://dbpedia.org/resource/Linked_Data" id="link-id17e65d38">Linked Data</a>) and then use the combination to build powerful lookup services that enrich the value of URLs (think: Web addressable reports holding query results) that you publish </li> <li> Bind all of the above to a domain that you own (e.g. a .Name domain) so that you have an attribution-friendly "authority" component for resource URLs and <a href="http://dbpedia.org/resource/Entity" id="link-id118a08d8">Entity</a> URIs published from your Personal Linked Data Space on the Web (or private HTTP network). </li> </ol> <p> In a nutshell, the AWS Cloud infrastructure simplifies the process of generating Federated presence on the <a href="http://dbpedia.org/resource/Internet" id="link-id1380af38">Internet</a> and/or <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id11633b10">World Wide Web</a>. Remember, centralized networking models always end up creating data silos, in some <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id142006f0">context</a>, ultimately! :-) </p>
2010-02-01T08:59:36-05:00
Virtuoso Chronicles from the Field: Nepomuk, KDE, and the quest for a sophisticated RDF DBMS.
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-01-28#1602
2010-01-28T16:14:04Z
<p>For this particular user experience chronicle, I've simply inserted the content of <a href="http://trueg.wordpress.com" id="link-id1368b4d8">Sebastian Trueg</a>'s post titled: <a href="http://trueg.wordpress.com/2010/01/26/what-we-did-last-summer-and-the-rest-of-2009-a-look-back-onto-the-nepomuk-development-year-with-an-obscenely-long-title/#comments" id="link-id139dddb0">What We Did Last Summer (And the Rest of 2009) – A Look Back Onto the Nepomuk Development Year ...</a>, directly into this post, without any additional commentary or modification.</p> <div class="snap_preview"> <p>2009 is over. <em>Yeah, sure, trueg, we know that, it has been over for a while now!</em> Ok, ok, I am a bit late, but still I would like to get this one out - if only for my archive. So here goes.</p> <h2> <a href="http://virtuoso.openlinksw.com" id="link-id64672f0">Virtuoso</a> </h2> <p>Let’s start with the major topic of 2009 (and also the beginning of 2010): The new Nepomuk database backend: <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/" id="link-id13cc47e0">Virtuoso</a>. Everybody who used Nepomuk had the same problems: you either used the <a href="http://openrdf.org/" id="link-id13a4ac88">sesame2</a> backend which depends on Java and steals all of your memory or you were stuck with <a href="http://librdf.org/" id="link-id11b6a550">Redland</a> which had the worst performance and missed some <a href="http://dbpedia.org/resource/SPARQL" id="link-id139d82b8">SPARQL</a> features making important parts of Nepomuk like queries unusable. So more than a year ago I had the idea to use the one GPL’ed database server out there that supported RDF in a professional manner: <a href="http://www.openlinksw.com/" id="link-id139fd948">OpenLin</a>k’s <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/" id="link-id12329590">Virtuoso</a>. It has all the features we need, has a very good performance, and scales up to dimensions we will probably never reach on the desktop (<em>yeah, right, and 64k main memory will be enough forever!</em>). So very early I started coding the necessary Soprano plugin which would talk to a locally running Virtuoso server through <a href="http://en.wikipedia.org/wiki/Open_Database_Connectivity" id="link-id14930d90">ODBC</a>. But since I ran into tons of small problems (as always) and got sidetracked by other tasks I did not finish it right away. OpenLink, however, was very interested in the idea of their server being part of every KDE installation (why wouldn’t they ;)). So they not only introduced a <a href="http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ini_Parameters" id="link-id136763c0">lite-mode</a> which makes Virtuoso suitable for the desktop but also helped in debugging all the problems that I had left. Many test runs, patches, and a Virtuoso 5.0.12 release later <a href="http://trueg.wordpress.com/2009/10/22/virtuoso-once-more-with-feeling/" id="link-id13c5a5a0">I could finally announce the Virtuoso integration</a> as usable.</p> <p>Then end of last year I dropped the support for sesame2 and redland. Virtuoso is now the only supported database backend. The reason is simple: Virtuoso is way more powerful than the rest - not only in terms of performance - and it is fully implemented in <a href="http://dbpedia.org/resource/C%2B%2B" id="link-id13a17cd8">C</a>(++) without any traces of Java. Maybe even more important is the integration of the full text index which makes the previously used CLucene index unnecessary. Thus, we can finally combine full text and graph queries in one SPARQL query. This results in a cleaner API and way faster return of search results since there is no need to combine the results from several queries anymore. A direct result of that is the new <a href="http://api.kde.org/4.x-api/kdelibs-apidocs/nepomuk/html/namespaceNepomuk_1_1Query.html" id="link-id149a9fd8">Nepomuk Query API</a> which I will discuss later.</p> <p>So now the only thing I am waiting for is the first bugfix release of Virtuoso 6, i.e. 6.0.1 which will fix the bugs that make 6.0.0 fail with Nepomuk. Should be out any day now. :)</p> <h2>The Nepomuk Query API</h2> <p>Querying <a href="http://dbpedia.org/resource/Data">data</a> in Nepomuk pre-KDE-4.4 could be done in one of two ways: 1. Use the very limited capabilities of the <a href="http://api.kde.org/4.x-api/kdelibs-apidocs/nepomuk/html/classNepomuk_1_1ResourceManager.html" id="link-id139ad3d0">ResourceManager</a> to list resources with certain properties or of a certain type; or 2. Write your own <a href="http://techbase.kde.org/Development/Tutorials/Metadata/Nepomuk/AdvancedQueries" id="link-id13c74608">SPARQL query using ugly QString::arg replacements</a>.</p> <p>With the introduction of Virtuoso and its awesome power we can now do pretty much everything in one query. This allowed <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id13c4cf18">me</a> to finally create a query API for KDE: <a href="http://api.kde.org/4.x-api/kdelibs-apidocs/nepomuk/html/classNepomuk_1_1Query_1_1Query.html" id="link-id602e818">Nepomuk::Query::Query</a> and friends. I won’t go into much detail here since I did that <a href="http://trueg.wordpress.com/2009/12/07/convenient-querying-in-libnepomuk/" id="link-id11282ff8">before</a>.</p> <p>All in all you should remember one thing: whenever you think about writing your own SPARQL query in a KDE application - have a look at libnepomukquery. It is very likely that you can avoid the hassle of debugging a query by using the query API.</p> <p>The first nice effect of the new API (apart from me using it all over the place obviously) is the new query interface in Dolphin. Internally it simply combines a bunch of <a href="http://api.kde.org/4.x-api/kdelibs-apidocs/nepomuk/html/classNepomuk_1_1Query_1_1Term.html" id="link-id11952270">Nepomuk::Query::Term</a> objects into a <a href="http://api.kde.org/4.x-api/kdelibs-apidocs/nepomuk/html/classNepomuk_1_1Query_1_1AndTerm.html" id="link-id13aa85b8">Nepomuk::Query::AndTerm</a>. All very readable and no ugly query strings.</p> <div id="attachment_234" class="wp-caption aligncenter" style="width: 610px"> <a href="http://trueg.files.wordpress.com/2010/01/dolphin-kde-4-4-search-panel.png" id="link-id11454028"><img class="size-full wp-image-234" title="Dolphin Search Panel in KDE SC 4.4" src="http://trueg.files.wordpress.com/2010/01/dolphin-kde-4-4-search-panel.png?w=600&h=208" alt="" width="600" height="208" /> </a> <p class="wp-caption-text">Dolphin Search Panel in KDE SC 4.4</p> </div> <h2>Shared Desktop Ontologies</h2> <p>An important part of the <a href="http://nepomuk.semanticdesktop.org/" id="link-id13a35a90">Nepomuk research project</a> was the creation of a set of <a href="http://www.semanticdesktop.org/ontologies/" id="link-id123a6700">ontologies</a> for describing desktop resources and their metadata. After the <a href="http://xesam.org/main/XesamAbout" id="link-id13c70ab8">Xesam</a> project under the umbrella of <a href="http://www.freedesktop.org/" id="link-id139e2108">freedesktop.org</a> had been convinced to use RDF for describing file metadata they developed their own ontology. Thanks to Evgeny (phreedom) Egorochkin and Antonie Mylka both the Xesam ontology and the Nepomuk <a href="http://dbpedia.org/resource/Information" id="link-id119be318">Information</a> Elements Ontology were already very close in design. Thus, it was relatively easy to merge the two and be left with only one ontology to support. Since then not only KDE but also <a href="http://strigi.sourceforge.net/" id="link-id123b63f0">Strigi</a> and <a href="http://projects.gnome.org/tracker/" id="link-id13d02a30">Tracker</a> are using the Nepomuk ontologies.</p> <p>At the Gran Canaria Desktop Summit I met some of the guys from Tracker and we tried to come up with a plan to create a joint project to maintain the ontologies. This got off to a rough start as nobody really felt responsible. So I simply took the initiative and released the <a href="http://sourceforge.net/projects/oscaf/files/" id="link-id148d7078">shared-desktop-ontologies</a> version 0.1 in November 2009. The result was a s***-load of hate-mails and bug reports due to me breaking KDE build. But in the end it was worth it. Now the package is established and other projects can start to pick it up to create data compatible to the Nepomuk system and Tracker.</p> <p>Today the ontologies (and the shared-desktop-ontologies package) are maintained in the <a href="http://sourceforge.net/apps/trac/oscaf/" id="link-id10ce1038">Oscaf project at Sourceforge</a>. The situation is far from perfect but it is a good start. If you need specific properties in the ontologies or are thinking about creating one for your own application - come and join us in the <a href="http://sourceforge.net/apps/trac/oscaf/report/1" id="link-id11413910">bug tracker</a>…</p> <h2>Timeline KIO Slave</h2> <p>It was at the Akonadi meeting that Will Stephenson and myself got into talking about mimicking some <a href="http://live.gnome.org/Zeitgeist" id="link-id116888b0">Zeitgeist</a> functionality through Nepomuk. Basically it meant gathering some data when opening and when saving files. We quickly came up with a hacky patch for KIO and <a href="http://api.kde.org/4.x-api/kdelibs-apidocs/kio/html/classKFileDialog.html" id="link-id13637348">KFileDialog</a> which covered most cases and allowed us to track when a file was modified and by which application. This little experiment did not leave that state though (it will, however, this year) but another one did: Zeitgeist also provides a fuse filesystem which allows to browse the files by modification dates. Well, whatever fuse can do, KIO can do as well. <a href="http://trueg.wordpress.com/2009/10/26/just-another-way-of-browsing-your-files/" id="link-id13cf58c0">Introducing the timeline:/ KIO slave</a> which gives a calendar view onto your files.</p> <p> <a href="http://trueg.wordpress.com/2009/10/26/just-another-way-of-browsing-your-files/" id="link-id113d4988"><img class="aligncenter size-medium wp-image-208" title="timeline-october" src="http://trueg.files.wordpress.com/2009/10/timeline-october.png?w=300&h=235" alt="" width="300" height="235" /> </a> </p> <h2>Tips And Tricks</h2> <p>Well, I thought I would mention the <a href="http://techbase.kde.org/Development/Tutorials/Metadata/Nepomuk/TipsAndTricks" id="link-id116357d0">Tips And Tricks</a> section I wrote for the <a href="http://techbase.kde.org/Development/Tutorials/Metadata/Nepomuk" id="link-id14473520">techbase</a>. It might not be a big deal but I think it contains some valuable information in case you are using Nepomuk as a developer.</p> <h2>Google Summer Of Code 2009</h2> <p>This time around I had the privilege to <a href="http://trueg.wordpress.com/2009/04/22/nepomuk-in-the-summer-x2/" id="link-id116b0cf8">mentor two students</a> in the Google Summer of Code. Alessandro Sivieri and Adam Kidder did outstanding work on <a href="http://trueg.wordpress.com/2009/08/25/gsoc-wrap-up-part-1/" id="link-id13c9f2f8">Improved Virtual Folders</a> and the <a href="http://trueg.wordpress.com/2009/08/28/gsoc-wrap-up-part-2/" id="link-id123bac00">Smart File Dialog</a>.</p> <p>Adam’s work lead me to some heavy improvements in the Nepomuk KIO slaves myself which I only finished this week (more details on that coming up). Alessandro continued his work on faceted file browsing in KDE and created:</p> <h3>Sembrowser</h3> <p>Alessandro is following up on his work to make faceted file browsing a reality in 2010 (and KDE SC 4.5). Since it was too late to get faceted browsing into KDE SC 4.4 he is working on <a href="http://www.kde-apps.org/content/show.php/Sembrowser?content=117692" id="link-id117c67d0">Sembrowser</a>, a stand-alone faceted file browser which will be the grounds for experiments until the code is merged into Dolphin.</p> <div id="attachment_238" class="wp-caption aligncenter" style="width: 310px"> <a href="http://trueg.files.wordpress.com/2010/01/sembrowser.png" id="link-id13aa8e80"><img class="size-medium wp-image-238" title="sembrowser" src="http://trueg.files.wordpress.com/2010/01/sembrowser.png?w=300&h=189" alt="" width="300" height="189" /> </a> <p class="wp-caption-text">Faceted Browsing in KDE with Sembrowser</p> </div> <h2>Nepomuk Workshops</h2> <p>In 2009 I organized the first Nepomuk workshop in Freiburg, Germany. And also the second one. While <a href="http://trueg.wordpress.com/2009/06/23/the-first-nepomuk-workshop-its-a-wrap/" id="link-id13b553e0">I reported properly on the first one</a> I still owe a summary for the second one. I will get around to that - sooner or later. ;)</p> <h2>CMake Magic</h2> <p> <a href="http://soprano.sourceforge.net/" id="link-id148bfad8">Soprano</a> gives us a nice command line tool to create a C++ namespace from an ontology file: <a href="http://soprano.sourceforge.net/apidox/trunk/soprano_devel_tools.html" id="link-iddac3b58">onto2vocabularyclass</a>. It produces nice convenience namespaces like <a href="http://soprano.sourceforge.net/apidox/trunk/namespaceSoprano_1_1Vocabulary_1_1NAO.html" id="link-idfd4b970">Soprano::Vocabulary::NAO</a>. Nepomuk adds another tool named <a href="http://techbase.kde.org/Development/Tutorials/Metadata/Nepomuk/ResourceGenerator" id="link-id11b60200">nepomuk-rcgen</a>. Both were a bit clumsy to use before. Now we have nice cmake macros which make it very simple to use both.</p> <p>See the <a href="http://techbase.kde.org/Development/Tutorials/Metadata/Nepomuk/ResourceGenerator" id="link-id11963490">techbase article</a> on how to use the new macros.</p> <h2>Bangarang</h2> <p>Without my <a href="http://dbpedia.org/resource/Knowledge" id="link-iddcbd7c8">knowledge</a> (imagine that!) Andrew Lake created an amazing new media player named <a href="http://bangarangkde.wordpress.com/" id="link-id113d9500">Bangarang</a> - <em>a Jamaican word for noise, chaos or disorder.</em> This player is Nepomuk-enabled in the sense that it has a media library which lets you browse your media files based on the Nepomuk data. It remembers the number of times a song or a video has been played and when it was played last. It allows to add detail such as the TV series name, season, episode number, or actors that are in the video - all through Nepomuk (I hope we will soon get <a href="http://thetvdb.com/" id="link-id1154d7a0">tvdb</a> integration).</p> <div id="attachment_242" class="wp-caption aligncenter" style="width: 310px"> <a href="http://trueg.files.wordpress.com/2010/01/bangarang2.png" id="link-id148bcdb8"><img class="size-medium wp-image-242" title="bangarang2" src="http://trueg.files.wordpress.com/2010/01/bangarang2.png?w=300&h=208" alt="" width="300" height="208" /> </a> <p class="wp-caption-text">Edit metadata directly in Bangarang</p> </div> <div id="attachment_243" class="wp-caption aligncenter" style="width: 303px"> <a href="http://trueg.files.wordpress.com/2010/01/bangarang-dolphin-fileinfo.png" id="link-id11c70a48"><img class="size-full wp-image-243" title="bangarang-dolphin-fileinfo" src="http://trueg.files.wordpress.com/2010/01/bangarang-dolphin-fileinfo.png?w=293&h=242" alt="" width="293" height="242" /> </a> <p class="wp-caption-text">Dolphin showing TV episode metadata created by Bangarang</p> </div> <div id="attachment_245" class="wp-caption aligncenter" style="width: 310px"> <a href="http://trueg.files.wordpress.com/2010/01/bangarang-dolphin-search.png" id="link-id149200f8"><img class="size-medium wp-image-245" title="bangarang-dolphin-search" src="http://trueg.files.wordpress.com/2010/01/bangarang-dolphin-search.png?w=300&h=212" alt="" width="300" height="212" /> </a> <p class="wp-caption-text">And of course searching for it works, too...</p> </div> <div id="attachment_244" class="wp-caption aligncenter" style="width: 310px"> <a href="http://trueg.files.wordpress.com/2010/01/bangarang1.png" id="link-id114f7c80"><img class="size-medium wp-image-244" title="bangarang1" src="http://trueg.files.wordpress.com/2010/01/bangarang1.png?w=300&h=225" alt="" width="300" height="225" /> </a> <p class="wp-caption-text">And it is pretty, too...</p> </div> <p>I am especially excited about this since finally applications not written or mentored by me start contributing Nepomuk data.</p> <h2>Gran Canaria Desktop Summit</h2> <p>2009 was also the year of the first Gnome-KDE joint-conference. Let me make a bulletin for completeness and refer to <a href="http://trueg.wordpress.com/2009/07/13/gran-canaria-desktop-summit-2009-the-nepomuk-perspective/" id="link-id143ff668">my previous blog post reporting on my experiences on the island</a>.</p> <p>Well, that was by far not all I did in 2009 but I think I covered most of the important topics. And after all it is ‘just a blog entry’ - there is no need for completeness. Thanks for reading.</p> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/trueg.wordpress.com/232/" id="link-id118a1950"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/comments/trueg.wordpress.com/232/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godelicious/trueg.wordpress.com/232/" id="link-id148ffb08"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/delicious/trueg.wordpress.com/232/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gostumble/trueg.wordpress.com/232/" id="link-id13c65a88"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/stumble/trueg.wordpress.com/232/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/godigg/trueg.wordpress.com/232/" id="link-id119b85a0"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/digg/trueg.wordpress.com/232/" /></a> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/goreddit/trueg.wordpress.com/232/" id="link-id13f5d6b8"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/reddit/trueg.wordpress.com/232/" /></a> <img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=trueg.wordpress.com&blog=6648236&post=232&subd=trueg&ref=&feed=1" /> </div>"
2010-02-01T09:02:55-05:00
One Technology That Will Rock 2010 (Update 1)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2010-01-02#1601
2010-01-02T17:30:38Z
<p>Thanks to the <a href="http://www.techcrunch.com/" id="link-id114eb070">TechCrunch</a> post titled: <a href="http://www.techcrunch.com/2010/01/01/ten-technologies-2010/" id="link-id1146e550">Ten Technologies That Will Rock 2010</a>, I've been able to quickly construct a derivative post that condenses the ten item list down to a Single Technology That Will Rock 2010 :-)</p> <p>Sticking with the TechCrunch layout, here is why all roads simply lead to <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11141d50">Linked Data</a> come 2010 and beyond: </p> <ol> <li> <strong>The Tablet: </strong>a new form factor addition re. <a href="http://dbpedia.org/resource/Internet" id="link-id13f09418">Internet</a> and <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> application hosts which is just another way of saying: Linked Data will be accessible from Tablet applications.</li> <li> <strong>Geo:</strong> GPS chips are now standard features of mobile phones, so <a href="http://www.techcrunch.com/2009/12/23/location-2010/" id="link-id112cfdd0">geolocation</a> is increasingly becoming a necessary feature for any killer app. Thus, GeoSpatial Linked Data and GeopSpatial Queries are going to be a critical success factor for any endeavor that seeks to engage mobile applications developers and ultimately their end-users. Basiacally, you want to be able to perform Esoteric Search from these devices of the form: Find Vendors of a Camcorder (e.g., with a Zoom Factor: Weight Ratio of X) within a 2km Radius of my current location. Or how many items from my WishList are available from a Vendor within a 2km radius of my current location. Conversely, provide Vendors with the ability to spot potential Customers within a 2km of a given "clicks & mortar" location (e.g. BestBuy store).</li> <li> <strong>Realtime Search: </strong>Rich Structured Profiles that leverage standards such as <a href="http://dbpedia.org/resource/Friend_of_a_friend" id="link-id140ece38">FOAF</a> and <a href="http://blogs.sun.com/bblfish/entry/foaf_ssl_creating_a_global" id="link-id11856318">FOAF+SSL</a> will enable Highly Personalized Realtime Search (HPRS) without compromisng privacy. Tecnically, this is about <a href="http://esw.w3.org/topic/WebID" id="link-id13ec6260">WebID</a>s securely bound to X.509 Certificates, providing access to verifiable and highly navigable Personal Profile <a href="http://dbpedia.org/resource/Data">Data</a> Spaces that also double as personal search index entry points.</li> <li> <strong>Chrome OS: </strong>Just another operating system for exploiting the burgeoning Web of Linked Data</li> <li> <strong>HTML5: </strong>Courtesy of <a href="http://dbpedia.org/resource/RDFa" id="link-id115b08f0">RDFa</a>, just another mechanism for exposing Linked Data by making HTML+RDFa a bona fide markup for <a href="http://dbpedia.org/resource/Metadata" id="link-id1195b070">metadata</a> (i.e., format for describing real world objects via their attribute-value graphs)</li> <li> <strong>Mobile Video:</strong> Simplifies the production and sharing of Video annotations (comments, reviews etc.) en route to creating rich Linked Discourse Data Spaces.</li> <li> <strong>Augmented Reality:</strong> Ditto</li> <li> <strong>Mobile Transactions:</strong> As per points 1&2 above, Vendor Discovery and Transaction Conusmation will increasingly be driven by high SDQ applications. The "Funnel Effect" (more choices based on individual preferences) will be a critical success factor for any one operating in the Mobile Transaction realm. Note, without Linked Data you cannot deliver scalable solutions that handle the combined requirements of: SDQ, "Funnel Effect", and Mobile Device form factor, will simply maginify the importance of Web accessible Linked Data.</li> <li> <strong>Android:</strong> An additional platform for items 1-8; basically, 2010 isn't going to be an iPhone only zone. Personally, this reminds <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id111ab5e8">me</a> of a battle from the past i.e., Microsoft vs Apple, re. desktop computing dominance. Google has studied history very well :-)</li> <li> <strong>Social CRM:</strong> this is simply about applying points 1-9 alongide the construction of Linked Data from eCRM Data Spaces.</li> </ol> <p>As I've stated in the past (across a variety of mediums), you cannot build applications that have long term value without addressing the following issues:</p> <ol> <li>Data Item or Object Identity</li> <li>Data Structure -- Data Models</li> <li>Data Representation -- Data Model <a href="http://dbpedia.org/resource/Entity" id="link-id1148eaf8">Entity</a> & Relationships Representation mechanism (as delivered by metadata oriented markup)</li> <li>Data Storage -- Database Management Systems</li> <li>Data Access -- Data Access Protocols </li> <li>Data Presentation -- How you present Views and Reports from Structured Data Sources</li> <li>Data Security -- Data Access Policies</li> </ol> <p>The items above basically showcase the very essence of the HTTP <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id1239af68">URI</a> abstraction that drives HTTP based Linked Data; which is also the basic payload unit that underlies <a href="http://dbpedia.org/resource/Representational_State_Transfer" id="link-id11489a98">REST</a>.</p> <h3>Conclusion</h3> <p>I simply hope that the next decade marks a period of broad appreciation and comprehension of Data Access, Integration, and Management issues on the parts of: application developers, integrators, analysts, end-users, and decision makers. Remember, without structured Data we cannot produce or share <a href="http://dbpedia.org/resource/Information" id="link-id13cb5040">Information</a>, and without Information, we cannot produce of share <a href="http://dbpedia.org/resource/Knowledge" id="link-id647abb0">Knowledge</a>.</p> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1567" id="link-id13fa3a20">HTTP URI Abstraction and Linked Data</a> </li> <li> <a href="http://www.dataflux.com/dfblog/?p=1458," id="link-id138f3ea8">First Law of Data Quality</a> </li> <li> <a href="http://walkingoncoals.blogspot.com/2009/12/whos-data-is-it-part-1.html" id="link-id13efccb8">Who's Data Is It?</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1442" id="link-id1355df68">Serendipitous Discovery Quotient</a> (SDQ)</li> <li> <a href="http://www.seangolliher.com/2009/linked-data/serendipitous-discovery-quotient-sdq-the-future-of-seo-or-an-abstract-concept/" id="link-id11217cb8">SDQ: The Future of SEO or an Abstract Concept?</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1587" id="link-id139cfbe0">SPARQL & GeoSpatial Indexing</a> (implications of <a href="http://dbpedia.org/resource/SPARQL" id="link-id13f51b78">SPARQL</a>-GEO)</li> <li> <a href="http://blog.jonudell.net/2009/09/09/talking-with-kingsley-idehen-about-mastering-your-own-search-index/" id="link-id13c5c248">Mastering Your Own Search Index</a> </li> <li> <a href="http://blog.jonudell.net/2009/11/23/talking-with-martin-hepp-about-solving-the-paradox-of-choice/" id="link-id135ba4d0">Solving the Paradox of Choice</a>.</li> </ul>
2010-02-01T09:02:41-05:00
Linked Data and Virtuoso in 2010
http://www.openlinksw.com:443/weblog/oerling/?date=2009-12-29#1603
2009-12-29T15:24:11Z
<p>It is again time for the end-of-year <a href="http://dbpedia.org/resource/Blog" id="link-id0x1f82a100">blog</a> post.</p> <p>In 2009, <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1dab31a0">RDF</a> <a href="http://www.openlinksw.com/weblog/oerling/?tag=scalability" id="link-id11951c50">scalability questions</a> were solved in their broad outline and the corresponding <a href="http://virtuoso.openlinksw.com" id="link-id0x116a7b20">Virtuoso</a> release was built and used in production internally. Its general availability is now imminent while it has been available on a case-by-case basis thus far.</p> <p>In 2010, we take on a new challenge: To bring RDF closer to parity with equivalent relational solutions. This will also entail some significant improvements to our relational technology.</p> <p>Storage density is a key ingredient of performance. Some of the advances will be in this area; other advances will be in increased parallelism of execution. Right now we run things in vectored batches in cluster situations where message latency forces operations to be shipped in large chunks. Next we will do this across the board, also in single servers. The advantages of this for <a href="http://dbpedia.org/resource/Cache" id="link-id0x1c7819f0">cache</a> behavior and other factors are known in the literature.</p> <p>Looking at environmental factors, we have a new <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1ed04da0">SPARQL</a> at a Working Draft stage. We have basic parity with <a href="http://dbpedia.org/resource/SQL" id="link-id0x1cd94b40">SQL</a> expressivity, which is a prerequisite for RDF to become a <a href="http://dbpedia.org/resource/Data" id="link-id0x16f2d758">data</a> model that can be an alternative to relational outside of very specialized contexts.</p> <p>As the standards process makes SPARQL closer to being an alternative to SQL for data integration, we will make the database engine technology such that RDF's inherent penalty in terms of storage overhead and processing time substantially decreases. This will make RDF a workable integration medium also in places where it was not such before. Of course, an application-specific <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x1e670768">schema</a> will retain some advantage over a generic one, but then one can have a purely relational application on Virtuoso as well. Just think of the possibility of an application-specific schema emerging by itself in a workload-driven fashion.</p> <p>As background data for an increasing number of fields becomes available as <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1e6e0848">linked data</a>, using this together with proprietary data for analytics and discovery becomes increasingly interesting. This is the initial line of RDF data warehousing. The biomedical field has many examples. The technologies we will release during 2010 will be geared towards enabling a second line of RDF applications, where ad hoc agile integration with RDF as a lingua franca becomes a real alternative to relational solutions with ETL point solutions for harvesting <a href="http://dbpedia.org/resource/Information" id="link-id0x1e22a3a0">information</a> from diverse systems. One may see how RDF's flexibility and expressivity may add to agility in any number of situations where data from heterogenous sources needs to be integrated. Which of today's business scenarios does not face this issue?</p> <p> <b>References:</b> </p> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1510" id="link-id13a91d58">Linked Data & The Year 2009</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1286" id="link-id11302a68">Retrospective and Outlook for 2008</a> </li> <li>Other <a href="http://www.openlinksw.com/weblog/oerling/?tag=scalability" id="link-id13c50578">Scalability</a> and <a href="http://www.openlinksw.com/weblog/oerling/?tag=benchmarking" id="link-id13a2d8a8">Benchmarking</a> posts</li> </ul>
2010-02-01T09:14:21.000002-05:00
The Business Of Linked Data (BOLD) Discussion Space
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-12-04#1600
2009-12-04T19:40:08Z
<p>I've created a new discussion space that's squarely focused on the business development and marketing aspects of "HTTP based <a href="http://dbpedia.org/resource/Linked_Data" id="link-id129e32d8">Linked Data" (Linked Data</a>). As its name indicates, It's a BOLD attempt to fill a VoiD. :-)</p> <h3>Background</h3> <p>A few months ago, <a href="http://blog.aldobucchi.com/#this" id="link-id1110eb30">Aldo Bucchi</a> posted a message to the <a href="http://esw.w3.org/topic/SweoIG/TaskForces/CommunityProjects/LinkingOpenData" id="link-id111d08a0">LOD</a> <a href="http://lists.w3.org/Archives/Public/public-lod/" id="link-id118b3778">mailing list</a> seeking a discussion space for more business and marketing oriented topic, in relation to Linked Data. At the time, my assumption was that the existing LOD mailing list served that purpose absolutely fine, but in due course I came to realize that Aldo's request had a much lager foundation than I initially suspected.</p> <h3>Historic Oversight</h3> <p>Linked Data, like its umbrella <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id16ceb618">Semantic Web</a> Project, has suffered from an inadvertent oversight on the parts of many of its enthusiasts (myself included): 100% of the discussion spaces are created by, geared towards, or dominated by researchers (from Academia primarily) and/or developers. Thus, at the very least, we've been operating in an echo chamber that only feed the existing void between the core community and those who are more interested in discussing business and marketing related topics.</p> <p>The new discussion space seeks to cover the following:</p> <ol> <li> Brainstorming Value Proposition Articulation</li> <li>War Story Exchanges</li> <li>Case Studies and Use-cases</li> <li>Market Research & Positioning (for instance Linked Data is killer technology that redefines <a href="http://dbpedia.org/resource/Data">Data</a> Integration, but none of the major research firms currently make that connection)</li>. </ol> <p>How Do I Join The Conversation? Simply sign up on the Google hosted <a href="http://groups.google.com/group/business-of-linked-data-bold" id="link-id129e4d08">BOLD mailing list</a>, introduce yourself (ideally), and then start conversing! :-)</p>
2010-02-01T09:02:27.000001-05:00
Personal and/or Service Specific Linked Data Spaces in the Cloud: DBpedia 3.4
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-11-16#1589
2009-11-16T18:17:46Z
<p> We have just released an Amazon EC2 based public Snapshot of <a href="http://dbpedia.org/resource/DBpedia" id="link-id18e899b8">DBpedia</a> 3.4. Thus, you can now instantiate a personal and/or service specific variant of the DBpedia 3.4 <a href="http://dbpedia.org/resource/Linked_Data" id="link-id168dec90">Linked Data</a> <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id18911268">Space</a>. Basically, you can replicate what we host, within minutes (as opposed to days). In addition, you no longer need to squabble --on an unpredictable basis with others-- for the infrastructure resources behind DBpedia's public instance, when using the <a href="http://dbpedia.org/resource/SPARQL" id="link-id18d5bd78">SPARQL</a> Endpoint, Faceted Search & Find Services, or HTML Browser Pages etc.</p> <h3>How Does It work?</h3> <ol> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtInstallationEC2" id="link-id115932b8">Instantiate a Virtuoso EC2 AMI</a> (paid variety, which is aggressively priced at $49.99 for setup and $19.99 per month thereafter)</li> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtAWSDBpedia34S" id="link-id182dc800"> Mount the shared DBpedia 3.4 public snapshot</a> </li> <li> Start Virtuoso Server</li> <li> Start exploiting the DBpedia Linked Data Space.</li> </ol> <h3>What Interfaces are exposed?</h3> <ol> <li> SPARQL Endpoint</li> <li> Linked Data Viewer Pages (as you see in the public DBpedia instance)</li> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtFacetBrowserInstallConfig" id="link-id117f6e80">Faceted Search & Find UI and Web Services</a> (REST or SOAP)</li> <li> All the inference rules for <a href="http://umbel.org/about/" id="link-id144b84a8">UMBEL</a>, SUMO, YAGO, <a href="http://dbpedia.org/resource/Cyc" id="link-id16b69da8">OpenCYC</a>, and DBpedia-OWL <a href="http://dbpedia.org/resource/Data">data</a> dictionaries</li> <li>Type Correlations Between DBpedia and Freebase</li> </ol> <p>Enjoy!</p>
2009-11-16T13:30:20-05:00
Personal and/or Service Specific Linked Data Spaces in the Cloud: DBpedia 3.4
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-11-16#1599
2009-11-16T18:17:46Z
<p> We have just released an Amazon EC2 based public Snapshot of <a href="http://dbpedia.org/resource/DBpedia" id="link-id18e899b8">DBpedia</a> 3.4. Thus, you can now instantiate a personal and/or service specific variant of the DBpedia 3.4 <a href="http://dbpedia.org/resource/Linked_Data" id="link-id168dec90">Linked Data</a> <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id18911268">Space</a>. Basically, you can replicate what we host, within minutes (as opposed to days). In addition, you no longer need to squabble --on an unpredictable basis with others-- for the infrastructure resources behind DBpedia's public instance, when using the <a href="http://dbpedia.org/resource/SPARQL" id="link-id18d5bd78">SPARQL</a> Endpoint, Faceted Search & Find Services, or HTML Browser Pages etc.</p> <h3>How Does It work?</h3> <ol> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtInstallationEC2" id="link-id115932b8">Instantiate a Virtuoso EC2 AMI</a> (paid variety, which is aggressively priced at $49.99 for setup and $19.99 per month thereafter)</li> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtAWSDBpedia34S" id="link-id182dc800"> Mount the shared DBpedia 3.4 public snapshot</a> </li> <li> Start Virtuoso Server</li> <li> Start exploiting the DBpedia Linked Data Space.</li> </ol> <h3>What Interfaces are exposed?</h3> <ol> <li> SPARQL Endpoint</li> <li> Linked Data Viewer Pages (as you see in the public DBpedia instance)</li> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtFacetBrowserInstallConfig" id="link-id117f6e80">Faceted Search & Find UI and Web Services</a> (REST or SOAP)</li> <li> All the inference rules for <a href="http://umbel.org/about/" id="link-id144b84a8">UMBEL</a>, SUMO, YAGO, <a href="http://dbpedia.org/resource/Cyc" id="link-id16b69da8">OpenCYC</a>, and DBpedia-OWL <a href="http://dbpedia.org/resource/Data">data</a> dictionaries</li> <li>Type Correlations Between DBpedia and Freebase</li> </ol> <p>Enjoy!</p>
2010-02-01T08:58:14-05:00
RDF Geography With Virtuoso
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-11-11#1588
2009-11-11T17:17:27Z
<p>We have just added a geometry <a href="http://dbpedia.org/resource/Data" id="link-id0x1c0e02b0">data</a> type and corresponding <a href="http://dbpedia.org/resource/R-tree" id="link-id0x1e093220">R</a>-tree index to <a href="http://virtuoso.openlinksw.com" id="link-id0x1ddccfe8">Virtuoso</a>. This follows the general scheme of <a href="http://dbpedia.org/resource/SQL" id="link-id0x1b88a580">SQL</a>/MM, as is implemented by <a href="http://dbpedia.org/resource/PostGIS" id="link-id0x1d271a90">PostGIS</a> and many others. We have all the engine-side stuff, including optimizer support for geometry cardinality sampling and good execution plans for combinations of spatial and other joins. We have however not yet implemented all the different geometry types and library function support for them, like shortest distance between two arbitrary shapes.</p> <p>The geometry support is for both SQL and <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1b8d4ca8">SPARQL</a>. On the SQL side, it works with the ISO/IEC 13249 SQL/MM API; with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1ed69318">RDF</a>, a geometry can occur as the object of a quad. If the object is a typed-literal of the <code>virtrdf:Geometry</code> type, it gets indexed in a geometry index over all geometries in quads; no special declarations are needed. After this, SQL MM predicates and functions can be used with SPARQL, like this:</p> <blockquote> <pre><code> PREFIX geo: <<a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1d2d0ae0">http</a>://www.w3.org/2003/01/geo/wgs84_pos#> SELECT ?class COUNT (*) WHERE { ?m geo:geometry ?geo . ?m a ?class . FILTER ( <bif:st_intersects> ( ?geo, <bif:st_point> (0, 52), 100 ) ) } GROUP BY ?class ORDER BY DESC 2 </code> </pre></blockquote> <p>This returns the counts of objects of each class occurring within 100 km of (0, 52), a point near London.</p> <p>For any data set with <a href="http://dbpedia.org/resource/World_Geodetic_System" id="link-id0x1ec00578">WGS 84</a> <code>geo:long</code> and <code>geo:lat</code> values, a simple SQL function makes a point geometry for each such coordinate pair and adds it as the <code>geo:geometry</code> property of the subject with the long/lat. This then enables fast spatial access to arbitrary location data in RDF.</p> <p>Right now, we hardly see any geometries other than points in RDF data, even though there are some efforts for vocabularies for more complex entities. As these get adopted we will support them.</p> <p>For scalability, we tried the implementation with <a href="http://www.openstreetmap.org/" id="link-id0x1c781e68">OpenStreetMap</a>'s 350 million or so points. The geometry implementation partitions well over a cluster, similarly to a full text index, i.e., every server has its slice of the geometries, partitioned by the geometry object's key, thus not by range of coordinates or such. Like this, the items are evenly spread even though the coordinate distribution is highly uneven.</p> <p>We can do spatial joins like —</p> <blockquote> <pre><code> SELECT ?s ( <sql:num_or_null> (?p) ) COUNT (*) WHERE { ?s <http://<a href="http://dbpedia.org/resource/DBpedia" id="link-id0x1f885868">dbpedia</a>.org/ontology/populationTotal> ?p . FILTER ( <sql:num_or_null> (?p) > 1000000 ) . ?s geo:geometry ?geo . FILTER ( <bif:st_intersects> ( ?pt, ?geo, 5 ) ) . ?xx geo:geometry ?pt } GROUP BY ?s ( <sql:num_or_null> (?p) ) ORDER BY DESC 3 LIMIT 20 </code> </pre></blockquote> <p>This takes the DBpedia subjects that have a population over 1 million and a geometry. We then count all the geometries within 5 km of the point location of the first geometry. With DBpedia (about 5 million points), <a href="http://www.geonames.org/" id="link-id0x1d4279b0">GeoNames</a> (7 million points), and OpenStreetMap (350 million points), we get the result:</p> <blockquote> <pre><code>http://dbpedia.org/resource/Munich 1356594 117280 http://dbpedia.org/resource/London 7355400 81486 http://dbpedia.org/resource/Davao_City 1363337 58640 http://dbpedia.org/resource/Belo_Horizonte 2412937 58640 http://dbpedia.org/resource/Chengde 3610000 58640 http://dbpedia.org/resource/Hamburg 1769117 51664 http://dbpedia.org/resource/San_Diego%2C_California 1266731 47685 http://dbpedia.org/resource/Bursa 1562828 47685 http://dbpedia.org/resource/Port-au-Prince 1082800 47685 http://dbpedia.org/resource/Oakland_County%2C_Michigan 1194156 45636 http://dbpedia.org/resource/Sana%27a 1747627 40923 http://dbpedia.org/resource/Milan 1303437 40923 http://dbpedia.org/resource/Campinas 1059420 40923 http://dbpedia.org/resource/Hohhot 2580000 40923 http://dbpedia.org/resource/Brussels 1031215 40923 http://dbpedia.org/resource/Bogra_District 2988567 40923 http://dbpedia.org/resource/Cort%C3%A9s_Department 1202510 40923 http://dbpedia.org/resource/Berlin 3416300 35668 http://dbpedia.org/resource/New_York_City 8274527 30810 http://dbpedia.org/resource/Los_Angeles%2C_California 3849378 25614<br /> 20 Rows. -- 1733 msec.<br /> Cluster 8 nodes, 1 s. 358 m/s 1596 KB/s 664% <a href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x1e6403b0">cpu</a> 2% read 16% clw threads 1r 0w 0i buffers 1124351 0 d 0 w 0 pfs </code></pre></blockquote> <p>This takes 1.7 seconds on a Virtuoso Cluster configured with 8 processes on a single dual-Xeon 5520 box, running at about 664% CPU with warm <a href="http://dbpedia.org/resource/Cache" id="link-id0x1e81f610">cache</a>. Fair enough for a first crack, this can obviously be optimized further. Still, the geo part of the processing is already as good as instantaneous.</p> <p>We will shortly have the geography features installed on DBpedia and the other data sets we host. As these come online we will show more demo queries.</p> <p>For more about SQL/MM, you can look to a couple of PDFs:</p> <ul> <li> <a href="http://www.fer.hr/_download/repository/SQLMM_Spatial-_The_Standard_to_Manage_Spatial_Data_in_Relational_Database_Systems.pdf" id="link-id133775f0">SQL/MM Spatial: The Standard to Manage Spatial Data in Relational Database Systems</a> by Knut Stolze</li> <li> <a href="http://www.sigmod.org/record/issues/0112/standards.pdf" id="link-id1433c5e0">SQL Multimedia and Application Packages (SQL/MM)</a> by Jim Melton and Andrew Eisenberg</li> </ul>
2010-02-01T09:14:29.000012-05:00
RDF Geography With Virtuoso
http://www.openlinksw.com:443/weblog/oerling/?date=2009-11-11#1587
2009-11-11T17:17:27Z
<p>We have just added a geometry <a href="http://dbpedia.org/resource/Data" id="link-id0x1c728fb0">data</a> type and corresponding <a href="http://dbpedia.org/resource/R-tree" id="link-id0x1d0e9368">R</a>-tree index to <a href="http://virtuoso.openlinksw.com" id="link-id0x1e999000">Virtuoso</a>. This follows the general scheme of <a href="http://dbpedia.org/resource/SQL" id="link-id0x1c1d2790">SQL</a>/MM, as is implemented by <a href="http://dbpedia.org/resource/PostGIS" id="link-id0x1c682eb8">PostGIS</a> and many others. We have all the engine-side stuff, including optimizer support for geometry cardinality sampling and good execution plans for combinations of spatial and other joins. We have however not yet implemented all the different geometry types and library function support for them, like shortest distance between two arbitrary shapes.</p> <p>The geometry support is for both SQL and <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1bba2728">SPARQL</a>. On the SQL side, it works with the ISO/IEC 13249 SQL/MM API; with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1b28c330">RDF</a>, a geometry can occur as the object of a quad. If the object is a typed-literal of the <code>virtrdf:Geometry</code> type, it gets indexed in a geometry index over all geometries in quads; no special declarations are needed. After this, SQL MM predicates and functions can be used with SPARQL, like this:</p> <blockquote> <pre><code> PREFIX geo: <<a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1e574ed8">http</a>://www.w3.org/2003/01/geo/wgs84_pos#> SELECT ?class COUNT (*) WHERE { ?m geo:geometry ?geo . ?m a ?class . FILTER ( <bif:st_intersects> ( ?geo, <bif:st_point> (0, 52), 100 ) ) } GROUP BY ?class ORDER BY DESC 2 </code> </pre></blockquote> <p>This returns the counts of objects of each class occurring within 100 km of (0, 52), a point near London.</p> <p>For any data set with <a href="http://dbpedia.org/resource/World_Geodetic_System" id="link-id0x1d7feae8">WGS 84</a> <code>geo:long</code> and <code>geo:lat</code> values, a simple SQL function makes a point geometry for each such coordinate pair and adds it as the <code>geo:geometry</code> property of the subject with the long/lat. This then enables fast spatial access to arbitrary location data in RDF.</p> <p>Right now, we hardly see any geometries other than points in RDF data, even though there are some efforts for vocabularies for more complex entities. As these get adopted we will support them.</p> <p>For scalability, we tried the implementation with <a href="http://www.openstreetmap.org/" id="link-id0x1cb79580">OpenStreetMap</a>'s 350 million or so points. The geometry implementation partitions well over a cluster, similarly to a full text index, i.e., every server has its slice of the geometries, partitioned by the geometry object's key, thus not by range of coordinates or such. Like this, the items are evenly spread even though the coordinate distribution is highly uneven.</p> <p>We can do spatial joins like —</p> <blockquote> <pre><code> SELECT ?s ( <sql:num_or_null> (?p) ) COUNT (*) WHERE { ?s <http://<a href="http://dbpedia.org/resource/DBpedia" id="link-id0x1d4179e8">dbpedia</a>.org/ontology/populationTotal> ?p . FILTER ( <sql:num_or_null> (?p) > 1000000 ) . ?s geo:geometry ?geo . FILTER ( <bif:st_intersects> ( ?pt, ?geo, 5 ) ) . ?xx geo:geometry ?pt } GROUP BY ?s ( <sql:num_or_null> (?p) ) ORDER BY DESC 3 LIMIT 20 </code> </pre></blockquote> <p>This takes the DBpedia subjects that have a population over 1 million and a geometry. We then count all the geometries within 5 km of the point location of the first geometry. With DBpedia (about 5 million points), <a href="http://www.geonames.org/" id="link-id0x19ee61b8">GeoNames</a> (7 million points), and OpenStreetMap (350 million points), we get the result:</p> <blockquote> <pre><code>http://dbpedia.org/resource/Munich 1356594 117280 http://dbpedia.org/resource/London 7355400 81486 http://dbpedia.org/resource/Davao_City 1363337 58640 http://dbpedia.org/resource/Belo_Horizonte 2412937 58640 http://dbpedia.org/resource/Chengde 3610000 58640 http://dbpedia.org/resource/Hamburg 1769117 51664 http://dbpedia.org/resource/San_Diego%2C_California 1266731 47685 http://dbpedia.org/resource/Bursa 1562828 47685 http://dbpedia.org/resource/Port-au-Prince 1082800 47685 http://dbpedia.org/resource/Oakland_County%2C_Michigan 1194156 45636 http://dbpedia.org/resource/Sana%27a 1747627 40923 http://dbpedia.org/resource/Milan 1303437 40923 http://dbpedia.org/resource/Campinas 1059420 40923 http://dbpedia.org/resource/Hohhot 2580000 40923 http://dbpedia.org/resource/Brussels 1031215 40923 http://dbpedia.org/resource/Bogra_District 2988567 40923 http://dbpedia.org/resource/Cort%C3%A9s_Department 1202510 40923 http://dbpedia.org/resource/Berlin 3416300 35668 http://dbpedia.org/resource/New_York_City 8274527 30810 http://dbpedia.org/resource/Los_Angeles%2C_California 3849378 25614<br /> 20 Rows. -- 1733 msec.<br /> Cluster 8 nodes, 1 s. 358 m/s 1596 KB/s 664% <a href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x120b8e30">cpu</a> 2% read 16% clw threads 1r 0w 0i buffers 1124351 0 d 0 w 0 pfs </code></pre></blockquote> <p>This takes 1.7 seconds on a Virtuoso Cluster configured with 8 processes on a single dual-Xeon 5520 box, running at about 664% CPU with warm <a href="http://dbpedia.org/resource/Cache" id="link-id0x1cb17380">cache</a>. Fair enough for a first crack, this can obviously be optimized further. Still, the geo part of the processing is already as good as instantaneous.</p> <p>We will shortly have the geography features installed on DBpedia and the other data sets we host. As these come online we will show more demo queries.</p> <p>For more about SQL/MM, you can look to a couple of PDFs:</p> <ul> <li> <a href="http://www.fer.hr/_download/repository/SQLMM_Spatial-_The_Standard_to_Manage_Spatial_Data_in_Relational_Database_Systems.pdf" id="link-id133775f0">SQL/MM Spatial: The Standard to Manage Spatial Data in Relational Database Systems</a> by Knut Stolze</li> <li> <a href="http://www.sigmod.org/record/issues/0112/standards.pdf" id="link-id1433c5e0">SQL Multimedia and Application Packages (SQL/MM)</a> by Jim Melton and Andrew Eisenberg</li> </ul>
2010-02-01T09:14:16.000001-05:00
European Commission and the Data Overflow
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-10-27#1586
2009-10-27T18:29:51Z
<p>The European Commission recently circulated a questionnaire to selected experts on what could be done for the future of big <a href="http://dbpedia.org/resource/Data" id="link-id0x43bae00">data</a>.</p> <p>Since the <a href="http://cordis.europa.eu/fp7/ict/content-knowledge/consultation_en.html" id="link-id1191c0f8">questionnaire is public</a>, I am publishing my answers below.</p> <ol type="1" start="1"> <li> <p> <b>Data and data types</b> </p> <ol type="a" start="1"> <li> <p> <b>What volumes of data are we dealing with today? What is the growth rate? Where can we expect to be in 2015? </b> </p> <p>Private data warehouses of corporations have more than doubled yearly for the past years; hundreds of TB is not exceptional. This will continue. The real shift is in structured data being published in increasing quantities with a minimum level of integrate-ability through use of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x5c7add0">RDF</a> and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x5c7adb8">linked data</a> principles. There are rewards for use of standard vocabularies and identifiers through search engines recognizing such data. There is convergence around <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x5c7ada0">DBpedia</a> identifiers for real-world entities, e.g., most things that would be in the news.</p> <p>This also means that internal data processes and silos may be enriched with this content. There is consequent pressure for accommodating more diversity of data, with more flexible <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x7d87a88">schema</a>.</p> <p>Ultimately, all content presently stored in RDBs and presented in public accessible dynamic web pages will end up on the web of linked data. Examples are product catalogs, price lists, event schedules and the like.</p> <p>The volume of the well known linked data sets is around 10 billion statements. With the above mentioned trends, growth by two or three orders of magnitude by 2015 seems reasonable, This is so especially if explicit semantics are extracted from the document web and if there is some further progress in the precision/recall of such extraction.</p> <p>Relevant sections of this mass of data are a potential addition to any present or future analytics application.</p> <p>Since arbitrary analytics over the database which is the web cannot be economically provided by a centralized search engine, a cloud model may be used for on-demand selection of relevant data and mixing it with private data. This will drive database innovation for the next years even more than the continued classical warehouse growth.</p> <p>Science data is another driver of the data overflow. For example, faster gene sequencing, more accurate measurements in high energy physics, better imaging, and remote sensing will produce large volumes of data. This data has highly regular structure but labeling this data with source and lineage calls for a flexible, schema-last, self-describing model, such as RDF and linked data. Data and <a href="http://dbpedia.org/resource/Metadata" id="link-id0x7a3fb40">metadata</a> should travel together but may have different data models.</p> <p>By and large, the metadata of science data will be another stream to the web of linked data, at least to the degree it is publicly accessible. Restricted circles can and likely will implement similar ideas.</p> </li> <li> <p> <b>What types of data can we deal with intelligently due to their inherent structure (geospatial, temporal, social or <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x5a48058">knowledge</a> graphs, 3D, sensor streams...)?</b> </p> <p>All the above types should be supported inside one DBMS so as to allow efficient querying combining conditions on all these types of data, e.g., <i>photos of sunsets taken last summer in Ibiza, with over 20 megapixels, by people I know.</i> </p> <p>Note that the test for being a sunset is an operation on the image blob that should be taken to the data; the images cannot be economically transferred.</p> <p>Interleaving of all database functions and types becomes increasingly important.</p> </li> </ol> </li> <li> <p> <b>Industries, communities</b> </p> <ol type="a" start="1"> <li> <p> <b>Who is producing these data and why? Could they do it better? How?</b> </p> <p>Right now, projects such as <a href="http://www.bio2rdf.org/" id="link-id0x2a29de8">Bio2RDF</a>, <a href="http://neurocommons.org/page/Main_Page" id="link-id0x7ddaed0">Neurocommons</a>, and DBPedia produce this data. The processes are in place and are reasonable. Incremental improvement is to be expected. These processes, along with the <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id0xbab4dfd0">linked data meme</a> generally taking off, drive demand for better <a href="http://dbpedia.org/resource/Natural_language_processing" id="link-id0x51f4e0">NLP</a> (<a href="http://dbpedia.org/resource/Natural_language_processing" id="link-id0x51a1b48">Natural Language Processing</a>), e.g., <a href="http://dbpedia.org/resource/Entity" id="link-id0x956680">entity</a> and relationship extraction, especially extraction that can produce instance data in given ontologies (e.g., events) using common identifiers (e.g., DBPedia URIs).</p> <p>Mapping of RDBs to RDF is possible, and a W3C working group is developing standards for this. The required baseline level has been reached; the rest is a matter of automating deployment. Within the enterprise, there are advantages to be gained for <a href="http://dbpedia.org/resource/Information" id="link-id0x7da9e80">information</a> integration; e.g., all entities in the CRM space can be integrated with all email and support tickets through giving everything a <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x71673f8">URI</a>. Some of this information may even be published on an <a href="http://dbpedia.org/resource/Extranet" id="link-id0x9aa6e0">extranet</a> for self-service and web-service interfaces. This has been done at small scales and the rest is a matter of spreading adoption and lowering the entry barrier. Incremental progress will take place, eventually resulting in qualitatively better integration along the value chain when adoption is sufficiently widespread.</p> </li> <li> <p> <b>Who is consuming these data and why? Could they do it better? How?</b> </p> <p>Consumers are various. The greatest need is for tools that summarize complex data and allow getting a bird's eye view of what data is in the first instance available. Consuming the data is hindered by the user not even necessarily knowing what data there is. This is somewhat new, as traditionally the business analyst did know the schema of the warehouse and was proficient with <a href="http://dbpedia.org/resource/SQL" id="link-id0x7f7b148">SQL</a> report generators and statistics packages.</p> <p>Where Web 2.0 made the <i>citizen journalist</i>, the web of linked data will make the <i>citizen analyst</i>. For this to happen, with benefits for individuals, enterprises, and governments alike, more work in user interfaces, knowledge discovery, and query composition will be useful. We may envision a "meshup economy" where data is plentiful, but the unit of value and exchange is the smart report that crystallizes actionable value from this ocean.</p> </li> <li> <p> <b>What industrial sectors in Europe could become more competitive if they became much better at managing data?</b> </p> <p>Any sector could benefit. Early adopters are seen in the biomedical field and to an extent in media. </p> </li> <li> <p> <b>Is the regulation landscape imposing constraints (privacy, compliance ...) that don't have today good tool support?</b> </p> <p>The regulation landscape drives database demand through data retention requirements and the like.</p> <p>With data integration, especially with privacy-sensitive data (as in medicine), there are issues of whether one dares put otherwise-shareable information online. Regulation is needed to protect individuals, but integration should still be possible for science.</p> <p>For this, we see a need for progress in applying policy-based approaches (e.g., row level security) to relatively schema-last data such as RDF. This is possible but needs some more work. Also, creating on-the-fly-anonymizing views on data might help.</p> <p>More research is needed for reconciling the need for security with the advantages of broad-based <i>ad hoc</i> integration. Ideally, data should be intelligent, aware of its origins and classification and cautious of whom it interacts with, all of this supported under the covers so that the user could ask anything but the data might refuse to answer or might restrict answers according to the user's profile. This is a tall order and implementing something of the sort is an open question.</p> </li> <li> <p> <b>What are the main practical problem identified for individuals and organizations? Please give examples and tell us about the main obstacles and barriers.</b> </p> <p>We have come across the following:</p> <ul> <li>Knowing that the data exists in the first place.</li> <li>If the data is found, figuring out the provenance, units and precision of measurement, identifiers, and the like.</li> <li>Compatible subject matter but incompatible representation: For example, one has numbers on a map with different maps for different points in time; another has time series of instrument data with geo-location for the instrument. It is only to be expected that the time interval between measurements is not the same. So there is need for a lot of one-off programming to align data.</li> </ul> <p>Other problems have to do with sheer volume, i.e., transfer of data even in a local area network is too slow, let alone over a wide area network. Computation needs to go to the data, and databases need to support this.</p> </li> </ol> </li> <li> <p> <b>Services, software stacks, protocols, standards, benchmarks</b> </p> <ol type="a" start="1"> <li> <p> <b>What combinations of components are needed to deal with these problems?</b> </p> <p>Recent times have seen a proliferation of special purpose databases. Since the data needs of the future are about combining data with maximum agility and minimum performance hit, there is need to gather the currently-separate functionality into an integrated system with sufficient flexibility. We see some of this in integration of map-reduce and scale-out databases. The former antagonists have become partners. Vertica, <a href="http://dbpedia.org/resource/Greenplum" id="link-id0x7a94e70">Greenplum</a>, and OpenLink <a href="http://virtuoso.openlinksw.com" id="link-id0x2ab2868">Virtuoso</a> are example of DBMS featuring work in this direction.</p> <p>Interoperability and at least <i>de facto</i> standards in ways of doing this will emerge.</p> </li> <li> <p> <b>What data exchange and processing mechanisms will be needed to work across platforms and programming languages?</b> </p> <p> <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x78a0458">HTTP</a>, <a href="http://dbpedia.org/resource/XML" id="link-id0x7ff2360">XML</a>, and RDF are in fact very verbose, yet these are the formats and models that have uptake. Thus, these will continue to be used even though one might think binary formats to be more efficient.</p> <p>There are of course science data set standards that are more compressed and these will continue, hopefully adding a practice of rich metadata in RDF.</p> <p>For internals of systems, MPI and TCP/IP with proprietary optimized wire formats will continue. Inter-system communication will likely continue to be HTTP, XML, and RDF as appropriate.</p> </li> <li> <p> <b>What data environments are today so wastefully messy that they would benefit from the development of standards?</b> </p> <p>RDF and <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x5643d70">OWL</a> are not messy but they could use some more performance; we are working on this. <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x152ab18">SPARQL</a> is finally acquiring the capabilities of a serious query language, so things are slowly coming together.</p> <p>Community process for developing application domain specific vocabularies works quite well, even though one could argue it is <i>ad hoc</i> and not up to what a modeling purist might wish.</p> <p>Top-down imposition of standards has a mixed history, with long and expensive development and sometimes no or little uptake, consider some WS* standards for example.</p> </li> <li> <p> <b>What kind of performance is expected or required of these systems? Who will measure it reliably? How?</b> </p> <p>Relational databases have a history of substantial investment in <a href="http://dbpedia.org/resource/Program_optimization" id="link-id0xecc100">optimization</a> and some of them are very good for what they do, e.g., the newer generation of analytics databases.</p> <p>The very large schema-last, no-SQL, sometimes eventually consistent key-value stores have a somewhat shorter history but do fill a real need.</p> <p>These trends will merge: Extreme scale, schema-last, complex queries, even more complex inference, custom code for in-database machine learning and other bulk processing.</p> <p>We find RDF augmented with some binary types at this crossroads. This point of the design space will have to provide performance roughly on the level of today's best relational solution for workloads that fit the relational model. The added cost of schema-last and inference must come down. We are working on this. Research work such as carried out with <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x7ae2890">MonetDB</a> gives clues as to how these aims can be reached.</p> <p>The separation of query language and inference is artificial. After the concepts are mature, these functions will merge and execute close to the data; there are clear evolutionary pressures in this direction.</p> <p>Benchmarks are key. Some gain can be had even from repurposing standard relational benchmarks like <a href="http://www.tpc.org/" id="link-id0x71eb528">TPC</a>-<a href="http://dbpedia.org/resource/TPC-H" id="link-id0x5e16a40">H</a>. But the TPC-H rules do not allow official reporting of such.</p> <p>Development of benchmarks for RDF, complex queries, and inference is needed. A bold challenge to the community, it should be rooted in real-life integration needs and involve high heterogeneity. A key-value store benchmark might also be conceived. A transaction benchmark like TPC-<a href="http://dbpedia.org/resource/C%2B%2B" id="link-id0x78562d0">C</a> might be the basis, maybe augmented with massive user-generated content like reviews and blogs.</p> <p>If benchmarks exist and are not too easy nor inaccessibly difficult nor too expensive to run — think of the high end TPC-C results — then TPC-style rules and processes would be quite adequate. The threshold to publish should be lowered: Everybody runs the TPC workloads internally but few publish.</p> <p>Some EC initiative for benchmarking could make sense, similar to the TREC initiative of the US government. Industry should be consulted for the specific content; possibly the answers to the present questionnaire can provide an approximate direction.</p> <p>Benchmarks should be run by software vendors on their own systems, tuned by themselves. But there should be a process of disclosure and auditing; the TPC rules give an example. Compliance should not be too expensive or time consuming. Some community development for automating these things would be a worthwhile target for EC funding.</p> </li> </ol> </li> <li> <p> <b>Usability and training</b> </p> <ol type="a" start="1"> <li> <p> <b>How difficult will it be for a developer of average competence to deploy components whose core is based on rather deep computer science? Do we all need to understand Monads and Continuations? What can be done to make it ever easier?</b> </p> <p>In the database world, huge advances in technology have taken place behind a relatively simple and stable interface: SQL. For the linked data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x7761e50">web</a>, the same will take place behind SPARQL.</p> <p>Beyond these, for example, programming with MPI with good utilization of a cluster platform for an arbitrary algorithm, is quite difficult. The casual amateur is hereby warned.</p> <p>There is no single solution. For automatic parallelization, since explicit, programmatic parallelization of things with MPI for example is very unscalable in terms of required skill, we should favor declarative and/or functional approaches.</p> <p>Developing a debugger and explanation engine for rule-based and description-logics-based inference would be an idea.</p> <p>For procedural workloads, things like Erlang may be good in cases and are not overly difficult in principle, especially if there are good debugging facilities.</p> <p>For shipping functions in a cluster or cloud, the <a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id0x5494b0">BOOM</a> (<a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id0x7f1f148">Berkeley Orders Of Magnitude</a>) approach or logic programming with explicit specification of compute location seem promising, surely more flexible than map-reduce. The question is whether a <a href="http://dbpedia.org/resource/PHP" id="link-id0x5c758c8">PHP</a> developer can be made to do logic programming.</p> <p>This bridge will be crossed only with actual need and even then reluctantly. We may look at the Web 2.0 practice of sharding <a href="http://dbpedia.org/resource/MySQL" id="link-id0x432f868">MySQL</a>, inconvenient as this may be, for an example. There is inertia and thus re-architecting is a constant process that is generally in reaction to facts, <i>post hoc</i>, often a point solution. One could argue that planning ahead would be smarter but by and large the world does not work so.</p> <p>One part of the answer is an infinitely-scalable SQL database that expands and shrinks in the clouds, with the usual semantics, maybe optional eventual consistency and built-in map reduce. If such a thing is inexpensive enough and syntax-level-compatible with present installed base, many developers do not have to learn very much more.</p> <p>This is maybe good for the bread-and-butter IT, but European competitiveness should not rest on this. Therefore we wish to go for bold new application types for which the client-server database application is not the model. Data-centric languages like BOOM, if they can be made very efficient and have good debugging support, are attractive there. These do require more intellectual investment but that is not a problem since the less-inquisitive part of the developer community is served by the first part of the answer.</p> </li> <li> <p> <b>How is a developer of average skills going to learn about these new advanced tools? How can we plan for excellent documentation and training, community mentoring, exchange of good practices, etc... across all EU countries?</b> </p> <p>For the most part, developers do not learn things for the sake of learning. When they have learned something and it is adequate, they stay with it for the most part and are even reluctant to engage in cross-camps interaction. The research world is often similarly insular. A new inflection in the application landscape is needed to drive learning. This inflection is provided by the <a href="https://wiki.mozilla.org/Labs/Ubiquity" id="link-id0x7f051c8">ubiquity</a> of mobile devices, sensor data, explicit semantics, NLP concept extraction, web of linked data, and such factors.</p> <p>RDFa is a good example of a new technique piggybacking on something everybody uses, namely HTML. These new things should, within possibility, be deployed in the usual technology stack, <a href="http://en.wikipedia.org/wiki/LAMP_%28software_bundle%29" id="link-id0x77151e0">LAMP</a> or Java. Of course these do not have to be LAMP or Java or HTML or HTTP themselves but they must manifest through these.</p> <p>A lot of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x7940cd0">semantic web</a> potential can be realized within the client-server database application model, thus no fundamental re-architecting, just some new data types and queries.</p> <p>For data- or processing-intensive tasks, an on-demand hookup to cloud-based servers with Erlang and/or BOOM for programming model would be easy enough to learn and utilize.</p> <p>The question is one of providing challenges. Addressing actual challenges with these techniques will lead to maturity, documentation, examples, and training. With virtual, Europe-wide distributed teams a reality in many places, Europe-wide dissemination is no longer insurmountable.</p> <p>As the data overflow proceeds, its victims will multiply and create demand for solutions. The EC could here encourage research project use cases gaining an extended life past the end of research projects, possibly being maintained and multiplied and spun off.</p> <p>If such things could be mutated into self-sustaining service businesses with pay-per-use revenue, say through a cloud SaaS business model, still primarily leveraging an open source technology stack, we could have self-propagating and self-supporting models for exploiting advanced IT. This would create interest, and interest would drive training and dissemination.</p> <p>The problem is creating the pull.</p> </li> </ol> </li> <li> <p> <b>Challenges</b> </p> <ol type="a" start="1"> <li> <p> <b>What should be, in this domain, the equivalent of the Netflix challenge, Ansari X Prize, <a href="http://dbpedia.org/resource/Google" id="link-id0x7e72f40">Google</a> Lunar X Prize, etc. ... ?</b> </p> <p>The EC itself no doubt suffers from data overflow in one function or another. Unless security/secrecy prohibits, simply publishing a large data set and a description of what operations should be done on it would be a start. The more real the data, the better — reality is consistently more complex and surprising than imagination. Since many interesting problems touch on fraud detection and law enforcement, there may be some security obstacles for using these application domains as subject matters of open challenges.</p> <p>Once there is a good benchmark, as discussed above, there can be some prize money allocated for the winners, specially if the race is tight.</p> <p>The Semantic Web Challenge and the Billion Triples Challenge exist and are useful as such, but do not seem to have any huge impact.</p> <p>The incentives should be sufficient and part of the expenses arising from running for such challenges could be funded. Otherwise investing in existing business development will be more interesting to industry. Some industry participation seems necessary; we would wish academia and industry to work closer. Also, having industry supply the baseline guarantees that academia actually does further the state of the art. This is not always certain.</p> <p>If challenges are based on actual problems, whether of the EC, its member governments, or private entities, and winning the challenge may lead to a contract for supplying an actual solution, these will naturally become more interesting for consortia involving integrators, specialist software vendors, and academia. Such a model would build actual capacity to deploy leading edge technologies in production, which is sorely needed.</p> </li> <li> <p> <b>What should one do to set up such a challenge, administer, and monitor it?</b> </p> <p>The EC should probably circulate a call for actual problem scenarios involving big data. If the matter of the overflow is as dire as represented, cases should be easy to find. A few should be selected and then anonymized if needed.</p> <p>The party with the use case would benefit by having hopefully the best work on it. The contestants would benefit from having real world needs guide R&D. The EC would not have to do very much, except possibly use some money for funding the best proposals. The winner would possibly get a large account and related sales and service income. The contestants would have to be teams possibly involving many organizations; for example, development and first-line services and support could come from different companies along a systems integrator model such as is widely used in the US.</p> <p>There may be a good benchmark at the time, possibly resulting from FP7 itself. In such a case, the EC could offer a prize for winners. Details would have to be worked out case by case. Such a challenge could be repeated a few times, as benchmark-driven progress in databases or TREC for example have taken some years to reach a point of slowdown in progress.</p> <p>Administrating such an activity should not be prohibitive, as most of the expertise can be found with the stakeholders.</p> </li> </ol> </li> </ol>
2009-10-27T14:57:31-04:00
European Commission and the Data Overflow
http://www.openlinksw.com:443/weblog/oerling/?date=2009-10-27#1585
2009-10-27T18:29:51Z
<p>The European Commission recently circulated a questionnaire to selected experts on what could be done for the future of big <a href="http://dbpedia.org/resource/Data" id="link-id0x79cfe58">data</a>.</p> <p>Since the <a href="http://cordis.europa.eu/fp7/ict/content-knowledge/consultation_en.html" id="link-id1191c0f8">questionnaire is public</a>, I am publishing my answers below.</p> <ol type="1" start="1"> <li> <p> <b>Data and data types</b> </p> <ol type="a" start="1"> <li> <p> <b>What volumes of data are we dealing with today? What is the growth rate? Where can we expect to be in 2015? </b> </p> <p>Private data warehouses of corporations have more than doubled yearly for the past years; hundreds of TB is not exceptional. This will continue. The real shift is in structured data being published in increasing quantities with a minimum level of integrate-ability through use of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x7d7e7a0">RDF</a> and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x7f2a788">linked data</a> principles. There are rewards for use of standard vocabularies and identifiers through search engines recognizing such data. There is convergence around <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x7dfbca8">DBpedia</a> identifiers for real-world entities, e.g., most things that would be in the news.</p> <p>This also means that internal data processes and silos may be enriched with this content. There is consequent pressure for accommodating more diversity of data, with more flexible <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x7babaf8">schema</a>.</p> <p>Ultimately, all content presently stored in RDBs and presented in public accessible dynamic web pages will end up on the web of linked data. Examples are product catalogs, price lists, event schedules and the like.</p> <p>The volume of the well known linked data sets is around 10 billion statements. With the above mentioned trends, growth by two or three orders of magnitude by 2015 seems reasonable, This is so especially if explicit semantics are extracted from the document web and if there is some further progress in the precision/recall of such extraction.</p> <p>Relevant sections of this mass of data are a potential addition to any present or future analytics application.</p> <p>Since arbitrary analytics over the database which is the web cannot be economically provided by a centralized search engine, a cloud model may be used for on-demand selection of relevant data and mixing it with private data. This will drive database innovation for the next years even more than the continued classical warehouse growth.</p> <p>Science data is another driver of the data overflow. For example, faster gene sequencing, more accurate measurements in high energy physics, better imaging, and remote sensing will produce large volumes of data. This data has highly regular structure but labeling this data with source and lineage calls for a flexible, schema-last, self-describing model, such as RDF and linked data. Data and <a href="http://dbpedia.org/resource/Metadata" id="link-id0x96ce60">metadata</a> should travel together but may have different data models.</p> <p>By and large, the metadata of science data will be another stream to the web of linked data, at least to the degree it is publicly accessible. Restricted circles can and likely will implement similar ideas.</p> </li> <li> <p> <b>What types of data can we deal with intelligently due to their inherent structure (geospatial, temporal, social or <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x7e8e248">knowledge</a> graphs, 3D, sensor streams...)?</b> </p> <p>All the above types should be supported inside one DBMS so as to allow efficient querying combining conditions on all these types of data, e.g., <i>photos of sunsets taken last summer in Ibiza, with over 20 megapixels, by people I know.</i> </p> <p>Note that the test for being a sunset is an operation on the image blob that should be taken to the data; the images cannot be economically transferred.</p> <p>Interleaving of all database functions and types becomes increasingly important.</p> </li> </ol> </li> <li> <p> <b>Industries, communities</b> </p> <ol type="a" start="1"> <li> <p> <b>Who is producing these data and why? Could they do it better? How?</b> </p> <p>Right now, projects such as <a href="http://www.bio2rdf.org/" id="link-id0x43bd098">Bio2RDF</a>, <a href="http://neurocommons.org/page/Main_Page" id="link-id0x5c074b0">Neurocommons</a>, and DBPedia produce this data. The processes are in place and are reasonable. Incremental improvement is to be expected. These processes, along with the <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id0x72131d0">linked data meme</a> generally taking off, drive demand for better <a href="http://dbpedia.org/resource/Natural_language_processing" id="link-id0x71e7798">NLP</a> (<a href="http://dbpedia.org/resource/Natural_language_processing" id="link-id0x7e0e2f0">Natural Language Processing</a>), e.g., <a href="http://dbpedia.org/resource/Entity" id="link-id0x71ab500">entity</a> and relationship extraction, especially extraction that can produce instance data in given ontologies (e.g., events) using common identifiers (e.g., DBPedia URIs).</p> <p>Mapping of RDBs to RDF is possible, and a W3C working group is developing standards for this. The required baseline level has been reached; the rest is a matter of automating deployment. Within the enterprise, there are advantages to be gained for <a href="http://dbpedia.org/resource/Information" id="link-id0x7a8e9a8">information</a> integration; e.g., all entities in the CRM space can be integrated with all email and support tickets through giving everything a <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x599f630">URI</a>. Some of this information may even be published on an <a href="http://dbpedia.org/resource/Extranet" id="link-id0x2a28f98">extranet</a> for self-service and web-service interfaces. This has been done at small scales and the rest is a matter of spreading adoption and lowering the entry barrier. Incremental progress will take place, eventually resulting in qualitatively better integration along the value chain when adoption is sufficiently widespread.</p> </li> <li> <p> <b>Who is consuming these data and why? Could they do it better? How?</b> </p> <p>Consumers are various. The greatest need is for tools that summarize complex data and allow getting a bird's eye view of what data is in the first instance available. Consuming the data is hindered by the user not even necessarily knowing what data there is. This is somewhat new, as traditionally the business analyst did know the schema of the warehouse and was proficient with <a href="http://dbpedia.org/resource/SQL" id="link-id0x5999558">SQL</a> report generators and statistics packages.</p> <p>Where Web 2.0 made the <i>citizen journalist</i>, the web of linked data will make the <i>citizen analyst</i>. For this to happen, with benefits for individuals, enterprises, and governments alike, more work in user interfaces, knowledge discovery, and query composition will be useful. We may envision a "meshup economy" where data is plentiful, but the unit of value and exchange is the smart report that crystallizes actionable value from this ocean.</p> </li> <li> <p> <b>What industrial sectors in Europe could become more competitive if they became much better at managing data?</b> </p> <p>Any sector could benefit. Early adopters are seen in the biomedical field and to an extent in media. </p> </li> <li> <p> <b>Is the regulation landscape imposing constraints (privacy, compliance ...) that don't have today good tool support?</b> </p> <p>The regulation landscape drives database demand through data retention requirements and the like.</p> <p>With data integration, especially with privacy-sensitive data (as in medicine), there are issues of whether one dares put otherwise-shareable information online. Regulation is needed to protect individuals, but integration should still be possible for science.</p> <p>For this, we see a need for progress in applying policy-based approaches (e.g., row level security) to relatively schema-last data such as RDF. This is possible but needs some more work. Also, creating on-the-fly-anonymizing views on data might help.</p> <p>More research is needed for reconciling the need for security with the advantages of broad-based <i>ad hoc</i> integration. Ideally, data should be intelligent, aware of its origins and classification and cautious of whom it interacts with, all of this supported under the covers so that the user could ask anything but the data might refuse to answer or might restrict answers according to the user's profile. This is a tall order and implementing something of the sort is an open question.</p> </li> <li> <p> <b>What are the main practical problem identified for individuals and organizations? Please give examples and tell us about the main obstacles and barriers.</b> </p> <p>We have come across the following:</p> <ul> <li>Knowing that the data exists in the first place.</li> <li>If the data is found, figuring out the provenance, units and precision of measurement, identifiers, and the like.</li> <li>Compatible subject matter but incompatible representation: For example, one has numbers on a map with different maps for different points in time; another has time series of instrument data with geo-location for the instrument. It is only to be expected that the time interval between measurements is not the same. So there is need for a lot of one-off programming to align data.</li> </ul> <p>Other problems have to do with sheer volume, i.e., transfer of data even in a local area network is too slow, let alone over a wide area network. Computation needs to go to the data, and databases need to support this.</p> </li> </ol> </li> <li> <p> <b>Services, software stacks, protocols, standards, benchmarks</b> </p> <ol type="a" start="1"> <li> <p> <b>What combinations of components are needed to deal with these problems?</b> </p> <p>Recent times have seen a proliferation of special purpose databases. Since the data needs of the future are about combining data with maximum agility and minimum performance hit, there is need to gather the currently-separate functionality into an integrated system with sufficient flexibility. We see some of this in integration of map-reduce and scale-out databases. The former antagonists have become partners. Vertica, <a href="http://dbpedia.org/resource/Greenplum" id="link-id0x45ecfa0">Greenplum</a>, and OpenLink <a href="http://virtuoso.openlinksw.com" id="link-id0x7f73fc8">Virtuoso</a> are example of DBMS featuring work in this direction.</p> <p>Interoperability and at least <i>de facto</i> standards in ways of doing this will emerge.</p> </li> <li> <p> <b>What data exchange and processing mechanisms will be needed to work across platforms and programming languages?</b> </p> <p> <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x776a1a0">HTTP</a>, <a href="http://dbpedia.org/resource/XML" id="link-id0x2a4e8d0">XML</a>, and RDF are in fact very verbose, yet these are the formats and models that have uptake. Thus, these will continue to be used even though one might think binary formats to be more efficient.</p> <p>There are of course science data set standards that are more compressed and these will continue, hopefully adding a practice of rich metadata in RDF.</p> <p>For internals of systems, MPI and TCP/IP with proprietary optimized wire formats will continue. Inter-system communication will likely continue to be HTTP, XML, and RDF as appropriate.</p> </li> <li> <p> <b>What data environments are today so wastefully messy that they would benefit from the development of standards?</b> </p> <p>RDF and <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x2a35960">OWL</a> are not messy but they could use some more performance; we are working on this. <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x12362e8">SPARQL</a> is finally acquiring the capabilities of a serious query language, so things are slowly coming together.</p> <p>Community process for developing application domain specific vocabularies works quite well, even though one could argue it is <i>ad hoc</i> and not up to what a modeling purist might wish.</p> <p>Top-down imposition of standards has a mixed history, with long and expensive development and sometimes no or little uptake, consider some WS* standards for example.</p> </li> <li> <p> <b>What kind of performance is expected or required of these systems? Who will measure it reliably? How?</b> </p> <p>Relational databases have a history of substantial investment in <a href="http://dbpedia.org/resource/Program_optimization" id="link-id0x7b2d7c8">optimization</a> and some of them are very good for what they do, e.g., the newer generation of analytics databases.</p> <p>The very large schema-last, no-SQL, sometimes eventually consistent key-value stores have a somewhat shorter history but do fill a real need.</p> <p>These trends will merge: Extreme scale, schema-last, complex queries, even more complex inference, custom code for in-database machine learning and other bulk processing.</p> <p>We find RDF augmented with some binary types at this crossroads. This point of the design space will have to provide performance roughly on the level of today's best relational solution for workloads that fit the relational model. The added cost of schema-last and inference must come down. We are working on this. Research work such as carried out with <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x794ee48">MonetDB</a> gives clues as to how these aims can be reached.</p> <p>The separation of query language and inference is artificial. After the concepts are mature, these functions will merge and execute close to the data; there are clear evolutionary pressures in this direction.</p> <p>Benchmarks are key. Some gain can be had even from repurposing standard relational benchmarks like <a href="http://www.tpc.org/" id="link-id0x7d45c58">TPC</a>-<a href="http://dbpedia.org/resource/TPC-H" id="link-id0x45b0198">H</a>. But the TPC-H rules do not allow official reporting of such.</p> <p>Development of benchmarks for RDF, complex queries, and inference is needed. A bold challenge to the community, it should be rooted in real-life integration needs and involve high heterogeneity. A key-value store benchmark might also be conceived. A transaction benchmark like TPC-<a href="http://dbpedia.org/resource/C%2B%2B" id="link-id0x7e32178">C</a> might be the basis, maybe augmented with massive user-generated content like reviews and blogs.</p> <p>If benchmarks exist and are not too easy nor inaccessibly difficult nor too expensive to run — think of the high end TPC-C results — then TPC-style rules and processes would be quite adequate. The threshold to publish should be lowered: Everybody runs the TPC workloads internally but few publish.</p> <p>Some EC initiative for benchmarking could make sense, similar to the TREC initiative of the US government. Industry should be consulted for the specific content; possibly the answers to the present questionnaire can provide an approximate direction.</p> <p>Benchmarks should be run by software vendors on their own systems, tuned by themselves. But there should be a process of disclosure and auditing; the TPC rules give an example. Compliance should not be too expensive or time consuming. Some community development for automating these things would be a worthwhile target for EC funding.</p> </li> </ol> </li> <li> <p> <b>Usability and training</b> </p> <ol type="a" start="1"> <li> <p> <b>How difficult will it be for a developer of average competence to deploy components whose core is based on rather deep computer science? Do we all need to understand Monads and Continuations? What can be done to make it ever easier?</b> </p> <p>In the database world, huge advances in technology have taken place behind a relatively simple and stable interface: SQL. For the linked data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x7e01618">web</a>, the same will take place behind SPARQL.</p> <p>Beyond these, for example, programming with MPI with good utilization of a cluster platform for an arbitrary algorithm, is quite difficult. The casual amateur is hereby warned.</p> <p>There is no single solution. For automatic parallelization, since explicit, programmatic parallelization of things with MPI for example is very unscalable in terms of required skill, we should favor declarative and/or functional approaches.</p> <p>Developing a debugger and explanation engine for rule-based and description-logics-based inference would be an idea.</p> <p>For procedural workloads, things like Erlang may be good in cases and are not overly difficult in principle, especially if there are good debugging facilities.</p> <p>For shipping functions in a cluster or cloud, the <a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id0x43665a8">BOOM</a> (<a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id0x7718f00">Berkeley Orders Of Magnitude</a>) approach or logic programming with explicit specification of compute location seem promising, surely more flexible than map-reduce. The question is whether a <a href="http://dbpedia.org/resource/PHP" id="link-id0x7d64f68">PHP</a> developer can be made to do logic programming.</p> <p>This bridge will be crossed only with actual need and even then reluctantly. We may look at the Web 2.0 practice of sharding <a href="http://dbpedia.org/resource/MySQL" id="link-id0xbab1ae98">MySQL</a>, inconvenient as this may be, for an example. There is inertia and thus re-architecting is a constant process that is generally in reaction to facts, <i>post hoc</i>, often a point solution. One could argue that planning ahead would be smarter but by and large the world does not work so.</p> <p>One part of the answer is an infinitely-scalable SQL database that expands and shrinks in the clouds, with the usual semantics, maybe optional eventual consistency and built-in map reduce. If such a thing is inexpensive enough and syntax-level-compatible with present installed base, many developers do not have to learn very much more.</p> <p>This is maybe good for the bread-and-butter IT, but European competitiveness should not rest on this. Therefore we wish to go for bold new application types for which the client-server database application is not the model. Data-centric languages like BOOM, if they can be made very efficient and have good debugging support, are attractive there. These do require more intellectual investment but that is not a problem since the less-inquisitive part of the developer community is served by the first part of the answer.</p> </li> <li> <p> <b>How is a developer of average skills going to learn about these new advanced tools? How can we plan for excellent documentation and training, community mentoring, exchange of good practices, etc... across all EU countries?</b> </p> <p>For the most part, developers do not learn things for the sake of learning. When they have learned something and it is adequate, they stay with it for the most part and are even reluctant to engage in cross-camps interaction. The research world is often similarly insular. A new inflection in the application landscape is needed to drive learning. This inflection is provided by the <a href="https://wiki.mozilla.org/Labs/Ubiquity" id="link-id0x770df38">ubiquity</a> of mobile devices, sensor data, explicit semantics, NLP concept extraction, web of linked data, and such factors.</p> <p>RDFa is a good example of a new technique piggybacking on something everybody uses, namely HTML. These new things should, within possibility, be deployed in the usual technology stack, <a href="http://en.wikipedia.org/wiki/LAMP_%28software_bundle%29" id="link-id0x55596a8">LAMP</a> or Java. Of course these do not have to be LAMP or Java or HTML or HTTP themselves but they must manifest through these.</p> <p>A lot of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x3d5378">semantic web</a> potential can be realized within the client-server database application model, thus no fundamental re-architecting, just some new data types and queries.</p> <p>For data- or processing-intensive tasks, an on-demand hookup to cloud-based servers with Erlang and/or BOOM for programming model would be easy enough to learn and utilize.</p> <p>The question is one of providing challenges. Addressing actual challenges with these techniques will lead to maturity, documentation, examples, and training. With virtual, Europe-wide distributed teams a reality in many places, Europe-wide dissemination is no longer insurmountable.</p> <p>As the data overflow proceeds, its victims will multiply and create demand for solutions. The EC could here encourage research project use cases gaining an extended life past the end of research projects, possibly being maintained and multiplied and spun off.</p> <p>If such things could be mutated into self-sustaining service businesses with pay-per-use revenue, say through a cloud SaaS business model, still primarily leveraging an open source technology stack, we could have self-propagating and self-supporting models for exploiting advanced IT. This would create interest, and interest would drive training and dissemination.</p> <p>The problem is creating the pull.</p> </li> </ol> </li> <li> <p> <b>Challenges</b> </p> <ol type="a" start="1"> <li> <p> <b>What should be, in this domain, the equivalent of the Netflix challenge, Ansari X Prize, <a href="http://dbpedia.org/resource/Google" id="link-id0x6a6c2b0">Google</a> Lunar X Prize, etc. ... ?</b> </p> <p>The EC itself no doubt suffers from data overflow in one function or another. Unless security/secrecy prohibits, simply publishing a large data set and a description of what operations should be done on it would be a start. The more real the data, the better — reality is consistently more complex and surprising than imagination. Since many interesting problems touch on fraud detection and law enforcement, there may be some security obstacles for using these application domains as subject matters of open challenges.</p> <p>Once there is a good benchmark, as discussed above, there can be some prize money allocated for the winners, specially if the race is tight.</p> <p>The Semantic Web Challenge and the Billion Triples Challenge exist and are useful as such, but do not seem to have any huge impact.</p> <p>The incentives should be sufficient and part of the expenses arising from running for such challenges could be funded. Otherwise investing in existing business development will be more interesting to industry. Some industry participation seems necessary; we would wish academia and industry to work closer. Also, having industry supply the baseline guarantees that academia actually does further the state of the art. This is not always certain.</p> <p>If challenges are based on actual problems, whether of the EC, its member governments, or private entities, and winning the challenge may lead to a contract for supplying an actual solution, these will naturally become more interesting for consortia involving integrators, specialist software vendors, and academia. Such a model would build actual capacity to deploy leading edge technologies in production, which is sorely needed.</p> </li> <li> <p> <b>What should one do to set up such a challenge, administer, and monitor it?</b> </p> <p>The EC should probably circulate a call for actual problem scenarios involving big data. If the matter of the overflow is as dire as represented, cases should be easy to find. A few should be selected and then anonymized if needed.</p> <p>The party with the use case would benefit by having hopefully the best work on it. The contestants would benefit from having real world needs guide R&D. The EC would not have to do very much, except possibly use some money for funding the best proposals. The winner would possibly get a large account and related sales and service income. The contestants would have to be teams possibly involving many organizations; for example, development and first-line services and support could come from different companies along a systems integrator model such as is widely used in the US.</p> <p>There may be a good benchmark at the time, possibly resulting from FP7 itself. In such a case, the EC could offer a prize for winners. Details would have to be worked out case by case. Such a challenge could be repeated a few times, as benchmark-driven progress in databases or TREC for example have taken some years to reach a point of slowdown in progress.</p> <p>Administrating such an activity should not be prohibitive, as most of the expertise can be found with the stakeholders.</p> </li> </ol> </li> </ol>
2009-10-27T14:57:28.000002-04:00
Conversation with Jon Udell: Are We There Yet Re. Web++ ?
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-09-10#1584
2009-09-10T15:03:01Z
<p> Personally, I believe that we've actually reached a watershed moment re. the evolution of the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> from a mesh of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id123169a8">Linked Data</a> Containers (Web of Linked Documents) to a mesh of Linked Data Items (entities or real world objects).</p> <p> The journey towards this watershed moment started with the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id14f69f48">Semantic Web</a> Project, gained focus and pragmatism via the <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id11155f78">Linked Data meme</a>, attained substance & credibility via efforts such as <a href="http://dbpedia.org/resource/DBpedia" id="link-id15857c78">DBpedia</a> and the resulting cloud of <a href="http://www4.wiwiss.fu-berlin.de/bizer/pub/lod-datasets_2009-07-14.html" id="link-id16adf918">Open Linked Data Spaces</a>, and finally arrived at the most important destination of all: broad comprehension and coherence, via <a href="http://dbpedia.org/resource/RDFa" id="link-id1229b960">RDFa</a>. </p> <p> Over the years, I've chronicled the journey above via entries in this particular <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id14f76338">data space</a> (my <a href="http://dbpedia.org/resource/Blog" id="link-idfd32c88">blog</a>) and most recently, via my rapid-fire comments and debates on <a href="http://twitter.com" id="link-id11339e80">Twitter</a> (basically hastag #linkeddata account: <a href="http://twitter.com/kidehen#this" id="link-id115e9af8">kidehen</a>). </p> <p> On a parallel front re. my chronicles, I've periodically had conversations with <a href="http://blog.jonudell.net/about/" id="link-id11829170">Jon Udell</a>, who has always provided a coherent sounding board and reconciliation framework for my world views and open data access vision; naturally, this has a lot to do with his holistic grasp of the big picture issues, associated technical details, and special communication prowess :-) </p> <p> Against this backdrop, I refer you to my <a href="http://itc.conversationsnetwork.org/shows/detail4233.html" id="link-id14ac9c08">most recent podcast conversation with Jon</a>, which is about how the tandem of HTML+RDFa and the <a href="http://www.heppnetz.de/projects/goodrelations/" id="link-id14279be8">GoodRelations vocabulary</a> deliver the critical missing links re. broad comprehension of the Semantic Web vision en route to mass exploitation. </p> <h3>Related</h3> <ul> <li> <a href="http://webbackplane.com/node/57" id="link-id113b5b00">Mark Birbeck Introduces RDFa</a> </li> <li> <a href="http://webbackplane.com/rdfa-handbook" id="link-id11b36ac0">RDFa Handbook</a> </li> <li> <a href="http://www.ebusiness-unibw.org/wiki/GoodRelations#CookBook:_GoodRelations_Recipes_and_Examples" id="link-id1519f458">GoodRelations Usage Examples & Templates</a> </li> <li> <a href="http://blog.jonudell.net/2009/09/09/talking-with-kingsley-idehen-about-mastering-your-own-search-index/" id="link-id11a62ce0">Be the master of your own search index</a> </li> <li> <a href="http://itc.conversationsnetwork.org/shows/detail4312.html" id="link-id115d54f0">Jon Udell Interviews Martin Hepp about GoodRelations, RDFa, and Esoteric Web Search</a> <br /> </li> </ul>
2010-02-01T08:58:04.000002-05:00
VLDB 2009 Web Scale Data Management Panel (5 of 5)
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-09-01#1583
2009-09-01T16:24:17Z
<blockquote> <p> <i>"The universe of cycles is not exactly one of literal cycles, but rather one of spirals," mused <a href="http://db.cs.berkeley.edu/jmh/" id="link-id117455a0">Joe Hellerstein</a> of UC Berkeley.</i> </p> <p> <i>"Come on, let's all drop some <a href="http://dbpedia.org/resource/ACID" id="link-id16b3db50">ACID</a>," interjected another.</i> </p> <p> <i>"It is not that we end up repeating the exact same things, rather even if some patterns seem to repeat, they do so at a higher level, enhanced by the experience gained," continued Joe.</i> </p> </blockquote> <p>Thus did the Web Scale <a href="http://dbpedia.org/resource/Data" id="link-id11061ae0">Data</a> Management panel conclude.</p> <p>Whether successive generations are made wiser by the ones that have gone before may be argued either way.</p> <p>The cycle in question was that of developers discovering ACID in the 1960s, i.e. Atomicity, Consistency, Integrity, Durability. Thus did the DBMS come into being. Then DBMSs kept becoming more complex until, as there will be a counter-force to each force, came the <a href="http://dbpedia.org/resource/Meme" id="link-id11076cc8">meme</a> of key value stores and BASE, no multiple-row transactions, eventual consistency, no query language but scaling to thousands of computers. So now, the DBMS community asks itself what went wrong.</p> <p>In the words of one panelist, another demonstrated a "shocking familiarity with the subject matter of substance abuse" when he called for the DBMS community to get on a <a href="http://dbpedia.org/resource/Twelve-step_program" id="link-id15d954a8">12 step program</a> and to look where addiction to certain ideas, among which ACID, had brought its life. Look at yourself: The influential papers in what ought to be your space by rights are coming from the OS community: <a href="http://dbpedia.org/resource/Google" id="link-id166675f0">Google</a> Bigtable, Amazon Dynamo, want more? When you ought to drive, you give excuses and play catch up! Stop denial, drop <a href="http://dbpedia.org/resource/SQL" id="link-id1105adf0">SQL</a>, drop ACID!</p> <p>The web developers have revolted against the time-honored principles of the DBMS. This is true. Sharded <a href="http://dbpedia.org/resource/MySQL" id="link-id1221c230">MySQL</a> is not the ticket — or is it? Must they rediscover the virtues of ACID, just like the previous generation did?</p> <p>Nothing under the sun is new. As in music and fashion, trends keep cycling also in science and engineering.</p> <p>But seriously, does the full-featured DBMS scale to web scale? <a href="http://dbpedia.org/resource/Microsoft" id="link-id10ffcaf8">Microsoft</a> says the Azure version of SQL server does. <a href="http://dbpedia.org/resource/Yahoo%21" id="link-id16b3f138">Yahoo</a> says they want no SQL but <a href="http://dbpedia.org/resource/Hadoop" id="link-id11046ef0">Hadoop</a> and <a href="http://research.yahoo.com/node/2304" id="link-id110a0040">PNUTS</a>.</p> <p>Twitter, Facebook, and other web names got their own discussion. Why do they not go to serious DBMS vendors for their data but make their own, like Facebook with Hive?</p> <p>Who can divine the mind of the web developer? What makes them go to <a href="http://www.danga.com/memcached/" id="link-id1109e280">memcached</a>, manually sharded MySQL, and <a href="http://dbpedia.org/resource/MapReduce" id="link-id1107cd60">MapReduce</a>, walking away from the 40 years of technology invested in declarative query and ACID? What is this highly visible but hard to grasp <a href="http://dbpedia.org/resource/Entity" id="link-id1105b6b8">entity</a>? My guess is that they want something they can understand, at least at the beginning. A DBMS, especially on a cluster, is complicated, and it is not so easy to say how it works and how its performance is determined. The big brands, if deployed on a thousand PCs, would also be prohibitively expensive. But if all you do with the DBMS is single row selects and updates, it is no longer so scary, but you end up doing all the distributed things in a middle layer, and abandoning expressive queries, transactions, and database-supported transparency of location. But at least now you know how it works and what it is good/not good for.</p> <p>This would be the case for those who make a conscious choice. But by and large the choice is not deliberate; it is something one drifts into: The application gains popularity; the single <a href="http://en.wikipedia.org/wiki/LAMP_%28software_bundle%29" id="link-iddc68d28">LAMP</a> can no longer keep all in memory; you need a second MySQL in the LAMP and you decide that users A–M go left and N–Z right (horizontal partitioning). This siren of sharding beckons you and all is good until you hit the reef of re-architecting. Memcached and duct-tape help, like aspirin helps with hangover, but the root cause of the headache lies unaddressed.</p> <p>The conclusion was that there ought to be something incrementally scalable from the get-go. Low cost of entry and built-in scale-out. No, the web developers do not hate SQL; they just have gotten the idea that it does not scale. But they would really wish it to. So, DBMS people, show there is life in you yet.</p> <p>Joe Hellerstein was the philosopher and paradigmatician of the panel. His team had developed a protocol-compatible Hadoop in a few months using a declarative logic programming style approach. His claim was that developers made the market. Thus, for writing applications against web scale data, there would have to be data centric languages. Why not? These are discussed in <a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id110ba0e0">Berkeley Orders Of Magnitude</a> (<a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id16aab768">BOOM</a>).</p> <p>I come from <a href="http://en.wikipedia.org/wiki/Lisp_%28programming_language%29" id="link-id10f2cd68">Lisp</a> myself, way back. I have since abandoned any desire to tell anybody what they ought to program in. This is a bit like religion: Attempting to impose or legislate or ram it on somebody just results in anything from lip service to rejection to war. The appeal exerted by the diverse language/paradigm -isms on their followers seems to be based on hitting a simplification of reality that coincides with a problem in the air. MapReduce is an example of this. <a href="http://dbpedia.org/resource/PHP" id="link-ide22cdd0">PHP</a> is another. A quick fix for a present need: Scripting web servers (PHP) or processing tons of files (MapReduce). The full database is not as quick a fix, even though it has many desirable features. It is also not as easy to tell what happens inside one, so MapReduce may give a greater feeling of control.</p> <p>Totally self-managing, dynamically-scalable <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id152864b0">RDF</a> would be a fix for not having to design or administer databases: Since it would be indexed on everything, complex queries would be possible; no full database scans would stop everything. For the mid-size segment of web sites this might be a fit. For the extreme ends of the spectrum, the choice is likely something custom built and much less expressive.</p> <p>The BOOM rule language for data-centric programming would be something very easy for us to implement, in fact we will get something of the sort essentially for free when we do the rule support already planned.</p> <p>The question is, can one induce web developers to do logic? The history is one of procedures, both in LAMP and MapReduce. On the other hand, the query languages that were ever universally adopted were declarative, i.e., keyword search and SQL. There certainly is a quest for an application model for the cloud space beyond just migrating apps. We'll see. More on this another time.</p>
2009-09-02T12:05:26-04:00
VLDB 2009 Web Scale Data Management Panel (5 of 5)
http://www.openlinksw.com:443/weblog/oerling/?date=2009-09-01#1582
2009-09-01T16:24:17Z
<blockquote> <p> <i>"The universe of cycles is not exactly one of literal cycles, but rather one of spirals," mused <a href="http://db.cs.berkeley.edu/jmh/" id="link-id117455a0">Joe Hellerstein</a> of UC Berkeley.</i> </p> <p> <i>"Come on, let's all drop some <a href="http://dbpedia.org/resource/ACID" id="link-id16b3db50">ACID</a>," interjected another.</i> </p> <p> <i>"It is not that we end up repeating the exact same things, rather even if some patterns seem to repeat, they do so at a higher level, enhanced by the experience gained," continued Joe.</i> </p> </blockquote> <p>Thus did the Web Scale <a href="http://dbpedia.org/resource/Data" id="link-id11061ae0">Data</a> Management panel conclude.</p> <p>Whether successive generations are made wiser by the ones that have gone before may be argued either way.</p> <p>The cycle in question was that of developers discovering ACID in the 1960s, i.e. Atomicity, Consistency, Integrity, Durability. Thus did the DBMS come into being. Then DBMSs kept becoming more complex until, as there will be a counter-force to each force, came the <a href="http://dbpedia.org/resource/Meme" id="link-id11076cc8">meme</a> of key value stores and BASE, no multiple-row transactions, eventual consistency, no query language but scaling to thousands of computers. So now, the DBMS community asks itself what went wrong.</p> <p>In the words of one panelist, another demonstrated a "shocking familiarity with the subject matter of substance abuse" when he called for the DBMS community to get on a <a href="http://dbpedia.org/resource/Twelve-step_program" id="link-id15d954a8">12 step program</a> and to look where addiction to certain ideas, among which ACID, had brought its life. Look at yourself: The influential papers in what ought to be your space by rights are coming from the OS community: <a href="http://dbpedia.org/resource/Google" id="link-id166675f0">Google</a> Bigtable, Amazon Dynamo, want more? When you ought to drive, you give excuses and play catch up! Stop denial, drop <a href="http://dbpedia.org/resource/SQL" id="link-id1105adf0">SQL</a>, drop ACID!</p> <p>The web developers have revolted against the time-honored principles of the DBMS. This is true. Sharded <a href="http://dbpedia.org/resource/MySQL" id="link-id1221c230">MySQL</a> is not the ticket — or is it? Must they rediscover the virtues of ACID, just like the previous generation did?</p> <p>Nothing under the sun is new. As in music and fashion, trends keep cycling also in science and engineering.</p> <p>But seriously, does the full-featured DBMS scale to web scale? <a href="http://dbpedia.org/resource/Microsoft" id="link-id10ffcaf8">Microsoft</a> says the Azure version of SQL server does. <a href="http://dbpedia.org/resource/Yahoo%21" id="link-id16b3f138">Yahoo</a> says they want no SQL but <a href="http://dbpedia.org/resource/Hadoop" id="link-id11046ef0">Hadoop</a> and <a href="http://research.yahoo.com/node/2304" id="link-id110a0040">PNUTS</a>.</p> <p>Twitter, Facebook, and other web names got their own discussion. Why do they not go to serious DBMS vendors for their data but make their own, like Facebook with Hive?</p> <p>Who can divine the mind of the web developer? What makes them go to <a href="http://www.danga.com/memcached/" id="link-id1109e280">memcached</a>, manually sharded MySQL, and <a href="http://dbpedia.org/resource/MapReduce" id="link-id1107cd60">MapReduce</a>, walking away from the 40 years of technology invested in declarative query and ACID? What is this highly visible but hard to grasp <a href="http://dbpedia.org/resource/Entity" id="link-id1105b6b8">entity</a>? My guess is that they want something they can understand, at least at the beginning. A DBMS, especially on a cluster, is complicated, and it is not so easy to say how it works and how its performance is determined. The big brands, if deployed on a thousand PCs, would also be prohibitively expensive. But if all you do with the DBMS is single row selects and updates, it is no longer so scary, but you end up doing all the distributed things in a middle layer, and abandoning expressive queries, transactions, and database-supported transparency of location. But at least now you know how it works and what it is good/not good for.</p> <p>This would be the case for those who make a conscious choice. But by and large the choice is not deliberate; it is something one drifts into: The application gains popularity; the single <a href="http://en.wikipedia.org/wiki/LAMP_%28software_bundle%29" id="link-iddc68d28">LAMP</a> can no longer keep all in memory; you need a second MySQL in the LAMP and you decide that users A–M go left and N–Z right (horizontal partitioning). This siren of sharding beckons you and all is good until you hit the reef of re-architecting. Memcached and duct-tape help, like aspirin helps with hangover, but the root cause of the headache lies unaddressed.</p> <p>The conclusion was that there ought to be something incrementally scalable from the get-go. Low cost of entry and built-in scale-out. No, the web developers do not hate SQL; they just have gotten the idea that it does not scale. But they would really wish it to. So, DBMS people, show there is life in you yet.</p> <p>Joe Hellerstein was the philosopher and paradigmatician of the panel. His team had developed a protocol-compatible Hadoop in a few months using a declarative logic programming style approach. His claim was that developers made the market. Thus, for writing applications against web scale data, there would have to be data centric languages. Why not? These are discussed in <a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id110ba0e0">Berkeley Orders Of Magnitude</a> (<a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id16aab768">BOOM</a>).</p> <p>I come from <a href="http://en.wikipedia.org/wiki/Lisp_%28programming_language%29" id="link-id10f2cd68">Lisp</a> myself, way back. I have since abandoned any desire to tell anybody what they ought to program in. This is a bit like religion: Attempting to impose or legislate or ram it on somebody just results in anything from lip service to rejection to war. The appeal exerted by the diverse language/paradigm -isms on their followers seems to be based on hitting a simplification of reality that coincides with a problem in the air. MapReduce is an example of this. <a href="http://dbpedia.org/resource/PHP" id="link-ide22cdd0">PHP</a> is another. A quick fix for a present need: Scripting web servers (PHP) or processing tons of files (MapReduce). The full database is not as quick a fix, even though it has many desirable features. It is also not as easy to tell what happens inside one, so MapReduce may give a greater feeling of control.</p> <p>Totally self-managing, dynamically-scalable <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id152864b0">RDF</a> would be a fix for not having to design or administer databases: Since it would be indexed on everything, complex queries would be possible; no full database scans would stop everything. For the mid-size segment of web sites this might be a fit. For the extreme ends of the spectrum, the choice is likely something custom built and much less expressive.</p> <p>The BOOM rule language for data-centric programming would be something very easy for us to implement, in fact we will get something of the sort essentially for free when we do the rule support already planned.</p> <p>The question is, can one induce web developers to do logic? The history is one of procedures, both in LAMP and MapReduce. On the other hand, the query languages that were ever universally adopted were declarative, i.e., keyword search and SQL. There certainly is a quest for an application model for the cloud space beyond just migrating apps. We'll see. More on this another time.</p>
2009-09-02T12:05:20.000001-04:00
VLDB 2009 Yahoo Keynote (4 of 5)
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-09-01#1581
2009-09-01T16:04:36Z
<p> <a href="http://dbpedia.org/resource/Raghu_Ramakrishnan" id="link-id0x177f3ef8">Raghu Ramakrishnan</a> of <a href="http://dbpedia.org/resource/Yahoo%21" id="link-id0x2a4aad0">Yahoo</a>! gave a keynote about <a href="http://research.yahoo.com/node/2304" id="link-id0x5584570">PNUTS</a>, the Yahoo solution for managing massive user <a href="http://dbpedia.org/resource/Data" id="link-id0x3805628">data</a>, from front page preferences to mail to social networks.</p> <p>Dynamic scale, wide area replication, and high availability are the issues. Transactions on multiple records, complex queries, and absolute consistency at all times are traded off. Also, the programming interfaces are lower level than with <a href="http://dbpedia.org/resource/SQL" id="link-id0x17bfc928">SQL</a>. Replication and consistency rules are choices for the application developer; the platform offers some basic alternatives. Implementation-wise, there is a <a href="http://dbpedia.org/resource/MySQL" id="link-id0x1862f7a8">MySQL</a> back-end and all the partitioning, query routing, replication, and balancing take place in a layer of front-ends.</p> <p>Now what do we say to this?</p> <p>In the Yahoo! case, even if complex queries were possible, which they are not, one would probably keep them off the online system since latency and availability are everything. A latency of some tens of milliseconds is however acceptable, which is not so terrible for single record operations: There is time for a couple of messages on the data center network and even maybe for a disk read.</p> <p>PNUTS is probably the fastest way of getting to the desired beachhead of simple access to data at infinite scale in multiple geographies. In the identical situation, I might have done something similar.</p> <p>But we are in a different situation, concerned with complex queries, a highly-normalized <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x25c942e8">schema</a>-last situation, i.e., index on everything, large objects normalized away, as is done in <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x4a3d080">RDF</a>. Then we are also in the relational situation. Infinite scale, fault tolerance, and wide-area replication do come up regularly in user needs. The applications for which people would like RDF are not only complex reasoning things but very big <a href="http://dbpedia.org/resource/Metadata" id="link-id0x19101128">metadata</a> stores for user generated content, social networks, and the like.</p> <p>Which of the PNUTS principles could we apply?</p> <ul> <li> <p> <b>Division in tablets:</b> When a partition of the data grows too big, it should split.</p> </li> <li> <p> <b>Migration of partitions:</b> as capacity/demand change, partitions should migrate so as to equalize load.</p> </li> <li> <p> <b>High availability:</b> This is divided in two — on one hand inside the data center; on the other between data centers. Inside the data center, storing partitions in duplicate and running them synchronously is possible. This is manifestly impossible in wide area settings, though. For this, we need a log-shipping style of asynchronous replication. But how does one deal with split networks and transfer of replication mastery?</p> </li> </ul> <p>PNUTS determines the master copy record by record. This makes sense when the record, for example, corresponds to a user. For RDF, doing this by the triple would be prohibitive. Doing this by the graph, or by the subject of a set of triples across all graphs, would be better. We would agree with PNUTS that transferring mastery by the storage chunk is not desired, as the chunk will contain arbitrary unrelated data.</p> <p> </p> <p>The eventual consistency mechanisms can be generalized to RDF readily enough. In a social RDF application, the graph is the most likely unit of data ownership and update authorization, so the graph would also be the unit of eventual consistency. Keeping a separate data structure listing recent inserts/deletes to a graph with timestamps would serve for establishing consistency. The size of this would be a small fraction of the size of the graph.</p> <p>RDF cannot do anything without joining between partitions, whereas for PNUTS the join between partitions is an application matter. But then PNUTS does have an extra step of RPC between the PNUTS infrastructure and the back-end. Doing query routing in the back-end gets rid of this. RDF does remain more dependent on even performance and short interconnect latencies, though. It also likely takes more space. But the essential consistency and availability features can be generalized to it, providing the merge of semi-structured data at infinite scale and availability with complex query.</p> <p>At any rate, repartitioning-on-demand and partition-migration remain the key agenda items for us, confirmed over and over at VLDB.</p>
2009-09-01T17:32:55.000002-04:00
VLDB 2009 Yahoo Keynote (4 of 5)
http://www.openlinksw.com:443/weblog/oerling/?date=2009-09-01#1577
2009-09-01T16:04:36Z
<p> <a href="http://dbpedia.org/resource/Raghu_Ramakrishnan" id="link-id0x19076030">Raghu Ramakrishnan</a> of <a href="http://dbpedia.org/resource/Yahoo%21" id="link-id0x47142b8">Yahoo</a>! gave a keynote about <a href="http://research.yahoo.com/node/2304" id="link-id0x186c1288">PNUTS</a>, the Yahoo solution for managing massive user <a href="http://dbpedia.org/resource/Data" id="link-id0x4e966e0">data</a>, from front page preferences to mail to social networks.</p> <p>Dynamic scale, wide area replication, and high availability are the issues. Transactions on multiple records, complex queries, and absolute consistency at all times are traded off. Also, the programming interfaces are lower level than with <a href="http://dbpedia.org/resource/SQL" id="link-id0x23e68948">SQL</a>. Replication and consistency rules are choices for the application developer; the platform offers some basic alternatives. Implementation-wise, there is a <a href="http://dbpedia.org/resource/MySQL" id="link-id0x182cead8">MySQL</a> back-end and all the partitioning, query routing, replication, and balancing take place in a layer of front-ends.</p> <p>Now what do we say to this?</p> <p>In the Yahoo! case, even if complex queries were possible, which they are not, one would probably keep them off the online system since latency and availability are everything. A latency of some tens of milliseconds is however acceptable, which is not so terrible for single record operations: There is time for a couple of messages on the data center network and even maybe for a disk read.</p> <p>PNUTS is probably the fastest way of getting to the desired beachhead of simple access to data at infinite scale in multiple geographies. In the identical situation, I might have done something similar.</p> <p>But we are in a different situation, concerned with complex queries, a highly-normalized <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x197b7948">schema</a>-last situation, i.e., index on everything, large objects normalized away, as is done in <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x385a900">RDF</a>. Then we are also in the relational situation. Infinite scale, fault tolerance, and wide-area replication do come up regularly in user needs. The applications for which people would like RDF are not only complex reasoning things but very big <a href="http://dbpedia.org/resource/Metadata" id="link-id0x25a30d98">metadata</a> stores for user generated content, social networks, and the like.</p> <p>Which of the PNUTS principles could we apply?</p> <ul> <li> <p> <b>Division in tablets:</b> When a partition of the data grows too big, it should split.</p> </li> <li> <p> <b>Migration of partitions:</b> as capacity/demand change, partitions should migrate so as to equalize load.</p> </li> <li> <p> <b>High availability:</b> This is divided in two — on one hand inside the data center; on the other between data centers. Inside the data center, storing partitions in duplicate and running them synchronously is possible. This is manifestly impossible in wide area settings, though. For this, we need a log-shipping style of asynchronous replication. But how does one deal with split networks and transfer of replication mastery?</p> </li> </ul> <p>PNUTS determines the master copy record by record. This makes sense when the record, for example, corresponds to a user. For RDF, doing this by the triple would be prohibitive. Doing this by the graph, or by the subject of a set of triples across all graphs, would be better. We would agree with PNUTS that transferring mastery by the storage chunk is not desired, as the chunk will contain arbitrary unrelated data.</p> <p> </p> <p>The eventual consistency mechanisms can be generalized to RDF readily enough. In a social RDF application, the graph is the most likely unit of data ownership and update authorization, so the graph would also be the unit of eventual consistency. Keeping a separate data structure listing recent inserts/deletes to a graph with timestamps would serve for establishing consistency. The size of this would be a small fraction of the size of the graph.</p> <p>RDF cannot do anything without joining between partitions, whereas for PNUTS the join between partitions is an application matter. But then PNUTS does have an extra step of RPC between the PNUTS infrastructure and the back-end. Doing query routing in the back-end gets rid of this. RDF does remain more dependent on even performance and short interconnect latencies, though. It also likely takes more space. But the essential consistency and availability features can be generalized to it, providing the merge of semi-structured data at infinite scale and availability with complex query.</p> <p>At any rate, repartitioning-on-demand and partition-migration remain the key agenda items for us, confirmed over and over at VLDB.</p>
2009-09-01T17:32:35.000002-04:00
VLDB 2009 TPC Workshop (3 of 5)
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-09-01#1580
2009-09-01T15:51:09Z
<p>Michael <a href="http://dbpedia.org/resource/Michael_Stonebraker" id="link-id0x1641ef70">Stonebraker</a> gave the keynote at the <a href="http://www.tpc.org/" id="link-id0x554d380">TPC</a> workshop. His message was that the TPC, at the venerable age of 21, was already a decade late in reinventing itself. From the height of relevance at the time of the debit/credit benchmark twenty years back, it was slipping into the sunset of irrelevance unless it paid attention.</p> <p>Now we are great fans of the TPC and while we have not published results by the TPC book, we have extensively used TPC material for guiding <a href="http://dbpedia.org/resource/Program_optimization" id="link-id0x16475bd8">optimization</a>, as has pretty much everybody else.</p> <p>It is true that the rules encourage unrealistic configurations. The emphasis on random access from disk that is built into the rules leads to disk configurations that are very improbable in practice, such as 1PB of disks for 3TB of <a href="http://dbpedia.org/resource/Data" id="link-id0x18f0b720">data</a>, just so there are enough disk arms in parallel. Stonebraker also pointed out that replication and failover were ubiquitous in real life and that roll forward from logs was unrealistic as a recovery model since it took so long. Benchmarks should therefore include replication.</p> <p>Further, Stonebraker challenged the TPC to go for the new frontier, which he described as the huge data sets in science and on big web sites. Scientists, the ones who would save our planet from the diverse ills confronting it, do not like relational databases. They avoid them when can. They want arrays for physics, and graphs for biology and chemistry. <a href="http://dbpedia.org/resource/MapReduce" id="link-id0x150376a8">MapReduce</a> is eating database's lunch; what will you do about this?</p> <p>I later suggested incorporating an <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x41cd4c0">RDF</a> <a href="http://dbpedia.org/resource/Metadata" id="link-id0x15904698">metadata</a> benchmark into the TPC suite. We'll see about this; we'll first have to come up with a suitable one. There is a great deal of pressure for making good RDF benchmarks but this is not yet in the center of the mainstream that TPC tends to cover.</p> <p>TPC's own talk was about the life cycle of benchmarks. A benchmark begins a bit ahead of the mainstream, with a problem that is difficult but not so difficult as to be uncommon. When the solution to this problem becomes commonplace, the benchmark's relevance gradually drops.</p> <p>There was a talk on robustness of query plans which was well to the point. Indeed, there are performance cliffs at certain points; for example, when passing from memory-only to disk-pageable data structures, or when switching from indexed access to table scans, or from loop to hash joins. Quite so. The analysis I really would have liked to see would have been one of what happens when passing from single server to a cluster, and from local joins to cross-partition ones. Also contrasting of <a href="http://dbpedia.org/resource/Cache" id="link-id0x16dd6710">cache</a> fusion and partitioning. We have our own data and experience but we find we don't have time to measure all the other systems.</p> <p>Anyway it is good to raise the question of smooth and predictable performance.</p>
2009-09-01T17:32:50.000003-04:00
VLDB 2009 TPC Workshop (3 of 5)
http://www.openlinksw.com:443/weblog/oerling/?date=2009-09-01#1576
2009-09-01T15:51:09Z
<p>Michael <a href="http://dbpedia.org/resource/Michael_Stonebraker" id="link-id0x15e5efe0">Stonebraker</a> gave the keynote at the <a href="http://www.tpc.org/" id="link-id0x18cee5f0">TPC</a> workshop. His message was that the TPC, at the venerable age of 21, was already a decade late in reinventing itself. From the height of relevance at the time of the debit/credit benchmark twenty years back, it was slipping into the sunset of irrelevance unless it paid attention.</p> <p>Now we are great fans of the TPC and while we have not published results by the TPC book, we have extensively used TPC material for guiding <a href="http://dbpedia.org/resource/Program_optimization" id="link-id0x4e55368">optimization</a>, as has pretty much everybody else.</p> <p>It is true that the rules encourage unrealistic configurations. The emphasis on random access from disk that is built into the rules leads to disk configurations that are very improbable in practice, such as 1PB of disks for 3TB of <a href="http://dbpedia.org/resource/Data" id="link-id0x191cd880">data</a>, just so there are enough disk arms in parallel. Stonebraker also pointed out that replication and failover were ubiquitous in real life and that roll forward from logs was unrealistic as a recovery model since it took so long. Benchmarks should therefore include replication.</p> <p>Further, Stonebraker challenged the TPC to go for the new frontier, which he described as the huge data sets in science and on big web sites. Scientists, the ones who would save our planet from the diverse ills confronting it, do not like relational databases. They avoid them when can. They want arrays for physics, and graphs for biology and chemistry. <a href="http://dbpedia.org/resource/MapReduce" id="link-id0x53f6040">MapReduce</a> is eating database's lunch; what will you do about this?</p> <p>I later suggested incorporating an <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x18902070">RDF</a> <a href="http://dbpedia.org/resource/Metadata" id="link-id0x3990af8">metadata</a> benchmark into the TPC suite. We'll see about this; we'll first have to come up with a suitable one. There is a great deal of pressure for making good RDF benchmarks but this is not yet in the center of the mainstream that TPC tends to cover.</p> <p>TPC's own talk was about the life cycle of benchmarks. A benchmark begins a bit ahead of the mainstream, with a problem that is difficult but not so difficult as to be uncommon. When the solution to this problem becomes commonplace, the benchmark's relevance gradually drops.</p> <p>There was a talk on robustness of query plans which was well to the point. Indeed, there are performance cliffs at certain points; for example, when passing from memory-only to disk-pageable data structures, or when switching from indexed access to table scans, or from loop to hash joins. Quite so. The analysis I really would have liked to see would have been one of what happens when passing from single server to a cluster, and from local joins to cross-partition ones. Also contrasting of <a href="http://dbpedia.org/resource/Cache" id="link-id0x1942aca8">cache</a> fusion and partitioning. We have our own data and experience but we find we don't have time to measure all the other systems.</p> <p>Anyway it is good to raise the question of smooth and predictable performance.</p>
2009-09-01T17:32:30-04:00
Some Interesting VLDB 2009 Papers (2 of 5)
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-09-01#1579
2009-09-01T15:46:14Z
<h3> <a href="http://dbpedia.org/resource/Intel_Corporation" id="link-id0x449c5e0">Intel</a> on <a href="http://dbpedia.org/resource/Hash_join" id="link-id0x4e82430">Hash Join</a> </h3> <p>Intel and <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x10bae5e8">Oracle</a> had measured hash and sort merge joins on Intel Core i7. The result was that hash join with both tables partitioned to match <a href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x3827798">CPU</a> <a href="http://dbpedia.org/resource/Cache" id="link-id0x2545b978">cache</a> was still the best but that sort/merge would catch up with more <a href="http://dbpedia.org/resource/SIMD" id="link-id0x32f4e40">SIMD</a> instructions in the future.</p> <p>We should probably experiment with this but the most important partitioning of hash joins is still between cluster nodes. Within the process, we will see. The tradeoff of doing all in cache-sized partitions is larger intermediate results which in turn will impact the working set of disk pages in RAM. For one-off queries this is OK; for online use this has an effect.</p> <h3>1000 TABLE Queries</h3> <p> <a href="http://dbpedia.org/resource/SAP_AG" id="link-id0x4ed7710">SAP</a> presented a paper about <a href="http://dbpedia.org/resource/Federated_database_system" id="link-id0x26827fd8">federating relational databases</a>. Queries would be expressed against VIEWs defined over remote TABLEs, UNIONed together and so forth. Traditional methods of <a href="http://dbpedia.org/resource/Program_optimization" id="link-id0x3838888">optimization</a> would run out of memory; a single 1000 TABLE plan is already a big thing. Enumerating multiple variations of such is not possible in practice. So the solution was to plan in two stages — first arrange the subqueries and derived TABLEs, and then do the JOIN orders locally. Further, local JOIN orders could even be adjusted at run time based on the actual <a href="http://dbpedia.org/resource/Data" id="link-id0x26033030">data</a>. Nice.</p> <h3>Oracle Subqueries and New Implementation of LOBs</h3> <p>Oracle presented some new <a href="http://dbpedia.org/resource/SQL" id="link-id0x23a0eb48">SQL</a> optimizations, combining and inlining subqueries and derived TABLEs. We do fairly similar things and might extend the repertoire of tricks in the direction outlined by Oracle as and when the need presents itself. This further confirms that SQL and other query optimization is really an incremental collection of specially recognized patterns. We still have not found any other way of doing it.</p> <p>Another interesting piece by Oracle was about their re-implementation of large object support, where they compared LOB loading to file system and raw device speeds.</p> <h3> <a href="http://dbpedia.org/resource/Amadeus_CRS" id="link-id0x1566d470">Amadeus CRS</a> booking system, steady query time for arbitrary single table queries</h3> <p>There was a paper about a memory-resident database that could give steady time for any kind of single-table scan query. The innovation was to not use indices, but to have one partition of the table per processor core, all in memory. Then each core would have exactly two cursors — one reading, the other writing. The write cursor should keep ahead of the read cursor. Like this, there would be no read/write contention on pages, no locking, no multiple threads splitting a tree at different points, none of the complexity of a multithreaded database engine. Then, when the cursor would hit a row, it would look at the set of queries or updates and add the result to the output if there was a result. The data indexes the queries, not the other way around. We have done something similar for detecting changes in a full text corpus but never thought of doing queries this way.</p> <p>Well, we are all about JOINs so this is not for us, but it deserves a mention for being original and clever. And indeed, anything one can ask about a table will likely be served with great predictability.</p> <h3> <a href="http://dbpedia.org/resource/Greenplum" id="link-id0x196b0538">Greenplum</a> </h3> <p> <a href="http://dbpedia.org/resource/Google" id="link-id0x108f8878">Google</a>'s chief economist said that the winning career choice would be to pick a scarce skill that made value from something that was plentiful. For the 2010s this career is that of the statistician/data analyst. We've said it before — the next web is analytics for all. The Greenplum talk was divided between the Fox use case, with 200TB of data about ads, web site traffic, and other things, growing 5TB a day. The message was that cubes and drill down are passé, that it is about complex statistical methods that have to run in the database, that the new kind of geek is the data geek, whose vocation it is to consume and spit out data, discover things in it, and so forth.</p> <p>The technical part was about Greenplum, a SQL database running on a cluster with a <a href="http://dbpedia.org/resource/PostgreSQL" id="link-id0x3106d00">PostgreSQL</a> back-end. The interesting points were embedding <a href="http://dbpedia.org/resource/MapReduce" id="link-id0x17968370">MapReduce</a> into SQL, and using relational tables for arrays and complex data types — pretty much what we also do. Greenplum emphasized scale-out and found column orientation more like a nice-to-have.</p> <h3> <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x119f7948">MonetDB</a>, optimizing database for CPU cache</h3> <p>The MonetDB people from <a href="http://dbpedia.org/resource/National_Research_Institute_for_Mathematics_and_Computer_Science" id="link-id0x3617658">CWI</a> in Amsterdam gave a 10 year best paper award talk about optimizing database for CPU cache. The key point was that if data is stored as columns, it ought also to be transferred as columns inside the execution engine. Materialize big chunks of state to cut down on interpretation overhead and use cache to best effect. They vector for CPU cache; we vector for scale-out, since the only way to ship operations is to ship many at a time. So we might as well vector also in single servers. This could be worth an experiment. Also we regularly visit the topic of <a href="http://dbpedia.org/resource/Column-oriented_DBMS" id="link-id0x38d43d8">column storage</a>. But we are not yet convinced that it would be better than row-style covering indices for <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x17e25760">RDF</a> quads. But something could certainly be tried, given time.</p>
2009-09-01T17:32:45.000003-04:00
Some Interesting VLDB 2009 Papers (2 of 5)
http://www.openlinksw.com:443/weblog/oerling/?date=2009-09-01#1575
2009-09-01T15:46:14Z
<h3> <a href="http://dbpedia.org/resource/Intel_Corporation" id="link-id0x3588e30">Intel</a> on <a href="http://dbpedia.org/resource/Hash_join" id="link-id0x1bc77c90">Hash Join</a> </h3> <p>Intel and <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x2f1d4d8">Oracle</a> had measured hash and sort merge joins on Intel Core i7. The result was that hash join with both tables partitioned to match <a href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x55b2b70">CPU</a> <a href="http://dbpedia.org/resource/Cache" id="link-id0x2a4fef8">cache</a> was still the best but that sort/merge would catch up with more <a href="http://dbpedia.org/resource/SIMD" id="link-id0x4fe8670">SIMD</a> instructions in the future.</p> <p>We should probably experiment with this but the most important partitioning of hash joins is still between cluster nodes. Within the process, we will see. The tradeoff of doing all in cache-sized partitions is larger intermediate results which in turn will impact the working set of disk pages in RAM. For one-off queries this is OK; for online use this has an effect.</p> <h3>1000 TABLE Queries</h3> <p> <a href="http://dbpedia.org/resource/SAP_AG" id="link-id0x55a1018">SAP</a> presented a paper about <a href="http://dbpedia.org/resource/Federated_database_system" id="link-id0x5500758">federating relational databases</a>. Queries would be expressed against VIEWs defined over remote TABLEs, UNIONed together and so forth. Traditional methods of <a href="http://dbpedia.org/resource/Program_optimization" id="link-id0x4f038f0">optimization</a> would run out of memory; a single 1000 TABLE plan is already a big thing. Enumerating multiple variations of such is not possible in practice. So the solution was to plan in two stages — first arrange the subqueries and derived TABLEs, and then do the JOIN orders locally. Further, local JOIN orders could even be adjusted at run time based on the actual <a href="http://dbpedia.org/resource/Data" id="link-id0x41d3560">data</a>. Nice.</p> <h3>Oracle Subqueries and New Implementation of LOBs</h3> <p>Oracle presented some new <a href="http://dbpedia.org/resource/SQL" id="link-id0x39ad838">SQL</a> optimizations, combining and inlining subqueries and derived TABLEs. We do fairly similar things and might extend the repertoire of tricks in the direction outlined by Oracle as and when the need presents itself. This further confirms that SQL and other query optimization is really an incremental collection of specially recognized patterns. We still have not found any other way of doing it.</p> <p>Another interesting piece by Oracle was about their re-implementation of large object support, where they compared LOB loading to file system and raw device speeds.</p> <h3> <a href="http://dbpedia.org/resource/Amadeus_CRS" id="link-id0x3aa1378">Amadeus CRS</a> booking system, steady query time for arbitrary single table queries</h3> <p>There was a paper about a memory-resident database that could give steady time for any kind of single-table scan query. The innovation was to not use indices, but to have one partition of the table per processor core, all in memory. Then each core would have exactly two cursors — one reading, the other writing. The write cursor should keep ahead of the read cursor. Like this, there would be no read/write contention on pages, no locking, no multiple threads splitting a tree at different points, none of the complexity of a multithreaded database engine. Then, when the cursor would hit a row, it would look at the set of queries or updates and add the result to the output if there was a result. The data indexes the queries, not the other way around. We have done something similar for detecting changes in a full text corpus but never thought of doing queries this way.</p> <p>Well, we are all about JOINs so this is not for us, but it deserves a mention for being original and clever. And indeed, anything one can ask about a table will likely be served with great predictability.</p> <h3> <a href="http://dbpedia.org/resource/Greenplum" id="link-id0x3670360">Greenplum</a> </h3> <p> <a href="http://dbpedia.org/resource/Google" id="link-id0x2c5dfb8">Google</a>'s chief economist said that the winning career choice would be to pick a scarce skill that made value from something that was plentiful. For the 2010s this career is that of the statistician/data analyst. We've said it before — the next web is analytics for all. The Greenplum talk was divided between the Fox use case, with 200TB of data about ads, web site traffic, and other things, growing 5TB a day. The message was that cubes and drill down are passé, that it is about complex statistical methods that have to run in the database, that the new kind of geek is the data geek, whose vocation it is to consume and spit out data, discover things in it, and so forth.</p> <p>The technical part was about Greenplum, a SQL database running on a cluster with a <a href="http://dbpedia.org/resource/PostgreSQL" id="link-id0x4e15798">PostgreSQL</a> back-end. The interesting points were embedding <a href="http://dbpedia.org/resource/MapReduce" id="link-id0x4fd3e00">MapReduce</a> into SQL, and using relational tables for arrays and complex data types — pretty much what we also do. Greenplum emphasized scale-out and found column orientation more like a nice-to-have.</p> <h3> <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x416d288">MonetDB</a>, optimizing database for CPU cache</h3> <p>The MonetDB people from <a href="http://dbpedia.org/resource/National_Research_Institute_for_Mathematics_and_Computer_Science" id="link-id0x4ebedb0">CWI</a> in Amsterdam gave a 10 year best paper award talk about optimizing database for CPU cache. The key point was that if data is stored as columns, it ought also to be transferred as columns inside the execution engine. Materialize big chunks of state to cut down on interpretation overhead and use cache to best effect. They vector for CPU cache; we vector for scale-out, since the only way to ship operations is to ship many at a time. So we might as well vector also in single servers. This could be worth an experiment. Also we regularly visit the topic of <a href="http://dbpedia.org/resource/Column-oriented_DBMS" id="link-id0x4d34cb0">column storage</a>. But we are not yet convinced that it would be better than row-style covering indices for <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x4d55d78">RDF</a> quads. But something could certainly be tried, given time.</p>
2009-09-01T17:32:24.000004-04:00
VLDB 2009 (1 of 5)
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-09-01#1578
2009-09-01T15:30:37Z
<p>I was at the <a href="http://vldb2009.org/" id="link-id0x77dd108">VLDB 2009</a> conference in Lyon, France. I will in the next few posts discuss some of the prominent themes and how they relate to our products or to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1a765238">RDF</a> and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x6966070">Linked Data</a>.</p> <p>Firstly, RDF was as good as absent from the presentations and discussions we saw. There were a few mentions in the panel on structured <a href="http://dbpedia.org/resource/Data" id="link-id0x3a536e8">data</a> on the web, however RDF was not in any way seen to be essential for this. There were also a couple of RDF mentions in questions at other sessions, but that was about it.</p> <p>It is a common perception that RDF and database people do not talk with each other. Evidence seems to bear this out.</p> <p>As a database developer I did get a lot of readily applicable ideas from the VLDB talks. These run across the whole range of DBMS topics, from <a href="http://dbpedia.org/resource/Data_compression" id="link-id0x6302f60">key compression</a> and <a href="http://dbpedia.org/resource/SQL" id="link-id0x69163c0">SQL</a> <a href="http://dbpedia.org/resource/Program_optimization" id="link-id0x63a5cf0">optimization</a>, to <a href="http://dbpedia.org/resource/Column-oriented_DBMS" id="link-id0x1b56daf8">column storage</a>, <a href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x57c6168">CPU</a> <a href="http://dbpedia.org/resource/Cache" id="link-id0x1c504710">cache</a> optimization, and the like. In this sense, VLDB is directly relevant to all we do. In a conversation, someone was mildly confused that I should on one hand mention I was doing RDF, and on the other hand also be concerned about database performance. These things are not seen to belong together, even though making RDF do something useful certainly depends on a great deal of database optimization.</p> <p>The question of all questions — that of infinite scale-out with complex queries, resilience, replication, and full database semantics — was strongly in the air.</p> <p>But it was in the air more as a question than as an answer. Not very much at all was said about the performance of distributed query plans, of <a href="http://dbpedia.org/resource/Two-phase_commit_protocol" id="link-id0x637c6b0">2pc</a> (<a href="http://dbpedia.org/resource/Two-phase_commit_protocol" id="link-id0x69386a8">two-phase commit</a>), of the impact of interconnect latency, and such things. On the other hand, people were talking quite liberally about optimizing CPU cache and local multi-core execution, not to mention SQL plans and rewrites. Also, almost nothing was said about transactions.</p> <p>Still, there is bound to be a great deal of work in scale-out of complex workloads by any number of players. Either these things are all figured out and considered self-evidently trivial, or they are so hot that people will go there only by way of allusion and vague reference. I think it is the latter.</p> <p>By and large, we were confirmed in our understanding that infinite scale-out on the go, with redundancy, is the ticket, especially if one can offer complex queries and transactional semantics coupled with instant data loading and <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x7f90a20">schema</a>-last.</p> <p>Column storage and cache optimizations seem to come right after these.</p> <p>Certainly the database space is diversifying.</p> <p> <a href="http://dbpedia.org/resource/MapReduce" id="link-id0x485bd40">MapReduce</a> was discussed quite a bit, as an intruder into what would be the database turf. We have no great problem with MapReduce; we do that in SQL procedures if one likes to program in this way. <a href="http://dbpedia.org/resource/Greenplum" id="link-id0x7cc58c8">Greenplum</a> also seems to have come by the same idea.</p> <p>As said before, RDF and RDF reasoning were ignored. Do these actually offer something to the database side? Certainly for search, discovery, integration, and resource discovery, linked data has evident advantages.</p> <p>Two points of the design space — the warehouse, and the web-scale key-value store — got a lot of attention. Would I do either in RDF? RDF is a slightly different design space point, like key-value with complex queries — on the surface, a fusion of the two. As opposed to RDF, the relational warehouse gains from fixed data-types and task-specific layout, whether row or column. The key-value store gains from having a concept of a semi-structured record, a bit like the RDF subject of a triple, but now with ad-hoc (if any) secondary indices, and inline blobs. The latter is much simpler and more compact than the generic RDF subject with graphs and all, and can be easily treated as a unit of version control and replication mastering. RDF, being more generic and more normalized, is representationally neither as ad-hoc nor as compact.</p> <p>But RDF will be the natural choice when complex queries and ad-hoc schema meet, for example in web-wide integrations of application data.</p> <p>There seems to be a huge divide in understanding between database-developing people and those who would be using databases. On one side, this has led to a back-to-basics movement with no SQL, no <a href="http://dbpedia.org/resource/ACID" id="link-id0x6390650">ACID</a>, key-value pairs instead of schema, MapReduce instead of fancy but hard-to-follow parallel execution plans. On the other side, the database space specializes more and more; it is no longer simply transactions vs. analytics, but many more points of specialization.</p> <p>Some frustration can be sensed in the ivory towers of science when it is seen that the ones most in need of database understanding in fact have the least. <a href="http://dbpedia.org/resource/Google" id="link-id0x1af4e7e0">Google</a>, <a href="http://dbpedia.org/resource/Yahoo%21" id="link-id0x75145d8">Yahoo</a>!, and <a href="http://dbpedia.org/resource/Microsoft" id="link-id0x17bd7d90">Microsoft</a> know what they are doing, with or without SQL, but the medium-size or fast-growing web sites seem to be in confusion when <a href="http://en.wikipedia.org/wiki/LAMP_%28software_bundle%29" id="link-id0x1bf238e0">LAMP</a> or <a href="http://dbpedia.org/resource/Ruby_programming_language" id="link-id0x6ca3848">Ruby</a> or the scripting-du-jour can no longer cut it.</p> <p>Can somebody using a database be expected to understand how it works? I would say no, not in general. Can a database be expected to unerringly self-configure based on workload? Sure, a database can suggest layouts, but it ought not restructure itself on the spur of the moment under full load.</p> <p>It is safe to say that the community at large no longer believes in "one size fits all". Since there is no general solution, there is a fragmented space of specific solutions. We will be looking at some of these issues in the following posts.</p>
2009-09-01T16:53:25-04:00
VLDB 2009 (1 of 5)
http://www.openlinksw.com:443/weblog/oerling/?date=2009-09-01#1574
2009-09-01T15:30:37Z
<p>I was at the <a href="http://vldb2009.org/" id="link-id0x6700588">VLDB 2009</a> conference in Lyon, France. I will in the next few posts discuss some of the prominent themes and how they relate to our products or to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x69386a8">RDF</a> and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x7537ce0">Linked Data</a>.</p> <p>Firstly, RDF was as good as absent from the presentations and discussions we saw. There were a few mentions in the panel on structured <a href="http://dbpedia.org/resource/Data" id="link-id0x1c201ed0">data</a> on the web, however RDF was not in any way seen to be essential for this. There were also a couple of RDF mentions in questions at other sessions, but that was about it.</p> <p>It is a common perception that RDF and database people do not talk with each other. Evidence seems to bear this out.</p> <p>As a database developer I did get a lot of readily applicable ideas from the VLDB talks. These run across the whole range of DBMS topics, from <a href="http://dbpedia.org/resource/Data_compression" id="link-id0x1b802010">key compression</a> and <a href="http://dbpedia.org/resource/SQL" id="link-id0x48bc820">SQL</a> <a href="http://dbpedia.org/resource/Program_optimization" id="link-id0x218bd558">optimization</a>, to <a href="http://dbpedia.org/resource/Column-oriented_DBMS" id="link-id0x238a39c8">column storage</a>, <a href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x6694538">CPU</a> <a href="http://dbpedia.org/resource/Cache" id="link-id0x4895568">cache</a> optimization, and the like. In this sense, VLDB is directly relevant to all we do. In a conversation, someone was mildly confused that I should on one hand mention I was doing RDF, and on the other hand also be concerned about database performance. These things are not seen to belong together, even though making RDF do something useful certainly depends on a great deal of database optimization.</p> <p>The question of all questions — that of infinite scale-out with complex queries, resilience, replication, and full database semantics — was strongly in the air.</p> <p>But it was in the air more as a question than as an answer. Not very much at all was said about the performance of distributed query plans, of <a href="http://dbpedia.org/resource/Two-phase_commit_protocol" id="link-id0x7a4b208">2pc</a> (<a href="http://dbpedia.org/resource/Two-phase_commit_protocol" id="link-id0x1a0e8ac8">two-phase commit</a>), of the impact of interconnect latency, and such things. On the other hand, people were talking quite liberally about optimizing CPU cache and local multi-core execution, not to mention SQL plans and rewrites. Also, almost nothing was said about transactions.</p> <p>Still, there is bound to be a great deal of work in scale-out of complex workloads by any number of players. Either these things are all figured out and considered self-evidently trivial, or they are so hot that people will go there only by way of allusion and vague reference. I think it is the latter.</p> <p>By and large, we were confirmed in our understanding that infinite scale-out on the go, with redundancy, is the ticket, especially if one can offer complex queries and transactional semantics coupled with instant data loading and <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x23a0c590">schema</a>-last.</p> <p>Column storage and cache optimizations seem to come right after these.</p> <p>Certainly the database space is diversifying.</p> <p> <a href="http://dbpedia.org/resource/MapReduce" id="link-id0x185e6370">MapReduce</a> was discussed quite a bit, as an intruder into what would be the database turf. We have no great problem with MapReduce; we do that in SQL procedures if one likes to program in this way. <a href="http://dbpedia.org/resource/Greenplum" id="link-id0x1ad96d68">Greenplum</a> also seems to have come by the same idea.</p> <p>As said before, RDF and RDF reasoning were ignored. Do these actually offer something to the database side? Certainly for search, discovery, integration, and resource discovery, linked data has evident advantages.</p> <p>Two points of the design space — the warehouse, and the web-scale key-value store — got a lot of attention. Would I do either in RDF? RDF is a slightly different design space point, like key-value with complex queries — on the surface, a fusion of the two. As opposed to RDF, the relational warehouse gains from fixed data-types and task-specific layout, whether row or column. The key-value store gains from having a concept of a semi-structured record, a bit like the RDF subject of a triple, but now with ad-hoc (if any) secondary indices, and inline blobs. The latter is much simpler and more compact than the generic RDF subject with graphs and all, and can be easily treated as a unit of version control and replication mastering. RDF, being more generic and more normalized, is representationally neither as ad-hoc nor as compact.</p> <p>But RDF will be the natural choice when complex queries and ad-hoc schema meet, for example in web-wide integrations of application data.</p> <p>There seems to be a huge divide in understanding between database-developing people and those who would be using databases. On one side, this has led to a back-to-basics movement with no SQL, no <a href="http://dbpedia.org/resource/ACID" id="link-id0x2ec4088">ACID</a>, key-value pairs instead of schema, MapReduce instead of fancy but hard-to-follow parallel execution plans. On the other side, the database space specializes more and more; it is no longer simply transactions vs. analytics, but many more points of specialization.</p> <p>Some frustration can be sensed in the ivory towers of science when it is seen that the ones most in need of database understanding in fact have the least. <a href="http://dbpedia.org/resource/Google" id="link-id0x7748540">Google</a>, <a href="http://dbpedia.org/resource/Yahoo%21" id="link-id0x1ba44020">Yahoo</a>!, and <a href="http://dbpedia.org/resource/Microsoft" id="link-id0x5788710">Microsoft</a> know what they are doing, with or without SQL, but the medium-size or fast-growing web sites seem to be in confusion when <a href="http://en.wikipedia.org/wiki/LAMP_%28software_bundle%29" id="link-id0x18098f18">LAMP</a> or <a href="http://dbpedia.org/resource/Ruby_programming_language" id="link-id0x4844138">Ruby</a> or the scripting-du-jour can no longer cut it.</p> <p>Can somebody using a database be expected to understand how it works? I would say no, not in general. Can a database be expected to unerringly self-configure based on workload? Sure, a database can suggest layouts, but it ought not restructure itself on the spur of the moment under full load.</p> <p>It is safe to say that the community at large no longer believes in "one size fits all". Since there is no general solution, there is a fragmented space of specific solutions. We will be looking at some of these issues in the following posts.</p>
2009-09-01T16:53:20-04:00
Provenance and Reification in Virtuoso
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-09-01#1573
2009-09-01T14:44:08Z
<p>These days, <a href="http://dbpedia.org/resource/Data" id="link-id0x37019c8">data</a> provenance is a big topic across the board, ranging from the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x53c3620">linked data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x4aa3848">web</a>, to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x385aff0">RDF</a> in general, to any kind of data integration, with or without RDF. Especially with scientific data we encounter the need for metadata and provenance, repeatability of experiments, etc. Data without context is worthless, yet the producers of said data do not always have a model or budget for metadata. And if they do, the approach is often a proprietary relational schema with web services in front.</p> <p>RDF and linked data principles could evidently be a great help. This is a large topic that goes into the culture of doing science and will deserve a more extensive treatment down the road.</p> <p>For now, I will talk about possible ways of dealing with provenance annotations in <a href="http://virtuoso.openlinksw.com" id="link-id0x51c4da0">Virtuoso</a> at a fairly technical level.</p> <p>If data comes many-triples-at-a-time from some source (e.g., library catalogue, user of a social network), then it is often easiest to put the data from each source/user into its own graph. Annotations can then be made on the graph. The graph IRI will simply occur as the subject of a triple in the same or some other graph. For example, all such annotations could go into a special annotations graph.</p> <p>On the query side, having lots of distinct graphs does not have to be a problem if the index scheme is the right one, i.e., the 4 index scheme <a href="http://docs.openlinksw.com/virtuoso/rdfperformancetuning.html#rdfperfindexes" id="link-id142a0798">discussed in the Virtuoso documentation</a>. If the query does not specify a graph, then triples in any graph will be considered when evaluating the query.</p> <p>One could write queries like —</p> <blockquote> <code><pre>SELECT ?pub WHERE { GRAPH ?g { ?person foaf:knows ?contact } ?contact foaf:name "Alice" . ?g xx:has_publisher ?pub }</pre> </code> </blockquote> <p>This would return the publishers of graphs that assert that somebody knows Alice.</p> <p>Of course, the <a href="http://www.w3.org/TR/2004/REC-rdf-primer-20040210/#reification" id="link-id14fa9488">RDF reification vocabulary</a> can be used as-is to say things about single triples. It is however very inefficient and is not supported by any specific optimization. Further, reification does not seem to get used very much; thus there is no great pressure to specially optimize it.</p> <p>If we have to say things about specific triples and this occurs frequently (i.e., for more than 10% or so of the triples), then modifying the quad table becomes an option. For all its inefficiency, the RDF reification vocabulary is applicable if reification is a rarity.</p> <p>Virtuoso's <code>RDF_QUAD</code> table can be altered to have more columns. The problem with this is that space usage is increased and the RDF loading and query functions will not know about the columns. A <a href="http://dbpedia.org/resource/SQL" id="link-id0x4784bf0">SQL</a> update statement can be used to set values for these additional columns if one knows the <code>G,S,P,O</code>. </p> <p>Suppose we annotated each quad with the user who inserted it and a timestamp. These would be columns in the <code>RDF_QUAD</code> table. The next choice would be whether these were primary key parts or dependent parts. If primary key parts, these would be non-<code>NULL</code> and would occur on every index. The same quad would exist for each distinct user and time this quad had been inserted. For loading functions to work, these columns would need a default. In practice, we think that having such metadata as a dependent part is more likely, so that <code>G,S,P,O</code> are the unique identifier of the quad. Whether one would then include these columns on indices other than the primary key would depend on how frequently they were accessed.</p> <p>In <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x4a8a7c0">SPARQL</a>, one could use an extension syntax like —</p> <blockquote> <code><pre>SELECT * WHERE { ?person foaf:knows ?connection OPTION ( time ?ts ) . ?connection foaf:name "Alice" . FILTER ( ?ts > "2009-08-08"^^xsd:datetime ) }</pre> </code> </blockquote> <p>This would return everybody who knows Alice since a date more recent than 2009-08-08. This presupposes that the quad table has been extended with a datetime column.</p> <p>The <code>OPTION (time ?ts)</code> syntax is not presently supported but we can easily add something of the sort if there is user demand for it. In practice, this would be an extension mechanism enabling one to access extension columns of <code>RDF_QUAD</code> via a column <code>?variable</code> syntax in the <code>OPTION</code> clause.</p> <p>If quad metadata were not for every quad but still relatively frequent, another possibility would be making a separate table with a key of <code>GSPO</code> and a dependent part of <code>R</code>, where <code>R</code> would be the reification <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x49e6108">URI</a> of the quad. Reification statements would then be made with <code>R</code> as a subject. This would be more compact than the reification vocabulary and would not modify the <code>RDF_QUAD</code> table. The syntax for referring to this could be something like —</p> <blockquote> <code><pre>SELECT * WHERE { ?person foaf:knows ?contact OPTION ( reify ?r ) . ?r xx:assertion_time ?ts . ?contact foaf:name "Alice" . FILTER ( ?ts > "2008-8-8"^^xsd:datetime ) }</pre> </code> </blockquote> <p>We could even recognize the reification vocabulary and convert it into the reify option if this were really necessary. But since it is so unwieldy I don't think there would be huge demand. Who knows? You tell us.</p>
2009-09-01T11:20:46.000006-04:00
Provenance and Reification in Virtuoso
http://www.openlinksw.com:443/weblog/oerling/?date=2009-09-01#1572
2009-09-01T14:44:08Z
<p>These days, <a href="http://dbpedia.org/resource/Data" id="link-id0x4a44870">data</a> provenance is a big topic across the board, ranging from the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x4e10e60">linked data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x4738350">web</a>, to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1fe33310">RDF</a> in general, to any kind of data integration, with or without RDF. Especially with scientific data we encounter the need for metadata and provenance, repeatability of experiments, etc. Data without context is worthless, yet the producers of said data do not always have a model or budget for metadata. And if they do, the approach is often a proprietary relational schema with web services in front.</p> <p>RDF and linked data principles could evidently be a great help. This is a large topic that goes into the culture of doing science and will deserve a more extensive treatment down the road.</p> <p>For now, I will talk about possible ways of dealing with provenance annotations in <a href="http://virtuoso.openlinksw.com" id="link-id0x36581e8">Virtuoso</a> at a fairly technical level.</p> <p>If data comes many-triples-at-a-time from some source (e.g., library catalogue, user of a social network), then it is often easiest to put the data from each source/user into its own graph. Annotations can then be made on the graph. The graph IRI will simply occur as the subject of a triple in the same or some other graph. For example, all such annotations could go into a special annotations graph.</p> <p>On the query side, having lots of distinct graphs does not have to be a problem if the index scheme is the right one, i.e., the 4 index scheme <a href="http://docs.openlinksw.com/virtuoso/rdfperformancetuning.html#rdfperfindexes" id="link-id142a0798">discussed in the Virtuoso documentation</a>. If the query does not specify a graph, then triples in any graph will be considered when evaluating the query.</p> <p>One could write queries like —</p> <blockquote> <code><pre>SELECT ?pub WHERE { GRAPH ?g { ?person foaf:knows ?contact } ?contact foaf:name "Alice" . ?g xx:has_publisher ?pub }</pre> </code> </blockquote> <p>This would return the publishers of graphs that assert that somebody knows Alice.</p> <p>Of course, the <a href="http://www.w3.org/TR/2004/REC-rdf-primer-20040210/#reification" id="link-id14fa9488">RDF reification vocabulary</a> can be used as-is to say things about single triples. It is however very inefficient and is not supported by any specific optimization. Further, reification does not seem to get used very much; thus there is no great pressure to specially optimize it.</p> <p>If we have to say things about specific triples and this occurs frequently (i.e., for more than 10% or so of the triples), then modifying the quad table becomes an option. For all its inefficiency, the RDF reification vocabulary is applicable if reification is a rarity.</p> <p>Virtuoso's <code>RDF_QUAD</code> table can be altered to have more columns. The problem with this is that space usage is increased and the RDF loading and query functions will not know about the columns. A <a href="http://dbpedia.org/resource/SQL" id="link-id0x4b1d938">SQL</a> update statement can be used to set values for these additional columns if one knows the <code>G,S,P,O</code>. </p> <p>Suppose we annotated each quad with the user who inserted it and a timestamp. These would be columns in the <code>RDF_QUAD</code> table. The next choice would be whether these were primary key parts or dependent parts. If primary key parts, these would be non-<code>NULL</code> and would occur on every index. The same quad would exist for each distinct user and time this quad had been inserted. For loading functions to work, these columns would need a default. In practice, we think that having such metadata as a dependent part is more likely, so that <code>G,S,P,O</code> are the unique identifier of the quad. Whether one would then include these columns on indices other than the primary key would depend on how frequently they were accessed.</p> <p>In <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x472afb0">SPARQL</a>, one could use an extension syntax like —</p> <blockquote> <code><pre>SELECT * WHERE { ?person foaf:knows ?connection OPTION ( time ?ts ) . ?connection foaf:name "Alice" . FILTER ( ?ts > "2009-08-08"^^xsd:datetime ) }</pre> </code> </blockquote> <p>This would return everybody who knows Alice since a date more recent than 2009-08-08. This presupposes that the quad table has been extended with a datetime column.</p> <p>The <code>OPTION (time ?ts)</code> syntax is not presently supported but we can easily add something of the sort if there is user demand for it. In practice, this would be an extension mechanism enabling one to access extension columns of <code>RDF_QUAD</code> via a column <code>?variable</code> syntax in the <code>OPTION</code> clause.</p> <p>If quad metadata were not for every quad but still relatively frequent, another possibility would be making a separate table with a key of <code>GSPO</code> and a dependent part of <code>R</code>, where <code>R</code> would be the reification <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x365b190">URI</a> of the quad. Reification statements would then be made with <code>R</code> as a subject. This would be more compact than the reification vocabulary and would not modify the <code>RDF_QUAD</code> table. The syntax for referring to this could be something like —</p> <blockquote> <code><pre>SELECT * WHERE { ?person foaf:knows ?contact OPTION ( reify ?r ) . ?r xx:assertion_time ?ts . ?contact foaf:name "Alice" . FILTER ( ?ts > "2008-8-8"^^xsd:datetime ) }</pre> </code> </blockquote> <p>We could even recognize the reification vocabulary and convert it into the reify option if this were really necessary. But since it is so unwieldy I don't think there would be huge demand. Who knows? You tell us.</p>
2015-08-20T17:26:18.015613-04:00
More On Parallel RDF/Text Query Evaluation
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-08-19#1571
2009-08-19T17:28:50Z
<p>We have received some more questions about <a href="http://virtuoso.openlinksw.com" id="link-id0x266cd288">Virtuoso</a>'s parallel query evaluation model.</p> <p>In answer, we will here explain how we do search engine style processing by writing <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x23c628b8">SPARQL</a>. There is no need for custom procedural code because the query optimizer does all the partitioning and the equivalent of map reduce.</p> <p>The point is that what used to require programming can often be done in a generic query language. The technical detail is that the implementation must be smart enough with respect to parallelizing queries for this to be of practical benefit. But by combining these two things, we are a step closer to the web being the database.</p> <p>I will here show how we do some joins combining full text, <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x22ff08b0">RDF</a> conditions, and aggregates and <code>ORDER BY</code>. The sample task is finding the top 20 entities with New York in some attribute value. Then we specify the search further by only taking actors associated with New York. The results are returned in the order of a composite of <a href="http://dbpedia.org/resource/Entity" id="link-id0x22da5258">entity</a> rank and text match score.</p> <p>The basic query is:</p> <blockquote> <code><pre> SELECT ( <sql:s_sum_page> ( <<a href="http://docs.openlinksw.com/virtuoso/fn_VECTOR_AGG.html" id="link-id0x2aab6151b368">sql:vector_agg</a>> ( <<a href="http://docs.openlinksw.com/virtuoso/fn_vector.html" id="link-id0x2aab607a70e8">bif:vector</a>> ( ?c1 , ?sm , ?g1 ) ), <bif:vector> ( 'NEW', 'YORK' ) ) ) AS ?res WHERE { { SELECT ( <SHORT_OR_LONG::> ( ?s1 ) ) AS ?c1 , ( <sql:S_SUM> ( <SHORT_OR_LONG::IRI_RANK> ( ?s1 ) , <SHORT_OR_LONG::> ( ?s1textp ) , <SHORT_OR_LONG::> ( ?o1 ) , ?sc ) ) AS ?sm , <SHORT_OR_LONG::> ( ?g ) AS ?g1 WHERE { QUAD MAP virtrdf:DefaultQuadMap { graph ?g { ?s1 ?s1textp ?o1 . ?o1 <<a href="http://docs.openlinksw.com/virtuoso/fn_contains.html" id="link-id0x2aab61f60d78">bif:contains</a>> '( NEW AND YORK )' OPTION ( SCORE ?sc ) } } } ORDER BY DESC ( <sql:sum_rank> (( <sql:S_SUM> ( <SHORT_OR_LONG::IRI_RANK> ( ?s1 ) , <SHORT_OR_LONG::> ( ?s1textp ) , <SHORT_OR_LONG::> ( ?o1 ) , ?sc ) )) ) LIMIT 20 OFFSET 0 } } </pre> </code> </blockquote> <p>This takes some explaining. The basic part is</p> <blockquote> <code><pre>{ ?s1 ?s1textp ?o1 . ?o1 <bif:contains> '( NEW AND YORK )' OPTION ( SCORE ?sc ) }</pre> </code> </blockquote> <p>This just makes tuples where <code>?s1</code> is the object (entity), <code>?s1textp</code> the property (attribute), and <code>?o1</code> the literal (value) which contains the strings <code>NEW</code> and <code>YORK</code>. For a single <code>?s1</code>, there can of course be many properties which all contain <code>NEW</code> and <code>YORK</code>.</p> <p>The rest of the query gathers all the "New York" containing properties of an entity into a single aggregate, and then gets the entity ranks of all such entities.</p> <p>After this, the aggregates are sorted by a sum of the entity rank and a combined text score calculated based on the individual text match scores between "New York" and the strings containing "New York". The text hit score is higher if the words repeat often and in close proximity.</p> <p>The <code>S_SUM</code> function is a user-defined aggregate (<a href="http://demo.openlinksw.com/DAV/VAD/fct/sum.sql" id="link-id0x2aab61e70b58">borrowed from the FCT application; subject to change over time</a>) which takes 4 arguments: The rank of the subject of the triple; the predicate of the triple containing the text; the object of the triple containing the text; and the text match score.</p> <p>These are grouped by the subject of the triple. After this, these are sorted by <code>sum_rank</code> of the aggregate constructed with <code>S_SUM</code>. The <code>sum_rank</code> (<a href="http://demo.openlinksw.com/DAV/VAD/fct/sum.sql" id="link-id0x2aab61563e28">also borrowed from the FCT application; subject to change over time</a>) is a <a href="http://dbpedia.org/resource/SQL" id="link-id0x2aab62ee4618">SQL</a> function combining the entity rank with the text scores of the different literals.</p> <p>This executes as one would expect: All partitions make a text index lookup, retrieving the object of the triple. The text index entries of an object are stored in the same partition as the object. But the entity rank is a property of the subject and is partitioned by the subject. Also the <code>GROUP BY</code> is by the subject. Thus the <a href="http://dbpedia.org/resource/Data" id="link-id0x24381030">data</a> is produced from all partitions, then streamed into the receiving partitions, determined by the subject. This partition can then get the score and group the matches by the subject. Since all these partial aggregates are partitioned by the subject, there is no need to merge them; thus, the top <code>k</code> sort can be done for each partition separately. Finally, the top 20 of each partition are merged into the global top 20. This is then passed to a final function <code>s_sum_page</code> (<a href="http://demo.openlinksw.com/DAV/VAD/fct/sum.sql" id="link-id0x2aab61d3da28">also borrowed from the FCT application; subject to change over time</a>) that turns this all into an <a href="http://dbpedia.org/resource/XML" id="link-id0x2363d6c0">XML</a> fragment that can be processed with XSLT for inclusion on a web page.</p> <p>This differs from the text search engine in that the query pipeline can contain arbitrary cross-partition joins. Also, the string "New York" is a common label that occurs in many distinct entities. Thus one text match, to one document, in the case the containing only the string "New York" will get many entities, likely all from different partitions.</p> <p>So, if we only want actors with a mention of "New York", we need to get the inner part of the query as:</p> <blockquote> <code><pre>{ ?s1 ?s1textp ?o1 . ?o1 <bif:contains> ' ( NEW AND YORK ) ' OPTION ( SCORE ?sc ) . ?s1 a <http://umbel.org/umbel/sc/Actor> }</pre> </code> </blockquote> <p>Whether an entity is an actor can be checked in the same partition as the rank of the entity. Thus the query plan gets this check right before getting the rank. This is natural since there is no point in getting the rank of something that is not an actor.</p> <p>The <code><SHORT_OR_LONG::sql:func></code> notation means that we call <code>func</code>, which is a SQL stored procedure with the arguments in their internal form. Thus, if a variable bound to an IRI is passed, the <code>SHORT_OR_LONG</code> specifies that it is passed as its internal ID and is not converted into its text form. This is essential, since there is no point getting the text of half a million IRIs when only 20 at most will be shown in the end.</p> <p>Now, when we run this on a collection of 4.5 billion triples of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x24381160">linked data</a>, once we have the working set, we can get the top 20 "New York" occurrences, with text summaries and all, in just 1.1s, with 12 of 16 cores busy. (The hardware is two boxes with two quad-core Xeon 5345 each.)</p> <p>If we run this query in two parallel sessions, we get both results in 1.9s, with 14 of 16 cores busy. This gets about 200K "New York" strings, which becomes about 400K entities with New York somewhere, for which a rank then has to be retrieved. After this, all the possibly-many occurrences of New York in the title, text, and other properties of the entity are aggregated together, resolving into some 220K groups. These are then sorted. This is internally over 1.5 million random lookups and some 40MB of traffic between processes. Restricting the type of the entity to actor drops the execution time of one query to 0.8s because there are then fewer ranks to retrieve and less data to aggregate and sort.</p> <p>By adding partitions and cores, we scale horizontally, as evaluating the query involves almost no central control, even though data are swapped between partitions. There is some flow control to avoid constructing overly-large intermediate results but generally partitions run independently and asynchronously. In the above case, there is just one fence at the point where all aggregates are complete, so that they can be sorted; otherwise, all is asynchronous.</p> <p>Doing <code>JOINs</code> between partitions and partitioned <code>GROUP BY</code>/<code>ORDER BY</code> is pretty regular database stuff. Applying this to RDF is a most natural thing.</p> <p>If we do not parallelize the user-defined aggregate for grouping all the "New York" occurrences, the query takes 8s instead of 1.1s. If we could not put SQL procedures as user-defined aggregates to be parallelized with the query, we'd have to either bring all the data to a central point before the top k, which would destroy performance, or we would have to do procedures with explicit parallel procedure calls which is hard to write, surely too hard for <i>ad hoc</i> queries.</p> <a href="http://bit.ly/1QpHoRj" id="link-id0x2aabc203ec28">Results of live execution</a> of <a href="http://bit.ly/1Ihmm8o" id="link-id0x2aabc2396c28">this query</a> may not be complete on initial load, as this link includes a "Virtuoso Anytime" timeout of 10 seconds. Running against a cold cache, these results may take much longer to return; a warm cache will deliver response times along the lines of those discussed above. <p>Engineering matters. If we wish to commoditize queries on a lot of data, such intelligence in the DBMS is necessary; it is very unscalable to require people to do procedural code or give query parallelization hints. If you need to optimize a workload of 10 different transactions, this is of course possible and even desirable, but for the infinity of all search or analysis, this will not happen.</p>
2015-06-03T12:03:52.939352-04:00
More On Parallel RDF/Text Query Evaluation
http://www.openlinksw.com:443/weblog/oerling/?date=2009-08-19#1570
2009-08-19T17:28:50Z
<p>We have received some more questions about <a href="http://virtuoso.openlinksw.com" id="link-id0x266cd288">Virtuoso</a>'s parallel query evaluation model.</p> <p>In answer, we will here explain how we do search engine style processing by writing <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x23c628b8">SPARQL</a>. There is no need for custom procedural code because the query optimizer does all the partitioning and the equivalent of map reduce.</p> <p>The point is that what used to require programming can often be done in a generic query language. The technical detail is that the implementation must be smart enough with respect to parallelizing queries for this to be of practical benefit. But by combining these two things, we are a step closer to the web being the database.</p> <p>I will here show how we do some joins combining full text, <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x22ff08b0">RDF</a> conditions, and aggregates and <code>ORDER BY</code>. The sample task is finding the top 20 entities with New York in some attribute value. Then we specify the search further by only taking actors associated with New York. The results are returned in the order of a composite of <a href="http://dbpedia.org/resource/Entity" id="link-id0x22da5258">entity</a> rank and text match score.</p> <p>The basic query is:</p> <blockquote> <code><pre> SELECT ( <sql:s_sum_page> ( <<a href="http://docs.openlinksw.com/virtuoso/fn_VECTOR_AGG.html" id="link-id0x2aab61593558">sql:vector_agg</a>> ( <<a href="http://docs.openlinksw.com/virtuoso/fn_vector.html" id="link-id0x2aab61593688">bif:vector</a>> ( ?c1 , ?sm , ?g1 ) ), <bif:vector> ( 'NEW', 'YORK' ) ) ) AS ?res WHERE { { SELECT ( <SHORT_OR_LONG::> ( ?s1 ) ) AS ?c1 , ( <sql:S_SUM> ( <SHORT_OR_LONG::IRI_RANK> ( ?s1 ) , <SHORT_OR_LONG::> ( ?s1textp ) , <SHORT_OR_LONG::> ( ?o1 ) , ?sc ) ) AS ?sm , <SHORT_OR_LONG::> ( ?g ) AS ?g1 WHERE { QUAD MAP virtrdf:DefaultQuadMap { graph ?g { ?s1 ?s1textp ?o1 . ?o1 <<a href="http://docs.openlinksw.com/virtuoso/fn_contains.html" id="link-id0x2aab61593788">bif:contains</a>> '( NEW AND YORK )' OPTION ( SCORE ?sc ) } } } ORDER BY DESC ( <sql:sum_rank> (( <sql:S_SUM> ( <SHORT_OR_LONG::IRI_RANK> ( ?s1 ) , <SHORT_OR_LONG::> ( ?s1textp ) , <SHORT_OR_LONG::> ( ?o1 ) , ?sc ) )) ) LIMIT 20 OFFSET 0 } } </pre> </code> </blockquote> <p>This takes some explaining. The basic part is</p> <blockquote> <code><pre>{ ?s1 ?s1textp ?o1 . ?o1 <bif:contains> '( NEW AND YORK )' OPTION ( SCORE ?sc ) }</pre> </code> </blockquote> <p>This just makes tuples where <code>?s1</code> is the object (entity), <code>?s1textp</code> the property (attribute), and <code>?o1</code> the literal (value) which contains the strings <code>NEW</code> and <code>YORK</code>. For a single <code>?s1</code>, there can of course be many properties which all contain <code>NEW</code> and <code>YORK</code>.</p> <p>The rest of the query gathers all the "New York" containing properties of an entity into a single aggregate, and then gets the entity ranks of all such entities.</p> <p>After this, the aggregates are sorted by a sum of the entity rank and a combined text score calculated based on the individual text match scores between "New York" and the strings containing "New York". The text hit score is higher if the words repeat often and in close proximity.</p> <p>The <code>S_SUM</code> function is a user-defined aggregate (<a href="http://demo.openlinksw.com/DAV/VAD/fct/sum.sql" id="link-id0x2aab61e65028">borrowed from the FCT application; subject to change over time</a>) which takes 4 arguments: The rank of the subject of the triple; the predicate of the triple containing the text; the object of the triple containing the text; and the text match score.</p> <p>These are grouped by the subject of the triple. After this, these are sorted by <code>sum_rank</code> of the aggregate constructed with <code>S_SUM</code>. The <code>sum_rank</code> (<a href="http://demo.openlinksw.com/DAV/VAD/fct/sum.sql" id="link-id0x2aab61e654d8">also borrowed from the FCT application; subject to change over time</a>) is a <a href="http://dbpedia.org/resource/SQL" id="link-id0x2aab61e65608">SQL</a> function combining the entity rank with the text scores of the different literals.</p> <p>This executes as one would expect: All partitions make a text index lookup, retrieving the object of the triple. The text index entries of an object are stored in the same partition as the object. But the entity rank is a property of the subject and is partitioned by the subject. Also the <code>GROUP BY</code> is by the subject. Thus the <a href="http://dbpedia.org/resource/Data" id="link-id0x24381030">data</a> is produced from all partitions, then streamed into the receiving partitions, determined by the subject. This partition can then get the score and group the matches by the subject. Since all these partial aggregates are partitioned by the subject, there is no need to merge them; thus, the top <code>k</code> sort can be done for each partition separately. Finally, the top 20 of each partition are merged into the global top 20. This is then passed to a final function <code>s_sum_page</code> (<a href="http://demo.openlinksw.com/DAV/VAD/fct/sum.sql" id="link-id0x2aab61533cf8">also borrowed from the FCT application; subject to change over time</a>) that turns this all into an <a href="http://dbpedia.org/resource/XML" id="link-id0x2363d6c0">XML</a> fragment that can be processed with XSLT for inclusion on a web page.</p> <p>This differs from the text search engine in that the query pipeline can contain arbitrary cross-partition joins. Also, the string "New York" is a common label that occurs in many distinct entities. Thus one text match, to one document, in the case the containing only the string "New York" will get many entities, likely all from different partitions.</p> <p>So, if we only want actors with a mention of "New York", we need to get the inner part of the query as:</p> <blockquote> <code><pre>{ ?s1 ?s1textp ?o1 . ?o1 <bif:contains> ' ( NEW AND YORK ) ' OPTION ( SCORE ?sc ) . ?s1 a <http://umbel.org/umbel/sc/Actor> }</pre> </code> </blockquote> <p>Whether an entity is an actor can be checked in the same partition as the rank of the entity. Thus the query plan gets this check right before getting the rank. This is natural since there is no point in getting the rank of something that is not an actor.</p> <p>The <code><SHORT_OR_LONG::sql:func></code> notation means that we call <code>func</code>, which is a SQL stored procedure with the arguments in their internal form. Thus, if a variable bound to an IRI is passed, the <code>SHORT_OR_LONG</code> specifies that it is passed as its internal ID and is not converted into its text form. This is essential, since there is no point getting the text of half a million IRIs when only 20 at most will be shown in the end.</p> <p>Now, when we run this on a collection of 4.5 billion triples of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x24381160">linked data</a>, once we have the working set, we can get the top 20 "New York" occurrences, with text summaries and all, in just 1.1s, with 12 of 16 cores busy. (The hardware is two boxes with two quad-core Xeon 5345 each.)</p> <p>If we run this query in two parallel sessions, we get both results in 1.9s, with 14 of 16 cores busy. This gets about 200K "New York" strings, which becomes about 400K entities with New York somewhere, for which a rank then has to be retrieved. After this, all the possibly-many occurrences of New York in the title, text, and other properties of the entity are aggregated together, resolving into some 220K groups. These are then sorted. This is internally over 1.5 million random lookups and some 40MB of traffic between processes. Restricting the type of the entity to actor drops the execution time of one query to 0.8s because there are then fewer ranks to retrieve and less data to aggregate and sort.</p> <p>By adding partitions and cores, we scale horizontally, as evaluating the query involves almost no central control, even though data are swapped between partitions. There is some flow control to avoid constructing overly-large intermediate results but generally partitions run independently and asynchronously. In the above case, there is just one fence at the point where all aggregates are complete, so that they can be sorted; otherwise, all is asynchronous.</p> <p>Doing <code>JOINs</code> between partitions and partitioned <code>GROUP BY</code>/<code>ORDER BY</code> is pretty regular database stuff. Applying this to RDF is a most natural thing.</p> <p>If we do not parallelize the user-defined aggregate for grouping all the "New York" occurrences, the query takes 8s instead of 1.1s. If we could not put SQL procedures as user-defined aggregates to be parallelized with the query, we'd have to either bring all the data to a central point before the top k, which would destroy performance, or we would have to do procedures with explicit parallel procedure calls which is hard to write, surely too hard for <i>ad hoc</i> queries.</p> <a href="http://bit.ly/1QpHoRj" id="link-id0x2aab552f1cd8">Results of live execution</a> of <a href="http://bit.ly/1Ihmm8o" id="link-id0x2aab55297ca8">this query</a> may not be complete on initial load, as this link includes a "Virtuoso Anytime" timeout of 10 seconds. Running against a cold cache, these results may take much longer to return; a warm cache will deliver response times along the lines of those discussed above. <p>Engineering matters. If we wish to commoditize queries on a lot of data, such intelligence in the DBMS is necessary; it is very unscalable to require people to do procedural code or give query parallelization hints. If you need to optimize a workload of 10 different transactions, this is of course possible and even desirable, but for the infinity of all search or analysis, this will not happen.</p>
2015-08-20T17:28:38.277168-04:00
Updated hardware improves LUBM 8000 load rate in Virtuoso 6
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-08-14#1569
2009-08-14T19:01:30Z
<p>We repeated the <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1562" id="link-id173d3068">earlier LUBM 8000 experiment</a> on a newer machine, with 2 x Xeon 5520 and 72G 1333MHz memory, and once again with the 2 machines as a networked cluster. Otherwise the settings were the same.</p> <p>The load rate is now 160,739 triples-per-second.</p> <table> <tr> <th></th> <td> </td> <th align="center"><a href="http://virtuoso.openlinksw.com" id="link-id0x240daf38">Virtuoso</a> 6 <br /> (previous run)</th> <td> </td> <th align="center">Virtuoso 6 <br /> (new run)</th> <td> </td> <th align="center">Virtuoso 6 <br /> (newest run)</th> </tr> <tr> <td align="left">blades</td> <td> </td> <td align="center">1 </td> <td> </td> <td align="center">1 </td> <td> </td> <td align="center">2</td> </tr> <tr> <td align="left">processors</td> <td> </td> <td align="center">2 x Xeon 5410</td> <td> </td> <td align="center">2 x Xeon 5520</td> <td> </td> <td align="center"> 2 x Xeon 5520 <br />+ <br />2 x Xeon 5410 <br />with 1x1GigE <br />interconnect </td> </tr> <tr> <td align="left">memory</td> <td> </td> <td align="center"> 16G 667 MHz</td> <td> </td> <td align="center">72G 1333 MHz</td> <td> </td> <td align="center">72G 1333 MHz <br />+ <br /> 16G 667 MHz <br /> respectively</td> </tr> <tr> <td align="left">reported load rate<br />triples-per-second</td> <td> </td> <td align="center"> 110,532 </td> <td> </td> <td align="center"> 160,739 </td> <td> </td> <td align="center"> 214,188 </td> </tr> </table> <p>Again, if others talk about loading LUBM, so must we. Otherwise, this metric is rather uninteresting.</p>
2009-08-15T15:27:29-04:00
Updated hardware improves LUBM 8000 load rate in Virtuoso 6
http://www.openlinksw.com:443/weblog/oerling/?date=2009-08-14#1568
2009-08-14T19:01:30Z
<p>We repeated the <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1562" id="link-id173d3068">earlier LUBM 8000 experiment</a> on a newer machine, with 2 x Xeon 5520 and 72G 1333MHz memory, and once again with the 2 machines as a networked cluster. Otherwise the settings were the same.</p> <p>The load rate is now 160,739 triples-per-second.</p> <table> <tr> <th></th> <td> </td> <th align="center"><a href="http://virtuoso.openlinksw.com" id="link-id0x199b9740">Virtuoso</a> 6 <br /> (previous run)</th> <td> </td> <th align="center">Virtuoso 6 <br /> (new run)</th> <td> </td> <th align="center">Virtuoso 6 <br /> (newest run)</th> </tr> <tr> <td align="left">blades</td> <td> </td> <td align="center">1 </td> <td> </td> <td align="center">1 </td> <td> </td> <td align="center">2</td> </tr> <tr> <td align="left">processors</td> <td> </td> <td align="center">2 x Xeon 5410</td> <td> </td> <td align="center">2 x Xeon 5520</td> <td> </td> <td align="center"> 2 x Xeon 5520 <br />+ <br />2 x Xeon 5410 <br />with 1x1GigE <br />interconnect </td> </tr> <tr> <td align="left">memory</td> <td> </td> <td align="center"> 16G 667 MHz</td> <td> </td> <td align="center">72G 1333 MHz</td> <td> </td> <td align="center">72G 1333 MHz <br />+ <br /> 16G 667 MHz <br /> respectively</td> </tr> <tr> <td align="left">reported load rate<br />triples-per-second</td> <td> </td> <td align="center"> 110,532 </td> <td> </td> <td align="center"> 160,739 </td> <td> </td> <td align="center"> 214,188 </td> </tr> </table> <p>Again, if others talk about loading LUBM, so must we. Otherwise, this metric is rather uninteresting.</p>
2009-08-15T15:27:25-04:00
The URI, URL, and Linked Data Meme's Generic HTTP URI (Updated)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-08-07#1567
2009-08-07T18:34:50Z
<h3> Situation Analysis</h3> <p> As the "<a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id12f96a00">Linked Data" meme</a> has gained momentum you've more than likely been on the receiving end of dialog with Linked Open <a href="http://dbpedia.org/resource/Data">Data</a> community members (myself included) that goes something like this:</p> <blockquote> <cite>"Do you have a <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id139252a0">URI</a>", "Get yourself a URI", "Give <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id140eab68">me</a> a de-referencable URI" etc..</cite> </blockquote> <p> And each time, you respond with a <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id112c1860">URL</a> -- which to the best of your <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> <a href="http://dbpedia.org/resource/Knowledge" id="link-id140b51c0">knowledge</a> is a bona fide URI. But to your utter confusion you are told: Nah! You gave me a Document URI instead of the URI of a real-world thing or object etc..</p> <h3> What's up with that?</h3> <p> Well our everyday use of the Web is an unfortunate conflation of two distinct things, which have Identity: Real World Objects (RWOs) & Address/Location of Documents (<a href="http://dbpedia.org/resource/Information" id="link-id144838b0">Information</a> bearing Resources).</p> <p> The "Linked Data" meme is about enhancing the Web by unobtrusively reintroducing its core essence: the generic HTTP URI, a vital piece of Web Architecture DNA. Basically, its about so realizing the full capabilities of the Web as a platform for Open Data Identification, Definition, Access, Storage, Representation, Presentation, and Integration.</p> <h3> What is a Real World Object?</h3> <p> People, Places, Music, Books, Cars, Ideas, Emotions etc..</p> <h3> What is a URI?</h3> <p> A Uniform Resource Identifier. A global identifier mechanism for network addressable data items. Its sole function is Name oriented Identification.</p> <h4> URI Generic Syntax</h4> <p> The constituent parts of a URI (from <a href="http://www.ietf.org/rfc/rfc2396.txt" id="link-id1180c700">URI Generic Syntax RFC</a>) are depicted below: <img alt="Image" src="http://virtuoso.openlinksw.com/images/generic_uri_syntax_image.png" /> </p> <h3> What is a URL?</h3> <p> A location oriented HTTP scheme based URI. The HTTP scheme introduces a powerful and inherent duality that delivers:</p> <ol> <li> Resource Address/Location Identifier</li> <li> Data Access mechanism for an Information bearing Resource (Document, File etc..)</li> </ol> <p> So far so good!</p> <h3> What is an HTTP based URI?</h3> <p> The kind of URI <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11100a28">Linked Data</a> aficionados mean when they use the term: URI.</p> <p> An HTTP URI is an HTTP scheme based URI. Unlike a URL, this kind of HTTP scheme URI is devoid of any Web Location orientation or specificity. Thus, Its inherent duality provides a more powerful level of abstraction. Hence, you can use this form of URI to assign Names/Identifiers to Real World Objects (RWO). Even better, courtesy of the Identity/Address duality of the HTTP scheme, a single URI can deliver the following:</p> <ol> <li> RWO Identfier/Name</li> <li> RWO Metadata document Locator (courtesy of URL aspect)</li> <li> Negotiable Representation of the Located Document (courtesy of HTTP's content negotiation feature).</li> </ol> <h3> What is Metadata?</h3> <p> Data about Data. Put differently, data that describes other data in a structured manner.</p> <h3> How Do we Model Metadata?</h3> <p> The predominant model for metadata is the <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id11193d30">Entity</a>-Attribute-Value + Classes & Relationships model (<a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id11725710">EAV</a>/CR). A model that's been with us since the inception of modern computing (long before the Web).</p> <h3> What about RDF?</h3> <p> The Resource Description Framework (RDF) is a framework for describing Web addressable resources. In a nutshell, its a framework for adding Metadata bearing Information Resources to the current Web. Its comprised of:</p> <ol> <li> Entity-Attribute-Value (aka. Subject-Predictate-Object) plus Classes & Relationships (<a href="http://dbpedia.org/resource/Data_dictionary" id="link-id138df0f8">Data Dictionaries</a> e.g., <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id116bf590">OWL</a>) metadata model</li> <li> A plethora of instance data representation formats that include: <a href="http://dbpedia.org/resource/RDFa" id="link-id13360b90">RDFa</a> (when doing so within (X)HTML docs), Turtle, N3, TriX, RDF/XML etc.</li> </ol> <h3> What's the Problem Today?</h3> <p> The ubiquitous use of the Web is primarily focused on a Linked Mesh of Information bearing Documents. URLs rather than generic HTTP URIs are the prime mechanism for Web tapestry; basically, we use URLs to conduct Information -- which is inherently subjective -- instead of using HTTP URIs to conduct "Raw Data" -- which is inherently objective.</p> <blockquote> <strong>Note:</strong> Information is "data in <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id1395ca50">context</a>", it isn't the same thing as "Raw Data". Thus, if we can link to Information via the Web, why shouldn't we be able to do the same for "Raw Data"?</blockquote> <h3> How Does the Link Data <a href="http://dbpedia.org/resource/Meme" id="link-id1160ab70">meme</a> solve the problem?</h3> <p> The meme simply provides a set of guidelines (best practices) for producing Web architecture friendly metadata. Meaning: when producing EAV/CR model based metadata, endow Subjects, their Attributes, and Attribute Values (optionally) with HTTP URIs. By doing so, a new level of Link Abstraction on the Web is possible i.e., "Data Item to Data Item" level links (aka <a href="http://dbpedia.org/resource/Linked_Data" id="link-id137a78a0">hyperdata</a> links). Even better, when you de-reference a RWO hyperdata link you end up with a negotiated representations of its metadata.</p> <h3> Conclusion</h3> <p> Linked Data is ultimately about an HTTP URI for each item in the <a href="http://dbpedia.org/resource/Data_hierarchy" id="link-id1393c3e0">Data Organization Hierarchy</a> :-)</p> <h3> Related</h3> <ol> <li> <a href="http://lists.w3.org/Archives/Public/www-tag/2009Aug/0000.html" id="link-id140c1e78">History of how "Resource" became part of URI</a> - historic account by <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id1172b128">TimBL</a> </li> <li> <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id1338cbd0">Linked Data Design Issues Document</a> - <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id13536ad8">TimBL</a>'s initial Linked Data Guide</li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1561" id="link-id116c1af8">Linked Data Rules Simplified</a> - My attempt at simplifying the Linked Data Meme without <a href="http://dbpedia.org/resource/SPARQL" id="link-id116c3b40">SPARQL</a> & RDF distraction</li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1547" id="link-id135dd1b8">Linked Data & Identity</a> - another related post</li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1565" id="link-id134afc50">The Linked Data Meme's Value Proposition</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1323" id="link-id1251e9248">So What Does "HREF" stand for anyway?</a> </li> <li> <a href="http://delicious.com/kidehen/identifier_scheme" id="link-id14cc7e18">My Del.icio.us hosted Bookmark Data Space for Identity Schemes</a> </li> <li> <a href="http://www.ted.com/talks/tim_berners_lee_on_the_next_web.html" id="link-id115a3748">TimBL's Ted Talk re. "Raw Linked Data"</a> </li> <li> <a href="http://www.infoq.com/articles/roa-rest-of-rest" id="link-id11b25558">Resource Oriented Architecture</a> </li> <li> <a href="http://www.blipfoto.com/view.php?id=465380&month=2&year=2010" id="link-id139824c8">More Famous Than Simon Cowell</a> .</li> </ol>
2010-03-28T12:19:00-04:00
Why Do We Put Stuff On The Web, Really?
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-07-24#1566
2009-07-24T15:54:26Z
<p>As espoused by the Ubuntu philosophy, no Human is an Island. Thus, although the objects of our sociality are vast and varied; that said, the basic foundation still centers on the pursuit and/or delivery of products and services.</p> <p>Today, the we put stuff on the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> because we want it do be discovered as part of a "sharing act". Likewise, we make regular use of Search Engine Services because we want to "Find" stuff in a productive manner.</p> <p>Putting, the above in <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id1340d970">context</a>, you don't need to be Einstein to figure out that to date the Web hasn't enabled vendors to describe their products and services clearly. Likewise, it hasn't enabled us to describe what we want, when we want it, and how much we are willing to pay etc. Basically, the SDQ of Web Content is excruciatingly low!</p> <p> The <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id1357e068">Linked Data meme</a> is about using the essence of the Web -- HTTP URIs -- as the mechanism for conducting <a href="http://dbpedia.org/resource/Data">data</a> across the Web that unambiguously unveils basic things like:</p> <ol> <li> Using a personal profile to describe exactly who I am, my interests, favorite things, what I want (wishlist), what I have to offer (offerlist) etc.</li> <li> Using an company profile to describe my entire product catalog, inventory levels, store locations, distributor and reseller networks, feature specs, price specs, deal terms and duration, and even opening and closing hours.</li> </ol> <h3>Conclusions</h3> <p>A Web of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id124f7778">Linked Data</a> enables a complete redefinition of eCommerce, and that's just for starters :-)</p> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1442" id="link-id112b62c0">Post Introducing SDQ</a> </li> <li> <a href="http://www.seangolliher.com/2009/linked-data/serendipitous-discovery-quotient-sdq-the-future-of-seo-or-an-abstract-concept/" id="link-id110cf500">Serendipitous Discovery Quotient (SDQ): The Future of SEO? Or an Abstract Concept?</a> </li> </ul>
2009-07-24T21:00:21-04:00
Exploring the Value Proposition of Linked Data
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-07-23#1565
2009-07-24T00:17:19Z
<h3>What is <a href="http://dbpedia.org/resource/Linked_Data" id="link-id138c9aa8">Linked Data</a>?</h3> <p> The primary topic of a <a href="http://dbpedia.org/resource/Meme" id="link-id12f86100">meme</a> penned by <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id115b4c98">TimBL</a> in the form of a <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id1333f300">Design Issues Doc</a> (note: this is how TimBL has shared his thoughts since the <a href="http://www.w3.org/DesignIssues/" id="link-id1128a1d0">Beginning of the Web</a>). </p> <p> There are a number of dimensions to the meme, but its primary purpose is the reintroduction of the HTTP <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id13c43cb8">URI</a> -- a vital component of the Web's core architecture. </p> <h3> What's Special about HTTP URIs?</h3> <p> They possess an intrinsic duality that combines persistent and unambiguous <a href="http://dbpedia.org/resource/Data">Data</a> Identity with platform & representation format independent Data Access. Thus, you can use a string of characters that look like a contemporary Web <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id119cd8a0">URL</a> to unambiguously achieve the following: </p> <ol> <li>Identity or Name Anything of Interest</li> <li>Describe Anything of Interest by associating the Description Subject's Identity with a constellation of Attribute and Value pairs (technically: an <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id1133e8a8">Entity</a>-Attribute-Value or Subject-Predicate-Object graph)</li> <li>Make the Description of Named Things of Interest discoverable on the Web by implicitly binding the aforementioned to Documents that hold their descriptions (technically: metadata documents or <a href="http://dbpedia.org/resource/Information" id="link-id1391da40">information</a> resources)</li> </ol> <h3>What's the basic value proposition of the <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id113bb690">Linked Data meme</a>?</h3> <p>Enabling more productive use of the Web by users and developers alike. All of which is achieved by tweaking the Web's Hyperlinking feature such that it now includes Hypertext and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1337a3f0">Hyperdata</a> as link types.</p> <p>Note: Hyperdata Linking is simply what an HTTP URI facilitates.</p> <p>Examples problems solved by injecting Linked Data into the Web:</p> <ol> <li>Federated Identity by enabling Individuals to unambiguously Identify themselves (Profiles++) courtesy of existing <a href="http://dbpedia.org/resource/Internet" id="link-id13926e28">Internet</a> and Web protocols (e.g., <a href="http://dbpedia.org/resource/Friend_of_a_friend" id="link-id13646ec8">FOAF</a>+SSL's WebIDs which combine Personal Identity with X.509 certificates and HTTPs based client side certification)</li> <li>Security and Privacy challenge alleviation by delivering a mechanism for policy based data access that feeds off federated individual identity and social network (graph) traversal</li> <li>Spam Busting via the above</li>. <li> Increasing the Serendipitous Discovery Quotient (SDQ) of Web accessible resources by embedding Rich Metadata into (X)HTML Documents e.g., structured descriptions of your "WishLists" and "OfferLists" via a common set of terms offered by vocabularies such as <a href="http://www.heppnetz.de/projects/goodrelations/" id="link-id1199b4d0">GoodRelations</a> and <a href="http://dbpedia.org/resource/SIOC" id="link-id1334cfb0">SIOC</a> </li> <li>Coherent integration of disparate data across the Web and/or within the Enterprise via "Data Meshing" rather than "Data Mashing"</li> <li>Moving beyond imprecise statistically driven "Keyword Search" (e.g. Page Rank) to "Precision Find" driven by typed link based <a href="http://dbpedia.org/resource/Entity" id="link-id135f6fe8">Entity</a> Rank plus Entity Type and Entity Property filters.</li> </ol> <h3>Conclusion</h3> <p>If all of the above still falls into the technical mumbo-jumbo realm, then simply consider Linked Data as delivering Open Data Access in granular form to Web accessible data -- that goes beyond data containers (documents or files).</p> <p>The value proposition of Linked Data is inextricably linked to the value proposition of the <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id1356f5c0">World Wide Web</a>. This is true, because the Linked Data meme is ultimately about an enhancement of the current Web; achieved by reintroducing its architectural essence -- in new <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id11300828">context</a> -- via a new level of link abstraction, courtesy of the Identity and Access duality of HTTP URIs.</p> <p>As a result of Linked Data, you can now have Links on the Web for a Person, Document, Music, Consumer Electronics, Products & Services, Business Opening & Closing Hours, Personal "WishLists" and "OfferList", an Idea, etc.. in addition to links for Properties (Attributes & Values) of the aforementioned. Ultimately, all of these links will be indexed in a myriad of ways providing the substrate for the next major period of Internet & Web driven innovation, within our larger human-ingenuity driven innovation continuum.</p> <h3>Related</h3> <ul> <li> <a href="http://www.ebusiness-unibw.org/wiki/GoodRelations#Recipes_and_Examples" id="link-id11386648">Recipes for Describing Your Business and its Offerings using the GoodRelations Vocabulary / Schema</a> </li> <li> <a href="http://slidesix.com/view/SolvingRealProblemsUsingLinkedData" id="link-id13658ee0">Solving Real Problems with RDF based Linked Data</a> </li> <li> <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=linked%20data&type=text&output=html" id="link-id1175a650">Other Linked Data Posts from this Blog oriented Linked Data Space</a> (goes back a few years!)</li> <li>Various practical <a href="http://delicious.com/kidehen/linked_data_demo" id="link-id13390cf8">Linked Data demo links from my Del.icio.us Bookmark oriented Data Space</a> </li> <li> <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id132cda80">My personal WebID</a> which is conduit to a Linked Data mesh covering vast variety of things I've opted to share with others via the Web (best viewed using a Linked Data aware User Agent like ODE).</li> </ul>
2009-07-24T08:20:01-04:00
Important Things to Note about the World Wide Web
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-07-23#1564
2009-07-23T13:27:11Z
<p>Based on the prevalence of confusion re. the <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id135eee50">Linked Data meme</a>, here are a few important points to remember about the <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id1118b210">World Wide Web</a>.</p> <ol> <li>Its an HTTP based Network Cluster within the <a href="http://dbpedia.org/resource/Internet" id="link-id1332a6e8">Internet</a> (remember: Networks are about meshes of Nodes connected by Links)</li> <li>Its underlying <a href="http://dbpedia.org/resource/Data">data</a> model is that of a Network (we've had Network Data models for eons. <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id13350310">EAV</a>/CR is an example)</li> <li>Links are facilitated via URIs</li> <li>Until recently the granularity of Networking on the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> was scoped to Data Containers (documents) (due to prevalence of <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id12f7c078">URL</a> style links</li> <li>The Linked Data meme adds Data Item (Datum) level granularity to World Wide Web networking via HTTP URIs</li> <li>Data Items become Web Reference-able when you Identify/Name them using HTTP based URIs</li> <li>An HTTP <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id135ffdb8">URI</a> implicitly binds a Web Reference-able Data Item (<a href="http://dbpedia.org/resource/Entity" id="link-id113afb60">Entity</a>, Datum, Data Object, Resource) to its Web Accessible Metadata</li> <li>Web Accessible Metadata resides within Data Containers (documents or <a href="http://dbpedia.org/resource/Information" id="link-id11402318">information</a> resources)</li> <li>The representation of a Web Accessible Metadata container is negotiable</li> <li>I am able to write and dispatch this <a href="http://dbpedia.org/resource/Blog" id="link-id0x770cbd0">blog</a> post courtesy of the Web features listed above</li> <li>You are able to explore the many dimensions to data exposed by this blog should you decide to explore the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x7acd540">Linked Data</a> mesh exposed by this post's HTTP URI (via its permalink permalink)</li> </ol> <p>The HTTP URI is the secret sauce of the Web that is powerfully and unobtrusively reintroduced via the Linked Data meme (classic back to the future act). This powerful sauce possess a unique power courtesy of its inherent duality i.e., how it uniquely combines Data Item Identity (think keys in traditional DBMS parlance) with Data Access (e.g. access to negotiable representations of associated metadata).</p> <p>As you can see, I've made no mention of RDF or <a href="http://dbpedia.org/resource/SPARQL" id="link-id1183bd48">SPARQL</a>, and I can still articulate the inherent value of the "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id132e7058">Linked Data</a>" dimension that the "Linked Data" meme adds to the World Wide Web.</p> <p>As per usual this post is a live demonstration of Linked Data (dog-food style) :-)</p> <h3>Related</h3> <ul> <li> <a href="http://www.semanticsincorporated.com/2009/07/if-linked-data-is-a-brand-it-has-big-problems-to-address.html" id="link-id1171d4e8">Greg Boutin's post about Linked Data Brand Management</a> </li> <li> <a href="http://iandavis.com/blog/2009/07/the-linked-data-brand" id="link-id12db0880">Ian Davis' "Linked Data Brand" post</a> </li> <li> <a href="http://cloudofdata.com/2009/07/does-linked-data-need-rdf/" id="link-id13537230">Paul Miller's "Does Linked Data need RDF" post</a> </li> </ul>
2009-07-23T10:33:58-04:00
Single Virtuoso host loads 110,500 triples-per-second on LUBM 8000
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-06-29#1563
2009-06-29T16:12:34Z
<p>LUBM load speed still seems to be a metric that is quoted in comparisons of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id142df6e8">RDF</a> stores. Consequently, we too measured the load time of LUBM 8000, 1,068-million triples, on the newest <a href="http://virtuoso.openlinksw.com" id="link-id1389dfa0">Virtuoso</a>.</p> <p>The real time for the load was 161m 3s. The rate was 110,532 triples-per-second. The hardware was one machine with 2 x Xeon 5410 (quad core, 2.33 GHz) and 16G 667 MHz RAM. The software was Virtuoso 6 Cluster, configured into 8 partitions (processes) — one partition per CPU core. Each partition had its database striped over 6 disks total; the 6 disks on the system were shared between the 8 database processes.</p> <p>The load was done on 8 streams, one per server process. At the beginning of the load, the CPU usage was 740% with no disk; at the end, it was around 700% with 25% disk wait. 100% counts here for one CPU core or one disk being constantly busy.</p> <p>The RDF store was configured with the default two indices over quads, these being GSPO and OGPS. Text indexing of literals was not enabled. No materialization of entailed triples was made.</p> <p>We think that LUBM loading is not a realistic benchmark for the world but since other people publish such numbers, so do we.</p>
2009-08-15T16:06:45.000001-04:00
Single Virtuoso host loads 110,500 triples-per-second on LUBM 8000
http://www.openlinksw.com:443/weblog/oerling/?date=2009-06-29#1562
2009-06-29T16:12:34Z
<p>LUBM load speed still seems to be a metric that is quoted in comparisons of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id142df6e8">RDF</a> stores. Consequently, we too measured the load time of LUBM 8000, 1,068-million triples, on the newest <a href="http://virtuoso.openlinksw.com" id="link-id1389dfa0">Virtuoso</a>.</p> <p>The real time for the load was 161m 3s. The rate was 110,532 triples-per-second. The hardware was one machine with 2 x Xeon 5410 (quad core, 2.33 GHz) and 16G 667 MHz RAM. The software was Virtuoso 6 Cluster, configured into 8 partitions (processes) — one partition per CPU core. Each partition had its database striped over 6 disks total; the 6 disks on the system were shared between the 8 database processes.</p> <p>The load was done on 8 streams, one per server process. At the beginning of the load, the CPU usage was 740% with no disk; at the end, it was around 700% with 25% disk wait. 100% counts here for one CPU core or one disk being constantly busy.</p> <p>The RDF store was configured with the default two indices over quads, these being GSPO and OGPS. Text indexing of literals was not enabled. No materialization of entailed triples was made.</p> <p>We think that LUBM loading is not a realistic benchmark for the world but since other people publish such numbers, so do we.</p>
2009-08-15T16:06:42.000001-04:00
Linked Data Rules Simplified
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-06-26#1561
2009-06-26T14:49:03Z
<p>As a compliment to the most recent <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id11a6a9b8">Linked Data Design Issues</a> note by <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id114c1ae8">TimBL</a>, I would like to add this subtle tweak to the enumerated rules:</p> <ol> <li> Identify or Name things using HTTP URIs </li> <li> Describe things using the RDF metadata model </li> <li> Increase link <a href="http://dbpedia.org/resource/Data">data</a> mesh density on the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> by linking (referring) to things in other data spaces using their HTTP URIs. </li> </ol> <p> If you perform the steps above, on any HTTP network (e.g. <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-iddbef1f0">World Wide Web</a>), you implicitly bind the Names/Identifiers of things to negotiable representations of their metadata (description) bearing documents. </p> <p> Also note, you can create and deploy the resulting RDF metadata using any of the following approaches:</p> <ol> <li> <a href="http://dbpedia.org/resource/RDFa" id="link-id14442c00">RDFa</a> within (X)HTML documents</li> <li> N3, Turtle, TriX, RDF/XML etc. based documents </li> <li>Programmatically generated variants of 1&2.</li> </ol> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1546" id="link-id1181ebf0">What is the Linked Data meme about?</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1543" id="link-id13039a98">Simple Explanation of RDF and Linked Data Dynamics</a> </li> </ul>
2009-06-26T23:18:24.000003-04:00
BBC Linked Data Meshup In 3 Steps
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-06-12#1560
2009-06-12T18:09:08Z
<h3>Situation Analysis:</h3> <p>Dr. Dre is one of the artists in the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1117a230">Linked Data</a> <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id10ff0fc0">Space</a> we host for the <a href="http://dbpedia.org/resource/BBC" id="link-id13cdba70">BBC</a>. He is also referenced in music oriented <a href="http://dbpedia.org/resource/Data">data</a> spaces such as <a href="http://dbpedia.org/resource/DBpedia" id="link-id119688a0">DBpedia</a>, <a href="http://musicbrainz.org/" id="link-id146f7d00">MusicBrainz</a> and <a href="http://last.FM" id="link-id15f50698">Last.FM</a> (to name a few). </p> <h3>Challenge:</h3> <p>How do I obtain a holistic view of the <a href="http://dbpedia.org/resource/Entity" id="link-id147a1490">entity</a> "Dr. Dre" across the BBC, MusicBrainz, and Last.FM data spaces? We know the BBC published Linked Data, but what about Last.FM and MusicBrainz? Both of these data spaces only expose XML or JSON data via REST APIs?</p> <h3>Solution:</h3> Simple 3 step Linked Data Meshup courtesy of <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtSponger" id="link-id147faf78">Virtuoso's in-built RDFizer Middleware</a> "the Sponger" (think <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id115ecea0">ODBC</a> Driver Manager for the Linked Data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id11806418">Web</a>) and its numerous Cartridges (think ODBC Drivers for the Linked Data Web). <h3>Steps:</h3> <ol> <li> Go to Last.FM and search using pattern: Dr. Dre (you will end up with this <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id11778f10">URL</a>: http://www.last.fm/music/Dr.+Dre)</li> <li> Go to the Virtuoso powered <a href="http://bbc.openlinksw.com" id="link-id14f40338">BBC Linked Data Space home page</a> and enter: http://bbc.openlinksw.com/about/html/http://www.last.fm/music/Dr.+Dre</li> <li> Go to the BBC Linked Data Space home page and type full text pattern (using default tab): Dr. Dre, then view <a href="http://bbc.openlinksw.com/fct/rdfdesc/usage.vsp?g=http%3A%2F%2Fwww.bbc.co.uk%2Fmusic%2Fartists%2F5f6ab597-f57a-40da-be9e-adad48708203%23artist&tp=4&sid=519&urilookup=&orig_refr=http://bbc.openlinksw.com/describe/?url=http://www.bbc.co.uk/music/artists/5f6ab597-f57a-40da-be9e-adad48708203" id="link-id119ac658">Dr. Dre's metadata via the Statistics Link</a>. </li> </ol> <h3>What Happened?</h3> <p>The following took place:</p> <ol> <li> Virtuoso <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id11a46fd8">Sponger</a> sent an HTTP GET to Last.FM</li> <li> Distilled the "Artist" entity "Dr. Dre" from the page, and made a <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id1297cc68">Linked Data graph</a> </li> <li> Inverse Functional Property and sameAs reasoning handled the Meshup (augmented graph from a conjunctive query processing pipeline)</li> <li>Links for "Dr. Dre" across <a href="http://bbc.openlinksw.com/describe/?url=http%3A%2F%2Fdbpedia.org%2Fresource%2FDr._Dre" id="link-id119e63e8">BBC (sameAs), Last.FM (seeAlso), via DBpedia URI</a>.</li> </ol> <p>The <a href="http://bbc.openlinksw.com/about/rdf/http/www.last.fm/music/Dr.+Dre#this" id="link-id111f6130">new enhanced URI for Dr. Dre</a> now provides a rich holistic view of the aforementioned "Artist" entity. This URI is usable anywhere on the Web for Linked Data Conduction :-)</p> <h3>Related (as in NearBy)</h3> <ul> <li> <a href="http://blogs.talis.com/n2/archives/617" id="link-idf3e0898">Augmenting Last.fm Data with BBC data on the Talis Platform</a> </li> </ul>
2009-06-12T16:38:34.000046-04:00
Understanding the BBC's Virtuoso Powered Linked Data Space
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-06-11#1559
2009-06-11T21:59:31Z
<p> The <a href="http://welcomebackstage.com/2009/06/bbc-backstage-sparql-endpoint/" id="link-id12969860">BBC's recently announced Linked Data space for Programmes and Music data</a>, joins a growing list of immediately useful "<a href="http://virtuoso.openlinksw.com" id="link-id110918f8">Virtuoso</a> Powered" linked data spaces, driving the burgeoning Web of Linked Data. Others include: <a href="http://dbpedia.org/resource/DBpedia" id="link-id12c0e720">DBpedia</a>, <a href="http://www.bio2rdf.org/" id="link-id14ee63a8">Bio2RDF</a>, NeuroCommons etc (the <a href="http://www4.wiwiss.fu-berlin.de/bizer/pub/lod-datasets_2009-03-27.html" id="link-id129a8588">click friendly version of the LOD-Cloud</a> diagram reveals a snapshot of other Virtuoso driven linked data spaces).</p> <h3>Why is it important?</h3> <p> As a leading media organization, the BBC's use of Linked Data provides a clear beacon to other media players re. the imminence of a serious Linked Data induced sector inflection. In a nutshell, every Web Site has to evolve into a Linked Data Space: a location on the Web that provides granular access to discrete data items in line with the core principles of the <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id11a14710">Linked Data meme</a>.</p> <p> Remember, the essence of the Linked Data meme is simply this: you reference data items and access their metadata, in variety of formats via a single HTTP based <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id1136b1c8">URI</a>. This approach to Web data publishing is compatible with any HTTP aware user agent (e.g., your Web Browser or tools & applications that provide abstracted access to HTTP).</p> <h3>How Do I use it?</h3> <p>There a number of very powerful things available to end-users and developers alike.</p> <h4>End-Users:</h4> <p> The most powerful feature of our variant of the BBC's Linked Data Space is the exposure of Faceted Find (think Search++ and beyond). Thus, you can go the the <a href="http://bbc.openlinksw.com" id="link-id12a32770">home page of the service</a> and commence data discovery and exploration via any of the following interfaces:</p> <ul> <li> <a href="http://dbpedia.org/resource/Full_text_search" id="link-id1179d618">Full Text Search</a> Tab -- type in a full text pattern and then experience <a href="http://bbc.openlinksw.com/facet_doc.html" id="link-id12c6dab0">Linked Data Entity Ranking as opposed to Page Ranking</a> </li> <li>URI Lookup (By Label) Tab -- type in part of a URI and let the system auto-complete by looking up Entity Labels</li> <li>URI Lookup (Raw String Pattern) Tab -- type in part of a URI and let the system auto-complete by looking up the raw URI</li> <li> <a href="http://bbc.openlinksw.com/ode/" id="link-id114b53c8">OpenLink Data Explorer Service</a> -- "deceptively simple" Linked Data explorer and Data Mesher (simply type in a URI or Text pattern, then view the data via a myriad of entity type specific viewer tabs).</li> </ul> <p>Once you are comfortable with at least one of the items above, you can exploit the system further by performing any of the following:</p> <ul> <li> <a href="http://bbc.openlinksw.com/page/void/Dataset" id="link-id117616c0">Explore the Linked Data Space via Data Dictionary</a> -- click on a Named Data Set URI and then <a href="http://bbc.openlinksw.com/describe/?url=http%3A%2F%2Fpurl.org%2Fontology%2Fpo%2FEpisode" id="link-id11664778">explore Class instances</a> (rdf:type property values) </li> <li> <a href="http://bbc.openlinksw.com/fct/rdfdesc/usage.vsp?g=http%3A%2F%2Fwww.bbc.co.uk%2Fmusic%2Fartists%2F5f6ab597-f57a-40da-be9e-adad48708203%23artist&tp=4&sid=519&urilookup=&orig_refr=http://bbc.openlinksw.com/describe/?url=http://www.bbc.co.uk/music/artists/5f6ab597-f57a-40da-be9e-adad48708203" id="link-id128a1aa8">Explore Entity Metadata</a> -- currently labeled "Statistics" but really is "Metadata" that describes data about an Entity (how you discern identifier co-reference, indirect identifiers, references from other data sets, and provenance/source graphs).</li> </ul> <h4> <a href="http://dbpedia.org/resource/Information" id="link-id1136cc60">Information</a> Architects & Developers</h4> <ul> <li>Bare bones <a href="http://dbpedia.org/resource/SPARQL" id="link-id13c15448">SPARQL</a> Endpoint -- usable by SPARQL aware user agents </li> <li> <a href="http://bbc.openlinksw.com/sparql_demo/" id="link-id114ed7f0">SPARQL Query Tool</a> -- type in SPARQL and interact with result pages that enable URI navigation (de-referencing)</li> <li> <a href="http://bbc.openlinksw.com/isparql" id="link-id12a25b38">iSPARQL Query By Example</a> -- paint your SPARQL Query and Learn <a href="http://wikis.openlinksw.com/dataspace/owiki/wiki/OATWikiWeb/InteractiveSparqlQueryBuilder" id="link-id13c0c578">SPARQL by Example</a> (just take defaults and then click "OK" to get in)</li> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtFacetBrowserInstallConfig" id="link-id15001fe0">Virtuoso Facets API</a> - REST API for Faceted Browsing & Navigation across Linked Data Set Dimensions.</li> </ul> <h3>Disambiguated Search (aka. Search++ or Find)</h3> <p> In line with the time-tested "embrace and extend" pattern, we provide Full Text search capability, but unlike Google, Yahoo!, Bing and other search engines, we don't use use "Page Rank" algorithm to sort results; instead, we use an "Entity Rank" algorithm since we are dealing with an RDF based Graph model DBMS where links exist between entities across instance data and data dictionary (vocabularies, schemas, ontologies) boundaries. In addition, when you get results (by clicking "show values" or "show values with distinct counts") that list entities associated with a full text search pattern, we take a quantum leap beyond search engines by allowing you to use "Entity Type" and/or "Entity Properties" (all of these have HTTP URIs too) to set your own <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id13c15c30">context</a> for what you seek.</p> <p>Much more to come in the form of <a href="http://bbc.openlinksw.com/fct/facet.vsp?cmd=featured&sid=423&no_qry=1" id="link-id128a0fd0">BBC specific demo queries</a> and tutorials :-)</p> <h3>Related</h3> <ul> <li> Live <a href="http://lod.openlinksw.com" id="link-id14eb3010">LOD Cloud Cache</a> instance that combines BBC data with other data sets from the LOD Cloud (in a single Virtuoso RDF DBMS hosting 5 Billion+ triples & counting) </li> </ul>
2009-06-26T23:15:13.000001-04:00
Comparing Virtuoso Performance on Different Processors
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-05-28#1558
2009-05-28T14:54:59Z
<p>Over the years we have run <a href="http://virtuoso.openlinksw.com" id="link-id0x16735e20">Virtuoso</a> on different hardware. We will here give a few figures that help identify the best price point for machines running Virtuoso.</p> <p>Our test is very simple: <i>Load 20 warehouses of <a href="http://dbpedia.org/resource/TPC-C" id="link-id0x16e0dba8">TPC-C</a> <a href="http://dbpedia.org/resource/Data" id="link-id0x14ff4f80">data</a>, and then run one client per warehouse for 10,000 new orders</i>. The way this is set up, disk I/O does not play a role and lock contention between the clients is minimal.</p> <p>The test essentially has 20 server and 20 client threads running the same workload in parallel. The load time gives the single thread number; the 20 clients run gives the multi-threaded number. The test uses about 2-3 GB of data, so all is in RAM but is large enough not to be all in processor cache.</p> <p>All times reported are real times, starting from the start of the first client and ending with the completion of the last client.</p> <p>Do not confuse these results with official TPC-C. The measurement protocols are entirely incomparable.</p> <style type="text/css"> TABLE { background: none; border: none } TH { text-align: center; font-weight: bold } TR.top { background: } TD { text-align: center; border: none } </style> <table align="center" cellspacing="10"> <tr> <th>Test</th> <th>Platform</th> <th>Load<br />(seconds)</th> <th>Run<br />(seconds)</th> <th>GHz / cores / threads</th> </tr> <tr> <td>1</td> <td>Amazon <a href="http://aws.amazon.com/ec2/" id="link-id0x15d68e20">EC2</a> Extra Large<br />(4 virtual cores)</td> <td>340</td> <td>42</td> <td>1.2 GHz? / 4 / 1</td> </tr> <tr> <td>1</td> <td>Amazon EC2 Extra Large<br />(4 virtual cores)</td> <td>305</td> <td>43.3</td> <td>1.2 GHz? / 4 / 1</td> </tr> <tr> <td>2</td> <td>1 x dual-core AMD 5900</td> <td>263</td> <td>58.2</td> <td>2.9 GHz / 2 / 1</td> </tr> <tr> <td>3</td> <td>2 x dual-core Xeon 5130 ("Woodcrest")</td> <td>245</td> <td>35.7</td> <td>2.0 GHz / 4 / 1</td> </tr> <tr> <td>4</td> <td>2 x quad-core Xeon 5410 ("Harpertown")</td> <td>237</td> <td>18.0</td> <td>2.33 GHz / 8 / 1</td> </tr> <tr> <td>5</td> <td>2 x quad-core Xeon 5520 ("Nehalem")</td> <td>162</td> <td>18.3</td> <td>2.26 GHz / 8 / 2</td> </tr> </table> <p>We tried two different EC2 instances to see if there would be variation. The variation was quite small. The tested EC2 instances costs 20 US cents per hour. The AMD dual-core costs 550 US dollars with 8G. The 3 Xeon configurations are Supermicro boards with 667MHz memory for the Xeon 5130 ("Woodcrest") and Xeon 5410 ("Harpertown"), and 800MHz memory for the Nehalem. The Xeon systems cost between 4000 and 7000 US dollars, with 5000 for a configuration with 2 x Xeon 5520 ("Nehalem"), 72 GB RAM, and 8 x 500 GB SATA disks.</p> <p> <i>Caveat: Due to slow memory (we could not get faster within available time), the results for the Nehalem do not take full advantage of its principal edge over the previous generation, i.e., memory subsystem. We'll see another time with faster memories.</i> </p> <p>The operating systems were various 64 bit Linux distributions.</p> <p>We did some further measurements comparing Harpertown and Nehalem processors. The Nehalem chip was a bit faster for a slightly lower clock but we did not see any of the twofold and greater differences advertised by Intel.</p> <p>We tried some <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1460b688">RDF</a> operations on the two last systems:</p> <table align="center" cellspacing="10"> <tr> <th>operation</th> <th> Harpertown</th> <th>Nehalem</th> </tr> <tr> <th>Build text index for <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x16a94590">DBpedia</a></th> <td>1080s</td> <td>770s</td> </tr> <tr> <th><a href="http://dbpedia.org/resource/Entity" id="link-id0xc37f380">Entity</a> Rank iteration</th> <td>263s</td> <td>251s</td> </tr> </table> <p>Then we tried to see if the core multithreading of Nehalem could be seen anywhere. To this effect, we ran the Fibonacci function in <a href="http://dbpedia.org/resource/SQL" id="link-id0x15842a20">SQL</a> to serve as an example of an all in-cache integer operation. 16 concurrent operations took exactly twice as long as 8 concurrent ones, as expected.</p> <p>For something that used memory, we took a count of RDF quads on two different indices, getting the same count. The database was a cluster setup with one process per core, so a count involved one thread per core. The counts in series took 5.02s and in parallel they took 4.27s.</p> <p>Then we took a more memory intensive piece that read the RDF quads table in the order of one index and for each row checked that there is the equal row on another, differently-partitioned index. This is a cross-partition join. One of the indices is read sequentially and the other at random. The throughput can be reported as random-lookups-per-second. The data was English DBpedia, about 140M triples. One such query takes a couple of minutes with a 650% CPU utilization. Running multiple such queries should show effects of core multithreading since we expect frequent cache misses.</p> <ol> <li>On the host OS of the Nehalem system — <table align="center" cellspacing="10"> <tr> <th>n</th> <th>cpu%</th> <th>rows per second</th> </tr> <tr> <th>1 query</th> <td>503</td> <td>906,413</td> </tr> <tr> <th>2 queries</th> <td>1263</td> <td>1,578,585</td> </tr> <tr> <th>3 queries</th> <td>1204</td> <td>1,566,849</td> </tr> </table> </li> <li>In a VM under Xen, on the Nehalem system — <table align="center" cellspacing="10"> <tr> <th>n</th> <th>cpu%</th> <th>rows per second</th> </tr> <tr> <th>1 query</th> <td>652</td> <td>799,293</td> </tr> <tr> <th>2 queries</th> <td>1266</td> <td>1,486,710</td> </tr> <tr> <th>3 queries</th> <td>1222</td> <td>1,484,093</td> </tr> </table> </li> <li> On the host OS of the Harpertown system — <table align="center" cellspacing="10"> <tr> <th>n</th> <th>cpu%</th> <th>rows per second</th> </tr> <tr> <th>1 query</th> <td> 648 </td> <td> 1,041,448 </td> </tr> <tr> <th>2 queries</th> <td> 708 </td> <td> 1,124,866 </td> </tr> </table> </li> </ol> <p>The CPU percentages are as reported by the OS: user + system CPU divided by real time.</p> <p>So, Nehalem is in general somewhat faster, around 20-30%, than Harpertown. The effect of core multithreading can be noticed but is not huge, another 20% or so for situations with more threads than cores. The join where Harpertown did better could be attributed to its larger cache — 12 MB vs 8 MB.</p> <p>We see that Xen has a measurable but not prohibitive overhead; count a little under 10% for everything, also tasks with no I/O. The VM was set up to have all CPU for the test and the queries did not do disk I/O.</p> <p>The executables were compiled with <code>gcc</code> with default settings. Specifying <code>-march=nocona</code> (Core 2 target) dropped the cross-partition join time mentioned above from 128s to 122s on Harpertown. We did not try this on Nehalem but presume the effect would be the same, since the out-of-order unit is not much different. We did not do anything about process-to-memory affinity on Nehalem, which is a non-uniform architecture. We would expect this to increase performance since we have many equal size processes with even load.</p> <p>The mainstay of the Nehalem value proposition is a better memory subsystem. Since the unit we got was at 800 MHz memory, we did not see any great improvement. So if you buy Nehalem, you should make sure it is with 1333 MHz memory, else the best case will not be over 50% over a 667 MHz Core 2-based Xeon.</p> <p>Nehalem remains a better deal for us because of more memory per board. One Nehalem box with 72 GB costs less than two Harpertown boxes with 32 GB and offers almost the same performance. Having a lot of memory in a small space is key. With faster memory, it might even outperform two Harpertown boxes, but this remains to be seen.</p> <p>If space were not a constraint, we could make a cluster of 12 small workstations for the price of our largest system and get still more memory and more processor power per unit of memory. The Nehalem box was almost 4x faster than the AMD box but then it has 9x the memory, so the CPU to memory ratio might be better with the smaller boxes.</p>
2009-05-28T11:15:41.000006-04:00
Comparing Virtuoso Performance on Different Processors
http://www.openlinksw.com:443/weblog/oerling/?date=2009-05-28#1557
2009-05-28T14:54:59Z
<p>Over the years we have run <a href="http://virtuoso.openlinksw.com" id="link-id0xd420b90">Virtuoso</a> on different hardware. We will here give a few figures that help identify the best price point for machines running Virtuoso.</p> <p>Our test is very simple: <i>Load 20 warehouses of <a href="http://dbpedia.org/resource/TPC-C" id="link-id0xdaaec90">TPC-C</a> <a href="http://dbpedia.org/resource/Data" id="link-id0xca1b7e0">data</a>, and then run one client per warehouse for 10,000 new orders</i>. The way this is set up, disk I/O does not play a role and lock contention between the clients is minimal.</p> <p>The test essentially has 20 server and 20 client threads running the same workload in parallel. The load time gives the single thread number; the 20 clients run gives the multi-threaded number. The test uses about 2-3 GB of data, so all is in RAM but is large enough not to be all in processor cache.</p> <p>All times reported are real times, starting from the start of the first client and ending with the completion of the last client.</p> <p>Do not confuse these results with official TPC-C. The measurement protocols are entirely incomparable.</p> <style type="text/css"> TABLE { background: none; border: none } TH { text-align: center; font-weight: bold } TR.top { background: } TD { text-align: center; border: none } </style> <table align="center" cellspacing="10"> <tr> <th>Test</th> <th>Platform</th> <th>Load<br />(seconds)</th> <th>Run<br />(seconds)</th> <th>GHz / cores / threads</th> </tr> <tr> <td>1</td> <td>Amazon <a href="http://aws.amazon.com/ec2/" id="link-id0xdaab030">EC2</a> Extra Large<br />(4 virtual cores)</td> <td>340</td> <td>42</td> <td>1.2 GHz? / 4 / 1</td> </tr> <tr> <td>1</td> <td>Amazon EC2 Extra Large<br />(4 virtual cores)</td> <td>305</td> <td>43.3</td> <td>1.2 GHz? / 4 / 1</td> </tr> <tr> <td>2</td> <td>1 x dual-core AMD 5900</td> <td>263</td> <td>58.2</td> <td>2.9 GHz / 2 / 1</td> </tr> <tr> <td>3</td> <td>2 x dual-core Xeon 5130 ("Woodcrest")</td> <td>245</td> <td>35.7</td> <td>2.0 GHz / 4 / 1</td> </tr> <tr> <td>4</td> <td>2 x quad-core Xeon 5410 ("Harpertown")</td> <td>237</td> <td>18.0</td> <td>2.33 GHz / 8 / 1</td> </tr> <tr> <td>5</td> <td>2 x quad-core Xeon 5520 ("Nehalem")</td> <td>162</td> <td>18.3</td> <td>2.26 GHz / 8 / 2</td> </tr> </table> <p>We tried two different EC2 instances to see if there would be variation. The variation was quite small. The tested EC2 instances costs 20 US cents per hour. The AMD dual-core costs 550 US dollars with 8G. The 3 Xeon configurations are Supermicro boards with 667MHz memory for the Xeon 5130 ("Woodcrest") and Xeon 5410 ("Harpertown"), and 800MHz memory for the Nehalem. The Xeon systems cost between 4000 and 7000 US dollars, with 5000 for a configuration with 2 x Xeon 5520 ("Nehalem"), 72 GB RAM, and 8 x 500 GB SATA disks.</p> <p> <i>Caveat: Due to slow memory (we could not get faster within available time), the results for the Nehalem do not take full advantage of its principal edge over the previous generation, i.e., memory subsystem. We'll see another time with faster memories.</i> </p> <p>The operating systems were various 64 bit Linux distributions.</p> <p>We did some further measurements comparing Harpertown and Nehalem processors. The Nehalem chip was a bit faster for a slightly lower clock but we did not see any of the twofold and greater differences advertised by Intel.</p> <p>We tried some <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xce85438">RDF</a> operations on the two last systems:</p> <table align="center" cellspacing="10"> <tr> <th>operation</th> <th> Harpertown</th> <th>Nehalem</th> </tr> <tr> <th>Build text index for <a href="http://dbpedia.org/resource/DBpedia" id="link-id0xab826a8">DBpedia</a></th> <td>1080s</td> <td>770s</td> </tr> <tr> <th><a href="http://dbpedia.org/resource/Entity" id="link-id0xcbb9938">Entity</a> Rank iteration</th> <td>263s</td> <td>251s</td> </tr> </table> <p>Then we tried to see if the core multithreading of Nehalem could be seen anywhere. To this effect, we ran the Fibonacci function in <a href="http://dbpedia.org/resource/SQL" id="link-id0xcd62218">SQL</a> to serve as an example of an all in-cache integer operation. 16 concurrent operations took exactly twice as long as 8 concurrent ones, as expected.</p> <p>For something that used memory, we took a count of RDF quads on two different indices, getting the same count. The database was a cluster setup with one process per core, so a count involved one thread per core. The counts in series took 5.02s and in parallel they took 4.27s.</p> <p>Then we took a more memory intensive piece that read the RDF quads table in the order of one index and for each row checked that there is the equal row on another, differently-partitioned index. This is a cross-partition join. One of the indices is read sequentially and the other at random. The throughput can be reported as random-lookups-per-second. The data was English DBpedia, about 140M triples. One such query takes a couple of minutes with a 650% CPU utilization. Running multiple such queries should show effects of core multithreading since we expect frequent cache misses.</p> <ol> <li>On the host OS of the Nehalem system — <table align="center" cellspacing="10"> <tr> <th>n</th> <th>cpu%</th> <th>rows per second</th> </tr> <tr> <th>1 query</th> <td>503</td> <td>906,413</td> </tr> <tr> <th>2 queries</th> <td>1263</td> <td>1,578,585</td> </tr> <tr> <th>3 queries</th> <td>1204</td> <td>1,566,849</td> </tr> </table> </li> <li>In a VM under Xen, on the Nehalem system — <table align="center" cellspacing="10"> <tr> <th>n</th> <th>cpu%</th> <th>rows per second</th> </tr> <tr> <th>1 query</th> <td>652</td> <td>799,293</td> </tr> <tr> <th>2 queries</th> <td>1266</td> <td>1,486,710</td> </tr> <tr> <th>3 queries</th> <td>1222</td> <td>1,484,093</td> </tr> </table> </li> <li> On the host OS of the Harpertown system — <table align="center" cellspacing="10"> <tr> <th>n</th> <th>cpu%</th> <th>rows per second</th> </tr> <tr> <th>1 query</th> <td> 648 </td> <td> 1,041,448 </td> </tr> <tr> <th>2 queries</th> <td> 708 </td> <td> 1,124,866 </td> </tr> </table> </li> </ol> <p>The CPU percentages are as reported by the OS: user + system CPU divided by real time.</p> <p>So, Nehalem is in general somewhat faster, around 20-30%, than Harpertown. The effect of core multithreading can be noticed but is not huge, another 20% or so for situations with more threads than cores. The join where Harpertown did better could be attributed to its larger cache — 12 MB vs 8 MB.</p> <p>We see that Xen has a measurable but not prohibitive overhead; count a little under 10% for everything, also tasks with no I/O. The VM was set up to have all CPU for the test and the queries did not do disk I/O.</p> <p>The executables were compiled with <code>gcc</code> with default settings. Specifying <code>-march=nocona</code> (Core 2 target) dropped the cross-partition join time mentioned above from 128s to 122s on Harpertown. We did not try this on Nehalem but presume the effect would be the same, since the out-of-order unit is not much different. We did not do anything about process-to-memory affinity on Nehalem, which is a non-uniform architecture. We would expect this to increase performance since we have many equal size processes with even load.</p> <p>The mainstay of the Nehalem value proposition is a better memory subsystem. Since the unit we got was at 800 MHz memory, we did not see any great improvement. So if you buy Nehalem, you should make sure it is with 1333 MHz memory, else the best case will not be over 50% over a 667 MHz Core 2-based Xeon.</p> <p>Nehalem remains a better deal for us because of more memory per board. One Nehalem box with 72 GB costs less than two Harpertown boxes with 32 GB and offers almost the same performance. Having a lot of memory in a small space is key. With faster memory, it might even outperform two Harpertown boxes, but this remains to be seen.</p> <p>If space were not a constraint, we could make a cluster of 12 small workstations for the price of our largest system and get still more memory and more processor power per unit of memory. The Nehalem box was almost 4x faster than the AMD box but then it has 9x the memory, so the CPU to memory ratio might be better with the smaller boxes.</p>
2015-08-20T17:28:06.354659-04:00
Library of Congress & Reasonable Linked Data
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-05-05#1556
2009-05-05T17:53:24Z
<p> While exploring the <a href="http://id.loc.gov/authorities/" id="link-id1488cca8">Subject Headings Linked Data Space</a> (LCSH) recently unveiled by the <a href="http://id.loc.gov/" id="link-id1672ad10">Library of Congress</a>, I noticed that the <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id158fef78">URI</a> for the subject heading: <a href="http://id.loc.gov/authorities/sh95000541#concept" id="link-id14c8d3e8">World Wide Web</a>, exposes an "owl:sameAs" link to resource URI: "info:lc/authorities/sh95000541" -- in fact, a URI.URN that isn't HTTP protocol scheme based.</p> <p> The observations above triggered a <a href="http://search.twitter.com/search?q=&ands=&phrase=&ors=&nots=&tag=linkeddata&lang=all&from=kidehen&to=edsu&ref=&near=&within=15&units=mi&since=2009-05-01&until=2009-05-05&rpp=10" id="link-id14e21ba0">discussion thread on Twitter</a> that involved: <a href="http://twitter.com/edsu" id="link-ide411808">@edsu</a>, <a href="http://twitter.com/iand" id="link-id11915ed0">@iand</a>, and <a href="http://twitter.com/kidehen" id="link-id1519c028">moi</a>. Naturally, it morphed into a live demonstration of: human vs machine, interpretation of claims expressed in the RDF graph.</p> <h3>What makes this whole thing interesting?</h3> <p>It showcases (in Man vs Machine style) the issue of unambiguously discerning the meaning of the owl:sameAs claim expressed in the LCSH <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id17004728">Linked Data Space</a>.</p> <h4>Perspectives & Potential Confusion</h4> <p> From the Linked Data perspective, it may spook a few people to see owl:sameAs values such as: "info:lc/authorities/sh95000541", that cannot be de-referenced using HTTP. </p> <p> It may confuse a few people or user agents that see URI de-referencing as not necessarily HTTP specific, thereby attempting to de-reference the URI.URN on the assumption that it's associated with a "<a href="http://www.handle.net/overviews/overview.html" id="link-id155517a8">handle system</a>", for instance.</p> <p> It may even confuse RDFizer / RDFization middleware that use owl:sameAs as a <a href="http://dbpedia.org/resource/Data">data</a> provider attribution mechanism via hint/nudge URI values derived from original content / data URI.URLs that de-reference to nothing e.g., an original resource URI.<a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id119e0d80">URL</a> plus "#this" which produces URI.URN-URL -- think of this pattern as "owl:shameAs" in a sense :-)</p> <h3> Unambiguously Discerning Meaning</h3> <p> Simply bring OWL reasoning (inference rules and reasoners) into the mix, thereby negating human dialogue about interpretation which ultimately unveils a mesh of orthogonal view points. Remember, OWL is all about infrastructure that ultimately enables you to express yourself clearly i.e., say what you mean, and mean what you say. </p> <h3>Path to Clarity (using <a href="http://virtuoso.openlinksw.com" id="link-id1537aa68">Virtuoso</a>, its in-built Sponger Middleware, and Inference Engine):</h3> <ol> <li>GET the data into the Virtuoso Quad store -- what the sponger does via its <a href="http://linkeddata.uriburner.com/about/html/http://id.loc.gov/authorities/sh95000541#concept" id="link-id1669fa40">URIBurner Service</a> (while following designated predicates such as owl:sameAs in case they point to other mesh-able data sources)</li> <li>Query the data in Quad Store with "owl:sameAs" inference rules enabled</li> <li>Repeat the last step with the inference rules excluded.</li> </ol> <h4>Actual <a href="http://dbpedia.org/resource/SPARQL" id="link-id17374110">SPARQL</a> Queries:</h4> <ul> <li> <a href="http://demo.openlinksw.com/DAV/home/demo/Public/Queries/DataWeb/lcsh_www_subject_heading.isparql" id="link-id16c986d0">SPARQL Query against the HTTP based Subject Heading URI for WWW</a> </li> <li> <a href="http://demo.openlinksw.com/DAV/home/demo/Public/Queries/DataWeb/lcsh_www_subject_heading_sameAs_inference_on.isparql" id="link-id16d4fea0">SPARQL Query (with reasoning via inference rule for owl:sameAs)</a> against the URN based Subject Heading URI for WWW</li> <li> <a href="http://demo.openlinksw.com/DAV/home/demo/Public/Queries/DataWeb/lcsh_www_subject_heading_no_sameAs_inference_on.isparql" id="link-id11bad768">SPARQL Query (*without* reasoning via inference rule for owl:sameAs)</a> against the URN based Subject Heading URI for WWW</li> </ul> <h4>Observations:</h4> <p> The <a href="http://dbpedia.org/resource/SPARQL" id="link-ide6acf68">SPARQL</a> queries against the Graph generated and automatically populated by the Sponger reveal -- without human intervention-- that: "info:lc/authorities/sh95000541", is just an alternative name for < xmlns="http" id.loc.gov="id.loc.gov" authorities="authorities" sh95000541="sh95000541" concept="concept">, and that the graph produced by LCSH is self-describing enough for an OWL reasoner to figure this all out courtesy of the <a href="http://linkeddata.uriburner.com/about/html/http://www.w3.org/2002/07/owl%23sameAs" id="link-id13e364b0">owl:sameAs</a> property :-).</p> <p>Hopefully, this post also provides a simple example of how <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id158a3fe8">OWL</a> facilitates "Reasonable Linked Data". </p> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1455" id="link-id164e19f8">State of the Linked Data Web</a> </li> <li> <a href="http://www.mkbergman.com/?p=474" id="link-id11973d10">Making Linked Data Reasonable Using Description Logics Series</a> - post by <a href="http://community.linkeddata.org/dataspace/person/mkbergman#this" id="link-id1184bfb8">Mike Bergman</a> </li> </ul>
2009-05-06T14:26:15.000034-04:00
Social Web Camp (#5 of 5)
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-04-30#1555
2009-04-30T16:14:02Z
<p>(Last of five posts related to the <a href="http://www2009.org/" id="link-id0x112efd58">WWW 2009</a> conference, held the week of April 20, 2009.) </p> <p>The social networks camp was interesting, with a special meeting around Twitter. Half jokingly, we (that is, the OpenLink folks attending) concluded that societies would never be completely classless, although mobility between, as well as criteria for membership in, given classes would vary with time and circumstance. Now, there would be a new class division between people for whom micro-blogging is obligatory and those for whom it is an option.</p> <p>By my experience, a great deal is possible in a short time, but this possibility depends on focus and concentration. These are increasingly rare. I am a great believer in core competence and focus. This is not only for geeks — one can have a lot of breadth-of-scope but this too depends on not getting sidetracked by constant <a href="http://dbpedia.org/resource/Information" id="link-id0x14e380b8">information</a> overload.</p> <p>Insofar as personal success depends on constant reaction to online social media, this comes at a cost in time and focus and this cost will have to be managed somehow, for example by automation or outsourcing. But if the social media is only automated fronts twitting and re-twitting among themselves, a bit like electronic trading systems do with securities, with or without human operators, the value of the medium decreases.</p> <p>There are contradictory requirements. On one hand, what is said in electronic media is essentially permanent, so one had best only say things that are well considered. On the other hand, one must say these things without adequate time for reflection or analysis. To cope with this, one must have a well-rehearsed position that is compacted so that it fits in a short format and is easy to remember and unambiguous to express. A culture of pre-cooked fast-food advertising cuts down on depth. Real-world things are complex and multifaceted. Besides, prevalent patterns of communication train the brain for a certain mode of functioning. If we train for rapid-fire 140-character messaging, we optimize one side but probably at the expense of another. In the meantime, the world continues developing increased complexity by all kinds of emergent effects. Connectivity is good but don't get lost in it.</p> <p>There is <a href="https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis/index.html" id="link-id170cb010">a CIA memorandum about how analysts misinterpret data and see what they want to see</a>. This is a relevant resource for understanding some psychology of perception and memory. With the information overload, largely driven by user generated content, interpreting fragmented and variously-biased real-time information is not only for the analyst but for everyone who needs to intelligently function in cyber-social space.</p> <p>I participated in discussions on security and privacy and on mobile social networks and context.</p> <p>For privacy, the main thing turned out to be whether people should be protected from themselves. Should information expire? Will it get buried by itself under huge volumes of new content? Well, for purposes of visibility, it will certainly get buried and will require constant management to stay visible. But for purposes of future finding of dirt, it will stay findable for those who are looking.</p> <p>There is also the corollary of setting security for resources, like documents, versus setting security for statements, i.e., structured data like social networks. As I have blogged before, policies <a id="link-id14aaff90">à la</a> <a href="http://dbpedia.org/resource/SQL" id="link-id0x13d77830">SQL</a> do not work well when schema is fluid and end-users can't be expected to formulate or understand these. Remember <a href="http://dbpedia.org/resource/Ted_Nelson" id="link-id0x156ceae0">Ted Nelson</a>? A user interface should be such that a beginner understands it in 10 seconds in an emergency. The user interaction question is how to present things so that the user understands who will have access to what content. Also, users should themselves be able to check what potentially sensitive information can be found out about them. A service along the lines of Garlic's Data Patrol should be a part of the social web infrastructure of the future.</p> <p>People at MIT have developed AIR (Accountability In <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x14e2abc0">RDF</a>) for expressing policies about what can be done with data and for explaining why access is denied if it is denied. However, if we at all look at the history of secrets, it is rather seldom that one hears that access to information about X is restricted to compartment so-and-so; it is much more common to hear that there is no X. I would say that a policy system that just leaves out information that is not supposed to be available will please the users more. This is not only so for organizations; it is fully plausible that an individual might not wish to expose even the existence of some selected inner circle of friends, their parties together, or whatever.</p> <p>In conclusion, there is no self-evident solution for careless use of social media. A site that requires people to confirm multiple times that they know what they are doing when publishing a photo will not get much use. We will see.</p> <p>For mobility, there was some talk about the context of usage. Again, this is difficult. For different contexts, one would for example disclose one's location at the granularity of the city; for some other purposes, one would say which conference room one is in.</p> <p>Embarrassing social situations may arise if mobile devices are too clever: If information about travel is pushed into the social network, one would feel like having to explain why one does not call on such-and-such a person and so on. Too much initiative in the mobile phone seems like a recipe for problems.</p> <p>There is a thin line between convenience and having IT infrastructure rule one's life. The complexities and subtleties of social situations ought not to be reduced to the level of if-then rules. People and their interactions are more complex than they themselves often realize. A system is not its own metasystem, as Gödel put it. Similarly, human self-<a href="http://dbpedia.org/resource/Knowledge" id="link-id0x70d82ff8">knowledge</a>, let alone knowledge about another, is by this very principle only approximate. Not to forget what psychology tells us about state-dependent recall and of how circumstance can evoke patterns of behavior before one even notices. The history of expert systems did show that people do not do very well at putting their skills in the form of if-then rules. Thus automating sociality past a certain point seems a problematic proposition.</p>
2009-04-30T12:51:54-04:00
Social Web Camp (#5 of 5)
http://www.openlinksw.com:443/weblog/oerling/?date=2009-04-30#1554
2009-04-30T16:14:02Z
<p>(Last of five posts related to the <a href="http://www2009.org/" id="link-id0xd28c860">WWW 2009</a> conference, held the week of April 20, 2009.) </p> <p>The social networks camp was interesting, with a special meeting around Twitter. Half jokingly, we (that is, the OpenLink folks attending) concluded that societies would never be completely classless, although mobility between, as well as criteria for membership in, given classes would vary with time and circumstance. Now, there would be a new class division between people for whom micro-blogging is obligatory and those for whom it is an option.</p> <p>By my experience, a great deal is possible in a short time, but this possibility depends on focus and concentration. These are increasingly rare. I am a great believer in core competence and focus. This is not only for geeks — one can have a lot of breadth-of-scope but this too depends on not getting sidetracked by constant <a href="http://dbpedia.org/resource/Information" id="link-id0x10019a70">information</a> overload.</p> <p>Insofar as personal success depends on constant reaction to online social media, this comes at a cost in time and focus and this cost will have to be managed somehow, for example by automation or outsourcing. But if the social media is only automated fronts twitting and re-twitting among themselves, a bit like electronic trading systems do with securities, with or without human operators, the value of the medium decreases.</p> <p>There are contradictory requirements. On one hand, what is said in electronic media is essentially permanent, so one had best only say things that are well considered. On the other hand, one must say these things without adequate time for reflection or analysis. To cope with this, one must have a well-rehearsed position that is compacted so that it fits in a short format and is easy to remember and unambiguous to express. A culture of pre-cooked fast-food advertising cuts down on depth. Real-world things are complex and multifaceted. Besides, prevalent patterns of communication train the brain for a certain mode of functioning. If we train for rapid-fire 140-character messaging, we optimize one side but probably at the expense of another. In the meantime, the world continues developing increased complexity by all kinds of emergent effects. Connectivity is good but don't get lost in it.</p> <p>There is <a href="https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis/index.html" id="link-id170cb010">a CIA memorandum about how analysts misinterpret data and see what they want to see</a>. This is a relevant resource for understanding some psychology of perception and memory. With the information overload, largely driven by user generated content, interpreting fragmented and variously-biased real-time information is not only for the analyst but for everyone who needs to intelligently function in cyber-social space.</p> <p>I participated in discussions on security and privacy and on mobile social networks and context.</p> <p>For privacy, the main thing turned out to be whether people should be protected from themselves. Should information expire? Will it get buried by itself under huge volumes of new content? Well, for purposes of visibility, it will certainly get buried and will require constant management to stay visible. But for purposes of future finding of dirt, it will stay findable for those who are looking.</p> <p>There is also the corollary of setting security for resources, like documents, versus setting security for statements, i.e., structured data like social networks. As I have blogged before, policies <a id="link-id14aaff90">à la</a> <a href="http://dbpedia.org/resource/SQL" id="link-id0x10b058d0">SQL</a> do not work well when schema is fluid and end-users can't be expected to formulate or understand these. Remember <a href="http://dbpedia.org/resource/Ted_Nelson" id="link-id0x145b3070">Ted Nelson</a>? A user interface should be such that a beginner understands it in 10 seconds in an emergency. The user interaction question is how to present things so that the user understands who will have access to what content. Also, users should themselves be able to check what potentially sensitive information can be found out about them. A service along the lines of Garlic's Data Patrol should be a part of the social web infrastructure of the future.</p> <p>People at MIT have developed AIR (Accountability In <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x10dec8f8">RDF</a>) for expressing policies about what can be done with data and for explaining why access is denied if it is denied. However, if we at all look at the history of secrets, it is rather seldom that one hears that access to information about X is restricted to compartment so-and-so; it is much more common to hear that there is no X. I would say that a policy system that just leaves out information that is not supposed to be available will please the users more. This is not only so for organizations; it is fully plausible that an individual might not wish to expose even the existence of some selected inner circle of friends, their parties together, or whatever.</p> <p>In conclusion, there is no self-evident solution for careless use of social media. A site that requires people to confirm multiple times that they know what they are doing when publishing a photo will not get much use. We will see.</p> <p>For mobility, there was some talk about the context of usage. Again, this is difficult. For different contexts, one would for example disclose one's location at the granularity of the city; for some other purposes, one would say which conference room one is in.</p> <p>Embarrassing social situations may arise if mobile devices are too clever: If information about travel is pushed into the social network, one would feel like having to explain why one does not call on such-and-such a person and so on. Too much initiative in the mobile phone seems like a recipe for problems.</p> <p>There is a thin line between convenience and having IT infrastructure rule one's life. The complexities and subtleties of social situations ought not to be reduced to the level of if-then rules. People and their interactions are more complex than they themselves often realize. A system is not its own metasystem, as Gödel put it. Similarly, human self-<a href="http://dbpedia.org/resource/Knowledge" id="link-id0xd7b1808">knowledge</a>, let alone knowledge about another, is by this very principle only approximate. Not to forget what psychology tells us about state-dependent recall and of how circumstance can evoke patterns of behavior before one even notices. The history of expert systems did show that people do not do very well at putting their skills in the form of if-then rules. Thus automating sociality past a certain point seems a problematic proposition.</p>
2009-04-30T12:51:49-04:00
Web Science and Keynotes at WWW 2009 (#4 of 5)
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-04-30#1553
2009-04-30T16:00:22Z
<p>(Fourth of five posts related to the <a href="http://www2009.org/" id="link-id0xdcaa1c0">WWW 2009</a> conference, held the week of April 20, 2009.) </p> <p>There was quite a bit of talk about what web science could or ought to be. I will here comment a bit on the <a href="http://www2009.org/panels.html" id="link-id1514ec30">panels</a> and <a href="http://www2009.org/keynote_abs.html" id="link-id11a5d620">keynotes</a>, in no special order. </p> <p>In the web science panel, Tim Berners-Lee said that the deliverable of the web science initiative could be a way of making sense of all the world's <a href="http://dbpedia.org/resource/Data" id="link-id0x121145b0">data</a> once the web had transformed into a database capable of answering arbitrary queries.</p> <p>Michael Brodie of Verizon said that one deliverable would be a well considered understanding of the issue of counter-terrorism and civil liberties: Everything, including terrorism, operates on the platform of the web. How do we understand an issue that is not one of privacy, intelligence, jurisprudence, or sociology, but of all these and more?</p> <p>I would add to this that it is not only a matter of governments keeping and analyzing vast amounts of private data, but of basically anybody who wants to do this being able to do so, even if at a smaller scale. In a way, the data web brings formerly government-only capabilities to the public, and is thus a democratization of intelligence and analytics. The citizen blogger increased the accountability of the press; the citizen analyst may have a similar effect. This is trickier though. We remember Jefferson's words about vigilance and the price of freedom. But vigilance is harder today, not because <a href="http://dbpedia.org/resource/Information" id="link-id0x10402da0">information</a> is not there but because there is so much of it, with diverse spins put on it.</p> <p>Tim B-L said at another panel that it seemed as if the new capabilities, especially the web as a database, were coming just in time to help us cope with the problems confronting the planet. With this, plus having everybody online, we would have more information, more creativity, more of everything at our disposal.</p> <p>I'd have to say that the web is dual use: The bulk of traffic may contribute to distraction more than to awareness, but then the same infrastructure and the social behaviors it supports may also create unprecedented value and in the best of cases also transparency. I have to think of "For whosoever hath, to him shall be given." [Matthew 13:12] This can mean many things; here I am talking about whoever hath a drive for <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x1201dff0">knowledge</a>.</p> <p>The web is both equalizing and polarizing: The equality is in the access; the polarity in the use made thereof. For a huge amount of noise there will be some crystallization of value that could not have arisen otherwise. Developments have unexpected effects. I would not have anticipated that gaming should advance supercomputing, for example.</p> <p>Wendy Hall gave a dinner speech about communities and conferences; how the original hypertext conferences, with lots of representation of the humanities, became the techie WWW conference series; and how now we have the pendulum swinging back to more diversity with the web science conferences. So it is with life. Aside from the facts that there are trends and pendulum effects, and that paths that cross usually cross again, it is very hard to say exactly how these things play out.</p> <p>At the "20 years of web" panel, there was a round of questions on how different people had been surprised by the web. Surprises ranged from the web's actual scalability to its rapid adoption and the culture of "if I do my part, others will do theirs." On the minus side, the emergence of spam and phishing were mentioned as unexpected developments.</p> <p>Questions of simplicity and complexity got a lot of attention, along with network effects. When things hit the right simplicity at the right place (e.g., HTML and <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x16e16d38">HTTP</a>, which hypertext-wise were nothing special), there is a tipping point.</p> <p>No barrier of entry, not too much modeling, was repeated quite a bit, also in relation to <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1033c750">semantic web</a> and ontology design. There is a magic of emergent effects when the pieces are simple enough: Organic chemistry out of a couple of dozen elements; all the world's information online with a few tags of markup and a couple of protocol verbs. But then this is where the real complexity starts — one half of it in the transport, the other in the applications, yet a narrow interface between the two.</p> <p>This then begs the question of content- and application-aware networks. The preponderance of opinion was for separation of powers — keep carriers and content apart.</p> <p>Michael Brodie commented in the questions to the first panel that simplicity was greatly overrated, that the world was in fact very complex. It seems to me that that any field of human endeavor develops enough complexity to fully occupy the cleverest minds who undertake said activity. The life-cycle between simplicity and complexity seems to be a universal feature. It is a bit like the Zen idea that "for the beginner, rivers are rivers and mountains are mountains, for the student these are imponderable mysteries of bewildering complexity and transcendent dimension but for the master these are again rivers and mountains." One way of seeing this is that the master, in spite of the actual complexity and interrelatedness of all things, sees where these complexities are significant and where not and knows to communicate concerning these as fits the situation.</p> <p>There is no fixed formula for saying where complexities and simplicities fit, relevance of detail is forever contextual. For technological systems, we find that there emerge relatively simple interfaces on either side of which there is huge complexity: The x86 instruction set, TCP/IP, <a href="http://dbpedia.org/resource/SQL" id="link-id0x107869d0">SQL</a>, to name a few. These are lucky breaks, it is very hard to say beforehand where these will emerge. Object oriented people would like to see such everywhere, which just leads to problems of modeling.</p> <p>There was a keynote from Telefonica about infrastructure. We heard that the power and cooling cost more than the equipment, that data centers ought to be scaled down from the football stadium and 20 megawatt scale, that systems must be designed for partitioning, to name a few topics. This is all well accepted. The new question is whether storage should go into the network infrastructure. We have blogged that the network will be the database, and it is no surprise that a telco should have the same idea, just with slightly different emphasis and wording. For Telefonica, this is about efficiency of bulk delivery, for us this is more about virtualized query-able dataspaces. Both will be distributed but issues of separation of powers may keep the two roles of network with storage separate.</p> <p>In conclusion, the network being the database was much more visible and accepted this year than last. The <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x12064438">linked data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x102cf670">web</a> was in Tim B-L's keynote as it was in the opening speech by the Prince of Asturias.</p>
2009-04-30T12:11:45.000004-04:00
Web Science and Keynotes at WWW 2009 (#4 of 5)
http://www.openlinksw.com:443/weblog/oerling/?date=2009-04-30#1551
2009-04-30T16:00:22Z
<p>(Fourth of five posts related to the <a href="http://www2009.org/" id="link-id0x1232b550">WWW 2009</a> conference, held the week of April 20, 2009.) </p> <p>There was quite a bit of talk about what web science could or ought to be. I will here comment a bit on the <a href="http://www2009.org/panels.html" id="link-id1514ec30">panels</a> and <a href="http://www2009.org/keynote_abs.html" id="link-id11a5d620">keynotes</a>, in no special order. </p> <p>In the web science panel, Tim Berners-Lee said that the deliverable of the web science initiative could be a way of making sense of all the world's <a href="http://dbpedia.org/resource/Data" id="link-id0xe01cd68">data</a> once the web had transformed into a database capable of answering arbitrary queries.</p> <p>Michael Brodie of Verizon said that one deliverable would be a well considered understanding of the issue of counter-terrorism and civil liberties: Everything, including terrorism, operates on the platform of the web. How do we understand an issue that is not one of privacy, intelligence, jurisprudence, or sociology, but of all these and more?</p> <p>I would add to this that it is not only a matter of governments keeping and analyzing vast amounts of private data, but of basically anybody who wants to do this being able to do so, even if at a smaller scale. In a way, the data web brings formerly government-only capabilities to the public, and is thus a democratization of intelligence and analytics. The citizen blogger increased the accountability of the press; the citizen analyst may have a similar effect. This is trickier though. We remember Jefferson's words about vigilance and the price of freedom. But vigilance is harder today, not because <a href="http://dbpedia.org/resource/Information" id="link-id0x130558b8">information</a> is not there but because there is so much of it, with diverse spins put on it.</p> <p>Tim B-L said at another panel that it seemed as if the new capabilities, especially the web as a database, were coming just in time to help us cope with the problems confronting the planet. With this, plus having everybody online, we would have more information, more creativity, more of everything at our disposal.</p> <p>I'd have to say that the web is dual use: The bulk of traffic may contribute to distraction more than to awareness, but then the same infrastructure and the social behaviors it supports may also create unprecedented value and in the best of cases also transparency. I have to think of "For whosoever hath, to him shall be given." [Matthew 13:12] This can mean many things; here I am talking about whoever hath a drive for <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x16032470">knowledge</a>.</p> <p>The web is both equalizing and polarizing: The equality is in the access; the polarity in the use made thereof. For a huge amount of noise there will be some crystallization of value that could not have arisen otherwise. Developments have unexpected effects. I would not have anticipated that gaming should advance supercomputing, for example.</p> <p>Wendy Hall gave a dinner speech about communities and conferences; how the original hypertext conferences, with lots of representation of the humanities, became the techie WWW conference series; and how now we have the pendulum swinging back to more diversity with the web science conferences. So it is with life. Aside from the facts that there are trends and pendulum effects, and that paths that cross usually cross again, it is very hard to say exactly how these things play out.</p> <p>At the "20 years of web" panel, there was a round of questions on how different people had been surprised by the web. Surprises ranged from the web's actual scalability to its rapid adoption and the culture of "if I do my part, others will do theirs." On the minus side, the emergence of spam and phishing were mentioned as unexpected developments.</p> <p>Questions of simplicity and complexity got a lot of attention, along with network effects. When things hit the right simplicity at the right place (e.g., HTML and <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1069cc18">HTTP</a>, which hypertext-wise were nothing special), there is a tipping point.</p> <p>No barrier of entry, not too much modeling, was repeated quite a bit, also in relation to <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x15d2c200">semantic web</a> and ontology design. There is a magic of emergent effects when the pieces are simple enough: Organic chemistry out of a couple of dozen elements; all the world's information online with a few tags of markup and a couple of protocol verbs. But then this is where the real complexity starts — one half of it in the transport, the other in the applications, yet a narrow interface between the two.</p> <p>This then begs the question of content- and application-aware networks. The preponderance of opinion was for separation of powers — keep carriers and content apart.</p> <p>Michael Brodie commented in the questions to the first panel that simplicity was greatly overrated, that the world was in fact very complex. It seems to me that that any field of human endeavor develops enough complexity to fully occupy the cleverest minds who undertake said activity. The life-cycle between simplicity and complexity seems to be a universal feature. It is a bit like the Zen idea that "for the beginner, rivers are rivers and mountains are mountains, for the student these are imponderable mysteries of bewildering complexity and transcendent dimension but for the master these are again rivers and mountains." One way of seeing this is that the master, in spite of the actual complexity and interrelatedness of all things, sees where these complexities are significant and where not and knows to communicate concerning these as fits the situation.</p> <p>There is no fixed formula for saying where complexities and simplicities fit, relevance of detail is forever contextual. For technological systems, we find that there emerge relatively simple interfaces on either side of which there is huge complexity: The x86 instruction set, TCP/IP, <a href="http://dbpedia.org/resource/SQL" id="link-id0x10363000">SQL</a>, to name a few. These are lucky breaks, it is very hard to say beforehand where these will emerge. Object oriented people would like to see such everywhere, which just leads to problems of modeling.</p> <p>There was a keynote from Telefonica about infrastructure. We heard that the power and cooling cost more than the equipment, that data centers ought to be scaled down from the football stadium and 20 megawatt scale, that systems must be designed for partitioning, to name a few topics. This is all well accepted. The new question is whether storage should go into the network infrastructure. We have blogged that the network will be the database, and it is no surprise that a telco should have the same idea, just with slightly different emphasis and wording. For Telefonica, this is about efficiency of bulk delivery, for us this is more about virtualized query-able dataspaces. Both will be distributed but issues of separation of powers may keep the two roles of network with storage separate.</p> <p>In conclusion, the network being the database was much more visible and accepted this year than last. The <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x100f4cf0">linked data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x15a55db8">web</a> was in Tim B-L's keynote as it was in the opening speech by the Prince of Asturias.</p>
2009-04-30T12:11:44-04:00
Short Recap of Virtuoso Basics (#3 of 5)
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-04-30#1552
2009-04-30T15:49:53Z
<p>(Third of five posts related to the <a href="http://www2009.org/" id="link-id0x14b582b8">WWW 2009</a> conference, held the week of April 20, 2009.) </p> <p>There are some points that came up in conversation at WWW 2009 that I will reiterate here. We find there is still some lack of clarity in the product image, so I will here condense it.</p> <p> <a href="http://virtuoso.openlinksw.com" id="link-id0x14bf48b8">Virtuoso</a> is a DBMS. We pitch it primarily to the <a href="http://dbpedia.org/resource/Data" id="link-id0x16bc4490">data</a> web space because this is where we see the emerging frontier. Virtuoso does both <a href="http://dbpedia.org/resource/SQL" id="link-id0x1223dc30">SQL</a> and <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x170eec88">SPARQL</a> and can do both at large scale and high performance. The popular perception of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x15a05fc0">RDF</a> and Relational models as mutually exclusive and antagonistic poles is based on the poor scalability of early RDF implementations. What we do is to have all the RDF specifics, like IRIs and typed literals as native SQL types, and to have a cost based optimizer that knows about this all.</p> <p>If you want application-specific data structures as opposed to a schema-agnostic quad-store model (triple + graph-name), then Virtuoso can give you this too. <a href="http://docs.openlinksw.com/virtuoso/rdfsparqlintegrationmiddleware.html#rdfviews" id="link-id14ddc7c8">Rendering application specific data structures as RDF</a> applies equally to relational data in non-Virtuoso databases because Virtuoso SQL can <a href="http://docs.openlinksw.com/virtuoso/qsvdbsrv.html" id="link-id14aaea70">federate tables from heterogenous DBMS</a>.</p> <p>On top of this, there is a <a href="http://docs.openlinksw.com/virtuoso/qswebserver.html" id="link-id16fcde60">web server built in</a>, so that no extra server is needed for web services, web pages, and the like.</p> <p>Installation is simple, just one exe and one config file. There is a huge amount of code in <a href="http://docs.openlinksw.com/virtuoso/installation.html" id="link-id16767b40">installers</a> — application code and test suites and such — but none of this is needed when you deploy. Scale goes from a 25MB memory footprint on the desktop to hundreds of gigabytes of RAM and endless terabytes of disk on shared-nothing clusters.</p> <p>Clusters (coming in Release 6) and SQL federation are <a href="http://download.openlinksw.com/download/product_matrix.vsp?p=l_os&c=39&df=16" id="link-id16722550">commercial only</a>; the rest can be had <a href="http://sourceforge.net/project/showfiles.php?group_id=161622" id="link-id131080a8">under GPL</a>.</p> <p>To condense further:</p> <ul> <li>Scalable Delivery of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1060ad98">Linked Data</a> </li> <li>SPARQL and SQL <ul> <li>Arbitrary RDF Data + Relational</li> <li>Also From 3rd Party <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x16bbce60">RDBMS</a> </li> </ul> </li> <li>Easy Deployment </li> <li>Standard Interfaces <ul> <li> <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x12e284d8">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0xb5e1400">JDBC</a>, OLE DB, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id0x15a55db8">ADO</a>.<a href="http://dbpedia.org/resource/.NET_Framework" id="link-id0x16beb070">NET</a>, XMLA</li> <li> <a href="http://jena.sourceforge.net/" id="link-id0x122b5008">Jena</a>, <a href="http://sourceforge.net/projects/sesame/" id="link-id0x148d4078">Sesame</a>, etc.</li> <li>All Web Protocols </li> </ul> </li> </ul>
2009-04-30T12:11:45-04:00
Short Recap of Virtuoso Basics (#3 of 5)
http://www.openlinksw.com:443/weblog/oerling/?date=2009-04-30#1550
2009-04-30T15:49:53Z
<p>(Third of five posts related to the <a href="http://www2009.org/" id="link-id0x1081fe40">WWW 2009</a> conference, held the week of April 20, 2009.) </p> <p>There are some points that came up in conversation at WWW 2009 that I will reiterate here. We find there is still some lack of clarity in the product image, so I will here condense it.</p> <p> <a href="http://virtuoso.openlinksw.com" id="link-id0xd0e85f0">Virtuoso</a> is a DBMS. We pitch it primarily to the <a href="http://dbpedia.org/resource/Data" id="link-id0x14a294d8">data</a> web space because this is where we see the emerging frontier. Virtuoso does both <a href="http://dbpedia.org/resource/SQL" id="link-id0x108042f8">SQL</a> and <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x10889878">SPARQL</a> and can do both at large scale and high performance. The popular perception of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x107d3b40">RDF</a> and Relational models as mutually exclusive and antagonistic poles is based on the poor scalability of early RDF implementations. What we do is to have all the RDF specifics, like IRIs and typed literals as native SQL types, and to have a cost based optimizer that knows about this all.</p> <p>If you want application-specific data structures as opposed to a schema-agnostic quad-store model (triple + graph-name), then Virtuoso can give you this too. <a href="http://docs.openlinksw.com/virtuoso/rdfsparqlintegrationmiddleware.html#rdfviews" id="link-id14ddc7c8">Rendering application specific data structures as RDF</a> applies equally to relational data in non-Virtuoso databases because Virtuoso SQL can <a href="http://docs.openlinksw.com/virtuoso/qsvdbsrv.html" id="link-id14aaea70">federate tables from heterogenous DBMS</a>.</p> <p>On top of this, there is a <a href="http://docs.openlinksw.com/virtuoso/qswebserver.html" id="link-id16fcde60">web server built in</a>, so that no extra server is needed for web services, web pages, and the like.</p> <p>Installation is simple, just one exe and one config file. There is a huge amount of code in <a href="http://docs.openlinksw.com/virtuoso/installation.html" id="link-id16767b40">installers</a> — application code and test suites and such — but none of this is needed when you deploy. Scale goes from a 25MB memory footprint on the desktop to hundreds of gigabytes of RAM and endless terabytes of disk on shared-nothing clusters.</p> <p>Clusters (coming in Release 6) and SQL federation are <a href="http://download.openlinksw.com/download/product_matrix.vsp?p=l_os&c=39&df=16" id="link-id16722550">commercial only</a>; the rest can be had <a href="http://sourceforge.net/project/showfiles.php?group_id=161622" id="link-id131080a8">under GPL</a>.</p> <p>To condense further:</p> <ul> <li>Scalable Delivery of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x12211da8">Linked Data</a> </li> <li>SPARQL and SQL <ul> <li>Arbitrary RDF Data + Relational</li> <li>Also From 3rd Party <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x168db0e0">RDBMS</a> </li> </ul> </li> <li>Easy Deployment </li> <li>Standard Interfaces <ul> <li> <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x10473bf0">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x12187f58">JDBC</a>, OLE DB, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id0x10354e48">ADO</a>.<a href="http://dbpedia.org/resource/.NET_Framework" id="link-id0x16eeadd0">NET</a>, XMLA</li> <li> <a href="http://jena.sourceforge.net/" id="link-id0x12e3fe08">Jena</a>, <a href="http://sourceforge.net/projects/sesame/" id="link-id0x15e62470">Sesame</a>, etc.</li> <li>All Web Protocols </li> </ul> </li> </ul>
2009-04-30T12:11:43.000001-04:00
Search at WWW 2009 (#2 of 5)
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-04-30#1549
2009-04-30T15:18:24Z
<p>(Second of five posts related to the <a href="http://www2009.org/" id="link-id124024c8">WWW 2009</a> conference, held the week of April 20, 2009.) </p> <p>There was a <a href="http://data.semanticweb.org/conference/www/2009/paper/109/html" id="link-id1207a3b0">workshop on semantic search</a> plus <a href="http://data.semanticweb.org/conference/www/2009/html" id="link-id1704ff48">a number of papers</a> and of course <a href="http://www2009.org/keynote.html" id="link-id11ec08d8">keynotes from Google and Yahoo</a>.</p> <p>A general topic was the use of and access to query logs. Are these the monopoly of GYM (Google, Yahoo, Microsoft) or should they be made more generally available? This is a privacy question. Use of query logs and click through of search results for improved ranking was mentioned many times throughout the conference.</p> <p>The <a href="http://data.semanticweb.org/conference/www/2009/paper/109/html" id="link-id120b7d38">semantic search workshop</a> was largely about benchmarks for keyword search in <a href="http://dbpedia.org/resource/Information" id="link-id171e2950">information</a> retrieval. For <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11e1a9b0">linked data</a>, which is a database proposition, these benchmarks are not really applicable. For document search aided by semantics derived by NLP, these are of course applicable. But there is a divide in approach.</p> <p> <a href="http://g1o.net/foaf.rdf#me" id="link-id11d1c7b0">Giovanni Tummarello</a> <a href="http://data.semanticweb.org/conference/www/2009/paper/59/html" id="link-id169add28">presented</a> <a href="http://sig.ma/" id="link-id11af0128">Sig.ma</a>, a service using <a href="http://sindice.com/" id="link-id11a69fa0">Sindice</a>'s <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id11f3a088">RDF</a> index for collecting all RDF statements about entities matching some set of keywords. One could then choose which sources and which entities were the right ones. One could further store such a query and embed it on a page. The point was that the filtering done manually could be persisted and republished, so as to create dynamic content aggregated from selected live sources. Further speculating, one could use such user feedback for adjusting ranking, even though Sig.ma did not. We may adopt the idea of manually excluding sources into our browser too. Fresnel lenses are another thing to look at.</p> <p>There was <a href="http://www2009.eprints.org/242/" id="link-id11dc7c68">a paper by Josep M. Pujol and Pablo Rodriguez, of Telefonica Research</a>, about returning search to the people by means of Porqpine, a peer-to-peer search implementation based on sharing search results from search engines among peers and indexing them locally as they were retrieved. For users with similar interests, this can give a community based ranking model but has issues of privacy. Another point was that with local processing and personal scale <a href="http://dbpedia.org/resource/Data" id="link-id171a8948">data</a> volumes various kinds of brute force processing were feasible that would cost a lot for the web scale. Much can be done web scale but it must be done cleverly, not with a shell script and not so ad hoc.</p> <p>As a counterpoint to this, there was <a href="http://www2009.eprints.org/220/" id="link-id120bf9e0">a talk about Hadoop and Hive</a>, a map-reduce-based <a href="http://dbpedia.org/resource/SQL" id="link-idee5d700">SQL</a>-like framework. One could do an SQL <code>GROUP BY</code> on text files with record parsing at run time, all spread over a Hadoop cluster. The issue is, if you have a petabyte of data, you may wish to run more than one ad hoc query on it. This means that joining between partitions and complex processing becomes important. This cannot be done without indices and complex query optimization, and needs a DBMS. Stonebraker and company are fully justified in their <a href="http://database.cs.brown.edu/sigmod09/" id="link-id11be1088">critique of map reduce</a>. It looks like each generation must get dazzled by the oversimplified and then retrace the same discoveries of complexity as the previous one.</p> <p>Some of our future plans were confirmed by what we saw, for example as concerns:</p> <ul> <li>Interactively selecting sources for search, showing the graphs, then interactively refining</li> <li>More social networks, more network analysis, and more work on social recommendation</li> <li>Real time indexing of new pings, filling the store by forwarding queries to search engines, and harvesting micro-formats from results</li> <li>Using <a href="http://dbpedia.org/resource/Entity" id="link-id16440770">entity</a> extraction</li> </ul> <p>These are all items in the pipeline, easy to do on top of the existing platform. For the machine learning and NLP parts, we will partner with others, details will be worked out while we work on the items we implement by ourselves.</p>
2009-04-30T12:51:53.000010-04:00
Search at WWW 2009 (#2 of 5)
http://www.openlinksw.com:443/weblog/oerling/?date=2009-04-30#1548
2009-04-30T15:18:24Z
<p>(Second of five posts related to the <a href="http://www2009.org/" id="link-id124024c8">WWW 2009</a> conference, held the week of April 20, 2009.) </p> <p>There was a <a href="http://data.semanticweb.org/conference/www/2009/paper/109/html" id="link-id1207a3b0">workshop on semantic search</a> plus <a href="http://data.semanticweb.org/conference/www/2009/html" id="link-id1704ff48">a number of papers</a> and of course <a href="http://www2009.org/keynote.html" id="link-id11ec08d8">keynotes from Google and Yahoo</a>.</p> <p>A general topic was the use of and access to query logs. Are these the monopoly of GYM (Google, Yahoo, Microsoft) or should they be made more generally available? This is a privacy question. Use of query logs and click through of search results for improved ranking was mentioned many times throughout the conference.</p> <p>The <a href="http://data.semanticweb.org/conference/www/2009/paper/109/html" id="link-id120b7d38">semantic search workshop</a> was largely about benchmarks for keyword search in <a href="http://dbpedia.org/resource/Information" id="link-id171e2950">information</a> retrieval. For <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11e1a9b0">linked data</a>, which is a database proposition, these benchmarks are not really applicable. For document search aided by semantics derived by NLP, these are of course applicable. But there is a divide in approach.</p> <p> <a href="http://g1o.net/foaf.rdf#me" id="link-id11d1c7b0">Giovanni Tummarello</a> <a href="http://data.semanticweb.org/conference/www/2009/paper/59/html" id="link-id169add28">presented</a> <a href="http://sig.ma/" id="link-id11af0128">Sig.ma</a>, a service using <a href="http://sindice.com/" id="link-id11a69fa0">Sindice</a>'s <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id11f3a088">RDF</a> index for collecting all RDF statements about entities matching some set of keywords. One could then choose which sources and which entities were the right ones. One could further store such a query and embed it on a page. The point was that the filtering done manually could be persisted and republished, so as to create dynamic content aggregated from selected live sources. Further speculating, one could use such user feedback for adjusting ranking, even though Sig.ma did not. We may adopt the idea of manually excluding sources into our browser too. Fresnel lenses are another thing to look at.</p> <p>There was <a href="http://www2009.eprints.org/242/" id="link-id11dc7c68">a paper by Josep M. Pujol and Pablo Rodriguez, of Telefonica Research</a>, about returning search to the people by means of Porqpine, a peer-to-peer search implementation based on sharing search results from search engines among peers and indexing them locally as they were retrieved. For users with similar interests, this can give a community based ranking model but has issues of privacy. Another point was that with local processing and personal scale <a href="http://dbpedia.org/resource/Data" id="link-id171a8948">data</a> volumes various kinds of brute force processing were feasible that would cost a lot for the web scale. Much can be done web scale but it must be done cleverly, not with a shell script and not so ad hoc.</p> <p>As a counterpoint to this, there was <a href="http://www2009.eprints.org/220/" id="link-id120bf9e0">a talk about Hadoop and Hive</a>, a map-reduce-based <a href="http://dbpedia.org/resource/SQL" id="link-idee5d700">SQL</a>-like framework. One could do an SQL <code>GROUP BY</code> on text files with record parsing at run time, all spread over a Hadoop cluster. The issue is, if you have a petabyte of data, you may wish to run more than one ad hoc query on it. This means that joining between partitions and complex processing becomes important. This cannot be done without indices and complex query optimization, and needs a DBMS. Stonebraker and company are fully justified in their <a href="http://database.cs.brown.edu/sigmod09/" id="link-id11be1088">critique of map reduce</a>. It looks like each generation must get dazzled by the oversimplified and then retrace the same discoveries of complexity as the previous one.</p> <p>Some of our future plans were confirmed by what we saw, for example as concerns:</p> <ul> <li>Interactively selecting sources for search, showing the graphs, then interactively refining</li> <li>More social networks, more network analysis, and more work on social recommendation</li> <li>Real time indexing of new pings, filling the store by forwarding queries to search engines, and harvesting micro-formats from results</li> <li>Using <a href="http://dbpedia.org/resource/Entity" id="link-id16440770">entity</a> extraction</li> </ul> <p>These are all items in the pipeline, easy to do on top of the existing platform. For the machine learning and NLP parts, we will partner with others, details will be worked out while we work on the items we implement by ourselves.</p>
2009-04-30T12:51:48-04:00
Linked Data & Identity
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-04-29#1547
2009-04-29T20:05:58Z
<blockquote> <cite>A person, organization, place, idea, subject matter topic/heading, and other real world things possess "identity" -- that is, a constellation of characteristics that distinguish them from any other identity. Associated with this abstraction can be a label used as a reference, or "identifier". This is the distinction between a thing and the name of the thing.</cite> </blockquote> <blockquote> <p>section from <a href="http://www.ietf.org/internet-drafts/draft-ietf-dkim-overview-11.txt" id="link-id15a13d40">IETF's Domain Keys spec</a>. (paraphrased by <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id13d88ed8">me</a>) </p>.</blockquote> <p>The <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id11d5b9a8">Linked Data meme</a> is based on the use of HTTP based URIs as reference / identifier labels associated with the "identity abstraction" referred to above. Thus, when you de-reference (request <a href="http://dbpedia.org/resource/Information" id="link-id14706fb8">information</a> about) an HTTP based <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id12b4ea50">URI</a> you ultimately end up with a resource <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id12127e20">URL</a> that exposes the "constellation of characteristics" mentioned above, in a representation negotiated at request time -- between an HTTP client and server e.g., (X)HTML, JSON, XML, RDF/XML, N3, Turtle, Trix, others :-)</p> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1546" id="link-id11b67288">What is the Linked Data meme About?</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1543" id="link-id151fa890">Simple Explanation of RDF & Linked Data Dynamics</a>.</li> <li> <a href="http://www.handle.net/" id="link-id11d9cd30">Handle</a> -- <a href="http://dbpedia.org/resource/Internet" id="link-id0x131986f0">Internet</a> wide Identity Scheme and Resolution System</li> </ul>
2009-05-01T12:25:49-04:00
What is the Linked Data Meme about?
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-04-29#1546
2009-04-29T15:32:49Z
<p>The act of using URIs to "refer to" (reference) <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> addressable <a href="http://dbpedia.org/resource/Data">data</a> objects. It's also the act of using the same <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id12b41fc0">URI</a> to de-reference the description of a referenced data object; in this case, the representation of the description is negotiated by a Web client and/or Web server. Thus, you can access the description of a data object via data representation formats such as: JSON, XML, (X)HTML, RDF/XML, N3, Turtle, TriX etc. </p> <p> <strong>Note:</strong> In proper Web parlance, a data object is referred to as a resource.</p> <h3>Simple example (using <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x131005a0">DBpedia</a>)</h3> <p>In the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x13299d20">Linked Data</a> realm, If you want to make a reference to the <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id0x188210a8">Linked Data meme</a> in a <a href="http://dbpedia.org/resource/Blog" id="link-id0x158a3fc0">blog</a> post, you are better off using the resource <strong>URI</strong>: http://dbpedia.org/resource/Linked_Data, instead of the Web page <strong><a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id0x142865b0">URL</a></strong>: http://dbpedia.org/page/Linked_Data, which is the address of a physical document (an <a href="http://dbpedia.org/resource/Information" id="link-id0x15884030">information</a> conveying artifact) that at best visually presents the negotiated representation of a resource description.</p> <h3>Why is this valuable?</h3> <p>In the simplest sense, you only have one focal point for referencing (referring to) and de-referencing (retrieving data about) a given Web resource. It protects you from the impact of Web document location changes (amongst many other things).</p> <p>Remember, a single URI is a conduit into a realm where the identity, access, representation, presentation, and storage of a resource (data object) are completely distinct. It's the mechanism for conducting data across network, machine, operating system, dbms engine, application, and service (API) boundaries. Thus, without "linked data meme" prescribed URI referencing and de-referencing, we are simply back to "business as usual" re. the industry at large, where networks, operating systems, dbms engines, applications, and services (APIs) become the basis for "data lock-in" and silo construction.</p> <h3>Going forward</h3> <p>Take a second to think about the profound virtues of the ubiquitous Web of Linked Document URLs that we have today, and then apply that thinking to the burgeoning Web of Linked Data URIs, that has just turned corner and heading in everyone's direction at full blast.</p> <p> <strong>Note to "Social Media" players:</strong> Who you know isn't the canonical object of sociality. What you are i.e., your description and the data objects it exposes, are real objects of your sociality :-)</p> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=%22Linked%20Data%22&type=text&output=html" id="link-id14d44430">Other post in this Blog Data Space associated with "Linked Data"</a>. </li> </ul>
2009-04-29T16:31:10-04:00
Linked Data at WWW 2009 (#1 of 5)
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-04-27#1545
2009-04-27T21:28:11Z
<p>(First of five posts related to the <a href="http://www2009.org/" id="link-id0x12d8ed90">WWW 2009</a> conference, held the week of April 20, 2009.)</p> <p>We gave a talk at the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x152bf430">Linked Open Data</a> workshop, <a href="http://events.linkeddata.org/ldow2009/" id="link-id0x191721c8">LDOW 2009</a>, at WWW 2009. I did not go very far into the technical points in the talk, as there was almost no time and the points are rather complex. Instead, I emphasized what new things had become possible with recent developments.</p> <p>The problem we do not cease hearing about is scale. We have solved most of it. There is scale in the schema: Put together, ontologies go over a million classes/properties. Which ones are relevant depends, and the user should have the choice. The instance <a href="http://dbpedia.org/resource/Data" id="link-id0x17c8f998">data</a> is in the tens of billions of triples, much derived from Web 2.0 sources but also much published as <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xd562090">RDF</a>.</p> <p>To make sense of this all, we need quick summaries and search. Without navigation via joins, the value will be limited. Fast joining, counting, grouping, and ranking are key.</p> <p>People will use different terms for the same thing. The issue of identity is philosophical. In order to do reasoning one needs strong identity; a statement like <i>x is a bit like y</i> is not very useful in a database context. Whether any x and y can be considered the same depends on the context. So leave this for query time. The conditions under which two people are considered the same will depend on whether you are doing marketing analysis or law enforcement. A general purpose data store cannot anticipate all the possibilities, so smush on demand, as you go, as has been said many times.</p> <p>Against this backdrop, we offer a solution with which anybody who so chooses can play with big data, whether a search or analytics player.</p> <p>We are going in the direction of more and more ad hoc processing at larger and larger scale. With good query parallelization, we can do big joins without complex programming. No explicit Map Reduce jobs or the like. What was done with special code with special parallel programming models, can now be done in <a href="http://dbpedia.org/resource/SQL" id="link-id0x60bd0c48">SQL</a> and <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x13db1ff0">SPARQL</a>.</p> <p>To showcase this, we do <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x10a5dde8">linked data</a> search, browsing, and so on, but are essentially a platform provider.</p> <p>Entry costs into relatively high end databases have dropped significantly. A cluster with 1 TB of RAM sells for $75K or so at today's retail prices and fits under a desk. For intermittent use, the rent for 1TB RAM is $1228 per day on <a href="http://aws.amazon.com/ec2/" id="link-id0xa59039d8">EC2</a>. With this on one side and <a href="http://virtuoso.openlinksw.com" id="link-id0x19f86c10">Virtuoso</a> on the other, a lot that was impractical in the past is now within reach. Like <a href="http://g1o.net/foaf.rdf#me" id="link-id0xa1853af8">Giovanni Tummarello</a> put it for airplanes, the physics are as they were for <a href="http://dbpedia.org/resource/Leonardo_da_Vinci" id="link-id0x12df02e0">da Vinci</a> but materials and engines had to develop a bit before there was commercial potential. So it is also with analytics for everyone.</p> <p>A remark from the audience was that all the stuff being shown, not limited to Virtuoso, was non-standard, having to do with text search, with ranking, with extensions, and was in fact not SPARQL and pure linked data principles. Further, by throwing this all together, one got something overcomplicated, too heavy.</p> <p>I answered as follows, which apparently cannot be repeated too much:</p> <p>First, everybody expects a text search box, and is conditioned to having one. No text search and no ranking is a non-starter. <i>Ceterum censeo</i>, for database, the next generation cannot be less expressive than the previous. All of SQL and then some is where SPARQL must be. The barest minimum is being able to say anything one can say in SQL, and then justify SPARQL by saying that it is better for heterogenous data, schema last, and so on. On top of this, transitivity and rules will not hurt. For now, the current SPARQL working group will at least reach basic SQL parity; the edge will still remain implementation dependent.</p> <p>Another remark was that joining is slow. Depends. Anything involving more complex disk access than linear reading of a blob is generally not good for interactive use. But with adequate memory, and with all hot spots in memory, we do some 3.2 million random-accesses-per-second on 12 cores, with easily 80% platform utilization for a single large query. The high utilization means that times drop as processing gets divided over more partitions.</p> <p>There was a talk about <a href="http://semanticweb.org/wiki/MashQL" id="link-id0x1642a780">MashQL</a> by <a href="http://data.semanticweb.org/person/mustafa-jarrar" id="link-id0x116e5af8">Mustafa Jarrar</a>, concerning an abstraction on top of SPARQL for easy composition of tree-structured queries. The idea was that such queries can be evaluated "on the fly" as they are being composed. As it happens, we already have an <a href="http://dbpedia.org/resource/XML" id="link-id0x11442520">XML</a>-based query abstraction layer incorporated into Virtuoso 6.0's built-in <a href="http://lod.openlinksw.com/fct/facet.vsp" id="link-id0x6a9ebfe0">Faceted Data Browser Service</a>, and the effects are probably quite similar. The most important point here is that by using XML, both of these approaches are interoperable against a Virtuoso back-end. Along similar lines, we did not get to talk to the G Facets people but our message to them is the same: <i>Use the <a href="http://lod.openlinksw.com/fct/facet.vsp" id="link-id0x1676e158">faceted browser service</a> to get vastly higher performance when querying against Linked Data, be it <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x12653418">DBpedia</a> or the <a href="http://dbpedia.org/resource/Entity" id="link-id0x10a61e78">entity</a> <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x164150d8">LOD</a> <a href="http://lod.openlinksw.com/" id="link-id0xc5ec918">Cloud</a>. Virtuoso 6.0 (Open Source Edition) "<a href="http://sourceforge.net/project/showfiles.php?group_id=161622&package_id=319652&release_id=677866" id="link-id12159728">TP1</a>" is now publicly available as a Technology Preview (beta).</i> </p> <p>We heard that there is an effort for porting Freebase's Parallax to SPARQL. The same thing applies to this. With a number of different data viewers on top of SPARQL, we come closer to broad-audience linked-data applications. These viewers are still too generic for the end user, though. We fully believe that for both search and transactions, application-domain-specific workflows will stay relevant. But these can be made to a fair degree by specializing generic linked-data-bound controls and gluing them together with some scripting.</p> <p>As said before, the application will interface the user to the vocabulary. The vocabulary development takes the modeling burden from the application and makes for interchangeable experience on the same data. The data in turn is "virtualized" into the database cloud or the local secure server, as the use case may require. </p> <p>For ease of adoption, open competition, and safety from lock-in, the community needs a SPARQL whose usability is not totally dependent on vendor extensions. But we might <i>de facto</i> have that in just a bit, whenever there is a working draft from the SPARQL WG.</p> <p>Another topic that we encounter often is the question of integration (or lack thereof) between communities. For example, database conferences reject <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x12563ea0">semantic web</a> papers and vice versa. Such politics would seem to emerge naturally but are nonetheless detrimental. We really should partner with people who write papers as their principal occupation. We ourselves do software products and use very little time for papers, so some of the bad reviews we have received do make a legitimate point. By rights, we should go for database venues but we cannot have this take too much time. So we are open to partnering for splitting the opportunity cost of multiple submissions.</p> <p>For future work, there is nothing radically new. We continue testing and productization of cluster databases. Just deliver what is in the pipeline. The essential nature of this is adding more and more cases of better and better parallelization in different query situations. The present usage patterns work well for finding bugs and performance bottlenecks. For presentation, our goal is to have third party viewers operate with our platform. We cannot completely leave data browsing and UI to third parties since we must from time to time introduce various unique functionality. Most interaction should however go via third party applications.</p>
2009-04-28T11:27:57-04:00
Linked Data at WWW 2009 (#1 of 5)
http://www.openlinksw.com:443/weblog/oerling/?date=2009-04-27#1544
2009-04-27T21:28:11Z
<p>(First of five posts related to the <a href="http://www2009.org/" id="link-id0x114c2450">WWW 2009</a> conference, held the week of April 20, 2009.)</p> <p>We gave a talk at the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x166e10f0">Linked Open Data</a> workshop, <a href="http://events.linkeddata.org/ldow2009/" id="link-id0x19c2b1f0">LDOW 2009</a>, at WWW 2009. I did not go very far into the technical points in the talk, as there was almost no time and the points are rather complex. Instead, I emphasized what new things had become possible with recent developments.</p> <p>The problem we do not cease hearing about is scale. We have solved most of it. There is scale in the schema: Put together, ontologies go over a million classes/properties. Which ones are relevant depends, and the user should have the choice. The instance <a href="http://dbpedia.org/resource/Data" id="link-id0x12c65250">data</a> is in the tens of billions of triples, much derived from Web 2.0 sources but also much published as <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x441128e0">RDF</a>.</p> <p>To make sense of this all, we need quick summaries and search. Without navigation via joins, the value will be limited. Fast joining, counting, grouping, and ranking are key.</p> <p>People will use different terms for the same thing. The issue of identity is philosophical. In order to do reasoning one needs strong identity; a statement like <i>x is a bit like y</i> is not very useful in a database context. Whether any x and y can be considered the same depends on the context. So leave this for query time. The conditions under which two people are considered the same will depend on whether you are doing marketing analysis or law enforcement. A general purpose data store cannot anticipate all the possibilities, so smush on demand, as you go, as has been said many times.</p> <p>Against this backdrop, we offer a solution with which anybody who so chooses can play with big data, whether a search or analytics player.</p> <p>We are going in the direction of more and more ad hoc processing at larger and larger scale. With good query parallelization, we can do big joins without complex programming. No explicit Map Reduce jobs or the like. What was done with special code with special parallel programming models, can now be done in <a href="http://dbpedia.org/resource/SQL" id="link-id0x16766eb0">SQL</a> and <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1645ddc8">SPARQL</a>.</p> <p>To showcase this, we do <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0xa167e698">linked data</a> search, browsing, and so on, but are essentially a platform provider.</p> <p>Entry costs into relatively high end databases have dropped significantly. A cluster with 1 TB of RAM sells for $75K or so at today's retail prices and fits under a desk. For intermittent use, the rent for 1TB RAM is $1228 per day on <a href="http://aws.amazon.com/ec2/" id="link-id0xa1a67b70">EC2</a>. With this on one side and <a href="http://virtuoso.openlinksw.com" id="link-id0x1622d4e0">Virtuoso</a> on the other, a lot that was impractical in the past is now within reach. Like <a href="http://g1o.net/foaf.rdf#me" id="link-id0x3d5c8b50">Giovanni Tummarello</a> put it for airplanes, the physics are as they were for <a href="http://dbpedia.org/resource/Leonardo_da_Vinci" id="link-id0x198e7cc0">da Vinci</a> but materials and engines had to develop a bit before there was commercial potential. So it is also with analytics for everyone.</p> <p>A remark from the audience was that all the stuff being shown, not limited to Virtuoso, was non-standard, having to do with text search, with ranking, with extensions, and was in fact not SPARQL and pure linked data principles. Further, by throwing this all together, one got something overcomplicated, too heavy.</p> <p>I answered as follows, which apparently cannot be repeated too much:</p> <p>First, everybody expects a text search box, and is conditioned to having one. No text search and no ranking is a non-starter. <i>Ceterum censeo</i>, for database, the next generation cannot be less expressive than the previous. All of SQL and then some is where SPARQL must be. The barest minimum is being able to say anything one can say in SQL, and then justify SPARQL by saying that it is better for heterogenous data, schema last, and so on. On top of this, transitivity and rules will not hurt. For now, the current SPARQL working group will at least reach basic SQL parity; the edge will still remain implementation dependent.</p> <p>Another remark was that joining is slow. Depends. Anything involving more complex disk access than linear reading of a blob is generally not good for interactive use. But with adequate memory, and with all hot spots in memory, we do some 3.2 million random-accesses-per-second on 12 cores, with easily 80% platform utilization for a single large query. The high utilization means that times drop as processing gets divided over more partitions.</p> <p>There was a talk about <a href="http://semanticweb.org/wiki/MashQL" id="link-id0x60bd57b0">MashQL</a> by <a href="http://data.semanticweb.org/person/mustafa-jarrar" id="link-id0xa1fb98d8">Mustafa Jarrar</a>, concerning an abstraction on top of SPARQL for easy composition of tree-structured queries. The idea was that such queries can be evaluated "on the fly" as they are being composed. As it happens, we already have an <a href="http://dbpedia.org/resource/XML" id="link-id0x1923a380">XML</a>-based query abstraction layer incorporated into Virtuoso 6.0's built-in <a href="http://lod.openlinksw.com/fct/facet.vsp" id="link-id0x67712740">Faceted Data Browser Service</a>, and the effects are probably quite similar. The most important point here is that by using XML, both of these approaches are interoperable against a Virtuoso back-end. Along similar lines, we did not get to talk to the G Facets people but our message to them is the same: <i>Use the <a href="http://lod.openlinksw.com/fct/facet.vsp" id="link-id0x70df2798">faceted browser service</a> to get vastly higher performance when querying against Linked Data, be it <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x1b3fd608">DBpedia</a> or the <a href="http://dbpedia.org/resource/Entity" id="link-id0x13ecd708">entity</a> <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x17f16970">LOD</a> <a href="http://lod.openlinksw.com/" id="link-id0x54334250">Cloud</a>. Virtuoso 6.0 (Open Source Edition) "<a href="http://sourceforge.net/project/showfiles.php?group_id=161622&package_id=319652&release_id=677866" id="link-id12159728">TP1</a>" is now publicly available as a Technology Preview (beta).</i> </p> <p>We heard that there is an effort for porting Freebase's Parallax to SPARQL. The same thing applies to this. With a number of different data viewers on top of SPARQL, we come closer to broad-audience linked-data applications. These viewers are still too generic for the end user, though. We fully believe that for both search and transactions, application-domain-specific workflows will stay relevant. But these can be made to a fair degree by specializing generic linked-data-bound controls and gluing them together with some scripting.</p> <p>As said before, the application will interface the user to the vocabulary. The vocabulary development takes the modeling burden from the application and makes for interchangeable experience on the same data. The data in turn is "virtualized" into the database cloud or the local secure server, as the use case may require. </p> <p>For ease of adoption, open competition, and safety from lock-in, the community needs a SPARQL whose usability is not totally dependent on vendor extensions. But we might <i>de facto</i> have that in just a bit, whenever there is a working draft from the SPARQL WG.</p> <p>Another topic that we encounter often is the question of integration (or lack thereof) between communities. For example, database conferences reject <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x185d6bf8">semantic web</a> papers and vice versa. Such politics would seem to emerge naturally but are nonetheless detrimental. We really should partner with people who write papers as their principal occupation. We ourselves do software products and use very little time for papers, so some of the bad reviews we have received do make a legitimate point. By rights, we should go for database venues but we cannot have this take too much time. So we are open to partnering for splitting the opportunity cost of multiple submissions.</p> <p>For future work, there is nothing radically new. We continue testing and productization of cluster databases. Just deliver what is in the pipeline. The essential nature of this is adding more and more cases of better and better parallelization in different query situations. The present usage patterns work well for finding bugs and performance bottlenecks. For presentation, our goal is to have third party viewers operate with our platform. We cannot completely leave data browsing and UI to third parties since we must from time to time introduce various unique functionality. Most interaction should however go via third party applications.</p>
2015-08-20T17:24:34.169932-04:00
Simple Explanation of RDF and Linked Data Dynamics
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-04-24#1543
2009-04-24T20:59:08Z
<h3>What is RDF?</h3> <p>The acronym stands for: Resource Description Framework. And that's just what it is.</p> <p>RDF is comprised of a <a href="http://dbpedia.org/resource/Data">Data</a> Model (<a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id11bb5cd8">EAV</a>/CR Graph) and Data Representation Formats such as: N3, Turtle, RDF/XML etc.</p> <p>RDF's essence is about: "Entities" and "Attributes" being <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id14362100">URI</a> based, while "Values" may be URI or Literals (typed or untyped) based. </p> <p>URIs are <a href="http://dbpedia.org/resource/Entity" id="link-id139066e8">Entity</a> Identifiers.</p> <h3>What is <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11ed5340">Linked Data</a>?</h3> <p>Short for "<a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> of Linked Data" or "Linked Data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id13f4b878">Web</a>".</p> <p>A term coined by <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id117b4310">TimBL</a> that describes an HTTP based "<a href="http://dbpedia.org/resource/Reference_(computer_science)" id="link-id138fed30">data access by reference</a> pattern" that uses a single pointer or handle for "referring to" and "obtaining actual data about" an entity.</p> <p>Linked Data uses the deceptively simple messaging scheme of HTTP to deliver a granular entity reference and access mechanism that transcends traditional computing boundaries such as: operating system, application, database engines, and networks.</p> <h3>How are Linked Data & RDF Related?</h3> <p>Linked Data simply mandates the following re. RDF:</p> <ul> <li>URIs should be HTTP based so that you can "refer to" (Reference) an Entity, its Attributes, or URI based Attribute values via the Web (infact any HTTP based network e.g., Intranets and Extranets)</li> <li> URIs should also be HTTP based so that you can use them to de-reference resource descriptions via the Web (or Intranets and Extranets).</li> </ul> <p> <strong>Note:</strong> by Entity I am also referring to: a resource (Web parlance), data item, data object, real-world object, or datum. </p> <p>Linked Data is also about, using URIs and HTTP's content negotiation feature to separate: presentation, representation, access, and identity of data items. Even better, content negotiation can be driven by user agent and/or data server based quality of service algorithms (representation preference order schemes).</p> <p>To conclude, Linked Data is ultimately about the realization that: Data is the new Electricity, and it's conductors are URIs :-)</p> <p> <strong>Tip to governments of the world</strong>: we are in exponential times, the current downturn is but one side of the "exponential times ledger", the other side of the "exponential times ledger" is simply about unleashing "raw data" -- in structured form -- into the Web, so that "citizen analysts" can blossom and ultimately deliver the transparency desperately sought at every level of the economic value chain. Think: "raw data ready" whenever you ponder about "shovel ready" infrastructure projects!</p>
2009-04-24T17:14:41-04:00
Take N: Yet Another OpenLink Data Spaces Introduction
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-04-22#1542
2009-04-22T18:46:18Z
<h3>Problem:</h3> <p>Your Life, Profession, <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a>, and <a href="http://dbpedia.org/resource/Internet" id="link-id0x1c6687f8">Internet</a> do not need to become mutually exclusive due to "<a href="http://dbpedia.org/resource/Information" id="link-id0x1c6696e8">information</a> overload".</p> <h3>Solution:</h3> <p> A platform or service that delivers a point of online presence that embodies the fundamental separation of: Identity, <a href="http://dbpedia.org/resource/Data">Data</a> Access, Data Representation, Data Presentation, by adhering to Web and Internet protocols.</p> <h3>How:</h3> <p> Typical post installation (Local or Cloud) task sequence:</p> <ol> <li> Identify myself (happens automatically by way of registration)</li> <li>If in an LDAP environment, import accounts or associate system with LDAP for account lookup and authentication</li> <li> Identify Online Accounts (by fleshing out profile) which also connects system to online accounts and their data</li> <li>Use Profile for granular description (Biography, Interests, WishList, OfferList, etc.)</li> <li>Optionally upstream or downstream data to and from my online accounts</li> <li>Create content Tagging Rules</li> <li>Create rules for associating Tags with formal URIs</li> <li>Create automatic Hyperlinking Rules for reuse when new content is created (e.g. <a href="http://dbpedia.org/resource/Blog" id="link-id11a7c660">Blog</a> posts)</li> <li>Exploit Data Portability virtues of RSS, Atom, OPML, <a href="http://dbpedia.org/resource/RDFa" id="link-id13f54d50">RDFa</a>, RDF/XML, and other formats for imports and exports</li> <li>Automatically <a href="http://dbpedia.org/resource/Tag" id="link-id121ddff0">tag</a> imported content</li> <li>Use function-specific helper application UIs for domain specific data generation e.g. AddressBook (optionally use vCard import), Calendar (optionally use iCalendar import), Email, File Storage (use WebDAV mount with copy and paste or HTTP GET), Feed Subscriptions (optionally import RSS/Atom/OPML feeds), Bookmarking (optionally import bookmark.html or XBEL) etc..</li> <li>Optionally enable "Conversation" feature (today: Social Media feature) across the relevant application domains (manage conversations under covers using NNTP, the standard for this functionality realm) </li> <li>Generate HTTP based <a href="http://dbpedia.org/resource/Entity" id="link-id13d5d378">Entity</a> IDs (URIs) for every piece of data in this burgeoning <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id11a69670">data space</a> </li> <li>Use REST based APIs to perform CRUD tasks against my data (local and remote) (<a href="http://dbpedia.org/resource/SPARQL" id="link-id11a76e10">SPARQL</a>, GData, Ubiquity Commands, Atom Publishing)</li> <li>Use OpenID, OAuth, <a href="http://dbpedia.org/resource/Friend_of_a_friend" id="link-id11c9b3e0">FOAF</a>+SSL, FOAF+SSL+OpenID for accessing data elsewhere</li> <li>Use OpenID, OAuth, FOAF+SSL, FOAF+SSL+OpenID for Controlling access to my data (Self Signed Certificate Generation, Browser Import of said Certificate & associated Private Key, plus persistence of Certificate to FOAF based profile data space in "one click")</li> <li>Have a simple UI for <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id14015bd0">Entity</a>-Attribute-Value or Subject-Predicate-Object arbitrary data annotations and creation since you can't pre model an "<a href="http://dbpedia.org/resource/Open_world_assumption" id="link-id11cd8548">Open World</a>" where the only constant is data flow</li> <li>Have my Personal <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id142beee8">URI</a> (Web ID) as the single entry point for controlled access to my HTTP accessible data space</li> </ol> <p> I've just outlined a snippet of the capabilities of the <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id13d64740">OpenLink Data Spaces</a> platform. A platform built using OpenLink <a href="http://virtuoso.openlinksw.com" id="link-id13d74170">Virtuoso</a>, architected to deliver: open, platform independent, multi-model, data access and data management across heterogeneous data sources. </p> <p> All you need to remember is your URI when seeking to interact with your data space.</p> <h3>Related</h3> <ol> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/GetAPersonalURIIn5MinutesOrLess" id="link-id13c97948">Get Yourself a URI (Web ID) in 5 Minutes or Less!</a> </li> <li> <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=%22data%20spaces%22&type=text&output=html" id="link-id1431e088">Various posts over the years about Data Spaces</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1415" id="link-id11f837f0">Future of Desktop Post</a> </li> <li> <a href="http://bnode.org/blog/2009/04/22/semantic-web-apps-to-simplify-my-life" id="link-id1393f8a8">Simplify My Life Post</a> by <a href="http://bnode.org/about" id="link-id11da0cc8">Bengee Nowack</a> </li> </ol>
2009-04-22T15:32:06.000020-04:00
Web Scale and Fault Tolerance
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-04-01#1541
2009-04-01T15:18:06Z
<p>One concern about <a href="http://virtuoso.openlinksw.com" id="link-id0x719d2f8">Virtuoso</a> Cluster is fault tolerance. This post talks about the basics of fault tolerance and what we can do with this, from improving resilience and optimizing performance to accommodating bulk loads without impacting interactive response. We will see that this is yet another step towards a 24/7 web-scale <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0xa9a1d8d8">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x25201030">Web</a>. We will see how large scale, continuous operation, and redundancy are related.</p> <p>It has been said many times — when things are large enough, failures become frequent. In view of this, basic storage of partitions in multiple copies is built into the Virtuoso cluster from the start. Until now, this feature has not been tested or used very extensively, aside from the trivial case of keeping all schema <a href="http://dbpedia.org/resource/Information" id="link-id0x4548898">information</a> in synchronous replicas on all servers.</p> <h2>Approaches to Fault Tolerance</h2> <p>Fault tolerance has many aspects but it starts with keeping <a href="http://dbpedia.org/resource/Data" id="link-id0x18757400">data</a> in at least two copies. There are shared-disk cluster databases like <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x711c900">Oracle</a> RAC that do not depend on partitioning. With these, as long as the disk image is intact, servers can come and go. The fault tolerance of the disk in turn comes from mirroring done by the disk controller. Raids other than mirrored disk are not really good for databases because of write speed.</p> <p>With shared-nothing setups like Virtuoso, fault tolerance is based on multiple servers keeping the same logical data. The copies are synchronized transaction-by-transaction but are not bit-for-bit identical nor write-by-write synchronous as is the case with mirrored disks.</p> <p>There are asynchronous replication schemes generally based on log shipping, where the replica replays the transaction log of the master copy. The master copy gets the updates, the replica replays them. Both can take queries. These do not guarantee an entirely ACID fail-over but for many applications they come close enough.</p> <p>In a tightly coupled cluster, it is possible to do synchronous, transactional updates on multiple copies without great added cost. Sending the message to two places instead of one does not make much difference since it is the latency that counts. But once we go to wide area networks, this becomes as good as unworkable for any sort of update volume. Thus, wide area replication must in practice be asynchronous.</p> <p>This is a subject for another discussion. For now, the short answer is that wide area log shipping must be adapted to the application's requirements for synchronicity and consistency. Also, exactly what content is shipped and to where depends on the application. Some application-specific logic will likely be involved; more than this one cannot say without a specific context.</p> <h2>Basics of Partition Fail-Over</h2> <p>For now, we will be concerned with redundancy protecting against broken hardware, software slowdown, or crashes inside a single site.</p> <p>The basic idea is simple: Writes go to all copies; reads that must be repeatable or serializable (i.e., locking) go to the first copy; reads that refer to committed state without guarantee of repeatability can be balanced among all copies. When a copy goes offline, nobody needs to know, as long as there is at least one copy online for each partition. The exception in practice is when there are open cursors or such stateful things as aggregations pending on a copy that goes down. Then the query or transaction will abort and the application can retry. This looks like a deadlock to the application.</p> <p>Coming back online is more complicated. This requires establishing that the recovering copy is actually in sync. In practice this requires a short window during which no transactions have uncommitted updates. Sometimes, forcing this can require aborting some transactions, which again looks like a deadlock to the application.</p> <p>When an error is seen, such as a process no longer accepting connections and dropping existing cluster connections, we in practice go via two stages. First, the operations that directly depended on this process are aborted, as well as any computation being done on behalf of the disconnected server. At this stage, attempting to read data from the partition of the failed server will go to another copy but writes will still try to update all copies and will fail if the failed copy continues to be offline. After it is established that the failed copy will stay off for some time, writes may be re-enabled — but now having the failed copy rejoin the cluster will be more complicated, requiring an atomic window to ensure sync, as mentioned earlier.</p> <p>For the DBA, there can be intermittent software crashes where a failed server automatically restarts itself, and there can be prolonged failures where this does not happen. Both are alerts but the first kind can wait. Since a system must essentially run itself, it will wait for some time for the failed server to restart itself. During this window, all reads of the failed partition go to the spare copy and writes give an error. If the spare does not come back up in time, the system will automatically re-enable writes on the spare but now the failed server may no longer rejoin the cluster without a complex sync cycle. This all can happen in well under a minute, faster than a human operator can react. The diagnostics can be done later.</p> <p>If the situation was a hardware failure, recovery consists of taking a spare server and copying the database from the surviving online copy. This done, the spare server can come on line. Copying the database can be done while online and accepting updates but this may take some time, maybe an hour for every 200G of data copied over a network. In principle this could be automated by scripting, but we would normally expect a human DBA to be involved.</p> <p>As a general rule, reacting to the failure goes automatically without disruption of service but bringing the failed copy online will usually require some operator action.</p> <h2>Levels of Tolerance and Performance</h2> <p>The only way to make failures totally invisible is to have all in duplicate and provisioned so that the system never runs at more than half the total capacity. This is often not economical or necessary. This is why we can do better, using the spare capacity for more than standby.</p> <p>Imagine keeping a repository of linked data. Most of the content will come in through periodic bulk replacement of data sets. Some data will come in through pings from applications publishing FOAF and similar. Some data will come through on-demand RDFization of resources.</p> <p>The performance of such a repository essentially depends on having enough memory. Having this memory in duplicate is just added cost. What we can do instead is have all copies store the whole partition but when routing queries, apply range partitioning on top of the basic hash partitioning. If one partition stores IDs 64K - 128K, the next partition 128K - 192K, and so forth, and all partitions are stored in two full copies, we can route reads to the first 32K IDs to the first copy and reads to the second 32K IDs to the second copy. In this way, the copies will keep different working sets. The RAM is used to full advantage.</p> <p>Of course, if there is a failure, then the working set will degrade, but if this is not often and not for long, this can be quite tolerable. The alternate expense is buying twice as much RAM, likely meaning twice as many servers. This workload is memory intensive, thus servers should have the maximum memory they can have without going to parts that are so expensive one gets a new server for the price of doubling memory.</p> <h2>Background Bulk Processing</h2> <p>When loading data, the system is online in principle, but query response can be quite bad. A large <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x19fd9c18">RDF</a> load will involve most memory and queries will miss the cache. The load will further keep most disks busy, so response is not good. This is the case as soon as a server's partition of the database is four times the size of RAM or greater. Whether the work is bulk-load or bulk-delete makes little difference.</p> <p>But if partitions are replicated, we can temporarily split the database so that the first copies serve queries and the second copies do the load. If the copies serving on line activities do some updates also, these updates will be committed on both copies. But the load will be committed on the second copy only. This is fully appropriate as long as the data are different. When the bulk load is done, the second copy of each partition will have the full up to date state, including changes that came in during the bulk load. The online activity can be now redirected to the second copies and the first copies can be overwritten in the background by the second copies, so as to again have all data in duplicate.</p> <p>Failures during such operations are not dangerous. If the copies doing the bulk load fail, the bulk load will have to be restarted. If the front end copies fail, the front end load goes to the copies doing the bulk load. Response times will be bad until the bulk load is stopped, but no data is lost.</p> <p>This technique applies to all data intensive background tasks — calculation of <a href="http://dbpedia.org/resource/Entity" id="link-id0x20b7a568">entity</a> search ranks, data cleansing, consistency checking, and so on. If two copies are needed to keep up with the online load, then data can be kept just as well in three copies instead of two. This method applies to any data-warehouse-style workload which must coexist with online access and occasional low volume updating.</p> <h2>Configurations of Redundancy</h2> <p>Right now, we can declare that two or more server processes in a cluster form a group. All data managed by one member of the group is stored by all others. The members of the group are interchangeable. Thus, if there is four-servers-worth of data, then there will be a minimum of eight servers. Each of these servers will have one server process per core. The first hardware failure will not affect operations. For the second failure, there is a 1/7 chance that it stops the whole system, if it falls on the server whose pair is down. If groups consist of three servers, for a total of 12, the two first failures are guaranteed not to interrupt operations; for the third, there is a 1/10 chance that it will.</p> <p>We note that for big databases, as said before, the RAM cache capacity is the sum of all the servers' RAM when in normal operation.</p> <p>There are other, more dynamic ways of splitting data among servers, so that partitions migrate between servers and spawn extra copies of themselves if not enough copies are online. The Google File System (GFS) does something of this sort at the file system level; Amazon's Dynamo does something similar at the database level. The analogies are not exact, though.</p> <p>If data is partitioned in this manner, for example into 1K slices, each in duplicate, with the rule that the two duplicates will not be on the same physical server, the first failure will not break operations but the second probably will. Without extra logic, there is a probability that the partitions formerly hosted by the failed server have their second copies randomly spread over the remaining servers. This scheme equalizes load better but is less resilient.</p> <h2>Maintenance and Continuity</h2> <p>Databases may benefit from defragmentation, rebalancing of indices, and so on. While these are possible online, by definition they affect the working set and make response times quite bad as soon as the database is significantly larger than RAM. With duplicate copies, the problem is largely solved. Also, software version changes need not involve downtime.</p> <h2>Present Status</h2> <p>The basics of replicated partitions are operational. The items to finalize are about system administration procedures and automatic synchronization of recovering copies. This must be automatic because if it is not, the operator will find a way to forget something or do some steps in the wrong order. This also requires a management view that shows what the different processes are doing and whether something is hung or failing repeatedly. All this is for the recovery part; taking failed partitions offline is easy.</p>
2009-04-01T11:18:54.000012-04:00
Web Scale and Fault Tolerance
http://www.openlinksw.com:443/weblog/oerling/?date=2009-04-01#1540
2009-04-01T15:18:06Z
<p>One concern about <a href="http://virtuoso.openlinksw.com" id="link-id0x3b82c38">Virtuoso</a> Cluster is fault tolerance. This post talks about the basics of fault tolerance and what we can do with this, from improving resilience and optimizing performance to accommodating bulk loads without impacting interactive response. We will see that this is yet another step towards a 24/7 web-scale <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x22c42e10">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x1e4f0b58">Web</a>. We will see how large scale, continuous operation, and redundancy are related.</p> <p>It has been said many times — when things are large enough, failures become frequent. In view of this, basic storage of partitions in multiple copies is built into the Virtuoso cluster from the start. Until now, this feature has not been tested or used very extensively, aside from the trivial case of keeping all schema <a href="http://dbpedia.org/resource/Information" id="link-id0x224401c0">information</a> in synchronous replicas on all servers.</p> <h2>Approaches to Fault Tolerance</h2> <p>Fault tolerance has many aspects but it starts with keeping <a href="http://dbpedia.org/resource/Data" id="link-id0x230b7500">data</a> in at least two copies. There are shared-disk cluster databases like <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0xa9a1d8d8">Oracle</a> RAC that do not depend on partitioning. With these, as long as the disk image is intact, servers can come and go. The fault tolerance of the disk in turn comes from mirroring done by the disk controller. Raids other than mirrored disk are not really good for databases because of write speed.</p> <p>With shared-nothing setups like Virtuoso, fault tolerance is based on multiple servers keeping the same logical data. The copies are synchronized transaction-by-transaction but are not bit-for-bit identical nor write-by-write synchronous as is the case with mirrored disks.</p> <p>There are asynchronous replication schemes generally based on log shipping, where the replica replays the transaction log of the master copy. The master copy gets the updates, the replica replays them. Both can take queries. These do not guarantee an entirely ACID fail-over but for many applications they come close enough.</p> <p>In a tightly coupled cluster, it is possible to do synchronous, transactional updates on multiple copies without great added cost. Sending the message to two places instead of one does not make much difference since it is the latency that counts. But once we go to wide area networks, this becomes as good as unworkable for any sort of update volume. Thus, wide area replication must in practice be asynchronous.</p> <p>This is a subject for another discussion. For now, the short answer is that wide area log shipping must be adapted to the application's requirements for synchronicity and consistency. Also, exactly what content is shipped and to where depends on the application. Some application-specific logic will likely be involved; more than this one cannot say without a specific context.</p> <h2>Basics of Partition Fail-Over</h2> <p>For now, we will be concerned with redundancy protecting against broken hardware, software slowdown, or crashes inside a single site.</p> <p>The basic idea is simple: Writes go to all copies; reads that must be repeatable or serializable (i.e., locking) go to the first copy; reads that refer to committed state without guarantee of repeatability can be balanced among all copies. When a copy goes offline, nobody needs to know, as long as there is at least one copy online for each partition. The exception in practice is when there are open cursors or such stateful things as aggregations pending on a copy that goes down. Then the query or transaction will abort and the application can retry. This looks like a deadlock to the application.</p> <p>Coming back online is more complicated. This requires establishing that the recovering copy is actually in sync. In practice this requires a short window during which no transactions have uncommitted updates. Sometimes, forcing this can require aborting some transactions, which again looks like a deadlock to the application.</p> <p>When an error is seen, such as a process no longer accepting connections and dropping existing cluster connections, we in practice go via two stages. First, the operations that directly depended on this process are aborted, as well as any computation being done on behalf of the disconnected server. At this stage, attempting to read data from the partition of the failed server will go to another copy but writes will still try to update all copies and will fail if the failed copy continues to be offline. After it is established that the failed copy will stay off for some time, writes may be re-enabled — but now having the failed copy rejoin the cluster will be more complicated, requiring an atomic window to ensure sync, as mentioned earlier.</p> <p>For the DBA, there can be intermittent software crashes where a failed server automatically restarts itself, and there can be prolonged failures where this does not happen. Both are alerts but the first kind can wait. Since a system must essentially run itself, it will wait for some time for the failed server to restart itself. During this window, all reads of the failed partition go to the spare copy and writes give an error. If the spare does not come back up in time, the system will automatically re-enable writes on the spare but now the failed server may no longer rejoin the cluster without a complex sync cycle. This all can happen in well under a minute, faster than a human operator can react. The diagnostics can be done later.</p> <p>If the situation was a hardware failure, recovery consists of taking a spare server and copying the database from the surviving online copy. This done, the spare server can come on line. Copying the database can be done while online and accepting updates but this may take some time, maybe an hour for every 200G of data copied over a network. In principle this could be automated by scripting, but we would normally expect a human DBA to be involved.</p> <p>As a general rule, reacting to the failure goes automatically without disruption of service but bringing the failed copy online will usually require some operator action.</p> <h2>Levels of Tolerance and Performance</h2> <p>The only way to make failures totally invisible is to have all in duplicate and provisioned so that the system never runs at more than half the total capacity. This is often not economical or necessary. This is why we can do better, using the spare capacity for more than standby.</p> <p>Imagine keeping a repository of linked data. Most of the content will come in through periodic bulk replacement of data sets. Some data will come in through pings from applications publishing FOAF and similar. Some data will come through on-demand RDFization of resources.</p> <p>The performance of such a repository essentially depends on having enough memory. Having this memory in duplicate is just added cost. What we can do instead is have all copies store the whole partition but when routing queries, apply range partitioning on top of the basic hash partitioning. If one partition stores IDs 64K - 128K, the next partition 128K - 192K, and so forth, and all partitions are stored in two full copies, we can route reads to the first 32K IDs to the first copy and reads to the second 32K IDs to the second copy. In this way, the copies will keep different working sets. The RAM is used to full advantage.</p> <p>Of course, if there is a failure, then the working set will degrade, but if this is not often and not for long, this can be quite tolerable. The alternate expense is buying twice as much RAM, likely meaning twice as many servers. This workload is memory intensive, thus servers should have the maximum memory they can have without going to parts that are so expensive one gets a new server for the price of doubling memory.</p> <h2>Background Bulk Processing</h2> <p>When loading data, the system is online in principle, but query response can be quite bad. A large <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x3c0cfb8">RDF</a> load will involve most memory and queries will miss the cache. The load will further keep most disks busy, so response is not good. This is the case as soon as a server's partition of the database is four times the size of RAM or greater. Whether the work is bulk-load or bulk-delete makes little difference.</p> <p>But if partitions are replicated, we can temporarily split the database so that the first copies serve queries and the second copies do the load. If the copies serving on line activities do some updates also, these updates will be committed on both copies. But the load will be committed on the second copy only. This is fully appropriate as long as the data are different. When the bulk load is done, the second copy of each partition will have the full up to date state, including changes that came in during the bulk load. The online activity can be now redirected to the second copies and the first copies can be overwritten in the background by the second copies, so as to again have all data in duplicate.</p> <p>Failures during such operations are not dangerous. If the copies doing the bulk load fail, the bulk load will have to be restarted. If the front end copies fail, the front end load goes to the copies doing the bulk load. Response times will be bad until the bulk load is stopped, but no data is lost.</p> <p>This technique applies to all data intensive background tasks — calculation of <a href="http://dbpedia.org/resource/Entity" id="link-id0x3b38ac0">entity</a> search ranks, data cleansing, consistency checking, and so on. If two copies are needed to keep up with the online load, then data can be kept just as well in three copies instead of two. This method applies to any data-warehouse-style workload which must coexist with online access and occasional low volume updating.</p> <h2>Configurations of Redundancy</h2> <p>Right now, we can declare that two or more server processes in a cluster form a group. All data managed by one member of the group is stored by all others. The members of the group are interchangeable. Thus, if there is four-servers-worth of data, then there will be a minimum of eight servers. Each of these servers will have one server process per core. The first hardware failure will not affect operations. For the second failure, there is a 1/7 chance that it stops the whole system, if it falls on the server whose pair is down. If groups consist of three servers, for a total of 12, the two first failures are guaranteed not to interrupt operations; for the third, there is a 1/10 chance that it will.</p> <p>We note that for big databases, as said before, the RAM cache capacity is the sum of all the servers' RAM when in normal operation.</p> <p>There are other, more dynamic ways of splitting data among servers, so that partitions migrate between servers and spawn extra copies of themselves if not enough copies are online. The Google File System (GFS) does something of this sort at the file system level; Amazon's Dynamo does something similar at the database level. The analogies are not exact, though.</p> <p>If data is partitioned in this manner, for example into 1K slices, each in duplicate, with the rule that the two duplicates will not be on the same physical server, the first failure will not break operations but the second probably will. Without extra logic, there is a probability that the partitions formerly hosted by the failed server have their second copies randomly spread over the remaining servers. This scheme equalizes load better but is less resilient.</p> <h2>Maintenance and Continuity</h2> <p>Databases may benefit from defragmentation, rebalancing of indices, and so on. While these are possible online, by definition they affect the working set and make response times quite bad as soon as the database is significantly larger than RAM. With duplicate copies, the problem is largely solved. Also, software version changes need not involve downtime.</p> <h2>Present Status</h2> <p>The basics of replicated partitions are operational. The items to finalize are about system administration procedures and automatic synchronization of recovering copies. This must be automatic because if it is not, the operator will find a way to forget something or do some steps in the wrong order. This also requires a management view that shows what the different processes are doing and whether something is hung or failing repeatedly. All this is for the recovery part; taking failed partitions offline is easy.</p>
2009-04-01T11:18:51.000014-04:00
Live Virtuoso instance hosting Linked Open Data (LOD) Cloud
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-03-30#1539
2009-03-30T16:27:26Z
<p>We have reached a beachead re. the <a href="http://lod.openlinksw.com" id="link-id11a035e0">Virtuoso instance hosting the Linked Open Data (LOD) Cloud</a>; meaning, we are not going to be performing any major updates and deletions short-term, bar incorporation of fresh data sets from the Freebase and <a href="http://www.bio2rdf.org/" id="link-id121d7278">Bio2RDF</a> projects (both communities a prepping new RDF data sets).</p> <p>At the current time we have loaded 100% of all the very large data sets from the <a href="http://www4.wiwiss.fu-berlin.de/bizer/pub/lod-datasets_2009-03-05.html" id="link-id1441f7e0">LOD Cloud</a>. As result, we can start the process of exposing <a href="http://dbpedia.org/resource/Linked_Data" id="link-id16c53de8">Linked Data</a> virtues in a manner that's palatable to users, developers, and database professionals across the <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id0x20165290">Web</a> 1.0, 2.0, and 3.0 spectrums.</p> <h3>What does this mean?</h3> <p>You can use the "Search & Find" or"<a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id119c6878">URI</a> Lookup" or <a href="http://dbpedia.org/resource/SPARQL" id="link-id157acde8">SPARQL</a> endpoint associated with the LOD cloud hosting instance to perform the following tasks:</p> <p> </p> <ol> <li>Find entities associated with <a href="http://dbpedia.org/resource/Full_text_search" id="link-id11a82f28">full text search</a> patterns -- Google Style, but with <a href="http://dbpedia.org/resource/Entity" id="link-id131b6380">Entity</a> & Text proximity Rank instead of Page Rank, since we are dealing with Entities rather than documents about entities</li> <li>Find and Lookup entities by Identifier (URI) -- which is helpful when locating URIs to use for identify entities in your own linked data spaces on the Web</li> <li>View entity descriptions via a variety of representation formats (HTML, <a href="http://dbpedia.org/resource/RDFa" id="link-id11e323b0">RDFa</a>, RDF/XML, N3, Turtle etc.)</li> <li>Determine uses of entity identifiers across the LOD cloud -- which helps you select preferred URIs based on usage statistics.</li> </ol> <h3>What does it offer Web 1.0 and 2.0 developers?</h3> <p> If you don't want to use the <a href="http://lod.openlinksw.com/sparql" id="link-id15c1ec30">SPARQL based Web Service</a>, or other Linked Data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id15ebd3b0">Web</a> oriented APIs for interacting with the LOD cloud programmatically, you can simply use the powerful <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtuosoFacetsWebService" id="link-id12e556a8">REST style Web Service</a> that provides <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id12138090">URL</a> parameters for performing full text oriented "Search", entity oriented "Find" queries, and faceted navigation over the huge data corpus with results data returned in JSON and XML formats.</p> <h3>Next Steps:</h3> <p> Amazon have agreed to add all the LOD Cloud data sets to their existing <a href="http://aws.amazon.com/publicdatasets" id="link-id11989aa8">public data sets collective</a>. Thus, the data sets we are loading will be available in "raw data" (RDF) format on the public data sets page via Named Elastic Block Storage (EBS) Snapshots); meaning, you can make an EC2 AMI (e.g. a Linux, Windows, Solaris) and install an RDF quad or triple store of choice into your AMI, then simply load data from the LOD cloud based on your needs.</p> <p> In addition to the above, we are also going to offer a <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtInstallationEC2" id="link-id13982a88">Virtuoso 6.0 Cluster Edition based LOD Cloud AMI</a> (as we've already done with <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtEC2AMIDBpediaInstall" id="link-id12cba108">DBpedia</a>, <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtEC2AMIMusicBrainzInstall" id="link-id1390d338">MusicBrainz</a>, <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtEC2AMINeuroCommonsInstall" id="link-id15801668">NeuroCommons</a>, and <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtEC2AMIBio2rdfInstall" id="link-id133e0840">Bio2Rdf</a>) that will enable you to simply instantiate a personal and service specific edition of Virtuoso with all the LOD data in place and fully tuned for performance and scalability; basically, you will simply press "Instantiate AMI" and a LOD cloud <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id15ccbf80">data space</a>, in true Linked Data from, will be at your disposal within minutes (i.e. the time it takes the DB to start).</p> <p>Work on the migration of the LOD data to EC2 starts this week. Thus, if you are interested in contributing an RDF based data set to the LOD cloud now is the time to get your archive links in place on the (see: <a href="http://esw.w3.org/topic/DataSetRDFDumps" id="link-id154d6f88">ESW Wiki page for LOD Data Sets</a>).</p>
2009-04-01T14:26:22.000002-04:00
Beyond Applications - Introducing the Planetary Datasphere (Part 2)
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-03-25#1538
2009-03-25T15:50:56Z
<p> <a href="http://www.openlinksw.com/weblog/oerling/?id=1535" id="link-id155e3bd0">We have looked at the general implications of the DataSphere</a>, a universal, ubiquitous database infrastructure, on end-user experience and application development and content. Now we will look at what this means at the back end, from hosting to security to server software and hardware.</p> <h2>Application Hosting</h2> <p>For the infrastructure provider, hosting the DataSphere is no different from hosting large Web 2.0 sites. This may be paid for by users, as in the cloud computing model where users rent capacity for their own purposes, or by advertisers, as in most of Web 2.0.</p> <p>Clouds play a role in this as places with high local connectivity. The DataSphere is the atmosphere; the Cloud is an atmospheric phenomenon.</p> <h2>What of Proprietary <a href="http://dbpedia.org/resource/Data" id="link-id0x10fd3e18">Data</a> and its Security?</h2> <p>Having proprietary data does not imply using a proprietary language. I would say that for any domain of discourse, no matter how private or specialized, at least some structural concepts can be borrowed from public, more generic sources. This lowers training thresholds and facilitates integration. Being able to integrate does not imply opening one's own data. To take an analogy, if you have a bunker with closed circuit air recycling, you still breathe air, even if that air is cut off from the atmosphere at large. For places with complex existing <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x13cae0b0">RDBMS</a> security, the best is to map the RDBMS to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x13deb7d8">RDF</a> on the fly, always running all requests through the RDBMS. This implicitly preserves any policy or label based security schemes.</p> <h2>What of Individual Privacy on the Open Web?</h2> <p>The more complex situations will be found in environments with mixed security needs, as in social networking with partly-open and partly-closed profiles. The FOAF+SSL solution with <code>https://</code> URIs is one approach. For query processing, we have a question of enforcing instance-level policies. In the DataSphere, granting privileges on tables and views no longer makes sense. In <a href="http://dbpedia.org/resource/SQL" id="link-id0x1211a490">SQL</a>, a policy means that behind the scenes the DBMS will add extra criteria to queries and updates depending on who is issuing them. The query processor adds conditions like getting the user's department ID and comparing it to the department ID on the payroll record. Labeled security is a scheme where data rows themselves contain security tags and the DBMS enforces these, row by row.</p> <p>I would say that these techniques are suited for highly-structured situations where the roles, compartments, and needs are clear, and where the organization has the database know-how to write, test, and deploy such rules by the table, row, and column. This does not sit well with schema-last. I would not bet much on an average developer's capacity for making airtight policies on RDF data where not even 100% schema-adherence is guaranteed.</p> <p>Doing security at the RDF graph level seems more appropriate. In many use cases, the graph is analogous to a photo album or a file system directory. A Data <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id0x13beff18">Space</a> can be divided into graphs to provide more granularity for expressing topic, provenance, or security. If policy conditions apply mostly to the graph, then things are not as likely to slip by, for example, policy rules missing some infrequent misuse of the schema. In these cases, the burden on the query processor is also not excessive: Just as with documents, the container (table, graph) is the object of access grants, not the individual sentences (DBMS records, RDF triples) in the document.</p> <p>It is left to the application to present a choice of graph level policies to the user. Exactly what these will be depends on the domain of discourse. A policy might restrict access to a meeting in a calendar to people whose OpenIDs figure in the attendee list, or limit access to a photo album to people mentioned in the owner's social network. Defining such policies is typically a task for the application developer.</p> <p>The difference between the Document Web and the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x13106cd0">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x13ca1050">Web</a> is that while the Document Web enforces security when a thing is returned to the user, Linked Data Web enforcement must occur whenever a query references something, even if this is an intermediate result not directly shown to the user.</p> <p>The DataSphere will offer a generic policy scheme, filtering what graphs are accessed in a given query situation. Other applications may then verify the safety of one's disclosed <a href="http://dbpedia.org/resource/Information" id="link-id0x13a02d60">information</a> using the same DataSphere infrastructure. Of course, the user must rely on the infrastructure provider to correctly enforce these rules. Then again, some users will operate and audit their own infrastructure anyway.</p> <h2>Federation vs. Centralization</h2> <p>On the open web, there is the question of federation vs. centralization. If an application is seen to be an interface to a vocabulary, it becomes more agnostic with respect to this. In practice, if we are talking about hosted services, what is hosted together joins much faster. Data Spaces with lots of interlinking, such as closely connected social networks, will tend to cluster together on the same cloud to facilitate joint operation. Data is ubiquitous and not location-conscious, but what one can efficiently do with it depends on location. Joint access patterns favor joint location. Due to technicalities of the matter, single database clusters will run complex queries within the cluster 100 to 1000 times faster than between clusters. The size of such data clouds may be in the hundreds-of-billions of triples. It seems to make sense to have data belonging to same-type or jointly-used applications close together. In practice, there will arise partitioning by type of usage, user profile, etc., but this is no longer airtight and applications more-or-less float on top of all of this.</p> <p>A search engine can host a copy of the Document Web and allow text lookups on it. But a text lookup is a single well-defined query that happens to parallelize and partition very well. A search engine can also have all the structured public data copied, but the problem there is that queries are a lot less predictable and may take orders of magnitude more resources than a single text lookup. As a partial answer, even now, we can set up a database so that the first million single-row joins cost the user nothing, but doing more requires a special subscription.</p> <p>The cost for hosting a trillion triples will vary radically in function of what throughput is promised. This may result in pricing per service level, a bit like ISP pricing varies in function of promised connectivity. Queries can be run for free if no throughput guarantee applies, and might cost more if the host promises at least five-million joins-per-second including infrequently-accessed data.</p> <p>Performance and cost dynamics will probably lead to the emergence of domain-specific clusters of colocated Data Spaces. The landscape will be hybrid, where usage drives data colocation. A single Google is not a practical solution to the world's spectrum of query needs.</p> <h2>What is the Cost of Schema-Last?</h2> <p>The DataSphere proposition is predicated on a worldwide database fabric that can store anything, just like a network can transport anything. It cannot enforce a fixed schema, just like TCP/IP cannot say that it will transport only email. This is continuous schema evolution. Well, TCP/IP can transport anything but it does transport a lot of HTML and email. Similarly, the DataSphere can optimize for some common vocabularies.</p> <p>We have seen that an application-specific relational schema is often 10 times more efficient than an equivalent completely generic RDF representation of the same thing. The gap may narrow, but task specific representations will keep an edge. We ought to know, as we do both.</p> <p>While anything can be represented, the masses are not that creative. For any data-hosting provider, making a specialized representation for the top 100 entities may cut data size in half or better. This is a behind-the-scenes optimization that will in time be a matter of course.</p> <p>Historically, our industry has been driven by two phenomena:</p> <ol> <li> <b>New PCs every 2 years.</b> To make this necessary, Windows has been getting bigger and bigger, and not upgrading is not an option if one must exchange documents with new data formats and keep up with security.</li> <li> <b>Agility, or <i>ad hoc</i> over planned.</b> The reason the RDBMS won over <a href="http://dbpedia.org/resource/CODASYL" id="link-id0x24ee5098">CODASYL</a> network databases was that one did not have to define what queries could be made when creating the database. With the Linked Data Web, we have one more step in this direction when we say that one does not have to decide what can be represented when creating the database.</li> </ol> <p>To summarize, there is some cost to schema-last, but then our industry needs more complexity to keep justifying constant investment. The cost is in this sense not all bad.</p> <p>Building the DataSphere may be the next great driver of server demand. As a case in point, Cisco, whose fortune was made when the network became ubiquitous, just entered the server game. It's in the air.</p> <h2>DataSphere Precursors</h2> <p>Right now, we have the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x13ea7938">Linked Open Data</a> movement with lots of new data being added. We have the drive for data- and reputation-portability. We have Freebase as a demonstrator of end-users actually producing structured data. We have convergence of terminology around <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x13ae45e8">DBpedia</a>, FOAF, SIOC, and more. We have demonstrators of useful data integration on the RDF stack in diverse fields, especially life sciences.</p> <p>We have a totally ubiquitous network for the distribution of this, plus database technology to make this work.</p> <p>We have a practical need for semantics, as search is getting saturated, email is getting killed by spam, and information overload is a constant. Social networks can be leveraged for solving a lot of this, if they can only be opened.</p> <p>Of course, there is a call for transparency in society at large. Well, the battle of transparency vs. spin is a permanent feature of human existence but even there, we cannot ignore the possibilities of open data.</p> <h2>Databases and Servers</h2> <p>Technically, what does this take? Mostly, this takes a lot of memory. The software is there and we are productizing it as we speak. As with other data intensive things, the key is scalable querying over clusters of commodity servers. Nothing we have not heard before. Of course, the DBMS must know about RDF specifics to get the right query plans and so on but this we have explained elsewhere.</p> <p>This all comes down to the cost of memory. No amount of CPU or network speed will make any difference if data is not in memory. Right now, a board with 8G and a dual core AMD X86-64 and 4 disks may cost about $700. 2 x 4 core Xeon and 16G and 8 disks may be $4000, counting just the components. In our experience, about 32G per billion triples is a minimum. This must be backed by a few independent disks so as to fill the cache in parallel. A cluster with 1 TB of RAM would be under $100K if built from low end boards.</p> <p>The workload is all about large joins across partitions. The queries parallelize well, thus using the largest and most expensive machines for building blocks is not cost efficient. Having absolutely everything in RAM is also not cost efficient, but it is necessary to have many disks to absorb the random access load. Disk access is predominantly random, unlike some analytics workloads that can read serially. If SSD's get a bit cheaper, one could have SSD for the database and disk for backup.</p> <p>With large data centers, redundancy becomes an issue. The most cost effective redundancy is simply storing partitions in duplicate or triplicate on different commodity servers. The DBMS software should handle the replication and fail-over.</p> <p>For operating such systems, scaling-on-demand is necessary. Data must move between servers, and adding or replacing servers should be an on-the-fly operation. Also, since access is essentially never uniform, the most commonly accessed partitions may benefit from being kept in more copies than less frequently accessed ones. The DBMS must be essentially self administrating since these things are quite complex and easily intractable if one does not have in depth understanding of this rather complex field.</p> <p>The best price point for hardware varies with time. Right now, the optimum is to have many basic motherboards with maximum memory in a rack unit, then another unit with local disks for each motherboard. Much cheaper than SAN's and Infiniband fabrics.</p> <h2>Conclusions and Next Steps</h2> <p>The ingredients and use cases are there. If server clusters with 1TB RAM begin under $100K, the cost of deployment is small compared to personnel costs.</p> <p>Bootstrapping the DataSphere from current Linked Open Data, such as DBpedia, <a href="http://dbpedia.org/resource/Cyc" id="link-id0x13c36da8">OpenCYC</a>, Freebase, and every sort of social network, is feasible. Aside from private data integration and analytics efforts and E-science, the use cases are liberating social networks and C2C and some aspects of search from silos, overcoming spam, and mass use of semantics extracted from text. Emergent effects will then carry the ball to places we have not yet been.</p> <p>The Linked Data Web has its origins in <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1405f0a8">Semantic Web</a> research, and many of the present participants come from these circles. Things may have been slowed down by a disconnect, only too typical of human activity, between Semantic Web research on one hand and database engineering on the other. Right now, the challenge is one of engineering. As documented on this <a href="http://dbpedia.org/resource/Blog" id="link-id0x2329f1a8">blog</a>, we have worked quite a bit on cluster databases, mostly but not exclusively with RDF use cases. The actual challenges of this are however not at all what is discussed in Semantic Web conferences. These have to do with complexities of parallelism, timing, message bottlenecks, transactions, and the like, i.e., hardcore engineering. These are difficult beyond what the casual onlooker might guess but not impossible. The details that remain to be worked out are nothing semantic, they are hardcore database, concerning automatic provisioning and such matters.</p> <p>It is as if the Semantic Web people look with envy at the Web 2.0 side where there are big deployments in production, yet they do not seem quite ready to take the step themselves. Well, I will write some other time about research and engineering. For now, the message is &mdash <i><b>go for it</b></i>. Stay tuned for more announcements, as we near production with our next generation of software.</p> <h2>Related</h2> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1535" id="link-id14e02bb0">Beyond Applications - Introducing the Planetary Datasphere (Part 1)</a> </li> <li> <a href="http://www.openlinksw.com/blog/~kidehen/?id=1442" id="link-id117dc518">Serendipitous Discovery Quotient (SDQ)</a> </li> <li> <a href="http://www.openlinksw.com/blog/~kidehen/?id=1534" id="link-id15c52410">How Linked Data will change Advertising</a> </li> <li> <a href="http://www.openlinksw.com/blog/~kidehen/?id=1519" id="link-id11e93658">The Time for RDBMS Primacy Downgrade is Nigh!</a> </li> <li> <a href="http://www.openlinksw.com/blog/~kidehen/?tag=DataSpace" id="link-id1491a588">Data Spaces</a> </li> </ul>
2009-03-25T12:31:58.000001-04:00
Beyond Applications - Introducing the Planetary Datasphere (Part 2)
http://www.openlinksw.com:443/weblog/oerling/?date=2009-03-25#1537
2009-03-25T15:50:56Z
<p> <a href="http://www.openlinksw.com/weblog/oerling/?id=1535" id="link-id155e3bd0">We have looked at the general implications of the DataSphere</a>, a universal, ubiquitous database infrastructure, on end-user experience and application development and content. Now we will look at what this means at the back end, from hosting to security to server software and hardware.</p> <h2>Application Hosting</h2> <p>For the infrastructure provider, hosting the DataSphere is no different from hosting large Web 2.0 sites. This may be paid for by users, as in the cloud computing model where users rent capacity for their own purposes, or by advertisers, as in most of Web 2.0.</p> <p>Clouds play a role in this as places with high local connectivity. The DataSphere is the atmosphere; the Cloud is an atmospheric phenomenon.</p> <h2>What of Proprietary <a href="http://dbpedia.org/resource/Data" id="link-id0x13b5b4a0">Data</a> and its Security?</h2> <p>Having proprietary data does not imply using a proprietary language. I would say that for any domain of discourse, no matter how private or specialized, at least some structural concepts can be borrowed from public, more generic sources. This lowers training thresholds and facilitates integration. Being able to integrate does not imply opening one's own data. To take an analogy, if you have a bunker with closed circuit air recycling, you still breathe air, even if that air is cut off from the atmosphere at large. For places with complex existing <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x24db80e0">RDBMS</a> security, the best is to map the RDBMS to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x24ea7c40">RDF</a> on the fly, always running all requests through the RDBMS. This implicitly preserves any policy or label based security schemes.</p> <h2>What of Individual Privacy on the Open Web?</h2> <p>The more complex situations will be found in environments with mixed security needs, as in social networking with partly-open and partly-closed profiles. The FOAF+SSL solution with <code>https://</code> URIs is one approach. For query processing, we have a question of enforcing instance-level policies. In the DataSphere, granting privileges on tables and views no longer makes sense. In <a href="http://dbpedia.org/resource/SQL" id="link-id0x24aaccc0">SQL</a>, a policy means that behind the scenes the DBMS will add extra criteria to queries and updates depending on who is issuing them. The query processor adds conditions like getting the user's department ID and comparing it to the department ID on the payroll record. Labeled security is a scheme where data rows themselves contain security tags and the DBMS enforces these, row by row.</p> <p>I would say that these techniques are suited for highly-structured situations where the roles, compartments, and needs are clear, and where the organization has the database know-how to write, test, and deploy such rules by the table, row, and column. This does not sit well with schema-last. I would not bet much on an average developer's capacity for making airtight policies on RDF data where not even 100% schema-adherence is guaranteed.</p> <p>Doing security at the RDF graph level seems more appropriate. In many use cases, the graph is analogous to a photo album or a file system directory. A Data <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id0x2396c058">Space</a> can be divided into graphs to provide more granularity for expressing topic, provenance, or security. If policy conditions apply mostly to the graph, then things are not as likely to slip by, for example, policy rules missing some infrequent misuse of the schema. In these cases, the burden on the query processor is also not excessive: Just as with documents, the container (table, graph) is the object of access grants, not the individual sentences (DBMS records, RDF triples) in the document.</p> <p>It is left to the application to present a choice of graph level policies to the user. Exactly what these will be depends on the domain of discourse. A policy might restrict access to a meeting in a calendar to people whose OpenIDs figure in the attendee list, or limit access to a photo album to people mentioned in the owner's social network. Defining such policies is typically a task for the application developer.</p> <p>The difference between the Document Web and the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x238a0098">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x23882280">Web</a> is that while the Document Web enforces security when a thing is returned to the user, Linked Data Web enforcement must occur whenever a query references something, even if this is an intermediate result not directly shown to the user.</p> <p>The DataSphere will offer a generic policy scheme, filtering what graphs are accessed in a given query situation. Other applications may then verify the safety of one's disclosed <a href="http://dbpedia.org/resource/Information" id="link-id0x2388e458">information</a> using the same DataSphere infrastructure. Of course, the user must rely on the infrastructure provider to correctly enforce these rules. Then again, some users will operate and audit their own infrastructure anyway.</p> <h2>Federation vs. Centralization</h2> <p>On the open web, there is the question of federation vs. centralization. If an application is seen to be an interface to a vocabulary, it becomes more agnostic with respect to this. In practice, if we are talking about hosted services, what is hosted together joins much faster. Data Spaces with lots of interlinking, such as closely connected social networks, will tend to cluster together on the same cloud to facilitate joint operation. Data is ubiquitous and not location-conscious, but what one can efficiently do with it depends on location. Joint access patterns favor joint location. Due to technicalities of the matter, single database clusters will run complex queries within the cluster 100 to 1000 times faster than between clusters. The size of such data clouds may be in the hundreds-of-billions of triples. It seems to make sense to have data belonging to same-type or jointly-used applications close together. In practice, there will arise partitioning by type of usage, user profile, etc., but this is no longer airtight and applications more-or-less float on top of all of this.</p> <p>A search engine can host a copy of the Document Web and allow text lookups on it. But a text lookup is a single well-defined query that happens to parallelize and partition very well. A search engine can also have all the structured public data copied, but the problem there is that queries are a lot less predictable and may take orders of magnitude more resources than a single text lookup. As a partial answer, even now, we can set up a database so that the first million single-row joins cost the user nothing, but doing more requires a special subscription.</p> <p>The cost for hosting a trillion triples will vary radically in function of what throughput is promised. This may result in pricing per service level, a bit like ISP pricing varies in function of promised connectivity. Queries can be run for free if no throughput guarantee applies, and might cost more if the host promises at least five-million joins-per-second including infrequently-accessed data.</p> <p>Performance and cost dynamics will probably lead to the emergence of domain-specific clusters of colocated Data Spaces. The landscape will be hybrid, where usage drives data colocation. A single Google is not a practical solution to the world's spectrum of query needs.</p> <h2>What is the Cost of Schema-Last?</h2> <p>The DataSphere proposition is predicated on a worldwide database fabric that can store anything, just like a network can transport anything. It cannot enforce a fixed schema, just like TCP/IP cannot say that it will transport only email. This is continuous schema evolution. Well, TCP/IP can transport anything but it does transport a lot of HTML and email. Similarly, the DataSphere can optimize for some common vocabularies.</p> <p>We have seen that an application-specific relational schema is often 10 times more efficient than an equivalent completely generic RDF representation of the same thing. The gap may narrow, but task specific representations will keep an edge. We ought to know, as we do both.</p> <p>While anything can be represented, the masses are not that creative. For any data-hosting provider, making a specialized representation for the top 100 entities may cut data size in half or better. This is a behind-the-scenes optimization that will in time be a matter of course.</p> <p>Historically, our industry has been driven by two phenomena:</p> <ol> <li> <b>New PCs every 2 years.</b> To make this necessary, Windows has been getting bigger and bigger, and not upgrading is not an option if one must exchange documents with new data formats and keep up with security.</li> <li> <b>Agility, or <i>ad hoc</i> over planned.</b> The reason the RDBMS won over <a href="http://dbpedia.org/resource/CODASYL" id="link-id0x13b23460">CODASYL</a> network databases was that one did not have to define what queries could be made when creating the database. With the Linked Data Web, we have one more step in this direction when we say that one does not have to decide what can be represented when creating the database.</li> </ol> <p>To summarize, there is some cost to schema-last, but then our industry needs more complexity to keep justifying constant investment. The cost is in this sense not all bad.</p> <p>Building the DataSphere may be the next great driver of server demand. As a case in point, Cisco, whose fortune was made when the network became ubiquitous, just entered the server game. It's in the air.</p> <h2>DataSphere Precursors</h2> <p>Right now, we have the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x236a9be8">Linked Open Data</a> movement with lots of new data being added. We have the drive for data- and reputation-portability. We have Freebase as a demonstrator of end-users actually producing structured data. We have convergence of terminology around <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x24db8350">DBpedia</a>, FOAF, SIOC, and more. We have demonstrators of useful data integration on the RDF stack in diverse fields, especially life sciences.</p> <p>We have a totally ubiquitous network for the distribution of this, plus database technology to make this work.</p> <p>We have a practical need for semantics, as search is getting saturated, email is getting killed by spam, and information overload is a constant. Social networks can be leveraged for solving a lot of this, if they can only be opened.</p> <p>Of course, there is a call for transparency in society at large. Well, the battle of transparency vs. spin is a permanent feature of human existence but even there, we cannot ignore the possibilities of open data.</p> <h2>Databases and Servers</h2> <p>Technically, what does this take? Mostly, this takes a lot of memory. The software is there and we are productizing it as we speak. As with other data intensive things, the key is scalable querying over clusters of commodity servers. Nothing we have not heard before. Of course, the DBMS must know about RDF specifics to get the right query plans and so on but this we have explained elsewhere.</p> <p>This all comes down to the cost of memory. No amount of CPU or network speed will make any difference if data is not in memory. Right now, a board with 8G and a dual core AMD X86-64 and 4 disks may cost about $700. 2 x 4 core Xeon and 16G and 8 disks may be $4000, counting just the components. In our experience, about 32G per billion triples is a minimum. This must be backed by a few independent disks so as to fill the cache in parallel. A cluster with 1 TB of RAM would be under $100K if built from low end boards.</p> <p>The workload is all about large joins across partitions. The queries parallelize well, thus using the largest and most expensive machines for building blocks is not cost efficient. Having absolutely everything in RAM is also not cost efficient, but it is necessary to have many disks to absorb the random access load. Disk access is predominantly random, unlike some analytics workloads that can read serially. If SSD's get a bit cheaper, one could have SSD for the database and disk for backup.</p> <p>With large data centers, redundancy becomes an issue. The most cost effective redundancy is simply storing partitions in duplicate or triplicate on different commodity servers. The DBMS software should handle the replication and fail-over.</p> <p>For operating such systems, scaling-on-demand is necessary. Data must move between servers, and adding or replacing servers should be an on-the-fly operation. Also, since access is essentially never uniform, the most commonly accessed partitions may benefit from being kept in more copies than less frequently accessed ones. The DBMS must be essentially self administrating since these things are quite complex and easily intractable if one does not have in depth understanding of this rather complex field.</p> <p>The best price point for hardware varies with time. Right now, the optimum is to have many basic motherboards with maximum memory in a rack unit, then another unit with local disks for each motherboard. Much cheaper than SAN's and Infiniband fabrics.</p> <h2>Conclusions and Next Steps</h2> <p>The ingredients and use cases are there. If server clusters with 1TB RAM begin under $100K, the cost of deployment is small compared to personnel costs.</p> <p>Bootstrapping the DataSphere from current Linked Open Data, such as DBpedia, <a href="http://dbpedia.org/resource/Cyc" id="link-id0x2396a038">OpenCYC</a>, Freebase, and every sort of social network, is feasible. Aside from private data integration and analytics efforts and E-science, the use cases are liberating social networks and C2C and some aspects of search from silos, overcoming spam, and mass use of semantics extracted from text. Emergent effects will then carry the ball to places we have not yet been.</p> <p>The Linked Data Web has its origins in <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x13ea7110">Semantic Web</a> research, and many of the present participants come from these circles. Things may have been slowed down by a disconnect, only too typical of human activity, between Semantic Web research on one hand and database engineering on the other. Right now, the challenge is one of engineering. As documented on this <a href="http://dbpedia.org/resource/Blog" id="link-id0x2388e368">blog</a>, we have worked quite a bit on cluster databases, mostly but not exclusively with RDF use cases. The actual challenges of this are however not at all what is discussed in Semantic Web conferences. These have to do with complexities of parallelism, timing, message bottlenecks, transactions, and the like, i.e., hardcore engineering. These are difficult beyond what the casual onlooker might guess but not impossible. The details that remain to be worked out are nothing semantic, they are hardcore database, concerning automatic provisioning and such matters.</p> <p>It is as if the Semantic Web people look with envy at the Web 2.0 side where there are big deployments in production, yet they do not seem quite ready to take the step themselves. Well, I will write some other time about research and engineering. For now, the message is &mdash <i><b>go for it</b></i>. Stay tuned for more announcements, as we near production with our next generation of software.</p> <h2>Related</h2> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1535" id="link-id14e02bb0">Beyond Applications - Introducing the Planetary Datasphere (Part 1)</a> </li> <li> <a href="http://www.openlinksw.com/blog/~kidehen/?id=1442" id="link-id117dc518">Serendipitous Discovery Quotient (SDQ)</a> </li> <li> <a href="http://www.openlinksw.com/blog/~kidehen/?id=1534" id="link-id15c52410">How Linked Data will change Advertising</a> </li> <li> <a href="http://www.openlinksw.com/blog/~kidehen/?id=1519" id="link-id11e93658">The Time for RDBMS Primacy Downgrade is Nigh!</a> </li> <li> <a href="http://www.openlinksw.com/blog/~kidehen/?tag=DataSpace" id="link-id1491a588">Data Spaces</a> </li> </ul>
2009-03-25T12:31:55-04:00
Beyond Applications - Introducing the Planetary Datasphere (Part 1)
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-03-24#1536
2009-03-24T14:38:57Z
<p>This is the first in a short series of <a href="http://dbpedia.org/resource/Blog" id="link-id0x12685210">blog</a> posts about what becomes possible when essentially unlimited <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x13112bc0">linked data</a> can be deployed on the open web and private intranets.</p> <p>The term <i>DataSphere</i> comes from Dan Simmons' <i><a href="http://dbpedia.org/resource/Hyperion_Cantos" id="link-id12ad4718">Hyperion</a></i> science fiction series, where it is a sort of pervasive computing capability that plays host to all sorts of processes, including what people do on the <a href="http://dbpedia.org/resource/.NET_Framework" id="link-id0x13398e38">net</a> today, and then some. I use this term here in order to emphasize the blurring of silo and application boundaries. The network is not only the computer but also the database. I will look at what effects the birth of a sort of linked data stratum can have on end-user experience, application development, application deployment and hosting, business models and advertising, and security; how cloud computing fits in; and how back-end software such as databases must evolve to support all of these.</p> <p>This is a mid-term vision. The components are coming into production as we speak, but the end result is not here quite yet.</p> <p>I use the word <i>DataSphere</i> to refer to a worldwide database fabric, a global Distributed DBMS collective, within which there are many <a href="http://dbpedia.org/resource/Data" id="link-id0x13209728">Data</a> Spaces, or Named Data Spaces. A <i>Data <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id0x137a0dc0">Space</a></i> is essentially a person's or organization's contribution to the DataSphere. I use <i>Linked Data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x23df2450">Web</a></i> to refer to component technologies and practices such as <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1286f7a0">RDF</a>, <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x132e70d0">SPARQL</a>, Linked Data practices, etc. The DataSphere does not have to be built on this technology stack <i>per se</i>, but this stack is still the best bet for it.</p> <h2>General</h2> <p>There exist applications for performing specialized functions such as social networking, shopping, document search, and C2C commerce at planetary scale. All these applications run on their own databases, each with a task specific schema. They communicate by web pages and by predefined messages for diverse application-specific transactions and reports.</p> <p>These silos are scalable because in general their data has some natural partitioning, and because the set of transactions is predetermined and the data structure is set up for this.</p> <p>The Linked Data Web proposes to create a data infrastructure that can hold anything, just like a network can transport anything. This is not a network with a memory of messages, but a whole that can answer arbitrary questions about what has been said. The prerequisite is that the questions are phrased in a vocabulary that is compatible with the vocabulary in which the statements themselves were made.</p> <p>In this setting, the vocabulary takes the place of the application. Of course, there continues to be a procedural element to applications; this has the function of translating statements between the domain vocabulary and a user interface. Examples are data import from existing applications, running predefined reports, composing new reports, and translating between natural language and the domain vocabulary.</p> <p>The big difference is that the database moves outside of the silo, at least in logical terms. The database will be like the network — horizontal and ubiquitous. The equivalent of TCP/IP will be the RDF/SPARQL combination. The equivalent of routing protocols between ISPs will be gateways between the specific DBMS engines supporting the services.</p> <h2>The place of the DBMS in the stack changes</h2> <p>The <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1320b9b0">RDBMS</a> in itself is eternal, or at least as eternal as a culture with heavy reliance on written records is. Any such culture will invent the RDBMS and use it where it best fits. We are not replacing this; we are building an abstracted worldwide data layer. This is to the RDBMS supporting line-of-business applications what the www was to enterprise content management systems.</p> <p>For transactions, the Web 2.0-style application-specific messages are fine. Also, any transactional system that must be audited must physically reside somewhere, have physical security, etc. It can't just be somewhere in the DataSphere, managed by some system with which one has no contract, just like Google's web page cache can't be relied on as a permanent repository of web content.</p> <p>Providing space on the Linked Data Web is like providing hosting on the Document Web. This may have varying service levels, pricing models, etc. The value of a queriable DataSphere is that a new application does not have to begin by building its own schema, database infrastructure, service hosting, etc. The application becomes more like a language <a href="http://dbpedia.org/resource/Meme" id="link-id0x13032ec8">meme</a>, a cultural form of interaction mediated by a relatively lightweight user-facing component, laterally open for unforeseen interaction with other applications from other domains of discourse.</p> <h2>End User Benefits</h2> <p>For the end user, the web will still look like a place where one can shop, discuss, date, whatever. These activities will be mediated by user interfaces as they are now. Right now, the end user's web presence is his/her blog or web site, and their contributions to diverse wikis, social web sites, and so forth. These are scattered. The user's Data Space is the collection of all these things, now presented in a queriable form. The user's Data Space is the user's statement of presence, referencing the diverse contributions of the user on diverse sites.</p> <p>The personal Data Space being a queriable, structured whole facilitates finding and being found, which is what brings individuals to the web in the first place. The best applications and sites are those which make this the easiest. The Linked Data Web allows saying what one wishes in a structured, queriable manner, across all application domains, independently of domain specific silos. The end user's interaction with the personal data space is through applications, like now. But these applications are just wrappers on top of self describing data, represented in domain specific vocabularies; one vocabulary is used for social networking, another for C2C commerce, and so on. The user is the master of their personal Data Space, free to take it where he or she wishes.</p> <p>Further benefits will include more ready referencing between these spaces, more uniform identity management, cross-application operations, and the emergence of "meta-applications," i.e., unified interfaces for managing many related applications/tasks.</p> <p>Of course, there is the increase in semantic richness, such as better contextuality derived from <a href="http://dbpedia.org/resource/Entity" id="link-id0x12b84120">entity</a> extraction from text. But this is also possible in a silo. The Linked Data Web angle is the sharing of identifiers for real world entities, which makes extracts of different sources by different parties potentially joinable. The user interaction will hardly ever be with the raw data. But the raw data being still at hand makes for better targeting of advertisements, better offering of related services, easier discovery of related content, and less noise overall.</p> <p> <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id0x136040d8">Kingsley Idehen</a> has coined the term <a href="http://www.openlinksw.com/blog/~kidehen/?id=1442" id="link-id0x12eadd10">SDQ</a>, for <a href="http://www.openlinksw.com/blog/~kidehen/?id=1442" id="link-id0x137014f8">Serendipitous Discovery Quotient</a>, to denote this. When applications expose explicit semantics, constructing a user experience that combines relevant data from many sources, including applications as well as highly targeted advertising, becomes natural. It is no longer a matter of "mashing up" web service interfaces with procedural code, but of "meshing" data through declarative queries across application spaces.</p> <h2>Applications in the DataSphere</h2> <p>The workflows supported by the DataSphere are essentially those taking place on the web now. The DataSphere dimension is expressed by bookmarklets, browser plugins, and the like, with ready access to related data and actions that are relevant for this data. Actions triggered by data can be anything from posting a comment to making an e-commerce purchase. Web 2.0 models fit right in.</p> <p>Web application development now consists of designing an application-specific database schema and writing web pages to interact with this schema. In the DataSphere, the database is abstracted away, as is a large part of the schema. The application floats on a sea of data instead of being tied to its own specific store and schema. Some local transaction data should still be handled in the old way, though.</p> <p>For the application developer, the question becomes one of vocabulary choice. How will the application synthesize URIs from the user interaction? Which URIs will be used, since pretty much anything will in practice have many names (e.g., <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x1337e5f0">DBpedia</a> Vs. Freebase identifiers). The end user will generally have no idea of this choice, nor of the various degrees of normalization, etc., in the vocabularies. Still, usage of such applications will produce data using some identifiers and vocabularies. Benefits of ready joining without translation will drive adoption. A vocabulary with instance data will get more instance data.</p> <p>The Linked Data Web infrastructure itself must support vocabulary and identifier choice by answering questions about who uses a particular identifier and where. Even now, we offer entity ranks and resolution of synonyms, queries on what graphs mention a certain identifier and so on. This is a means of finding the most commonly used term for each situation. Convergence of terminology cuts down on translation and makes for easier and more efficient querying.</p> <h2>Advertising</h2> <p>The application developer is, for purposes of advertising, in the position of the inventory owner, just like a traditional publisher, whether web or other. But with smarter data, it is not a matter of static keywords but of the semantically explicit data behind each individual user impression driving the ads. Data itself carries no ads but the user impression will still go through a display layer that can show ads. If the application relies on reuse of licensed content, such as media, then the content provider may get a cut of the ad revenue even if it is not the direct owner of the inventory. The specifics of implementing and enforcing this are to be worked out.</p> <h2>Content Providers, License, and Attribution</h2> <p>For the content provider, the <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x116b9ae8">URI</a> is the brand carrier. If the data is well linked and queriable, this will drive usage and traffic to the services of the content provider. This is true of any provider, whether a media publisher, e-commerce business, government agency, or anything else.</p> <p>Intellectual property considerations will make the URI a first class citizen. Just like the URI is a part of the document web experience, it is a part of the Linked Data Web experience. Just like Creative Commons licenses allow the licensor to define what type of attribution is required, a data publisher can mandate that a user experience mediated by whatever application should expose the source as a dereferenceable URI. </p> <p>One element of data dereferencing must be linking to applications that facilitate human interaction with the data. A generic data browser is a developer tool; the end user experience must still be mediated by interfaces tailored to the domain. This layer can take care of making the brand visible and can show advertising or be monetized on a usage basis.</p> <p>Next we will look at the service provider and infrastructure side of this.</p> <h2>Related</h2> <ul> <li> <a href="http://www.openlinksw.com/blog/~kidehen/?id=1442" id="link-id148ea4e0">Serendipitous Discovery Quotient (SDQ)</a> </li> <li> <a href="http://www.openlinksw.com/blog/~kidehen/?id=1534" id="link-id14b07f88">How Linked Data will change Advertising</a> </li> <li> <a href="http://www.openlinksw.com/blog/~kidehen/?id=1519" id="link-id117c6608">The Time for RDBMS Primacy Downgrade is Nigh!</a> </li> <li> <a href="http://www.openlinksw.com/blog/~kidehen/?tag=DataSpace" id="link-id154e1d58">Data Spaces</a> </li> </ul>
2009-03-24T10:50:16-04:00
Beyond Applications - Introducing the Planetary Datasphere (Part 1)
http://www.openlinksw.com:443/weblog/oerling/?date=2009-03-24#1535
2009-03-24T14:38:57Z
<p>This is the first in a short series of <a href="http://dbpedia.org/resource/Blog" id="link-id0x12c91d60">blog</a> posts about what becomes possible when essentially unlimited <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x2375f488">linked data</a> can be deployed on the open web and private intranets.</p> <p>The term <i>DataSphere</i> comes from Dan Simmons' <i><a href="http://dbpedia.org/resource/Hyperion_Cantos" id="link-id12ad4718">Hyperion</a></i> science fiction series, where it is a sort of pervasive computing capability that plays host to all sorts of processes, including what people do on the <a href="http://dbpedia.org/resource/.NET_Framework" id="link-id0x13084f08">net</a> today, and then some. I use this term here in order to emphasize the blurring of silo and application boundaries. The network is not only the computer but also the database. I will look at what effects the birth of a sort of linked data stratum can have on end-user experience, application development, application deployment and hosting, business models and advertising, and security; how cloud computing fits in; and how back-end software such as databases must evolve to support all of these.</p> <p>This is a mid-term vision. The components are coming into production as we speak, but the end result is not here quite yet.</p> <p>I use the word <i>DataSphere</i> to refer to a worldwide database fabric, a global Distributed DBMS collective, within which there are many <a href="http://dbpedia.org/resource/Data" id="link-id0x2504fff8">Data</a> Spaces, or Named Data Spaces. A <i>Data <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id0x81175fa0">Space</a></i> is essentially a person's or organization's contribution to the DataSphere. I use <i>Linked Data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x70f4e190">Web</a></i> to refer to component technologies and practices such as <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x3a5ddcd8">RDF</a>, <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x23b049e0">SPARQL</a>, Linked Data practices, etc. The DataSphere does not have to be built on this technology stack <i>per se</i>, but this stack is still the best bet for it.</p> <h2>General</h2> <p>There exist applications for performing specialized functions such as social networking, shopping, document search, and C2C commerce at planetary scale. All these applications run on their own databases, each with a task specific schema. They communicate by web pages and by predefined messages for diverse application-specific transactions and reports.</p> <p>These silos are scalable because in general their data has some natural partitioning, and because the set of transactions is predetermined and the data structure is set up for this.</p> <p>The Linked Data Web proposes to create a data infrastructure that can hold anything, just like a network can transport anything. This is not a network with a memory of messages, but a whole that can answer arbitrary questions about what has been said. The prerequisite is that the questions are phrased in a vocabulary that is compatible with the vocabulary in which the statements themselves were made.</p> <p>In this setting, the vocabulary takes the place of the application. Of course, there continues to be a procedural element to applications; this has the function of translating statements between the domain vocabulary and a user interface. Examples are data import from existing applications, running predefined reports, composing new reports, and translating between natural language and the domain vocabulary.</p> <p>The big difference is that the database moves outside of the silo, at least in logical terms. The database will be like the network — horizontal and ubiquitous. The equivalent of TCP/IP will be the RDF/SPARQL combination. The equivalent of routing protocols between ISPs will be gateways between the specific DBMS engines supporting the services.</p> <h2>The place of the DBMS in the stack changes</h2> <p>The <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x10082590">RDBMS</a> in itself is eternal, or at least as eternal as a culture with heavy reliance on written records is. Any such culture will invent the RDBMS and use it where it best fits. We are not replacing this; we are building an abstracted worldwide data layer. This is to the RDBMS supporting line-of-business applications what the www was to enterprise content management systems.</p> <p>For transactions, the Web 2.0-style application-specific messages are fine. Also, any transactional system that must be audited must physically reside somewhere, have physical security, etc. It can't just be somewhere in the DataSphere, managed by some system with which one has no contract, just like Google's web page cache can't be relied on as a permanent repository of web content.</p> <p>Providing space on the Linked Data Web is like providing hosting on the Document Web. This may have varying service levels, pricing models, etc. The value of a queriable DataSphere is that a new application does not have to begin by building its own schema, database infrastructure, service hosting, etc. The application becomes more like a language <a href="http://dbpedia.org/resource/Meme" id="link-id0x23c85e68">meme</a>, a cultural form of interaction mediated by a relatively lightweight user-facing component, laterally open for unforeseen interaction with other applications from other domains of discourse.</p> <h2>End User Benefits</h2> <p>For the end user, the web will still look like a place where one can shop, discuss, date, whatever. These activities will be mediated by user interfaces as they are now. Right now, the end user's web presence is his/her blog or web site, and their contributions to diverse wikis, social web sites, and so forth. These are scattered. The user's Data Space is the collection of all these things, now presented in a queriable form. The user's Data Space is the user's statement of presence, referencing the diverse contributions of the user on diverse sites.</p> <p>The personal Data Space being a queriable, structured whole facilitates finding and being found, which is what brings individuals to the web in the first place. The best applications and sites are those which make this the easiest. The Linked Data Web allows saying what one wishes in a structured, queriable manner, across all application domains, independently of domain specific silos. The end user's interaction with the personal data space is through applications, like now. But these applications are just wrappers on top of self describing data, represented in domain specific vocabularies; one vocabulary is used for social networking, another for C2C commerce, and so on. The user is the master of their personal Data Space, free to take it where he or she wishes.</p> <p>Further benefits will include more ready referencing between these spaces, more uniform identity management, cross-application operations, and the emergence of "meta-applications," i.e., unified interfaces for managing many related applications/tasks.</p> <p>Of course, there is the increase in semantic richness, such as better contextuality derived from <a href="http://dbpedia.org/resource/Entity" id="link-id0x23904698">entity</a> extraction from text. But this is also possible in a silo. The Linked Data Web angle is the sharing of identifiers for real world entities, which makes extracts of different sources by different parties potentially joinable. The user interaction will hardly ever be with the raw data. But the raw data being still at hand makes for better targeting of advertisements, better offering of related services, easier discovery of related content, and less noise overall.</p> <p> <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id0x37342a60">Kingsley Idehen</a> has coined the term <a href="http://www.openlinksw.com/blog/~kidehen/?id=1442" id="link-id0x3a56e4e8">SDQ</a>, for <a href="http://www.openlinksw.com/blog/~kidehen/?id=1442" id="link-id0x23649b70">Serendipitous Discovery Quotient</a>, to denote this. When applications expose explicit semantics, constructing a user experience that combines relevant data from many sources, including applications as well as highly targeted advertising, becomes natural. It is no longer a matter of "mashing up" web service interfaces with procedural code, but of "meshing" data through declarative queries across application spaces.</p> <h2>Applications in the DataSphere</h2> <p>The workflows supported by the DataSphere are essentially those taking place on the web now. The DataSphere dimension is expressed by bookmarklets, browser plugins, and the like, with ready access to related data and actions that are relevant for this data. Actions triggered by data can be anything from posting a comment to making an e-commerce purchase. Web 2.0 models fit right in.</p> <p>Web application development now consists of designing an application-specific database schema and writing web pages to interact with this schema. In the DataSphere, the database is abstracted away, as is a large part of the schema. The application floats on a sea of data instead of being tied to its own specific store and schema. Some local transaction data should still be handled in the old way, though.</p> <p>For the application developer, the question becomes one of vocabulary choice. How will the application synthesize URIs from the user interaction? Which URIs will be used, since pretty much anything will in practice have many names (e.g., <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x2364eae8">DBpedia</a> Vs. Freebase identifiers). The end user will generally have no idea of this choice, nor of the various degrees of normalization, etc., in the vocabularies. Still, usage of such applications will produce data using some identifiers and vocabularies. Benefits of ready joining without translation will drive adoption. A vocabulary with instance data will get more instance data.</p> <p>The Linked Data Web infrastructure itself must support vocabulary and identifier choice by answering questions about who uses a particular identifier and where. Even now, we offer entity ranks and resolution of synonyms, queries on what graphs mention a certain identifier and so on. This is a means of finding the most commonly used term for each situation. Convergence of terminology cuts down on translation and makes for easier and more efficient querying.</p> <h2>Advertising</h2> <p>The application developer is, for purposes of advertising, in the position of the inventory owner, just like a traditional publisher, whether web or other. But with smarter data, it is not a matter of static keywords but of the semantically explicit data behind each individual user impression driving the ads. Data itself carries no ads but the user impression will still go through a display layer that can show ads. If the application relies on reuse of licensed content, such as media, then the content provider may get a cut of the ad revenue even if it is not the direct owner of the inventory. The specifics of implementing and enforcing this are to be worked out.</p> <h2>Content Providers, License, and Attribution</h2> <p>For the content provider, the <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0xa9abc2f8">URI</a> is the brand carrier. If the data is well linked and queriable, this will drive usage and traffic to the services of the content provider. This is true of any provider, whether a media publisher, e-commerce business, government agency, or anything else.</p> <p>Intellectual property considerations will make the URI a first class citizen. Just like the URI is a part of the document web experience, it is a part of the Linked Data Web experience. Just like Creative Commons licenses allow the licensor to define what type of attribution is required, a data publisher can mandate that a user experience mediated by whatever application should expose the source as a dereferenceable URI. </p> <p>One element of data dereferencing must be linking to applications that facilitate human interaction with the data. A generic data browser is a developer tool; the end user experience must still be mediated by interfaces tailored to the domain. This layer can take care of making the brand visible and can show advertising or be monetized on a usage basis.</p> <p>Next we will look at the service provider and infrastructure side of this.</p> <h2>Related</h2> <ul> <li> <a href="http://www.openlinksw.com/blog/~kidehen/?id=1442" id="link-id148ea4e0">Serendipitous Discovery Quotient (SDQ)</a> </li> <li> <a href="http://www.openlinksw.com/blog/~kidehen/?id=1534" id="link-id14b07f88">How Linked Data will change Advertising</a> </li> <li> <a href="http://www.openlinksw.com/blog/~kidehen/?id=1519" id="link-id117c6608">The Time for RDBMS Primacy Downgrade is Nigh!</a> </li> <li> <a href="http://www.openlinksw.com/blog/~kidehen/?tag=DataSpace" id="link-id154e1d58">Data Spaces</a> </li> </ul>
2009-03-24T10:50:13-04:00
How Linked Data will change Advertising
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-03-22#1534
2009-03-23T04:39:49Z
<p>This post is a reply to <a href="http://www.jasonkolb.com/" id="link-id11f11e90">Jason Kolb</a>'s post titled: <a href="http://www.jasonkolb.com/weblog/2009/03/using-advertising-to-take-over-the-world.html" id="link-id15528ae8">Using Advertising to Take Over the World</a>. Jason's post is a response to <a href="http://scobleizer.com/" id="link-id11a41fd0">Robert Scoble</a>'s post titled: <a href="http://scobleizer.com/2009/03/21/why-facebook-has-never-listened-and-why-it-definitely-wont-start-now/" id="link-id143e2d88">Why Facebook has never listened and why it definitely won’t start now.</a> </p> <p>Jason:</p> <p>Scoble is sensing what comes next, but in my opinion, describes it using an old obtrusive advertising model anecdote.</p> <p>I've penned a post or two about the "<a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1458" id="link-id15247e90">Magic of You</a>" which is all about the new <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id0x20b2da18">Web</a> power broker (<a href="http://dbpedia.org/resource/Entity" id="link-id15552ba0">Entity</a>: "You").</p> <p>Personally, I've long envisaged a complete overhaul of advertising where obtrusive advertising simply withers away; ultimately replaced by an unobtrusive model that is driven by individualized relevance and high doses of serendipity. Basically, this is ultimately about "taking the Ad out of item placement in Web pages".</p> <p>The fundamental ingredients of an unobtrusive advertising landscape would include the following Human facts:</p> <ol> <li>We are social beings and need stuff from time to time </li> <li>We know what we need and would like to "Find stuff" when we are in "I Need Stuff" mode.</li> </ol> <p>Ideally, we would like to be able to simply state the following, via a Web accessible profile:</p> <ol> <li> Here are my "Wants" or "Needs" (my Wish-List) </li> <li> Here are the products and services that I "Offer" (my Offer-List).</li> </ol> <p>Now put the above into the <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id157388c8">context</a> of an evolving Web where <a href="http://dbpedia.org/resource/Data" id="link-id0x226b34d0">data</a> items are becoming more visible by the second, courtesy of the "<a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id11ab8f80">Linked Data" meme</a>. Thus, things that weren't discernable via the Web: "People", "Places", "Music", "Books", "Products", etc., become much easier to identify and describe.</p> <p>Assuming the comments above hold true re. the Web's evolution into a collection of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11bf4830">Linked Data</a> Spaces, and the following occur:</p> <ol> <li> Structured profile pages become the basic units of Web presence</li> <li> Wish-Lists and Offer-Lists are exposed by profile pages</li> </ol> <p>Wish-Lists and Offer-Lists will gradually start bonding with increasing degrees of serendipity courtesy of exponential growth in Linked Data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id154a92f8">Web</a> density. </p> <p>So based on what I've stated so far, Scoble would simply browse the Web or visit his profile page, and in either scenario enjoy a "<a href="http://www.youtube.com/watch?v=nQbVD5hlddk" id="link-id118d3878">minority report</a>" style of experience albeit all under his control (since he is the one driving his Web user agent).</p> <p>What I describe above simply comes down to "Wish-lists" and associated recommendations becoming the norm outside the confines of Amazon's <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id11a6c710">data space</a> on the Web. Serendipitous discovery, intelligent lookups, and linkages are going to be the fundamental essence of Linked Data Web oriented applications, services, agents.</p> <p>Beyond Scoble, it's also important to note that access to data will be controlled by entity "You". Your data space on the Web will be something you will controll access to in a myriad of ways, and it will include the option to provide licensed access to commercial entities on your terms. Naturally, you will also determine the currency that facilitates the value exchange :-)</p> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1458" id="link-id11799a58">The Numerati & The Magic of You!</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1442" id="link-id15246d50">Serendipitous Discovery Quotient (SDQ) Explained</a> </li> <li> <a href="http://www.youtube.com/watch?v=nQbVD5hlddk" id="link-id1360f6d0">Minority Report Clip</a> </li> </ul>
2009-03-25T08:30:58-04:00
Linked Data Workshop "Faceted Views" Paper Update
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-03-16#1533
2009-03-16T13:16:57Z
<p>Here is an update of <a href="http://www.openlinksw.com/weblog/oerling/lodw2.pdf" id="link-id105bef28">Faceted Views over Large-Scale Linked Data</a>. This will be presented at the <a href="http://events.linkeddata.org/ldow2009/" id="link-id16290958">Linked Data workshop</a> of <a href="http://www2009.org/" id="link-id119d8be0">WWW 2009</a> in <a href="http://dbpedia.org/resource/Madrid" id="link-id119b2120">Madrid</a> next month. </p> <p>This update adds more explanation and some comments on how we rank entities.</p>
2009-03-17T11:29:36.000006-04:00
Linked Data Workshop "Faceted Views" Paper Update
http://www.openlinksw.com:443/weblog/oerling/?date=2009-03-16#1532
2009-03-16T13:16:57Z
<p>Here is an update of <a href="http://www.openlinksw.com/weblog/oerling/lodw2.pdf" id="link-id105bef28">Faceted Views over Large-Scale Linked Data</a>. This will be presented at the <a href="http://events.linkeddata.org/ldow2009/" id="link-id16290958">Linked Data workshop</a> of <a href="http://www2009.org/" id="link-id119d8be0">WWW 2009</a> in <a href="http://dbpedia.org/resource/Madrid" id="link-id119b2120">Madrid</a> next month. </p> <p>This update adds more explanation and some comments on how we rank entities.</p>
2009-03-17T11:29:27.000002-04:00
Simple Compare & Contrast of Web 1.0, 2.0, and 3.0 (Update 1)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-03-14#1531
2009-03-14T18:20:00Z
<p>Here is a tabulated "compare and contrast" of <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> usage patterns 1.0, 2.0, and 3.0.</p> <table border="1" width="715" height="286"> <tbody> <tr> <td> </td> <td><strong>Web 1.0</strong></td> <td><strong>Web 2.0</strong></td> <td><strong>Web 3.0</strong></td> </tr> <tr> <td><strong>Simple Definition</strong></td> <td>Interactive / Visual Web</td> <td>Programmable Web</td> <td><a href="http://dbpedia.org/resource/Linked_Data" id="link-id117a9a98">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id146bcdb0">Web</a></td> </tr> <tr> <td><strong>Unit of Presence</strong></td> <td>Web Page</td> <td>Web Service Endpoint</td> <td><a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id11a66c60">Data Space</a> (named structured data enclave)</td> </tr> <tr> <td><strong>Unit of Value Exchange</strong></td> <td>Page <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id146083f8">URL</a></td> <td>Endpoint URL for API</td> <td>Resource / <a href="http://dbpedia.org/resource/Entity" id="link-id121b2148">Entity</a> / Object <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id1467ed00">URI</a></td> </tr> <tr> <td><strong>Data Granularity</strong></td> <td>Low (HTML)</td> <td>Medium (XML)</td> <td>High (RDF)</td> </tr> <tr> <td><strong>Defining Services</strong></td> <td>Search </td> <td>Community (Blogs to Social Networks) </td> <td>Find</td> </tr> <tr> <td><strong>Participation Quotient</strong></td> <td>Low</td> <td>Medium</td> <td>High</td> </tr> <tr> <td><strong>Serendipitous Discovery Quotient </strong></td> <td>Low</td> <td>Medium</td> <td>High</td> </tr> <tr> <td><strong>Data Referencability Quotient </strong></td> <td>Low (Documents)</td> <td>Medium (Documents)</td> <td>High (Documents and their constituent Data)</td> </tr> <tr> <td><strong>Subjectivity Quotient</strong></td> <td>High</td> <td>Medium (from A-list bloggers to select source and partner lists)</td> <td>Low (everything is discovered via URIs)</td> </tr> <tr> <td> <strong><a href="http://dbpedia.org/resource/Transclusion" id="link-id155308d8">Transclusence</a> </strong></td> <td>Low</td> <td>Medium (Code driven Mashups)</td> <td>HIgh (Data driven Meshups)</td> </tr> <tr> <td><strong>What You See Is What You Prefer (WYSIWYP)</strong></td> <td>Low</td> <td>Medium </td> <td>High (negotiated representation of resource descriptions)</td> </tr> <tr> <td><strong>Open Data Access (Data Accessibility)</strong></td> <td>Low</td> <td>Medium (Silos)</td> <td>High (no Silos)</td> </tr> <tr> <td><strong>Identity Issues Handling</strong></td> <td>Low</td> <td>Medium (<a href="http://dbpedia.org/resource/OpenID" id="link-id119d77f8">OpenID</a>)</td> <td><p>High (<a href="http://esw.w3.org/topic/foaf+ssl" id="link-id135cc348">FOAF+SSL</a>)</p></td> </tr> <tr> <td><strong>Solution Deployment Model</strong></td> <td>Centralized</td> <td>Centralized with sprinklings of Federation</td> <td>Federated with function specific Centralization (e.g. Lookup hubs like <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id1496d1d0">LOD</a> Cloud or <a href="http://dbpedia.org/resource/DBpedia" id="link-id1571f690">DBpedia</a>)</td> </tr> <tr> <td><strong>Data Model Orientation</strong></td> <td>Logical (Tree based DOM)</td> <td>Logical (Tree based XML)</td> <td>Conceptual (Graph based RDF)</td> </tr> <tr> <td><strong>User Interface Issues</strong></td> <td>Dynamically generated static interfaces</td> <td>Dyanically generated interafaces with semi-dynamic interfaces (courtesy of XSLT or <a href="http://dbpedia.org/resource/XQuery" id="link-id118399e8">XQuery</a>/<a href="http://dbpedia.org/resource/XPath" id="link-id14b00ba0">XPath</a>)</td> <td>Dynamic Interfaces (pre- and post-generation) courtesy of self-describing nature of RDF</td> </tr> <tr> <td><strong>Data Querying</strong></td> <td><a href="http://dbpedia.org/resource/Full_text_search" id="link-id14fdd948">Full Text Search</a></td> <td>Full Text Search</td> <td>Full Text Search + Structured Graph Pattern Query Language (<a href="http://dbpedia.org/resource/SPARQL" id="link-id154a9368">SPARQL</a>)</td> </tr> <tr> <td><strong>What Each Delivers</strong></td> <td>Democratized Publishing</td> <td>Democratized Journalism & Commentary (Citizen Journalists & Commentators)</td> <td>Democratized Analysis (Citizen Data Analysts)</td> </tr> <tr> <td> <strong><a href="http://dbpedia.org/resource/Star_Wars" id="link-id155ce920">Star Wars Edition Analogy</a> </strong></td> <td>Star Wars (original fight for decentralization via rebellion)</td> <td>Empire Strikes Back (centralization and data silos make comeback)</td> <td>Return of the JEDI (<a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1474" id="link-id11706640">FORCE</a> emerges and facilitates decentralization from "Identity" all the way to "Open Data Access" and "Negotiable Descriptive Data Representation")</td> </tr> </tbody> </table> <p>Naturally, I am not expecting everyone to agree with <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id15be20c0">me</a>. I am simply making my contribution to what will remain facinating discourse for a long time to come :-)</p> <h3>Related</h3> <ul> <li> <a href="http://novaspivack.typepad.com/nova_spivacks_weblog/2007/10/web-30----the-a.html" id="link-id14a9d738">Web 3.0 The Best Official Definition Imaginable</a> -- Nova Spivack's </li> </ul>
2009-04-29T13:21:25.000004-04:00
Important Movie and Ultimate Linked Data Documentary (Update 3)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-03-06#1530
2009-03-06T20:04:37Z
<p>If you are still grappling with the "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id15cfb138">Semantic Web</a> Project" and one of its more distinguished deliverables: <a href="http://dbpedia.org/resource/Linked_Data" id="link-id15340548">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id141f88b8">Web</a>, then please make time to watch and digest the imminence of this <a href="http://video.google.com/videoplay?docid=7190175107515525470" id="link-id15394f88">1990 documentary</a> about <a href="http://dbpedia.org/resource/Hyperland" id="link-id153951e0">Hypermedia</a> titled: <a href="http://dbpedia.org/resource/Hyperland" id="link-id153f7998">Hyperland</a>, by the late <a href="http://dbpedia.org/resource/Douglas_Adams" id="link-id15d2ac88">Douglas Adams</a>.</p> <h3>Related</h3> <ul> <li> <a href="http://www.youtube.com/watch?v=rOsPKjbMvxY" id="link-id117ab018">Hyperland Documentary</a> -- Youtube </li> <li> <a href="http://hyperworlds.org/" id="link-id152dd5d0">Hyperworlds</a> - Ted Nelson Presentation</li> <li> <a href="http://www.youtube.com/watch?v=1IQFjTnDozo" id="link-id1533dba0">The Invention of the World Wide Web</a> </li> <li> <a href="http://www.youtube.com/watch?v=zAvNlh2Z0GI" id="link-id117a0238">The Web's Secret Stories</a> - TED Presentation (basically about using the Web reveal [connections] commonality via [dots] individuality pre. Twitter</li> <li> <a href="http://www.ted.com/index.php/talks/pattie_maes_demos_the_sixth_sense.html" id="link-id154b32f0">Pattie Mae demonstrates 6th sense</a> - an example of what will be done with Linked Data re. user interaction.</li> <li> <a href="http://conferences.ted.com/TED2009/" id="link-id11ea40c8">TimBL's TED 2009 Linked Data Presentation</a> </li> </ul>
2009-03-15T10:35:49.000003-04:00
An Update on Virtuoso Development
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-03-05#1529
2009-03-05T10:23:49Z
<p>It is time for an update on <a href="http://virtuoso.openlinksw.com" id="link-id0x151e89e8">Virtuoso</a> developments.</p> <p>We continue enhancing our hosting of the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1464e168">Linked Open Data</a> (<a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x151e7f38">LOD</a>) cloud at <a href="http://lod.openlinksw.com" id="link-id11ac2448">http://lod.openlinksw.com</a>.</p> <p>We have now added result ranking for both text and URIs. Text hit scores are based on word frequency and proximity; <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x14f9fae0">URI</a> scores are based on link density.</p> <p>We calculate each URI's rank by adding up references and weighing these by the score of the referrer. This is like in web search. Each iteration of the ranking will join every referred to each of its referrers. We do about 1.2 million such joins per second, across partitions, over 2.2 billion triples and 400M distinct subjects without any great optimization, just using <a href="http://dbpedia.org/resource/SQL" id="link-id0xaa36a458">SQL</a> stored procedures and partitioned function calls. This is a sort of SQL map-reduce. We would do over twice as fast if it were all in <a href="http://dbpedia.org/resource/C%2B%2B" id="link-id0x1571e270">C</a> but this is adequate for now. The more interesting bit will be tuning the scoring based on what type of link we have. This is what the web search engines cannot do as well, since document links are untyped.</p> <p>We are moving toward a decent user interface for the LOD hosting, including offering ready-made domain-specific queries, e.g., biomedical.</p> <p>Things like "URI finding with autocomplete" are done and just have to be put online.</p> <p>With <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x14325e08">linked data</a>, there is the whole question of identifier choice. We will have a special page just for this. There we show reference statistics, synonyms declared by <code><a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x638b3900">owl</a>:sameAs</code>, synonyms determined by shared property values, etc. In this way we become a terminology lookup service.</p> <p>Copies of the LOD cluster system are available for evaluators, on a case by case basis. We will make this publicly available on EC2 also in not too long.</p> <p>Otherwise, we continue working on productization, primarily things like reliability and recovery. One exercise is running <a href="http://dbpedia.org/resource/TPC-C" id="link-id0x144d00f0">TPC-C</a> with intentionally stupid partitioning, so that almost all joins and deadlocks are distributed. Then we simulate a cluster interconnect that drops messages now and then, sometimes kill server processes, and still keep full ACID properties. Cloud capable, also in bad weather.</p> <p>The open source release of Virtuoso 6 (no cluster) is basically ready to go, mostly this is a question of logistics.</p> <p>I will talk about these things in greater individual detail next week.</p>
2009-03-05T09:58:16-05:00
An Update on Virtuoso Development
http://www.openlinksw.com:443/weblog/oerling/?date=2009-03-05#1528
2009-03-05T10:23:49Z
<p>It is time for an update on <a href="http://virtuoso.openlinksw.com" id="link-id0x14b05090">Virtuoso</a> developments.</p> <p>We continue enhancing our hosting of the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x14395f00">Linked Open Data</a> (<a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x13e0cf98">LOD</a>) cloud at <a href="http://lod.openlinksw.com" id="link-id11ac2448">http://lod.openlinksw.com</a>.</p> <p>We have now added result ranking for both text and URIs. Text hit scores are based on word frequency and proximity; <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0xa9f5ff20">URI</a> scores are based on link density.</p> <p>We calculate each URI's rank by adding up references and weighing these by the score of the referrer. This is like in web search. Each iteration of the ranking will join every referred to each of its referrers. We do about 1.2 million such joins per second, across partitions, over 2.2 billion triples and 400M distinct subjects without any great optimization, just using <a href="http://dbpedia.org/resource/SQL" id="link-id0x79e77aa0">SQL</a> stored procedures and partitioned function calls. This is a sort of SQL map-reduce. We would do over twice as fast if it were all in <a href="http://dbpedia.org/resource/C%2B%2B" id="link-id0x11a3f3e8">C</a> but this is adequate for now. The more interesting bit will be tuning the scoring based on what type of link we have. This is what the web search engines cannot do as well, since document links are untyped.</p> <p>We are moving toward a decent user interface for the LOD hosting, including offering ready-made domain-specific queries, e.g., biomedical.</p> <p>Things like "URI finding with autocomplete" are done and just have to be put online.</p> <p>With <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x14334a48">linked data</a>, there is the whole question of identifier choice. We will have a special page just for this. There we show reference statistics, synonyms declared by <code><a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x59a757b0">owl</a>:sameAs</code>, synonyms determined by shared property values, etc. In this way we become a terminology lookup service.</p> <p>Copies of the LOD cluster system are available for evaluators, on a case by case basis. We will make this publicly available on EC2 also in not too long.</p> <p>Otherwise, we continue working on productization, primarily things like reliability and recovery. One exercise is running <a href="http://dbpedia.org/resource/TPC-C" id="link-id0x1f6797c0">TPC-C</a> with intentionally stupid partitioning, so that almost all joins and deadlocks are distributed. Then we simulate a cluster interconnect that drops messages now and then, sometimes kill server processes, and still keep full ACID properties. Cloud capable, also in bad weather.</p> <p>The open source release of Virtuoso 6 (no cluster) is basically ready to go, mostly this is a question of logistics.</p> <p>I will talk about these things in greater individual detail next week.</p>
2009-03-05T09:58:07-05:00
Facets and Large Ontologies of the LOD Cloud
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-02-16#1527
2009-02-16T11:21:05Z
<p>We have just submitted <a href="http://www.openlinksw.com/weblog/oerling/lodw.pdf" id="link-id13d9bc68">this paper</a> to the WWW09 <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x51c2fd00">Linked Open Data</a> Workshop.</p> <p>The thing is intermittently live with both <a href="http://dbpedia.org/resource/DBpedia" id="link-id0xa16f87a0">Dbpedia</a> on one instance and a <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1a8212d8">LOD</a> Cloud <a href="http://dbpedia.org/resource/Data" id="link-id0xc1097a8">data</a> collection of about 2 billion triples on another. We will give out the links once we have tested a bit more.</p> <p>The present activity is all about testing <a href="http://virtuoso.openlinksw.com" id="link-id0x17d64f90">Virtuoso</a> 6 for release, cluster and otherwise.</p>
2009-02-17T16:24:34.000004-05:00
Facets and Large Ontologies of the LOD Cloud
http://www.openlinksw.com:443/weblog/oerling/?date=2009-02-16#1526
2009-02-16T11:21:05Z
<p>We have just submitted <a href="http://www.openlinksw.com/weblog/oerling/lodw.pdf" id="link-id13d9bc68">this paper</a> to the WWW09 <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1a2b37a0">Linked Open Data</a> Workshop.</p> <p>The thing is intermittently live with both <a href="http://dbpedia.org/resource/DBpedia" id="link-id0xc0dd578">Dbpedia</a> on one instance and a <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1d721b60">LOD</a> Cloud <a href="http://dbpedia.org/resource/Data" id="link-id0xa141b238">data</a> collection of about 2 billion triples on another. We will give out the links once we have tested a bit more.</p> <p>The present activity is all about testing <a href="http://virtuoso.openlinksw.com" id="link-id0xd69ff70">Virtuoso</a> 6 for release, cluster and otherwise.</p>
2009-02-17T16:24:27.000009-05:00
Response to: What is Web 3.0 and Why Should I Care?
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-01-29#1524
2009-01-29T18:16:44Z
<p>Another post done in response to lost comments. This time, the comments relate to Robin Bloor's article titled: <a href="http://havemacwillblog.com/2008/12/16/what-is-web-30-and-why-should-i-care/" id="link-id12e79d70">What is Web 3.0 and Why Should I Care?</a> </p> <p>Robin:</p> <p> <a href="http://dbpedia.org/resource/Web_3.0" id="link-id12db8fb0">Web 3.0 </a>is fundamentally about the World Wid Web becoming a structured database equipped with a formal <a href="http://dbpedia.org/resource/Data">data</a> model (RDF which is a moniker for <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id11490990">Entity-Attribute-Value</a> with Classes & Relationships based Graph Model), query language, and a protocol for handling divrerse data representational requirements via negotiation</p>. <p>Web 3.0 is about a Web that facilitates serendipitous discovery of relevant things; thereby making serendipitous discovery quotient (SDQ), rather than search engine optimization (SEO), the critical success factor that drives how resources get published on the Web.</p> <p>Personally, I believe we are on the cusp of a major industry inflection re. how we interact with data hosted in computing spaces. In a nutshell, the conceptual model interaction based on real-world entities such as people, places, and other things (including abstract subject matter) will usurp traditional logical model interaction based on rows and columns of typed and/or untyped literal values exemplified by relational data access and management systems.</p> <p>Labels such as "Web 3.0", "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id13664538">Linked Data</a>", and "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id157ff968">Semantic Web</a>", are simply about the aforementioned model transition playing out on the <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id114bd0e8">World Wide Web</a> and across private Linked Data Webs such as Intranets & Extranets, as exemplified emergence of the "Master Data Management" label/buzzword.</p> <h3>What's the critical infrastructure supporting Web 3.0?</h3> <p>As was the case with Web Services re. Web 2.0, there is a critical piece of infrastructure driving the evolution in question, and in this case it comes down to the evolution of Hyperlinking.</p> <p>We now have a new and complimentary variant of Hyperlinking commonly referred to as "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id152ed150">Hyperdata</a>" that now sits alongside "Hypertext". Hyperdata when used in conjunction with HTTP based URIs as Data Source Names (or Identifiers), delivers a potent and granular data access mechanism scoped down to the datum (object or record) level; which is much different from the document (record or <a href="http://dbpedia.org/resource/Entity" id="link-id1141e830">entity</a> container) level linkage that Hypertext accords.</p> <p>In addition, the incorporation of HTTP into this new and enhanced granular Data Source Naming mechanism also addresses past challenges relating to separation of data, data representation, and data transmission protocols -- remember XDR woes familiar to all sockets level programmers -- courtesy of in-built content negotiation. Hence, via a simple HTTP GET --against a Data Source Name exposed by a Hyperdata link -- I can negotiate (from client or server sides) the exact representation of the description (entity-attribute-value graph) of an Entity / Data Object / Resource, dispatched by a data server.</p> <blockquote>For example, this is how a description of entity "<strong><a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id141ce520">Me</a></strong>" ends up being available in (X)HTML or RDF document representations (as you will observe when you click on that link to my Personal <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id15f9fed0">URI</a>).</blockquote> <p> The foundation of what I describe above comes from:</p> <ol> <li> Entity-Attribute-Value & Class Relationship Data Model (originating from LISP era with detours via the <a href="http://dbpedia.org/resource/Object_database" id="link-id12db8fb0">Object Database</a> era. into the Triples approach in RDF) </li> <li>Use of HTTP based Identifiers in the Entity <a href="http://dbpedia.org/resource/Identity_%28object-oriented_programming%29" id="link-id1193af48">ID</a> construction process</li> <li> <a href="http://dbpedia.org/resource/SPARQL" id="link-id1348f188">SPARQL</a> query language for the Data Model.</li> </ol> <p>Some live examples from <a href="http://dbpedia.org/resource/DBpedia" id="link-id12e62a50">DBpedia</a>:</p> <ul> <li> http://dbpedia.org/resource/Linked_Data</li> <li>http://dbpedia.org/resource/Hyperdata</li> <li>http://dbpedia.org/resource/Entity-attribute-value_model</li> <li>http://dbpedia.org/resource/Benjamin_Franklin</li> </ul> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1519?sid=5097848d70f69738bd366e2b6374672c&realm=wa" id="link-id13c31500">The End of RDBMS Primacy is Nigh</a> </li> <li> <a href="http://esw.w3.org/topic/SweoIG/TaskForces/CommunityProjects/LinkingOpenData" id="link-id1356e6a0">Linking Open Data Community</a> </li> </ul>
2009-01-29T13:45:11-05:00
ebiz RDF & Data Integration Article Retort
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-01-29#1522
2009-01-29T15:12:15Z
<p>Yesterday, I stumbled across an <a href="http://www.ebizq.net" id="link-id13e41be8">ebiz</a> article by <a href="http://www.ebizq.net/MT4/mt-cp.cgi?__mode=view&blog_id=43&id=16" id="link-id11c080a0">David Linthicum</a> titled:<a href="http://www.ebizq.net/blogs/linthicum/2009/01/rdf_and_data_integration.php" id="link-id13620940"> RDF & Data Integration</a>. Naturally, I read it, and while reading encountered a number of inaccuracies that compelled <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id152f4828">me</a> to comment on the post. </p> <p>Today, I revisited the same article -- and to my shock and horror -- my comments do not exist (note: the site did accept my comments yesterday!). Even more frustrating for me, I now have to expend time I don't have re-writing my comments due to the depth and danger of the inaccuracies in this post re. RDF in general.</p> <h3>Important Note to ebiz and David: </h3> <p>Please look into what happened to my comments. It's too early for me to conclude that subjective censorship is a play on the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> -- which isn't a hard copy journalistic format style of platform where editors get away with such shenanigans. The Web is a sticky database, and outer joining is well and truly functional (meaning: exclusion and omission ultimately come back to bite via full outer join query results against the Web DB).</p> <p>By the way, if you publish the comments I made to the post (yesterday), I will add a note to this post, accordingly.</p> <p>Yes! David just confirmed to me via <a href="http://linkeddata.uriburner.com/about/html/http://dbpedia.org/resource/Category:Data_modeling" id="link-id15293c20">Twitter</a> that this is yet another comment system related issue and absolutely no intent to censor etc. His words <a href="http://linkeddata.uriburner.com/about/html/http://linkeddata.uriburner.com/about/rdf/http://twitter.com/DavidLinthicum/status/1159201301%23this" id="link-id14e5ac98">Twervatim</a> :-) </p> <p>For sake of clarity, I've itemized the inaccuracies and applied my correction comments (inline) accordingly:</p> <blockquote> <h3>Inaccuracy #1:</h3> <p>Resource Description Framework (RDF), a part of the XML story, provides interoperability between applications that exchange <a href="http://dbpedia.org/resource/Information" id="link-id15f48080">information</a>. </p> <h3>Correction #1: </h3> <p>RDF and XML are not inextricably linked in any way. RDF is part Data Model (<a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id119a7300">EAV</a>/CR style Graph) with associated markup and data serialization formats that include: N3, Turtle, TriX, RDF/XML etc.</p> <h3>Inaccuracy #2:</h3> <p>RDF uses XML to define a foundation for processing metadata and to provide a standard metadata infrastructure for both the Web and the enterprise. </p> <h3>Correction #2: </h3> <p>RDF/XML is an XML based markup and data serialization format. As a markup language it can be used for creating RDF model records/statements (using Subject, Predicate, Object or <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id15120c28">Entity</a>, Attribute, Value). As a serialization format, it provides a mechanism for marshaling RDF data across data managers and data consumers.</p> <h3>Inaccuracy #3:</h3> <p>The difference between the two is that XML is used to transport data using a common format, while RDF is layered on top of XML defining a broad category of data. </p> <h3>Correction #3:</h3> <p>See earlier corrections above.</p> <h3>Inaccuracy #4:</h3> <p>When the XML data is declared to be of the RDF format, applications are then able to understand the data without understanding who sent it. </p> <h3>Correction #4:</h3> <p>You do not declare data to be of RDF format. RDF isn't a format it is a data model (as stated above). You can "up lift" or map data from XML to RDF (hierarchical to graph model mapping). Likewise you can "down shift" or map data from RDF to XML (example: <a href="http://dbpedia.org/resource/SPARQL" id="link-id146966c0">SPARQL</a> SELECT query patterns "down shift" to SPARQL Results XML, which isn't RDF/XML, while keeping access to graphs via URIs or <a href="http://dbpedia.org/resource/Entity" id="link-id14282528">Entity</a> Identifiers that reside within the serialization).</p> <h3>Inaccuracy #5:</h3> <p>RDF extends the XML model and syntax to be specified for describing either resources or a collection of information. (XML points to a resource in order to scope and uniquely identify a set of properties known as the schema.).</p> <h3>Correction #5:</h3> <p>See earlier comments. </p> </blockquote> <p>The single accurate paragraph in this ebiz article lies right at the end and it states the following:</p> <blockquote> <cite>"I've always thought RDF has been underutilized for data integration, and it's really an old standard. Now that we're focused on both understanding and integrating data, perhaps RDF should make a comeback."</cite> </blockquote> <h3>Related:</h3> <ul> <li> <a href="http://www.w3.org/2001/sw/SW-FAQ#whrdfxml" id="link-id1534cdc8">Semantic Web FAQ fragment re. RDF and XML</a> </li> <li> <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=rdf%20data%20integration&type=text&output=html" id="link-id15a7dbc0">Various posts re. RDF and Data Integration</a> from this <a href="http://dbpedia.org/resource/Blog" id="link-id15da4618">Blog</a> <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id1120d810">Data Space</a>.</li> </ul>
2009-01-29T16:25:58-05:00
Time for RDBMS Primacy Downgrade is Nigh! (No Embedded Images Edition - Update 1)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-01-27#1520
2009-01-27T19:19:44Z
<p> As the world works it way through a "once in a generation" economic crisis, the long overdue downgrade of the <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id15750540">RDBMS</a>, from its pivotal position at the apex of the <a href="http://dbpedia.org/resource/Data" id="link-id0x24ea3650">data</a> access and data management pyramid is nigh.</p> <h3>What is the Data Access, and Data Management Value Pyramid?</h3> <p> As depicted below, a top-down view of the data access and data management value chain. The term: apex, simply indicates value primacy, which takes the form of a data access API based entry point into a DBMS realm -- aligned to an underlying data model. Examples of data access APIs include: Native Call Level Interfaces (CLIs), <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id11c254c0">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id149b16a8">JDBC</a>, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id11451eb0">ADO</a>.NET, <a href="http://dbpedia.org/resource/OLE_DB" id="link-id15b02478">OLE-DB</a>, <a href="http://dbpedia.org/resource/XML_for_Analysis" id="link-id1181fa10">XMLA</a>, and <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id0x1f8394a8">Web</a> Services.</p> See: <a href="http://virtuoso.openlinksw.com/images/Agility_Value_Factors_Pyramid.png" id="link-id146cadd8"> AVF Pyramid Diagram.</a> <p> The degree to which ad-hoc views of data managed by a DBMS can be produced and dispatched to relevant data consumers (e.g. people), without compromising concurrency, data durability, and security, collectively determine the "Agility Value Factor" (AVF) of a given DBMS. Remember, agility as the cornerstone of environmental adaptation is as old as the concept of evolution, and intrinsic to all pursuits of primacy. </p> <p>In simpler business oriented terms, look at AVF as the degree to which DBMS technology affects the ability to effectively implement "Market Leadership Discipline" along the following pathways: innovation, operation excellence, or customer intimacy. </p> <h3>Why has RDBMS Primacy has Endured?</h3> <p> Historically, at least since the late '80s, the RDBMS genre of DBMS has consistently offered the highest AVF relative to other DBMS genres en route to primacy within the value pyramid. The desire to improve on paper reports and spreadsheets is basically what DBMS technology has fundamentally addressed to date, even though conceptual level interaction with data has never been its forte.</p> See: <a href="http://virtuoso.openlinksw.com/images/Old_RDBMS_Primacy_Pyramid.png" id="link-id134dab90"> RDBMS Primacy Diagram.</a> <p> For more then 10 years -- at the very least -- limitations of the traditional RDBMS in the realm of conceptual level interaction with data across diverse data sources and schemas (enterprise, Web, and <a href="http://dbpedia.org/resource/Internet" id="link-id116001c0">Internet</a>) has been crystal clear to many RDBMS technology practitioners, as indicated by some of the quotes excerpted below:</p> <blockquote> <cite> <p> "Future of Database Research is excellent, but what is the future of data?" </p> "..it is hard for <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id14932398">me</a> to disagree with the conclusions in this report. It captures exactly the right thoughts, and should be a must read for everyone involved in the area of databases and database research in particular." <p>-- <a href="http://jhingran.typepad.com/anant_jhingrans_musings/" id="link-id11334c50">Dr. Anant Jingran</a>, CTO, IBM <a href="http://dbpedia.org/resource/Information" id="link-id150c7970">Information</a> Management Systems, commenting on the <a href="http://db.cs.berkeley.edu/claremont/" id="link-id11c3b408">2007 RDBMS technology retreat</a> attended by a number of key DBMS technology pioneers and researchers.</p> </cite> </blockquote> <blockquote> <cite> <p> "<a href="http://www.databasecolumn.com/2007/09/one-size-fits-all.html" id="link-id15c14f08">One size fits all: A concept whose time has come and gone</a> </p> <p> </p> <ol> <li> They are direct descendants of System R and <a href="http://dbpedia.org/resource/Ingres" id="link-id146da780">Ingres</a> and were architected more than 25 years ago</li> <li> They are advocating "one size fits all"; i.e. a single engine that solves all DBMS needs. </li> </ol> <p>-- Prof. <a href="http://en.wikipedia.org/wiki/Michael_Stonebraker" id="link-id145c4e28">Michael Stonebreaker</a>, one of the founding fathers of the RDBMS industry.</p> </cite> </blockquote> <p>Until this point in time, the requisite confluence of "circumstantial pain" and "open standards" based technology required to enable an objective "compare and contrast" of RDBMS engine virtues and viable alternatives hasn't occurred. Thus, the RDBMS has endured it position of primacy albeit on a "one size fits all basis". </p> <h4>Circumstantial Pain</h4> <p> As mentioned earlier, we are in the midst of an economic crisis that is ultimately about a consistent inability to connect dots across a substrate of interlinked data sources that transcend traditional data access boundaries with high doses of schematic heterogeneity. Ironically, in a era of the dot-com, we haven't been able to make meaningful connections between relevant "real-world things" that extend beyond primitive data hosted database tables and content management style document containers; we've struggled to achieve this in the most basic sense, let alone evolve our ability to connect inline with the <a href="http://www.vldb2007.org/program/slides/s1161-brodie.pdf" id="link-id11a0dcf0">exponential rate at which the Internet & Web are spawning "universes of discourse" (data spaces) that emanate from user activity</a> (within the enterprise and across the Internet & Web). In a nutshell, we haven't been able to upgrade our interaction with data such that "conceptual models" and resulting "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id12da4b00">context</a> lenses" (or facets) become concrete; by this I mean: real-world <a href="http://dbpedia.org/resource/Entity" id="link-id146a48a8">entity</a> interaction making its way into the computer realm as opposed to the impedance we all suffer today when we transition from conceptual model interaction (real-world) to logical model interaction (when dealing with RDBMS based data access and data management). </p> <p>Here are some simple examples of what I can only best describe as: "critical dots unconnected", resulting from an inability to interact with data conceptually:</p> <strong>Government (Globally) -</strong> <p> Financial regulatory bodies couldn't effectively discern that a <a href="http://dbpedia.org/resource/Credit_default_swap" id="link-id115ba0e0">Credit Default Swap</a> is an Insurance policy in all but literal name. And in not doing so the cost of an unregulated <a href="http://dbpedia.org/resource/Insurance" id="link-id158d4960">insurance policy</a> laid the foundation for exacerbating the toxicity of fatally flawed mortgage backed securities. Put simply: a flawed insurance policy was the fallback on a toxic security that financiers found exotic based on superficial packaging.</p> <strong>Enterprises - </strong> <p> Banks still don't understand that capital really does exists in tangible and intangible forms; with the intangible being the variant that is inherently dynamic. For example, a tech companies intellectual capital far exceeds the value of fixture, fittings, and buildings, but you be amazed to find that in most cases this vital asset has not significant value when banks get down to the nitty gritty of debt collateral; instead, a buffer of flawed securitization has occurred atop a borderline static asset class covering the aforementioned buildings, fixtures, and fittings. </p> <p> In the general enterprise arena, IT executives continued to "rip and replace" existing technology without ever effectively addressing the timeless inability to connect data across disparate data silos generated by internal enterprise applications, let alone the broader need to mesh data from the inside with external data sources. No correlations made between the growth of buzzwords and the compounding nature of data integration challenges. It's 2009 and only a miniscule number of executives dare fantasize about being anywhere within distance of the: relevant information at your fingertips vision. </p> <p> Looking more holistically at data interaction in general, whether you interact with data in the enterprise space (i.e., at work) or on the Internet or Web, you ultimately are delving into a mishmash of disparate computer systems, applications, service (Web or SOA), and databases (of the RDBMS variety in a majority of cases) associated with a plethora of disparate schemas. Yes, but even today "rip and replace" is still the norm pushed by most vendors; pitting one mono culture against another as exemplified by irrelevances such as: FOSS/LAMP vs Commercial or Web vs. Enterprise, when none of this matters if the data access and integration issues are recognized let alone addressed (see: <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1497?sid=0df0294caee8b37925c6a888bbbca136&realm=wa" id="link-id15c27300">Applications are Like Fish and Data Like Wine</a>). </p> <p> Like the current credit-crunch, exponential growth of data originating from disparate application databases and associated schemas, within shrinking processing time frames, has triggered a rethinking of what defines data access and data management value today en route to an inevitable RDBMS downgrade within the value pyramid.</p> <h3>Technology</h3> <p>There have been many attempts to address real-world modeling requirements across the broader DBMS community from Object Databases to Object-Relational Databases, and more recently the emergence of simple <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id1128dad0">Entity</a>-Attribute-Value model DBMS engines. In all cases failure has come down to the existence of one or more of the following deficiencies, across each potential alternative:</p> <ol> <li>Query language standardization - nothing close to <a href="http://dbpedia.org/resource/SQL" id="link-id16002d60">SQL</a> standardization</li> <li>Data Access API standardization - nothing close to ODBC, JDBC, OLE-DB, or ADO.NET</li> <li>Wire protocol standardization - nothing close to HTTP</li> <li>Distributed Identity infrastructure - nothing close to the non-repudiatable digital Identity that <a href="http://dbpedia.org/resource/Friend_of_a_friend" id="link-id14926b18">foaf</a>+ssl accords</li> <li>Use of Identifiers as network based pointers to data sources - nothing close to RDF based <a href="http://dbpedia.org/resource/Linked_Data" id="link-id16180a28">Linked Data</a> </li> <li>Negotiable data representation - nothing close to Mime and HTTP based Content Negotiation</li> <li>Scalability especially in the era of Internet & Web scale.</li> </ol> <h4>Entity-Attribute-Value with Classes & Relationships (<a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id13e741b8">EAV</a>/CR) data models</h4> <p>A common characteristic shared by all post-relational DBMS management systems (from Object Relational to pure Object) is an orientation towards variations of EAV/CR based data models. Unfortunately, all efforts in the EAV/CR realm have typically suffered from at least one of the deficiencies listed above. In addition, the same "one DBMS model fits all" approach that lies at the heart of the RDBMS downgrade also exists in the EAV/CR realm.</p> <h3>What Comes Next?</h3> <p>The RDBMS is not going away (ever), but its era of primacy -- by virtue of its placement at the apex of the data access and data management value pyramid -- is over! I make this bold claim for the following reasons: </p> <ol> <li> The Internet aided "Global Village" has brought "<a href="http://en.wikipedia.org/wiki/Open_World_Assumption" id="link-id1148e560">Open World</a>" vs "<a href="http://en.wikipedia.org/wiki/Closed_World_Assumption" id="link-id11967cd0">Closed World</a>" assumption issues to the fore e.g., the current global economic crisis remains centered on the inability to connect dots across "Open World" and "Closed World" data frontiers </li> <li> Entity-Attribute-Value with Classes & Relationships (EAV/CR) based DBMS models are more effective when dealing with disparate data associated with disparate schemas, across disparate DBMS engines, host operating systems, and networks. </li> </ol> <p>Based on the above, it is crystal clear that a different kind of DBMS -- one with higher AVF relative to the RDBMS -- needs to sit atop today's data access and data management value pyramid. The characteristics of this DBMS must include the following:</p> <ol> <li> Every item of data (Datum/Entity/Object/Resource) has Identity</li> <li> Identity is achieved via Identifiers that aren't locked at the DBMS, OS, Network, or Application levels</li> <li> Object Identifiers and Object values are independent (extricably linked by association)</li> <li> Object values should be de-referencable via Object Identifier</li> <li> Representation of de-referenced value graph (entity, attributes, and values mesh) must be negotiable (i.e. content negotiation)</li> <li>Structured query language must provide mechanism for Creation, Deletion, Updates, and Querying of data objects</li> <li> Performance & Scalability across "Closed World" (enterprise) and "Open World" (Internet & Web) realms.</li> </ol> <p>Quick recap, I am not saying that RDBMS engine technology is dead or obsolete. I am simply stating that the era of RDBMS primacy within the data access and data management value pyramid is over. </p> <p>The problem domain (conceptual model views over heterogeneous data sources) at the apex of the aforementioned pyramid has simply evolved beyond the natural capabilities of the RDBMS which is rooted in "Closed World" assumptions re., data definition, access, and management. The need to maintain domain based conceptual interaction with data is now palpable at every echelon within our "Global Village" - Internet, Web, Enterprise, Government etc.</p> <p>It is my personal view that an EAV/CR model based DBMS, with support for the seven items enumerated above, can trigger the long anticipated RDBMS downgrade. Such a DBMS would be inherently multi-model because you would need to the best of RDBMS and EAV/CR model engines in a single product, with in-built support for HTTP and other Internet protocols in order to effectively address data representation and serialization issues.</p> <h4>EAV/CR Oriented Data Access & Management Technology</h4> <p>Examples of contemporary EAV/CR frameworks that provide concrete conceptual layers for data access and data management currently include:</p> <ul> <li> <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id115d1cb0"> Resource Description Framework</a> (RDF) - an EAV/CR based framework</li> <li> <a href="http://dbpedia.org/resource/Linked_Data" id="link-id116cf810">RDF Linked Data </a>- EAV/CR based framework that mandates de-referencable HTTP based Identifiers</li> <li> <a href="http://dbpedia.org/resource/ADO.NET_Entity_Framework" id="link-id13daa160">ADO.NET Entity Frameworks</a> - Microsoft .NET based EAV/CR framework</li> <li> <a href="http://dbpedia.org/page/Core_Data" id="link-id11111838">Core Data Services </a>- Mac OS X based EAV/CR framework that evolved from NeXT's <a href="http://dbpedia.org/resource/Enterprise_Objects_Framework" id="link-id15c27df0">Enterprise Object Frameworks</a> (EOF).</li> </ul> <p>The frameworks above provide the basis for a revised AVF pyramid, as depicted below, that reflects today's data access and management realities i.e., an Internet & Web driven global village comprised of interlinked distributed data objects, compatible with "Open World" assumptions.</p> See: <a href="http://virtuoso.openlinksw.com/images/New_EAV_RDBMS_Pyramid.png" id="link-id158e0760">New EAV/CR Primacy Diagram.</a> <h3>Related</h3> <ul> <li> <a href="http://dynamicorange.com/2009/01/22/blueblog-how-and-why-glue-is-using-amazon-simpledb-instead-of-a-relational-database/" id="link-id15e07c10">How & Why Glue is Using Amazon SimpleDB</a> </li> <li> <a href="http://www.cs.cmu.edu/afs/cs.cmu.edu/user/clamen/OODBMS/Manifesto/htManifesto/node4.html#SECTION00022000000000000000" id="link-id116cf450">Object Database Manifesto (Identity excerpt)</a> </li> <li> <a href="http://www.unixspace.com/context/databases.html" id="link-id150b2c20">Database Models Overview</a> </li> <li> <a href="http://www.youtube.com/watch?v=WEj9vqVvHPc&feature=related" id="link-id0x1135d978">Ted Nelson Explaining Irregularity and Idiosyncrasy of Data Structures</a> - ZigZag Demo </li> </ul>
2009-03-17T11:50:58-04:00
The Time for RDBMS Primacy Downgrade is Nigh!
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-01-24#1519
2009-01-25T00:04:00Z
<p> As the world works it way through a "once in a generation" economic crisis, the long overdue downgrade of the <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id15750540">RDBMS</a>, from its pivotal position at the apex of the <a href="http://dbpedia.org/resource/Data" id="link-id0x66a74b8">data</a> access and data management pyramid is nigh.</p> <h3>What is the Data Access, and Data Management Value Pyramid?</h3> <p> As depicted below, a top-down view of the data access and data management value chain. The term: apex, simply indicates value primacy, which takes the form of a data access API based entry point into a DBMS realm -- aligned to an underlying data model. Examples of data access APIs include: Native Call Level Interfaces (CLIs), <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id11c254c0">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id149b16a8">JDBC</a>, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id11451eb0">ADO</a>.NET, <a href="http://dbpedia.org/resource/OLE_DB" id="link-id15b02478">OLE-DB</a>, <a href="http://dbpedia.org/resource/XML_for_Analysis" id="link-id1181fa10">XMLA</a>, and <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id0x2fef498">Web</a> Services.</p> <div> <img alt="Image" src="http://virtuoso.openlinksw.com/images/Agility_Value_Factors_Pyramid.png" /> </div> <p> The degree to which ad-hoc views of data managed by a DBMS can be produced and dispatched to relevant data consumers (e.g. people), without compromising concurrency, data durability, and security, collectively determine the "Agility Value Factor" (AVF) of a given DBMS. Remember, agility as the cornerstone of environmental adaptation is as old as the concept of evolution, and intrinsic to all pursuits of primacy. </p> <p>In simpler business oriented terms, look at AVF as the degree to which DBMS technology affects the ability to effectively implement "Market Leadership Discipline" along the following pathways: innovation, operation excellence, or customer intimacy. </p> <h3>Why has RDBMS Primacy has Endured?</h3> <p> Historically, at least since the late '80s, the RDBMS genre of DBMS has consistently offered the highest AVF relative to other DBMS genres en route to primacy within the value pyramid. The desire to improve on paper reports and spreadsheets is basically what DBMS technology has fundamentally addressed to date, even though conceptual level interaction with data has never been its forte.</p> <div> <img alt="Image" src="http://virtuoso.openlinksw.com/images/Old_RDBMS_Primacy_Pyramid.png" /> </div> <p> For more then 10 years -- at the very least -- limitations of the traditional RDBMS in the realm of conceptual level interaction with data across diverse data sources and schemas (enterprise, Web, and <a href="http://dbpedia.org/resource/Internet" id="link-id116001c0">Internet</a>) has been crystal clear to many RDBMS technology practitioners, as indicated by some of the quotes excerpted below:</p> <blockquote> <cite> <p> "Future of Database Research is excellent, but what is the future of data?" </p> "..it is hard for <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id14932398">me</a> to disagree with the conclusions in this report. It captures exactly the right thoughts, and should be a must read for everyone involved in the area of databases and database research in particular." <p>-- <a href="http://jhingran.typepad.com/anant_jhingrans_musings/" id="link-id11334c50">Dr. Anant Jingran</a>, CTO, IBM <a href="http://dbpedia.org/resource/Information" id="link-id150c7970">Information</a> Management Systems, commenting on the <a href="http://db.cs.berkeley.edu/claremont/" id="link-id11c3b408">2007 RDBMS technology retreat</a> attended by a number of key DBMS technology pioneers and researchers.</p> </cite> </blockquote> <blockquote> <cite> <p> "<a href="http://www.databasecolumn.com/2007/09/one-size-fits-all.html" id="link-id15c14f08">One size fits all: A concept whose time has come and gone</a> </p> <p> </p> <ol> <li> They are direct descendants of System R and <a href="http://dbpedia.org/resource/Ingres" id="link-id146da780">Ingres</a> and were architected more than 25 years ago</li> <li> They are advocating "one size fits all"; i.e. a single engine that solves all DBMS needs. </li> </ol> <p>-- Prof. <a href="http://en.wikipedia.org/wiki/Michael_Stonebraker" id="link-id145c4e28">Michael Stonebreaker</a>, one of the founding fathers of the RDBMS industry.</p> </cite> </blockquote> <p>Until this point in time, the requisite confluence of "circumstantial pain" and "open standards" based technology required to enable an objective "compare and contrast" of RDBMS engine virtues and viable alternatives hasn't occurred. Thus, the RDBMS has endured it position of primacy albeit on a "one size fits all basis". </p> <h4>Circumstantial Pain</h4> <p> As mentioned earlier, we are in the midst of an economic crisis that is ultimately about a consistent inability to connect dots across a substrate of interlinked data sources that transcend traditional data access boundaries with high doses of schematic heterogeneity. Ironically, in a era of the dot-com, we haven't been able to make meaningful connections between relevant "real-world things" that extend beyond primitive data hosted database tables and content management style document containers; we've struggled to achieve this in the most basic sense, let alone evolve our ability to connect inline with the <a href="http://www.vldb2007.org/program/slides/s1161-brodie.pdf" id="link-id11a0dcf0">exponential rate at which the Internet & Web are spawning "universes of discourse" (data spaces) that emanate from user activity</a> (within the enterprise and across the Internet & Web). In a nutshell, we haven't been able to upgrade our interaction with data such that "conceptual models" and resulting "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id12da4b00">context</a> lenses" (or facets) become concrete; by this I mean: real-world <a href="http://dbpedia.org/resource/Entity" id="link-id146a48a8">entity</a> interaction making its way into the computer realm as opposed to the impedance we all suffer today when we transition from conceptual model interaction (real-world) to logical model interaction (when dealing with RDBMS based data access and data management). </p> <p>Here are some simple examples of what I can only best describe as: "critical dots unconnected", resulting from an inability to interact with data conceptually:</p> <strong>Government (Globally) -</strong> <p> Financial regulatory bodies couldn't effectively discern that a <a href="http://dbpedia.org/resource/Credit_default_swap" id="link-id115ba0e0">Credit Default Swap</a> is an Insurance policy in all but literal name. And in not doing so the cost of an unregulated <a href="http://dbpedia.org/resource/Insurance" id="link-id158d4960">insurance policy</a> laid the foundation for exacerbating the toxicity of fatally flawed mortgage backed securities. Put simply: a flawed insurance policy was the fallback on a toxic security that financiers found exotic based on superficial packaging.</p> <strong>Enterprises - </strong> <p> Banks still don't understand that capital really does exists in tangible and intangible forms; with the intangible being the variant that is inherently dynamic. For example, a tech companies intellectual capital far exceeds the value of fixture, fittings, and buildings, but you be amazed to find that in most cases this vital asset has not significant value when banks get down to the nitty gritty of debt collateral; instead, a buffer of flawed securitization has occurred atop a borderline static asset class covering the aforementioned buildings, fixtures, and fittings. </p> <p> In the general enterprise arena, IT executives continued to "rip and replace" existing technology without ever effectively addressing the timeless inability to connect data across disparate data silos generated by internal enterprise applications, let alone the broader need to mesh data from the inside with external data sources. No correlations made between the growth of buzzwords and the compounding nature of data integration challenges. It's 2009 and only a miniscule number of executives dare fantasize about being anywhere within distance of the: relevant information at your fingertips vision. </p> <p> Looking more holistically at data interaction in general, whether you interact with data in the enterprise space (i.e., at work) or on the Internet or Web, you ultimately are delving into a mishmash of disparate computer systems, applications, service (Web or SOA), and databases (of the RDBMS variety in a majority of cases) associated with a plethora of disparate schemas. Yes, but even today "rip and replace" is still the norm pushed by most vendors; pitting one mono culture against another as exemplified by irrelevances such as: FOSS/LAMP vs Commercial or Web vs. Enterprise, when none of this matters if the data access and integration issues are recognized let alone addressed (see: <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1497?sid=0df0294caee8b37925c6a888bbbca136&realm=wa" id="link-id15c27300">Applications are Like Fish and Data Like Wine</a>). </p> <p> Like the current credit-crunch, exponential growth of data originating from disparate application databases and associated schemas, within shrinking processing time frames, has triggered a rethinking of what defines data access and data management value today en route to an inevitable RDBMS downgrade within the value pyramid.</p> <h3>Technology</h3> <p>There have been many attempts to address real-world modeling requirements across the broader DBMS community from Object Databases to Object-Relational Databases, and more recently the emergence of simple <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id1128dad0">Entity</a>-Attribute-Value model DBMS engines. In all cases failure has come down to the existence of one or more of the following deficiencies, across each potential alternative:</p> <ol> <li>Query language standardization - nothing close to <a href="http://dbpedia.org/resource/SQL" id="link-id16002d60">SQL</a> standardization</li> <li>Data Access API standardization - nothing close to ODBC, JDBC, OLE-DB, or ADO.NET</li> <li>Wire protocol standardization - nothing close to HTTP</li> <li>Distributed Identity infrastructure - nothing close to the non-repudiatable digital Identity that <a href="http://dbpedia.org/resource/Friend_of_a_friend" id="link-id14926b18">foaf</a>+ssl accords</li> <li>Use of Identifiers as network based pointers to data sources - nothing close to RDF based <a href="http://dbpedia.org/resource/Linked_Data" id="link-id16180a28">Linked Data</a> </li> <li>Negotiable data representation - nothing close to Mime and HTTP based Content Negotiation</li> <li>Scalability especially in the era of Internet & Web scale.</li> </ol> <h4>Entity-Attribute-Value with Classes & Relationships (<a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id13e741b8">EAV</a>/CR) data models</h4> <p>A common characteristic shared by all post-relational DBMS management systems (from Object Relational to pure Object) is an orientation towards variations of EAV/CR based data models. Unfortunately, all efforts in the EAV/CR realm have typically suffered from at least one of the deficiencies listed above. In addition, the same "one DBMS model fits all" approach that lies at the heart of the RDBMS downgrade also exists in the EAV/CR realm.</p> <h3>What Comes Next?</h3> <p>The RDBMS is not going away (ever), but its era of primacy -- by virtue of its placement at the apex of the data access and data management value pyramid -- is over! I make this bold claim for the following reasons: </p> <ol> <li> The Internet aided "Global Village" has brought "<a href="http://en.wikipedia.org/wiki/Open_World_Assumption" id="link-id1148e560">Open World</a>" vs "<a href="http://en.wikipedia.org/wiki/Closed_World_Assumption" id="link-id11967cd0">Closed World</a>" assumption issues to the fore e.g., the current global economic crisis remains centered on the inability to connect dots across "Open World" and "Closed World" data frontiers </li> <li> Entity-Attribute-Value with Classes & Relationships (EAV/CR) based DBMS models are more effective when dealing with disparate data associated with disparate schemas, across disparate DBMS engines, host operating systems, and networks. </li> </ol> <p>Based on the above, it is crystal clear that a different kind of DBMS -- one with higher AVF relative to the RDBMS -- needs to sit atop today's data access and data management value pyramid. The characteristics of this DBMS must include the following:</p> <ol> <li> Every item of data (Datum/Entity/Object/Resource) has Identity</li> <li> Identity is achieved via Identifiers that aren't locked at the DBMS, OS, Network, or Application levels</li> <li> Object Identifiers and Object values are independent (extricably linked by association)</li> <li> Object values should be de-referencable via Object Identifier</li> <li> Representation of de-referenced value graph (entity, attributes, and values mesh) must be negotiable (i.e. content negotiation)</li> <li>Structured query language must provide mechanism for Creation, Deletion, Updates, and Querying of data objects</li> <li> Performance & Scalability across "Closed World" (enterprise) and "Open World" (Internet & Web) realms.</li> </ol> <p>Quick recap, I am not saying that RDBMS engine technology is dead or obsolete. I am simply stating that the era of RDBMS primacy within the data access and data management value pyramid is over. </p> <p>The problem domain (conceptual model views over heterogeneous data sources) at the apex of the aforementioned pyramid has simply evolved beyond the natural capabilities of the RDBMS which is rooted in "Closed World" assumptions re., data definition, access, and management. The need to maintain domain based conceptual interaction with data is now palpable at every echelon within our "Global Village" - Internet, Web, Enterprise, Government etc.</p> <p>It is my personal view that an EAV/CR model based DBMS, with support for the seven items enumerated above, can trigger the long anticipated RDBMS downgrade. Such a DBMS would be inherently multi-model because you would need to the best of RDBMS and EAV/CR model engines in a single product, with in-built support for HTTP and other Internet protocols in order to effectively address data representation and serialization issues.</p> <h4>EAV/CR Oriented Data Access & Management Technology</h4> <p>Examples of contemporary EAV/CR frameworks that provide concrete conceptual layers for data access and data management currently include:</p> <ul> <li> <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id115d1cb0"> Resource Description Framework</a> (RDF) - an EAV/CR based framework</li> <li> <a href="http://dbpedia.org/resource/Linked_Data" id="link-id116cf810">RDF Linked Data </a>- EAV/CR based framework that mandates de-referencable HTTP based Identifiers</li> <li> <a href="http://dbpedia.org/resource/ADO.NET_Entity_Framework" id="link-id13daa160">ADO.NET Entity Frameworks</a> - Microsoft .NET based EAV/CR framework</li> <li> <a href="http://dbpedia.org/page/Core_Data" id="link-id11111838">Core Data Services </a>- Mac OS X based EAV/CR framework that evolved from NeXT's <a href="http://dbpedia.org/resource/Enterprise_Objects_Framework" id="link-id15c27df0">Enterprise Object Frameworks</a> (EOF).</li> </ul> <p>The frameworks above provide the basis for a revised AVF pyramid, as depicted below, that reflects today's data access and management realities i.e., an Internet & Web driven global village comprised of interlinked distributed data objects, compatible with "Open World" assumptions.</p> <div> <image src="http://virtuoso.openlinksw.com/images/New_EAV_RDBMS_Pyramid.png"></image> </div> <h3>Related</h3> <ul> <li> <a href="http://allanslibrary.blogspot.com/2009/06/semantic-way.html" id="link-id0xb8c5e498">The Semantic Way</a> - Alan Cho's Summary of <a href="http://www.pwc.com/extweb/home.nsf/docid/1308AF8EA7929CCA852575BA00720F26" id="link-id0xb80f5e10">PwC 2009 tech forecast report on the Semantic Web</a> </li> <li> <a href="http://www.readwriteweb.com/archives/is_the_relational_database_doomed.php" id="link-id0xb8c20658">Is the RDBMS Doomed</a> - <a href="http://www.readwriteweb.com">ReadWriteWeb</a> Article</li> <li> <a href="http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores/" id="link-id0x1ab4778">Anti-RDBMS: a list of Distributed Key-Value Stores</a> - by <a href="http://www.last.fm/user/RJ" id="link-id0x5a968060">Richard Jones</a> (CTO Last.FM)</li> <li> <a href="http://dynamicorange.com/2009/01/22/blueblog-how-and-why-glue-is-using-amazon-simpledb-instead-of-a-relational-database/" id="link-id15e07c10">How & Why Glue is Using Amazon SimpleDB</a> </li> <li> <a href="http://www.cs.cmu.edu/afs/cs.cmu.edu/user/clamen/OODBMS/Manifesto/htManifesto/node4.html#SECTION00022000000000000000" id="link-id116cf450">Object Database Manifesto (Identity excerpt)</a> </li> <li> <a href="http://www.unixspace.com/context/databases.html" id="link-id150b2c20">Database Models Overview</a> </li> <li> <a href="http://www.youtube.com/watch?v=WEj9vqVvHPc&feature=related" id="link-id0x66b0850">Ted Nelson Explaining Irregularity and Idiosyncrasy of Data Structures</a> - ZigZag Demo </li> </ul>
2009-06-03T18:09:58.000001-04:00
In Response to: This is Not the Future (Update #3)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-01-13#1518
2009-01-13T04:18:12Z
<p>As I cannot post directly to Glenn's <a href="http://dbpedia.org/resource/Blog" id="link-id149ad010">blog</a> titled: <a href="http://www.furia.com/page.cgi?type=log&id=319" id="link-id113ed070">This is Not the Near Future (Either)</a>, I have to basically respond to him here, in blog post form :-(</p> <p>What is our <a href="http://b3s.openlinksw.com/fct/facet.vsp" id="link-id10fbeec0">"Search" and "Find" demonstration</a> about? It is about how you use the "Description" of "Things" to unambiguously locate things in a database at <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> Scale.</p> <p>To our perpetual chagrin, we are trying to demonstrate an engine -- not UI prowess -- but the immediate response is to jump to the UI aesthetics.</p> <p>Google, Yahoo etc.. offer a simple input form for <a href="http://dbpedia.org/resource/Full_text_search" id="link-id14296fb8">full text search</a> patterns, they have a processing window for completing full text searches across Web Content indexed on their servers. Once the search patterns are processed, you get a page ranked result set (collection of Web pages basically that claim/state: we found N pages out of a document corpus of about M indexed pages). </p> <p> <strong>Note:</strong> the estimate aspect of traditional search results in like "advertising small print" the user lives with the illusion that all possible documents on the Web (or even <a href="http://dbpedia.org/resource/Internet" id="link-id13436b50">Internet</a>) have been searched whereas in reality: 25% of the possible total is a major stretch; since the Web and Internet are <a href="http://en.wikipedia.org/wiki/Fractal_dimension_on_networks" id="link-id1105ec48">fractal networks</a> and <a href="http://en.wikipedia.org/wiki/Scale-free_network" id="link-id111ad558">scale-free</a>, inherently growing at exponential rates "ad infinitum" across boundless dimensions of human comprehension.</p> <p> The power of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id111dc7c8">Linked Data</a> ultimately comes down to the fact that the user constructs the path to what they seek via the properties of the "Things" in question. The routes are not hardwired since <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id15cbc6f8">URI</a> de-referencing (follow your nose pattern) is available to Linked Data aware query engines and crawlers. </p> <p>We are simply trying to demonstrate how you can combine the best of full text search with the best of structured querying while reusing familiar interaction patterns from Google/Yahoo. Thus, you start with full text search, find get all the entities associated with the pattern, then use the <a href="http://dbpedia.org/resource/Entity" id="link-id1512c8a8">entity</a> types or entity properties to find what you seek.</p> <p>You state in your post:</p> <blockquote> <cite>"To state the obvious caveat, the claim OpenLink is making about this demo is not that it delivers better search-term relevance, therefore the ranking of searching results is not the main criteria on which it is intended to be assessed." </cite> </blockquote> <p> Correct. </p> <blockquote> <cite> "On the other hand, one of the things they are bragging about is that their server will automatically cut off long-running queries. So how do you like your first page of results?". </cite> </blockquote> <p> Not exactly correct. We are performing aggregates using a configurable interactive time factor. Example: tell <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id12fb67c0">me</a> how many entities of type: Person, with interest: <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id159bffc8">Semantic Web</a>, exist in this database within 2 seconds. Also understand that you could retry the same query and get different numbers within the same interactive time factor. It isn't your basic "query cut-off". </p> <blockquote> <cite> "And on the other other hand, the big claim OpenLink is making about this demo is that the aggregate experience of using it is better than the aggregate experience of using "traditional" search. So go ahead, use it. If you can."</cite> </blockquote> <p>Yes, "Microsoft" was a poor example for sure, the example could have been pattern: "glenn mcdonald", which should demonstrate the fundamental utility of what we are trying to demonstrate i.e., entity disambiguation courtesy of entity properties and/or entity type filtering.</p> <p> <a href="http://www.google.com/search?q=glenn+mcdonald" id="link-id15e4dbc8">Compare Googles results for: Glenn McDonald</a> with those from our demo (which dissambiguate "Glenn McDonald" via associated properties and/or types), assuming we both agree that your Web Site or Blog Home isn't the center of your entity graph or personal <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id15754528">data space</a> (i.e., data about you); so getting your home page at the top of the Google page rank offers limited value, in reality.</p> <p>What are we bragging about? A little more than what you attempt to explain. Yes, we are showing that we can find stuff within a processing window, but understand the following:</p> <ul> <li> Processing Time Window (or interactive time) is configurable </li> <li> Data Corpus is a Billion+ Triples (from <a href="http://challenge.semanticweb.org/" id="link-id149a25e0">Billion Triples Challenge Data Set</a>) </li> <li> <a href="http://dbpedia.org/resource/SPARQL" id="link-id15e13180">SPARQL</a> doesn't have Aggregation capabilities by default (we have implemented <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSArticleBISPARQL2" id="link-id1593a550">SPARQL-BI</a> to deliver aggregates for analytics against large data sets, we even handle the TPC-H industry standard benchmark with SPARQL-BI)</li> <li> Paging isn't possible without aggregates, and doing aggregates on a Billion+ triples as part of a query processing cycle isn't trivial stuff (otherwise it would be everywhere due to inherent and obvious necessity).</li> </ul> <p>I hope I've clarified what's going on with our demo? If not, pose your challenge via examples and I will respond with solutions or simply cry out loud: "no mas!".</p> <p>As for your "Mac OX X Leopard" comments, I can only say this: I emphasized that this is a demo, the data is pretty old, and the input data has issues (i.e. some of the input data is bad as your example shows). The purpose of this demo is not about the text per se., it's about the size of the data corpus and faceted querying. We are going to have the entire <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id15dfec08">LOD</a> Cloud loaded into the real thing, and in addition to that our Sponger Middleware will be enabled, and then you can take issue with data quality as per your reference to "Cyndi Lauper" (btw - it takes one property filter to find <a href="http://b3s.openlinksw.com/fct/facet.vsp?cmd=set_view&sid=552&type=text-properties&limit=20&offset=0" id="link-id1496d2a0">information about her quickly</a> using "<strong>dbpprop:name</strong>" after filtering for properties with text values).</p> <p>Of all things, this demo had nothing to do with UI and <a href="http://dbpedia.org/resource/Information" id="link-id11009090">Information</a> presentation aesthetics. It was all about combining full text search and structured queries (sparql behind the scenes) against a huge data corpus en route to solving challenges associated with faceted browsing over large data sets. We have built a service that resides inside <a href="http://virtuoso.openlinksw.com" id="link-id118b36a8">Virtuoso</a>. The Service is naturally of the "Web Service" variety and can be used from any consumer / client environment that speaks HTTP (directly or indirectly).</p> <p>To be continued ...</p>
2009-01-21T19:02:47-05:00
A Linked Data Web Approach To Semantic "Search" & "Find" (Updated)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-01-09#1517
2009-01-09T23:34:50Z
<p>The first salvo of what we've been hinting about re. server side faceted browsing over Unlimited <a href="http://dbpedia.org/resource/Data">Data</a> within configurable Interactive Time-frames is now available for experimentation at: <a href="http://b3s.openlinksw.com/fct/facet.vsp" id="link-ide41d210">http://b3s.openlinksw.com/fct/facet.vsp</a>.</p> <h3>Simple example / demo:</h3> <p>Enter search pattern: Microsoft</p> <p>You will get the usual result from a full text pattern search i.e., hits and text excerpts with matching patterns in boldface. This first step is akin to throwing your net out to sea while fishing.</p> <p> Now you have your catch, what next? Basically, this is where traditional text search value ends since <a href="http://dbpedia.org/resource/regular_expression" id="link-id113b6840">regex</a> or <a href="http://dbpedia.org/resource/XPath" id="link-id1151c140">xpath</a>/<a href="http://dbpedia.org/resource/XQuery" id="link-id14565db8">xquery</a> offer little when the structure of literal text is the key to filtering or categorization based analysis of real-world entities. Naturally, this is where the value of structured querying of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11bc8208">linked data</a> starts, as you seek to use <a href="http://dbpedia.org/resource/Entity" id="link-id150e7298">entity</a> descriptions (combination of attribute and relationship properties) to "Find relevant things".</p> <p>Continuing with the demo.</p> <p>Click on "Properties" link within the Navigation section of the browser page which results in a distillation and aggregation of the properties of the entities associated with the search results. Then use the "Next" link to page through the properties until to find the properties that best match what you seek. Note, this particular step is akin to using the properties of the catch (using fishing analogy) for query filtering, with each subsequent property link click narrowing your selection further.</p> <p>Using property based filtering is just one perspective on the data corpus associated with the text search pattern; thus, you can alter perspectives by clicking on the "Class" link so that you can filter you search results by entity type. Of course, in a number of scenarios you would use a combination of entity types and entity properties filters to locate the entities of interest to you. </p> <h3>A Few Notes about this demo instance of <a href="http://virtuoso.openlinksw.com" id="link-id14453088">Virtuoso</a>:</h3> <ul> <li> Lookup Data Size (Local Linked Data Corpus): 2 Billion+ Triples (<a href="http://dbpedia.org/page/Entity-attribute-value_model" id="link-id13447558">entity-attribute-value</a> tuples)</li> <li> This is a *temporary* teaser / precursor to the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id14e3bfc8">LOD</a> (Linking Open Data Cloud) variant of our Linked Data driven "Search" & "Find" service; we decided to implement this functionality prior to commissioning a larger and more up to date instance based on the entire LOD Cloud</li> <li> The browser is simply using a <a href="http://virtuoso.openlinksw.com" id="link-id138b5688">Virtuoso</a> PL function that also exists in <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> Service form for loose binding by 3rd parties that have a UI orientation and focus (our UI is deliberately bare boned).</li> <li>The properties and entity types (classes) links expose formal definitions and dictionary provenance <a href="http://dbpedia.org/resource/Information" id="link-id10ecc8e0">information</a> materialized in an HTML page (of course your browser or any other HTTP user agent can negotiation alternative representations of this descriptive information)</li> <li> <a href="http://umbel.org/about/" id="link-id117b95e0">UMBEL</a> based inference rules are enabled, giving you a live and simple demonstration of the virtues of Linked Data Dictionaries for example: click on the description link of any property or class from the <a href="http://dbpedia.org/resource/Friend_of_a_friend" id="link-id1595dd88">foaf</a> (friend-of-a-friend vocabulary), <a href="http://dbpedia.org/resource/SIOC" id="link-id151315e8">sioc</a> (semantically-interlinked-online-communities ontology), <a href="http://musicontology.com/" id="link-id15b9d6e8">mo</a> (music ontology), <a href="http://bibliontology.com/" id="link-id114257e8">bibo</a> (bibliographic data ontology) namespaces to see how the data between these lower level vocabularies or ontologies are meshed with <a href="http://dbpedia.org/resource/Cyc" id="link-id15b9be80">OpenCyc</a>'s upper level ontology. </li> </ul> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1515" id="link-id14694eb8">Faceted Search: Unlimited Data in Interactive Time</a> </li> <li> <a href="http://blogs.usnet.private:8893/Virtuoso Anytime: No Query Is Too Complex (updated)" id="link-id1356c630">Virtuoso Anytime: No Query Is Too Complex</a> </li> </ul>
2009-01-10T13:55:56.000001-05:00
Faceted Search: Unlimited Data in Interactive Time
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-01-09#1516
2009-01-09T22:03:11Z
<p>Why not see the whole world of <a href="http://dbpedia.org/resource/Data" id="link-id0xc3f6b38">data</a> as facets? Well, we'd like to, but there is the feeling that this is not practical.</p> <p>The old problem has been that it is not really practical to pre-compute counts of everything for all possible combinations of search conditions and counting/grouping/sorting. The actual matches take time.</p> <p>Well, neither is in fact necessary. When there are large numbers of items matching the conditions, counting them can take time but then this is the beginning of the search, and the user is not even likely to look very closely at the counts. It is enough to see that there are many of one and few of another. If the user already knows the precise predicate or class to look for, then the top-level faceted view is not even needed. The faceted view for guiding search and precise analytics are two different problems.</p> <p>There are client-side faceted views like Exhibit or our own <a href="http://ode.openlinksw.com/" id="link-id0x1bc1cfe0">ODE</a>. The problem with these is that there are a few orders of magnitude difference between the actual database size and what fits on the user agent. This is compounded by the fact that one does not know what to cache on the user agent because of the open nature of the data web. If this were about a fixed workflow, then a good guess would be possible — but we are talking about the data web, the very soul of serendipity and unexpected discovery.</p> <p>So we made a web service that will do faceted search on arbitrary <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xbb62170">RDF</a>. If it does not get complete results within a timeout, it will return what it has counted so far, using <a href="http://virtuoso.openlinksw.com" id="link-id0xb122b00">Virtuoso</a>'s <a href="http://www.openlinksw.com/weblog/oerling/?id=1494" id="link-id117b0df0"><b>Anytime</b></a> feature. Looking for subjects with some specific combination of properties is however a bit limited, so this will also do <code>JOINs</code>. Many features are one or two <code>JOINs</code> away; take geographical locations or social networks, for example.</p> <p>Yet a faceted search should be point-and-click, and should not involve a full query construction. We put the compromise at starting with full text or property or class, then navigating down properties or classes, to arbitrary depth, tree-wise. At each step, one can see the matching instances or their classes or properties, all with counts, faceted-style.</p> <p>This is good enough for queries like 'what do Harry Potter fans also like' or 'who are the authors of articles tagged <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0xbee32d8">semantic web</a> and machine learning and published in 2008'. For complex grouping, sub-queries, arithmetic or such, one must write the actual query.</p> <p>But one can begin with facets, and then continue refining the query by hand since the service also returns <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xbcc9f38">SPARQL</a> text. We made a small web interface on top of the service with all logic server side. This proves that the web service is usable and that an interface with no AJAX, and no problems with browser interoperability or such, is possible and easy. Also, the problem of syncing between a user-agent-based store and a database is entirely gone.</p> <p>If we are working with a known data structure, the user interface should choose the display by the data type and offer links to related reports. This is all easy to build as web pages or AJAX. We show how the generic interface is done in Virtuoso PL, and you can adapt that or rewrite it in <a href="http://dbpedia.org/resource/PHP" id="link-id0xcdbe268">PHP</a>, Java, JavaScript, or anything else, to accommodate use-case specific navigation needs such as data format.</p> <p>The web service takes an <a href="http://dbpedia.org/resource/XML" id="link-id0xc019c08">XML</a> representation of the search, which is more restricted and easier to process by machine than the SPARQL syntax. The web service returns the results, the SPARQL query it generated, whether the results are complete or not, and some resource use statistics.</p> <p>The source of the PL functions, Web Service and Virtuoso Server Page (HTML UI) will be available as part of Virtuoso 6.0 and higher. A Programmer's Guide will be available as part of the standard Virtuoso Documentation collection, including the Virtuoso Open Source Edition Website.</p>
2009-01-09T17:15:39-05:00
Faceted Search: Unlimited Data in Interactive Time
http://www.openlinksw.com:443/weblog/oerling/?date=2009-01-09#1515
2009-01-09T22:03:11Z
<p>Why not see the whole world of <a href="http://dbpedia.org/resource/Data" id="link-id0x1a0319c0">data</a> as facets? Well, we'd like to, but there is the feeling that this is not practical.</p> <p>The old problem has been that it is not really practical to pre-compute counts of everything for all possible combinations of search conditions and counting/grouping/sorting. The actual matches take time.</p> <p>Well, neither is in fact necessary. When there are large numbers of items matching the conditions, counting them can take time but then this is the beginning of the search, and the user is not even likely to look very closely at the counts. It is enough to see that there are many of one and few of another. If the user already knows the precise predicate or class to look for, then the top-level faceted view is not even needed. The faceted view for guiding search and precise analytics are two different problems.</p> <p>There are client-side faceted views like Exhibit or our own <a href="http://ode.openlinksw.com/" id="link-id0xc3db130">ODE</a>. The problem with these is that there are a few orders of magnitude difference between the actual database size and what fits on the user agent. This is compounded by the fact that one does not know what to cache on the user agent because of the open nature of the data web. If this were about a fixed workflow, then a good guess would be possible — but we are talking about the data web, the very soul of serendipity and unexpected discovery.</p> <p>So we made a web service that will do faceted search on arbitrary <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xbdbf198">RDF</a>. If it does not get complete results within a timeout, it will return what it has counted so far, using <a href="http://virtuoso.openlinksw.com" id="link-id0x17691878">Virtuoso</a>'s <a href="http://www.openlinksw.com/weblog/oerling/?id=1494" id="link-id117b0df0"><b>Anytime</b></a> feature. Looking for subjects with some specific combination of properties is however a bit limited, so this will also do <code>JOINs</code>. Many features are one or two <code>JOINs</code> away; take geographical locations or social networks, for example.</p> <p>Yet a faceted search should be point-and-click, and should not involve a full query construction. We put the compromise at starting with full text or property or class, then navigating down properties or classes, to arbitrary depth, tree-wise. At each step, one can see the matching instances or their classes or properties, all with counts, faceted-style.</p> <p>This is good enough for queries like 'what do Harry Potter fans also like' or 'who are the authors of articles tagged <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x160eb950">semantic web</a> and machine learning and published in 2008'. For complex grouping, sub-queries, arithmetic or such, one must write the actual query.</p> <p>But one can begin with facets, and then continue refining the query by hand since the service also returns <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x17e82228">SPARQL</a> text. We made a small web interface on top of the service with all logic server side. This proves that the web service is usable and that an interface with no AJAX, and no problems with browser interoperability or such, is possible and easy. Also, the problem of syncing between a user-agent-based store and a database is entirely gone.</p> <p>If we are working with a known data structure, the user interface should choose the display by the data type and offer links to related reports. This is all easy to build as web pages or AJAX. We show how the generic interface is done in Virtuoso PL, and you can adapt that or rewrite it in <a href="http://dbpedia.org/resource/PHP" id="link-id0xbe41d60">PHP</a>, Java, JavaScript, or anything else, to accommodate use-case specific navigation needs such as data format.</p> <p>The web service takes an <a href="http://dbpedia.org/resource/XML" id="link-id0xc2fc358">XML</a> representation of the search, which is more restricted and easier to process by machine than the SPARQL syntax. The web service returns the results, the SPARQL query it generated, whether the results are complete or not, and some resource use statistics.</p> <p>The source of the PL functions, Web Service and Virtuoso Server Page (HTML UI) will be available as part of Virtuoso 6.0 and higher. A Programmer's Guide will be available as part of the standard Virtuoso Documentation collection, including the Virtuoso Open Source Edition Website.</p>
2015-08-20T17:24:08.122703-04:00
New ADO.NET 3.x Provider for Virtuoso Released (Update 2)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-01-08#1514
2009-01-08T04:36:47Z
<p>I am pleased to announce the immediate availability of the <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtAdoNet35Provider" id="link-id142e7390">Virtuoso ADO.NET 3.5 data provider</a> for Microsoft's .NET platform.</p> <h3>What is it?</h3> <p>A data access driver/provider that provides conceptual <a href="http://dbpedia.org/resource/Entity" id="link-id11c36c00">entity</a> oriented access to <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id12fb8618">RDBMS</a> data managed by Virtuoso. Naturally, it also uses Virtuoso's in-built virtual / <a href="http://dbpedia.org/resource/federated_database_system" id="link-id115bedc8">federated database</a> layer to provide access to <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id15153c08">ODBC</a> and <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id13418908">JDBC</a> accessible RDBMS engines such as: <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id134d72f0">Oracle</a> (7.x to latest), <a href="http://dbpedia.org/resource/SQL" id="link-id15757b88">SQL</a> Server (4.2 to latest), <a href="http://dbpedia.org/resource/Sybase" id="link-id15ef8d48">Sybase</a>, IBM <a href="http://dbpedia.org/resource/IBM_Informix" id="link-id12f56aa0">Informix</a> (5.x to latest), IBM <a href="http://dbpedia.org/resource/IBM_DB2" id="link-id119feb38">DB2</a>, <a href="http://dbpedia.org/resource/Ingres" id="link-id14e3d6c8">Ingres</a> (6.x to latest), Progress (7.x to OpenEdge), <a href="http://dbpedia.org/resource/MySQL" id="link-id11295630">MySQL</a>, PostgreSQL, <a href="http://dbpedia.org/resource/Firebird_database_server" id="link-id12f40448">Firebird</a>, and others using our ODBC or JDBC bridge drivers.</p> <h3>Benefits?</h3> <h4>Technical:</h4> <p>It delivers an <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id14012040">Entity-Attribute-Value + Classes & Relationships model</a> over disparate data sources that are materialized as .NET Entity Framework Objects, which are then consumable via ADO.NET Data Object Services, LINQ for Entities, and other ADO.NET data consumers.</p> <p>The provider is fully integrated into Visual Studio 2008 and delivers the same "ease of use" offered by Microsoft's own SQL Server provider, but across Virtuoso, Oracle, Sybase, DB2, Informix, Ingres, <a href="http://dbpedia.org/resource/Progress_4GL" id="link-id158d1fe8">Progress (OpenEdge</a>), MySQL, PostgreSQL, Firebird, and others. The same benefits also apply uniformly to Entity Frameworks compatibility.</p> <p> Bearing in mind that Virtuoso is a multi-model (hybrid) data manager, this also implies that you can use .NET Entity Frameworks against all data managed by Virtuoso. Remember, Virtuoso's SQL channel is a conduit to Virtuoso's core; thus, RDF (courtesy of <a href="http://esw.w3.org/topic/SPASQL" id="link-id133c9b70">SPASQL</a> as already implemented re. <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtJenaProvider" id="link-id11380b80">Jena</a>/<a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtSesame2Provider" id="link-id10fc0c88">Sesame</a>/<a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtRDFDriverRedland" id="link-id1390f730">Redland</a> providers), XML, and other data forms stored in Virtuoso also become accessible via .NET's Entity Frameworks.</p> <br /> <h4>Strategic:</h4> <p>You can choose which entity oriented data access model works best for you: RDF <a href="http://dbpedia.org/resource/Linked_Data" id="link-id151354f0">Linked Data</a> & <a href="http://dbpedia.org/resource/SPARQL" id="link-id15dc5eb0">SPARQL</a> or .NET Entity Frameworks & <a href="http://en.wikipedia.org/wiki/ADO.NET_Entity_Framework#Entity_SQL" id="link-id14404e80">Entity SQL</a>. Either way, Virtuoso delivers a commercial grade, high-performance, secure, and scalable solution.</p> <br /> <h3>How do I use it?</h3> Simply follow one of guides below: <ul> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtEntityFrameworkSchoolDbWinFormApp" id="link-id15e5c580">Using Visual Studio 2008 & Virtuoso to build an Entity Frameworks based Windows forms application</a> </li> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtUsingMsAdoNetDataServicesWithVirtuoso" id="link-id157912b0">Using Visual Studio 2008 & Virtuoso to build an ADO.NET Data Services based application</a> </li> </ul> <p> <b>Note:</b> When working with external or 3rd party databases, simply use the Virtuoso Conductor to link the external data source into Virtuoso. Once linked, the remote tables will simply be treated as though they are native Virtuoso tables leaving the <a href="http://dbpedia.org/resource/Virtual_Database" id="link-id15b04b18">virtual database</a> engine to handle the rest. This is similar to the role the Microsoft JET engine played in the early days of ODBC, so if you've ever linked an ODBC data source into Microsoft Access, you are ready to do the same using Virtuoso.</p> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1420" id="link-id160afdd0">Entity Oriented Data Access</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1474" id="link-id113eeb50">Yoda & the Data FORCE.</a> </li> </ul>
2009-01-08T09:12:50.000006-05:00
Linked Data Web Collaborators: Introducing Structured Dynamics
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-01-03#1513
2009-01-03T04:03:33Z
<p>As indicated in posts from Fred Giasson and <a href="http://community.linkeddata.org/dataspace/person/mkbergman#this" id="link-id152486c0">Mike Bergman</a>, the <a href="http://zitgist.com/about/" id="link-id1163fb28">Zitgist</a> incubation effort that contributed to the delivery of vital <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1163ff68">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id112a1338">Web</a> infrastructure components such as <a href="http://www.talkdigger.com/" id="link-id11938fe8">TalkDigger</a> (discourse discovery and participation), <a href="http://www.pingthesemanticweb.com/about/" id="link-id15da46f0">PingTheSemanticWeb</a> (ground-zero <a href="http://dbpedia.org/resource/Data">data</a> source for most <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id15ff68f0">Semantic Web</a> search engines), <a href="http://umbel.org/about/" id="link-id112fddb0">UMBEL</a> (binding layer for Upper and Lower Ontologies amongst other things), <a href="http://musicontology.com" id="link-id157ff9e0">Music Ontology</a> (enabling meaningful description of Music), and <a href="http://bibliontology.com/" id="link-id11459180">Bibliographic Ontology</a> (enabling meaningful description of Bibliographic content), is now ready to continue its business development and technology growth as a going concern known as <a href="http://www.structureddynamics.com/" id="link-id110c3b50">Structured Dynamics</a>.</p> <p>With great joy and pride, I wish Structured Dynamics all the success they deserve. Naturally, the collaborations and close relationship between <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id11849528">OpenLink Software</a> and its latest technology partner will continue -- especially as we collectively work towards a more comprehendible and pragmatic Web of Linked Data for developers (across Web 1.0, 2.0, 3.0, and beyond), end-users (<a href="http://dbpedia.org/resource/Information" id="link-id15246af8">information</a>- and <a href="http://dbpedia.org/resource/Knowledge" id="link-id15d27888">knowledge</a>-workers), and entrepreneurs (driven by quality and tangible value contribution).</p> <h3>Related</h3> <ul> <li> <a href="http://fgiasson.com/blog/index.php/2009/01/02/structured-dynamics-for-the-new-year/" id="link-id13bf7fd0">Structured Dynamics for the New Year</a> </li> <li> <a href="http://www.mkbergman.com/?p=468" id="link-id111e9e88">A New Year, a New Beginning and a New Venture</a> </li> </ul>
2009-01-02T23:27:26-05:00
My Hopes for Linked Data in 2009 (Update #2)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2009-01-02#1512
2009-01-02T18:39:23Z
<p>Happy New Year!</p> <p>In 2009 I hope the following happens re. "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id15acc7d0">Linked Data</a>":</p> <ol> <li>We realize it's a <a href="http://dbpedia.org/resource/Meme" id="link-id1101eb90">Meme</a> </li> <li>We collectively connect the Meme to the concept of granular hyperlinks between <a href="http://dbpedia.org/resource/Data">data</a> entities/objects (datum to datum linkage aka. <a href="http://dbpedia.org/resource/Linked_Data" id="link-id113d96a0">Hyperdata</a> Linking)</li> <li>We generally connect the Meme to technology ancestry such as the <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id1136d980">Entity-Attribute-Value with Classes & Relationships</a> (EAV/CR) data model (then broader commonality with erstwhile unrelated realms will be unveiled e.g., <a href="http://dbpedia.org/resource/ADO.NET_Entity_Framework" id="link-id1122ab80">Entity Frameworks from Microsoft</a>, <a href="http://dbpedia.org/resource/Core_Data" id="link-id138b5b28">Core Data from Apple</a>, <a href="http://en.wikipedia.org/wiki/Simpledb" id="link-id118576d0">SimpleDB</a> from Amazon, and the <a href="http://en.wikipedia.org/wiki/Freebase_(database)" id="link-id19107a90">Freebase Graph Model DB</a> amongst others)</li> <li>We instinctively connect the Meme to the concept of Entity Oriented Data Access and Management (RDF based Linked Data is basically EAV/CR scheme that uses HTTP based Pointers for Entity, Attribute, and Relationship Identifiers)</li> <li>We naturally connect the Meme with the notion that an identifier for a unit of data (aka. Datum) should be the conduit to a negotiable representation of said Datum's description (i.e., it's attribute and relationship properties in HTML, XHTML, <a href="http://dbpedia.org/resource/RDFa" id="link-id151cc688">RDFa</a>, Turtle, N3, RDF/XML etc., for example)</li> <li>We ultimately connect the Meme with a conceptual-level approach to data integration across disparate data sources (also known as <a href="http://dbpedia.org/resource/Master_Data_Management" id="link-id1596b8d8">Master Data Management</a> (MDM) ).</li> </ol> <p> 2009 is about a reboot on a monumental scale. We need new thinking, new technology, new approaches, and new solutions. No matter what route we take, we can't negate the importance of "Data". When dealing with organic or inorganic computers systems -- Data is simply everything!</p> <p> The ability of individuals and enterprises to access, mesh, and disseminate data to relevant nodes across public and private networks will ultimately determine the winners and losers in the new frontier, ushered in by 2009.</p> <p> Do not take data access and data management technology for granted. User interfaces come and ago, application logic comes and goes, but your data stays with you forever. If you are mystified by data access technology then make 2009 the year of data access technology demystification :-) </p> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/?id=1510" id="link-id11246da8">Linked Data & The Year 2009</a> </li> <li> <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=data%20access&type=text&output=html" id="link-id11848a20">Various posts from my blog space</a> </li> </ul>
2009-01-06T21:35:19.000002-05:00
Linked Data & The Year 2009 (updated)
http://www.openlinksw.com:443/blog/vdb/blog/?date=2009-01-02#1511
2009-01-02T16:17:06Z
<p>As is fitting for the season, I will editorialize a bit about what has gone before and what is to come.</p> <p> <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id1119f250">Sir Tim</a> said it at WWW08 in <a href="http://www2008.org/" id="link-id0x1dcb93a0">Beijing</a> — <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x13a3efb8">linked data</a> and the linked data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x13a44cd0">web</a> is the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x10d25788">semantic web</a> and the Web done right.</p> <p>The grail of <i>ad hoc</i> analytics on infinite <a href="http://dbpedia.org/resource/Data" id="link-id0xa201d518">data</a> has lost none of its appeal. We have seen fresh evidence of this in the realm of data warehousing products, as well as storage in general.</p> <p>The benefits of a data model more abstract than the relational are being increasingly appreciated also outside the data web circles. Microsoft's <a href="http://dbpedia.org/resource/Entity" id="link-id0x12fa4e40">Entity</a> Frameworks technology is an example. Agility has been a buzzword for a long time. Everything should be offered in a service based business model and should interoperate and integrate with everything else — business needs first; schema last.</p> <p>Not to forget that when money is tight, reuse of existing assets and paying on a usage basis are naturally emphasized. <a href="http://dbpedia.org/resource/Information" id="link-id0x175b32e8">Information</a>, as the asset it is, is none the less important, on the contrary. But even with information, value should be realized economically, which, among other things, entails not reinventing the wheel.</p> <p>It is against this backdrop that this year will play out.</p> <p>As concerns research, I will <a href="http://www.openlinksw.com/weblog/oerling/?id=1374" id="link-id1151b128">again quote</a> <a href="http://www.ibiblio.org/hhalpin/#" id="link-id141cb740">Harry Halpin</a> at <a href="http://www.eswc2008.org/" id="link-id0x18a8a858">ESWC 2008</a>: "Men will fight in a war, and even lose a war, for what they believe just. And it may come to pass that later, even though the war were lost, the things then fought for will emerge under another name and establish themselves as the prevailing reality" [or words to this effect].</p> <p>Something like the data web, and even the semantic web, will happen. Harry's question was whether this would be the descendant of what is today called semantic web research.</p> <p>I heard in conversation about a project for making a very large metadata store. I also heard that the makers did not particularly insist on this being <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x3c39ed80">RDF</a>-based, though.</p> <p>Why should such a thing be RDF-based? If it is already accepted that there will be <i>ad hoc</i> schema and that queries ought to be able to view the data from all angles, not be limited by having indices one way and not another way, then why not RDF?</p> <p>The justification of RDF is in reusing and linking-to data and terminology out there. Another justification is that by using an RDF store, one is spared a lot of work and tons of compromises which attend making an <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id0x14a77880">entity</a>-attribute-value (<a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id0x5f978e88">EAV</a>, i.e., triple) store on a generic <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x391bdcd8">RDBMS</a>. The sem-web world has been there, trust me. We came out well because we put all inside the RDBMS, lowest level, which you can't do unless you own the RDBMS. Source access is not enough; you also need the <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x138a3a00">knowledge</a>.</p> <p>Technicalities aside, the question is one of proprietary vs. standards-based. This is not only so with software components, where standards have consistently demonstrated benefits, but now also with the data. <a href="http://www.zemanta.com/" id="link-id0x5f92cb38">Zemanta</a> and <a href="http://www.opencalais.com/" id="link-id0x139c3200">OpenCalais</a> serving <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x1731dc78">DBpedia</a> URIs are examples. Even in entirely closed applications, there is benefit in reusing open vocabularies and identifiers: One does not need to create a secret language for writing a secret memo.</p> <p>Where data is a carrier of value, its value is enhanced by it being easy to repurpose (i.e., standard vocabularies) and to discover (i.e., data set metadata). As on the web, so on the enterprise <a href="http://dbpedia.org/resource/Intranet" id="link-id0x1324ada8">intranet</a>. In this lies the strength of RDF as opposed to proprietary flexible database schemes. This is a qualitative distinction.</p> <p align="center"> <a href="http://esw.w3.org/topic/SweoIG/TaskForces/CommunityProjects/LinkingOpenData" id="link-id117178a8"><img src="http://www.openlinksw.com/images/logos/LoDLogo.gif" alt="Linking Open Data project logo" /> </a> <br /> <a href="http://dbpedia.org/resource/In_hoc_signo_vinces" id="link-id115f47e8"><i>In hoc signo vinces.</i> </a> </p> <p>In this light, we welcome the <a href="http://semanticweb.org/wiki/VoiD" id="link-id0x67cf560">voiD</a> (<a href="http://semanticweb.org/wiki/VoiD" id="link-id0x1898c908">VOcabulary of Interlinked Data</a>), which is the first promise of making federatable data discoverable. Now that there is a point of focus for these efforts, the needed expressivity will no doubt accrete around the voiD core.</p> <p>For data as a service, we clearly see the value of open terminologies as prerequisites for service interchangeability, i.e., creating a marketplace. <a href="http://dbpedia.org/resource/XML" id="link-id0x1588d6a8">XML</a> is for the transaction; RDF is for the discovery, query, and analytics. As with databases in general, first there was the transaction; then there was the query. Same here. For monetizing the query, there are models ranging from renting data sets and server capacity in the clouds to hosted services where one pays for processing past a certain quota. For the hosted case, we just removed a major barrier to offering unlimited query against unlimited data when we completed the <a href="http://www.openlinksw.com/weblog/oerling/?id=1374" id="link-id110b8668">Virtuoso Anytime</a> feature. With this, the user gets what is found within a set time, which is already something, and in case of needing more, one can pay for the usage. Of course, we do not forget advertising. When data has explicit semantics, contextuality is better than with keywords.</p> <p>For these visions to materialize on top of the linked data platform, linked data must join the world of data. This means messaging that is geared towards the database public. They know the problem, but the RDF proposition is still not well enough understood for it to connect.</p> <p>For the relational IT world, we offer passage to the data web and its promise of integration through RDF mapping. We are also bringing out new Microsoft Entity <a href="http://dbpedia.org/resource/ADO.NET_Entity_Framework" id="link-id0x13a50fd8">Framework</a> components. This goes in the direction of defining a unified database frontier with RDF and non-RDF entity models side by side.</p> <p>For <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id0x1d2ea7f0">OpenLink Software</a>, 2008 was about developing technology for scale, RDF as well as generic relational. We did show a tiny preview with the <a href="http://challenge.semanticweb.org/" id="link-id0x658fbc8">Billion Triples Challenge</a> demo. Now we are set to come out with the real thing, featuring, among other things, faceted search at the billion triple scale. We <a href="http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?id=1489" id="link-id150c6090">started offering ready-to-go Virtuoso-hosted linked open data sets</a> on Amazon EC2 in December. Now we continue doing this based on our next-generation server, as well as make Virtuoso 6 Cluster commercially available. Technical specifics are amply discussed on this <a href="http://dbpedia.org/resource/Blog" id="link-id0x1424ec20">blog</a>. There are still some new technology things to be developed this year; first among these are strong <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x14b8ca88">SPARQL</a> federation, and on-the-fly resizing of server clusters. On the research partnerships side, we have an EU grant for working with the OntoWiki project from the University of Leipzig, and we are partners in DERI's <a href="https://lion.deri.ie/" id="link-id115c02f8">Líon project</a>. These will provide platforms for further demonstrating the "web" in data web, as in web-scale smart databasing.</p> <p>2009 will see change through scale. The things that exist will start interconnecting and there will be emergent value. Deployments will be larger and scale will be readily available through a services model or by installation at one's own facilities. We may see the start of Search becoming Find, like <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id14e43050">Kingsley</a> says, meaning semantics of data guiding search. Entity extraction will multiply data volumes and bring parts of the data web to real time.</p> <p>Exciting 2009 to all.</p>
2009-01-02T13:26:42.000003-05:00
Linked Data & The Year 2009 (updated)
http://www.openlinksw.com:443/weblog/oerling/?date=2009-01-02#1510
2009-01-02T16:17:06Z
<p>As is fitting for the season, I will editorialize a bit about what has gone before and what is to come.</p> <p> <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id1119f250">Sir Tim</a> said it at WWW08 in <a href="http://www2008.org/" id="link-id0x14ab66b0">Beijing</a> — <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x115a4588">linked data</a> and the linked data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0xa5c678">web</a> is the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x7cbe5540">semantic web</a> and the Web done right.</p> <p>The grail of <i>ad hoc</i> analytics on infinite <a href="http://dbpedia.org/resource/Data" id="link-id0xa4b25428">data</a> has lost none of its appeal. We have seen fresh evidence of this in the realm of data warehousing products, as well as storage in general.</p> <p>The benefits of a data model more abstract than the relational are being increasingly appreciated also outside the data web circles. Microsoft's <a href="http://dbpedia.org/resource/Entity" id="link-id0x1c3c72b0">Entity</a> Frameworks technology is an example. Agility has been a buzzword for a long time. Everything should be offered in a service based business model and should interoperate and integrate with everything else — business needs first; schema last.</p> <p>Not to forget that when money is tight, reuse of existing assets and paying on a usage basis are naturally emphasized. <a href="http://dbpedia.org/resource/Information" id="link-id0xa0743bd8">Information</a>, as the asset it is, is none the less important, on the contrary. But even with information, value should be realized economically, which, among other things, entails not reinventing the wheel.</p> <p>It is against this backdrop that this year will play out.</p> <p>As concerns research, I will <a href="http://www.openlinksw.com/weblog/oerling/?id=1374" id="link-id1151b128">again quote</a> <a href="http://www.ibiblio.org/hhalpin/#" id="link-id141cb740">Harry Halpin</a> at <a href="http://www.eswc2008.org/" id="link-id0x28f68040">ESWC 2008</a>: "Men will fight in a war, and even lose a war, for what they believe just. And it may come to pass that later, even though the war were lost, the things then fought for will emerge under another name and establish themselves as the prevailing reality" [or words to this effect].</p> <p>Something like the data web, and even the semantic web, will happen. Harry's question was whether this would be the descendant of what is today called semantic web research.</p> <p>I heard in conversation about a project for making a very large metadata store. I also heard that the makers did not particularly insist on this being <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x13c8af68">RDF</a>-based, though.</p> <p>Why should such a thing be RDF-based? If it is already accepted that there will be <i>ad hoc</i> schema and that queries ought to be able to view the data from all angles, not be limited by having indices one way and not another way, then why not RDF?</p> <p>The justification of RDF is in reusing and linking-to data and terminology out there. Another justification is that by using an RDF store, one is spared a lot of work and tons of compromises which attend making an <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id0x1ca17b20">entity</a>-attribute-value (<a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id0x1c9d6050">EAV</a>, i.e., triple) store on a generic <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x557dff0">RDBMS</a>. The sem-web world has been there, trust me. We came out well because we put all inside the RDBMS, lowest level, which you can't do unless you own the RDBMS. Source access is not enough; you also need the <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x1470c748">knowledge</a>.</p> <p>Technicalities aside, the question is one of proprietary vs. standards-based. This is not only so with software components, where standards have consistently demonstrated benefits, but now also with the data. <a href="http://www.zemanta.com/" id="link-id0x524bea0">Zemanta</a> and <a href="http://www.opencalais.com/" id="link-id0x46132d38">OpenCalais</a> serving <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x13624fb8">DBpedia</a> URIs are examples. Even in entirely closed applications, there is benefit in reusing open vocabularies and identifiers: One does not need to create a secret language for writing a secret memo.</p> <p>Where data is a carrier of value, its value is enhanced by it being easy to repurpose (i.e., standard vocabularies) and to discover (i.e., data set metadata). As on the web, so on the enterprise <a href="http://dbpedia.org/resource/Intranet" id="link-id0xa1392eb8">intranet</a>. In this lies the strength of RDF as opposed to proprietary flexible database schemes. This is a qualitative distinction.</p> <p align="center"> <a href="http://esw.w3.org/topic/SweoIG/TaskForces/CommunityProjects/LinkingOpenData" id="link-id117178a8"><img src="http://www.openlinksw.com/images/logos/LoDLogo.gif" alt="Linking Open Data project logo" /> </a> <br /> <a href="http://dbpedia.org/resource/In_hoc_signo_vinces" id="link-id115f47e8"><i>In hoc signo vinces.</i> </a> </p> <p>In this light, we welcome the <a href="http://semanticweb.org/wiki/VoiD" id="link-id0x12352cc0">voiD</a> (<a href="http://semanticweb.org/wiki/VoiD" id="link-id0x722c18">VOcabulary of Interlinked Data</a>), which is the first promise of making federatable data discoverable. Now that there is a point of focus for these efforts, the needed expressivity will no doubt accrete around the voiD core.</p> <p>For data as a service, we clearly see the value of open terminologies as prerequisites for service interchangeability, i.e., creating a marketplace. <a href="http://dbpedia.org/resource/XML" id="link-id0x2c21c00">XML</a> is for the transaction; RDF is for the discovery, query, and analytics. As with databases in general, first there was the transaction; then there was the query. Same here. For monetizing the query, there are models ranging from renting data sets and server capacity in the clouds to hosted services where one pays for processing past a certain quota. For the hosted case, we just removed a major barrier to offering unlimited query against unlimited data when we completed the <a href="http://www.openlinksw.com/weblog/oerling/?id=1374" id="link-id110b8668">Virtuoso Anytime</a> feature. With this, the user gets what is found within a set time, which is already something, and in case of needing more, one can pay for the usage. Of course, we do not forget advertising. When data has explicit semantics, contextuality is better than with keywords.</p> <p>For these visions to materialize on top of the linked data platform, linked data must join the world of data. This means messaging that is geared towards the database public. They know the problem, but the RDF proposition is still not well enough understood for it to connect.</p> <p>For the relational IT world, we offer passage to the data web and its promise of integration through RDF mapping. We are also bringing out new Microsoft Entity <a href="http://dbpedia.org/resource/ADO.NET_Entity_Framework" id="link-id0x723080">Framework</a> components. This goes in the direction of defining a unified database frontier with RDF and non-RDF entity models side by side.</p> <p>For <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id0x11e1dfc0">OpenLink Software</a>, 2008 was about developing technology for scale, RDF as well as generic relational. We did show a tiny preview with the <a href="http://challenge.semanticweb.org/" id="link-id0x722d08">Billion Triples Challenge</a> demo. Now we are set to come out with the real thing, featuring, among other things, faceted search at the billion triple scale. We <a href="http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?id=1489" id="link-id150c6090">started offering ready-to-go Virtuoso-hosted linked open data sets</a> on Amazon EC2 in December. Now we continue doing this based on our next-generation server, as well as make Virtuoso 6 Cluster commercially available. Technical specifics are amply discussed on this <a href="http://dbpedia.org/resource/Blog" id="link-id0x10fc1930">blog</a>. There are still some new technology things to be developed this year; first among these are strong <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x7fd25590">SPARQL</a> federation, and on-the-fly resizing of server clusters. On the research partnerships side, we have an EU grant for working with the OntoWiki project from the University of Leipzig, and we are partners in DERI's <a href="https://lion.deri.ie/" id="link-id115c02f8">Líon project</a>. These will provide platforms for further demonstrating the "web" in data web, as in web-scale smart databasing.</p> <p>2009 will see change through scale. The things that exist will start interconnecting and there will be emergent value. Deployments will be larger and scale will be readily available through a services model or by installation at one's own facilities. We may see the start of Search becoming Find, like <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id14e43050">Kingsley</a> says, meaning semantics of data guiding search. Entity extraction will multiply data volumes and bring parts of the data web to real time.</p> <p>Exciting 2009 to all.</p>
2009-01-02T13:26:35-05:00
Is Linked Data Always Relevant?
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-12-29#1509
2008-12-29T22:32:00Z
<p>I pose the question above because I stumbled across an interesting claim about <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id1193b2b0">OpenLink Software</a> and its representatives expressed in the <a href="http://www.readwriteweb.com" id="link-id14e4e730">ReadWriteWeb</a> post titled: <a href="http://www.readwriteweb.com/archives/xbrl_mashing_up_financial_statements.php" id="link-id1119ecd8">XBRL: Mashing Up Financial Statements</a>, where the following claim is made:</p> <blockquote> <cite>"..There is evidence that they promote <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11933ec0">LINKED DATA</a> at any expense without understanding the rationale behind other approaches...".</cite> </blockquote> <p> To answer the question above, Linked Data is always relevant as long as we are actually talking about "<a href="http://dbpedia.org/resource/Data" id="link-id0x199ba780">Data</a>" which is simply the case all of the time, irrespective of interaction medium.</p> <p>If XBRL can be disconnected in anyway from Linked Data, I desperately would like to be enlightened (as per my comments to the post). Why wouldn't anyone desire the ability to navigate the linked data inherent in any financial report? Every <a href="http://dbpedia.org/resource/Entity" id="link-id1145b868">entity</a> in an XBRL instance document is an entity, directly or indirectly related to other entities. Why "Mash" the data when you can harmonize XBRL data via a Generic Financial Dictionary (schema or ontology) such that descriptions of Balance Sheet, P&L, and other entities are navigable via their attributes and relationships? In short, why "Mash" (code based brute force joining across disparately shaped data) when you can "Mesh" (natural joining of structured data entities)?</p> <p>"Linked Data" is about the ability to connect all our observations (data)? , perceptions (<a href="http://dbpedia.org/resource/Information" id="link-id11b79e98">information</a>), and inferences / conclusions (<a href="http://dbpedia.org/resource/Knowledge" id="link-id143e0aa8">knowledge</a>) across a spectrum of interaction media. And it just so happens that the RDF data model (<a href="http://en.wikipedia.org/wiki/Entity-Attribute-Value_model" id="link-id114e68b0">Entity-Attribute-Vaue</a> + Class Relationships + HTTP based Object Identifiers), a range of RDF data model serialization formats, and <a href="http://dbpedia.org/resource/SPARQL" id="link-id115bca28">SPARQL</a> (Query Language and <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id0x1c1ef2c8">Web</a> Service combo) actually make this possible, in a manner consistent with the essence of the global space we know as the <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id13dc10d8">World Wide Web</a>.</p> <h3>Related</h3> <ul> <li> <a href="http://www.bbc.co.uk/britainfromabove/stories/visualisations/communication.shtml" id="link-id115f3858">BBC's Britain from Above</a> (core message: Data is Everything).</li> </ul>
2008-12-31T12:57:41-05:00
Bio2Rdf EC2 AMI is now Ready! (Updated)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-12-23#1508
2008-12-23T15:37:45Z
<p>Adding to the collection of Amazon EC2 AMI based knowledgebases already unveiled for <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtEC2AMIDBpediaInstall" id="link-id117a3710">DBpedia</a> and <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtEC2AMINeuroCommonsInstall" id="link-id11293c10">NeuroCommons</a>, we now have a <a href="http://www.bio2rdf.org/" id="link-id14ba6338">Bio2Rdf</a> knowledgebase AMI.</p> <h3>What is Bio2Rdf?</h3> <p>A community developed knowledgebase comprised of Bio Informatics <a href="http://dbpedia.org/resource/Data">data</a> from across <a href="http://www.freebase.com/view/user/bio2rdf/public/sparql" id="link-id1468fb90">30 or so public data sources</a>. The standard deployment of Bio2Rdf includes a a federation of <a href="http://dbpedia.org/resource/SPARQL" id="link-id111962a8">SPARQL</a> endpoints provided by project members and collaborators.</p> <h3>What is the Bio2Rdf EC2 AMI?</h3> <p> An Amazon EC2 hosted variant of the Bio2Rdf knowledgebase. In addition to providing a SPARQL endpoint, the data exposed by the Amazon AMI is published in compliance with <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1560ac50">Linked Data</a> publishing best practices espoused by the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id153aff30">Linking Open Data community</a> (<a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id1390d968">LOD</a>).</p> <h3>Benefits?</h3> <p>The ability to instantiate a personal or service-specific variant of this powerful knowledgebase via the Amazon EC2 Cloud. Instead of a 22+ hour error prone odyssey - you simply get down to the task of data analysis and integration within 1.5 hrs (when setting up you AMI for the first time).</p> <h3>How do I get going?</h3> Just follow the instructions in the <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtEC2AMIBio2rdfInstall" id="link-id114fc4a8">Bio2Rdf EC2 AMI installation guide</a>. <h3>Related</h3> <ul> <li> <a href="http://bio2rdf.wiki.sourceforge.net/" id="link-id19109ed8">Bio2Rdf Wiki</a> </li> <li> <a href="http://bio2rdf.wiki.sourceforge.net/Demo+queries" id="link-id1134c988">Sample Bio2Rdf Queries Collection</a> </li> <li> <a href="http://virtuoso.openlinksw.com/wiki/main/Main/VirtuosoEC2AMI" id="link-id11c28e08">Virtuoso's Amazon Machine Image (AMI) for EC2 Home Page</a> </li> <li> <a href="http://www.youtube.com/watch?v=XdBd14rjcs0" id="link-id14b4a390">Cloud Computing Explanation Video</a> </li> </ul>
2008-12-24T11:05:13-05:00
Virtuoso 6 FAQ directory
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-12-18#1507
2008-12-18T15:46:18Z
<p>We have received various inquiries on high-end metadata stores. I will here go through some salient questions. The requested features include:</p> <ul> <li>Scaling to trillions of triples</li> <li>Running on clusters of commodity servers</li> <li>Running in federated environments, possibly over wide area networks</li> <li>Built-in inference</li> <li>Transactions</li> <li>Security</li> <li>Support for extra triple level metadata, such as security attributes</li> </ul> <p>Q: What is the storage cost per triple? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#StorageCostPerTriple" id="link-id147f61e8">answer</a> </p> <p>Q: What is the cost to insert a triple? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#TripleInsertionCost" id="link-id112e2488">answer</a> </p> <p>Q: What is the cost to delete a triple? (For the insertion itself, as well as for updating any indices) <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#TripleDeletionCost" id="link-id11728528">answer</a> </p> <p>Q: What is the cost to search on a given property? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#PropertySearchCost" id="link-id1586e360">answer</a> </p> <p>Q: What <a href="http://dbpedia.org/resource/Data" id="link-id14688e38">data</a> types are supported? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#SupportedDataTypes" id="link-id1593dbf0">answer</a> </p> <p>Q: What inferencing is supported? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#SupportedInferencing" id="link-id112f3248">answer</a> </p> <p>Q: Is the inferencing dynamic or is an extra step required before inferencing can be used? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#InferencingDynamism" id="link-id1477e2e0">answer</a> </p> <p>Q: Do you support <a href="http://dbpedia.org/resource/Full_text_search" id="link-id1177b198">full text search</a>? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#FullTextSearchSupport" id="link-id1543b170">answer</a> </p> <p>Q: What programming interfaces are supported? Do you support standard <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id14bb69c0">SPARQL protocol</a>? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#SupportedProgrammingInterfaces" id="link-id14d4eb18">answer</a> </p> <p>Q: How can data be partitioned across multiple servers? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#MultipleServerDataPartitioning" id="link-id13722e00">answer</a> </p> <p>Q: How many triples can a single server handle? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#SingleServerTripleLimits" id="link-id14046e58">answer</a> </p> <p>Q: What is the performance impact of going from the billion to the trillion triples? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#PerformanceImpactBillionToTrillion" id="link-id113cfc10">answer</a> </p> <p>Q: Do you support additional metadata for triples, such as timestamps, security tags etc? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#TripleMetadataSupport" id="link-id14c75fa8">answer</a> </p> <p>Q: Should we use <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id11342010">RDF</a> for our large metadata store? What are the alternatives? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#LargeMetadataStoreFormat" id="link-id1478db38">answer</a> </p> <p>Q: How multithreaded is <a href="http://virtuoso.openlinksw.com" id="link-id1651d028">Virtuoso</a>? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#VirtuosoMultiThreading" id="link-id152ad310">answer</a> </p> <p>Q: Can multiple servers run off a single shared disk database? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#MultipleServersOneDiskDatabase" id="link-id14d9d528">answer</a> </p> <p>Q: Can Virtuoso run on a SAN? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#VirtuosoOnSAN" id="link-id111b55d0">answer</a> </p> <p>Q: How does Virtuoso join across partitions? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#CrossPartitionJoins" id="link-id11094db8">answer</a> </p> <p>Q: Does Virtuoso support federated triple stores? If there are multiple <a href="http://dbpedia.org/resource/SPARQL" id="link-id19156b48">SPARQL</a> end points, can Virtuoso be used to do queries joining between these? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#FederatedTripleStoresAndQueries" id="link-id15447ef8">answer</a> </p> <p>Q: How many servers can a cluster contain? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#ClusterServerLimit" id="link-id125fe0d0">answer</a> </p> <p>Q: How do I reconfigure a cluster, adding and removing machines, etc? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#ClusterReconfiguration" id="link-id1150c448">answer</a> </p> <p>Q: How will Virtuoso handle regional clusters? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#RegionalClustering" id="link-id1596ca48">answer</a> </p> <p>Q: Is there a mechanism for terminating long running queries? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#TerminatingLongRunningQueries" id="link-id116bbd60">answer</a> </p> <p>Q: Can the user be asynchronously notified when a long running query terminates? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#AsynchNotificationOfQueryTermination" id="link-id15a59a50">answer</a> </p> <p>Q: How many concurrent queries can Virtuoso handle? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#ConcurrentQueryLimits" id="link-id110a8c00">answer</a> </p> <p>Q: What is the relative performance of SPARQL queries vs. native relational queries <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#RelativePerformanceSparqlVsSql" id="link-id110914f8">answer</a> </p> <p>Q: Does Virtuoso support property tables? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#PropertyTableSupport" id="link-id1581f8c8">answer</a> </p> <p>Q: What performance metrics does Virtuoso offer? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#PerformanceMetricSupport" id="link-id14e92300">answer</a> </p> <p>Q: What support do you provide for concurrency/multithreading operation? Is your interface thread-safe? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#ConcurrencyAndThreadSafety" id="link-id15964b80">answer</a> </p> <p>Q: What level of ACID properties are supported? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#AcidComplianceLevel" id="link-id11035ac0">answer</a> </p> <p>Q: Do you provide the ability to atomically add a set of triples, where either all are added or none are added? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#AtomicTripleInsertion" id="link-id15290e68">answer</a> </p> <p>Q: Do you provide the ability to add a set of triples, respecting the isolation property (so concurrent accessors either see none of the triple values, or all of them)? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#IsolationDuringInsertion" id="link-id15855df0">answer</a> </p> <p>Q: What is the time to start a database, create/open a graph? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#StartupTimes" id="link-id14227f40">answer</a> </p> <p>Q: What sort of security features are built into Virtuoso? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#BuiltInSecurity" id="link-id11927810">answer</a> </p>
2008-12-22T14:30:07.000004-05:00
Virtuoso 6 FAQ directory
http://www.openlinksw.com:443/weblog/oerling/?date=2008-12-18#1506
2008-12-18T15:46:18Z
<p>We have received various inquiries on high-end metadata stores. I will here go through some salient questions. The requested features include:</p> <ul> <li>Scaling to trillions of triples</li> <li>Running on clusters of commodity servers</li> <li>Running in federated environments, possibly over wide area networks</li> <li>Built-in inference</li> <li>Transactions</li> <li>Security</li> <li>Support for extra triple level metadata, such as security attributes</li> </ul> <p>Q: What is the storage cost per triple? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#StorageCostPerTriple" id="link-id147f61e8">answer</a> </p> <p>Q: What is the cost to insert a triple? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#TripleInsertionCost" id="link-id112e2488">answer</a> </p> <p>Q: What is the cost to delete a triple? (For the insertion itself, as well as for updating any indices) <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#TripleDeletionCost" id="link-id11728528">answer</a> </p> <p>Q: What is the cost to search on a given property? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#PropertySearchCost" id="link-id1586e360">answer</a> </p> <p>Q: What <a href="http://dbpedia.org/resource/Data" id="link-id14688e38">data</a> types are supported? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#SupportedDataTypes" id="link-id1593dbf0">answer</a> </p> <p>Q: What inferencing is supported? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#SupportedInferencing" id="link-id112f3248">answer</a> </p> <p>Q: Is the inferencing dynamic or is an extra step required before inferencing can be used? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#InferencingDynamism" id="link-id1477e2e0">answer</a> </p> <p>Q: Do you support <a href="http://dbpedia.org/resource/Full_text_search" id="link-id1177b198">full text search</a>? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#FullTextSearchSupport" id="link-id1543b170">answer</a> </p> <p>Q: What programming interfaces are supported? Do you support standard <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id14bb69c0">SPARQL protocol</a>? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#SupportedProgrammingInterfaces" id="link-id14d4eb18">answer</a> </p> <p>Q: How can data be partitioned across multiple servers? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#MultipleServerDataPartitioning" id="link-id13722e00">answer</a> </p> <p>Q: How many triples can a single server handle? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#SingleServerTripleLimits" id="link-id14046e58">answer</a> </p> <p>Q: What is the performance impact of going from the billion to the trillion triples? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#PerformanceImpactBillionToTrillion" id="link-id113cfc10">answer</a> </p> <p>Q: Do you support additional metadata for triples, such as timestamps, security tags etc? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#TripleMetadataSupport" id="link-id14c75fa8">answer</a> </p> <p>Q: Should we use <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id11342010">RDF</a> for our large metadata store? What are the alternatives? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#LargeMetadataStoreFormat" id="link-id1478db38">answer</a> </p> <p>Q: How multithreaded is <a href="http://virtuoso.openlinksw.com" id="link-id1651d028">Virtuoso</a>? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#VirtuosoMultiThreading" id="link-id152ad310">answer</a> </p> <p>Q: Can multiple servers run off a single shared disk database? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#MultipleServersOneDiskDatabase" id="link-id14d9d528">answer</a> </p> <p>Q: Can Virtuoso run on a SAN? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#VirtuosoOnSAN" id="link-id111b55d0">answer</a> </p> <p>Q: How does Virtuoso join across partitions? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#CrossPartitionJoins" id="link-id11094db8">answer</a> </p> <p>Q: Does Virtuoso support federated triple stores? If there are multiple <a href="http://dbpedia.org/resource/SPARQL" id="link-id19156b48">SPARQL</a> end points, can Virtuoso be used to do queries joining between these? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#FederatedTripleStoresAndQueries" id="link-id15447ef8">answer</a> </p> <p>Q: How many servers can a cluster contain? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#ClusterServerLimit" id="link-id125fe0d0">answer</a> </p> <p>Q: How do I reconfigure a cluster, adding and removing machines, etc? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#ClusterReconfiguration" id="link-id1150c448">answer</a> </p> <p>Q: How will Virtuoso handle regional clusters? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#RegionalClustering" id="link-id1596ca48">answer</a> </p> <p>Q: Is there a mechanism for terminating long running queries? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#TerminatingLongRunningQueries" id="link-id116bbd60">answer</a> </p> <p>Q: Can the user be asynchronously notified when a long running query terminates? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#AsynchNotificationOfQueryTermination" id="link-id15a59a50">answer</a> </p> <p>Q: How many concurrent queries can Virtuoso handle? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#ConcurrentQueryLimits" id="link-id110a8c00">answer</a> </p> <p>Q: What is the relative performance of SPARQL queries vs. native relational queries <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#RelativePerformanceSparqlVsSql" id="link-id110914f8">answer</a> </p> <p>Q: Does Virtuoso support property tables? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#PropertyTableSupport" id="link-id1581f8c8">answer</a> </p> <p>Q: What performance metrics does Virtuoso offer? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#PerformanceMetricSupport" id="link-id14e92300">answer</a> </p> <p>Q: What support do you provide for concurrency/multithreading operation? Is your interface thread-safe? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#ConcurrencyAndThreadSafety" id="link-id15964b80">answer</a> </p> <p>Q: What level of ACID properties are supported? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#AcidComplianceLevel" id="link-id11035ac0">answer</a> </p> <p>Q: Do you provide the ability to atomically add a set of triples, where either all are added or none are added? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#AtomicTripleInsertion" id="link-id15290e68">answer</a> </p> <p>Q: Do you provide the ability to add a set of triples, respecting the isolation property (so concurrent accessors either see none of the triple values, or all of them)? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#IsolationDuringInsertion" id="link-id15855df0">answer</a> </p> <p>Q: What is the time to start a database, create/open a graph? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#StartupTimes" id="link-id14227f40">answer</a> </p> <p>Q: What sort of security features are built into Virtuoso? <a href="http://virtuoso.openlinksw.com/Whitepapers/html/Virt6FAQ.html#BuiltInSecurity" id="link-id11927810">answer</a> </p>
2008-12-22T14:30:00-05:00
Virtuoso RDF: A Getting Started Guide for the Developer
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-12-17#1505
2008-12-17T12:31:34Z
<p>It is a long standing promise of mine to dispel the false impression that using <a href="http://virtuoso.openlinksw.com/" id="link-id113506d0">Virtuoso</a> to work with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id115d9528">RDF</a> is complicated.</p> <p>The purpose of this presentation is to show a programmer how to put RDF into Virtuoso and how to query it. This is done programmatically, with no confusing user interfaces.</p> <p>You should have a Virtuoso Open Source tree built and installed. We will look at the LUBM benchmark demo that comes with the package. All you need is a Unix shell. Running the shell under emacs (<code>m-x shell</code>) is the best. But the open source <code>isql</code> utility should have command line editing also. The emacs shell is however convenient for cutting and pasting things between shell and files.</p> <p>To get started, cd into <code>binsrc/tests/lubm</code>.</p> <p>To verify that this works, you can do </p> <blockquote> <pre>./test_server.sh virtuoso-t</pre></blockquote> <p>This will test the server with the LUBM queries. This should report 45 tests passed. After this we will do the tests step-by-step.</p> <h2>Loading the <a href="http://dbpedia.org/resource/Data" id="link-id10f7bd90">Data</a> </h2> <p>The file <code>lubm-load.sql</code> contains the commands for loading the LUBM single university qualification database.</p> <p>The data files themselves are in <code>lubm_8000</code>, 15 files in RDFXML.</p> <p>There is also a little ontology called <code>inf.nt</code>. This declares the subclass and subproperty relations used in the benchmark.</p> <p>So now let's go through this procedure.</p> <p>Start the server:</p> <blockquote> <pre>$ virtuoso-t -f & </pre></blockquote> <p>This starts the server in foreground mode, and puts it in the background of the shell.</p> <p>Now we connect to it with the isql utility.</p> <blockquote> <pre>$ isql 1111 dba dba </pre></blockquote> <p>This gives a <code>SQL></code> prompt. The default username and password are both <code>dba</code>.</p> <p>When a command is <a href="http://dbpedia.org/resource/SQL" id="link-id1176ce70">SQL</a>, it is entered directly. If it is <a href="http://dbpedia.org/resource/SPARQL" id="link-id156df468">SPARQL</a>, it is prefixed with the keyword <code>sparql</code>. This is how all the SQL clients work. Any SQL client, such as any <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id152d0a00">ODBC</a> or <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id157ad6a0">JDBC</a> application, can use SPARQL if the SQL string starts with this keyword.</p> <p>The <code>lubm-load.sql</code> file is quite self-explanatory. It begins with defining an SQL procedure that calls the RDF/XML load function, <code>DB..RDF_LOAD_RDFXML</code>, for each file in a directory.</p> <p>Next it calls this function for the <code>lubm_8000</code> directory under the server's working directory.</p> <blockquote> <pre>sparql CLEAR GRAPH <lubm>; sparql CLEAR GRAPH <inf>; load_lubm ( server_root() || '/lubm_8000/' ); </pre></blockquote> <p>Then it verifies that the right number of triples is found in the <lubm> graph.</p> <blockquote> <pre>sparql SELECT COUNT(*) FROM <lubm> WHERE { ?x ?y ?z } ; </pre></blockquote> <p>The echo commands below this are interpreted by the isql utility, and produce output to show whether the test was passed. They can be ignored for now.</p> <p>Then it adds some implied <code>subOrganizationOf</code> triples. This is part of setting up the LUBM test database.</p> <blockquote> <pre>sparql PREFIX ub: <http://www.lehigh.edu/~zhp2/2004/0401/univ-bench.owl#> INSERT INTO GRAPH <lubm> { ?x ub:subOrganizationOf ?z } FROM <lubm> WHERE { ?x ub:subOrganizationOf ?y . ?y ub:subOrganizationOf ?z . }; </pre></blockquote> <p>Then it loads the ontology file, <code>inf.nt</code>, using the Turtle load function, <code>DB.DBA.TTLP</code>. The arguments of the function are the text to load, the default namespace prefix, and the <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id15835550">URI</a> of the target graph.</p> <blockquote> <pre>DB.DBA.TTLP ( file_to_string ( 'inf.nt' ), 'http://www.lehigh.edu/~zhp2/2004/0401/univ-bench.owl', 'inf' ) ; sparql SELECT COUNT(*) FROM <inf> WHERE { ?x ?y ?z } ; </pre></blockquote> <p>Then we declare that the triples in the <code><inf></code> graph can be used for inference at run time. To enable this, a SPARQL query will declare that it uses the <code>'inft'</code> rule set. Otherwise this has no effect.</p> <blockquote> <pre>rdfs_rule_set ('inft', 'inf'); </pre></blockquote> <p>This is just a log checkpoint to finalize the work and truncate the transaction log. The server would also eventually do this in its own time.</p> <blockquote> <pre>checkpoint; </pre></blockquote> <p>Now we are ready for querying.</p> <h2>Querying the Data</h2> <p>The queries are given in 3 different versions: The first file, <code>lubm.sql</code>, has the queries with most inference open coded as <code>UNIONs</code>. The second file, <code>lubm-inf.sql</code>, has the inference performed at run time using the ontology <a href="http://dbpedia.org/resource/Information" id="link-id1109faf0">information</a> in the <code><inf></code> graph we just loaded. The last, <code>lubm-phys.sql</code>, relies on having the entailed triples physically present in the <code><lubm></code> graph. These entailed triples are inserted by the SPARUL commands in the <code>lubm-cp.sql</code> file.</p> <p>If you wish to run all the commands in a SQL file, you can type <code>load <filename>;</code> (e.g., <code>load lubm-cp.sql;</code>) at the <code>SQL></code> prompt. If you wish to try individual statements, you can paste them to the command line.</p> <p>For example: </p> <blockquote> <pre>SQL> sparql PREFIX ub: <http://www.lehigh.edu/~zhp2/2004/0401/univ-bench.owl#> SELECT * FROM <lubm> WHERE { ?x a ub:Publication . ?x ub:publicationAuthor <http://www.Department0.University0.edu/AssistantProfessor0> }; VARCHAR _______________________________________________________________________ http://www.Department0.University0.edu/AssistantProfessor0/Publication0 http://www.Department0.University0.edu/AssistantProfessor0/Publication1 http://www.Department0.University0.edu/AssistantProfessor0/Publication2 http://www.Department0.University0.edu/AssistantProfessor0/Publication3 http://www.Department0.University0.edu/AssistantProfessor0/Publication4 http://www.Department0.University0.edu/AssistantProfessor0/Publication5 6 Rows. -- 4 msec. </pre></blockquote> <p>To stop the server, simply type <code>shutdown;</code> at the <code>SQL></code> prompt.</p> <p>If you wish to use a <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id11384668">SPARQL protocol</a> end point, just enable the HTTP listener. This is done by adding a stanza like —</p> <blockquote> <pre>[HTTPServer] ServerPort = 8421 ServerRoot = . ServerThreads = 2 </pre></blockquote> <p>— to the end of the <code>virtuoso.ini</code> file in the <code>lubm</code> directory. Then shutdown and restart (type <code>shutdown;</code> at the <code>SQL></code> prompt and then <code>virtuoso-t -f &</code> at the shell prompt).</p> <p>Now you can connect to the end point with a web browser. The <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id113d02d8">URL</a> is <code>http://localhost:8421/sparql</code>. Without parameters, this will show a human readable form. With parameters, this will execute SPARQL.</p> <p>We have shown how to load and query RDF with Virtuoso using the most basic SQL tools. Next you can access RDF from, for example, <a href="http://dbpedia.org/resource/PHP" id="link-id142d0ba0">PHP</a>, using the PHP ODBC interface.</p> <p>To see how to use <a href="http://jena.sourceforge.net/" id="link-id117074f0">Jena</a> or <a href="http://sourceforge.net/projects/sesame/" id="link-id1103c9b0">Sesame</a> with Virtuoso, look at <a href="http://docs.openlinksw.com/virtuoso/rdfnativestorageproviders.html" id="link-id15488ce8">Native RDF Storage Providers</a>. To see how RDF data types are supported, see <a href="http://docs.openlinksw.com/virtuoso/VirtuosoDriverJDBC.html#jdbcrdf" id="link-id15784a40">Extension datatype for RDF</a> </p> <p>To work with large volumes of data, you must add memory to the configuration file and use the row-autocommit mode, i.e., do <code>log_enable (2);</code> before the load command. Otherwise Virtuoso will do the entire load as a single transaction, and will run out of rollback space. See <a href="http://docs.openlinksw.com/virtuoso/" id="link-id111410f0">documentation</a> for more.</p>
2008-12-17T12:41:27.000006-05:00
Virtuoso RDF: A Getting Started Guide for the Developer
http://www.openlinksw.com:443/weblog/oerling/?date=2008-12-17#1504
2008-12-17T12:31:34Z
<p>It is a long standing promise of mine to dispel the false impression that using <a href="http://virtuoso.openlinksw.com/" id="link-id113506d0">Virtuoso</a> to work with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id115d9528">RDF</a> is complicated.</p> <p>The purpose of this presentation is to show a programmer how to put RDF into Virtuoso and how to query it. This is done programmatically, with no confusing user interfaces.</p> <p>You should have a Virtuoso Open Source tree built and installed. We will look at the LUBM benchmark demo that comes with the package. All you need is a Unix shell. Running the shell under emacs (<code>m-x shell</code>) is the best. But the open source <code>isql</code> utility should have command line editing also. The emacs shell is however convenient for cutting and pasting things between shell and files.</p> <p>To get started, cd into <code>binsrc/tests/lubm</code>.</p> <p>To verify that this works, you can do </p> <blockquote> <pre>./test_server.sh virtuoso-t</pre></blockquote> <p>This will test the server with the LUBM queries. This should report 45 tests passed. After this we will do the tests step-by-step.</p> <h2>Loading the <a href="http://dbpedia.org/resource/Data" id="link-id10f7bd90">Data</a> </h2> <p>The file <code>lubm-load.sql</code> contains the commands for loading the LUBM single university qualification database.</p> <p>The data files themselves are in <code>lubm_8000</code>, 15 files in RDFXML.</p> <p>There is also a little ontology called <code>inf.nt</code>. This declares the subclass and subproperty relations used in the benchmark.</p> <p>So now let's go through this procedure.</p> <p>Start the server:</p> <blockquote> <pre>$ virtuoso-t -f & </pre></blockquote> <p>This starts the server in foreground mode, and puts it in the background of the shell.</p> <p>Now we connect to it with the isql utility.</p> <blockquote> <pre>$ isql 1111 dba dba </pre></blockquote> <p>This gives a <code>SQL></code> prompt. The default username and password are both <code>dba</code>.</p> <p>When a command is <a href="http://dbpedia.org/resource/SQL" id="link-id1176ce70">SQL</a>, it is entered directly. If it is <a href="http://dbpedia.org/resource/SPARQL" id="link-id156df468">SPARQL</a>, it is prefixed with the keyword <code>sparql</code>. This is how all the SQL clients work. Any SQL client, such as any <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id152d0a00">ODBC</a> or <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id157ad6a0">JDBC</a> application, can use SPARQL if the SQL string starts with this keyword.</p> <p>The <code>lubm-load.sql</code> file is quite self-explanatory. It begins with defining an SQL procedure that calls the RDF/XML load function, <code>DB..RDF_LOAD_RDFXML</code>, for each file in a directory.</p> <p>Next it calls this function for the <code>lubm_8000</code> directory under the server's working directory.</p> <blockquote> <pre>sparql CLEAR GRAPH <lubm>; sparql CLEAR GRAPH <inf>; load_lubm ( server_root() || '/lubm_8000/' ); </pre></blockquote> <p>Then it verifies that the right number of triples is found in the <lubm> graph.</p> <blockquote> <pre>sparql SELECT COUNT(*) FROM <lubm> WHERE { ?x ?y ?z } ; </pre></blockquote> <p>The echo commands below this are interpreted by the isql utility, and produce output to show whether the test was passed. They can be ignored for now.</p> <p>Then it adds some implied <code>subOrganizationOf</code> triples. This is part of setting up the LUBM test database.</p> <blockquote> <pre>sparql PREFIX ub: <http://www.lehigh.edu/~zhp2/2004/0401/univ-bench.owl#> INSERT INTO GRAPH <lubm> { ?x ub:subOrganizationOf ?z } FROM <lubm> WHERE { ?x ub:subOrganizationOf ?y . ?y ub:subOrganizationOf ?z . }; </pre></blockquote> <p>Then it loads the ontology file, <code>inf.nt</code>, using the Turtle load function, <code>DB.DBA.TTLP</code>. The arguments of the function are the text to load, the default namespace prefix, and the <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id15835550">URI</a> of the target graph.</p> <blockquote> <pre>DB.DBA.TTLP ( file_to_string ( 'inf.nt' ), 'http://www.lehigh.edu/~zhp2/2004/0401/univ-bench.owl', 'inf' ) ; sparql SELECT COUNT(*) FROM <inf> WHERE { ?x ?y ?z } ; </pre></blockquote> <p>Then we declare that the triples in the <code><inf></code> graph can be used for inference at run time. To enable this, a SPARQL query will declare that it uses the <code>'inft'</code> rule set. Otherwise this has no effect.</p> <blockquote> <pre>rdfs_rule_set ('inft', 'inf'); </pre></blockquote> <p>This is just a log checkpoint to finalize the work and truncate the transaction log. The server would also eventually do this in its own time.</p> <blockquote> <pre>checkpoint; </pre></blockquote> <p>Now we are ready for querying.</p> <h2>Querying the Data</h2> <p>The queries are given in 3 different versions: The first file, <code>lubm.sql</code>, has the queries with most inference open coded as <code>UNIONs</code>. The second file, <code>lubm-inf.sql</code>, has the inference performed at run time using the ontology <a href="http://dbpedia.org/resource/Information" id="link-id1109faf0">information</a> in the <code><inf></code> graph we just loaded. The last, <code>lubm-phys.sql</code>, relies on having the entailed triples physically present in the <code><lubm></code> graph. These entailed triples are inserted by the SPARUL commands in the <code>lubm-cp.sql</code> file.</p> <p>If you wish to run all the commands in a SQL file, you can type <code>load <filename>;</code> (e.g., <code>load lubm-cp.sql;</code>) at the <code>SQL></code> prompt. If you wish to try individual statements, you can paste them to the command line.</p> <p>For example: </p> <blockquote> <pre>SQL> sparql PREFIX ub: <http://www.lehigh.edu/~zhp2/2004/0401/univ-bench.owl#> SELECT * FROM <lubm> WHERE { ?x a ub:Publication . ?x ub:publicationAuthor <http://www.Department0.University0.edu/AssistantProfessor0> }; VARCHAR _______________________________________________________________________ http://www.Department0.University0.edu/AssistantProfessor0/Publication0 http://www.Department0.University0.edu/AssistantProfessor0/Publication1 http://www.Department0.University0.edu/AssistantProfessor0/Publication2 http://www.Department0.University0.edu/AssistantProfessor0/Publication3 http://www.Department0.University0.edu/AssistantProfessor0/Publication4 http://www.Department0.University0.edu/AssistantProfessor0/Publication5 6 Rows. -- 4 msec. </pre></blockquote> <p>To stop the server, simply type <code>shutdown;</code> at the <code>SQL></code> prompt.</p> <p>If you wish to use a <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id11384668">SPARQL protocol</a> end point, just enable the HTTP listener. This is done by adding a stanza like —</p> <blockquote> <pre>[HTTPServer] ServerPort = 8421 ServerRoot = . ServerThreads = 2 </pre></blockquote> <p>— to the end of the <code>virtuoso.ini</code> file in the <code>lubm</code> directory. Then shutdown and restart (type <code>shutdown;</code> at the <code>SQL></code> prompt and then <code>virtuoso-t -f &</code> at the shell prompt).</p> <p>Now you can connect to the end point with a web browser. The <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id113d02d8">URL</a> is <code>http://localhost:8421/sparql</code>. Without parameters, this will show a human readable form. With parameters, this will execute SPARQL.</p> <p>We have shown how to load and query RDF with Virtuoso using the most basic SQL tools. Next you can access RDF from, for example, <a href="http://dbpedia.org/resource/PHP" id="link-id142d0ba0">PHP</a>, using the PHP ODBC interface.</p> <p>To see how to use <a href="http://jena.sourceforge.net/" id="link-id117074f0">Jena</a> or <a href="http://sourceforge.net/projects/sesame/" id="link-id1103c9b0">Sesame</a> with Virtuoso, look at <a href="http://docs.openlinksw.com/virtuoso/rdfnativestorageproviders.html" id="link-id15488ce8">Native RDF Storage Providers</a>. To see how RDF data types are supported, see <a href="http://docs.openlinksw.com/virtuoso/VirtuosoDriverJDBC.html#jdbcrdf" id="link-id15784a40">Extension datatype for RDF</a> </p> <p>To work with large volumes of data, you must add memory to the configuration file and use the row-autocommit mode, i.e., do <code>log_enable (2);</code> before the load command. Otherwise Virtuoso will do the entire load as a single transaction, and will run out of rollback space. See <a href="http://docs.openlinksw.com/virtuoso/" id="link-id111410f0">documentation</a> for more.</p>
2008-12-17T12:41:21.000001-05:00
See the Lite: Embeddable/Background Virtuoso starts at 25MB
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-12-17#1503
2008-12-17T09:34:12Z
<p>We have received many requests for an embeddable-scale <a href="http://virtuoso.openlinksw.com" id="link-id0x1cd69650">Virtuoso</a>. In response to this, we have added a Lite mode, where the initial size of a server process is a tiny fraction of what the initial size would be with default settings. With 2MB of disk cache buffers (ini file setting, <code>NumberOfBuffers = 256</code>), the process size stays under 30MB on 32-bit Linux.</p> <p>The value of this is that one can now have <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1ce89340">RDF</a> and full text indexing on the desktop without running a Java VM or any other memory-intensive software. And of course, all of <a href="http://dbpedia.org/resource/SQL" id="link-id0x1cfc9288">SQL</a> (transactions, stored procedures, etc.) is in the same embeddably-sized container.</p> <p>The Lite executable is a full Virtuoso executable; the Lite mode is controlled by a switch in the configuration file. The executable size is about 10MB for 32-bit Linux. A database created in the Lite mode will be converted into a fully-featured database (tables and indexes are added, among other things) if the server is started with the Lite setting "off"; functionality can be reverted to Lite mode, though it will now consume somewhat more memory, etc.</p> <p>Lite mode offers full SQL and <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1c511da8">SPARQL</a>/SPARUL (via SPASQL), but disables all <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1dac1950">HTTP</a>-based services (WebDAV, application hosting, etc.). Clients can still use all typical database access mechanisms (i.e., <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0xb19a488">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x1d93ee40">JDBC</a>, OLE-DB, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id0x1ce391c0">ADO</a>.<a href="http://dbpedia.org/resource/.NET_Framework" id="link-id0xacf1168">NET</a>, and XMLA) to connect, including the <a href="http://jena.sourceforge.net/" id="link-id0xaaf5b58">Jena</a> and <a href="http://sourceforge.net/projects/sesame/" id="link-id0x1b1e4328">Sesame</a> frameworks for RDF. ODBC now offers full support of RDF <a href="http://dbpedia.org/resource/Data" id="link-id0x1cfc9f78">data</a> types for <a href="http://dbpedia.org/resource/C%2B%2B" id="link-id0xa6059d8">C</a>-based clients. A Redland-compatible API also exists, for use with Redland v1.0.8 and later. </p> <p>Especially for embedded use, we now allow restricting the listener to be a Unix socket, which allows client connections only from the localhost.</p> <p>Shipping an embedded Virtuoso is easy. It just takes one executable and one configuration file. Performance is generally comparable to "normal" mode, except that Lite will be somewhat less scalable on multicore systems.</p> <p>The Lite mode will be included in the next Virtuoso 5 Open Source release.</p>
2008-12-17T12:03:49-05:00
See the Lite: Embeddable/Background Virtuoso starts at 25MB
http://www.openlinksw.com:443/weblog/oerling/?date=2008-12-17#1502
2008-12-17T09:34:12Z
<p>We have received many requests for an embeddable-scale <a href="http://virtuoso.openlinksw.com" id="link-id0xa5aa1b38">Virtuoso</a>. In response to this, we have added a Lite mode, where the initial size of a server process is a tiny fraction of what the initial size would be with default settings. With 2MB of disk cache buffers (ini file setting, <code>NumberOfBuffers = 256</code>), the process size stays under 30MB on 32-bit Linux.</p> <p>The value of this is that one can now have <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1db79ac8">RDF</a> and full text indexing on the desktop without running a Java VM or any other memory-intensive software. And of course, all of <a href="http://dbpedia.org/resource/SQL" id="link-id0xa923298">SQL</a> (transactions, stored procedures, etc.) is in the same embeddably-sized container.</p> <p>The Lite executable is a full Virtuoso executable; the Lite mode is controlled by a switch in the configuration file. The executable size is about 10MB for 32-bit Linux. A database created in the Lite mode will be converted into a fully-featured database (tables and indexes are added, among other things) if the server is started with the Lite setting "off"; functionality can be reverted to Lite mode, though it will now consume somewhat more memory, etc.</p> <p>Lite mode offers full SQL and <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1b388830">SPARQL</a>/SPARUL (via SPASQL), but disables all <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1d56b618">HTTP</a>-based services (WebDAV, application hosting, etc.). Clients can still use all typical database access mechanisms (i.e., <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x1c5abc38">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x1dade1f8">JDBC</a>, OLE-DB, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id0x25d8e0f0">ADO</a>.<a href="http://dbpedia.org/resource/.NET_Framework" id="link-id0x1d7a1a28">NET</a>, and XMLA) to connect, including the <a href="http://jena.sourceforge.net/" id="link-id0x1d929b98">Jena</a> and <a href="http://sourceforge.net/projects/sesame/" id="link-id0x1b7a9088">Sesame</a> frameworks for RDF. ODBC now offers full support of RDF <a href="http://dbpedia.org/resource/Data" id="link-id0xaf62aa0">data</a> types for <a href="http://dbpedia.org/resource/C%2B%2B" id="link-id0xa8784b0">C</a>-based clients. A Redland-compatible API also exists, for use with Redland v1.0.8 and later. </p> <p>Especially for embedded use, we now allow restricting the listener to be a Unix socket, which allows client connections only from the localhost.</p> <p>Shipping an embedded Virtuoso is easy. It just takes one executable and one configuration file. Performance is generally comparable to "normal" mode, except that Lite will be somewhat less scalable on multicore systems.</p> <p>The Lite mode will be included in the next Virtuoso 5 Open Source release.</p>
2008-12-17T12:03:43-05:00
SPARQL and Scalable Inference on Demand paper
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-12-16#1501
2008-12-16T19:00:41Z
<p>We recently submitted a paper, <i><a href="http://www.openlinksw.com/weblog/oerling/scalable_inference.pdf" id="link-id115e1618">SPARQL and Scalable Inference on Demand</a></i>, for next year's European <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id14c7cf60">Semantic Web</a> Conference, <a href="http://www.eswc2009.org/" id="link-id115004c8">ESWC 2009</a>.</p>
2008-12-16T16:11:48.000002-05:00
SPARQL and Scalable Inference on Demand paper
http://www.openlinksw.com:443/weblog/oerling/?date=2008-12-16#1500
2008-12-16T19:00:41Z
<p>We recently submitted a paper, <i><a href="http://www.openlinksw.com/weblog/oerling/scalable_inference.pdf" id="link-id115e1618">SPARQL and Scalable Inference on Demand</a></i>, for next year's European <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id14c7cf60">Semantic Web</a> Conference, <a href="http://www.eswc2009.org/" id="link-id115004c8">ESWC 2009</a>.</p>
2008-12-16T16:11:42.000002-05:00
"E Pluribus Unum", or "Inversely Functional Identity", or "Smooshing Without the Stickiness" (re-updated)
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-12-16#1499
2008-12-16T14:14:43Z
<p>What a terrible word, smooshing... I have understood it to mean that when you have two names for one thing, you give each all the attributes of the other. This smooshes them together, makes them interchangeable.</p> <p>This is complex, so I will begin with the point and the interested may read on for the details and implications. Starting with soon to be released version 6, <a href="http://virtuoso.openlinksw.com" id="link-id15718cb8">Virtuoso</a> allows you to say that two things, if they share a uniquely identifying property, are the same. Examples of uniquely identifying properties would be a book's ISBN number, or a person's social security plus full name. In relational language this is a <i>unique key</i>, and in <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id145ed998">RDF</a> parlance, an <i>inverse functional property</i>.</p> <p>In most systems, such problems are dealt with as a preprocessing step before querying. For example, all the items that are considered the same will get the same properties or at load time all identifiers will be normalized according to some application rules. This is good if the rules are clear and understood. This is so in closed situations, where things tend to have standard identifiers to begin with. But on the open web this is not so clear cut.</p> <p>In this post, we show how to do these things <i>ad hoc</i>, without materializing anything. At the end, we also show how to materialize identity and what the consequences of this are with open web <a href="http://dbpedia.org/resource/Data" id="link-id11726358">data</a>. We use real live web crawls from the <a href="http://challenge.semanticweb.org/" id="link-id14f40448">Billion Triples Challenge</a> data set.</p> <p>On the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id156e2b10">linked data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id1106ce08">web</a>, there are independently arising descriptions of the same thing and thus arises the need to smoosh, if these are to be somehow integrated. But this is only the beginning of the problems.</p> <p>To address these, we have added the option of specifying that some property will be considered inversely functional in a query. This is done at run time and the property does not really have to be inversely functional in the pure sense. <code>foaf:name</code> will do for an example. This simply means that for purposes of the query concerned, two subjects which have at least one <code>foaf:name</code> in common are considered the same. In this way, we can join between FOAF files. With the same database, a query about music preferences might consider having the same name as "same enough," but a query about criminal prosecution would obviously need to be more precise about sameness.</p> <p>Our ontology is defined like this:</p> <blockquote> <pre>-- Populate a named graph with the triples you want to use in query time inferencing<br /> ttlp ( ' @prefix foaf: <xmlns="http" xmlns.com="xmlns.com" foaf="foaf"> </> @prefix owl: <xmlns="http" www.w3.org="www.w3.org" owl="owl"> </> foaf:mbox_sha1sum a owl:InverseFunctionalProperty . foaf:name a owl:InverseFunctionalProperty . ', 'xx', 'b3sifp' );<br /> -- Declare that the graph contains an ontology for use in query time inferencing <br /> rdfs_rule_set ( 'http://example.com/rules/b3sifp#', 'b3sifp' ); </pre></blockquote> <p>Then use it:</p> <blockquote> <pre>sparql DEFINE input:inference "http://example.com/rules/b3sifp#" SELECT DISTINCT ?k ?f1 ?f2 WHERE { ?k foaf:name ?n . ?n bif:contains "'Kjetil Kjernsmo'" . ?k foaf:knows ?f1 . ?f1 foaf:knows ?f2 };<br /> VARCHAR VARCHAR VARCHAR ______________________________________ _______________________________________________ ______________________________<br /> http://www.kjetil.kjernsmo.net/foaf#me http://norman.walsh.name/knows/who/robin-berjon http://twitter.com/dajobe http://www.kjetil.kjernsmo.net/foaf#me http://norman.walsh.name/knows/who/robin-berjon http://twitter.com/net_twitter http://www.kjetil.kjernsmo.net/foaf#me http://norman.walsh.name/knows/who/robin-berjon http://twitter.com/amyvdh http://www.kjetil.kjernsmo.net/foaf#me http://norman.walsh.name/knows/who/robin-berjon http://twitter.com/pom http://www.kjetil.kjernsmo.net/foaf#me http://norman.walsh.name/knows/who/robin-berjon http://twitter.com/mattb http://www.kjetil.kjernsmo.net/foaf#me http://norman.walsh.name/knows/who/robin-berjon http://twitter.com/davorg http://www.kjetil.kjernsmo.net/foaf#me http://norman.walsh.name/knows/who/robin-berjon http://twitter.com/distobj http://www.kjetil.kjernsmo.net/foaf#me http://norman.walsh.name/knows/who/robin-berjon http://twitter.com/perigrin .... </pre></blockquote> <p>Without the inference, we get no matches. This is because the data in question has one graph per FOAF file, and blank nodes for persons. No graph references any person outside the ones in the graph. So if somebody is mentioned as known, then without the inference there is no way to get to what that person's FOAF file says, since the same individual will be a different blank node there. The declaration in the context named <code>b3sifp</code> just means that all things with a matching <code>foaf:name</code> or <code>foaf:mbox_sha1sum</code> are the same.</p> <p>Sameness means that two are the same for purposes of <code>DISTINCT</code> or <code>GROUP BY</code>, and if two are the same, then both have the <code>UNION</code> of all of the properties of both.</p> <p>If this were a naive smoosh, then the individuals would have all the same properties but would not be the same for <code>DISTINCT</code>.</p> <p>If we have complex application rules for determining whether individuals are the same, then one can materialize <code>owl:sameAs</code> triples and keep them in a separate graph. In this way, the original data is not contaminated and the materialized volume stays reasonable — nothing like the blow-up of duplicating properties across instances.</p> <p>The pro-smoosh argument is that if every duplicate makes exactly the same statements, then there is no great blow-up. Best and worst cases will always depend on the data. In rough terms, the more <i>ad hoc</i> the use, the less desirable the materialization. If the usage pattern is really set, then a relational-style application-specific representation with identity resolved at load time will perform best. We can do that too, but so can others.</p> <p>The principal point is about agility as concerns the inference. Run time is more agile than materialization, and if the rules change or if different users have different needs, then materialization runs into trouble. When talking web scale, having multiple users is a given; it is very uneconomical to give everybody their own copy, and the likelihood of a user accessing any significant part of the corpus is minimal. Even if the queries were not limited, the user would typically not wait for the answer of a query doing a scan or aggregation over 1 billion <a href="http://dbpedia.org/resource/Blog" id="link-id1156a550">blog</a> posts or something of the sort. So queries will typically be selective. Selective means that they do not access all of the data, hence do not benefit from ready-made materialization for things they do not even look at. </p> <p>The exception is corpus-wide statistics queries. But these will not be done in interactive time anyway, and will not be done very often. Plus, since these do not typically run all in memory, these are disk bound. And when things are disk bound, size matters. Reading extra entailment on the way is just a performance penalty.</p> <p>Enough talk. Time for an experiment. We take the Yahoo and Falcon web crawls from the Billion Triples Challenge set, and do two things with the FOAF data in them:</p> <ol> <li>Resolve identity at insert time. We remove duplicate person URIs, and give the single <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id11317008">URI</a> all the properties of all the duplicate URIs. We expect these to be most often repeats. If a person references another person, we normalize this reference to go to the single URI of the referenced person.</li> <li>Give every duplicate URI of a person all the properties of all the duplicates. If these are the same value, the data should not get much bigger, or so we think.</li> </ol> <p>For the experiment, we will consider two people the same if they have the same <code>foaf:name</code> and are both instances of <code>foaf:Person</code>. This gets some extra hits but should not be statistically significant.</p> <p>The following is a commented <a href="http://dbpedia.org/resource/SQL" id="link-id110945b0">SQL</a> script performing the smoosh. We play with internal IDs of things, thus some of these operations cannot be done in SPARQL alone. We use SPARQL where possible for readability. As the documentation states, <code>iri_to_id</code> converts from the qualified name of an IRI to its ID and <code>id_to_iri</code> does the reverse.</p> <p>We count the triples that enter into the smoosh:</p> <blockquote> <pre>-- the name is an existence because else we'd get several times more due to -- the names occurring in many graphs <br /> sparql SELECT COUNT(*) WHERE { { SELECT DISTINCT ?person WHERE { ?person a foaf:Person } } . FILTER ( bif:exists ( SELECT (1) WHERE { ?person foaf:name ?nn } ) ) . ?person ?p ?o };<br /> -- We get 3284674 </pre></blockquote> <p>We make a few tables for intermediate results.</p> <blockquote> <pre>-- For each distinct name, gather the properties and objects from -- all subjects with this name <br /> CREATE TABLE name_prop ( np_name ANY, np_p IRI_ID_8, np_o ANY, PRIMARY KEY ( np_name, np_p, np_o ) ); ALTER INDEX name_prop ON name_prop PARTITION ( np_name VARCHAR (-1, 0hexffff) );<br /> -- Map from name to canonical IRI used for the name <br /> CREATE TABLE name_iri ( ni_name ANY PRIMARY KEY, ni_s IRI_ID_8 ); ALTER INDEX name_iri ON name_iri PARTITION ( ni_name VARCHAR (-1, 0hexffff) );<br /> -- Map from person IRI to canonical person IRI<br /> CREATE TABLE pref_iri ( i IRI_ID_8, pref IRI_ID_8, PRIMARY KEY ( i ) ); ALTER INDEX pref_iri ON pref_iri PARTITION ( i INT (0hexffff00) );<br /> -- a table for the materialization where all aliases get all properties of every other <br /> CREATE TABLE smoosh_ct ( s IRI_ID_8, p IRI_ID_8, o ANY, PRIMARY KEY ( s, p, o ) ); ALTER INDEX smoosh_ct ON smoosh_ct PARTITION ( s INT (0hexffff00) );<br /> -- disable transaction log and enable row auto-commit. This is necessary, otherwise -- bulk operations are done transactionally and they will run out of rollback space.<br /> LOG_ENABLE (2);<br /> -- Gather all the properties of all persons with a name under that name. -- INSERT SOFT means that duplicates are ignored <br /> INSERT SOFT name_prop SELECT "n", "p", "o" FROM ( sparql DEFINE output:valmode "LONG" SELECT ?n ?p ?o WHERE { ?x a foaf:Person . ?x foaf:name ?n . ?x ?p ?o } ) xx ;<br /> -- Now choose for each name the canonical IRI <br /> INSERT INTO name_iri SELECT np_name, ( SELECT MIN (s) FROM rdf_quad WHERE o = np_name AND p = IRI_TO_ID ('http://xmlns.com/foaf/0.1/name') ) AS mini FROM name_prop WHERE np_p = IRI_TO_ID ('http://xmlns.com/foaf/0.1/name') ;<br /> -- For each person IRI, map to the canonical IRI of that person <br /> INSERT SOFT pref_iri (i, pref) SELECT s, ni_s FROM name_iri, rdf_quad WHERE o = ni_name AND p = IRI_TO_ID ('http://xmlns.com/foaf/0.1/name') ;<br /> -- Make a graph where all persons have one iri with all the properties of all aliases -- and where person-to-person refs are canonicalized<br /> INSERT SOFT rdf_quad (g,s,p,o) SELECT IRI_TO_ID ('psmoosh'), ni_s, np_p, COALESCE ( ( SELECT pref FROM pref_iri WHERE i = np_o ), np_o ) FROM name_prop, name_iri WHERE ni_name = np_name OPTION ( loop, quietcast ) ;<br /> -- A little explanation: The properties of names are copied into rdf_quad with the name -- replaced with its canonical IRI. If the object has a canonical IRI, this is used as -- the object, else the object is unmodified. This is the COALESCE with the sub-query.<br /> -- This takes a little time. To check on the progress, take another connection to the -- server and do <br /> STATUS ('cluster');<br /> -- It will return something like -- Cluster 4 nodes, 35 s. 108 m/s 1001 KB/s 75% cpu 186% read 12% clw threads 5r 0w 0i -- buffers 549481 253929 d 8 w 0 pfs<br /> -- Now finalize the state; this makes it permanent. Else the work will be lost on server -- failure, since there was no transaction log <br /> CL_EXEC ('checkpoint');<br /> -- See what we got<br /> sparql SELECT COUNT (*) FROM <psmoosh> WHERE {?s ?p ?o};<br /> -- This is 2253102<br /> -- Now make the copy where all have the properties of all synonyms. This takes so much -- space we do not insert it as RDF quads, but make a special table for it so that we can -- run some statistics. This saves time.<br /> INSERT SOFT smoosh_ct (s, p, o) SELECT s, np_p, np_o FROM name_prop, rdf_quad WHERE o = np_name AND p = IRI_TO_ID ('http://xmlns.com/foaf/0.1/name') ;<br /> -- as above, INSERT SOFT so as to ignore duplicates <br /> SELECT COUNT (*) FROM smoosh_ct;<br /> -- This is 167360324<br /> -- Find out where the bloat comes from <br /> SELECT TOP 20 COUNT (*), ID_TO_IRI (p) FROM smoosh_ct GROUP BY p ORDER BY 1 DESC; </pre></blockquote> <p>The results are:</p> <blockquote> <pre>54728777 http://www.w3.org/2002/07/owl#sameAs 48543153 http://xmlns.com/foaf/0.1/knows 13930234 http://www.w3.org/2000/01/rdf-schema#seeAlso 12268512 http://xmlns.com/foaf/0.1/interest 11415867 http://xmlns.com/foaf/0.1/nick 6683963 http://xmlns.com/foaf/0.1/weblog 6650093 http://xmlns.com/foaf/0.1/depiction 4231946 http://xmlns.com/foaf/0.1/mbox_sha1sum 4129629 http://xmlns.com/foaf/0.1/homepage 1776555 http://xmlns.com/foaf/0.1/holdsAccount 1219525 http://xmlns.com/foaf/0.1/based_near 305522 http://www.w3.org/1999/02/22-rdf-syntax-ns#type 274965 http://xmlns.com/foaf/0.1/name 155131 http://xmlns.com/foaf/0.1/dateOfBirth 153001 http://xmlns.com/foaf/0.1/img 111130 http://www.w3.org/2001/vcard-rdf/3.0#ADR 52930 http://xmlns.com/foaf/0.1/gender 48517 http://www.w3.org/2004/02/skos/core#subject 45697 http://www.w3.org/2000/01/rdf-schema#label 44860 http://purl.org/vocab/bio/0.1/olb </pre></blockquote> <p>Now compare with the predicate distribution of the smoosh with identities canonicalized </p> <blockquote> <pre>sparql SELECT COUNT (*) ?p FROM <psmoosh> WHERE { ?s ?p ?o } GROUP BY ?p ORDER BY 1 DESC LIMIT 20;</pre></blockquote> <p>Results are:</p> <blockquote> <pre>748311 http://xmlns.com/foaf/0.1/knows 548391 http://xmlns.com/foaf/0.1/interest 140531 http://www.w3.org/2000/01/rdf-schema#seeAlso 105273 http://www.w3.org/1999/02/22-rdf-syntax-ns#type 78497 http://xmlns.com/foaf/0.1/name 48099 http://www.w3.org/2004/02/skos/core#subject 45179 http://xmlns.com/foaf/0.1/depiction 40229 http://www.w3.org/2000/01/rdf-schema#comment 38272 http://www.w3.org/2000/01/rdf-schema#label 37378 http://xmlns.com/foaf/0.1/nick 37186 http://dbpedia.org/property/abstract 34003 http://xmlns.com/foaf/0.1/img 26182 http://xmlns.com/foaf/0.1/homepage 23795 http://www.w3.org/2002/07/owl#sameAs 17651 http://xmlns.com/foaf/0.1/mbox_sha1sum 17430 http://xmlns.com/foaf/0.1/dateOfBirth 15586 http://xmlns.com/foaf/0.1/page 12869 http://dbpedia.org/property/reference 12497 http://xmlns.com/foaf/0.1/weblog 12329 http://blogs.yandex.ru/schema/foaf/school </pre></blockquote> <p>We can drop the <code>owl:sameAs</code> triples from the count, so the bloat is a bit less by that but it still is tens of times larger than the canonicalized copy or the initial state.</p> <p>Now, when we try using the psmoosh graph, we still get different results from the results with the original data. This is because <code>foaf:knows</code> relations to things with no <code>foaf:name</code> are not represented in the smoosh. The exist:</p> <blockquote> <pre>sparql SELECT COUNT (*) WHERE { ?s foaf:knows ?thing . FILTER ( !bif:exists ( SELECT (1) WHERE { ?thing foaf:name ?nn } ) ) };<br /> -- 1393940 </pre></blockquote> <p>So the smoosh graph is not an accurate rendition of the social network. It would have to be smooshed further to be that, since the data in the sample is quite irregular. But we do not go that far here.</p> <p>Finally, we calculate the smoosh blow up factors. We do not include <code>owl:sameAs</code> triples in the counts.</p> <blockquote> <pre>select (167360324 - 54728777) / 3284674.0; 34.290022997716059<br /> select 2229307 / 3284674.0; = 0.678699621332284 </pre></blockquote> <p>So, to get a smoosh that is not really the equivalent of the original, either multiply the original triple count by 34 or 0.68, depending on whether synonyms are collapsed or not.</p> <p>Making the smooshes does not take very long, some minutes for the small one. Inserting the big one would be longer, a couple of hours maybe. It was 33 minutes for filling the <code>smoosh_ct</code> table. The metrics were not with optimal tuning so the performance numbers just serve to show that smooshing takes time. Probably more time than allowable in an interactive situation, no matter how the process is optimized.</p>
2008-12-16T15:01:36.000003-05:00
"E Pluribus Unum", or "Inversely Functional Identity", or "Smooshing Without the Stickiness" (re-updated)
http://www.openlinksw.com:443/weblog/oerling/?date=2008-12-16#1498
2008-12-16T14:14:43Z
<p>What a terrible word, smooshing... I have understood it to mean that when you have two names for one thing, you give each all the attributes of the other. This smooshes them together, makes them interchangeable.</p> <p>This is complex, so I will begin with the point and the interested may read on for the details and implications. Starting with soon to be released version 6, <a href="http://virtuoso.openlinksw.com" id="link-id15718cb8">Virtuoso</a> allows you to say that two things, if they share a uniquely identifying property, are the same. Examples of uniquely identifying properties would be a book's ISBN number, or a person's social security plus full name. In relational language this is a <i>unique key</i>, and in <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id145ed998">RDF</a> parlance, an <i>inverse functional property</i>.</p> <p>In most systems, such problems are dealt with as a preprocessing step before querying. For example, all the items that are considered the same will get the same properties or at load time all identifiers will be normalized according to some application rules. This is good if the rules are clear and understood. This is so in closed situations, where things tend to have standard identifiers to begin with. But on the open web this is not so clear cut.</p> <p>In this post, we show how to do these things <i>ad hoc</i>, without materializing anything. At the end, we also show how to materialize identity and what the consequences of this are with open web <a href="http://dbpedia.org/resource/Data" id="link-id11726358">data</a>. We use real live web crawls from the <a href="http://challenge.semanticweb.org/" id="link-id14f40448">Billion Triples Challenge</a> data set.</p> <p>On the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id156e2b10">linked data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id1106ce08">web</a>, there are independently arising descriptions of the same thing and thus arises the need to smoosh, if these are to be somehow integrated. But this is only the beginning of the problems.</p> <p>To address these, we have added the option of specifying that some property will be considered inversely functional in a query. This is done at run time and the property does not really have to be inversely functional in the pure sense. <code>foaf:name</code> will do for an example. This simply means that for purposes of the query concerned, two subjects which have at least one <code>foaf:name</code> in common are considered the same. In this way, we can join between FOAF files. With the same database, a query about music preferences might consider having the same name as "same enough," but a query about criminal prosecution would obviously need to be more precise about sameness.</p> <p>Our ontology is defined like this:</p> <blockquote> <pre>-- Populate a named graph with the triples you want to use in query time inferencing<br /> ttlp ( ' @prefix foaf: <xmlns="http" xmlns.com="xmlns.com" foaf="foaf"> </> @prefix owl: <xmlns="http" www.w3.org="www.w3.org" owl="owl"> </> foaf:mbox_sha1sum a owl:InverseFunctionalProperty . foaf:name a owl:InverseFunctionalProperty . ', 'xx', 'b3sifp' );<br /> -- Declare that the graph contains an ontology for use in query time inferencing <br /> rdfs_rule_set ( 'http://example.com/rules/b3sifp#', 'b3sifp' ); </pre></blockquote> <p>Then use it:</p> <blockquote> <pre>sparql DEFINE input:inference "http://example.com/rules/b3sifp#" SELECT DISTINCT ?k ?f1 ?f2 WHERE { ?k foaf:name ?n . ?n bif:contains "'Kjetil Kjernsmo'" . ?k foaf:knows ?f1 . ?f1 foaf:knows ?f2 };<br /> VARCHAR VARCHAR VARCHAR ______________________________________ _______________________________________________ ______________________________<br /> http://www.kjetil.kjernsmo.net/foaf#me http://norman.walsh.name/knows/who/robin-berjon http://twitter.com/dajobe http://www.kjetil.kjernsmo.net/foaf#me http://norman.walsh.name/knows/who/robin-berjon http://twitter.com/net_twitter http://www.kjetil.kjernsmo.net/foaf#me http://norman.walsh.name/knows/who/robin-berjon http://twitter.com/amyvdh http://www.kjetil.kjernsmo.net/foaf#me http://norman.walsh.name/knows/who/robin-berjon http://twitter.com/pom http://www.kjetil.kjernsmo.net/foaf#me http://norman.walsh.name/knows/who/robin-berjon http://twitter.com/mattb http://www.kjetil.kjernsmo.net/foaf#me http://norman.walsh.name/knows/who/robin-berjon http://twitter.com/davorg http://www.kjetil.kjernsmo.net/foaf#me http://norman.walsh.name/knows/who/robin-berjon http://twitter.com/distobj http://www.kjetil.kjernsmo.net/foaf#me http://norman.walsh.name/knows/who/robin-berjon http://twitter.com/perigrin .... </pre></blockquote> <p>Without the inference, we get no matches. This is because the data in question has one graph per FOAF file, and blank nodes for persons. No graph references any person outside the ones in the graph. So if somebody is mentioned as known, then without the inference there is no way to get to what that person's FOAF file says, since the same individual will be a different blank node there. The declaration in the context named <code>b3sifp</code> just means that all things with a matching <code>foaf:name</code> or <code>foaf:mbox_sha1sum</code> are the same.</p> <p>Sameness means that two are the same for purposes of <code>DISTINCT</code> or <code>GROUP BY</code>, and if two are the same, then both have the <code>UNION</code> of all of the properties of both.</p> <p>If this were a naive smoosh, then the individuals would have all the same properties but would not be the same for <code>DISTINCT</code>.</p> <p>If we have complex application rules for determining whether individuals are the same, then one can materialize <code>owl:sameAs</code> triples and keep them in a separate graph. In this way, the original data is not contaminated and the materialized volume stays reasonable — nothing like the blow-up of duplicating properties across instances.</p> <p>The pro-smoosh argument is that if every duplicate makes exactly the same statements, then there is no great blow-up. Best and worst cases will always depend on the data. In rough terms, the more <i>ad hoc</i> the use, the less desirable the materialization. If the usage pattern is really set, then a relational-style application-specific representation with identity resolved at load time will perform best. We can do that too, but so can others.</p> <p>The principal point is about agility as concerns the inference. Run time is more agile than materialization, and if the rules change or if different users have different needs, then materialization runs into trouble. When talking web scale, having multiple users is a given; it is very uneconomical to give everybody their own copy, and the likelihood of a user accessing any significant part of the corpus is minimal. Even if the queries were not limited, the user would typically not wait for the answer of a query doing a scan or aggregation over 1 billion <a href="http://dbpedia.org/resource/Blog" id="link-id1156a550">blog</a> posts or something of the sort. So queries will typically be selective. Selective means that they do not access all of the data, hence do not benefit from ready-made materialization for things they do not even look at. </p> <p>The exception is corpus-wide statistics queries. But these will not be done in interactive time anyway, and will not be done very often. Plus, since these do not typically run all in memory, these are disk bound. And when things are disk bound, size matters. Reading extra entailment on the way is just a performance penalty.</p> <p>Enough talk. Time for an experiment. We take the Yahoo and Falcon web crawls from the Billion Triples Challenge set, and do two things with the FOAF data in them:</p> <ol> <li>Resolve identity at insert time. We remove duplicate person URIs, and give the single <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id11317008">URI</a> all the properties of all the duplicate URIs. We expect these to be most often repeats. If a person references another person, we normalize this reference to go to the single URI of the referenced person.</li> <li>Give every duplicate URI of a person all the properties of all the duplicates. If these are the same value, the data should not get much bigger, or so we think.</li> </ol> <p>For the experiment, we will consider two people the same if they have the same <code>foaf:name</code> and are both instances of <code>foaf:Person</code>. This gets some extra hits but should not be statistically significant.</p> <p>The following is a commented <a href="http://dbpedia.org/resource/SQL" id="link-id110945b0">SQL</a> script performing the smoosh. We play with internal IDs of things, thus some of these operations cannot be done in SPARQL alone. We use SPARQL where possible for readability. As the documentation states, <code>iri_to_id</code> converts from the qualified name of an IRI to its ID and <code>id_to_iri</code> does the reverse.</p> <p>We count the triples that enter into the smoosh:</p> <blockquote> <pre>-- the name is an existence because else we'd get several times more due to -- the names occurring in many graphs <br /> sparql SELECT COUNT(*) WHERE { { SELECT DISTINCT ?person WHERE { ?person a foaf:Person } } . FILTER ( bif:exists ( SELECT (1) WHERE { ?person foaf:name ?nn } ) ) . ?person ?p ?o };<br /> -- We get 3284674 </pre></blockquote> <p>We make a few tables for intermediate results.</p> <blockquote> <pre>-- For each distinct name, gather the properties and objects from -- all subjects with this name <br /> CREATE TABLE name_prop ( np_name ANY, np_p IRI_ID_8, np_o ANY, PRIMARY KEY ( np_name, np_p, np_o ) ); ALTER INDEX name_prop ON name_prop PARTITION ( np_name VARCHAR (-1, 0hexffff) );<br /> -- Map from name to canonical IRI used for the name <br /> CREATE TABLE name_iri ( ni_name ANY PRIMARY KEY, ni_s IRI_ID_8 ); ALTER INDEX name_iri ON name_iri PARTITION ( ni_name VARCHAR (-1, 0hexffff) );<br /> -- Map from person IRI to canonical person IRI<br /> CREATE TABLE pref_iri ( i IRI_ID_8, pref IRI_ID_8, PRIMARY KEY ( i ) ); ALTER INDEX pref_iri ON pref_iri PARTITION ( i INT (0hexffff00) );<br /> -- a table for the materialization where all aliases get all properties of every other <br /> CREATE TABLE smoosh_ct ( s IRI_ID_8, p IRI_ID_8, o ANY, PRIMARY KEY ( s, p, o ) ); ALTER INDEX smoosh_ct ON smoosh_ct PARTITION ( s INT (0hexffff00) );<br /> -- disable transaction log and enable row auto-commit. This is necessary, otherwise -- bulk operations are done transactionally and they will run out of rollback space.<br /> LOG_ENABLE (2);<br /> -- Gather all the properties of all persons with a name under that name. -- INSERT SOFT means that duplicates are ignored <br /> INSERT SOFT name_prop SELECT "n", "p", "o" FROM ( sparql DEFINE output:valmode "LONG" SELECT ?n ?p ?o WHERE { ?x a foaf:Person . ?x foaf:name ?n . ?x ?p ?o } ) xx ;<br /> -- Now choose for each name the canonical IRI <br /> INSERT INTO name_iri SELECT np_name, ( SELECT MIN (s) FROM rdf_quad WHERE o = np_name AND p = IRI_TO_ID ('http://xmlns.com/foaf/0.1/name') ) AS mini FROM name_prop WHERE np_p = IRI_TO_ID ('http://xmlns.com/foaf/0.1/name') ;<br /> -- For each person IRI, map to the canonical IRI of that person <br /> INSERT SOFT pref_iri (i, pref) SELECT s, ni_s FROM name_iri, rdf_quad WHERE o = ni_name AND p = IRI_TO_ID ('http://xmlns.com/foaf/0.1/name') ;<br /> -- Make a graph where all persons have one iri with all the properties of all aliases -- and where person-to-person refs are canonicalized<br /> INSERT SOFT rdf_quad (g,s,p,o) SELECT IRI_TO_ID ('psmoosh'), ni_s, np_p, COALESCE ( ( SELECT pref FROM pref_iri WHERE i = np_o ), np_o ) FROM name_prop, name_iri WHERE ni_name = np_name OPTION ( loop, quietcast ) ;<br /> -- A little explanation: The properties of names are copied into rdf_quad with the name -- replaced with its canonical IRI. If the object has a canonical IRI, this is used as -- the object, else the object is unmodified. This is the COALESCE with the sub-query.<br /> -- This takes a little time. To check on the progress, take another connection to the -- server and do <br /> STATUS ('cluster');<br /> -- It will return something like -- Cluster 4 nodes, 35 s. 108 m/s 1001 KB/s 75% cpu 186% read 12% clw threads 5r 0w 0i -- buffers 549481 253929 d 8 w 0 pfs<br /> -- Now finalize the state; this makes it permanent. Else the work will be lost on server -- failure, since there was no transaction log <br /> CL_EXEC ('checkpoint');<br /> -- See what we got<br /> sparql SELECT COUNT (*) FROM <psmoosh> WHERE {?s ?p ?o};<br /> -- This is 2253102<br /> -- Now make the copy where all have the properties of all synonyms. This takes so much -- space we do not insert it as RDF quads, but make a special table for it so that we can -- run some statistics. This saves time.<br /> INSERT SOFT smoosh_ct (s, p, o) SELECT s, np_p, np_o FROM name_prop, rdf_quad WHERE o = np_name AND p = IRI_TO_ID ('http://xmlns.com/foaf/0.1/name') ;<br /> -- as above, INSERT SOFT so as to ignore duplicates <br /> SELECT COUNT (*) FROM smoosh_ct;<br /> -- This is 167360324<br /> -- Find out where the bloat comes from <br /> SELECT TOP 20 COUNT (*), ID_TO_IRI (p) FROM smoosh_ct GROUP BY p ORDER BY 1 DESC; </pre></blockquote> <p>The results are:</p> <blockquote> <pre>54728777 http://www.w3.org/2002/07/owl#sameAs 48543153 http://xmlns.com/foaf/0.1/knows 13930234 http://www.w3.org/2000/01/rdf-schema#seeAlso 12268512 http://xmlns.com/foaf/0.1/interest 11415867 http://xmlns.com/foaf/0.1/nick 6683963 http://xmlns.com/foaf/0.1/weblog 6650093 http://xmlns.com/foaf/0.1/depiction 4231946 http://xmlns.com/foaf/0.1/mbox_sha1sum 4129629 http://xmlns.com/foaf/0.1/homepage 1776555 http://xmlns.com/foaf/0.1/holdsAccount 1219525 http://xmlns.com/foaf/0.1/based_near 305522 http://www.w3.org/1999/02/22-rdf-syntax-ns#type 274965 http://xmlns.com/foaf/0.1/name 155131 http://xmlns.com/foaf/0.1/dateOfBirth 153001 http://xmlns.com/foaf/0.1/img 111130 http://www.w3.org/2001/vcard-rdf/3.0#ADR 52930 http://xmlns.com/foaf/0.1/gender 48517 http://www.w3.org/2004/02/skos/core#subject 45697 http://www.w3.org/2000/01/rdf-schema#label 44860 http://purl.org/vocab/bio/0.1/olb </pre></blockquote> <p>Now compare with the predicate distribution of the smoosh with identities canonicalized </p> <blockquote> <pre>sparql SELECT COUNT (*) ?p FROM <psmoosh> WHERE { ?s ?p ?o } GROUP BY ?p ORDER BY 1 DESC LIMIT 20;</pre></blockquote> <p>Results are:</p> <blockquote> <pre>748311 http://xmlns.com/foaf/0.1/knows 548391 http://xmlns.com/foaf/0.1/interest 140531 http://www.w3.org/2000/01/rdf-schema#seeAlso 105273 http://www.w3.org/1999/02/22-rdf-syntax-ns#type 78497 http://xmlns.com/foaf/0.1/name 48099 http://www.w3.org/2004/02/skos/core#subject 45179 http://xmlns.com/foaf/0.1/depiction 40229 http://www.w3.org/2000/01/rdf-schema#comment 38272 http://www.w3.org/2000/01/rdf-schema#label 37378 http://xmlns.com/foaf/0.1/nick 37186 http://dbpedia.org/property/abstract 34003 http://xmlns.com/foaf/0.1/img 26182 http://xmlns.com/foaf/0.1/homepage 23795 http://www.w3.org/2002/07/owl#sameAs 17651 http://xmlns.com/foaf/0.1/mbox_sha1sum 17430 http://xmlns.com/foaf/0.1/dateOfBirth 15586 http://xmlns.com/foaf/0.1/page 12869 http://dbpedia.org/property/reference 12497 http://xmlns.com/foaf/0.1/weblog 12329 http://blogs.yandex.ru/schema/foaf/school </pre></blockquote> <p>We can drop the <code>owl:sameAs</code> triples from the count, so the bloat is a bit less by that but it still is tens of times larger than the canonicalized copy or the initial state.</p> <p>Now, when we try using the psmoosh graph, we still get different results from the results with the original data. This is because <code>foaf:knows</code> relations to things with no <code>foaf:name</code> are not represented in the smoosh. The exist:</p> <blockquote> <pre>sparql SELECT COUNT (*) WHERE { ?s foaf:knows ?thing . FILTER ( !bif:exists ( SELECT (1) WHERE { ?thing foaf:name ?nn } ) ) };<br /> -- 1393940 </pre></blockquote> <p>So the smoosh graph is not an accurate rendition of the social network. It would have to be smooshed further to be that, since the data in the sample is quite irregular. But we do not go that far here.</p> <p>Finally, we calculate the smoosh blow up factors. We do not include <code>owl:sameAs</code> triples in the counts.</p> <blockquote> <pre>select (167360324 - 54728777) / 3284674.0; 34.290022997716059<br /> select 2229307 / 3284674.0; = 0.678699621332284 </pre></blockquote> <p>So, to get a smoosh that is not really the equivalent of the original, either multiply the original triple count by 34 or 0.68, depending on whether synonyms are collapsed or not.</p> <p>Making the smooshes does not take very long, some minutes for the small one. Inserting the big one would be longer, a couple of hours maybe. It was 33 minutes for filling the <code>smoosh_ct</code> table. The metrics were not with optimal tuning so the performance numbers just serve to show that smooshing takes time. Probably more time than allowable in an interactive situation, no matter how the process is optimized.</p>
2008-12-16T15:01:30-05:00
Cool URIs, Fish, and Wine
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-12-12#1497
2008-12-12T21:41:03Z
<p>I've just read <a href="http://www.redmonk.com/jgovernor/about/" id="link-id152d07f0">James Governor</a>'s insightful post titled: <a href="http://www.redmonk.com/jgovernor/2007/04/05/why-applications-are-like-fish-and-data-is-like-wine/" id="link-id14e9a200">Why Applications Are Like Fish and Data is Like Wine</a>, where he sums up the comparative value of applications (code containers) and data as follows:</p> <blockquote>"<cite>Only one improves with age. With apologies to the originator of the phrase - “Hardware is like fish, operating systems are like wine.</cite>”</blockquote> <p>Yes! Applications are like Fish and Data like Wine, which is basically what <a href="http://dbpedia.org/resource/Linked_Data" id="link-id115defd0">Linked Data</a> is fundamentally about, especially when you inject memes such as "<a href="http://www.w3.org/TR/cooluris/" id="link-id1438f878">Cool URIs</a>" into the mix. Remember, the essence of Linked Data is all about a <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> of Linked Data Objects endowed with <a href="http://dbpedia.org/resource/Identity_(object-oriented_programming)" id="link-id114df410">Identifiers</a> that don't change i.e., they occupy one place in public (e.g. <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id1195e010">World Wide Web</a>) or private (your corporate <a href="http://dbpedia.org/resource/Intranet" id="link-id1149f1b0">Intranet</a> or <a href="http://dbpedia.org/resource/Extranet" id="link-id11927c80">Extranet</a>) networks, keeping the data that they expose relevant (as in fresh), accessible, and usable in many forms courtesy of the data access & representation dexterity that HTTP facilitates, when incorporated into object identifiers.</p> <p> Here is another excerpt from his post that rings true (amongst many others): </p> <blockquote> <cite>What am I talking about? Processes change, and need to change. Baking data into the application is a bad idea because the data can’t then be extended in useful, and “unexpected ways”. But not expecting corporate data to be used in new ways is kind of like not expecting the Spanish Inquisition. But… “NOBODY expects the Spanish Inquisition! Amongst our weaponry are such diverse elements as: fear, surprise, ruthless efficiency, an almost fanatical devotion to the Pope.” (sounds like Enterprise Architecture ...).</cite> </blockquote> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1482" id="link-id111b6618">Master Data Management & RDF based Linked Data</a> </li> </ul>
2009-01-23T17:22:00.000005-05:00
The future of the Semantic Web? It?s already here | Cision Blog
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-12-11#1496
2008-12-11T20:46:14Z
<blockquote> <p>December 11, 2008</p> <h2 id="title"> <a href="http://blog.us.cision.com/2008/12/the-future-of-the-semantic-web-its-already-here/" title="The future of the Semantic Web? It’s already here" id="link-id0xb4a00c8">The future of the Semantic Web? It’s already here</a> </h2> <p>Author: Jay Krall</p> <h3>An expert weighs in on how Web 3.0 is about to make media monitoring easier</h3> <p>For public relations professionals, finding mentions about a particular brand or product is getting more challenging as the vast clutter of the Web continues to grow. While paid <a href="http://us.cision.com/products_services/cisionpoint/monitor/enews/enews_overview.asp" target="_blank" id="link-id0x1d682460">monitoring services like those offered by Cision</a> and others can help, for those using free-text search engines like Google for media monitoring, combing through pages of irrelevant search results has become routine. For example, acronyms pose a problem: how many instances of the term “HP” referring to “horsepower” do you have to sift through to find articles about <a href="http://hp.com" target="_blank" id="link-id0xaf5bd40">Hewlett-Packard</a> products? Plenty.</p> <p> <span id="more-348"></span> </p> <p>Worse yet, the longer your queries get, the harder it is for search engines to find what you really want. It’s almost 2009. With all this technological innovation happening so fast, why does it seem like computers still can’t read very well? If they were more literate, the monitoring of media and social media for brand mentions would be a lot easier for everyone.</p> <p>That’s just one practical argument for the importance of the Semantic Web. First described in 1999 by <a href="http://www.w3.org/" target="_blank" id="link-id0x1d1499c0">World Wide Web Consortium</a> director <a href="http://www.w3.org/People/Berners-Lee/" target="_blank" id="link-id0x1c755008">Tim Berners-Lee</a>, the Semantic Web, also referred to as Web 3.0, is often described as a vision for the next generation of the Web: pages that can search each other and pull from each other’s data intelligently, melding Web sites and news feeds into precisely honed, individual Web experiences. But actually, the technologies of the Semantic Web are already hard at work, thanks to a group of computer scientists from around the world who are making Berners-Lee’s vision a reality.</p> <p> <a href="http://www.openlinksw.com/blog/%7Ekidehen/" target="_blank" id="link-id0x1e2c25d0">Kingsley Idehen</a>, CEO of <a href="http://www.openlinksw.com/" target="_blank" id="link-id0xa650340">OpenLink Software</a>, is one of those pioneers. He is one of the creators of <a href="http://dbpedia.org" target="_blank" id="link-id0xae0add0">DBpedia</a>, a Semantic Web tool that culls data from <a href="http://wikipedia.org" target="_blank" id="link-id0x1b20c1a8">Wikipedia</a> in amazingly precise ways. The project is a collaboration of OpenLink Software, the <a href="http://www.informatik.uni-leipzig.de/%7Eauer/" target="_blank" id="link-id0x19b8a758">University of Leipzig</a> and <a href="http://www.wiwiss.fu-berlin.de/en/institute/pwo/bizer/index.html" target="_blank" id="link-id0x1d289c70">Freie University Berlin</a>. Simply put, it divides up the site’s information into tags, and uses those tags to develop searches in which the subject is clearly defined, using a computer language that could soon be applied all across the Web. Beginning in late 2006, a program assigned 274 million tags describing nearly 1 billion facts to catalog Wikipedia in this way using the Resource Description Framework (RDF), a commonly accepted format for Semantic Web applications.</p> </blockquote> <p>( <a href="http://blog.us.cision.com/2008/12/the-future-of-the-semantic-web-its-already-here/" id="link-id0xa610bf8">Full story ...</a> )</p>
2008-12-11T15:46:14.000002-05:00
Virtuoso Anytime: No Query Is Too Complex (updated)
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-12-11#1495
2008-12-11T16:13:10Z
<p>A persistent argument against the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1199d5f8">linked data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id116f2730">web</a> has been the cost, scalability, and vulnerability of <a href="http://dbpedia.org/resource/SPARQL" id="link-id14e423c0">SPARQL</a> end points, should the linked data web gain serious mass and traffic.</p> <p>As we are on the brink of hosting the whole <a href="http://dbpedia.org/resource/DBpedia" id="link-id1376a8b0">DBpedia</a> <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id113c8d20">Linked Open Data</a> cloud in <a href="http://virtuoso.openlinksw.com" id="link-id11425a78">Virtuoso</a> Cluster, we have had to think of what we'll do if, for example, somebody decides to count all the triples in the set.</p> <p>How can we encourage clever use of <a href="http://dbpedia.org/resource/Data" id="link-id116f1210">data</a>, yet not die if somebody, whether through malice, lack of understanding, or simple bad luck, submits impossible queries?</p> <p>Restricting the language is not the way; any language beyond text search can express queries that will take forever to execute. Also, just returning a timeout after the first second (or whatever arbitrary time period) leaves people in the dark and does not produce an impression of responsiveness. So we decided to allow arbitrary queries, and if a quota of time or resources is exceeded, we return partial results and indicate how much processing was done.</p> <p>Here we are looking for the top 10 people whom people claim to know without being known in return, like this:</p> <blockquote> <pre>SQL> sparql SELECT ?celeb, COUNT (*) WHERE { ?claimant foaf:knows ?celeb . FILTER (!bif:exists ( SELECT (1) WHERE { ?celeb foaf:knows ?claimant } ) ) } GROUP BY ?celeb ORDER BY DESC 2 LIMIT 10;<br /> celeb callret-1 VARCHAR VARCHAR ________________________________________ _________<br /> http://twitter.com/BarackObama 252 http://twitter.com/brianshaler 183 http://twitter.com/newmediajim 101 http://twitter.com/HenryRollins 95 http://twitter.com/wilw 81 http://twitter.com/stevegarfield 78 http://twitter.com/cote 66 mailto:adam.westerski@deri.org 66 mailto:michal.zaremba@deri.org 66 http://twitter.com/dsifry 65<br /> *** Error S1TAT: [Virtuoso Driver][Virtuoso Server]RC...: Returning incomplete results, query interrupted by result timeout. Activity: 1R rnd 0R seq 0P disk 1.346KB / 3 messages<br /> SQL> sparql SELECT ?celeb, COUNT (*) WHERE { ?claimant foaf:knows ?celeb . FILTER (!bif:exists ( SELECT (1) WHERE { ?celeb foaf:knows ?claimant } ) ) } GROUP BY ?celeb ORDER BY DESC 2 LIMIT 10;<br /> celeb callret-1 VARCHAR VARCHAR ________________________________________ _________<br /> http://twitter.com/JasonCalacanis 496 http://twitter.com/Twitterrific 466 http://twitter.com/ev 442 http://twitter.com/BarackObama 356 http://twitter.com/laughingsquid 317 http://twitter.com/gruber 294 http://twitter.com/chrispirillo 259 http://twitter.com/ambermacarthur 224 http://twitter.com/t 219 http://twitter.com/johnedwards 188<br /> *** Error S1TAT: [Virtuoso Driver][Virtuoso Server]RC...: Returning incomplete results, query interrupted by result timeout. Activity: 329R rnd 44.6KR seq 342P disk 638.4KB / 46 messages</pre></blockquote> <p>The first query read all data from disk; the second run had the working set from the first and could read some more before time ran out, hence the results were better. But the response time was the same.</p> <p>If one has a query that just loops over consecutive joins, like in basic SPARQL, interrupting the processing after a set time period is simple. But such queries are not very interesting. To give meaningful partial answers with nested aggregation and sub-queries requires some more tricks. The basic idea is to terminate the innermost active sub-query/aggregation at the first timeout, and extend the timeout a bit so that accumulated results get fed to the next aggregation, like from the <code>GROUP BY</code> to the <code>ORDER BY</code>. If this again times out, we continue with the next outer layer. This guarantees that results are delivered if there were any results found for which the query pattern is true. False results are not produced, except in cases where there is comparison with a count and the count is smaller than it would be with the full evaluation.</p> <p>One can also use this as a basis for paid services. The cutoff does not have to be time; it can also be in other units, making it insensitive to concurrent usage and variations of working set.</p> <p>This system will be deployed on our <a href="http://challenge.semanticweb.org/" id="link-id11500a58">Billion Triples Challenge</a> <a href="http://b3s.openlinksw.com/" id="link-id11683120">demo instance</a> in a few days, after some more testing. When Virtuoso 6 ships, all <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id1157a500">LOD</a> Cloud AMIs and OpenLink-hosted LOD Cloud SPARQL endpoints will have this enabled by default. (AMI users will be able to disable the feature, if desired.) The feature works with Virtuoso 6 in both single server and cluster deployment.</p>
2008-12-12T10:29:23-05:00
Virtuoso Anytime: No Query Is Too Complex (updated)
http://www.openlinksw.com:443/weblog/oerling/?date=2008-12-11#1494
2008-12-11T16:13:10Z
<p>A persistent argument against the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1199d5f8">linked data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id116f2730">web</a> has been the cost, scalability, and vulnerability of <a href="http://dbpedia.org/resource/SPARQL" id="link-id14e423c0">SPARQL</a> end points, should the linked data web gain serious mass and traffic.</p> <p>As we are on the brink of hosting the whole <a href="http://dbpedia.org/resource/DBpedia" id="link-id1376a8b0">DBpedia</a> <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id113c8d20">Linked Open Data</a> cloud in <a href="http://virtuoso.openlinksw.com" id="link-id11425a78">Virtuoso</a> Cluster, we have had to think of what we'll do if, for example, somebody decides to count all the triples in the set.</p> <p>How can we encourage clever use of <a href="http://dbpedia.org/resource/Data" id="link-id116f1210">data</a>, yet not die if somebody, whether through malice, lack of understanding, or simple bad luck, submits impossible queries?</p> <p>Restricting the language is not the way; any language beyond text search can express queries that will take forever to execute. Also, just returning a timeout after the first second (or whatever arbitrary time period) leaves people in the dark and does not produce an impression of responsiveness. So we decided to allow arbitrary queries, and if a quota of time or resources is exceeded, we return partial results and indicate how much processing was done.</p> <p>Here we are looking for the top 10 people whom people claim to know without being known in return, like this:</p> <blockquote> <pre>SQL> sparql SELECT ?celeb, COUNT (*) WHERE { ?claimant foaf:knows ?celeb . FILTER (!bif:exists ( SELECT (1) WHERE { ?celeb foaf:knows ?claimant } ) ) } GROUP BY ?celeb ORDER BY DESC 2 LIMIT 10;<br /> celeb callret-1 VARCHAR VARCHAR ________________________________________ _________<br /> http://twitter.com/BarackObama 252 http://twitter.com/brianshaler 183 http://twitter.com/newmediajim 101 http://twitter.com/HenryRollins 95 http://twitter.com/wilw 81 http://twitter.com/stevegarfield 78 http://twitter.com/cote 66 mailto:adam.westerski@deri.org 66 mailto:michal.zaremba@deri.org 66 http://twitter.com/dsifry 65<br /> *** Error S1TAT: [Virtuoso Driver][Virtuoso Server]RC...: Returning incomplete results, query interrupted by result timeout. Activity: 1R rnd 0R seq 0P disk 1.346KB / 3 messages<br /> SQL> sparql SELECT ?celeb, COUNT (*) WHERE { ?claimant foaf:knows ?celeb . FILTER (!bif:exists ( SELECT (1) WHERE { ?celeb foaf:knows ?claimant } ) ) } GROUP BY ?celeb ORDER BY DESC 2 LIMIT 10;<br /> celeb callret-1 VARCHAR VARCHAR ________________________________________ _________<br /> http://twitter.com/JasonCalacanis 496 http://twitter.com/Twitterrific 466 http://twitter.com/ev 442 http://twitter.com/BarackObama 356 http://twitter.com/laughingsquid 317 http://twitter.com/gruber 294 http://twitter.com/chrispirillo 259 http://twitter.com/ambermacarthur 224 http://twitter.com/t 219 http://twitter.com/johnedwards 188<br /> *** Error S1TAT: [Virtuoso Driver][Virtuoso Server]RC...: Returning incomplete results, query interrupted by result timeout. Activity: 329R rnd 44.6KR seq 342P disk 638.4KB / 46 messages</pre></blockquote> <p>The first query read all data from disk; the second run had the working set from the first and could read some more before time ran out, hence the results were better. But the response time was the same.</p> <p>If one has a query that just loops over consecutive joins, like in basic SPARQL, interrupting the processing after a set time period is simple. But such queries are not very interesting. To give meaningful partial answers with nested aggregation and sub-queries requires some more tricks. The basic idea is to terminate the innermost active sub-query/aggregation at the first timeout, and extend the timeout a bit so that accumulated results get fed to the next aggregation, like from the <code>GROUP BY</code> to the <code>ORDER BY</code>. If this again times out, we continue with the next outer layer. This guarantees that results are delivered if there were any results found for which the query pattern is true. False results are not produced, except in cases where there is comparison with a count and the count is smaller than it would be with the full evaluation.</p> <p>One can also use this as a basis for paid services. The cutoff does not have to be time; it can also be in other units, making it insensitive to concurrent usage and variations of working set.</p> <p>This system will be deployed on our <a href="http://challenge.semanticweb.org/" id="link-id11500a58">Billion Triples Challenge</a> <a href="http://b3s.openlinksw.com/" id="link-id11683120">demo instance</a> in a few days, after some more testing. When Virtuoso 6 ships, all <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id1157a500">LOD</a> Cloud AMIs and OpenLink-hosted LOD Cloud SPARQL endpoints will have this enabled by default. (AMI users will be able to disable the feature, if desired.) The feature works with Virtuoso 6 in both single server and cluster deployment.</p>
2008-12-12T10:29:18-05:00
DataSpaces Bulletin: December issue now online!
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-12-09#1493
2008-12-09T18:21:00Z
<p>The highly anticipated <a href="http://support.openlinksw.com/supportweb/DataSpacesBulletin-2008-12" id="link-id0x1ec42588">December 2008 issue of the DataSpaces Bulletin is now available</a>!</p> <p>This month's DataSpaces contains material of interest to the Virtuoso developer and UDA user community alike —</p> <ol> <li>Introduction to Virtuoso Universal Server (Cloud Edition).</li> <li>Links to Virtuoso and Linked Data mailing lists.</li> <li>UDA license management tips and tricks.</li> </ol>
2008-12-09T15:06:51-05:00
DataSpaces Bulletin: December issue now online!
http://www.openlinksw.com:443/weblog/uda/135/?date=2008-12-09#1492
2008-12-09T18:09:00Z
<p>The highly anticipated <a href="http://support.openlinksw.com/supportweb/DataSpacesBulletin-2008-12" id="link-id0x2108a398">December 2008 issue of the DataSpaces Bulletin is now available</a>!</p> <p>This month's DataSpaces contains material of interest to the Virtuoso developer and UDA user community alike —</p> <ol> <li>Introduction to Virtuoso Universal Server (Cloud Edition).</li> <li>Links to Virtuoso and Linked Data mailing lists.</li> <li>UDA license management tips and tricks.</li> </ol>
2008-12-09T15:06:53-05:00
Virtuoso+Neurocommons EC2 AMI released! (Update - 1)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-12-08#1491
2008-12-08T21:13:04Z
<h3>What is Neurocommons?</h3> <p>Excerpted from the <a href="http://neurocommons.org/" id="link-id142131b8">project home page</a>:</p> <blockquote> <cite>The NeuroCommons project seeks to make all scientific research materials - research articles, annotations, <a href="http://dbpedia.org/resource/Data">data</a>, physical materials - as available and as useable as they can be. We do this by both fostering practices that render <a href="http://dbpedia.org/resource/Information" id="link-id112f8418">information</a> in a form that promotes uniform access by computational agents - sometimes called "interoperability". We want <a href="http://dbpedia.org/resource/Knowledge" id="link-id1195f9b0">knowledge</a> sources to combine meaningfully, enabling semantically precise queries that span multiple information sources.</cite> </blockquote> <p>In a nutshell, a great project that makes practical use of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1e945010">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id115de818">Web</a> technology in the areas of computational biology and neuroscience.</p> <h3>What is <a href="http://virtuoso.openlinksw.com" id="link-id0xa1eda880">Virtuoso</a> and Neurocommons AMI for EC2?</h3> <p>A pre-installed and fully tuned edition of Virtuoso that includes a fully configured Neurocommons Knowledgebase (in RDF <a href="http://dbpedia.org/resource/Linked_Data" id="link-id114d8c88">Linked Data</a> form) on Amazon's EC2 Cloud platform.</p> <h3>Benefits?</h3> <p>Generally, it provides a no-hassles mechanism for instantiating personal-, organization-, or service-specific instances of a very powerful research knowledgebase within approximately 1.15 hours compared to a lengthy rebuild from RDF source data alternative that takes 14 hours or more, depending on machine hardware configuration and host operating system resources.</p> <h3>Features:</h3> <ol> <li> <a href="http://blogs.usnet.private:8893/sparql.neurocommons.org" id="link-id154c5710">Neurocommons public instance</a> functionality replica (re. RDF and (X)HTML resource description representations & <a href="http://dbpedia.org/resource/SPARQL" id="link-id1188e5f0">SPARQL</a> endpoint)</li> <li> Local <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id117092a8">URI</a> de-referencing (so no contention with public endpoint) as part of the RDF Linked Data Deployment</li> <li> Fully tuned Virtuoso instance for neurocommons knowledgebase. </li> </ol> <h3>Installation Guide</h3> Simply read the <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtEC2AMINeuroCommonsInstall" id="link-id15267570">Virtuoso+NeuroCommons EC2 AMI installation guide</a>. <h3>Related</h3> <ul> <li> <a href="http://sciencecommons.org/about/science-commons-dylan-video/" id="link-id14cb22f0">Science Commons Video</a> </li> </ul>
2008-12-10T22:48:49-05:00
Virtuoso+DBpedia AMI for EC2 now Live!
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-12-01#1490
2008-12-01T16:04:28Z
<h3>What is <a href="http://virtuoso.openlinksw.com" id="link-id11015c60">Virtuoso</a>+<a href="http://dbpedia.org/resource/DBpedia" id="link-id1140b6f0">DBpedia</a> AMI for EC2?</h3> <p>A pre-installed and fully tuned edition of Virtuoso that includes a fully configured DBpedia instance on Amazon's EC2 Cloud platform.</p> <h3>Benefits?</h3> <p>Generally, it provides a no hassles mechanism for instantiating personal, organization, or service specific instances of DBpedia within approximately 1.5 hours as opposed to a lengthy rebuild from RDF source <a href="http://dbpedia.org/resource/Data">data</a> that takes between 8 - 22 hours depending on machine hardware configuration and host operating system resources.</p> <p>From a <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> Entrepreneur perspective it offers all of the generic benefits of a <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/ODSInstallationEC2" id="link-id1148ac90">Virtuoso EC2 AMI</a> plus the following:</p> <ol> <li> Instant bootstrap of a dense Lookup Hub for <a href="http://dbpedia.org/resource/Linked_Data" id="link-id14c94590">Linked Data Web</a> oriented solutions</li> <li> No exposure to any of the complexities and nuances associated with deployment of dereferencable URIs (you have a DBpedia replica)</li> <li> Predictable performance and scalability due localization of query processing (you aren't sharing the public DBpedia server with the rest of the world). </li> </ol> <h3>Features:</h3> <ol> <li> DBpedia public instance functionality replica (re. RDF and (X)HTML resource description representations & <a href="http://dbpedia.org/resource/SPARQL" id="link-id1188e5f0">SPARQL</a> endpoint)</li> <li> Local <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id117092a8">URI</a> de-referencing (so no contention with public endpoint) as part of the Linked Data Deployment</li> <li> Fully tuned Virtuoso instance for DBpedia data set hosting. </li> </ol> <h3>How Do I Get Started?</h3> Simply read the <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtEC2AMIDBpediaInstall" id="link-id15836e90">Virtuoso-DBpedia EC2 AMI installation guide</a>. <p>Here are a few live examples of DBpedia resource URIs deployed and de-referencable via one of my EC2 based <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id14930ab0">personal data spaces</a>:</p> <ul> <li> <a href="http://kingsley.idehen.name/resource/Linked_Data" id="link-id1104a740">Linked Data</a> </li> <li> <a href="http://kingsley.idehen.name/resource/Entity-attribute-value_model" id="link-id11200f48">Entity-Attribute-Value</a> (aka. Triples) Model</li> <li> <a href="http://kingsley.idehen.name/resource/Hyperdata" id="link-id11235ef0">Hyperdata</a> Linking (aka. <a href="http://kingsley.idehen.name/resource/Object_hyperlinking" id="link-id15493b90">Object Hyperlinking</a>)</li> <li> <a href="http://kingsley.idehen.name/resource/Barack_Obama" id="link-id15497580">Barack Obama</a> </li> </ul>
2008-12-12T11:22:27-05:00
Introducing Virtuoso Universal Server (Cloud Edition) for Amazon EC2
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-11-28#1489
2008-11-28T19:27:12Z
<h3>What is it?</h3> <p>A pre-installed edition of <a href="http://virtuoso.openlinksw.com" id="link-id14bea838">Virtuoso</a> for Amazon's EC2 Cloud platform.</p> <h3>What does it offer?</h3> From a <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> Entrepreneur perspective it offers: <ol> <li> Low cost entry point to a game-changing Web 3.0+ (and beyond) platform that combines <a href="http://dbpedia.org/resource/SQL" id="link-id11309b38">SQL</a>, <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id135f7988">RDF</a>, XML, and Web Services functionality</li> <li> Flexible variable cost model (courtesy of <a href="http://aws.amazon.com/devpay/" id="link-id17941018">EC2 DevPay</a>) tightly bound to revenue generated by your services</li> <li> Delivers federated and/or centralized model flexibility for you SaaS based solutions</li> <li> Simple entry point for developing and deploying sophisticated database driven applications (SQL or RDF <a href="http://dbpedia.org/resource/Linked_Data" id="link-id14ea6b10">Linked Data Web</a> oriented)</li> <li> Complete framework for exploiting OpenID, OAuth (including Role enhancements) that simplifies exploitation of these vital Identity and <a href="http://dbpedia.org/resource/Data">Data</a> Access technologies</li> <li>Easily implement RDF Linked Data based Mail, Blogging, Wikis, Bookmarks, Calendaring, Discussion Forums, Tagging, Social-Networking as <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id11519928">Data Space</a> (data containers) features of your application or service offering</li> <li>Instant alleviation of challenges (e.g. service costs and agility) associated with <a href="http://dbpedia.org/resource/DataPortability" id="link-id111cb610">Data Portability</a> and Open Data Access across Web 2.0 data silos</li> <li> LDAP integration for <a href="http://dbpedia.org/resource/Intranet" id="link-id114a8270">Intranet</a> / <a href="http://dbpedia.org/resource/Extranet" id="link-id10fe4f08">Extranet</a> style applications.</li> </ol> <p>From the DBMS engine perspective it provides you with one or more pre-configured instances of Virtuoso that enable immediate exploitation of the following services:</p> <ol> <li> RDF Database (a Quad Store with <a href="http://dbpedia.org/resource/SPARQL" id="link-id11911bf8">SPARQL</a> & SPARUL Language & Protocol support)</li> <li> <a href="http://dbpedia.org/resource/SQL" id="link-id110544c8">SQL</a> Database (with <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id1524c7d0">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id14cfb658">JDBC</a>, OLE-DB, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id110ec6c8">ADO</a>.NET, and XMLA driver access)</li> <li>XML Database (XML Schema, <a href="http://dbpedia.org/resource/XQuery" id="link-id10ebf218">XQuery</a>/<a href="http://dbpedia.org/resource/XPath" id="link-id142a7898">Xpath</a>, XSLT, Full Text Indexing)</li> <li>Full Text Indexing.</li> </ol> <p>From a Middleware perspective it provides:</p> <ol> <li> RDF Views (Wrappers / Semantic Covers) over SQL, XML, and other data sources accessible via SOAP or REST style Web Services</li> <li> Sponger Service for converting non RDF <a href="http://dbpedia.org/resource/Information" id="link-id11931c60">information</a> resources into RDF <a href="http://dbpedia.org/resource/Linked_Data" id="link-id118f7168">Linked Data</a> "on the fly" via a large collection of pre-installed RDFizer Cartridges.</li> </ol> <p>From the Web Server Platform perspective it provides an alternative to LAMP stack components such as <a href="http://dbpedia.org/resource/MySQL" id="link-id10f7b780">MySQL</a> and Apace by offering</p> <ol> <li> HTTP Web Server</li> <li> WebDAV Server</li> <li> Web <a href="http://dbpedia.org/resource/Application_server" id="link-id1268daa8">Application Server</a> (includes <a href="http://dbpedia.org/resource/PHP" id="link-id1585d238">PHP</a> runtime hosting)</li> <li> SOAP or REST style Web Services Deployment</li> <li> RDF Linked Data Deployment</li> <li> SPARQL (SPARQL Query Language) and SPARUL (SPARQL Update Language) endpoints</li> <li>Virtuoso Hosted PHP packages for <a href="http://dbpedia.org/resource/MediaWiki" id="link-id15568818">MediaWiki</a>, <a href="http://dbpedia.org/resource/Drupal" id="link-id110bd7a8">Drupal</a>, <a href="http://dbpedia.org/resource/WordPress" id="link-id10f66918">Wordpress</a>, and <a href="http://dbpedia.org/resource/PhpBB" id="link-id13fda4d0">phpBB3</a> (just install the relevant Virtuoso Distro. Package). </li> </ol> <p>From the general System Administrator's perspective it provides:</p> <ol> <li> Online Backups (Backup Set dispatched to S3 buckets, FTP, or HTTP/WebDAV server locations)</li> <li>Synchronized Incremental Backups to Backup Set locations</li> <li>Backup Restore from Backup Set location (without exiting to EC2 shell).</li> </ol> <p>Higher level user oriented offerings include:</p> <ol> <li>OpenLink Data Explorer front-end for exploring the burgeoning Linked Data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id11646dc8">Web</a> </li> <li> Ajax based SPARQL Query Builder (iSPARQL) that enables SPARQL Query construction by Example</li> <li>Ajax based SQL Query Builder (QBE) that enables SQL Query construction by Example.</li> </ol> <p>For Web 2.0 / 3.0 users, developers, and entrepreneurs it offers it includes Distributed Collaboration Tools & Social Media realm functionality courtesy of <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id11009930">ODS</a> that includes:</p> <ol> <li> Point of presence on the Linked Data Web that meshes your Identity and your Data via URIs</li> <li> System generated Social Network Profile & Contact Data via <a href="http://dbpedia.org/resource/Friend_of_a_friend" id="link-id1185a1c0">FOAF</a>?</li> <li> System generated <a href="http://dbpedia.org/resource/SIOC" id="link-id14791890">SIOC</a> (Semantically Interconnected Online Community) <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id1577cad8">Data Space</a> (that includes a Social Graph) exposing all your Web data in RDF Linked Data form</li> <li> System generated OpenID and automatic integration with FOAF</li> <li> Transparent Data Integration across Facebook, Digg, LinkedIn, FriendFeed, Twitter, and any other Web 2.0 data space equipped with RSS / Atom support and/or REST style Web Services</li> <li> In-built support for SyncML which enables data synchronization with Mobile Phones.</li> </ol> <h3>How Do I Get Going with It?</h3> <ul> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/ODSInstallationEC2" id="link-id114e1600">Standard Installation Guide</a> </li> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtEC2AMIDBpediaInstall" id="link-id110a98e8">Personal or Service Specific DBpedia Installation Guide</a> </li> </ul>
2008-11-28T16:06:02.000006-05:00
An Example of RDF Scalability
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-11-27#1488
2008-11-27T11:23:47Z
<p>We hear it to exhaustion, where is <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x14e828d8">RDF</a> scalability? We have been suggesting for a while that this is a solved question. I will here give some concrete numbers to back this.</p> <p>The scalability dream is to add hardware and get increased performance in proportion to the power the added component has when measured by itself. A corollary dream is to take scalability effects that are measured in a simple task and see them in a complex task.</p> <p>Below we show how we do 3.3 million random triple lookups per second on two 8 core commodity servers producing complete results, joining across partitions. On a single 4 core server, the figure is about 1 million lookups per second. With a single thread, it is about 250K lookups per second. This is the good case. But even our worse case is quite decent.</p> <p>We took a simple <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x14fef850">SPARQL</a> query, counting how many people say they reciprocally know each other. In the <a href="http://challenge.semanticweb.org/" id="link-id0x1bca04d0">Billion Triples Challenge</a> <a href="http://dbpedia.org/resource/Data" id="link-id0x1be84e88">data</a> set, there are 25M <code>foaf:knows</code> quads of which 92K are reciprocal. <i>Reciprocal</i> here means that when x knows y in some graph, y knows x in the same or any other graph.</p> <pre>SELECT COUNT (*) WHERE { ?p1 foaf:knows ?p2 . ?p2 foaf:knows ?p1 }</pre> <p>There is no guarantee that the triple of <code>x knows y</code> is in the same partition as the triple y knows x. Thus the join is randomly distributed, n partitions to n partitions.</p> <p>We left this out of the Billion Triples Challenge demo because this did not run fast enough for our liking. Since then, we have corrected this.</p> <p>If run on a single thread, this query would be a loop over all the quads with a predicate of <code>foaf:knows</code>, and an inner loop looking for a quad with 3 of 4 fields given (<code>SPO</code>). If we have a partitioned situation, we have a loop over all the <code>foaf:knows</code> quads in each partition, and an inner lookup looking for the reciprocal <code>foaf:knows</code> quad in whatever partition it may be found.</p> <p>We have implemented this with two different message patterns: </p> <ol> <li> <p> <b>Centralized:</b> One process reads all the <code>foaf:knows</code> quads from all processes. Every 50K quads, it sends a batch of reciprocal quad checks to each partition that could contain a reciprocal quad. Each partition keeps the count of found reciprocal quads, and these are gathered and added up at the end.</p> </li> <li> <p> <b>Symmetrical:</b> Each process reads the <code>foaf:knows</code> quads in its partition, and sends a batch of checks to each process that could have the reciprocal <code>foaf:knows</code> quad every 50K quads. At the end, the counts are gathered from all partitions. There is some additional control traffic but we do not go into its details here.</p> </li> </ol> <p>Below is the result measured on 2 machines each with 2 x Xeon 5345 (quad core; total 8 cores), 16G RAM, and each machine running 6 <a href="http://virtuoso.openlinksw.com" id="link-id0x16642a90">Virtuoso</a> instances. The interconnect is dual 1-Gbit ethernet. Numbers are with warm cache.</p> <blockquote> <code>Centralized: 35,543 msec, 728,634 sequential + random lookups per second <br /> Cluster 12 nodes, 35 s. 1072 m/s 39,085 KB/s 316% cpu ... <br /> <br /> Symmetrical: 7706 msec, 3,360,740 sequential + random lookups per second <br /> Cluster 12 nodes, 7 s. 572 m/s 16,983 KB/s 1137% cpu ...</code> </blockquote> <p>The second line is the summary from the cluster status report for the duration of the query. The interesting numbers are the KB/s and the %CPU. The former is the cross-sectional data transfer rate for intra-cluster communication; the latter is the consolidated CPU utilization, where a constantly-busy core counts for 100%. The point to note is that the symmetrical approach takes 4x less real time with under half the data transfer rate. Further, when using multiple machines, the speed of a single interface does not limit the overall throughput as it does in the centralized situation.</p> <p>These figures represent the best and worst cases of distributed <code>JOIN</code>ing. If we have a straight sequence of <code>JOIN</code>s, with single pattern optionals and existences and the order in which results are produced is not significant (i.e., there is aggregation, existence test, or <code>ORDER BY</code>), the symmetrical pattern is applicable. On the other hand, if there are multiple triple pattern optionals, complex sub-queries, <code>DISTINCT</code>s in the middle of the query, or results have to be produced in the order of an index, then the centralized approach must be used at least part of the time.</p> <p>Also, if we must make transitive closures, which can be thought of as an extension of a <code>DISTINCT</code> in a subquery, we must pass the data through a single point before moving the bindings to the next <code>JOIN</code> in the sequence. This happens for example in resolving <code><a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x14e1a160">owl</a>:sameAs</code> at run time. However, the good news is that performance does not fall much below the centralized figure even when there are complex nested structures with intermediate transitive closures, <code>DISTINCT</code>s, complex existence tests, etc., that require passing all intermediate results through a central point. No matter the complexity, it is always possible to vector some tens-of-thousands of variable bindings into a single message exchange. And if there are not that many intermediate results, then single query execution time is not a problem anyhow.</p> <p>For our sample query, we would get still more speed by using a partitioned hash join, filling the hash from the <code>foaf:knows</code> relations and then running the <code>foaf:knows</code> relations through the hash. If the hash size is right, a hash lookup is somewhat better than an index lookup. The problem is that when the hash join is not the right solution, it is an expensive mistake: the best case is good; the worst case is very bad. But if there is no index then hash join is better than nothing. One problem of hash joins is that they make temporary data structures which, if large, will skew the working set. One must be quite sure of the cardinality before it is safe to try a hash join. So we do not do hash joins with RDF, but we do use them sometimes with relational data. </p> <p>These same methods apply to relational data just as well. This does not make generic RDF storage outperform an application-specific relational representation on the same platform, as the latter benefits from all the same optimizations, but in terms of sheer numbers, this makes RDF representation an option where it was not an option before. RDF is all about not needing to design the schema around the queries, and not needing to limit what joins with what else.</p>
2008-12-01T12:09:55.000008-05:00
An Example of RDF Scalability
http://www.openlinksw.com:443/weblog/oerling/?date=2008-11-27#1487
2008-11-27T11:23:47Z
<p>We hear it to exhaustion, where is <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1eab4128">RDF</a> scalability? We have been suggesting for a while that this is a solved question. I will here give some concrete numbers to back this.</p> <p>The scalability dream is to add hardware and get increased performance in proportion to the power the added component has when measured by itself. A corollary dream is to take scalability effects that are measured in a simple task and see them in a complex task.</p> <p>Below we show how we do 3.3 million random triple lookups per second on two 8 core commodity servers producing complete results, joining across partitions. On a single 4 core server, the figure is about 1 million lookups per second. With a single thread, it is about 250K lookups per second. This is the good case. But even our worse case is quite decent.</p> <p>We took a simple <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x15cb3da8">SPARQL</a> query, counting how many people say they reciprocally know each other. In the <a href="http://challenge.semanticweb.org/" id="link-id0x1bfb7a00">Billion Triples Challenge</a> <a href="http://dbpedia.org/resource/Data" id="link-id0xa57187d8">data</a> set, there are 25M <code>foaf:knows</code> quads of which 92K are reciprocal. <i>Reciprocal</i> here means that when x knows y in some graph, y knows x in the same or any other graph.</p> <pre>SELECT COUNT (*) WHERE { ?p1 foaf:knows ?p2 . ?p2 foaf:knows ?p1 }</pre> <p>There is no guarantee that the triple of <code>x knows y</code> is in the same partition as the triple y knows x. Thus the join is randomly distributed, n partitions to n partitions.</p> <p>We left this out of the Billion Triples Challenge demo because this did not run fast enough for our liking. Since then, we have corrected this.</p> <p>If run on a single thread, this query would be a loop over all the quads with a predicate of <code>foaf:knows</code>, and an inner loop looking for a quad with 3 of 4 fields given (<code>SPO</code>). If we have a partitioned situation, we have a loop over all the <code>foaf:knows</code> quads in each partition, and an inner lookup looking for the reciprocal <code>foaf:knows</code> quad in whatever partition it may be found.</p> <p>We have implemented this with two different message patterns: </p> <ol> <li> <p> <b>Centralized:</b> One process reads all the <code>foaf:knows</code> quads from all processes. Every 50K quads, it sends a batch of reciprocal quad checks to each partition that could contain a reciprocal quad. Each partition keeps the count of found reciprocal quads, and these are gathered and added up at the end.</p> </li> <li> <p> <b>Symmetrical:</b> Each process reads the <code>foaf:knows</code> quads in its partition, and sends a batch of checks to each process that could have the reciprocal <code>foaf:knows</code> quad every 50K quads. At the end, the counts are gathered from all partitions. There is some additional control traffic but we do not go into its details here.</p> </li> </ol> <p>Below is the result measured on 2 machines each with 2 x Xeon 5345 (quad core; total 8 cores), 16G RAM, and each machine running 6 <a href="http://virtuoso.openlinksw.com" id="link-id0x1c0c94a8">Virtuoso</a> instances. The interconnect is dual 1-Gbit ethernet. Numbers are with warm cache.</p> <blockquote> <code>Centralized: 35,543 msec, 728,634 sequential + random lookups per second <br /> Cluster 12 nodes, 35 s. 1072 m/s 39,085 KB/s 316% cpu ... <br /> <br /> Symmetrical: 7706 msec, 3,360,740 sequential + random lookups per second <br /> Cluster 12 nodes, 7 s. 572 m/s 16,983 KB/s 1137% cpu ...</code> </blockquote> <p>The second line is the summary from the cluster status report for the duration of the query. The interesting numbers are the KB/s and the %CPU. The former is the cross-sectional data transfer rate for intra-cluster communication; the latter is the consolidated CPU utilization, where a constantly-busy core counts for 100%. The point to note is that the symmetrical approach takes 4x less real time with under half the data transfer rate. Further, when using multiple machines, the speed of a single interface does not limit the overall throughput as it does in the centralized situation.</p> <p>These figures represent the best and worst cases of distributed <code>JOIN</code>ing. If we have a straight sequence of <code>JOIN</code>s, with single pattern optionals and existences and the order in which results are produced is not significant (i.e., there is aggregation, existence test, or <code>ORDER BY</code>), the symmetrical pattern is applicable. On the other hand, if there are multiple triple pattern optionals, complex sub-queries, <code>DISTINCT</code>s in the middle of the query, or results have to be produced in the order of an index, then the centralized approach must be used at least part of the time.</p> <p>Also, if we must make transitive closures, which can be thought of as an extension of a <code>DISTINCT</code> in a subquery, we must pass the data through a single point before moving the bindings to the next <code>JOIN</code> in the sequence. This happens for example in resolving <code><a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x28005280">owl</a>:sameAs</code> at run time. However, the good news is that performance does not fall much below the centralized figure even when there are complex nested structures with intermediate transitive closures, <code>DISTINCT</code>s, complex existence tests, etc., that require passing all intermediate results through a central point. No matter the complexity, it is always possible to vector some tens-of-thousands of variable bindings into a single message exchange. And if there are not that many intermediate results, then single query execution time is not a problem anyhow.</p> <p>For our sample query, we would get still more speed by using a partitioned hash join, filling the hash from the <code>foaf:knows</code> relations and then running the <code>foaf:knows</code> relations through the hash. If the hash size is right, a hash lookup is somewhat better than an index lookup. The problem is that when the hash join is not the right solution, it is an expensive mistake: the best case is good; the worst case is very bad. But if there is no index then hash join is better than nothing. One problem of hash joins is that they make temporary data structures which, if large, will skew the working set. One must be quite sure of the cardinality before it is safe to try a hash join. So we do not do hash joins with RDF, but we do use them sometimes with relational data. </p> <p>These same methods apply to relational data just as well. This does not make generic RDF storage outperform an application-specific relational representation on the same platform, as the latter benefits from all the same optimizations, but in terms of sheer numbers, this makes RDF representation an option where it was not an option before. RDF is all about not needing to design the schema around the queries, and not needing to limit what joins with what else.</p>
2015-08-20T17:27:01.932709-04:00
Your Personal Edition of DBpedia in the Clouds
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-11-25#1486
2008-11-25T23:23:26Z
<p>We are just about done with an end-to-end workflow pattern that enables reconstitution of <a href="http://dbpedia.org/resource/DBpedia" id="link-id112a4aa0">DBpedia</a> 3.2 instances in the Clouds courtesy of <a href="http://virtuoso.openlinksw.com" id="link-id11969f70">Virtuoso</a> and EC2.</p> <p>Basically this is how it works.</p> <ol> <li>Instantiate a Virtuoso EC2 AMI (paid variety)</li> <li>Install the special EC2 extensions (ec2ext_dav.vad) VAD via the Conductor UI or iSQL</li> <li>Restore the Virtuoso+DBpedia backup from our S3 bucket</li> <li>After approx. 1 hr, you will have a complete DBpedia replica in your own <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id1169c358">data space</a> on the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10fd39a8">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id141d0c80">Web</a>.</li> </ol> <p>DBpedia replica implies:</p> <ol> <li> <a href="http://dbpedia.org/resource/SPARQL" id="link-id116dfd78">SPARQL</a> Endpoint</li> <li>Linked Data Viewer Pages (as you see in the public DBpedia instance)</li> <li>All requisite re-write rules for <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id114ab148">URI</a> de-referencing and attribution (i.e., low cost triples that links back to main DBpedia using terms from our little Attribution Ontology) </li> <li>All the inference rules for <a href="http://umbel.org/about/" id="link-id112d4860">UMBEL</a>, YAGO, <a href="http://dbpedia.org/resource/Cyc" id="link-id14612e80">OpenCYC</a>, and DBpedia-OWL data dictionaries </li> <li>All Full Text Indexes</li> <li>All Bitmap Indexes.</li> </ol> <p>Tomorrow is the official go live day (due to last minute price changes), but you can instantiate a paid <a href="https://aws-portal.amazon.com/gp/aws/user/subscription/index.html?ie=UTF8&offeringCode=6CB89F71" id="link-id115da1a8">Virtuoso AMI</a> starting now :-)</p> <p>To be continued...</p>
2008-11-25T18:55:55-05:00
Virtuoso Vs. MySQL: Setting the Berlin Record Straight (update 2)
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-11-20#1485
2008-11-20T11:06:11Z
<p>In the context of the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0xa322b58">Berlin SPARQL Benchmark</a>, I have repeatedly written about measurement procedures and steady state. The point is that the numbers at larger scales are unreliable due to cache behavior if one is not careful about measurement and does not have adequate warmup. Thus it came to pass that one cut of the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x9524730">BSBM</a> paper had 3 seconds for <a href="http://dbpedia.org/resource/MySQL" id="link-id0x2ba8db0">MySQL</a> and 100 for <a href="http://virtuoso.openlinksw.com" id="link-id0xa9137d0">Virtuoso</a>, basically through ignoring cache effects.</p> <p>So we decided to do it ourselves.</p> <p>The score is (updated with revised <code>innodb_buffer_pool_size</code> setting, based on advice noted down below):</p> <table border="1" cellspacing="2" cellpadding="5"> <tr> <th>n-clients</th> <th>Virtuoso</th> <th>MySQL <br /> (with increased buffer pool size)</th> <th>MySQL <br /> (with default buffer poll size)</th> </tr> <tr align="right"> <td>1</td> <td> 41,161.33</td> <td> 27,023.11 </td> <td> 12,171.41</td> </tr> <tr align="right"> <td>4</td> <td> 127,918.30</td> <td> (pending) </td> <td> 37,566.82</td> </tr> <tr align="right"> <td>8</td> <td> 218,162.29 </td> <td> 105,524.23 </td> <td> 51,104.39 </td> </tr> <tr align="right"> <td>16</td> <td> 214,763.58 </td> <td> 98,852.42 </td> <td> 47,589.18 </td> </tr> </table> <p>The metric is the query mixes per hour from the BSBM test driver output. For the interested, the complete output is <a href="http://www.openlinksw.com/weblog/oerling/texts/bsbmres.txt" id="link-id1119f770">here</a>.</p> <p>The benchmark is pure <a href="http://dbpedia.org/resource/SQL" id="link-id0x2b61c88">SQL</a>, nothing to do with <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x17a6d408">SPARQL</a> or <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x9a0a968">RDF</a>.</p> <p>The hardware is 2 x Xeon 5345 (2 x quad core, 2.33 GHz), 16 G RAM. The OS is 64-bit Debian Linux.</p> <p>The benchmark was run at a scale of 200,000. Each run had 2000 warm-up query mixes and 500 measured query mixes, which gives steady state, eliminating any effects of OS disk cache and the like. Both databases were configured to use 8G for disk cache. The test effectively runs from memory. We ran an analyze table on each MySQL table but noticed that this had no effect. Virtuoso does the stats sampling on the go; possibly MySQL also since the explicit stats did not make any difference. The MySQL tables were served by the InnoDB engine. MySQL appears to cache results of queries in some cases. This was not apparent in the tests.</p> <p>The versions are 5.09 for Virtuoso and 5.1.29 for MySQL. You can download and examine --</p> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/texts/virtuoso.ini" id="link-id14fe17f0">Virtuoso configuration file</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/texts/my.cnf" id="link-id116fe490">MySQL configuration file</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/texts/create_tables_and_rdf_view.sql" id="link-id14ce9268">Table definitions & RDF views</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/texts/mysqlinx.sql" id="link-id1535e298">Indexes on MySQL tables</a> </li> </ul> <p> <strike>MySQL ought to do better. We suspect that here, just as in the TPC-D experiment we made way back, the query plans are not quite right. Also we rarely saw over 300% CPU utilization for MySQL. It is possible there is a config parameter that affects this. The public is invited to tell us about such.</strike> </p> <p> <b>Update:</b> </p> <p>Andreas Schultz of the BSBM team advised us to increase the <code>innodb_buffer_pool_size</code> setting in the MySQL config. We did and it produced some improvement. Indeed, this is more like it, as we now see CPU utilization around 700% instead of the 300% in the previously published run, which rendered it suspect. Also, our experiments with TPC-D led us to expect better. We ran these things a few times so as to have warm cache.</p> <p>On the first run, we noticed that the Innodb warm up time was somewhere well in excess of 2000 query mixes. Another time, we should make a graph of throughput as a function of time for both MySQL and Virtuoso. We recently made a greedy prefetch hack that should give us some mileage there. For the next BSBM, all we can advise is to run larger scale system for half an hour first and then measure and then measure again. If the second measurement is the same as the first then it is good.</p> <p>As always, since MySQL is not our specialty, we confidently invite the public to tell us how to make it run faster. So, unless something more turns up, our next trial is a revisit of <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x17a20498">TPC-H</a>.</p>
2008-11-24T10:15:11.000021-05:00
Virtuoso Vs. MySQL: Setting the Berlin Record Straight (update 2)
http://www.openlinksw.com:443/weblog/oerling/?date=2008-11-20#1484
2008-11-20T11:06:11Z
<p>In the context of the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0xa5314d8">Berlin SPARQL Benchmark</a>, I have repeatedly written about measurement procedures and steady state. The point is that the numbers at larger scales are unreliable due to cache behavior if one is not careful about measurement and does not have adequate warmup. Thus it came to pass that one cut of the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x18482c20">BSBM</a> paper had 3 seconds for <a href="http://dbpedia.org/resource/MySQL" id="link-id0xb8c54de8">MySQL</a> and 100 for <a href="http://virtuoso.openlinksw.com" id="link-id0x189b2210">Virtuoso</a>, basically through ignoring cache effects.</p> <p>So we decided to do it ourselves.</p> <p>The score is (updated with revised <code>innodb_buffer_pool_size</code> setting, based on advice noted down below):</p> <table border="1" cellspacing="2" cellpadding="5"> <tr> <th>n-clients</th> <th>Virtuoso</th> <th>MySQL <br /> (with increased buffer pool size)</th> <th>MySQL <br /> (with default buffer poll size)</th> </tr> <tr align="right"> <td>1</td> <td> 41,161.33</td> <td> 27,023.11 </td> <td> 12,171.41</td> </tr> <tr align="right"> <td>4</td> <td> 127,918.30</td> <td> (pending) </td> <td> 37,566.82</td> </tr> <tr align="right"> <td>8</td> <td> 218,162.29 </td> <td> 105,524.23 </td> <td> 51,104.39 </td> </tr> <tr align="right"> <td>16</td> <td> 214,763.58 </td> <td> 98,852.42 </td> <td> 47,589.18 </td> </tr> </table> <p>The metric is the query mixes per hour from the BSBM test driver output. For the interested, the complete output is <a href="http://www.openlinksw.com/weblog/oerling/texts/bsbmres.txt" id="link-id1119f770">here</a>.</p> <p>The benchmark is pure <a href="http://dbpedia.org/resource/SQL" id="link-id0x5257718">SQL</a>, nothing to do with <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xb8c463e0">SPARQL</a> or <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x16e68d50">RDF</a>.</p> <p>The hardware is 2 x Xeon 5345 (2 x quad core, 2.33 GHz), 16 G RAM. The OS is 64-bit Debian Linux.</p> <p>The benchmark was run at a scale of 200,000. Each run had 2000 warm-up query mixes and 500 measured query mixes, which gives steady state, eliminating any effects of OS disk cache and the like. Both databases were configured to use 8G for disk cache. The test effectively runs from memory. We ran an analyze table on each MySQL table but noticed that this had no effect. Virtuoso does the stats sampling on the go; possibly MySQL also since the explicit stats did not make any difference. The MySQL tables were served by the InnoDB engine. MySQL appears to cache results of queries in some cases. This was not apparent in the tests.</p> <p>The versions are 5.09 for Virtuoso and 5.1.29 for MySQL. You can download and examine --</p> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/texts/virtuoso.ini" id="link-id14fe17f0">Virtuoso configuration file</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/texts/my.cnf" id="link-id116fe490">MySQL configuration file</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/texts/create_tables_and_rdf_view.sql" id="link-id14ce9268">Table definitions & RDF views</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/texts/mysqlinx.sql" id="link-id1535e298">Indexes on MySQL tables</a> </li> </ul> <p> <strike>MySQL ought to do better. We suspect that here, just as in the TPC-D experiment we made way back, the query plans are not quite right. Also we rarely saw over 300% CPU utilization for MySQL. It is possible there is a config parameter that affects this. The public is invited to tell us about such.</strike> </p> <p> <b>Update:</b> </p> <p>Andreas Schultz of the BSBM team advised us to increase the <code>innodb_buffer_pool_size</code> setting in the MySQL config. We did and it produced some improvement. Indeed, this is more like it, as we now see CPU utilization around 700% instead of the 300% in the previously published run, which rendered it suspect. Also, our experiments with TPC-D led us to expect better. We ran these things a few times so as to have warm cache.</p> <p>On the first run, we noticed that the Innodb warm up time was somewhere well in excess of 2000 query mixes. Another time, we should make a graph of throughput as a function of time for both MySQL and Virtuoso. We recently made a greedy prefetch hack that should give us some mileage there. For the next BSBM, all we can advise is to run larger scale system for half an hour first and then measure and then measure again. If the second measurement is the same as the first then it is good.</p> <p>As always, since MySQL is not our specialty, we confidently invite the public to tell us how to make it run faster. So, unless something more turns up, our next trial is a revisit of <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x122eaa00">TPC-H</a>.</p>
2008-11-24T10:15:05-05:00
Cool Fractal Animations
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-11-08#1483
2008-11-08T16:20:33Z
<p>The <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> is essentially <a href="http://en.wikipedia.org/wiki/Fractal" id="link-id16c7e290">Fractal</a> in form, so when thinking about the Web it sometimes helps to have cool <a href="http://fractalanimation.com/" id="link-id1b189108">fractal animations</a> at one's disposal. </p> <p>Also, when you watch these fractal animations, you should ultimately understand why a centralized approach to "Web Presence" is inherently flawed :-)</p>
2008-11-08T16:07:33.000006-05:00
Master Data Management (MDM) & RDF based Linked Data
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-11-05#1482
2008-11-05T22:53:13Z
<p>It is getting clearer by the second that <a href="http://dbpedia.org/resource/Master_Data_Management" id="link-id167265a8">Master Data Management</a> and RDF based <a href="http://dbpedia.org/resource/Linked_Data" id="link-id17940750">Linked data</a> are two realms separated by a common desire to provide "<a href="http://dbpedia.org/resource/Entity" id="link-id1be08e68">Entity</a> Oriented Data Access" to heterogeneous data sources (within the enterprise and/or across the <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id19b8bdd8">World Wide Web</a>).</p> <p>Here is how I see Linked Data providing tangible value to MDM tools vendors and users:</p> <ol> <li> Open access to Entities across MDM instances served up by different MDM solutions acting as Linked Data publishers (i.e., expose MDM Entities as RDF resources endowed with de-referencable URIs thereby enabling <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11137b48">Hyperdata</a>-style linking) </li> <li> Use of RDF-ization middleware to hook disparate data sources (<a href="http://dbpedia.org/resource/SQL" id="link-id13154ae0">SQL</a>, XML, and other data sources) into existing MDM packages (i.e., the MDM solutions become consumers of RDF Linked Data).</li> </ol> <p> Of course <a href="http://virtuoso.openlinksw.com" id="link-id13b70e20">Virtuoso</a> was designed and developed to deliver the above from day one (circa. 1998 re. the core and 2005 re. the use of RDF for the final mile) as depicted below: </p> <div> <img alt="Image" src="http://virtuoso.openlinksw.com/images/vconc650.jpg" /> </div> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=mdm&type=text&output=html" id="link-id15f8abd8">Other MDM related posts</a> </li> </ul>
2008-11-05T18:19:02-05:00
ISWC 2008: Some Questions
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-11-04#1481
2008-11-04T15:54:42Z
<h2>Inference: Is it always forward chaining?</h2> <p>We got a number of questions about <a href="http://virtuoso.openlinksw.com" id="link-id0x13c64b60">Virtuoso</a>'s inference support. It seems that we are the odd one out, as we do not take it for granted that inference ought to consist of materializing entailment.</p> <p>Firstly, of course one can materialize all one wants with Virtuoso. The simplest way to do this is using SPARUL. With the recent transitivity extensions to <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x14d17778">SPARQL</a>, it is also easy to materialize implications of transitivity with a single statement. Our point is that for trivial entailment such as subclass, sub-property, single transitive property, and <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x128e55d0">owl</a>:sameAs, we do not require materialization, as we can resolve these at run time also, with backward-chaining built into the engine.</p> <p>For more complex situations, one needs to materialize the entailment. At the present time, we know how to generalize our transitive feature to run arbitrary backward-chaining rules, including recursive ones. We could have a sort of Datalog backward-chaining embedded in our <a href="http://dbpedia.org/resource/SQL" id="link-id0x12614770">SQL</a>/SPARQL and could run this with good parallelism, as the transitive feature already works with clustering and partitioning without dying of message latency. Exactly when and how we do this will be seen. Even if users want entailment to be materialized, such a rule system could be used for producing the materialization at good speed.</p> <p>We had a word with <a href="http://web.comlab.ox.ac.uk/people/Ian.Horrocks/" id="link-id117c99d0">Ian Horrocks</a> on the question. He noted that it is often naive on behalf of the community to tend to equate description of semantics with description of algorithm. The <a href="http://dbpedia.org/resource/Data" id="link-id0x145b2980">data</a> need not always be blown up.</p> <p>The advantage of not always materializing is that the working set stays better. Once the working set is no longer in memory, response times jump disproportionately. Also, if the data changes or is retracted or is unreliable, one can end up doing a lot of extra work with materialization. Consider the effect of one malicious sameAs statement. This can lead to a lot of effects that are hard to retract. On the other hand, if running in memory with static data such as the LUBM benchmark, the queries run some 20% faster if entailment subclasses and sub-properties are materialized rather than done at run time.</p> <h2>Genetic Algorithms for SPARQL?</h2> <p>Our compliments for the wildest idea of the conference go to <a href="http://www.eyaloren.org/" id="link-id1a203af8">Eyal Oren</a>, <a href="http://www.few.vu.nl/~cgueret/" id="link-id16208758">Christophe Guéret</a>, and <a href="http://www.few.vu.nl/~schlobac/" id="link-id111923e0">Stefan Schlobach</a>, <i>et al</i>, for their <a href="http://www.informatik.uni-trier.de/~ley/db/conf/semweb/iswc2008.html#OrenGS08" id="link-id11793540">paper on using genetic algorithms for guessing how variables in a SPARQL query ought to be instantiated</a>. Prisoners of our "conventional wisdom" as we are, this might never have occurred to us.</p> <h2>Schema Last?</h2> <p>It is interesting to see how the industry comes to the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x12b57e90">semantic web</a> conferences talking about schema last while at the same time the traditional semantic web people stress enforcing schema constraints and making more predictably performing and database friendlier logics. So do the extremes converge.</p> <p>There is a point to schema last. <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x12a8ff48">RDF</a> is very good for getting a view of ad hoc or unknown data. One can just load and look at what there is. Also, additions of unforeseen optional properties or relations to the schema are easy and efficient. However, it seems that a really high traffic online application would always benefit from having some application specific data structures. Such could also save considerably in hardware.</p> <p>It is not a sharp divide between RDF and relational application oriented representation. We have the capabilities in our RDB to RDF mapping. We just need to show this and have SPARUL and data loading</p>
2008-11-04T14:37:01-05:00
ISWC 2008: Some Questions
http://www.openlinksw.com:443/weblog/oerling/?date=2008-11-04#1479
2008-11-04T15:54:42Z
<h2>Inference: Is it always forward chaining?</h2> <p>We got a number of questions about <a href="http://virtuoso.openlinksw.com" id="link-id0x131604a8">Virtuoso</a>'s inference support. It seems that we are the odd one out, as we do not take it for granted that inference ought to consist of materializing entailment.</p> <p>Firstly, of course one can materialize all one wants with Virtuoso. The simplest way to do this is using SPARUL. With the recent transitivity extensions to <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1422f910">SPARQL</a>, it is also easy to materialize implications of transitivity with a single statement. Our point is that for trivial entailment such as subclass, sub-property, single transitive property, and <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x145894a8">owl</a>:sameAs, we do not require materialization, as we can resolve these at run time also, with backward-chaining built into the engine.</p> <p>For more complex situations, one needs to materialize the entailment. At the present time, we know how to generalize our transitive feature to run arbitrary backward-chaining rules, including recursive ones. We could have a sort of Datalog backward-chaining embedded in our <a href="http://dbpedia.org/resource/SQL" id="link-id0x1458a288">SQL</a>/SPARQL and could run this with good parallelism, as the transitive feature already works with clustering and partitioning without dying of message latency. Exactly when and how we do this will be seen. Even if users want entailment to be materialized, such a rule system could be used for producing the materialization at good speed.</p> <p>We had a word with <a href="http://web.comlab.ox.ac.uk/people/Ian.Horrocks/" id="link-id117c99d0">Ian Horrocks</a> on the question. He noted that it is often naive on behalf of the community to tend to equate description of semantics with description of algorithm. The <a href="http://dbpedia.org/resource/Data" id="link-id0x14cf0b18">data</a> need not always be blown up.</p> <p>The advantage of not always materializing is that the working set stays better. Once the working set is no longer in memory, response times jump disproportionately. Also, if the data changes or is retracted or is unreliable, one can end up doing a lot of extra work with materialization. Consider the effect of one malicious sameAs statement. This can lead to a lot of effects that are hard to retract. On the other hand, if running in memory with static data such as the LUBM benchmark, the queries run some 20% faster if entailment subclasses and sub-properties are materialized rather than done at run time.</p> <h2>Genetic Algorithms for SPARQL?</h2> <p>Our compliments for the wildest idea of the conference go to <a href="http://www.eyaloren.org/" id="link-id1a203af8">Eyal Oren</a>, <a href="http://www.few.vu.nl/~cgueret/" id="link-id16208758">Christophe Guéret</a>, and <a href="http://www.few.vu.nl/~schlobac/" id="link-id111923e0">Stefan Schlobach</a>, <i>et al</i>, for their <a href="http://www.informatik.uni-trier.de/~ley/db/conf/semweb/iswc2008.html#OrenGS08" id="link-id11793540">paper on using genetic algorithms for guessing how variables in a SPARQL query ought to be instantiated</a>. Prisoners of our "conventional wisdom" as we are, this might never have occurred to us.</p> <h2>Schema Last?</h2> <p>It is interesting to see how the industry comes to the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1154c1b0">semantic web</a> conferences talking about schema last while at the same time the traditional semantic web people stress enforcing schema constraints and making more predictably performing and database friendlier logics. So do the extremes converge.</p> <p>There is a point to schema last. <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x14c6a930">RDF</a> is very good for getting a view of ad hoc or unknown data. One can just load and look at what there is. Also, additions of unforeseen optional properties or relations to the schema are easy and efficient. However, it seems that a really high traffic online application would always benefit from having some application specific data structures. Such could also save considerably in hardware.</p> <p>It is not a sharp divide between RDF and relational application oriented representation. We have the capabilities in our RDB to RDF mapping. We just need to show this and have SPARUL and data loading</p>
2008-11-04T14:36:50.000010-05:00
ISWC 2008: Billion Triples Challenge
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-11-04#1480
2008-11-04T15:52:11Z
<p>We showed our billion triples demo at the <a href="http://iswc2008.semanticweb.org/" id="link-id0x14898200">ISWC 2008</a> poster session. Generally people liked what they saw, as we basically did what one always had wanted to do with <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x12c56820">SPARQL</a> but never could. This means firstly full <a href="http://dbpedia.org/resource/SQL" id="link-id0x13a86e38">SQL</a> parity, with sub-queries, aggregation, full text, etc. Beyond SQL, we have transitive sub-queries, <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x12842500">owl</a>:sameAs at run time, and other inference things, all on demand.</p> <p>The live demo is at <a href="http://b3s.openlinksw.com/" id="link-id14ba36e0">http://b3s.openlinksw.com/</a>. This site is under development and may not be on all the time. We are taking it in the direction of hosting the whole <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x14329f58">LOD</a> cloud. This is an evolving operation where we will continue showcasing how one can ask increasingly interesting questions from a growing online database, in the spirit of the billion triples charter.</p> <p>In the words of <a href="http://www.cs.rpi.edu/~hendler/" id="link-id111ad740">Jim Hendler</a>, we were not selected for the finale because this would have made the challenge a database shootout instead of a more research-oriented event. There is some point to this since if the event becomes like the TPC benchmarks, this will limit the entrance to full time database players. Anyway, we got a special mention in the intro of the challenge track.</p> <p>The winner was Semaplorer, a federated SPARQL query system. There is some merit to this, as we ourselves are not convinced that centralization is always the right direction. As discussed in the <i><a href="http://www.openlinksw.com/weblog/oerling/?id=1376" id="link-id1831cce0">DARQ Matter of Federation</a></i> post, we have a notion of how to do this production-strength with our cluster engine, now also over wide area networks. We shall see.</p> <h2>Why Not Just Join?</h2> <p>The entries from Deri and LARKC (<a href="http://www.larkc.eu/marvin/" id="link-id1bb42778">MaRVIN</a>, "Massive <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id19c15d30">RDF</a> Versatile Inference Network") were doing materialization of inference results in a cluster environment. The thing they were not doing was joining across partitions. Thus, the <a href="http://dbpedia.org/resource/Data" id="link-id0x147fb970">data</a> was partitioned on whatever criterion and then the data in each partition was further refined according to rules known to all partitions. Deri did not address joining further.</p> <p>"Nature shall be the guide of the alchemist," goes the old adage. We can look at MaRVIN as an example of this dictum. Networks of people are low bandwidth, not nearly fully connected. Asking a colleague for <a href="http://dbpedia.org/resource/Information" id="link-id0x14c02be8">information</a> is expensive and subject to misunderstanding; asking another research group might never produce an answer.</p> <p>Even looking at one individual, we have no reason to think that the human expert would do complete reasoning. Indeed, the brain is a sort of compute cluster, but it does not have flat latency point to point connectivity — some joins are fast; others are not even tried, for all we know.</p> <p>A database running on a cluster is a sort of counter-example. A database with RDF workload will end up joining across partitions pretty much all of the time.</p> <p>MaRVIN's approach to joining could be likened to a country dance: Boys get to take a whirl with different girls according to a complex pattern. For match-making, some matches are produced early but one never knows if the love of a lifetime might be just around the corner. Also, if the dancers are inexperienced, they will have little ability to evaluate how good a match they have with their partner. A few times around the dance floor are needed to get the hang of things.</p> <p>The question is, at what point will it no longer be possible to join across the database? This depends on the interconnect latency. The higher the latency, the more useful the square-dancing approach becomes.</p> <p>Another practical consideration is the fact that RDF reasoners are not usually built for distributed memory multiprocessors. If the reasoner must be a plug-in component, then it cannot be expected to be written for grids.</p> <p>We can think of a product safety use case: Find cosmetics that have ingredients that are considered toxic in the amounts they are present in each product. This can be done as a database query with some transitive operations, like running through a cosmetics taxonomy and a poisons database. If the business logic deciding whether the presence of an ingredient in the product is a health hazard is very complex, we can get a lot of joins.</p> <p>The MaRVIN way would be to set up a ball where each lipstick and eyeliner dances with every poison and then see if matches are made. The matching logic could be arbitrarily complex since it would run locally. Of course here, some domain <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x14a9f8c0">knowledge</a> is needed in order to set up the processing so that each product and poison carry all the associated information with them. Dancing with half a partner can bias one's perceptions: Again, it is like nature, sometimes not all cards are on the table.</p> <p>It would seem that there is some setup involved before answering a question: Composition of partitions, frequency of result exchange, etc. How critical the domain knowledge implicit in the setup is for the quality of results is an interesting question.</p> <p>The question is, at what point will a cluster using <a href="http://dbpedia.org/resource/federated_database_system" id="link-id0x12b31cf0">distributed database</a> operations for inference become impractical? Of course, it is impractical from the get-go if the reasoners and query processors are not made for this. But what if they are? We are presently evaluating different message patterns for joining between partitions. The baseline is some 250,000 random single-triple lookups per second per core. Using a cluster increases this throughput. The increase is more or less linear depending on whether all intermediate results pass via one coordinating node (worst case) or whether each node can decide which other node will do the next join step for each result (best case). For example, a <code>DISTINCT</code> operation requires that data passes through a single place but <code>JOIN</code>ing and aggregation in general do not.</p> <p>We will still publish numbers during this November.</p>
2008-11-04T14:36:56-05:00
ISWC 2008: Billion Triples Challenge
http://www.openlinksw.com:443/weblog/oerling/?date=2008-11-04#1478
2008-11-04T15:52:11Z
<p>We showed our billion triples demo at the <a href="http://iswc2008.semanticweb.org/" id="link-id0x13a0a520">ISWC 2008</a> poster session. Generally people liked what they saw, as we basically did what one always had wanted to do with <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x138f5798">SPARQL</a> but never could. This means firstly full <a href="http://dbpedia.org/resource/SQL" id="link-id0x1264a688">SQL</a> parity, with sub-queries, aggregation, full text, etc. Beyond SQL, we have transitive sub-queries, <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x1e084138">owl</a>:sameAs at run time, and other inference things, all on demand.</p> <p>The live demo is at <a href="http://b3s.openlinksw.com/" id="link-id14ba36e0">http://b3s.openlinksw.com/</a>. This site is under development and may not be on all the time. We are taking it in the direction of hosting the whole <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x13355f80">LOD</a> cloud. This is an evolving operation where we will continue showcasing how one can ask increasingly interesting questions from a growing online database, in the spirit of the billion triples charter.</p> <p>In the words of <a href="http://www.cs.rpi.edu/~hendler/" id="link-id111ad740">Jim Hendler</a>, we were not selected for the finale because this would have made the challenge a database shootout instead of a more research-oriented event. There is some point to this since if the event becomes like the TPC benchmarks, this will limit the entrance to full time database players. Anyway, we got a special mention in the intro of the challenge track.</p> <p>The winner was Semaplorer, a federated SPARQL query system. There is some merit to this, as we ourselves are not convinced that centralization is always the right direction. As discussed in the <i><a href="http://www.openlinksw.com/weblog/oerling/?id=1376" id="link-id1831cce0">DARQ Matter of Federation</a></i> post, we have a notion of how to do this production-strength with our cluster engine, now also over wide area networks. We shall see.</p> <h2>Why Not Just Join?</h2> <p>The entries from Deri and LARKC (<a href="http://www.larkc.eu/marvin/" id="link-id1bb42778">MaRVIN</a>, "Massive <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id19c15d30">RDF</a> Versatile Inference Network") were doing materialization of inference results in a cluster environment. The thing they were not doing was joining across partitions. Thus, the <a href="http://dbpedia.org/resource/Data" id="link-id0x1d3c1a38">data</a> was partitioned on whatever criterion and then the data in each partition was further refined according to rules known to all partitions. Deri did not address joining further.</p> <p>"Nature shall be the guide of the alchemist," goes the old adage. We can look at MaRVIN as an example of this dictum. Networks of people are low bandwidth, not nearly fully connected. Asking a colleague for <a href="http://dbpedia.org/resource/Information" id="link-id0x125dd698">information</a> is expensive and subject to misunderstanding; asking another research group might never produce an answer.</p> <p>Even looking at one individual, we have no reason to think that the human expert would do complete reasoning. Indeed, the brain is a sort of compute cluster, but it does not have flat latency point to point connectivity — some joins are fast; others are not even tried, for all we know.</p> <p>A database running on a cluster is a sort of counter-example. A database with RDF workload will end up joining across partitions pretty much all of the time.</p> <p>MaRVIN's approach to joining could be likened to a country dance: Boys get to take a whirl with different girls according to a complex pattern. For match-making, some matches are produced early but one never knows if the love of a lifetime might be just around the corner. Also, if the dancers are inexperienced, they will have little ability to evaluate how good a match they have with their partner. A few times around the dance floor are needed to get the hang of things.</p> <p>The question is, at what point will it no longer be possible to join across the database? This depends on the interconnect latency. The higher the latency, the more useful the square-dancing approach becomes.</p> <p>Another practical consideration is the fact that RDF reasoners are not usually built for distributed memory multiprocessors. If the reasoner must be a plug-in component, then it cannot be expected to be written for grids.</p> <p>We can think of a product safety use case: Find cosmetics that have ingredients that are considered toxic in the amounts they are present in each product. This can be done as a database query with some transitive operations, like running through a cosmetics taxonomy and a poisons database. If the business logic deciding whether the presence of an ingredient in the product is a health hazard is very complex, we can get a lot of joins.</p> <p>The MaRVIN way would be to set up a ball where each lipstick and eyeliner dances with every poison and then see if matches are made. The matching logic could be arbitrarily complex since it would run locally. Of course here, some domain <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x133b84b8">knowledge</a> is needed in order to set up the processing so that each product and poison carry all the associated information with them. Dancing with half a partner can bias one's perceptions: Again, it is like nature, sometimes not all cards are on the table.</p> <p>It would seem that there is some setup involved before answering a question: Composition of partitions, frequency of result exchange, etc. How critical the domain knowledge implicit in the setup is for the quality of results is an interesting question.</p> <p>The question is, at what point will a cluster using <a href="http://dbpedia.org/resource/federated_database_system" id="link-id0x1466c1c0">distributed database</a> operations for inference become impractical? Of course, it is impractical from the get-go if the reasoners and query processors are not made for this. But what if they are? We are presently evaluating different message patterns for joining between partitions. The baseline is some 250,000 random single-triple lookups per second per core. Using a cluster increases this throughput. The increase is more or less linear depending on whether all intermediate results pass via one coordinating node (worst case) or whether each node can decide which other node will do the next join step for each result (best case). For example, a <code>DISTINCT</code> operation requires that data passes through a single place but <code>JOIN</code>ing and aggregation in general do not.</p> <p>We will still publish numbers during this November.</p>
2015-08-20T17:23:25.633905-04:00
ISWC 2008: RDB2RDF Face-to-Face
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-11-04#1477
2008-11-04T13:26:19Z
<p>The W3C's RDB-to-<a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x153bdcf8">RDF</a> mapping incubator group (<a href="http://www.w3.org/2005/Incubator/rdb2rdf/" id="link-id0x13e3e6b8">RDB2RDF XG</a>) met in <a href="http://dbpedia.org/resource/Karlsruhe" id="link-id0x15236b08">Karlsruhe</a> after <a href="http://iswc2008.semanticweb.org/" id="link-id0x2450fba8">ISWC 2008</a>.</p> <p>The meeting was about writing a charter for a working group that would define a standard for mapping relational databases to RDF, either for purposes of import into RDF stores or of query mapping from <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x14c84338">SPARQL</a> to <a href="http://dbpedia.org/resource/SQL" id="link-id0x146db368">SQL</a>. There was a lot of agreement and the meeting even finished ahead of the allotted time.</p> <h2>Whose Identifiers?</h2> <p>There was discussion concerning using the <a href="http://dbpedia.org/resource/Entity" id="link-id0x12c15e58">Entity</a> Name Service from the Okkam project for assigning URIs to entities mapped from relational databases. This makes sense when talking about long-lived, legal entities, such as people or companies or geography. Of course, there are cases where this makes no sense; for example, a purchase order or maintenance call hardly needs an identifier registered with the ENS. The problem is, in practice, a CRM could mention customers that have an ENS registered ID (or even several such IDs) and others that have none. Of course, the CRM's reference cannot depend on any registration. Also, even when there is a stable <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x12b7b5c0">URI</a> for the entity, a CRM may need a key that specifies some administrative subdivision of the customer.</p> <p>Also we note that an on-demand RDB-to-RDF mapping may have some trouble dealing with "same as" assertions. If names that are anything other than string forms of the keys in the system must be returned, there will have to be a lookup added to the RDB. This is an administrative issue. Certainly going over the network to ask for names of items returned by queries has a prohibitive cost. It would be good for ad hoc integration to use shared URIs when possible. The trouble of adding and maintaining lookups for these, however, makes this more expensive than just mapping to RDF and using literals for joining between independently maintained systems.</p> <h2> <a href="http://dbpedia.org/resource/XML" id="link-id0x14bf7da0">XML</a> or RDF?</h2> <p>We talked about having a language for human consumption and another for discovery and machine processing of mappings. Would this latter be XML or RDF based? Describing every detail of syntax for a mapping as RDF is really tedious. Also such descriptions are very hard to query, just as <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x1493ffc0">OWL</a> ontologies are. One solution is to have opaque strings embedded into RDF, just like XSLT has <a href="http://dbpedia.org/resource/XPath" id="link-id0x1400fe98">XPath</a> in string form embedded into XML. Maybe it will end up in this way here also. Having a complete XML mapping of the parse tree for mappings, XQueryX-style, could be nice for automatic generation of mappings with XSLT from an XML view of the <a href="http://dbpedia.org/resource/Information" id="link-id0x14c846d8">information</a> schema. But then XSLT can also produce text, so an XML syntax that has every detail of a mapping language as distinct elements is not really necessary for this.</p> <p>Another matter is then describing the RDF generated by the mapping in terms of RDFS or OWL. This would be a by-product of declaring the mapping. Most often, I would presume the target ontology to be given, though, reducing the need for this feature. But if RDF mapping is used for discovery of <a href="http://dbpedia.org/resource/Data" id="link-id0x14f6f128">data</a>, such a description of the exposed data is essential.</p> <h2>Interoperability</h2> <p>We agreed with <a href="http://www.informatik.uni-leipzig.de/~auer/foaf.rdf#me" id="link-id0x1e776730">Sören Auer</a> that we could make <a href="http://virtuoso.openlinksw.com" id="link-id0x1477ad18">Virtuoso</a>'s mapping language compatible with <a href="http://triplify.org/" id="link-id0x15514388">Triplify</a>. Triplify is very simple, extraction only, no SPARQL, but does have the benefit of expressing everything in SQL. As it happens, I would be the last person to tell a web developer what language to program in. So if it is SQL, then let it stay SQL. Technically, a lot of the information the Virtuoso mapping expresses is contained in the Triplify SQL statements, but not all. Some extra declarations are needed still but can have reasonable defaults.</p> <p>There are two ways of stating a mapping. Virtuoso starts with the triple and says which tables and columns will produce the triple. Triplify starts with the SQL statement and says what triples it produces. These are fairly equivalent. For the web developer, the latter is likely more self-evident, while the former may be more compact and have less repetition.</p> <p>Virtuoso and Triplify alone would give us the two interoperable implementations required from a working group, supposing the language were annotations on top of SQL. This would be a guarantee of delivery, as we would be close enough to the result from the get go.</p> <h2>Related Web resources</h2> <ul> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSSQL2RDF" id="link-id14e27040">OpenLink Virtuoso: Open-Source Edition: Mapping SQL Data to RDF</a> </li> <li> <a href="http://virtuoso.openlinksw.com/Whitepapers/pdf/Virtuoso_SQL_to_RDF_Mapping.pdf" id="link-id1baad3a8">Virtuoso RDF Views — Getting Started Guide (PDF)</a> </li> </ul>
2008-11-04T17:20:35-05:00
ISWC 2008: RDB2RDF Face-to-Face
http://www.openlinksw.com:443/weblog/oerling/?date=2008-11-04#1476
2008-11-04T13:26:19Z
<p>The W3C's RDB-to-<a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x141f0470">RDF</a> mapping incubator group (<a href="http://www.w3.org/2005/Incubator/rdb2rdf/" id="link-id0x13b8d018">RDB2RDF XG</a>) met in <a href="http://dbpedia.org/resource/Karlsruhe" id="link-id0x1e748060">Karlsruhe</a> after <a href="http://iswc2008.semanticweb.org/" id="link-id0x1eba8468">ISWC 2008</a>.</p> <p>The meeting was about writing a charter for a working group that would define a standard for mapping relational databases to RDF, either for purposes of import into RDF stores or of query mapping from <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1e5abe10">SPARQL</a> to <a href="http://dbpedia.org/resource/SQL" id="link-id0x13930368">SQL</a>. There was a lot of agreement and the meeting even finished ahead of the allotted time.</p> <h2>Whose Identifiers?</h2> <p>There was discussion concerning using the <a href="http://dbpedia.org/resource/Entity" id="link-id0x15587978">Entity</a> Name Service from the Okkam project for assigning URIs to entities mapped from relational databases. This makes sense when talking about long-lived, legal entities, such as people or companies or geography. Of course, there are cases where this makes no sense; for example, a purchase order or maintenance call hardly needs an identifier registered with the ENS. The problem is, in practice, a CRM could mention customers that have an ENS registered ID (or even several such IDs) and others that have none. Of course, the CRM's reference cannot depend on any registration. Also, even when there is a stable <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x144660f8">URI</a> for the entity, a CRM may need a key that specifies some administrative subdivision of the customer.</p> <p>Also we note that an on-demand RDB-to-RDF mapping may have some trouble dealing with "same as" assertions. If names that are anything other than string forms of the keys in the system must be returned, there will have to be a lookup added to the RDB. This is an administrative issue. Certainly going over the network to ask for names of items returned by queries has a prohibitive cost. It would be good for ad hoc integration to use shared URIs when possible. The trouble of adding and maintaining lookups for these, however, makes this more expensive than just mapping to RDF and using literals for joining between independently maintained systems.</p> <h2> <a href="http://dbpedia.org/resource/XML" id="link-id0x1edb8170">XML</a> or RDF?</h2> <p>We talked about having a language for human consumption and another for discovery and machine processing of mappings. Would this latter be XML or RDF based? Describing every detail of syntax for a mapping as RDF is really tedious. Also such descriptions are very hard to query, just as <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x2450fba8">OWL</a> ontologies are. One solution is to have opaque strings embedded into RDF, just like XSLT has <a href="http://dbpedia.org/resource/XPath" id="link-id0x234e5478">XPath</a> in string form embedded into XML. Maybe it will end up in this way here also. Having a complete XML mapping of the parse tree for mappings, XQueryX-style, could be nice for automatic generation of mappings with XSLT from an XML view of the <a href="http://dbpedia.org/resource/Information" id="link-id0x22e129f8">information</a> schema. But then XSLT can also produce text, so an XML syntax that has every detail of a mapping language as distinct elements is not really necessary for this.</p> <p>Another matter is then describing the RDF generated by the mapping in terms of RDFS or OWL. This would be a by-product of declaring the mapping. Most often, I would presume the target ontology to be given, though, reducing the need for this feature. But if RDF mapping is used for discovery of <a href="http://dbpedia.org/resource/Data" id="link-id0x155139c0">data</a>, such a description of the exposed data is essential.</p> <h2>Interoperability</h2> <p>We agreed with <a href="http://www.informatik.uni-leipzig.de/~auer/foaf.rdf#me" id="link-id0x132a64e0">Sören Auer</a> that we could make <a href="http://virtuoso.openlinksw.com" id="link-id0x1272c988">Virtuoso</a>'s mapping language compatible with <a href="http://triplify.org/" id="link-id0x12622738">Triplify</a>. Triplify is very simple, extraction only, no SPARQL, but does have the benefit of expressing everything in SQL. As it happens, I would be the last person to tell a web developer what language to program in. So if it is SQL, then let it stay SQL. Technically, a lot of the information the Virtuoso mapping expresses is contained in the Triplify SQL statements, but not all. Some extra declarations are needed still but can have reasonable defaults.</p> <p>There are two ways of stating a mapping. Virtuoso starts with the triple and says which tables and columns will produce the triple. Triplify starts with the SQL statement and says what triples it produces. These are fairly equivalent. For the web developer, the latter is likely more self-evident, while the former may be more compact and have less repetition.</p> <p>Virtuoso and Triplify alone would give us the two interoperable implementations required from a working group, supposing the language were annotations on top of SQL. This would be a guarantee of delivery, as we would be close enough to the result from the get go.</p> <h2>Related Web resources</h2> <ul> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSSQL2RDF" id="link-id14e27040">OpenLink Virtuoso: Open-Source Edition: Mapping SQL Data to RDF</a> </li> <li> <a href="http://virtuoso.openlinksw.com/Whitepapers/pdf/Virtuoso_SQL_to_RDF_Mapping.pdf" id="link-id1baad3a8">Virtuoso RDF Views — Getting Started Guide (PDF)</a> </li> </ul>
2008-11-04T17:20:30.000002-05:00
YODA & the Data FORCE
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-11-03#1474
2008-11-03T17:32:49Z
<p> The original <a href="http://www.w3.org/History/1989/proposal.html" id="link-id13b25ba8">design document</a> (by <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id181e4c70">TimBL</a>) that lead to the WWW (*an important read*) was very clear about the need to create an "<a href="http://dbpedia.org/resource/Information" id="link-id10f23918">information</a> space" that connects heterogeneous <a href="http://dbpedia.org/resource/Data">data</a> sources. Unfortunately, in trying to create a moniker to distinguish one aspect of the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> (the Linked Document Web) from the part that was overlooked (the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11096818">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id1b9c6b98">Web</a>), we ended up with a project code name that's fundamentally a misnomer in the form of: "The <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id10ffe228">Semantic Web</a>".</p> <p> If we could just take "The Semantic Web" moniker for what it was -- a code name for an aspect of the Web -- and move on, things will get much clearer, fast!</p> <p> Basically, what is/was the "Semantic Web" should really have been code named: ("You" Oriented Data Access) as a play on: Yoda's appreciation of the FORCE (Fact ORiented Connected Entities) -- the power of inter galactic, interlinked, structured data, fashioned by the <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id191b22e0">World Wide Web</a> courtesy of the HTTP protocol.</p> <div> <img alt="Image" src="http://motivationalspeaker1.files.wordpress.com/2008/03/yoda.jpg" /> </div> <p> As stated in a earlier post, the next phase of the Web is all about the magic of <a href="http://dbpedia.org/resource/Entity" id="link-id1a7395f0">entity</a> "You". The single most important item of reference to every Web user would be the Person Entity <a href="http://dbpedia.org/resource/Identity_%28object-oriented_programming%29" id="link-id16ab9308">ID</a> (<a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id1d403c88">URI</a>). Just by remembering your Entity ID, you will have intelligent pathways across, and into, the FORCE that the Linked Data Web delivers. The quality of the pathways and increased density of the FORCE are the keys to high <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1442" id="link-id1c549b28">SDQ</a> (tomorrows SEO). Thus, the SDQ of URIs will ultimately be the unit determinant of value to Web Users, along the following personal lines, hence the critical platform questions:</p> <ul> <li> Does your platform give <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id175afe00">me</a> Identity (a URI) with high SDQ?</li> <li> Do the Data Source Names (URIs) in your Data Spaces deliver high SDQ?</li> </ul> <p> While most industry commentators continue to ponder and pontificate about what "The Semantic Web" is (unfortunately), the real thing (the "FORCE") is already here, and self-enhancing rapidly.</p> <p> Assuming we now accept the FORCE is simply an RDF based Linked Data moniker, and that RDF Linked Data is all about the Web as a structured database, we should start to move our attention over to practical exploitation of this burgeoning global database, and in doing so we should not discard <a href="http://dbpedia.org/resource/Knowledge" id="link-id19e2c6e0">knowledge</a> from the past such as the many great examples available gratis from the Relational Database realm. For instance, we should start paying attention to the discovery, development, and deployment of high level tools such as query builders, report writers, and intelligence oriented analytic tools, none of which should -- at first point of interaction -- expose raw RDF or the <a href="http://dbpedia.org/resource/SPARQL" id="link-id117921f0">SPARQL</a> query language. Along similar lines of thinking, we also need development environments and frameworks that are counterparts to Visual Studio, ACCESS, File Maker, and the like.</p> <h3> Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1458" id="link-id1cec1a40">Numerati & The Magic of You!</a> </li> </ul>
2010-07-20T13:53:06-04:00
Entity Oriented Data Access
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-11-03#1475
2008-11-03T17:32:08Z
<p>Recent <a href="http://en.wiktionary.org/wiki/perturbation" id="link-id1bdb9ec8">perturbations</a> in <a href="http://dbpedia.org/resource/Data">Data</a> Access and Data Management technology realms are clear signs of an imminent inflection. In a nutshell, the focus of data access is moving from the "Logical Level" (what you see if you've ever looked at a DBMS schema derived from an <a href="http://dbpedia.org/resource/Entity" id="link-id18735f38">Entity</a> Data Model) to the "Conceptual Level" (i.e., the Entity Model becoming concrete).</p> <p>In recent times I've stumbled across Master Data Management (MDM) which is all about entities that provide holistic views of enterprise data (or what I call: <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id18f07ec8">Context</a> Lenses). I've also stumbled across emerging tensions in the .NET realm between Linq to Entities and Linq to <a href="http://dbpedia.org/resource/SQL" id="link-id19429e88">SQL</a>, where in either case the fundamental issues comes down to the optimal paths "Conceptual Level Access" over the "Logical Logical Level" when dealing with data access in the .NET realm.</p> <p> Strangely, the emerging realm of RDF <a href="http://dbpedia.org/resource/Linked_Data" id="link-id115b3780">Linked Data</a>, MDM, and .NET's Entity Frameworks, remain strangely disconnected.</p> <p>Another oddity is the obvious, but barely acknowledged, blurring of the lines between the "traditional enterprise employee" and the "individual <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> <a href="http://dbpedia.org/resource/Netizen" id="link-id0x1ffd8640">netizen</a>". The fusion between these entities is one of the most defining characteristics of how the Web is reshaping the data landscape.</p> <p>At the current time, I tend to crystalize my data access world view under the moniker: <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1474" id="link-id1544ee60">YODA</a> ("You" Oriented Data Access), based on the following:</p> <ol> <li> Entities are the new focal point of data access, management, and integration </li> <li> "You" are the entry point (Data Source Name) into this new realm of inter connected Entities that the Web exposes</li> <li> "You" the "Person" Entity is associated with many other "Things" such as "Organizations", "Other People", "Books", "Music", "Subject Matter" etc. </li> <li> "You" the "Person" needs Identity in this new global database, which is why "You" need to Identify "Yourself" using an an HTTP based Entity <a href="http://dbpedia.org/resource/Identity_%28object-oriented_programming%29" id="link-id145d0438">ID</a> (aka. <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id1873ad08">URI</a>) </li> <li> When "You" have an ID for "Yourself" it becomes much easier for the essence of "You" to be discovered via the Web </li> <li> When "Others" have IDs for "Themselves" on the Web it becomes much easier for "You" to serendipitously discover or explicitly "Find" things on the Web. </li> </ol> <h3>Related</h3> <ul> <li> <a href="http://www.infoq.com/news/2008/11/DLINQ-Future" id="link-id17501eb0">Is LINQ to SQL truly dead?</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1420" id="link-id10fbf920">Virtuoso, Linked Data, and Linq2Rdf</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1224" id="link-id19c44b00">Enterprise 0.0, Linked Data, and the Semantic Data Web</a> (*an old post*)</li> </ul>
2008-11-03T22:51:48-05:00
ISWC 2008: The Scalable Knowledge Systems Workshop
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-11-03#1473
2008-11-03T13:16:47Z
<p>Mike Dean of <a href="http://dbpedia.org/resource/BBN_Technologies" id="link-id0x25699878">BBN Technologies</a> opened the Scalable <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x1ed01750">Knowledge</a> Systems Workshop with an invited talk. He reminded us of the facts of nature as concern the cost of distributed computing and running out of space for the working set. Developers in the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x21fbb9a8">semantic web</a> field deplorably often ignore these facts, or alternatively recognize them and admit that they are unbeatable, that one just can't join across partitions.</p> <p>I gave a talk about the <a href="http://virtuoso.openlinksw.com" id="link-id0x20b6e020">Virtuoso</a> Cluster edition, wherein I repeated essentially the same ground facts as Mike and outlined how we (in spite of these) profit from distributed memory multiprocessing. To those not intimate with these questions, let me affirm that deriving benefit from threading in a symmetric multiprocessor box, let alone a cluster connected by a network, totally depends on having many relatively long running things going at a time and blocking as seldom as possible.</p> <p>Further, Mike Dean talked about <a href="http://www.asio.bbn.com/" id="link-id0x222252f0">ASIO</a>, the BBN suite of semantic web tools. His most challenging statement was about the storage engine, a network-database-inspired triple-store using memory-mapped files. </p> <p>Will the <a href="http://dbpedia.org/resource/CODASYL" id="link-id0x222d8730">CODASYL</a> days come back, and will the linked list on disk be the way to store triples/quads? I would say that this will have, especially with a memory-mapped file, probably a better best-case as a B-tree but that this also will be less predictable with fragmentation. With Virtuoso, using a B-tree index, we see about 20-30% of CPU time spent on index lookup when running LUBM queries. With a disk-based memory-mapped linked-list storage, we would see some improvements in this while getting hit probably worse than now in the case of fragmentation. Plus compaction on the fly would not be nearly as easy and surely far less local, if there were pointers between pages. So it is my intuition that trees are a safer bet with varying workloads while linked lists can be faster in a query-dominated in-memory situation.</p> <p>Chris Bizer presented the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x22e41c40">Berlin SPARQL Benchmark</a> (<a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1c909960">BSBM</a>), which has already been discussed here in some detail. He did acknowledge that the next round of the race must have a real steady-state rule. This just means that the benchmark must be run long enough for the system under test to reach a state where the cache is full and the performance remains indefinitely at the same level. Reaching steady state can take 20-30 minutes in some cases.</p> <p>Regardless of steady state, BSBM has two generally valid conclusions: </p> <ol> <li>mapping relational to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x21d01890">RDF</a>, where possible, is faster than triple storage; and </li> <li>the equivalent relational solution can be some 10x faster than the pure triples representation.</li> </ol> <p>Mike Dean asked whether BSBM was a case of a setup to have triple stores fail. Not necessarily, I would say; we should understand that one motivation of BSBM is testing mapping technologies. Therefore it must have a workload where mapping makes sense. Of course there are workloads where triples are unchallenged — take the <a href="http://challenge.semanticweb.org/" id="link-id0x1feb9250">Billion Triples Challenge</a> <a href="http://dbpedia.org/resource/Data" id="link-id0x1fe12b60">data</a> set for one.</p> <p>Also, with BSBM, once should note that the query optimization time plays a fairly large role since most queries touch relatively little data. Also, even if the scale is large, the working set is not nearly the size of the database. This in fact penalizes mapping technologies against native <a href="http://dbpedia.org/resource/SQL" id="link-id0x1e275c88">SQL</a> since the difference there is compiling the query, especially since parameters are not used. So, Chris, since we both like to map, let's make a benchmark that shows mapping closer to native SQL.</p> <h2>Bridging the 10x Gap?</h2> <p>When we run Virtuoso relational against Virtuoso triple store with the <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x22046d88">TPC-H</a> workload, we see that the relational case is significantly faster. These are long queries, thus query optimization time is negligible; we are here comparing memory-based access times. Why is this? The answer is that a single index lookup gives multiple column values with almost no penalty for the extra column. Also, since the number of total joins is lower, the overhead coming from moving from join to next join is likewise lower. This is just a meter of count of executed instructions.</p> <p>A column store joins in principle just as much as a triple store. However, since the BI workload often consists of scanning over large tables, the joins tend to be local, the needed lookup can often use the previous location as a starting point. A triple store can do the same if queries have high locality. We do this in some SQL situations and can try this with triples also. The RDF workload is typically more random in its access pattern, though. The other factor is the length of control path. A column store has a simpler control flow if it knows that the column will have exactly one value per row. With RDF, this is not a given. Also, the column store's row is identified by a single number and not a multipart key. These two factors give the column store running with a fixed schema some edge over the more generic RDF quad store.</p> <p>There was some discussion on how much closer a triple store could come to a relational one. Some gains are undoubtedly possible. We will see. For the ideal row store workload, the <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x22f837c0">RDBMS</a> will continue to have some edge. Large online systems typically have a large part of the workload that is simple and repetitive. There is nothing to prevent one having special indices for supporting such workload, even while retaining the possibility of arbitrary triples elsewhere. Some degree of application-specific data structure does make sense. We just need to show how this is done. In this way, we have a continuum and not an either/or choice of triples vs. tables.</p> <h2>Scale, Where Next?</h2> <p>Concerning the future direction of the workshop, there were a few directions suggested. One of the more interesting ones was Mike Dean's suggestion about dealing with a large volume of same-as assertions, specifically a volume where materializing all the entailed triples was no longer practical. Of course, there is the question of scale. This time, we were the only ones focusing on a parallel database with no restrictions on joining.</p>
2008-11-03T12:33:54-05:00
ISWC 2008: The Scalable Knowledge Systems Workshop
http://www.openlinksw.com:443/weblog/oerling/?date=2008-11-03#1471
2008-11-03T13:16:47Z
<p>Mike Dean of <a href="http://dbpedia.org/resource/BBN_Technologies" id="link-id0x21d04768">BBN Technologies</a> opened the Scalable <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x22348c58">Knowledge</a> Systems Workshop with an invited talk. He reminded us of the facts of nature as concern the cost of distributed computing and running out of space for the working set. Developers in the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x22570328">semantic web</a> field deplorably often ignore these facts, or alternatively recognize them and admit that they are unbeatable, that one just can't join across partitions.</p> <p>I gave a talk about the <a href="http://virtuoso.openlinksw.com" id="link-id0x23f313f0">Virtuoso</a> Cluster edition, wherein I repeated essentially the same ground facts as Mike and outlined how we (in spite of these) profit from distributed memory multiprocessing. To those not intimate with these questions, let me affirm that deriving benefit from threading in a symmetric multiprocessor box, let alone a cluster connected by a network, totally depends on having many relatively long running things going at a time and blocking as seldom as possible.</p> <p>Further, Mike Dean talked about <a href="http://www.asio.bbn.com/" id="link-id0x1d74c108">ASIO</a>, the BBN suite of semantic web tools. His most challenging statement was about the storage engine, a network-database-inspired triple-store using memory-mapped files. </p> <p>Will the <a href="http://dbpedia.org/resource/CODASYL" id="link-id0x1f8ee860">CODASYL</a> days come back, and will the linked list on disk be the way to store triples/quads? I would say that this will have, especially with a memory-mapped file, probably a better best-case as a B-tree but that this also will be less predictable with fragmentation. With Virtuoso, using a B-tree index, we see about 20-30% of CPU time spent on index lookup when running LUBM queries. With a disk-based memory-mapped linked-list storage, we would see some improvements in this while getting hit probably worse than now in the case of fragmentation. Plus compaction on the fly would not be nearly as easy and surely far less local, if there were pointers between pages. So it is my intuition that trees are a safer bet with varying workloads while linked lists can be faster in a query-dominated in-memory situation.</p> <p>Chris Bizer presented the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1d670da0">Berlin SPARQL Benchmark</a> (<a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x21928808">BSBM</a>), which has already been discussed here in some detail. He did acknowledge that the next round of the race must have a real steady-state rule. This just means that the benchmark must be run long enough for the system under test to reach a state where the cache is full and the performance remains indefinitely at the same level. Reaching steady state can take 20-30 minutes in some cases.</p> <p>Regardless of steady state, BSBM has two generally valid conclusions: </p> <ol> <li>mapping relational to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xab811020">RDF</a>, where possible, is faster than triple storage; and </li> <li>the equivalent relational solution can be some 10x faster than the pure triples representation.</li> </ol> <p>Mike Dean asked whether BSBM was a case of a setup to have triple stores fail. Not necessarily, I would say; we should understand that one motivation of BSBM is testing mapping technologies. Therefore it must have a workload where mapping makes sense. Of course there are workloads where triples are unchallenged — take the <a href="http://challenge.semanticweb.org/" id="link-id0x2538c3b8">Billion Triples Challenge</a> <a href="http://dbpedia.org/resource/Data" id="link-id0x1d673760">data</a> set for one.</p> <p>Also, with BSBM, once should note that the query optimization time plays a fairly large role since most queries touch relatively little data. Also, even if the scale is large, the working set is not nearly the size of the database. This in fact penalizes mapping technologies against native <a href="http://dbpedia.org/resource/SQL" id="link-id0xac16cc10">SQL</a> since the difference there is compiling the query, especially since parameters are not used. So, Chris, since we both like to map, let's make a benchmark that shows mapping closer to native SQL.</p> <h2>Bridging the 10x Gap?</h2> <p>When we run Virtuoso relational against Virtuoso triple store with the <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x1d7dc518">TPC-H</a> workload, we see that the relational case is significantly faster. These are long queries, thus query optimization time is negligible; we are here comparing memory-based access times. Why is this? The answer is that a single index lookup gives multiple column values with almost no penalty for the extra column. Also, since the number of total joins is lower, the overhead coming from moving from join to next join is likewise lower. This is just a meter of count of executed instructions.</p> <p>A column store joins in principle just as much as a triple store. However, since the BI workload often consists of scanning over large tables, the joins tend to be local, the needed lookup can often use the previous location as a starting point. A triple store can do the same if queries have high locality. We do this in some SQL situations and can try this with triples also. The RDF workload is typically more random in its access pattern, though. The other factor is the length of control path. A column store has a simpler control flow if it knows that the column will have exactly one value per row. With RDF, this is not a given. Also, the column store's row is identified by a single number and not a multipart key. These two factors give the column store running with a fixed schema some edge over the more generic RDF quad store.</p> <p>There was some discussion on how much closer a triple store could come to a relational one. Some gains are undoubtedly possible. We will see. For the ideal row store workload, the <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x22e5b6f8">RDBMS</a> will continue to have some edge. Large online systems typically have a large part of the workload that is simple and repetitive. There is nothing to prevent one having special indices for supporting such workload, even while retaining the possibility of arbitrary triples elsewhere. Some degree of application-specific data structure does make sense. We just need to show how this is done. In this way, we have a continuum and not an either/or choice of triples vs. tables.</p> <h2>Scale, Where Next?</h2> <p>Concerning the future direction of the workshop, there were a few directions suggested. One of the more interesting ones was Mike Dean's suggestion about dealing with a large volume of same-as assertions, specifically a volume where materializing all the entailed triples was no longer practical. Of course, there is the question of scale. This time, we were the only ones focusing on a parallel database with no restrictions on joining.</p>
2008-11-03T12:33:49-05:00
ISWC 2008 Conference reports
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-11-03#1472
2008-11-03T13:12:18Z
<p>In the next few posts, I will talk about the <a href="http://iswc2008.semanticweb.org/" id="link-id0x1154d6a8">ISWC 2008</a> conference of last week in <a href="http://dbpedia.org/resource/Karlsruhe" id="link-id0x139eec40">Karlsruhe</a>, <a href="http://dbpedia.org/resource/Germany" id="link-id0x138b5dc8">Germany</a>.</p> <p>For us, this is divided into</p> <ul> <li>Workshop on scalable <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x23561d00">knowledge</a> systems</li> <li>General Conference </li> <li>The <a href="http://challenge.semanticweb.org/" id="link-id0x127a0d88">Billion Triples Challenge</a> </li> <li>The W3C <a href="http://www.w3.org/2005/Incubator/rdb2rdf/" id="link-id0x212e2390">RDB2RDF XG</a> Face-to-Face</li> </ul> <p>I will talk about each in turn.</p>
2008-11-03T09:33:06.000004-05:00
ISWC 2008 Conference reports
http://www.openlinksw.com:443/weblog/oerling/?date=2008-11-03#1470
2008-11-03T13:12:18Z
<p>In the next few posts, I will talk about the <a href="http://iswc2008.semanticweb.org/" id="link-id0x136691d8">ISWC 2008</a> conference of last week in <a href="http://dbpedia.org/resource/Karlsruhe" id="link-id0x1d375a98">Karlsruhe</a>, <a href="http://dbpedia.org/resource/Germany" id="link-id0x13496660">Germany</a>.</p> <p>For us, this is divided into</p> <ul> <li>Workshop on scalable <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x1d99cd18">knowledge</a> systems</li> <li>General Conference </li> <li>The <a href="http://challenge.semanticweb.org/" id="link-id0x1d310ab8">Billion Triples Challenge</a> </li> <li>The W3C <a href="http://www.w3.org/2005/Incubator/rdb2rdf/" id="link-id0x201271a8">RDB2RDF XG</a> Face-to-Face</li> </ul> <p>I will talk about each in turn.</p>
2008-11-03T09:33:06-05:00
Virtuoso Installation Screencasts
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-11-02#1469
2008-11-02T01:44:27Z
<p>As promised in an earlier post titled: <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1461" id="link-id1c412298">Virtuoso, PHP 3.5 Runtime Hosting, phpBB3, and Linked Data</a>, here are direct links to the "silent movies" mentioned in the past:</p> <ul> <li> <a href="http://my-movies.s3.amazonaws.com/Virtuoso_PHPBB3_Vista_Linked_Data_Demo.mov" id="link-id13ea5790">Installing Virtuoso on Vista with PHP Hosting</a> </li> <li> <a href="http://my-movies.s3.amazonaws.com/Virtuoso_PHPBB3_MacOSX_Linked_Data_Demo.mov" id="link-id135299d8">Installing Virtuoso on Mac OS X (Leopard) with PHP Hosting</a> </li> <li> <a href="http://my-movies.s3.amazonaws.com/Virtuoso_PHPBB3_MacOSX_Linked_Data_Demo.mov" id="link-id1275fd88">EC2 Installation Part 1</a> (*AMIs take about 5 minutes to get assembled*)</li> <li> <a href="http://my-movies.s3.amazonaws.com/Virtuoso_PHPBB3_MacOSX_Linked_Data_Demo.mov" id="link-id10f78ae8">EC2 Installation Part 2</a> (*post AMI creation part*)</li> </ul> <p>Virtuoso is an extremely compact product that is very easy to install. The ease of installation carries over to the PHP runtime when bound to Virtuoso.</p>
2008-11-02T16:20:21-05:00
Welcoming Freebase to the Linked Data Web
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-10-31#1468
2008-10-31T15:02:00Z
Finally! That's all I can say re. Freebase :-) They've now plugged their database and their community driven <a href="http://dbpedia.org/resource/Data">data</a> curation efforts into the burgeoning <a href="http://dbpedia.org/resource/Linked_Data" id="link-id111fe3b0">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id1cd46860">Web</a>. <p>Here are some examples of how we distill Entities (People, Places, Music, and other things) from Freebase (X)HTML pages (meaning: we don't have to start from RDF <a href="http://dbpedia.org/resource/Information" id="link-id1115cfe8">information</a> resources as data sources for the eventual RDF Linked Data we generate):</p> <ul> <li> <a href="http://linkeddata.uriburner.com/about/html/http://www.freebase.com/view/en/barack_obama" id="link-id1957da00">Barack Obama</a> </li> <li> <a href="http://linkeddata.uriburner.com/about/html/http://www.freebase.com/view/en/hillary_rodham_clinton" id="link-id175786d8">Hillary Rodham Clinton</a> </li> <li> <a href="http://linkeddata.uriburner.com/about/html/http://www.freebase.com/view/en/john_mccain" id="link-id1c7ada58">Johan McCain</a> </li> </ul> <p>Tip: Install our <a href="http://ode.openlinksw.com" id="link-id17a69a20">OpenLink Data Explorer</a> extension for Firefox. Once installed, simply browse through Freebase, and whenever you encounter a page about something of interest, simply use the following sequences to distill (via the Page Description feature) the entities from the page you are reading:</p> <ul> <li> CTRL-Click (Mac OS X) </li> <li> Right+Click (Windows & Linux) </li> </ul> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1455" id="link-id17758840">State of the Linked Data Web</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1454" id="link-idea627e8">Dynamic Linked Data Web Constellation</a> </li> </ul>
2008-10-31T11:23:35.000002-04:00
Virtuoso - Are We Too Clever for Our Own Good? (updated)
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-10-26#1467
2008-10-26T12:15:35Z
<p>"Physician, heal thyself," it is said. We profess to say what the messaging of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1b4a25f0">semantic web</a> ought to be, but is our own perfect?</p> <p>I will here engage in some critical introspection as well as amplify on some answers given to <a href="http://virtuoso.openlinksw.com" id="link-id0x1e4f9928">Virtuoso</a>-related questions in recent times.</p> <p>I use some conversations from the <a href="http://dbpedia.org/resource/Vienna" id="link-id0x1e6c0ca8">Vienna</a> <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1e56df88">Linked Data</a> Practitioners meeting as a starting point. These views are mine and are limited to the Virtuoso server. These do not apply to the <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id0x1e680440">ODS</a> (<a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id0x1e140068">OpenLink Data Spaces</a>) applications line, <a href="http://oat.openlinksw.com/" id="link-id0x1f4ba630">OAT</a> (<a href="http://oat.openlinksw.com/" id="link-id0x1ba4bac8">OpenLink Ajax Toolkit</a>), or <a href="http://ode.openlinksw.com/" id="link-id0x1d4159b0">ODE</a> (<a href="http://ode.openlinksw.com/" id="link-id0x1e973c80">OpenLink Data Explorer</a>).</p> <h3>"It is not always clear what the main thrust is, we get the impression that you are spread too thin," said <a href="http://www.informatik.uni-leipzig.de/~auer/foaf.rdf#me" id="link-id0x1f8bafe0">Sören Auer</a>.</h3> <p>Well, personally, I am all for core competence. This is why I do not participate in all the online conversations and groups as much as I could, for example. Time and energy are critical resources and must be invested where they make a difference. In this case, the real core competence is running in the database race. This in itself, come to think of it, is a pretty broad concept.</p> <p>This is why we put a lot of emphasis on Linked Data and the <a href="http://dbpedia.org/resource/Data" id="link-id0x200bd1f0">Data</a> Web for now, as this is the emerging game. This is a deliberate choice, not an outside imperative or built-in limitation. More specifically, this means exposing any pre-existing relational data as linked data plus being the definitive <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1fb03528">RDF</a> store.</p> <p>We can do this because we own our database and <a href="http://dbpedia.org/resource/SQL" id="link-id0x1e7dcc70">SQL</a> and data access middleware and have a history of connecting to any <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1e9baf18">RDBMS</a> out there.</p> <p>The principal message we have been hearing from the RDF field is the call for scale of triple storage. This is even louder than the call for relational mapping. We believe that in time mapping will exceed triple storage as such, once we get some real production strength mappings deployed, enough to outperform RDF warehousing.</p> <p>There are also RDF middleware things like RDF-ization and demand-driven web harvesting (i.e, the so-called Sponger). These are <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1f5f6b78">SPARQL</a> options, thus accessed via standard interfaces. We have little desire to create our own languages or APIs, or to tell people how to program. This is why we recently introduced <a href="http://sourceforge.net/projects/sesame/" id="link-id0x206818c8">Sesame</a>- and <a href="http://jena.sourceforge.net/" id="link-id0x202b3348">Jena</a>-compatible APIs to our RDF store. From what we hear, these work. On the other hand, we do not hesitate to move beyond the standards when there is obvious value or necessity. This is why we brought SPARQL up to and beyond SQL expressivity. It is not a case of E3 (Embrace, Extend, Extinguish).</p> <p>Now, this message could be better reflected in our material on the web. This <a href="http://dbpedia.org/resource/Blog" id="link-id0x1c82e508">blog</a> is a rather informal step in this direction; more is to come. For now we concentrate on delivering.</p> <p>The conventional communications wisdom is to split the message by target audience. For this, we should split the RDF, relational, and web services messages from each other. We believe that a challenger, like the semantic web technology stack, must have a compelling message to tell for it to be interesting. This is not a question of research prototypes. The new technology cannot lack something the installed technology takes for granted.</p> <p>This is why we do not tend to show things like how to insert and query a few triples: No business out there will insert and query triples for the sake of triples. There must be a more compelling story — for example, turning the whole world into a database. This is why our examples start with things like turning the <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x20832510">TPC-H</a> database into RDF, queries and all. Anything less is not interesting. Why would an enterprise that has business intelligence and integration issues way more complex than the rather stereotypical TPC-H even look at a technology that pretends to be all for integration and all for expressivity of queries, yet cannot answer the first question of the entry exam?</p> <p>The world out there is complex. But maybe we ought to make some simple tutorials? So, as a call to the people out there, tell us what a good tutorial would be. The question is more about figuring out what is out there and adapting these and making a sort of compatibility list. Jena and Sesame stuff ought to run as is. We could offer a webinar to all the data web luminaries showing how to promote the data web message with Virtuoso. After all, why not show it on the best platform?</p> <h3>"You are arrogant. When I read your papers or documentation, the impression I get is that you say you are smart and the reader is stupid."</h3> <p>We should answer in multiple parts.</p> <p>For general collateral, like web sites and documentation:</p> <p>The web site gives a confused product image. For the Virtuoso product, we should divide at the top into</p> <ul> <li> Data web and RDF - Host linked data, expose relational assets as linked data;</li> <li> Relational Database - Full function, high performance, open source, Federated/Virtual Relational DBMS, expose heterogeneous RDB assets through one point of contact for integration;</li> <li> Web Services - access all the above over standard protocols, dynamic web pages, web hosting.</li> </ul> <p>For each point, one simple statement. We all know what the above things mean?</p> <p>Then we add a new point about scalability that impacts all the above, namely the Virtuoso version 6 Cluster, meaning that you can do all these things at 10 to 1000 times the scale. This means this much more data or in some cases this much more requests per second. This too is clear.</p> <p>Far as I am concerned, hosting Java or .<a href="http://dbpedia.org/resource/.NET_Framework" id="link-id0x20283a88">NET</a> does not have to be on the front page. Also, we have no great interest in going against <a href="http://dbpedia.org/resource/Apache" id="link-id0x2024a068">Apache</a> when it comes to a web server only situation. The fact that we have a web listener is important for some things but our claim to fame does not rest on this.</p> <p>Then for documentation and training materials: The documentation should be better. Specifically it should have more of a how-to dimension since nobody reads the whole thing anyhow. About online tutorials, the order of presentation should be different. They do not really reflect what is important at the present moment either.</p> <p>Now for conference papers: Since taking the data web as a focus area, we have submitted some papers and had some rejected because these do not have enough references and do not explain what is obvious to ourselves.</p> <p>I think that the communications failure in this case is that we want to talk about end to end solutions and the reviewers expect research. For us, the solution is interesting and exists only if there is an adequate functionality mix for addressing a specific use case. This is why we do not make a paper about query cost model alone because the cost model, while indispensable, is a thing that is taken for granted where we come from. So we mention RDF adaptations to cost model, as these are important to the whole but do not find these to be the justification for a whole paper. If we made papers on this basis, we would have to make five times as many. Maybe we ought to.</p> <h3>"Virtuoso is very big and very difficult"</h3> <p>One thing that is not obvious from the Virtuoso packaging is that the minimum installation is an executable under 10MB and a config file. Two files.</p> <p>This gives you SQL and SPARQL out of the box. Adding <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x1ee61058">ODBC</a> and <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x1b8c31c0">JDBC</a> clients is as simple as it gets. After this, there is basic database functionality. Tuning is a matter of a few parameters that are explained on this blog and elsewhere. Also, the full scale installation is available as an Amazon EC2 image, so no installation required.</p> <p>Now for the difficult side:</p> <p>Use SQL and SPARQL; use stored procedures whenever there is server side business logic. For some time critical web pages, use VSP. Do not use VSPX. Otherwise, use whatever you are used to — <a href="http://dbpedia.org/resource/PHP" id="link-id0x20a13c00">PHP</a> or Java or anything else. For web services, simple is best. Stick to basics. "The engineer is one who can invent a simple thing." Use SQL statements rather than admin UI.</p> <p>Know that you can start a server with no database file and you get an initial database with nothing extra. The demo database, the way it is produced by installers is cluttered.</p> <p>We should put this into a couple of use case oriented how-tos.</p> <p>Also, we should create a network of "friendly local virtuoso geeks" for providing basic training and services so we do not have to explain these things all the time. To all you data-web-ers out there — please sign up and we will provide instructions, etc. Contact Yrjänä Rankka (ghard[at-sign]openlinksw.com), or go through the mailing lists; do not contact me directly.</p> <h3>"OK, we understand that you may be good at the large end of the spectrum but how do you reconcile this with the lightweight or embedded end, like the semantic desktop?"</h3> <p>Now, what is good for one end is usually good for the other. Namely, a database, no matter the scale, needs to have space efficient storage, fast index lookup, and correct query plans. Then there are things that occur only at the high-end, like clustering, but these are separate things. For embedding, the initial memory footprint needs to be small. With Virtuoso, this is accomplished by leaving out some 200 built-in tables and 100,000 lines of SQL procedures that are normally in by default, supporting things such as DAV and diverse other protocols. After all, if SPARQL is all one wants these are not needed.</p> <p>If one really wants to do one's server logic (like web listener and thread dispatching) oneself, this is not impossible but requires some advice from us. On the other hand, if one wants to have logic for security close to the data, then using stored procedures is recommended; these execute right next to the data, and support inline SPARQL and SQL. Depending on the license status of the other code, some special licensing arrangements may apply.</p> <p>We are talking about such things with different parties at present.</p> <h3>"How webby are you? What is webby?"</h3> <p>"Webby means distributed, heterogeneous, open; not monolithic consolidation of everything."</p> <p>We are philosophically webby. We come from open standards; we are after all called OpenLink; our history consists of connecting things. We believe in choice — the user should be able to pick the best of breed for components and have them work together. We cannot and do not wish to force replacement of existing assets. Transforming data on the fly and connecting systems, leaving data where it originally resides, is the first preference. For the data web, the first preference is a federation of independent SPARQL end points. When there is harvesting, we prefer to do it on demand, as with our Sponger. With the immense amount of data out there we believe in finding what is relevant <i>when</i> it is relevant, preferably close at hand, leveraging things like social networks. With a data web, many things which are now siloized, such as marketplaces and social networks, will return to the open.</p> <p>Google-style crawling of everything becomes less practical if one needs to run complex <i>ad hoc</i> queries against the mass of data. For these types of scenarios, if one needs to warehouse, the data cloud will offer solutions where one pays for database on demand. While we believe in loosely coupled federation where possible, we have serious work on the scalability side for the data center and the compute-on-demand cloud.</p> <h3>"How does OpenLink see the next five years unfolding?"</h3> <p>Personally, I think we have the basics for the birth of a new inflection in the <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x1fb9ae58">knowledge</a> economy. The <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x1f07c648">URI</a> is the unit of exchange; its value and competitive edge lie in the data it links you with. A name without context is worth little, but as a name gets more use, more <a href="http://dbpedia.org/resource/Information" id="link-id0x1f007d60">information</a> can be found through that name. This is anything from financial statistics, to legal precedents, to news reporting or government data. Right now, if the SEC just added one line of markup to the XBRL template, this would instantaneously make all SEC-mandated reporting into linked data via GRDDL.</p> <p>The URI is a carrier of brand. An information brand gets traffic and references, and this can be monetized in diverse ways. The key word is <i>context</i>. Information overload is here to stay, and only better context offers the needed increase in productivity to stay ahead of the flood.</p> <p>Semantic technologies on the whole can help with this. Why these should be semantic web or data web technologies as opposed to just semantic is the linked data value proposition. Even smart islands are still islands. Agility, scale, and scope, depend on the possibility of combining things. Therefore common terminologies and dereferenceability and discoverability are important. Without these, we are at best dealing with closed systems even if they were smart. The expert systems of the 1980s are a case in point.</p> <p>Ever since the .com era, the <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id0x2048e670">URL</a> has been a brand. Now it becomes a URI. Thus, entirely hiding the URI from the user experience is not always desirable. The URI is a sort of handle on the provenance and where more can be found; besides, people are already used to these.</p> <p>With linked data, information value-add products become easy to build and deploy. They can be basically just canned SPARQL queries combining data in a useful and insightful manner. And where there is traffic there can be monetization, whether by advertizing, subscription, or other means. Such possibilities are a natural adjunct to the blogosphere. To publish analysis, one no longer needs to be a think tank or media company. We could call this scenario the birth of a meshup economy.</p> <p>For OpenLink itself, this is our roadmap. The immediate future is about getting our high end offerings like clustered RDF storage generally available, both on the cloud and for private data centers. Ourselves, we will offer the whole <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1c696170">Linked Open Data</a> cloud as a database. The single feature to come in version 2 of this is fully automatic partitioning and repartitioning for on-demand scale; now, you have to choose how many partitions you have.</p> <p>This makes some things possible that were hard thus far.</p> <p>On the mapping front, we go for real-scale data integration scenarios where we can show that SPARQL can unify terms and concepts across databases, yet bring no added cost for complex queries. Enterprises can use their existing warehouses and have an added level of abstraction, the possibility of cross systems interlinking, the advantages of using the same taxonomies and ontologies across systems, and so forth.</p> <p>Then there will be developments in the direction of smarter web harvesting on demand with the Virtuoso <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id0x206ab780">Sponger</a>, and federation of heterogeneous SPARQL end points. The federation is not so unlike clustering, except the time scales are 2 orders of magnitude longer. The work on SPARQL end point statistics and data set description and discovery is a good development in the community.</p> <p>Then there will be NLP integration, as exemplified by the Open Calais linked data wrapper and more.</p> <p>Can we pull this off or is this being spread too thin? We know from experience that all this can be accomplished. Scale is already here; we show it with the billion triples set. Mapping is here; we showed it last in the Berlin Benchmark. We will also show some TPC-H results after we get a little quiet after the ISWC event. Then there is ongoing maintenance but with this we have shown a steady turnaround and quick time to fix for pretty much anything.</p>
2008-10-27T12:07:58-04:00
Virtuoso - Are We Too Clever for Our Own Good? (updated)
http://www.openlinksw.com:443/weblog/oerling/?date=2008-10-26#1465
2008-10-26T12:15:35Z
<p>"Physician, heal thyself," it is said. We profess to say what the messaging of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1fa3da18">semantic web</a> ought to be, but is our own perfect?</p> <p>I will here engage in some critical introspection as well as amplify on some answers given to <a href="http://virtuoso.openlinksw.com" id="link-id0x1e1eecf0">Virtuoso</a>-related questions in recent times.</p> <p>I use some conversations from the <a href="http://dbpedia.org/resource/Vienna" id="link-id0x1ec0b2e0">Vienna</a> <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x2045ac10">Linked Data</a> Practitioners meeting as a starting point. These views are mine and are limited to the Virtuoso server. These do not apply to the <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id0x2045ac38">ODS</a> (<a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id0x14f63c58">OpenLink Data Spaces</a>) applications line, <a href="http://oat.openlinksw.com/" id="link-id0x14f63c80">OAT</a> (<a href="http://oat.openlinksw.com/" id="link-id0x1e536928">OpenLink Ajax Toolkit</a>), or <a href="http://ode.openlinksw.com/" id="link-id0x1eaed7f8">ODE</a> (<a href="http://ode.openlinksw.com/" id="link-id0x1edfff88">OpenLink Data Explorer</a>).</p> <h3>"It is not always clear what the main thrust is, we get the impression that you are spread too thin," said <a href="http://www.informatik.uni-leipzig.de/~auer/foaf.rdf#me" id="link-id0x1b8a9580">Sören Auer</a>.</h3> <p>Well, personally, I am all for core competence. This is why I do not participate in all the online conversations and groups as much as I could, for example. Time and energy are critical resources and must be invested where they make a difference. In this case, the real core competence is running in the database race. This in itself, come to think of it, is a pretty broad concept.</p> <p>This is why we put a lot of emphasis on Linked Data and the <a href="http://dbpedia.org/resource/Data" id="link-id0x1b85fa38">Data</a> Web for now, as this is the emerging game. This is a deliberate choice, not an outside imperative or built-in limitation. More specifically, this means exposing any pre-existing relational data as linked data plus being the definitive <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1f5b4468">RDF</a> store.</p> <p>We can do this because we own our database and <a href="http://dbpedia.org/resource/SQL" id="link-id0x20076468">SQL</a> and data access middleware and have a history of connecting to any <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1ffd6f98">RDBMS</a> out there.</p> <p>The principal message we have been hearing from the RDF field is the call for scale of triple storage. This is even louder than the call for relational mapping. We believe that in time mapping will exceed triple storage as such, once we get some real production strength mappings deployed, enough to outperform RDF warehousing.</p> <p>There are also RDF middleware things like RDF-ization and demand-driven web harvesting (i.e, the so-called Sponger). These are <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1316f720">SPARQL</a> options, thus accessed via standard interfaces. We have little desire to create our own languages or APIs, or to tell people how to program. This is why we recently introduced <a href="http://sourceforge.net/projects/sesame/" id="link-id0x20756a68">Sesame</a>- and <a href="http://jena.sourceforge.net/" id="link-id0x1ec01ac0">Jena</a>-compatible APIs to our RDF store. From what we hear, these work. On the other hand, we do not hesitate to move beyond the standards when there is obvious value or necessity. This is why we brought SPARQL up to and beyond SQL expressivity. It is not a case of E3 (Embrace, Extend, Extinguish).</p> <p>Now, this message could be better reflected in our material on the web. This <a href="http://dbpedia.org/resource/Blog" id="link-id0x2027b410">blog</a> is a rather informal step in this direction; more is to come. For now we concentrate on delivering.</p> <p>The conventional communications wisdom is to split the message by target audience. For this, we should split the RDF, relational, and web services messages from each other. We believe that a challenger, like the semantic web technology stack, must have a compelling message to tell for it to be interesting. This is not a question of research prototypes. The new technology cannot lack something the installed technology takes for granted.</p> <p>This is why we do not tend to show things like how to insert and query a few triples: No business out there will insert and query triples for the sake of triples. There must be a more compelling story — for example, turning the whole world into a database. This is why our examples start with things like turning the <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x2051ff98">TPC-H</a> database into RDF, queries and all. Anything less is not interesting. Why would an enterprise that has business intelligence and integration issues way more complex than the rather stereotypical TPC-H even look at a technology that pretends to be all for integration and all for expressivity of queries, yet cannot answer the first question of the entry exam?</p> <p>The world out there is complex. But maybe we ought to make some simple tutorials? So, as a call to the people out there, tell us what a good tutorial would be. The question is more about figuring out what is out there and adapting these and making a sort of compatibility list. Jena and Sesame stuff ought to run as is. We could offer a webinar to all the data web luminaries showing how to promote the data web message with Virtuoso. After all, why not show it on the best platform?</p> <h3>"You are arrogant. When I read your papers or documentation, the impression I get is that you say you are smart and the reader is stupid."</h3> <p>We should answer in multiple parts.</p> <p>For general collateral, like web sites and documentation:</p> <p>The web site gives a confused product image. For the Virtuoso product, we should divide at the top into</p> <ul> <li> Data web and RDF - Host linked data, expose relational assets as linked data;</li> <li> Relational Database - Full function, high performance, open source, Federated/Virtual Relational DBMS, expose heterogeneous RDB assets through one point of contact for integration;</li> <li> Web Services - access all the above over standard protocols, dynamic web pages, web hosting.</li> </ul> <p>For each point, one simple statement. We all know what the above things mean?</p> <p>Then we add a new point about scalability that impacts all the above, namely the Virtuoso version 6 Cluster, meaning that you can do all these things at 10 to 1000 times the scale. This means this much more data or in some cases this much more requests per second. This too is clear.</p> <p>Far as I am concerned, hosting Java or .<a href="http://dbpedia.org/resource/.NET_Framework" id="link-id0x1f297540">NET</a> does not have to be on the front page. Also, we have no great interest in going against <a href="http://dbpedia.org/resource/Apache" id="link-id0x1ea29578">Apache</a> when it comes to a web server only situation. The fact that we have a web listener is important for some things but our claim to fame does not rest on this.</p> <p>Then for documentation and training materials: The documentation should be better. Specifically it should have more of a how-to dimension since nobody reads the whole thing anyhow. About online tutorials, the order of presentation should be different. They do not really reflect what is important at the present moment either.</p> <p>Now for conference papers: Since taking the data web as a focus area, we have submitted some papers and had some rejected because these do not have enough references and do not explain what is obvious to ourselves.</p> <p>I think that the communications failure in this case is that we want to talk about end to end solutions and the reviewers expect research. For us, the solution is interesting and exists only if there is an adequate functionality mix for addressing a specific use case. This is why we do not make a paper about query cost model alone because the cost model, while indispensable, is a thing that is taken for granted where we come from. So we mention RDF adaptations to cost model, as these are important to the whole but do not find these to be the justification for a whole paper. If we made papers on this basis, we would have to make five times as many. Maybe we ought to.</p> <h3>"Virtuoso is very big and very difficult"</h3> <p>One thing that is not obvious from the Virtuoso packaging is that the minimum installation is an executable under 10MB and a config file. Two files.</p> <p>This gives you SQL and SPARQL out of the box. Adding <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x20a2e7d0">ODBC</a> and <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x1e4cceb8">JDBC</a> clients is as simple as it gets. After this, there is basic database functionality. Tuning is a matter of a few parameters that are explained on this blog and elsewhere. Also, the full scale installation is available as an Amazon EC2 image, so no installation required.</p> <p>Now for the difficult side:</p> <p>Use SQL and SPARQL; use stored procedures whenever there is server side business logic. For some time critical web pages, use VSP. Do not use VSPX. Otherwise, use whatever you are used to — <a href="http://dbpedia.org/resource/PHP" id="link-id0x20b03f08">PHP</a> or Java or anything else. For web services, simple is best. Stick to basics. "The engineer is one who can invent a simple thing." Use SQL statements rather than admin UI.</p> <p>Know that you can start a server with no database file and you get an initial database with nothing extra. The demo database, the way it is produced by installers is cluttered.</p> <p>We should put this into a couple of use case oriented how-tos.</p> <p>Also, we should create a network of "friendly local virtuoso geeks" for providing basic training and services so we do not have to explain these things all the time. To all you data-web-ers out there — please sign up and we will provide instructions, etc. Contact Yrjänä Rankka (ghard[at-sign]openlinksw.com), or go through the mailing lists; do not contact me directly.</p> <h3>"OK, we understand that you may be good at the large end of the spectrum but how do you reconcile this with the lightweight or embedded end, like the semantic desktop?"</h3> <p>Now, what is good for one end is usually good for the other. Namely, a database, no matter the scale, needs to have space efficient storage, fast index lookup, and correct query plans. Then there are things that occur only at the high-end, like clustering, but these are separate things. For embedding, the initial memory footprint needs to be small. With Virtuoso, this is accomplished by leaving out some 200 built-in tables and 100,000 lines of SQL procedures that are normally in by default, supporting things such as DAV and diverse other protocols. After all, if SPARQL is all one wants these are not needed.</p> <p>If one really wants to do one's server logic (like web listener and thread dispatching) oneself, this is not impossible but requires some advice from us. On the other hand, if one wants to have logic for security close to the data, then using stored procedures is recommended; these execute right next to the data, and support inline SPARQL and SQL. Depending on the license status of the other code, some special licensing arrangements may apply.</p> <p>We are talking about such things with different parties at present.</p> <h3>"How webby are you? What is webby?"</h3> <p>"Webby means distributed, heterogeneous, open; not monolithic consolidation of everything."</p> <p>We are philosophically webby. We come from open standards; we are after all called OpenLink; our history consists of connecting things. We believe in choice — the user should be able to pick the best of breed for components and have them work together. We cannot and do not wish to force replacement of existing assets. Transforming data on the fly and connecting systems, leaving data where it originally resides, is the first preference. For the data web, the first preference is a federation of independent SPARQL end points. When there is harvesting, we prefer to do it on demand, as with our Sponger. With the immense amount of data out there we believe in finding what is relevant <i>when</i> it is relevant, preferably close at hand, leveraging things like social networks. With a data web, many things which are now siloized, such as marketplaces and social networks, will return to the open.</p> <p>Google-style crawling of everything becomes less practical if one needs to run complex <i>ad hoc</i> queries against the mass of data. For these types of scenarios, if one needs to warehouse, the data cloud will offer solutions where one pays for database on demand. While we believe in loosely coupled federation where possible, we have serious work on the scalability side for the data center and the compute-on-demand cloud.</p> <h3>"How does OpenLink see the next five years unfolding?"</h3> <p>Personally, I think we have the basics for the birth of a new inflection in the <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x2018bd98">knowledge</a> economy. The <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x1ec110d8">URI</a> is the unit of exchange; its value and competitive edge lie in the data it links you with. A name without context is worth little, but as a name gets more use, more <a href="http://dbpedia.org/resource/Information" id="link-id0x1ecfba08">information</a> can be found through that name. This is anything from financial statistics, to legal precedents, to news reporting or government data. Right now, if the SEC just added one line of markup to the XBRL template, this would instantaneously make all SEC-mandated reporting into linked data via GRDDL.</p> <p>The URI is a carrier of brand. An information brand gets traffic and references, and this can be monetized in diverse ways. The key word is <i>context</i>. Information overload is here to stay, and only better context offers the needed increase in productivity to stay ahead of the flood.</p> <p>Semantic technologies on the whole can help with this. Why these should be semantic web or data web technologies as opposed to just semantic is the linked data value proposition. Even smart islands are still islands. Agility, scale, and scope, depend on the possibility of combining things. Therefore common terminologies and dereferenceability and discoverability are important. Without these, we are at best dealing with closed systems even if they were smart. The expert systems of the 1980s are a case in point.</p> <p>Ever since the .com era, the <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id0x1c4c9248">URL</a> has been a brand. Now it becomes a URI. Thus, entirely hiding the URI from the user experience is not always desirable. The URI is a sort of handle on the provenance and where more can be found; besides, people are already used to these.</p> <p>With linked data, information value-add products become easy to build and deploy. They can be basically just canned SPARQL queries combining data in a useful and insightful manner. And where there is traffic there can be monetization, whether by advertizing, subscription, or other means. Such possibilities are a natural adjunct to the blogosphere. To publish analysis, one no longer needs to be a think tank or media company. We could call this scenario the birth of a meshup economy.</p> <p>For OpenLink itself, this is our roadmap. The immediate future is about getting our high end offerings like clustered RDF storage generally available, both on the cloud and for private data centers. Ourselves, we will offer the whole <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x20791bf0">Linked Open Data</a> cloud as a database. The single feature to come in version 2 of this is fully automatic partitioning and repartitioning for on-demand scale; now, you have to choose how many partitions you have.</p> <p>This makes some things possible that were hard thus far.</p> <p>On the mapping front, we go for real-scale data integration scenarios where we can show that SPARQL can unify terms and concepts across databases, yet bring no added cost for complex queries. Enterprises can use their existing warehouses and have an added level of abstraction, the possibility of cross systems interlinking, the advantages of using the same taxonomies and ontologies across systems, and so forth.</p> <p>Then there will be developments in the direction of smarter web harvesting on demand with the Virtuoso <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id0x1f27e6d8">Sponger</a>, and federation of heterogeneous SPARQL end points. The federation is not so unlike clustering, except the time scales are 2 orders of magnitude longer. The work on SPARQL end point statistics and data set description and discovery is a good development in the community.</p> <p>Then there will be NLP integration, as exemplified by the Open Calais linked data wrapper and more.</p> <p>Can we pull this off or is this being spread too thin? We know from experience that all this can be accomplished. Scale is already here; we show it with the billion triples set. Mapping is here; we showed it last in the Berlin Benchmark. We will also show some TPC-H results after we get a little quiet after the ISWC event. Then there is ongoing maintenance but with this we have shown a steady turnaround and quick time to fix for pretty much anything.</p>
2008-10-27T12:07:52-04:00
State of the Semantic Web, Part 2 - The Technical Questions (updated)
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-10-26#1466
2008-10-26T12:02:43Z
<p>Here I will talk about some more technical questions that came up. This is mostly general; <a href="http://virtuoso.openlinksw.com" id="link-id0x205901a0">Virtuoso</a> specific questions and answers are separate. </p> <h3>"How to Bootstrap? Where will the triples come from?"</h3> <p>There are already wrappers producing <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x13519ac8">RDF</a> from many applications. Since any structured or semi-structured <a href="http://dbpedia.org/resource/Data" id="link-id0x1c93b418">data</a> can be converted to RDF and often there is even a pre-existing terminology for the application domain, the availability of the data <i>per se</i> is not the concern.</p> <p>The triples may come from any application or database, but they will not come from the end user directly. There was a good talk about photograph annotation in <a href="http://dbpedia.org/resource/Vienna" id="link-id0x1ea9d150">Vienna</a>, describing many ways of deriving metadata for photos. The essential wisdom is annotating on the spot and wherever possible doing so automatically. The consumer is very unlikely to go annotate photos after the fact. Further, one can infer that photos made with the same camera around the same time are from the same location. There are other such heuristics. In this use case, the end user does not need to see triples. There is some benefit though in using commonly used geographical terminology for linking to other data sources.</p> <h3>"How will one develop applications?"</h3> <p>I'd say one will develop them much the same way as thus far. In <a href="http://dbpedia.org/resource/PHP" id="link-id0x207fca00">PHP</a>, for example. Whether one's query language is <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x20a5fde0">SPARQL</a> or <a href="http://dbpedia.org/resource/SQL" id="link-id0x1a0bb5e0">SQL</a> does not make a large difference in how basic web UI is made.</p> <p>A SPARQL end-point is no more an end-user item than a SQL command-line is.</p> <p>A common mistake among techies is that they think the data structure and user experience can or ought to be of the same structure. The UI dialogs do not, for example, have to have a 1:1 correspondence with SQL tables.</p> <p>The idea of generating UI from data, whether relational or data-web, is so seductive that generation upon generation of developers fall for it, repeatedly. Even I, at OpenLink, after supposedly having been around the block a couple of times made some experiments around the topic. What does make sense is putting a thin wrapper or HTML around the application, using XSLT and such for formatting. Since the model does allow for unforeseen properties of data, one can build a viewer for these alongside the regular forms. For this, Ajax technologies like <a href="http://oat.openlinksw.com/" id="link-id0x1e91d118">OAT</a> (the <a href="http://oat.openlinksw.com/" id="link-id0x174b7950">OpenLink AJAX Toolkit</a>) will be good.</p> <p>The UI ought not to completely hide the URIs of the data from the user. It should offer a drill down to faceted views of the triples for example. Remember when Xerox talked about graphical user interfaces in 1980? "Don't mode me in" was the slogan, as I recall.</p> <p>Since then, we have vacillated between modal and non-modal interaction models. Repetitive workflows like order entry go best modally and are anyway being replaced by web services. Also workflows that are very infrequent benefit from modality; take personal network setup wizards, for example. But enabling the <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x1ea14610">knowledge</a> worker is a domain that by its nature must retain some respect for human intelligence and not kill this by denying access to the underlying data, including provenance and URIs. Face it: the world is not getting simpler. It is increasingly data dependent and when this is so, having semantics and flexibility of access for the data is important.</p> <p>For a real-time task-oriented user interface like a fighter plane cockpit, one will not show URIs unless specifically requested. For planning fighter sorties though, there is some potential benefit in having all data such as friendly and hostile assets, geography, organizational structure, etc., as <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x207bcd20">linked data</a>. It makes for more flexible querying. Linked data does not <i>per se</i> mean open, so one can be joinable with open data through using the same identifiers even while maintaining arbitrary levels of security and compartmentalization.</p> <p>For automating tasks that every time involve the same data and queries, RDF has no intrinsic superiority. Thus the user interfaces in places where RDF will have real edge must be more capable of <i>ad hoc</i> viewing and navigation than regular real-time or line of business user interfaces.</p> <p>The <a href="http://ode.openlinksw.com/" id="link-id0x2083a6f0">OpenLink Data Explorer</a> idea of a "data behind the web page" view goes in this direction. Read the web as before, then hit a switch to go to the data view. There are and will be separate clarifications and demos about this.</p> <h3>"What of the proliferation of standards? Does this not look too tangled, no clear identity? How would one know where to begin?"</h3> <p>When <a href="http://www.w3.org/2001/sw/sweo/" id="link-id0x1e8eac68">SWEO</a> was beginning, there was an endlessly protracted discussion of the so-called layer cake. This acronym jungle is not good messaging. Just say linked, flexibly repurpose-able data, and rich vocabularies and structure. Just the right amount of structure for the application, less rigid and easier to change than relational.</p> <p>Do not even mention the different serialization formats. Just say that it fits on top of the accepted web infrastructure — <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1e3806b8">HTTP</a>, URIs, and <a href="http://dbpedia.org/resource/XML" id="link-id0x1f547288">XML</a> where desired.</p> <p>It is misleading to say inference is a box at some specific place in the diagram. Inference of different types may or may not take place at diverse points, whether presentation or storage, on demand or as a preprocessing step. Since there is structure and semantics, inference is possible if desired.</p> <h3>"Can I make a social network application in RDF only, with no <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x20553ee0">RDBMS</a>?"</h3> <p>Yes, in principle, but what do you have in mind? The answer is very context dependent. The person posing the question had an E-learning system in mind, with things such as course catalogues, course material, etc. In such a case, RDF is a great match, especially since the user count will not be in the millions. No university has that many students and anyway they do not hang online browsing the course catalogue.</p> <p>On the other hand, if I think of making a social network site with RDF as the exclusive data model, I see things that would be very inefficient. For example, keeping a count of logins or the last time of login would be by default several times less efficient than with a RDBMS.</p> <p>If some application is really large scale and has a knowable workload profile, like any social network does, then some task-specific data structure is simply economical. This does not mean that the application language cannot be SPARQL but this means that the storage format must be tuned to favor some operations over others, relational style. This is a matter of cost more than of feasibility. Ten servers cost less than a hundred and have failures ten times less frequently.</p> <p>In the near term we will see the birth of an application paradigm for the data web. The data will be open, exposed, first-class citizen; yet the user experience will not have to be in a 1:1 image of the data.</p>
2008-10-27T11:28:14-04:00
State of the Semantic Web, Part 2 - The Technical Questions (updated)
http://www.openlinksw.com:443/weblog/oerling/?date=2008-10-26#1464
2008-10-26T12:02:43Z
<p>Here I will talk about some more technical questions that came up. This is mostly general; <a href="http://virtuoso.openlinksw.com" id="link-id0x1f53d1a0">Virtuoso</a> specific questions and answers are separate. </p> <h3>"How to Bootstrap? Where will the triples come from?"</h3> <p>There are already wrappers producing <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1beda278">RDF</a> from many applications. Since any structured or semi-structured <a href="http://dbpedia.org/resource/Data" id="link-id0x1e57c648">data</a> can be converted to RDF and often there is even a pre-existing terminology for the application domain, the availability of the data <i>per se</i> is not the concern.</p> <p>The triples may come from any application or database, but they will not come from the end user directly. There was a good talk about photograph annotation in <a href="http://dbpedia.org/resource/Vienna" id="link-id0x2028b7e8">Vienna</a>, describing many ways of deriving metadata for photos. The essential wisdom is annotating on the spot and wherever possible doing so automatically. The consumer is very unlikely to go annotate photos after the fact. Further, one can infer that photos made with the same camera around the same time are from the same location. There are other such heuristics. In this use case, the end user does not need to see triples. There is some benefit though in using commonly used geographical terminology for linking to other data sources.</p> <h3>"How will one develop applications?"</h3> <p>I'd say one will develop them much the same way as thus far. In <a href="http://dbpedia.org/resource/PHP" id="link-id0x1eff1748">PHP</a>, for example. Whether one's query language is <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1d83dff8">SPARQL</a> or <a href="http://dbpedia.org/resource/SQL" id="link-id0x1e9f4e88">SQL</a> does not make a large difference in how basic web UI is made.</p> <p>A SPARQL end-point is no more an end-user item than a SQL command-line is.</p> <p>A common mistake among techies is that they think the data structure and user experience can or ought to be of the same structure. The UI dialogs do not, for example, have to have a 1:1 correspondence with SQL tables.</p> <p>The idea of generating UI from data, whether relational or data-web, is so seductive that generation upon generation of developers fall for it, repeatedly. Even I, at OpenLink, after supposedly having been around the block a couple of times made some experiments around the topic. What does make sense is putting a thin wrapper or HTML around the application, using XSLT and such for formatting. Since the model does allow for unforeseen properties of data, one can build a viewer for these alongside the regular forms. For this, Ajax technologies like <a href="http://oat.openlinksw.com/" id="link-id0x1d780520">OAT</a> (the <a href="http://oat.openlinksw.com/" id="link-id0x20943788">OpenLink AJAX Toolkit</a>) will be good.</p> <p>The UI ought not to completely hide the URIs of the data from the user. It should offer a drill down to faceted views of the triples for example. Remember when Xerox talked about graphical user interfaces in 1980? "Don't mode me in" was the slogan, as I recall.</p> <p>Since then, we have vacillated between modal and non-modal interaction models. Repetitive workflows like order entry go best modally and are anyway being replaced by web services. Also workflows that are very infrequent benefit from modality; take personal network setup wizards, for example. But enabling the <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x1e14eb88">knowledge</a> worker is a domain that by its nature must retain some respect for human intelligence and not kill this by denying access to the underlying data, including provenance and URIs. Face it: the world is not getting simpler. It is increasingly data dependent and when this is so, having semantics and flexibility of access for the data is important.</p> <p>For a real-time task-oriented user interface like a fighter plane cockpit, one will not show URIs unless specifically requested. For planning fighter sorties though, there is some potential benefit in having all data such as friendly and hostile assets, geography, organizational structure, etc., as <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1e91d118">linked data</a>. It makes for more flexible querying. Linked data does not <i>per se</i> mean open, so one can be joinable with open data through using the same identifiers even while maintaining arbitrary levels of security and compartmentalization.</p> <p>For automating tasks that every time involve the same data and queries, RDF has no intrinsic superiority. Thus the user interfaces in places where RDF will have real edge must be more capable of <i>ad hoc</i> viewing and navigation than regular real-time or line of business user interfaces.</p> <p>The <a href="http://ode.openlinksw.com/" id="link-id0x1c7f8ee0">OpenLink Data Explorer</a> idea of a "data behind the web page" view goes in this direction. Read the web as before, then hit a switch to go to the data view. There are and will be separate clarifications and demos about this.</p> <h3>"What of the proliferation of standards? Does this not look too tangled, no clear identity? How would one know where to begin?"</h3> <p>When <a href="http://www.w3.org/2001/sw/sweo/" id="link-id0x1d73c268">SWEO</a> was beginning, there was an endlessly protracted discussion of the so-called layer cake. This acronym jungle is not good messaging. Just say linked, flexibly repurpose-able data, and rich vocabularies and structure. Just the right amount of structure for the application, less rigid and easier to change than relational.</p> <p>Do not even mention the different serialization formats. Just say that it fits on top of the accepted web infrastructure — <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1efefed0">HTTP</a>, URIs, and <a href="http://dbpedia.org/resource/XML" id="link-id0x1af89b18">XML</a> where desired.</p> <p>It is misleading to say inference is a box at some specific place in the diagram. Inference of different types may or may not take place at diverse points, whether presentation or storage, on demand or as a preprocessing step. Since there is structure and semantics, inference is possible if desired.</p> <h3>"Can I make a social network application in RDF only, with no <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1cb62cd8">RDBMS</a>?"</h3> <p>Yes, in principle, but what do you have in mind? The answer is very context dependent. The person posing the question had an E-learning system in mind, with things such as course catalogues, course material, etc. In such a case, RDF is a great match, especially since the user count will not be in the millions. No university has that many students and anyway they do not hang online browsing the course catalogue.</p> <p>On the other hand, if I think of making a social network site with RDF as the exclusive data model, I see things that would be very inefficient. For example, keeping a count of logins or the last time of login would be by default several times less efficient than with a RDBMS.</p> <p>If some application is really large scale and has a knowable workload profile, like any social network does, then some task-specific data structure is simply economical. This does not mean that the application language cannot be SPARQL but this means that the storage format must be tuned to favor some operations over others, relational style. This is a matter of cost more than of feasibility. Ten servers cost less than a hundred and have failures ten times less frequently.</p> <p>In the near term we will see the birth of an application paradigm for the data web. The data will be open, exposed, first-class citizen; yet the user experience will not have to be in a 1:1 image of the data.</p>
2008-10-27T11:28:01-04:00
Dog-fooding: Linked Data and OpenLink Product Portfolio
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-10-24#1463
2008-10-24T22:05:42Z
<p>Thanks to RDF and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1cf5c700">Linked Data</a>, it's becoming a lot easier for us to explain and reveal the depth of the OpenLink technology portfolio.</p> <p>Here is a look at our offerings by product family:</p> <ul> <li> <a href="http://data.openlinksw.com/about/html/http://data.openlinksw.com/oplweb/product_family/uda" id="link-id1161c6d0">Universal Data Access Drivers</a> </li> <li> <a href="http://data.openlinksw.com/about/html/http://data.openlinksw.com/oplweb/product_family/virtuoso" id="link-id17945fc8">Virtuoso</a> </li> <li> <a href="http://data.openlinksw.com/about/html/http://data.openlinksw.com/oplweb/product_family/dca" id="link-id10f899c0">Distributed Collaborative Applications</a> (DCA)</li> <li> <a href="http://data.openlinksw.com/oplweb/product_family/development" id="link-id1c55ac70">Developer Kits & Frameworks</a> </li> <li> <a href="http://data.openlinksw.com/oplweb/product_family/utilities" id="link-id1a735e50">Benchamark & Diagnostic Utilities</a> </li> </ul> <p>As you explore the <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id10fc4af8">Linked Data graph</a> exposed via our product portfolio, I expect you to experience, or at least spot, the virtuous potential of high SDQ (<a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1442" id="link-id13847698">Serendipitous Discovery Quotient</a>) courtesy of Linked Data, which is <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> 3.0's answer to <a href="http://dbpedia.org/resource/Search_engine_optimization" id="link-id115ad4f0">SEO</a>. For instance, how <a href="http://linkeddata.uriburner.com/about/html/http://data.openlinksw.com/oplweb/dbms_family/Oracle" id="link-id1cda63c8">Database</a>, <a href="http://data.openlinksw.com/oplweb/opsys_family/Windows" id="link-id1a803f18">Operating System</a>, and <a href="http://data.openlinksw.com/oplweb/processor/universal_1" id="link-id19cbaba0">Processor</a> family paths in the product portfolio graph (data network) unveil a lot more about <a href="http://linkeddata.uriburner.com/about/html/http://www.openlinksw.com/dataspace/organization/openlink%23this" id="link-ide9b7070">OpenLink Software</a> than meets the proverbial "eye" :-)</p>
2008-10-24T18:13:50-04:00
Virtuoso, PHP Runtime Hosting: phpBB, Wordpress, Drupal, MediaWiki, and Linked Data
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-10-24#1461
2008-10-24T19:55:00Z
<p> Runtime hosting is functionality realm of <a href="http://virtuoso.openlinksw.com" id="link-id1189fee8">Virtuoso</a> that is sometimes easily overlooked. In this post I want to provide a simple no-hassles HOWTO guide for installing Virtuoso on Windows (32 or 64 Bit), Mac OS X (Universal or Native 64 Bit), and Linux (32 or 64 Bit). The installation guide also covers the instantiation of <a href="http://dbpedia.org/resource/PhpBB" id="link-id118af3a8">phpBB3</a> as verification of the Virtuoso hosted <a href="http://dbpedia.org/resource/PHP" id="link-id12736b88">PHP</a> 3.5 runtime.</p> <h3> What are the benefits of PHP Runtime Hosting?</h3> <p> Like <a href="http://dbpedia.org/resource/Apache" id="link-id111ca408">Apache</a>, Virtuoso is a bona-fide <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> <a href="http://dbpedia.org/resource/Application_server" id="link-id110d2aa8">Application Server</a> for PHP based applications. Unlike Apache, Virtuoso is also the following:</p> <ul> <li> a Hybrid Native DBMS Engine (Relational, RDF-Graph, and Document models) that is accessible via industry standard interfaces (solely)</li> <li> a Virtual DBMS or Master <a href="http://dbpedia.org/resource/Data">Data</a> Manager (MDM) that virtualizes heterogeneous data sources (<a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x22b6f0c8">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x23af98c8">JDBC</a>, Web Services, Hypermedia Resources, Non Hypermedia Resources)</li> <li> an <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=rdf%20middleware&type=text&output=html" id="link-id1116aad8">RDF Middleware</a> solution for RDF-zation of non RDF resources across the Web and enterprise Intranets and/or Extranets (in the form of Cartridges for data exposed via REST or SOA oriented SOAP interfaces)</li> <li> an RDF <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10fbe088">Linked Data</a> Server (meaning it can deploy RDF Linked Data based on its native and/or virtualized data)</li> </ul> <p> As result of the above, when you deploy a PHP application using Virtuoso, you inherit the following benefits:</p> <ol> <li> Use of PHP-<a href="http://www.iodbc.org" id="link-id1159e070">iODBC</a> for in-process communication with Virtuoso</li> <li> Easy generation of RDF Linked Data Views atop the <a href="http://dbpedia.org/resource/SQL" id="link-id0x24f44c98">SQL</a> schemas of PHP applications</li> <li> Easy deployment of RDF Linked Data from virtualized data sources</li> <li> Less <a href="http://dbpedia.org/resource/LAMP_stack" id="link-id1179dff0">LAMP</a> monoculture (*there is no such thing as virtuous monoculture*) when dealing with PHP based Web applications.</li> </ol> <p> As indicated in prior posts, producing RDF Linked Data from the existing Web, where a lot of content is deployed by PHP based content managers, should simply come down to RDF Views over the SQL Schemas and deployment / publishing of the RDF Views in RDF Linked data form. In a nutshell, this is what Virtuoso delivers via its PHP runtime hosting and pre packaged VADs (Virtuoso Application Distribution packages), for popular PHP based applications such as: <a href="http://blogs.usnet.private:8893/weblog/kidehen@openlinksw.com/127/dbpedia.org/resource/PhpBB" id="link-id120cc6368">phpBB3</a>, <a href="http://dbpedia.org/resource/Drupal" id="link-id111ff1c0">Drupal</a>, <a href="http://dbpedia.org/resource/WordPress" id="link-id111e26f8">WordPress</a>, and <a href="http://dbpedia.org/resource/MediaWiki" id="link-id10ea0258">MediaWiki</a>.</p> <p> In addition, to the RDF Linked Data deployment, we've also taken the traditional LAMP installation tedium out of the typical PHP application deployment process. For instance, you don't have to rebuild PHP 3.5 (32 or 64 Bit) on Windows, Mac OS X, or Linux to get going, simply install Virtuoso, and then select a VAD package for the relevant application and you're set. If the application of choice isn't pre packaged by us, simply install as you would when using Apache, which comes dow to situating the PHP files in your Web structure under the Web Application's root directory.</p> <h3> Installation Guide</h3> <ol> <li> Download the Virtuoso installer for Windows (<a href="http://download.openlinksw.com/downwiz/login.vsp?pfam=2&pform=26&pcat=47&prod=virtuoso-uim-unisvr-ent&os=i686-generic-win-32&os2=i686-generic-win-32&xpfam=virtuoso&xpform=personal&xpcat=unisvr&xos=i686-generic-win-32&release-dbms=6.1-virt61" id="link-id11d084578">32 Bit msi file</a> or <a href="http://download.openlinksw.com/downwiz/login.vsp?pfam=2&pform=26&pcat=47&prod=virtuoso-uim-unisvr-ent&os=x86_64-generic-win-64&os2=x86_64-generic-win-64&xpfam=virtuoso&xpform=personal&xpcat=unisvr&xos=x86_64-generic-win-64&release-dbms=6.1-virt61" id="link-id11aea67a8">64 Bit msi file</a>), Mac OS X (<a href="http://download.openlinksw.com/downwiz/login.vsp?pfam=2&pform=26&pcat=47&prod=virtuoso-uim-unisvr-ent&os=universal-apple-macosx10.6-32&os2=universal-apple-macosx10.6-32&xpfam=virtuoso&xpform=personal&xpcat=unisvr&xos=universal-apple-macosx10.6-32&release-dbms=6.1-virt61" id="link-id11a93bef8">Universal Binary dmg file</a>), or instantiate the <a href="http://www.openlinksw.com/oat/wiki/main/Main/ODSInstallationEC2" id="link-id111fe248">Virtuoso EC2 AMI</a> (*search for pattern: "Virtuoso when using the Firefox extension for EC2 as the AMI ID is currently: ami-7c31d515 and name: virtuoso-test/virtuoso-cloud-beta-9-i386.manifest.xml, for latest cut*)</li> <li> Run the installer (or download the movies using the links in the related section below)</li> <li> Go to the Virtuoso Conductor (*which will show up at the end of the installation process* or go to http://localhost:8890/conductor)</li> <li> Go to the "Admin" tab within the (X)HTML based UI and select the "Packages" sub-menu item (a Tab)</li> <li> Pick phpBB3 (or any other pre-packaged PHP app) and then click on "Install/Upgrase"</li> <li> The watch one of my silent movies or read the initial startup guides for Virtuoso hosted phpBB3, Drupal, Wordpress, MediaWiki.</li> </ol> <h3> Related</h3> <p> At the current time, I've only provided links to ZIP files containing the Virtuoso installation "silent movies". This approach is a short-term solution to some of my current movie publishing challenges re. YouTube and Vimeo -- where the compressed output hasn't been of acceptable visual quality. Once resolved, I will publish much more "Multimedia Web" friendly movies :-)</p> <ul> <li> <a href="http://my-movies.s3.amazonaws.com/Virtuoso_PHPBB3_Vista_Linked_Data_Demo.mov.zip" id="link-id11642450">Windows Vista (x64) Installation Movie</a> </li> <li> <a href="http://my-movies.s3.amazonaws.com/Virtuoso_PHPBB3_MacOSX_Linked_Data_Demo.mov.zip" id="link-id11210498">Mac OS X (x64 & Universal binary) Installation Movie</a> </li> <li> <a href="http://my-movies.s3.amazonaws.com/Virtuoso_PHPBB3_EC2_AMI_Linked_Data_Demo.zip" id="link-id111ff268">Virtuoso EC2 Cloud Edition Installation Movie</a> </li> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtuosoPHP" id="link-id12038b6c8">Guide for PHP based Application Deployment using Virtuoso</a> </li> </ul>
2010-03-25T21:19:59-04:00
The Virtuous Web of Linked Data -- Business Perspective (Updated)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-10-24#1462
2008-10-24T15:56:55Z
<a href="http://www.openlinksw.com/weblog/oerling" id="link-id115d8420">Orri Erling</a> (Program Manager: OpenLink <a href="http://virtuoso.openlinksw.com" id="link-id111293d8">Virtuoso</a>) has dropped a well explained reiteration of the essence of the "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id115d85a0">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id1161b138">Web</a>" or "<a href="http://dbpedia.org/resource/Data">Data</a> Web" with an emphasis on the business value. His post is titled: <a href="http://www.openlinksw.com/weblog/oerling/?id=1459" id="link-id1109d340">State of the Semantic Web (Part 1) - Sociology, Business, and Messaging</a>. <p>Typically, Orri's post are targeted at the hard core RDF and <a href="http://dbpedia.org/resource/SQL" id="link-id115e2818">SQL</a> DBMS audiences, but in this particular post, he shoots straight at the business community revealing "Opportunity Cost" containment as the invisible driver behind the business aspects of any market inflection.</p> <p>Remember, the Web isn't ubiquitous because its users mastered the mechanics and virtues of HTML and/or HTTP. Web ubiquity is a function of the opportunity cost of not being on the Web, courtesy of the network effects of hyperlinked documents -- i.e., the instant gratification of traversing documents on the Web via a single click action. In similar fashion, the Linked Data Web's ubiquity will simply come down to the opportunity cost of not being "inside the Web", courtesy of the network effects of hyperlinked entities (documents, people, music, books, and other "Things"). </p> <p>Here are some excerpts from Orri's post:</p> <blockquote> <cite>Every time there is a major shift in technology, this shift needs to be motivated by addressing a new class of problem. This means doing something that could not be done before. The last time this happened was when the relational database became the dominant IT technology. At that time, the questions involved putting the enterprise in the database and building a cluster of line of business applications around the database. The argument for the <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id113779e8">RDBMS</a> was that you did not have to constrain the set of queries that might later be made, when designing the database. In other words, it was making things more ad hoc. This was opposed then on grounds of being less efficient than the hierarchical and network databases which the relational eventually replaced.</cite> <cite>Today, the point of the Data Web is that you do not have to constrain what your data can join or integrate with, when you design your database. The counter-argument is that this is slow and geeky and not scalable. See the similarity?</cite> <cite>A difference is that we are not specifically aiming at replacing the RDBMS. In fact, if you know exactly what you will query and have a well defined workload, a relational representation optimized for the workload will give you about 10x the performance of the equivalent RDF warehouse. OLTP remains a relational-only domain. </cite> <cite>However, when we are talking about doing queries and analytics against the Web, or even against more than a handful of relational systems, the things which make RDBMS good become problematic.</cite> </blockquote> <p>If we think about Web 1.0 as a period where the distinguishing noun was: "Author", and Web 2.0 the noun: "Journalist", we should be able to see that what comes next is the noun: "Analyst". This new generation analyst would be equipped with de-referencable Web Identity courtesy of their Person <a href="http://dbpedia.org/resource/Entity" id="link-id111ab7d0">Entity</a> <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id10f23220">URI</a>. The analyst's URI would also be the critical component of Web based low cost attribution ecosystem; one that ultimately turns the URI into the analyst's brand emblem / imprint.</p> <h3>Related</h3> <ul> <a href="http://blog.whatfettle.com/about/" id="link-id1120fb88">Paul Downey</a> - <a href="http://blog.whatfettle.com/2008/10/24/on-the-vanity-of-demanding-attribution/" id="link-id111590b8">Vanity of Demanding Attribution</a> </ul>
2008-10-24T14:49:18-04:00
State of the Semantic Web, Part 1 - Sociology, Business, and Messaging (update 2)
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-10-24#1460
2008-10-24T10:19:03Z
<p>I was in <a href="http://dbpedia.org/resource/Vienna" id="link-id0x1f18a540">Vienna</a> for the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1ec788a0">Linked Data</a> Practitioners gathering this week. Danny Ayers asked me if I would <a href="http://dbpedia.org/resource/Blog" id="link-id0x20838238">blog</a> about the State of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x20694ed8">Semantic Web</a> or write the <i>This Week's Semantic Web</i> column. I don't have the time to cover all that may have happened during the past week but I will editorialize about the questions that again were raised in Vienna. How these things relate to <a href="http://virtuoso.openlinksw.com" id="link-id0x20b1cd38">Virtuoso</a> will be covered separately. This is about the overarching questions of the times, not the finer points of geek craft.</p> <p> <a href="http://www.informatik.uni-leipzig.de/~auer/foaf.rdf#me" id="link-id0x1ff31b30">Sören Auer</a> asked me to say a few things about relational to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1f8118e0">RDF</a> mapping. I will cite some highlights from this, as they pertain to the general scene. There was an "open hacking" session Wednesday night featuring lightning talks. I will use some of these too as a starting point.</p> <h3>The messaging?</h3> <p>The <a href="http://www.w3.org/2001/sw/sweo/" id="link-id0x1dc39210">SWEO</a> (Semantic Web Education and Outreach) interest group of the W3C spent some time looking for an elevator pitch for the Semantic Web. It became "<a href="http://dbpedia.org/resource/Data" id="link-id0x1f24dd98">Data</a> Unleashed." Why not? Let's give this some context.</p> <p>So, if we are holding a <i>Semantic Web 101</i> session, where should we begin? I hazard to guess that we should not begin by writing a FOAF file in Turtle by hand, as this is one thing that is not likely to happen in the real world.</p> <p>Of course, the social aspect of the Data Web is the most immediately engaging, so a demo might be to go make an account with <a href="http://myopenlink.net/" id="link-id0x1f5e0198">myopenlink</a>.<a href="http://dbpedia.org/resource/.NET_Framework" id="link-id0x1ec49a00">net</a> and see that after one has entered the data one normally enters for any social network, one has become a Data Web citizen. This means that one can be found, just like this, with a query against the set of data spaces hosted on the system. Then we just need a few pages that repurpose this data and relate it to other data. We show some samples of queries like this in our <a href="http://challenge.semanticweb.org/" id="link-id0x1ee35f70">Billion Triples Challenge</a> demo. We will make a webcast about this to make it all clearer.</p> <p>Behold: The Data Web is about the world becoming a database; writing <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x20644808">SPARQL</a> queries or triples is incidental. You will write FOAF files by hand just as little as you now write <a href="http://dbpedia.org/resource/SQL" id="link-id0x1fd9fbc0">SQL</a> insert statements for filling in your account <a href="http://dbpedia.org/resource/Information" id="link-id0x1dfd3540">information</a> on Myspace.</p> <p>Every time there is a major shift in technology, this shift needs to be motivated by addressing a new class of problem. This means doing something that could not be done before. The last time this happened was when the relational database became the dominant IT technology. At that time, the questions involved putting the enterprise in the database and building a cluster of Line Of Business (LOB) applications around the database. The argument for the <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1e920868">RDBMS</a> was that you did not have to constrain the set of queries that might later be made, when designing the database. In other words, it was making things more <i>ad hoc</i>. This was opposed then on grounds of being less efficient than the hierarchical and network databases which the relational eventually replaced.</p> <p>Today, the point of the Data Web is that you do not have to constrain what your data can join or integrate with, when you design your database. The counter-argument is that this is slow and geeky and not scalable. See the similarity?</p> <p>A difference is that we are not specifically aiming at replacing the RDBMS. In fact, if you know exactly what you will query and have a well defined workload, a relational representation optimized for the workload will give you about 10x the performance of the equivalent RDF warehouse. OLTP remains a relational-only domain.</p> <p>However, when we are talking about doing queries and analytics against the Web, or even against more than a handful of relational systems, the things which make RDBMS good become problematic.</p> <h3>What is the business value of this?</h3> <p>The most reliable of human drives is the drive to make oneself known. This drives all, from any social scene to business communications to politics. Today, when you want to proclaim you exist, you do so first on the Web. The Web did not become the prevalent media because business loved it for its own sake, it became prevalent because business could not afford not to assert their presence there. If anything, the Web eroded the communications dominance of a lot of players, which was not welcome but still had to be dealt with, by embracing the Web.</p> <p>Today, in a world driven by data, the Data Web will be catalyzed by similar factors: If your data is not there, you will not figure in query results. Search engines will play some role there but also many social applications will have reports that are driven by published data. Also consider any e-commerce, any marketplace, and so forth. The Data Portability movement is a case in point: Users want to own their own content; silo operators want to capitalize on holding it. Right now, we see these things in silos; the Data Web will create bridges between these, and what is now in silo data centers will be increasingly available on an ad hoc basis with Open Data.</p> <p>Again, we see a movement from the specialized to the generic: What LinkedIn does in its data center can be done with ad hoc queries with <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1e715138">linked open data</a>. Of course, LinkedIn does these things somewhat more efficiently because their system is built just for this task, but the linked data approach has the built-in readiness to join with everything else at almost no cost, without making a new data warehouse for each new business question.</p> <p>We could call this the sociological aspect of the thing. Getting to more concrete business, we see an economy that, we could say, without being alarmists, is confronted with some issues. Well, generally when times are bad, this results in consolidation of property and power. Businesses fail and get split up and sold off in pieces, government adds controls and regulations and so forth. This means ad hoc data integration, as control without data is just pretense. If times are lean, this also means that there is little readiness to do wholesale replacement of systems, which will take years before producing anything. So we must play with what there is and make it deliver, in ways and conditions that were not necessarily anticipated. The agility of the Data Web, if correctly understood, can be of great benefit there, especially on the reporting and business intelligence side. Specifically mapping line-of-business systems into RDF on the fly will help with integration, making the specialized warehouse the slower and more expensive alternative. But this too is needed at times.</p> <p>But for the RDF community to be taken seriously there, the messaging must be geared in this direction. Writing FOAF files by hand is not where you begin the pitch. Well, what is more natural then having a global, queriable information space, when you have a global information driven economy?</p> <p>The Data Web is about making this happen. First with doing this in published generally available data; next with the enterprises having their private data for their own use but still linking toward the outside, even though private data stays private: You can still use standard terms and taxonomies, where they apply, when talking of proprietary information.</p> <h3>But let's get back to more specific issues</h3> <p>At the lightning talks in Vienna, one participant said, "Man's enemy is not the lion that eats men, it's his own brother. Semantic Web's enemy is the <a href="http://dbpedia.org/resource/XML" id="link-id0x1aeb61b8">XML</a> Web services stack that ate its lunch." There is some truth to the first part. The second part deserves some comment. The Web services stack is about transactions. When you have a fixed, often repeating task, it is a natural thing to make this a Web service. Even though SOA is not really prevalent in enterprise IT, it has value in things like managing supply-chain logistics with partners, etc. Lots of standard messages with unambiguous meaning. To make a parallel with the database world: first there was OLTP; then there was business intelligence. Of course, you must first have the transactions, to have something to analyze.</p> <p>SOA is for the transactions; the Data Web is for integration, analysis, and discovery. It is the <i>ad hoc</i> component of the real time enterprise, if you will. It is not a competitor against a transaction oriented SOA. In fact, RDF has no special genius for transactions. Another mistake that often gets made is stretching things beyond their natural niche. Doing transactions in RDF is this sort of over-stretching without real benefit.</p> <p>"I made an ontology and it really did solve a problem. How do I convince the enterprise people, the MBA who says it's too complex, the developer who says it is not what he's used to, and so on?"</p> <p>This is an education question. One of the findings of SWEO's enterprise survey was that there was awareness that difficult problems existed. There were and are corporate ontologies and taxonomies, diversely implemented. Some of these needs are recognized. RDF based technologies offer to make these more open standards based. open standards have proven economical in the past. What we also hear is that major enterprises do not even know what their information and human resources assets are: Experts can't be found even when they are in the next department, or reports and analysis gets buried in wikis, spreadsheets, and emails.</p> <p>Just as when SQL took off, we need vendors to do workshops on getting started with a technology. The affair in Vienna was a step in this direction. Another type of event specially focusing on vertical problems and their Data Web solutions is a next step. For example, one could do a workshop on integrating supply chain information with Data Web technologies. Or one on making enterprise <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x1fbd3398">knowledge</a> bases from HR, CRM, office automation, wikis, etc. The good thing is that all these things are additions to, not replacements of, the existing mission-critical infrastructure. And better use of what you already have ought to be the theme of the day.</p>
2008-10-27T11:28:07-04:00
State of the Semantic Web, Part 1 - Sociology, Business, and Messaging (update 2)
http://www.openlinksw.com:443/weblog/oerling/?date=2008-10-24#1459
2008-10-24T10:19:03Z
<p>I was in <a href="http://dbpedia.org/resource/Vienna" id="link-id0x28471870">Vienna</a> for the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x26f0ec28">Linked Data</a> Practitioners gathering this week. Danny Ayers asked me if I would <a href="http://dbpedia.org/resource/Blog" id="link-id0x26cf7678">blog</a> about the State of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x273087e0">Semantic Web</a> or write the <i>This Week's Semantic Web</i> column. I don't have the time to cover all that may have happened during the past week but I will editorialize about the questions that again were raised in Vienna. How these things relate to <a href="http://virtuoso.openlinksw.com" id="link-id0x264e11b8">Virtuoso</a> will be covered separately. This is about the overarching questions of the times, not the finer points of geek craft.</p> <p> <a href="http://www.informatik.uni-leipzig.de/~auer/foaf.rdf#me" id="link-id0x2787de70">Sören Auer</a> asked me to say a few things about relational to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x280b12f8">RDF</a> mapping. I will cite some highlights from this, as they pertain to the general scene. There was an "open hacking" session Wednesday night featuring lightning talks. I will use some of these too as a starting point.</p> <h3>The messaging?</h3> <p>The <a href="http://www.w3.org/2001/sw/sweo/" id="link-id0x28078030">SWEO</a> (Semantic Web Education and Outreach) interest group of the W3C spent some time looking for an elevator pitch for the Semantic Web. It became "<a href="http://dbpedia.org/resource/Data" id="link-id0x290a48c0">Data</a> Unleashed." Why not? Let's give this some context.</p> <p>So, if we are holding a <i>Semantic Web 101</i> session, where should we begin? I hazard to guess that we should not begin by writing a FOAF file in Turtle by hand, as this is one thing that is not likely to happen in the real world.</p> <p>Of course, the social aspect of the Data Web is the most immediately engaging, so a demo might be to go make an account with <a href="http://myopenlink.net/" id="link-id0x272ed6d0">myopenlink</a>.<a href="http://dbpedia.org/resource/.NET_Framework" id="link-id0x277dbbd0">net</a> and see that after one has entered the data one normally enters for any social network, one has become a Data Web citizen. This means that one can be found, just like this, with a query against the set of data spaces hosted on the system. Then we just need a few pages that repurpose this data and relate it to other data. We show some samples of queries like this in our <a href="http://challenge.semanticweb.org/" id="link-id0x25fda5c8">Billion Triples Challenge</a> demo. We will make a webcast about this to make it all clearer.</p> <p>Behold: The Data Web is about the world becoming a database; writing <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x278c3878">SPARQL</a> queries or triples is incidental. You will write FOAF files by hand just as little as you now write <a href="http://dbpedia.org/resource/SQL" id="link-id0x27e6be18">SQL</a> insert statements for filling in your account <a href="http://dbpedia.org/resource/Information" id="link-id0x2727a278">information</a> on Myspace.</p> <p>Every time there is a major shift in technology, this shift needs to be motivated by addressing a new class of problem. This means doing something that could not be done before. The last time this happened was when the relational database became the dominant IT technology. At that time, the questions involved putting the enterprise in the database and building a cluster of Line Of Business (LOB) applications around the database. The argument for the <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x26020128">RDBMS</a> was that you did not have to constrain the set of queries that might later be made, when designing the database. In other words, it was making things more <i>ad hoc</i>. This was opposed then on grounds of being less efficient than the hierarchical and network databases which the relational eventually replaced.</p> <p>Today, the point of the Data Web is that you do not have to constrain what your data can join or integrate with, when you design your database. The counter-argument is that this is slow and geeky and not scalable. See the similarity?</p> <p>A difference is that we are not specifically aiming at replacing the RDBMS. In fact, if you know exactly what you will query and have a well defined workload, a relational representation optimized for the workload will give you about 10x the performance of the equivalent RDF warehouse. OLTP remains a relational-only domain.</p> <p>However, when we are talking about doing queries and analytics against the Web, or even against more than a handful of relational systems, the things which make RDBMS good become problematic.</p> <h3>What is the business value of this?</h3> <p>The most reliable of human drives is the drive to make oneself known. This drives all, from any social scene to business communications to politics. Today, when you want to proclaim you exist, you do so first on the Web. The Web did not become the prevalent media because business loved it for its own sake, it became prevalent because business could not afford not to assert their presence there. If anything, the Web eroded the communications dominance of a lot of players, which was not welcome but still had to be dealt with, by embracing the Web.</p> <p>Today, in a world driven by data, the Data Web will be catalyzed by similar factors: If your data is not there, you will not figure in query results. Search engines will play some role there but also many social applications will have reports that are driven by published data. Also consider any e-commerce, any marketplace, and so forth. The Data Portability movement is a case in point: Users want to own their own content; silo operators want to capitalize on holding it. Right now, we see these things in silos; the Data Web will create bridges between these, and what is now in silo data centers will be increasingly available on an ad hoc basis with Open Data.</p> <p>Again, we see a movement from the specialized to the generic: What LinkedIn does in its data center can be done with ad hoc queries with <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x261c7bc8">linked open data</a>. Of course, LinkedIn does these things somewhat more efficiently because their system is built just for this task, but the linked data approach has the built-in readiness to join with everything else at almost no cost, without making a new data warehouse for each new business question.</p> <p>We could call this the sociological aspect of the thing. Getting to more concrete business, we see an economy that, we could say, without being alarmists, is confronted with some issues. Well, generally when times are bad, this results in consolidation of property and power. Businesses fail and get split up and sold off in pieces, government adds controls and regulations and so forth. This means ad hoc data integration, as control without data is just pretense. If times are lean, this also means that there is little readiness to do wholesale replacement of systems, which will take years before producing anything. So we must play with what there is and make it deliver, in ways and conditions that were not necessarily anticipated. The agility of the Data Web, if correctly understood, can be of great benefit there, especially on the reporting and business intelligence side. Specifically mapping line-of-business systems into RDF on the fly will help with integration, making the specialized warehouse the slower and more expensive alternative. But this too is needed at times.</p> <p>But for the RDF community to be taken seriously there, the messaging must be geared in this direction. Writing FOAF files by hand is not where you begin the pitch. Well, what is more natural then having a global, queriable information space, when you have a global information driven economy?</p> <p>The Data Web is about making this happen. First with doing this in published generally available data; next with the enterprises having their private data for their own use but still linking toward the outside, even though private data stays private: You can still use standard terms and taxonomies, where they apply, when talking of proprietary information.</p> <h3>But let's get back to more specific issues</h3> <p>At the lightning talks in Vienna, one participant said, "Man's enemy is not the lion that eats men, it's his own brother. Semantic Web's enemy is the <a href="http://dbpedia.org/resource/XML" id="link-id0x26273118">XML</a> Web services stack that ate its lunch." There is some truth to the first part. The second part deserves some comment. The Web services stack is about transactions. When you have a fixed, often repeating task, it is a natural thing to make this a Web service. Even though SOA is not really prevalent in enterprise IT, it has value in things like managing supply-chain logistics with partners, etc. Lots of standard messages with unambiguous meaning. To make a parallel with the database world: first there was OLTP; then there was business intelligence. Of course, you must first have the transactions, to have something to analyze.</p> <p>SOA is for the transactions; the Data Web is for integration, analysis, and discovery. It is the <i>ad hoc</i> component of the real time enterprise, if you will. It is not a competitor against a transaction oriented SOA. In fact, RDF has no special genius for transactions. Another mistake that often gets made is stretching things beyond their natural niche. Doing transactions in RDF is this sort of over-stretching without real benefit.</p> <p>"I made an ontology and it really did solve a problem. How do I convince the enterprise people, the MBA who says it's too complex, the developer who says it is not what he's used to, and so on?"</p> <p>This is an education question. One of the findings of SWEO's enterprise survey was that there was awareness that difficult problems existed. There were and are corporate ontologies and taxonomies, diversely implemented. Some of these needs are recognized. RDF based technologies offer to make these more open standards based. open standards have proven economical in the past. What we also hear is that major enterprises do not even know what their information and human resources assets are: Experts can't be found even when they are in the next department, or reports and analysis gets buried in wikis, spreadsheets, and emails.</p> <p>Just as when SQL took off, we need vendors to do workshops on getting started with a technology. The affair in Vienna was a step in this direction. Another type of event specially focusing on vertical problems and their Data Web solutions is a next step. For example, one could do a workshop on integrating supply chain information with Data Web technologies. Or one on making enterprise <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x260172a8">knowledge</a> bases from HR, CRM, office automation, wikis, etc. The good thing is that all these things are additions to, not replacements of, the existing mission-critical infrastructure. And better use of what you already have ought to be the theme of the day.</p>
2015-08-20T17:21:50.348583-04:00
The Numerati & The Magic of You!
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-10-21#1458
2008-10-21T15:42:52Z
<p>In response to <a href="http://www.readwriteweb.com" id="link-id111d6ae8">ReadWriteWeb</a>'s post titled: <a href="http://www.readwriteweb.com/archives/who_will_control_your_data_web30.php" id="link-id113c27e0">Who will own your Data in Web 3.0 World?</a>. My simple answer: You!</p> <p>You will control your data in the Web 3.0 realm. If somehow this remains somewhat incomprehensible and nebulous (as is typical in this emerging realm) then simply think about this as: The Magic of You!</p> <p>Remember, "You" was the <a href="http://en.wikipedia.org/wiki/Time_Magazine" id="link-id144c52a8">Times</a> person of the year as an acknowledgement of the Web 2.0 phenomenon, and maybe this time next year it would simply be the "Magic of Being You" that's the person of the year :-)</p> <p>Web 3.0 brings databasing to the Web (as a feature). The single most important action item at this stage is the act of creating a record for yourself, in this new <a href="http://dbpedia.org/resource/federated_database_system" id="link-id11540b50">distributed database</a> held together by an HTTP based Network (e.g., the <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id115a02f8">World Wide Web</a>).</p> <h3>Related:</h3> <ol> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/GetAPersonalURIIn5MinutesOrLess" id="link-id113aead0">Get yourself a Web Database ID in 5 minutes or less</a> </li> <li> 2006 Callout from <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id118acdd8">TimBL</a>: <a href="http://dig.csail.mit.edu/breadcrumbs/node/71" id="link-id11126580">Get Yourself a URI</a> </li> <li> Just watch the <a href="http://www.youtube.com/watch?v=jBqByfoLGdU" id="link-id13d19568">Numerati Video</a> </li> </ol>
2010-02-01T08:55:22.000017-05:00
The Trouble with Labels (Contd.): Data Integration & SOA
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-10-12#1457
2008-10-12T18:53:44Z
<p>I just stumbled across an post from <a href="http://www.itbusinessedge.com" id="link-id10f82f50">ITBusines Edge</a> titled: <a href="http://www.itbusinessedge.com/item/?ci=48119" id="link-id10f37b90">How Semantic Technology Can Help Companies with Integration</a>. While reading the post I encountered the term: <a href="http://dbpedia.org/resource/Master_Data_Management" id="link-id11055eb8">Master Data Manager (MDM)</a>, and wondered to myself, "what's that?" only to realize it's the very same thing I described as a <a href="http://dbpedia.org/resource/Federated_database_system" id="link-id13985af0">Data Virtualization</a> or <a href="http://dbpedia.org/resource/Virtual_Database" id="link-id1167c720">Virtual Database technology</a> (circa. 1998).</p> <p>Now, if re-labeling can confuse <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id14aaaaf0">me</a> when applied to a realm I've been intimately involved with for eons (<a href="http://dbpedia.org/resource/Internet" id="link-id112042f0">internet</a> time). I don't want to imagine what it does for others who aren't that intimately involved with the important data access and data integration realms. </p> <p>On the more refreshing side, the article does shed some light on the potency of RDF and OWL when applied to the construction of conceptual views of heterogeneous data sources.</p> <blockquote> <cite>"How do you know that data coming from one place calculates net revenue the same way that data coming from another place does? You’ve got people using the same term for different things and different terms for the same things. How do you reconcile all of that? That’s really what semantic integration is about." </cite> </blockquote> <p>BTW - I discovered this article via another titled: <a href="http://www.itbusinessedge.com/blogs/mia/?p=485" id="link-id11134098">Understanding Integration And How It Can Help with SOA</a>, that covers SOA and Integration matters. Again, in this piece I feel the gradual realization of the virtues that RDF, OWL, and RDF <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11048740">Linked Data</a> bring to bear in the vital realm of data integration across heterogeneous data silos.</p> <h3>Conclusion</h3> <p>A number of events, at the micro and macro economic levels, are forcing attention back to the issue of productive use of existing IT resources. The trouble with the aforementioned quest is that it ultimately unveils the global IT affliction known as: heterogeneous data silos, and the challenges of pain alleviation, that have been ignored forever or approached inadequately as clearly shown by the rapid build up of SOA horror stories in the data integration realm.</p> <p>Data Integration via conceptualization of heterogenous data sources, that result in concrete conceptual layer data access and management, remains the greatest and most potent application of technologies associated with the "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id10fa5050">Semantic Web</a>" and/or "Linked Data" monikers.</p> <h3>Related</h3> <ul> <li> <a href="http://www.infoworld.com/article/03/05/23/21FEinnovidehen_1.html" id="link-id118c9c00">InforWorld 2003 Innovator article</a> </li> <li> <a href="http://weblog.infoworld.com/udell/2006/04/28.html" id="link-id11057298">2006 Podcast Interview with Jon Udell</a> </li> <li> <a href="http://dbpedia.org/resource/Enterprise_Information_Integration" id="link-id13f89030">Enterprise Information Integration</a> </li> <li>One of <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=data%20integration&type=text&output=html" id="link-id11048b98">several posts</a> about our <a href="http://virtuoso.openlinksw.com" id="link-id10fef0e0">Virtuoso</a> <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id10e5a068">Universal Server</a> and <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1406" id="link-id111d5aa8">Conceptual Model based data integration</a> </li> <li> <a href="http://virtuoso.openlinksw.com/wiki/main/Main/VOSHistory" id="link-id11020108">History of Virtuoso</a> </li> <li> <a href="http://www.mkbergman.com/me/" id="link-id1101e7b0">Mike Bergman</a>'s post titled: <a href="http://www.mkbergman.com/?p=459" id="link-id10fdb640">WOA: A New Enterprise Partner for Linked Data</a> </li> </ul>
2008-10-12T18:54:22-04:00
The Calamitous Nature of Opportunity
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-10-10#1456
2008-10-10T16:30:53Z
<p>As articulated in timeless fashion by <a href="http://dbpedia.org/resource/Albert_Einstein" id="link-id160b76b8">Albert Einstein</a>: </p> <blockquote> <cite>The significant problems we face cannot be solved at the same level of thinking we were at when we created them. </cite> </blockquote> <p>This quote also applies to the current global financial mess because the essence of this crisis remains inextricably linked to dependency on outdated "<a href="http://dbpedia.org/resource/Closed_world_assumption" id="link-id14a6b6c0">closed world</a>" systems.</p> <h3>How we got here (5,000 ft. view)</h3> <p>We have a global human network that depends on systems driven by, and confined to, <a href="http://dbpedia.org/resource/Data">data</a> silos! Every time you hear a CEO, Government Official, work colleague, neighbor, sibling, or relative tell you they didn't see it coming, just remember: </p> <ul> <li>For every action, there is an equal and opposite reaction</li> <li>For every debit there is a credit</li> <li>What goes around, comes around</li> <li> <a href="http://www.quotedb.com/quotes/245" id="link-id12ace758">No man is an Island</a> (little tweak: Human)</li> <li>We are all Linked whether we like it or not</li> <li>System preserving reboots are a feature of all intelligently designed systems.</li> </ul> <h3>Why there won't be a Depression</h3> <p>There won't be a depression because we can't afford one. Just like we couldn't afford to continue with the manner in which our systems work today. Unlike the '30s, we all know that there are no absolute safe havens right now, we have enough <a href="http://dbpedia.org/resource/Information" id="link-id13d0c258">information</a> at our disposal to eventually understand (post panic) that stuffing the mattress isn't an option (even government bonds won't cut it, ditto money market accounts).</p> <h3>The Opportunity</h3> <p>Take a deep breadth and tell traditional media to "shut up". As per usual, the traditional mass media wants to have it both ways by stoking the panic and maxing out on the frenzy with reckless abandon (as per usual). If there is a time to appreciate the blogosphere and quality journalism etc.. It's now.</p> <p> Anyway, as the saying goes: "It's always darkest before dawn", and as bizarre as this may sound in some quarters, things will ultimately change for the better. It just so happened that a really big cane was required in order for us to change our dysfunctional ways :-(</p> <p>I recently wrote a post about "<a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1440" id="link-id115387f8">zero based cognition</a>" that sought to bring attention to the power of "Human Thought" in relation to value creation.</p> <p>Innovative creation and dissemination of value is how we will eventually get out of the current mess (as we've done in the past). The predictability of the aforementioned reality is significantly increased by the sheer link density and resulting "network effects" potential of the <a href="http://dbpedia.org/resource/Internet" id="link-id14a595e8">Internet</a> and <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id1112a570">World Wide Web</a>. Our ability to "connect the dots" as part of our value creation, dissemination, and consumption processing pipelines is what will ultimately separate the winners from the losers (individuals, enterprises, nations).</p> <h3>Related</h3> <ul> <li> <a href="http://yihongs-research.blogspot.com" id="link-id14b0fb90">Yihong Ding</a>'s insightful <a href="http://yihongs-research.blogspot.com/2008/10/financial-crisis-who-will-be-winner.html" id="link-id112197b0">perspectives</a> </li> <li> <a href="http://www.jasonkolb.com/" id="link-id112d4ad8">Jason Kolb</a>'s poignant piece titled: <a href="http://www.jasonkolb.com/weblog/2008/10/the-year-the-innovation-died.html" id="link-id10fe7008">The Year Innovation Died</a> </li> <li> <a href="http://www.portfolio.com/views/blogs/the-tech-observer/2008/10/09/tech-start-ups-and-the-economys-best-hope?tid=true" id="link-id14a80788">Tech Start-ups and the Economy's Best Hope</a> </li> <li> <a href="http://video.google.com/videoplay?docid=-9050474362583451279" id="link-id11053b90">Money as Debt</a> - (a documentary spotted by <a href="http://hyperdata.org/blog/" id="link-id114c0e30">Danja</a>)</li> <li> <a href="http://www.alleyinsider.com/peter_kafka" id="link-id10f01b10">Peter Kalfka</a>'s post: <a href="http://www.alleyinsider.com/2008/12/startup-advice-how-to-make-the-collapse-work-for-you" id="link-id10de8058">Smart Startup Advice: Don't Panic - Profit</a> </li> <li> <a href="http://www.huffingtonpost.com/nathan-gardels/soros-end-of-financial-cr_b_134008.html" id="link-id10fef1e8">George Soros Interview</a> </li> <li>Mark Cuban (<a href="http://blogmaverick.com/" id="link-ide8b5298">Blog Maverick</a>) echoing "<a href="http://blogmaverick.com/2008/10/23/the-cure-to-our-economic-problems/" id="link-id10e630d8">Entrepreneurship is the key"</a> sentiment.</li> </ul>
2008-10-23T22:20:17-04:00
State of the Linked Data Web
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-10-10#1455
2008-10-10T02:27:44Z
<p> The evolution of the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> into a <a href="http://dbpedia.org/resource/federated_database_system" id="link-id13d825f8">federated database</a>, <a href="http://dbpedia.org/resource/Information" id="link-id11821e18">information</a> space, and <a href="http://dbpedia.org/resource/Knowledge" id="link-id147f5d20">knowledge</a>-base hybrid continues at frenetic pace.</p> <p> As more <a href="http://dbpedia.org/resource/Linked_Data" id="link-id14a805a8">Linked Data</a> is injected into the Web from the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id114ebeb8">Linking Open Data community</a> and other initiatives, it's important to note that "Linked Data" is available in a variety of forms such as:</p> <ul> <li> <a href="http://dbpedia.org/resource/Data">Data</a> Model Definition oriented Linked Data (aka. Data Dictionary)</li> <li> Data Model Instance Data (aka. Instance Data)</li> <li> Linked Data oriented solutions that leverage the smart data substrate that Models and Instance Data meshes deliver.</li> </ul> <p> Note: The common glue across the different types of Linked Data remains the commitment to data object (<a href="http://dbpedia.org/resource/Entity" id="link-id1103afe8">entity</a>) identification and access via de-referencable URIs (aka. record / entity level data source names).</p> <p> As stated in my recent post titled: <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id11743278">Semantic Web</a>: <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1444" id="link-id10f44ce0">Travails to Harmony Illustrated</a>. Harmonious intersections of instance data, data dictionaries (schemas, ontologies, rules etc.) provide a powerful substrate (smart data) for the development and deployment of "People" and/or "Machine" oriented solutions. Of course, others have commented on these matters and expressed similar views (see related section below).</p> <p> The clickable venn diagram below, provides a simple exploration path that exposes the linkage that already exists, across the different Linked Data types, within the burgeoning <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id1132fe60">Linked Data Web</a>.</p> <div> <map name="LiveCloud"> <area coords="356,136,120" href="http://umbel.org/images/lod_constellation.html" shape="circle" /> <area coords="140,136,120" href="http://www4.wiwiss.fu-berlin.de/bizer/pub/lod-datasets_2008-09-18.html" shape="circle" /> <area coords="248,280,120" href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/ClickableVirtSpongerCloud" shape="circle" /> </map> <img alt="Image" border="0" src="http://virtuoso.openlinksw.com/ldp_presentation/images/linked_data_people_schems_venn.png" usemap="#LiveCloud" /> </div> <h3> Related</h3> <ul> <li> <a href="http://jhingran.typepad.com/" id="link-id14aeb438">Anant Jingran</a>'s insightful <a href="http://intranet.usnet.private:8893/anant_jhingrans_musings/2008/08/future-of-database-research-is-excellent-but-what-is-the-future-of-data.html" id="link-id1158ca98">LDP Conference Trip report</a> </li> <li> Anant's recent post about the <a href="http://jhingran.typepad.com/anant_jhingrans_musings/2008/08/future-of-database-research-is-excellent-but-what-is-the-future-of-data.html" id="link-id1128fd78">future of Data</a> </li> <li> <a href="http://www.mkbergman.com/me/" id="link-id1114d330">Mike Bergman</a> - <a href="http://www.mkbergman.com/457/a-new-constellation-in-the-linking-open-data-lod-sky/" id="link-id114780f8">A New Constellation in the Linking Open Data (LOD) Sky</a> </li> <li> <a href="http://fgiasson.com/me/" id="link-id14aedaf0">Frederick Giasson</a> - <a href="http://fgiasson.com/blog/index.php/2008/09/04/exploding-dbpedias-domain-using-umbel" id="link-id12daa6d0">Exploding DBpedia Domain using UMBEL</a> </li> </ul>
2010-03-28T18:25:19-04:00
Dynamic Linked Data Constellation
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-10-09#1454
2008-10-09T21:23:25Z
<p>Now that the virtues of dynamic generation of RDF based <a href="http://dbpedia.org/resource/Linked_Data" id="link-id14c429a0">Linked Data</a> are becoming clearer, I guess it's time to unveil the <a href="http://virtuoso.openlinksw.com" id="link-id13d7c7e0">Virtuoso</a> <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id115d0c20">Sponger</a> driven Dynamic Linked Data constellation diagram.</p> <p>Our diagram depicts the myriad of <a href="http://dbpedia.org/resource/Data">data</a> sources from which RDF Linked Data is generated "on the fly" via our data source specific RDF-zation cartridges/drivers. It also unveils how the sponger leverages the Linked Data constellations of <a href="http://umbel.org/" id="link-id14bd5700">UMBEL</a>, <a href="http://dbpedia.org/resource/DBpedia" id="link-id110f5a48">DBpedia</a>, <a href="http://www.bio2rdf.org/" id="link-id11494bc0">Bio2Rdf</a>, and others for lookups.</p> <map name="GraffleExport"> <area shape="circle" coords="723,292,36" href="http://linkeddata.uriburner.com/about/html/http://xbel.sourceforge.net/bookmarks/xbel.xbel" /> <area shape="circle" coords="423,309,36" href="http://linkeddata.uriburner.com/about/html/http://www.w3.org/2001/sw/grddl-wg/doc29/hotel-data.html" /> <area shape="circle" coords="592,285,36" href="http://linkeddata.uriburner.com/about/html/http://farm4.static.flickr.com/3067/2590298570_304a594899_t.jpg" /> <area shape="circle" coords="685,220,36" href="http://linkeddata.uriburner.com/about/html/http://microformats.org/feed/" /> <area shape="circle" coords="817,182,36" href="http://linkeddata.uriburner.com/about/html/http://en.wikipedia.org/wiki/London" /> <area shape="circle" coords="745,447,36" href="http://linkeddata.uriburner.com/about/http://www.sec.gov/Archives/edgar/data/51143/000110465908059468/ibm-20080429.xml" /> <area shape="circle" coords="597,387,36" href="http://linkeddata.uriburner.com/about/html/http://www.twine.com/twine/1p2dqhdx-1jg/nova-spivack-my-public-twine" /> <area shape="circle" coords="659,459,36" href="http://linkeddata.uriburner.com/about/html/http://www.theyworkforyou.com/mp/diane_abbott/hackney_north_and_stoke_newington" /> <area shape="circle" coords="585,494,36" href="http://linkeddata.uriburner.com/about/html/http://www.discogs.com/release/634302" /> <area shape="circle" coords="817,503,36" href="http://linkeddata.uriburner.com/about/html/http://getsatisfaction.com/mozilla/topics/ubiquity_mostly_fails_on_mac_ppc" /> <area shape="circle" coords="753,563,36" href="http://linkeddata.uriburner.com/about/html/http://www.sec.gov/Archives/edgar/data/789019/000119312507259490/msft-20070930.xml" /> <area shape="circle" coords="676,559,36" href="http://linkeddata.uriburner.com/about/html/http://twitter.com/statuses/user_timeline/633273.rss" /> <area shape="circle" coords="845,592,36" href="http://linkeddata.uriburner.com/about/html/http://ns.hr-xml.org/2_5/HR-XML-2_5/SEP/ResumeExample.xml" /> <area shape="circle" coords="845,431,36" href="http://linkeddata.uriburner.com/about/html/http://www.w3.org/2003/12/rdf-in-xhtml-xslts/complete-example.html" /> <area shape="circle" coords="701,375,36" href="http://linkeddata.uriburner.com/about/html/http://www.molly.com/people.php" /> <area shape="circle" coords="829,342,36" href="http://linkeddata.uriburner.com/about/html/http://upload.wikimedia.org/wikipedia/commons/b/b0/Hydroxide_lone_pairs-2D.svg" /> <area shape="circle" coords="341,259,36" href="http://linkeddata.uriburner.com/about/html/http://friendfeed.com/search?q=linked+data" /> <area shape="circle" coords="294,170,36" href="http://linkeddata.uriburner.com/about/html/http://local.yahooapis.com/MapsService/V1/trafficData?appid=Jz0wAU7V34Ff7gCzblxD.86qjGTEn._H3KA3PEajNE3xBMIbQtEZqIqHRh0G&street=701+First+Street&city=Sunnyvale&state=CA" /> <area shape="circle" coords="829,262,36" href="http://linkeddata.uriburner.com/about/html/http://blog.disqus.net/2008/08/25/reblog-comments-can-be-blog-posts/" /> <area shape="circle" coords="407,526,36" href="http://linkeddata.uriburner.com/about/html/http://finance.yahoo.com/q?s=AAPL" /> <area shape="circle" coords="503,292,36" href="http://linkeddata.uriburner.com/about/html/http://isbndb.com/d/person/berners_lee_tim.html" /> <area shape="circle" coords="310,501,36" href="http://linkeddata.uriburner.com/about/html/http://musicbrainz.org/release/37e955d4-a53c-45aa-a812-1b23b88dbc13.html" /> <area shape="circle" coords="413,221,36" href="http://linkeddata.uriburner.com/about/html/http://www.freebase.com/view/en/abraham_lincoln" /> <area shape="circle" coords="553,213,36" href="http://linkeddata.uriburner.com/about/html/http://slideshow.rubyforge.org/microformats.html" /> <area shape="circle" coords="445,93,36" href="http://linkeddata.uriburner.com/about/html/http://farm1.static.flickr.com/212/496684670_7122c831ed.jpg" /> <area shape="circle" coords="680,663,36" href="http://linkeddata.uriburner.com/about/html/http://cgi.ebay.com/Eclipse-by-Stephenie-Meyer-2007_W0QQitemZ250283828939QQcmdZViewItem?hash=item250283828939&_trkparms=240%3A1318&_trksid=p4295" /> <area shape="circle" coords="553,663,36" href="http://linkeddata.uriburner.com/about/html/http://digg.com/general_sciences/at_last-stem_cells_without_side_effects_" /> <area shape="circle" coords="248,651,36" href="http://linkeddata.uriburner.com/about/html/http://delicious.com/popular/blog" /> <area shape="circle" coords="158,554,36" href="http://linkeddata.uriburner.com/about/html/http://www.crunchbase.com/person/sergey-brin" /> <area shape="circle" coords="214,477,36" href="http://linkeddata.uriburner.com/about/html/http://virtuoso.openlinksw.com/presentations/Creating_Deploying_Exploiting_Linked_Data2/Creating_Deploying_Exploiting_Linked_Data2_TimBL_v3.html" /> <area shape="circle" coords="209,339,36" href="http://linkeddata.uriburner.com/about/html/https://bugzilla.mozilla.org/show_bug.cgi?id=251714" /> <area shape="circle" coords="264,405,36" href="http://linkeddata.uriburner.com/about/html/http://ma.gnolia.com/rss/full/tags/microformats" /> <area shape="circle" coords="417,663,36" href="http://linkeddata.uriburner.com/about/html/http://www.lawyer-directory.net/lawyer/Grand-Rapids-Bar-Association-Grand-Rapids-law434032.htm" /> <area shape="circle" coords="497,733,36" href="http://linkeddata.uriburner.com/about/html/http://www.concertbuzz.net/genres/classic-rock/jethro-tull.html" /> <area shape="circle" coords="373,437,36" href="http://linkeddata.uriburner.com/about/html/http://openmediaweb.org/index.php/2008/01/13/publishing-my-workout-music-in-haudio/" /> <area shape="circle" coords="373,121,36" href="http://linkeddata.uriburner.com/about/html/https://apml.engagd.com/apml/danielabarbosa.myopenid.com.apml" /> <area shape="circle" coords="245,559,36" href="http://linkeddata.uriburner.com/about/html/http://www.w3.org/2001/sw/grddl-wg/doc29/hotel-data.html" /> <area shape="circle" coords="170,650,36" href="http://linkeddata.uriburner.com/about/html/http://www.amazon.co.uk/Owl-Representing-Information-Ontology-Language/dp/1412034485/ref=sr_1_1/203-3289794-0348766?ie=UTF8&s=books&qid=1177155160&sr=1-1" /> <area shape="circle" coords="327,703,36" href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtSponger" /> <area shape="circle" coords="481,170,36" href="http://linkeddata.uriburner.com/about/html/http://www.w3.org/2004/08/TalkFiles/iCal/TalksAll.ics" /> <area shape="circle" coords="487,615,36" href="http://linkeddata.uriburner.com/about/html/http://openlibrary.org/b/OL7983950M" /> <area shape="circle" coords="158,417,36" href="http://linkeddata.uriburner.com/about/html/http://xml.weather.yahoo.com/forecastrss?p=USCA1116&u=f" /> <area shape="circle" coords="608,729,36" href="http://linkeddata.uriburner.com/about/html/http://www.youtube.com" /> <area shape="circle" coords="726,148,36" href="http://linkeddata.uriburner.com/about/html/http://www.lespetitescases.net/semantique-et-xhtml" /> <area shape="circle" coords="773,663,36" href="http://linkeddata.uriburner.com/about/html/http://upcoming.org/event/130719/" /> <area shape="circle" coords="553,118,36" href="http://linkeddata.uriburner.com/about/html/http://tech.yahoo.com/pr/apple-ipod-video-30gb-black-mp3-player/1992981873" /> <area shape="circle" coords="608,609,36" href="http://linkeddata.uriburner.com/about/html/http://socialgraph.apis.google.com/otherme?pretty=1&q=www.openlinksw.com/blog/~kidehen/" /> <area shape="circle" coords="487,516,36" href="http://linkeddata.uriburner.com/about/html/oai:dcmi.ischool.washington.edu:article/8" /> <area shape="circle" coords="635,158,36" href="http://linkeddata.uriburner.com/about/html/http://news.bbc.co.uk/rss/feeds.opml" /> <area shape="circle" coords="256,261,36" href="http://linkeddata.uriburner.com/about/html/http://www.google.com/base/feeds/snippets?bq=%20%5bemployer:%20Hewlett-Packard%5d%20%20%5bjob%20type:full-time%5d" /> <area shape="circle" coords="120,248,36" href="http://linkeddata.uriburner.com/about/html/http://www.apple.com/" /> <area shape="circle" coords="120,339,36" href="http://linkeddata.uriburner.com/about/html/http://www.fileinfo.net/extension/atom" /> <area shape="circle" coords="337,343,36" href="http://linkeddata.uriburner.com/about/html/urn:lsid:ubio.org:namebank:12292" /> <area shape="circle" coords="340,598,36" href="http://linkeddata.uriburner.com/about/html/http://www.maine.gov/portal/government/calendar.shtml" /> <area shape="circle" coords="337,343,36" href="http://linkeddata.uriburner.com/about/html/urn:lsid:ubio.org:namebank:12292" /> <area shape="circle" coords="585,494,36" href="http://linkeddata.uriburner.com/about/html/http://www.discogs.com/release/634302" /> <area shape="circle" coords="491,409,36" href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtSponger" /> <area shape="poly" coords="992,603,1015,557,1061,557,1100,592,1076,638,1030,638,992,603" href="http://bio2rdf.org/wiki/attach/Main/bio2rdfmap_blanc.png" /> <area shape="poly" coords="995,454,1018,408,1064,408,1103,443,1079,489,1033,489,995,454" href="http://www4.wiwiss.fu-berlin.de/bizer/pub/lod-datasets_2008-09-18.html" /> <area shape="poly" coords="919,117,943,71,989,71,1027,106,1004,152,958,152,919,117" href="http://umbel.org/images/081005_lod_constellation.png" /> </map> <img alt="Image" border="0" src="http://virtuoso.openlinksw.com/images/sponger-cloud.png" usemap="#GraffleExport" />
2008-10-17T10:45:53.000004-04:00
What is Linked Data oriented RDF-ization?
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-10-06#1453
2008-10-06T20:14:26Z
<p>RDF-ization is a term used by the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id14b4ebd0">Semantic Web</a> community to describe the process of generating RDF from non RDF <a href="http://dbpedia.org/resource/Data">Data</a> Sources such as (X)HTML, Weblogs, Shared Bookmark Collections, Photo Galleries, Calendars, Contact Managers, Feed Subscriptions, Wikis, and other <a href="http://dbpedia.org/resource/Information" id="link-id13f2a2e0">information</a> resource collections. </p> <p>If the RDF generated, results in an <a href="http://dbpedia.org/resource/Entity" id="link-id11281278">entity</a>-to-entity level network (graph) in which each entity is endowed with a de-referencable HTTP based ID (a <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id148200f0">URI</a>), we end up with an enhancement to the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> that adds <a href="http://dbpedia.org/resource/Linked_Data" id="link-id112a5980">Hyperdata</a> linking across extracted entities, to the existing Hypertext based Web of linked documents (pages, images, and other information resource types). Thus, I can use the same <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id10ebc630">URL</a> linking mechanism to reference a broader range of "Things" i.e., documents, things that documents are about, or things loosely associated with documents.</p> <p>The <a href="http://virtuoso.openlinksw.com" id="link-id144304a8">Virtuoso</a> <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id14a96400">Sponger</a> is an example of an <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=rdf%20middleware&type=text&output=html" id="link-id14d36938">RDF Middleware</a> solution from <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id117e5c50">OpenLink Software</a>. It's an in-built component of the Virtuoso <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id14b4d0e0">Universal Server</a>, and deployable in many forms e.g., Software as Service (SaaS) or traditional software installation. It delivers RDF-ization services via a collection of Web information resource specific Cartridges/Providers/Drivers covering Wikipedia, Freebase, CrunchBase, WikiCompany, OpenLibrary, Digg, eBay, Amazon, RSS/Atom/OPML feed sources, XBRL, and many more.</p> <p>RDF-ization alone doesn't ensure valuable RDF based <a href="http://dbpedia.org/resource/Linked_Data" id="link-id14a75b48">Linked Data</a> on the Web. The process of producing RDF Linked Data is ultimately about the art of effectively describing resources with an eye for <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id1442fea0">context</a>. </p> <h3>RDF-ization Processing Steps</h3> <ol> <li> Entity Extraction</li> <li> Vocabulary/Schema/Ontology (Data Dictionary) mapping</li> <li> HTTP based Proxy URI generation</li> <li>Linked Data Cloud Lookups (e.g., perform <a href="http://umbel.org/about/" id="link-id14432f00">UMBEL</a> lookup to add "isAbout" fidelity to graph and then lookup <a href="http://dbpedia.org/resource/DBpedia" id="link-id14485f40">DBpedia</a> and other <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id115ea410">LOD</a> instance data enclaves for Identical individuals and connect via "owl:sameAs")</li> <li> RDF <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id14ae31c0">Linked Data Graph</a> projection that uses the description of the container information resource to expose the URIs of the distilled entities.</li> </ol> <p>The animation that follows illustrates the process (5,000 feet view), from grabbing resources via HTTP GET, to injecting RDF Linked Data back into the Web cloud:</p> <div> <embed src="http://virtuoso.openlinksw.com/screencasts/virtuoso-rdf-middleware.swf" type="application/x-shockwave-flash" width="800" height="600"></embed> </div> <p>Note: the Shredder is a Generic Cartridge, so you would have one of these per data source type (information resource type).</p>
2008-10-07T17:35:24-04:00
Virtuoso Cluster Paper Update
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-10-02#1451
2008-10-02T10:02:33Z
<p>An updated version of the paper about <a href="http://virtuoso.openlinksw.com" id="link-id0xc0abc50">Virtuoso</a> Cluster is available at <a href="http://www.openlinksw.com/weblog/oerling/2008webscale_rdf.pdf" id="link-id16459248">2008webscale_rdf.pdf</a> </p>
2008-10-03T04:38:06-04:00
Virtuoso Update, Billion Triples and Outlook
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-10-02#1450
2008-10-02T10:02:32Z
<div> <div style="display:none;">Virtuoso Update, Billion Triples and Outlook</div> <p>I will say a few things about what we have been doing and where we can go.</p> <p>Firstly, we have a fairly scalable platform with <a href="http://virtuoso.openlinksw.com" id="link-id0x1aa82dc0">Virtuoso</a> 6 Cluster. It was most recently tested with the workload discussed in the previous <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1445" id="link-id1638a5b8">Billion Triples post</a>.</p> <p>There is an updated version of <a href="http://www.openlinksw.com/weblog/oerling/2008webscale_rdf.pdf" id="link-id16280a68">the paper about this</a>. This will be presented at the web scale workshop of ISWC 2008 in Karlsruhe.</p> <p>Right now, we are polishing some things in Virtuoso 6 -- some optimizations for smarter balancing of interconnect traffic over multiple network interfaces, and some more <a href="http://dbpedia.org/resource/SQL" id="link-id0x1abd3f38">SQL</a> optimizations specific to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1adbe410">RDF</a>. The must-have basics, like parallel running of sub-queries and aggregates, and all-around unrolling of loops of every kind into large partitioned batches, is all there and proven to work.</p> <p>We spent a lot of time around the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1aaa0e78">Berlin SPARQL Benchmark</a> story, so we got to the more advanced stuff like the <a href="http://challenge.semanticweb.org/" id="link-id0x1a860a50">Billion Triples Challenge</a> rather late. We did along the way also run <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1a27f2a8">BSBM</a> with an <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x1ad5c918">Oracle</a> back-end, with Virtuoso mapping <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1cf0e4a0">SPARQL</a> to SQL. This merits its own analysis in the near future. This will be the basic how-to of mapping OLTP systems to RDF. Depending on the case, one can use this for lookups in real-time or ETL.</p> <p>RDF will deliver value in complex situations. An example of a complex relational mapping use case came from Ordnance Survey, presented at the <a href="http://www.w3.org/2005/Incubator/rdb2rdf/" id="link-id0x1ab96bb0">RDB2RDF XG</a>. Examples of complex warehouses include the <a href="http://neurocommons.org/page/Main_Page" id="link-id0x1adb2db0">Neurocommons</a> database, the Billion Triples Challenge, and the <a href="http://www.garlik.com/" id="link-id0x1925c7b0">Garlik DataPatrol</a>.</p> <p>In comparison, the Berlin workload is really simple and one where RDF is not at its best, as amply discussed on the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1c6d1480">Linked Data</a> forum. BSBM's primary value is as a demonstrator for the basic mapping tasks that will be repeated over and over for pretty much any online system when presence on the <a href="http://dbpedia.org/resource/Data" id="link-id0x1a937400">data</a> web becomes as indispensable as presence on the HTML web.</p> <p>I will now talk about the complex warehouse/web-harvesting side. I will come to the mapping in another post.</p> <p>Now, all the things shown in the <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1445" id="link-id14de1d18">Billion Triples post</a> can be done with a relational system specially built for each purpose. Since we are a general purpose <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1a457c70">RDBMS</a>, we use this capability where it makes sense. For example, storing statistics about which tags or interests occur with which other tags or interests as RDF blank nodes makes no sense. We do not even make the experiment; we know ahead of time that the result is at least an order of magnitude in favor of the relational row-oriented solution in both space and time.</p> <p>Whenever there is a data structure specially made for answering one specific question, like joint occurrence of tags, RDB and mapping is the way to go. With Virtuoso, this can fully-well coexist with physical triples, and can still be accessed in SPARQL and mixed with triples. This is territory that we have not extensively covered yet, but we will be giving some examples about this later.</p> <p>The real value of RDF is in agility. When there is no time to design and load a new warehouse for every new question, RDF is unparalleled. Also SPARQL, once it has the necessary extensions of aggregating and sub-queries, is nicer than SQL, especially when we have sub-classes and sub-properties, transitivity, and "same as" enabled. These things have some run time cost and if there is a report one is hitting absolutely all the time, then chances are that resolving terms and identity at load-time and using materialized views in SQL is the reasonable thing. If one is inventing a new report every time, then RDF has a lot more convenience and flexibility.</p> <p>We are just beginning to explore what we can do with data sets such as the online conversation space, linked data, and the open ontologies of <a href="http://umbel.org/about/" id="link-id0x1aa5ea18">UMBEL</a> and <a href="http://dbpedia.org/resource/Cyc" id="link-id0x1a631a20">OpenCyc</a>. It is safe to say that we can run with real world scale without loss of query expressivity. There is an incremental cost for performance but this is not prohibitive. Serving the whole billion triples set from memory would cost about $32K in hardware. $8K will do if one can wait for disk part of the time. One can use these numbers as a basis for costing larger systems. For online search applications, one will note that running the indexes pretty much from memory is necessary for flat response time. For back office analytics this is not necessarily as critical. It all depends on the use case.</p> <p>We expect to be able to combine geography, social proximity, subject matter, and <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id0x1aebdcc8">named entities</a>, with hierarchical taxonomies and traditional full text, and to present this through a simple user interface.</p> <p>We expect to do this with online response times if we have a limited set of starting points and do not navigate more than 2 or 3 steps from each starting point. An example would be to have a full text pattern and news group, and get the cloud of interests from the authors of matching posts. Another would be to make a faceted view of the properties of the 1000 people most closely connected to one person.</p> <p>Queries like finding the fastest online responders to questions about romance across the global board-scape, or finding the person who initiates the most long running conversations about crime, take a bit longer but are entirely possible.</p> <p>The genius of RDF is to be able to do these things within a general purpose database, ad hoc, in a single query language, mostly without materializing intermediate results. Any of these things could be done with arbitrary efficiency in a custom built system. But what is special now is that the cost of access to this type of <a href="http://dbpedia.org/resource/Information" id="link-id0x1ab88490">information</a> and far beyond drops dramatically as we can do these things in a far less labor intensive way, with a general purpose system, with no redesigning and reloading of warehouses at every turn. The query becomes a commodity.</p> <p>Still, one must know what to ask. In this respect, the self-describing nature of RDF is unmatched. A query like <i>list the top 10 attributes with the most distinct values for all persons</i> cannot be done in SQL. SQL simply does not allow the columns to be variable.</p> <p>Further, we can accept queries as text, the way people are used to supplying them, and use structure for drill-down or result-relevance, and also recognize named entities and subject matter concepts in query text. Very simple NLP will go a long way towards keeping SPARQL out of the user experience.</p> <p>The other way of keeping query complexity hidden is to publish hand-written SPARQL as parameter-fed canned reports.</p> <p>Between now and ISWC 2008, the last week of October, we will put out demos showing some of these things. Stay tuned.</p> </div>
2008-10-02T12:47:07.000004-04:00
Virtuoso Cluster Paper Update
http://www.openlinksw.com:443/weblog/oerling/?date=2008-10-02#1449
2008-10-02T09:38:14Z
<p>An updated version of the paper about <a href="http://virtuoso.openlinksw.com" id="link-id0x17b3d2c8">Virtuoso</a> Cluster is available at <a href="http://www.openlinksw.com/weblog/oerling/2008webscale_rdf.pdf" id="link-id16459248">2008webscale_rdf.pdf</a> </p>
2008-10-03T04:38:01-04:00
Virtuoso Update, Billion Triples and Outlook
http://www.openlinksw.com:443/weblog/oerling/?date=2008-10-02#1448
2008-10-02T09:31:17Z
<p>I will say a few things about what we have been doing and where we can go.</p> <p>Firstly, we have a fairly scalable platform with <a href="http://virtuoso.openlinksw.com" id="link-id0xa412e450">Virtuoso</a> 6 Cluster. It was most recently tested with the workload discussed in the previous <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1445" id="link-id1638a5b8">Billion Triples post</a>.</p> <p>There is an updated version of <a href="http://www.openlinksw.com/weblog/oerling/2008webscale_rdf.pdf" id="link-id16280a68">the paper about this</a>. This will be presented at the web scale workshop of ISWC 2008 in Karlsruhe.</p> <p>Right now, we are polishing some things in Virtuoso 6 -- some optimizations for smarter balancing of interconnect traffic over multiple network interfaces, and some more <a href="http://dbpedia.org/resource/SQL" id="link-id0x1c1c5f48">SQL</a> optimizations specific to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1bcb6108">RDF</a>. The must-have basics, like parallel running of sub-queries and aggregates, and all-around unrolling of loops of every kind into large partitioned batches, is all there and proven to work.</p> <p>We spent a lot of time around the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x3a4e17c8">Berlin SPARQL Benchmark</a> story, so we got to the more advanced stuff like the <a href="http://challenge.semanticweb.org/" id="link-id0x1a66c568">Billion Triples Challenge</a> rather late. We did along the way also run <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x188c2608">BSBM</a> with an <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x1aa97f98">Oracle</a> back-end, with Virtuoso mapping <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1abd87a0">SPARQL</a> to SQL. This merits its own analysis in the near future. This will be the basic how-to of mapping OLTP systems to RDF. Depending on the case, one can use this for lookups in real-time or ETL.</p> <p>RDF will deliver value in complex situations. An example of a complex relational mapping use case came from Ordnance Survey, presented at the <a href="http://www.w3.org/2005/Incubator/rdb2rdf/" id="link-id0x1a941678">RDB2RDF XG</a>. Examples of complex warehouses include the <a href="http://neurocommons.org/page/Main_Page" id="link-id0x1aa5a9f8">Neurocommons</a> database, the Billion Triples Challenge, and the <a href="http://www.garlik.com/" id="link-id0x372df7b0">Garlik DataPatrol</a>.</p> <p>In comparison, the Berlin workload is really simple and one where RDF is not at its best, as amply discussed on the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1a671cf0">Linked Data</a> forum. BSBM's primary value is as a demonstrator for the basic mapping tasks that will be repeated over and over for pretty much any online system when presence on the <a href="http://dbpedia.org/resource/Data" id="link-id0x1ab83dd0">data</a> web becomes as indispensable as presence on the HTML web.</p> <p>I will now talk about the complex warehouse/web-harvesting side. I will come to the mapping in another post.</p> <p>Now, all the things shown in the <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1445" id="link-id14de1d18">Billion Triples post</a> can be done with a relational system specially built for each purpose. Since we are a general purpose <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x340d3470">RDBMS</a>, we use this capability where it makes sense. For example, storing statistics about which tags or interests occur with which other tags or interests as RDF blank nodes makes no sense. We do not even make the experiment; we know ahead of time that the result is at least an order of magnitude in favor of the relational row-oriented solution in both space and time.</p> <p>Whenever there is a data structure specially made for answering one specific question, like joint occurrence of tags, RDB and mapping is the way to go. With Virtuoso, this can fully-well coexist with physical triples, and can still be accessed in SPARQL and mixed with triples. This is territory that we have not extensively covered yet, but we will be giving some examples about this later.</p> <p>The real value of RDF is in agility. When there is no time to design and load a new warehouse for every new question, RDF is unparalleled. Also SPARQL, once it has the necessary extensions of aggregating and sub-queries, is nicer than SQL, especially when we have sub-classes and sub-properties, transitivity, and "same as" enabled. These things have some run time cost and if there is a report one is hitting absolutely all the time, then chances are that resolving terms and identity at load-time and using materialized views in SQL is the reasonable thing. If one is inventing a new report every time, then RDF has a lot more convenience and flexibility.</p> <p>We are just beginning to explore what we can do with data sets such as the online conversation space, linked data, and the open ontologies of <a href="http://umbel.org/about/" id="link-id0x19cabf38">UMBEL</a> and <a href="http://dbpedia.org/resource/Cyc" id="link-id0x19cecd10">OpenCyc</a>. It is safe to say that we can run with real world scale without loss of query expressivity. There is an incremental cost for performance but this is not prohibitive. Serving the whole billion triples set from memory would cost about $32K in hardware. $8K will do if one can wait for disk part of the time. One can use these numbers as a basis for costing larger systems. For online search applications, one will note that running the indexes pretty much from memory is necessary for flat response time. For back office analytics this is not necessarily as critical. It all depends on the use case.</p> <p>We expect to be able to combine geography, social proximity, subject matter, and <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id0x1a8202e8">named entities</a>, with hierarchical taxonomies and traditional full text, and to present this through a simple user interface.</p> <p>We expect to do this with online response times if we have a limited set of starting points and do not navigate more than 2 or 3 steps from each starting point. An example would be to have a full text pattern and news group, and get the cloud of interests from the authors of matching posts. Another would be to make a faceted view of the properties of the 1000 people most closely connected to one person.</p> <p>Queries like finding the fastest online responders to questions about romance across the global board-scape, or finding the person who initiates the most long running conversations about crime, take a bit longer but are entirely possible.</p> <p>The genius of RDF is to be able to do these things within a general purpose database, ad hoc, in a single query language, mostly without materializing intermediate results. Any of these things could be done with arbitrary efficiency in a custom built system. But what is special now is that the cost of access to this type of <a href="http://dbpedia.org/resource/Information" id="link-id0x1ab0a918">information</a> and far beyond drops dramatically as we can do these things in a far less labor intensive way, with a general purpose system, with no redesigning and reloading of warehouses at every turn. The query becomes a commodity.</p> <p>Still, one must know what to ask. In this respect, the self-describing nature of RDF is unmatched. A query like <i>list the top 10 attributes with the most distinct values for all persons</i> cannot be done in SQL. SQL simply does not allow the columns to be variable.</p> <p>Further, we can accept queries as text, the way people are used to supplying them, and use structure for drill-down or result-relevance, and also recognize named entities and subject matter concepts in query text. Very simple NLP will go a long way towards keeping SPARQL out of the user experience.</p> <p>The other way of keeping query complexity hidden is to publish hand-written SPARQL as parameter-fed canned reports.</p> <p>Between now and ISWC 2008, the last week of October, we will put out demos showing some of these things. Stay tuned.</p>
2015-08-20T17:32:28.645488-04:00
Where Are All the RDF-based Semantic Web Applications?
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-10-01#1447
2008-10-01T23:09:00Z
<p> In response to the "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id15971040">Semantic Web</a> Technology" application classification scheme espoused by <a href="http://www.readwriteweb.com" id="link-id16391540">ReadWriteWeb</a> (RWW), emphasized in the post titled: <a href="http://www.readwriteweb.com/archives/rdf_semantic_web_apps.php" id="link-id1157eaa0">Where are all the RDF-based Semantic Web Apps?</a>, here is my attempt to clarify and reintroduce what <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id15a43758">OpenLink Software</a> offers (today) in relation to Semantic Web technology. </p> <p> From the RWW Top-Down category, which I interpret as: technologies that produce RDF from non RDF <a href="http://dbpedia.org/resource/Data">data</a> sources. Our product portfolio is comprised of the following; <a href="http://virtuoso.openlinksw.com" id="link-id14f05818">Virtuoso Universal Server</a>, <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id162c8630">OpenLink Data Spaces</a>, <a href="http://oat.openlinksw.com" id="link-id134e1a00">OpenLink Ajax Toolkit</a>, and <a href="http://ode.openlinksw.com" id="link-id160b3bf8">OpenLink Data Explorer</a> (which includes ubiquity commands).</p> <h3>Virtuoso Universal Server functionality summary:</h3> <ol> <li>Generation of RDF <a href="http://dbpedia.org/resource/Linked_Data" id="link-id161d5f50">Linked Data</a> Views of <a href="http://dbpedia.org/resource/SQL" id="link-id161d5978">SQL</a>, XML, and <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> Services in general </li> <li>Deployment of RDF Linked Data </li> <li>"On the Fly" generation of RDF Linked Data from Document Web <a href="http://www.w3.org/TR/webarch/" id="link-id178bbc08">information resources</a> (i.e. distillation of entities from their containers e.g. Web pages) via Cartridges / Drivers</li> <li> <a href="http://dbpedia.org/resource/SPARQL" id="link-id162c2118">SPARQL</a> query language support </li> <li>SPARQL extensions that bring SPARQL closer to SQL e.g Aggregates, Update, Insert, Delete Named Graph support (i.e. use of logical names to partition RDF data within Virtuoso's multi-model dbms engine) </li> <li>Inference Engine (currently in use re. <a href="http://dbpedia.org/resource/DBpedia" id="link-id14f563c0">DBpedia</a> via Yago and <a href="http://umbel.org/about/" id="link-id113273b8">UMBEL</a>)</li> <li>Host and exposes data from <a href="http://dbpedia.org/resource/Drupal" id="link-id123d3bd8">Drupal</a>, <a href="http://dbpedia.org/resource/WordPress" id="link-id141adf40">Wordpress</a>, <a href="http://dbpedia.org/resource/MediaWiki" id="link-id1604b450">MediaWiki</a>, <a href="http://dbpedia.org/resource/PhpBB" id="link-id141013a8">phpBB3</a> as RDF Linked Data via in-built support for <a href="http://dbpedia.org/resource/PHP" id="link-id14661e58">PHP</a> runtime</li> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/ODSInstallationEC2" id="link-id146c84d0">Available as an EC2 AMI</a> </li> <li>etc..</li> </ol> <h3>OpenLink Data Spaces functionality summary:</h3> <ol> <li>Simple mechanism for Linked Data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id15473770">Web</a> enabling yourself by giving you an <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/GetAPersonalURIIn5MinutesOrLess" id="link-id15f6d278">HTTP based User ID</a> (a de-referencable <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id15aaeb68">URI</a>) that is linked to a <a href="http://myopenlink.net/dataspace/person/kidehen" id="link-id15a7a840">FOAF based Profile page</a> and OpenID</li> <li>Binds all your data sources (blogs, wikis, bookmarks, photos, calendar items etc. ) to your URI so can "Find" things by only remembering your URI</li> <li>Makes your profile page and personal URI the focal point of Linked Data Web presence</li> <li>Delivers Data Portability (using data access by value or <a href="http://dbpedia.org/resource/Reference_(computer_science)" id="link-id16212838">data access by reference</a>) across data silos (e.g. Web 2.0 style social networks)</li> <li>Allows you make annotations about anything in your own <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id14668010">Data Space</a>(s) on the Web without exposure to RDF markup</li> <li>A Briefcase feature that provides a WebDAV driven RDF Linked Data variant of functionality seen in Mac OS X Spotlight and WinFS with the addition of SPARQL compliance</li> <li>Automatically generates <a href="http://dbpedia.org/resource/RDFa" id="link-id14691440">RDFa</a> in its (X)HTML pages</li> <li> <a href="http://dbpedia.org/resource/Blog" id="link-id14fae7b8">Blog</a>, Wiki, WebDAV File Server, Shared Bookmarks, Calendar, and other applications that look and feel like Web 2.0 counterparts but emitt RDF Linked Data amongst a plethora of data exchange formats</li> <li>Available as an EC2 AMI</li> <li>etc..</li> </ol> <h3>OpenLink Ajax Toolkit functionality summary:</h3> <ol> <li>Provides binding to SQL, RDF, XML, and Web Services via Ajax Database Connectivity Layer (you only need an <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id11550548">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id13ae5f68">JDBC</a>, OLE-DB, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id162803e8">ADO</a>.NET, XMLA Driver, or Web Service on the backend for dynamic data access from Javascript)</li> <li>All controls are Ajax Database Connectivity bound (widgets get their data from Ajax Database Connectivity data sources)</li> <li>Bundled with Virtuoso and <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id161dfe90">ODS</a> installations.</li> <li>etc.</li> </ol> <h3>OpenLink Data Explorer functionality summary</h3> <ol> <li>Distills entities associated with information resource style containers (e.g. Web Pages or files) as RDF Linked Data</li> <li>Exposes the RDF based <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id12a42ed8">Linked Data graph</a> associated with information resources (see the Linked Data behind Web pages)</li> <li>Ubiquity commands for invoking the above</li> <li>Available as a <a href="http://linkeddata.uriburner.com/ode" id="link-id15a0d2b0">Hosted Service</a> or <a href="http://ode.openlinksw.com" id="link-id138b9fa8">Firefox Extension</a> </li> <li>Bundled with Virtuoso and ODS installations</li> <li>etc.</li> </ol> <h3>Note:</h3> <p>Of course you could have simply looked up <a href="http://www.openlinksw.com/dataspace/organization/openlink" id="link-id14ef2c10">OpenLink Software's FOAF based Profile page</a> (*note the Linked Data Explorer tab*), or simply passed the <a href="http://dbpedia.org/resource/Friend_of_a_friend" id="link-id14cbf5c8">FOAF</a> profile page <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id16453e28">URL</a> to a Linked Data aware client application such as: <a href="http://linkeddata.uriburner.com/ode" id="link-id15a80500">OpenLink Data Explorer</a>, <a href="http://zitgist.com/about/" id="link-id1586a360">Zitgist</a> <a href="http://dataviewer.zitgist.com" id="link-id16249f60">Data Viewer</a>, <a href="http://beckr.org/marbles" id="link-id15993fb0">Marbles</a>, and <a href="http://dig.csail.mit.edu/2005/ajar/release/tabulator/0.8/tab.html" id="link-id14d63048">Tabulator</a>, and obtained information. Remember, <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id138ba838">OpenLink Software</a> is an <a href="http://dbpedia.org/resource/Entity" id="link-id1173e120">Entity</a> of Type: <a href="http://xmlns.com/foaf/0.1/Organization" id="link-id138b87b8">foaf:Organization</a>, on the burgeoning Linked Data Web :-)</p> <h3>Related</h3> <ul> <li> <a href="http://virtuoso.openlinksw.com/presentations/Creating_Deploying_Exploiting_Linked_Data2/Creating_Deploying_Exploiting_Linked_Data2_TimBL_v3.html" id="link-id163a0c88">Linked Data Planet Keynote</a> (RDFa based remix edition)</li> <li> <a href="http://semanticbusiness.blogspot.com/2008/09/report-on-cusp-global-review-of.html" id="link-id11471a40">On The Cusp: A Global Review of the Semantic Web Industry.</a> </li> </ul>
2008-10-02T15:27:41-04:00
OpenLink Software's Virtuoso Submission to the Billion Triples Challenge
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-09-30#1446
2008-09-30T16:24:34Z
<div> <h2>Introduction</h2> <p>We use <a href="http://virtuoso.openlinksw.com" id="link-id0xb03e418">Virtuoso</a> 6 Cluster Edition to demonstrate the following:</p> <ul> <li>Text and structured <a href="http://dbpedia.org/resource/Information" id="link-id0xbd9dae8">information</a> based lookups</li> <li>Analytics queries</li> <li>Analysis of co-occurrence of features like interests and tags.</li> <li>Dealing with identity of multiple IRI's (<a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0xb383dd8">owl</a>:sameAs)</li> </ul> <p>The demo is based on a set of canned <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xbda6298">SPARQL</a> queries that can be invoked using the <a href="http://ode.openlinksw.com/" id="link-id0xbb292f0">OpenLink Data Explorer</a> (<a href="http://ode.openlinksw.com/" id="link-id0xc263528">ODE</a>) Firefox extension.</p> <p>The demo queries can also be run directly against the SPARQL end point.</p> <p>The demo is being worked on at the time of submission and may be shown online by appointment.</p> <p>Automatic annotation of the <a href="http://dbpedia.org/resource/Data" id="link-id0xa173378">data</a> based on <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id0xbdda558">named entity extraction</a> is being worked on at the time of this submission. By the time of ISWC 2008 the set of sample queries will be enhanced with queries based on extracted <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id0xa66fbe0">named entities</a> and their relationships in the <a href="http://umbel.org/about/" id="link-id0xa06e2c8">UMBEL</a> and Open CYC ontologies. </p> <p>Also examples involving owl:sameAs are being added, likewise with similarity metrics and search hit scores.</p> <h2>The Data</h2> <p>The database consists of the billion triples data sets and some additions like Umbel. Also the Freebase extract is newer than the challenge original.</p> <p>The triple count is 1115 million.</p> <p>In the case of web harvested resources, the data is loaded in one graph per resource.</p> <p>In the case of larger data sets like <a href="http://dbpedia.org/resource/DBpedia" id="link-id0xc2bf770">Dbpedia</a> or the US census, all triples of the provenance share a data set specific graph.</p> <p>All string literals are additionally indexed in a full text index. No stop words are used.</p> <p>Most queries do not specify a graph. Thus they are evaluated against the union of all the graphs in the database. The indexing scheme is SPOG, GPOS, POGS, OPGS. All indices ending in S are bitmap indices. </p> <h2>The Queries </h2> <p>The demo uses Virtuoso SPARQL extensions in most queries. These extensions consist on one hand of well known <a href="http://dbpedia.org/resource/SQL" id="link-id0xaf8cb40">SQL</a> features like aggregation with grouping and existence and value subqueries and on the other of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xafdceb8">RDF</a> specific features. The latter include run time RDFS and OWL inferencing support and backward chaining subclasses and transitivity. </p> <h3>Simple Lookups</h3> <pre>sparql select ?s ?p (bif:search_excerpt (bif:vector ('<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0xbb64dd0">semantic', 'web</a>'), ?o)) where { ?s ?p ?o . filter (bif:contains (?o, "'semantic web'")) } limit 10 ; </pre> <p>This looks up triples with semantic web in the object and makes a search hit summary of the literal, highlighting the search terms. </p> <pre>sparql select ?tp count(*) where { ?s ?p2 ?o2 . ?o2 a ?tp . ?s foaf:nick ?o . filter (bif:contains (?o, "plaid_skirt")) } group by ?tp order by desc 2 limit 40 ; </pre> <p>This looks at what sorts of things are referenced by the properties of the foaf handle plaid_skirt.</p> <p>What are these things called?</p> <pre>sparql select ?lbl count(*) where { ?s ?p2 ?o2 . ?o2 rdfs:label ?lbl . ?s foaf:nick ?o . filter (bif:contains (?o, "plaid_skirt")) } group by ?lbl order by desc 2 ; </pre> <p>Many of these things do not have a rdfs:label. Let us use a more general concept of lable which groups dc:title, foaf:name and other name-like properties together. The subproperties are resolved at run time, there is no materialization. </p> <pre>sparql define input:inference 'b3s' select ?lbl count(*) where { ?s ?p2 ?o2 . ?o2 b3s:label ?lbl . ?s foaf:nick ?o . filter (bif:contains (?o, "plaid_skirt")) } group by ?lbl order by desc 2 ; </pre> <p>We can list sources by the topics they contain. Below we look for graphs that mention terrorist bombing. </p> <pre>sparql select ?g count(*) where { graph ?g { ?s ?p ?o . filter (bif:contains (?o, "'terrorist bombing'")) } } group by ?g order by desc 2 ; </pre> <p>Now some web 2.0 tagging of search results. The <a href="http://dbpedia.org/resource/Tag" id="link-id0xa8b89f8">tag</a> cloud of "computer"</p> <pre>sparql select ?lbl count (*) where { ?s ?p ?o . ?o bif:contains "computer" . ?s sioc:topic ?tg . optional { ?tg rdfs:label ?lbl } } group by ?lbl order by desc 2 limit 40 ; </pre> <p>This query will find the posters who talk the most about sex.</p> <pre>sparql select ?auth count (*) where { ?d dc:creator ?auth . ?d ?p ?o filter (bif:contains (?o, "sex")) } group by ?auth order by desc 2 ; </pre> <h3>Analytics </h3> <p>We look for people who are joined by having relatively uncommon interests but do not know each other.</p> <pre>sparql select ?i ?cnt ?n1 ?n2 ?p1 ?p2 where { { select ?i count (*) as ?cnt where { ?p foaf:interest ?i } group by ?i } filter ( ?cnt > 1 && ?cnt < 10) . ?p1 foaf:interest ?i . ?p2 foaf:interest ?i . filter (?p1 != ?p2 && !bif:exists ((select (1) where {?p1 foaf:knows ?p2 })) && !bif:exists ((select (1) where {?p2 foaf:knows ?p1 }))) . ?p1 foaf:nick ?n1 . ?p2 foaf:nick ?n2 . } order by ?cnt limit 50 ; </pre> <p>The query takes a fairly long time, mostly spent counting the interested in 25M interest triples. It then takes people that share the interest and checks that neither claims to know the other. It then sorts the results rarest interest first. The query can be written more efficently but is here just to show that database-wide scans of the population are possible ad hoc. </p> <p>Now we go to SQL to make a tag co-occurrence matrix. This can be used for showing a Technorati-style related tags line at the bottom of a search result page. This showcases the use of SQL together with SPARQL. The half-matrix of tags t1, t2 with the co-occurrence count at the intersection is much more efficiently done in SQL, specially since it gets updated as the data changes. This is an example of materialized intermediate results based on warehoused RDF. </p> <pre>create table tag_count (tcn_tag iri_id_8, tcn_count int, primary key (tcn_tag)); alter index tag_count on tag_count partition (tcn_tag int (0hexffff00)); create table tag_coincidence (tc_t1 iri_id_8, tc_t2 iri_id_8, tc_count int, tc_t1_count int, tc_t2_count int, primary key (tc_t1, tc_t2)) alter index tag_coincidence on tag_coincidence partition (tc_t1 int (0hexffff00)); create index tc2 on tag_coincidence (tc_t2, tc_t1) partition (tc_t2 int (0hexffff00)); </pre> <p>How many times each topic is mentioned?</p> <pre> insert into tag_count select * from (sparql define output:valmode "LONG" select ?t count (*) as ?cnt where { ?s sioc:topic ?t } group by ?t) xx option (quietcast); </pre> <p>Take all t1, t2 where t1 and t2 are tags of the same subject, store only the permutation where the internal id of t1 < that of t2.</p> <pre>insert into tag_coincidence (tc_t1, tc_t2, tc_count) select "t1", "t2", cnt from (select "t1", "t2", count (*) as cnt from (sparql define output:valmode "LONG" select ?t1 ?t2 where { ?s sioc:topic ?t1 . ?s sioc:topic ?t2 }) tags where "t1" < "t2" group by "t1", "t2") xx where isiri_id ("t1") and isiri_id ("t2") option (quietcast); </pre> <p>Now put the individual occurrence counts into the same table with the co-occurrence. This denormalization makes the related tags lookup faster. </p> <pre>update tag_coincidence set tc_t1_count = (select tcn_count from tag_count where tcn_tag = tc_t1), tc_t2_count = (select tcn_count from tag_count where tcn_tag = tc_t2); </pre> <p>Now each tag_coincidence row has the joint occurrence count and individual occurrence counts. A single select will return a Technorati-style related tags listing. </p> <p>To show the <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x9d4bc60">URI</a>'s of the tags: </p> <pre>select top 10 id_to_iri (tc_T1), id_to_iri (tc_t2), tc_count from tag_coincidence order by tc_count desc; </pre> <h3>Social Networks </h3> <p>We look at what interests people have </p> <pre>sparql select ?o ?cnt where { { select ?o count (*) as ?cnt where { ?s foaf:interest ?o } group by ?o } filter (?cnt > 100) } order by desc 2 limit 100 ; </pre> <p>Now the same for the Harry Potter fans </p> <pre>sparql select ?i2 count (*) where { ?p foaf:interest <<a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0xba0b390">http</a>://www.livejournal.com/interests.bml?int=harry+potter> . ?p foaf:interest ?i2 } group by ?i2 order by desc 2 limit 20 ; </pre> <p>We see whether knows relations are symmmetrical. We return the top n people that others claim to know without being reciprocally known.</p> <pre>sparql select ?celeb, count (*) where { ?claimant foaf:knows ?celeb . filter (!bif:exists ((select (1) where { ?celeb foaf:knows ?claimant }))) } group by ?celeb order by desc 2 limit 10 ; </pre> <p>We look for a well connected person to start from.</p> <pre>sparql select ?p count (*) where { ?p foaf:knows ?k } group by ?p order by desc 2 limit 50 ; </pre> <p>We look for the most connected of the many online identities of Stefan Decker.</p> <pre>sparql select ?sd count (distinct ?xx) where { ?sd a foaf:Person . ?sd ?name ?ns . filter (bif:contains (?ns, "'Stefan Decker'")) . ?sd foaf:knows ?xx } group by ?sd order by desc 2 ; </pre> <p>We count the transitive closure of Stefan Decker's connections </p> <pre>sparql select count (*) where { { select * where { ?s foaf:knows ?o } } option (transitive, t_distinct, t_in(?s), t_out(?o)) . filter (?s = <mailto:stefan.decker@deri.org>) } ; </pre> <p>Now we do the same while following owl:sameAs links.</p> <pre>sparql define input:same-as "yes" select count (*) where { { select * where { ?s foaf:knows ?o } } option (transitive, t_distinct, t_in(?s), t_out(?o)) . filter (?s = <mailto:stefan.decker@deri.org>) } ; </pre> <h2>Demo System</h2> <p>The system runs on Virtuoso 6 Cluster Edition. The database is partitioned into 12 partitions, each served by a distinct server process. The system demonstrated hosts these 12 servers on 2 machines, each with 2 xXeon 5345 and 16GB memory and 4 SATA disks. For scaling, the processes and corresponding partitions can be spread over a larger number of machines. If each ran on its own server with 16GB RAM, the whole data set could be served from memory. This is desirable for search engine or fast analytics applications. Most of the demonstrated queries run in memory on second invocation. The timing difference between first and second run is easily an order of magnitude. </p> </div>
2008-10-03T06:20:48.000094-04:00
OpenLink Software's Virtuoso Submission to the Billion Triples Challenge
http://www.openlinksw.com:443/weblog/oerling/?date=2008-09-30#1445
2008-09-30T15:39:26Z
<div> <h2>Introduction</h2> <p>We use <a href="http://virtuoso.openlinksw.com" id="link-id0xa278560">Virtuoso</a> 6 Cluster Edition to demonstrate the following:</p> <ul> <li>Text and structured <a href="http://dbpedia.org/resource/Information" id="link-id0xb3a4490">information</a> based lookups</li> <li>Analytics queries</li> <li>Analysis of co-occurrence of features like interests and tags.</li> <li>Dealing with identity of multiple IRI's (<a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0xa904bd8">owl</a>:sameAs)</li> </ul> <p>The demo is based on a set of canned <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xac185d0">SPARQL</a> queries that can be invoked using the <a href="http://ode.openlinksw.com/" id="link-id0xb8efe28">OpenLink Data Explorer</a> (<a href="http://ode.openlinksw.com/" id="link-id0xb341808">ODE</a>) Firefox extension.</p> <p>The demo queries can also be run directly against the SPARQL end point.</p> <p>The demo is being worked on at the time of submission and may be shown online by appointment.</p> <p>Automatic annotation of the <a href="http://dbpedia.org/resource/Data" id="link-id0xa2fcc88">data</a> based on <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id0xc085440">named entity extraction</a> is being worked on at the time of this submission. By the time of ISWC 2008 the set of sample queries will be enhanced with queries based on extracted <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id0xa92b3e0">named entities</a> and their relationships in the <a href="http://umbel.org/about/" id="link-id0xa1c7c38">UMBEL</a> and Open CYC ontologies. </p> <p>Also examples involving owl:sameAs are being added, likewise with similarity metrics and search hit scores.</p> <h2>The Data</h2> <p>The database consists of the billion triples data sets and some additions like Umbel. Also the Freebase extract is newer than the challenge original.</p> <p>The triple count is 1115 million.</p> <p>In the case of web harvested resources, the data is loaded in one graph per resource.</p> <p>In the case of larger data sets like <a href="http://dbpedia.org/resource/DBpedia" id="link-id0xa949850">Dbpedia</a> or the US census, all triples of the provenance share a data set specific graph.</p> <p>All string literals are additionally indexed in a full text index. No stop words are used.</p> <p>Most queries do not specify a graph. Thus they are evaluated against the union of all the graphs in the database. The indexing scheme is SPOG, GPOS, POGS, OPGS. All indices ending in S are bitmap indices. </p> <h2>The Queries </h2> <p>The demo uses Virtuoso SPARQL extensions in most queries. These extensions consist on one hand of well known <a href="http://dbpedia.org/resource/SQL" id="link-id0xc116190">SQL</a> features like aggregation with grouping and existence and value subqueries and on the other of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xa9047f0">RDF</a> specific features. The latter include run time RDFS and OWL inferencing support and backward chaining subclasses and transitivity. </p> <h3>Simple Lookups</h3> <pre>sparql select ?s ?p (bif:search_excerpt (bif:vector ('<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0xbe38070">semantic', 'web</a>'), ?o)) where { ?s ?p ?o . filter (bif:contains (?o, "'semantic web'")) } limit 10 ; </pre> <p>This looks up triples with semantic web in the object and makes a search hit summary of the literal, highlighting the search terms. </p> <pre>sparql select ?tp count(*) where { ?s ?p2 ?o2 . ?o2 a ?tp . ?s foaf:nick ?o . filter (bif:contains (?o, "plaid_skirt")) } group by ?tp order by desc 2 limit 40 ; </pre> <p>This looks at what sorts of things are referenced by the properties of the foaf handle plaid_skirt.</p> <p>What are these things called?</p> <pre>sparql select ?lbl count(*) where { ?s ?p2 ?o2 . ?o2 rdfs:label ?lbl . ?s foaf:nick ?o . filter (bif:contains (?o, "plaid_skirt")) } group by ?lbl order by desc 2 ; </pre> <p>Many of these things do not have a rdfs:label. Let us use a more general concept of lable which groups dc:title, foaf:name and other name-like properties together. The subproperties are resolved at run time, there is no materialization. </p> <pre>sparql define input:inference 'b3s' select ?lbl count(*) where { ?s ?p2 ?o2 . ?o2 b3s:label ?lbl . ?s foaf:nick ?o . filter (bif:contains (?o, "plaid_skirt")) } group by ?lbl order by desc 2 ; </pre> <p>We can list sources by the topics they contain. Below we look for graphs that mention terrorist bombing. </p> <pre>sparql select ?g count(*) where { graph ?g { ?s ?p ?o . filter (bif:contains (?o, "'terrorist bombing'")) } } group by ?g order by desc 2 ; </pre> <p>Now some web 2.0 tagging of search results. The <a href="http://dbpedia.org/resource/Tag" id="link-id0xa366510">tag</a> cloud of "computer"</p> <pre>sparql select ?lbl count (*) where { ?s ?p ?o . ?o bif:contains "computer" . ?s sioc:topic ?tg . optional { ?tg rdfs:label ?lbl } } group by ?lbl order by desc 2 limit 40 ; </pre> <p>This query will find the posters who talk the most about sex.</p> <pre>sparql select ?auth count (*) where { ?d dc:creator ?auth . ?d ?p ?o filter (bif:contains (?o, "sex")) } group by ?auth order by desc 2 ; </pre> <h3>Analytics </h3> <p>We look for people who are joined by having relatively uncommon interests but do not know each other.</p> <pre>sparql select ?i ?cnt ?n1 ?n2 ?p1 ?p2 where { { select ?i count (*) as ?cnt where { ?p foaf:interest ?i } group by ?i } filter ( ?cnt > 1 && ?cnt < 10) . ?p1 foaf:interest ?i . ?p2 foaf:interest ?i . filter (?p1 != ?p2 && !bif:exists ((select (1) where {?p1 foaf:knows ?p2 })) && !bif:exists ((select (1) where {?p2 foaf:knows ?p1 }))) . ?p1 foaf:nick ?n1 . ?p2 foaf:nick ?n2 . } order by ?cnt limit 50 ; </pre> <p>The query takes a fairly long time, mostly spent counting the interested in 25M interest triples. It then takes people that share the interest and checks that neither claims to know the other. It then sorts the results rarest interest first. The query can be written more efficently but is here just to show that database-wide scans of the population are possible ad hoc. </p> <p>Now we go to SQL to make a tag co-occurrence matrix. This can be used for showing a Technorati-style related tags line at the bottom of a search result page. This showcases the use of SQL together with SPARQL. The half-matrix of tags t1, t2 with the co-occurrence count at the intersection is much more efficiently done in SQL, specially since it gets updated as the data changes. This is an example of materialized intermediate results based on warehoused RDF. </p> <pre>create table tag_count (tcn_tag iri_id_8, tcn_count int, primary key (tcn_tag)); alter index tag_count on tag_count partition (tcn_tag int (0hexffff00)); create table tag_coincidence (tc_t1 iri_id_8, tc_t2 iri_id_8, tc_count int, tc_t1_count int, tc_t2_count int, primary key (tc_t1, tc_t2)) alter index tag_coincidence on tag_coincidence partition (tc_t1 int (0hexffff00)); create index tc2 on tag_coincidence (tc_t2, tc_t1) partition (tc_t2 int (0hexffff00)); </pre> <p>How many times each topic is mentioned?</p> <pre> insert into tag_count select * from (sparql define output:valmode "LONG" select ?t count (*) as ?cnt where { ?s sioc:topic ?t } group by ?t) xx option (quietcast); </pre> <p>Take all t1, t2 where t1 and t2 are tags of the same subject, store only the permutation where the internal id of t1 < that of t2.</p> <pre>insert into tag_coincidence (tc_t1, tc_t2, tc_count) select "t1", "t2", cnt from (select "t1", "t2", count (*) as cnt from (sparql define output:valmode "LONG" select ?t1 ?t2 where { ?s sioc:topic ?t1 . ?s sioc:topic ?t2 }) tags where "t1" < "t2" group by "t1", "t2") xx where isiri_id ("t1") and isiri_id ("t2") option (quietcast); </pre> <p>Now put the individual occurrence counts into the same table with the co-occurrence. This denormalization makes the related tags lookup faster. </p> <pre>update tag_coincidence set tc_t1_count = (select tcn_count from tag_count where tcn_tag = tc_t1), tc_t2_count = (select tcn_count from tag_count where tcn_tag = tc_t2); </pre> <p>Now each tag_coincidence row has the joint occurrence count and individual occurrence counts. A single select will return a Technorati-style related tags listing. </p> <p>To show the <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0xaf355c8">URI</a>'s of the tags: </p> <pre>select top 10 id_to_iri (tc_T1), id_to_iri (tc_t2), tc_count from tag_coincidence order by tc_count desc; </pre> <h3>Social Networks </h3> <p>We look at what interests people have </p> <pre>sparql select ?o ?cnt where { { select ?o count (*) as ?cnt where { ?s foaf:interest ?o } group by ?o } filter (?cnt > 100) } order by desc 2 limit 100 ; </pre> <p>Now the same for the Harry Potter fans </p> <pre>sparql select ?i2 count (*) where { ?p foaf:interest <<a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0xa274410">http</a>://www.livejournal.com/interests.bml?int=harry+potter> . ?p foaf:interest ?i2 } group by ?i2 order by desc 2 limit 20 ; </pre> <p>We see whether knows relations are symmmetrical. We return the top n people that others claim to know without being reciprocally known.</p> <pre>sparql select ?celeb, count (*) where { ?claimant foaf:knows ?celeb . filter (!bif:exists ((select (1) where { ?celeb foaf:knows ?claimant }))) } group by ?celeb order by desc 2 limit 10 ; </pre> <p>We look for a well connected person to start from.</p> <pre>sparql select ?p count (*) where { ?p foaf:knows ?k } group by ?p order by desc 2 limit 50 ; </pre> <p>We look for the most connected of the many online identities of Stefan Decker.</p> <pre>sparql select ?sd count (distinct ?xx) where { ?sd a foaf:Person . ?sd ?name ?ns . filter (bif:contains (?ns, "'Stefan Decker'")) . ?sd foaf:knows ?xx } group by ?sd order by desc 2 ; </pre> <p>We count the transitive closure of Stefan Decker's connections </p> <pre>sparql select count (*) where { { select * where { ?s foaf:knows ?o } } option (transitive, t_distinct, t_in(?s), t_out(?o)) . filter (?s = <mailto:stefan.decker@deri.org>) } ; </pre> <p>Now we do the same while following owl:sameAs links.</p> <pre>sparql define input:same-as "yes" select count (*) where { { select * where { ?s foaf:knows ?o } } option (transitive, t_distinct, t_in(?s), t_out(?o)) . filter (?s = <mailto:stefan.decker@deri.org>) } ; </pre> <h2>Demo System</h2> <p>The system runs on Virtuoso 6 Cluster Edition. The database is partitioned into 12 partitions, each served by a distinct server process. The system demonstrated hosts these 12 servers on 2 machines, each with 2 xXeon 5345 and 16GB memory and 4 SATA disks. For scaling, the processes and corresponding partitions can be spread over a larger number of machines. If each ran on its own server with 16GB RAM, the whole data set could be served from memory. This is desirable for search engine or fast analytics applications. Most of the demonstrated queries run in memory on second invocation. The timing difference between first and second run is easily an order of magnitude. </p> </div>
2008-10-03T06:20:43.000002-04:00
Semantic Web: Travails to Harmony Illustrated (Updated)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-09-27#1444
2008-09-27T19:14:48Z
<h3>All about <a href="http://dbpedia.org/resource/Data">Data</a> Dictionary issues</h3> <p>Over emphasis on <a href="http://dbpedia.org/resource/Data_dictionary" id="link-id10e99460"><a href="http://dbpedia.org/resource/Description_logic" id="link-id0xa2800c0">Description Logics</a></a> (RDFS, OWL, Inference & Reasoning etc) matters without any actual real-world instance <a href="http://dbpedia.org/resource/Data" id="link-id0x9d3a838">data</a> (e.g., lot's of reasoning over RDF in zip files or local drives).</p> <img alt="Image" src="http://virtuoso.openlinksw.com/ldp_presentation/images/linked_data_people_schems_venn2_missing_lod.png" /> <h3>All about Linking Openly accessible RDF Data Sets</h3> <p>Over emphasis on Instance Data without Data Dictionary appreciation and utilization (e.g., <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10ea0728">Linked Data</a> instance level linkage via "<a href="http://www.w3.org/TR/owl-ref/#sameAs-def" id="link-id10f2f650">owl:sameAs</a>"). </p> <img alt="Image" src="http://virtuoso.openlinksw.com/ldp_presentation/images/linked_data_people_schems_venn2_missing_dict.png" /> <h3>All about Applications & Frameworks</h3> <p>Here we are dealing with numerous applications and frameworks that inextricably bind Instance Data Management and Data Dictionaries. Basically, an all or nothing proposition, if you want to delve into the RDF <a href="http://dbpedia.org/resource/Linked_Data" id="link-id110b4970">Linked Data</a> solutions realm.</p> <img alt="Image" src="http://virtuoso.openlinksw.com/ldp_presentation/images/linked_data_people_schems_venn2_missing_modularity.png" /> <p>Often overlooked, is the fact that the <a href="http://dbpedia.org/resource/Linked_Data" id="link-ide398d40">Linked Data Web</a> - as an aspect of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id19653440">Semantic Web</a> innovation continuum - is fundamentally about designing and constructing an "<a href="http://dbpedia.org/resource/Open_world_assumption" id="link-id19cac3a0">Open World</a>" compatible DBMS for the <a href="http://dbpedia.org/resource/Internet" id="link-id127fd198">Internet</a>. Thus, erstwhile "<a href="http://dbpedia.org/resource/Closed_world_assumption" id="link-id1252b338">Closed World</a>" DBMS components such as Data Dictionaries (handlers of Data Definition, Referential Integrity etc.) and actual Instance Data, are now distributed and loosely coupled. Thus, your data could be in one <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id12bf6338">Data Space</a> while the data dictionary resides in another. In actual fact, you could have several loosely bound data dictionaries that serve the specific Inference and Reasoning needs of a variety of applications, services, or agents. </p> <img alt="Image" src="http://virtuoso.openlinksw.com/ldp_presentation/images/linked_data_people_schems_venn2.png" />
2008-09-28T15:18:53-04:00
The Linked Data Market via a BCG Matrix (Updated)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-09-25#1442
2008-09-25T20:42:49Z
<p>The sweet spot of <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> 3.0 (or any other Web.vNext moniker) is all about providing Web Users with a structured and interlinked <a href="http://dbpedia.org/resource/Data">data</a> substrate that facilitates serendipitous discovery of relevant "Things" i.e., a <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10db3b48">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id170db618">Web</a> -- a Web of Linkable Entities that goes beyond documents and other <a href="http://dbpedia.org/resource/Information" id="link-id110a5d30">information</a> resource (data containers) types.</p> <p>Understanding potential <a href="http://dbpedia.org/resource/Linked_Data" id="link-id19e21c60">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id16d008d0">Web</a> business models, relative to other Web based market segments, is best pursued via a<a href="http://dbpedia.org/resource/BCG_diagram" id="link-id14734148"> BCG Matrix</a> diagram, such as the one I've constructed below:</p> <br /> <img alt="Image" src="http://virtuoso.openlinksw.com/ldp_presentation/images/linked_data_sdq_quadarant.png" /> <br /> <h3>Notes:</h3> <h4>Link Density</h4> <ul> <li>Web 1.0's collection of "Web Sites" have relatively low link density relative to Web 2.0's user-activity driven generation of semi-structured <a href="http://dbpedia.org/resource/Linked_Data" id="link-id14c302d8">linked data</a> spaces (e.g., Blogs, Wikis, Shared Bookmarks, RSS/Atom Feeds, Photo Galleries, Discussion Forums etc..)</li> <li>Semantic Technologies (i.e. "<strong>Semantics Inside</strong> style solutions") which are primarily about "Semantic Meaning" culled from Web 1.0 Pages also have limited linked density relative to Web 2.0</li> <li>The <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1286ab58">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-ide81ab20">Web</a>, courtesy of the open-ended linking capacity of URIs, matches and ultimately exceeds Web 2.0 link density.</li> </ul> <h4>Relevance</h4> <ul> <li>Web 1.0 and 2.0 are low relevance realms driven by hyperlinks to <a href="http://dbpedia.org/resource/Information" id="link-id173db890">information</a> resources ((X)HTML, RSS, Atom, OPML, XML, Images, Audio files etc.) associated with Literal Labels and Tagging schemes devoid of explicit property based resource description thereby making the pursuit of relevance mercurial at best</li> <li>Semantic Technologies offer more relevance than Web 1.0 and 2.0 based on the increased <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id124de510">context</a> that semantic analysis of Web pages accords</li> <li>The <a href="http://dbpedia.org/resource/Linked_Data" id="link-id111c4850">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id16e4e4c0">Web</a>, courtesy of URIs that expose self-describing data entities, match the relevance levels attained by Semantic Technologies.</li> </ul> <h4>Serendipity Quotient (SDQ)</h4> <ul> <li>Web 1.0 has next to no serendipity, the closest thing is <a href="http://google.com" id="link-id16dceec8">Google's "I'm Feeling Lucky" button</a> </li> <li>Web 2.0 possess higher potential for serendipitous discovery than Web 1.0, but such potential is neutralized by inherent subjectivity due to its human-interaction-focused literal foundation (e.g., tags, voting schemes, wiki editors etc.)</li> <li>Semantic Technologies produce islands-of-relevance with little scope for serendipitous discovery due to <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id18078e60">URI</a> invisibility, since the prime focus is delivering more <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id1253cc38">context</a> to Web search relative to traditional Web 1.0 search engines.</li> <li>The <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x201d0ae8">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10c7fb70">Web</a>'s use of URIs as the naming and resolution mechanism for exposing structured and interlinked resources provides the highest potential for serendipitous discovery of relevant "Things"</li> </ul> <p>To conclude, the <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x23ebbf90">Web</a>'s market opportunities are all about the evolution of the Web into a powerful substrate that offers a unique intersection of "Link Density" and "Relevance", exploitable across horizontal and vertical market segments to solutions providers. Put differently, SDQ is how you take "The Ad" out of "Advertising" when matching Web users to relevant things :-)</p>
2008-09-26T12:36:56-04:00
View Plurality Deficiency & Programming Language Autism
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-09-17#1441
2008-09-17T14:38:20Z
<p>I've just read a really nice post by <a href="http://demo.openlinksw.com/about/html/http://bblfish.net/people/henry/card%23me" id="link-idea954a0">Henry Story</a> titled: <a href="http://blogs.sun.com/bblfish/entry/are_oo_languages_autistic" id="link-id110164a8">Are OO Languages Autistic?</a> </p> <p>In typical style, Henry walks you through his point of view using simple but powerful illustrations. Here is a key statement in his post that really struck <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id10f82150">me</a>:</p> <blockquote> <cite>"In order to be able to have a mental theory one needs to be able to <strong>understand that other people may have a different view of the world</strong>. On a narrow three dimensional understanding of 'view', this reveals itself in that people at different locations in a room will see different things. One person may be able to see a cat behind a tree that will be hidden to another. In some sense though these two views can easily be merged into a coherent description."</cite> </blockquote> <p>Opaque <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> pages (e.g., generated by Semantic Technology inside offerings that will not expose or share <a href="http://dbpedia.org/resource/Data">data</a> <a href="http://dbpedia.org/resource/Entity" id="link-id10f81420">entity</a> URIs), irrespective of how smart the underlying page generation and visualization technology may be, a fundamentally autistic and counter intuitive as we move toward a Web of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10cc3d80">Linked Data</a>.</p> <p>Preoccupation with the "V" aspect of the <a href="http://dbpedia.org/resource/Model-view-controller" id="link-id10fa86b0">M-V-C</a> trinity is inadvertently compounding and the problem of digital autism on the Web. Unbeknownst to the purveyors of data silos and proprietary service lock-in, digital autism on the Web ultimately implies Web business model autism.</p>
2008-09-17T10:54:48.000004-04:00
Zero-based Cognition (Difference between Humans & Machines)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-09-17#1440
2008-09-17T13:43:21Z
<p>Human beings, courtesy of the gift of <a href="http://dbpedia.org/resource/Cognition" id="link-id10f3d278">cognition</a>, are capable of creating reusable <a href="http://dbpedia.org/resource/Data">data</a>, <a href="http://dbpedia.org/resource/Information" id="link-id10f76078">information</a>, <a href="http://dbpedia.org/resource/Knowledge" id="link-id10c10518">knowledge</a> from simple or complex observations in an abstract realm. A machine on the other hand can only discover and infere based on a substrate of structured and interlinked data, <a href="http://dbpedia.org/resource/Information" id="link-id0xdda2b88">information</a>, or <a href="http://dbpedia.org/resource/Knowledge" id="link-id0xcf58b18">knowledge</a> in a concrete human created realm e.g., a <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10f3d640">Linked Data</a>.</p> As is quite common these days, <a href="http://yihongs-research.blogspot.com/" id="link-id10ecf268">Yihong Ding</a> has written another great piece titled: <a href="http://www.internetevolution.com/author.asp?section_id=542&doc_id=163916l" id="link-id10fb6558">A New Take on Internet-Based AI</a>, that delves into this specific matter. Yihong expresses an vital insight as excerpted below: <blockquote> <cite>"Artificial intelligence is supposed to let machines do things for people. The risk is that we may rely too much on them. Two months ago, for instance, writer Nicolas Carr asked whether <a href="http://www.theatlantic.com/doc/200807/google" id="link-id115e9bb8">Google is making us stupid</a>. In my recent <a href="http://dbpedia.org/resource/Blog" id="link-id10d9ceb0">blog</a> series "<a href="http://altsearchengines.com/2008/08/22/the-age-of-google-part-1-of-4/" id="link-id10c8de60">The Age of Google</a>," I extended Carr’s discussion. Due to the success of Google, we are relying more on objective search than on active thinking to answer questions. In consequence, the more Google has advanced its service, the farther Google users have drifted from active thinking</cite>."</blockquote> <blockquote> <cite>"But at least one form of human thinking cannot be replaced by machines. I am not talking about inference/discovery (which machines may be capable of doing) but about creation/generation-from-nothing (which I don’t believe machines may ever do)."</cite> </blockquote> <p>I tend to describe our ability to create/generate-from-nothing as "Zero-based Cognition", which is initially about "thought" and the eventually about "speed of thought dissemination" and "global thought meshing". </p> <p>In a peculiar sense, Zero-based cognition is analogous to <a href="http://dbpedia.org/resource/Zero_Based_Budgeting" id="link-id10ccbf78">Zero-based budgeting</a> from the accounting realm :-)</p>
2008-10-17T07:23:42.000003-04:00
Web 1.0, 2.0, and 3.0 (Yet Again)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-09-15#1439
2008-09-15T17:33:44Z
<p>If your <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> presence doesn't extend beyond (X)HTML web pages, you are only participating in Web usage Dimension 1.0.</p> <p>If your Web presence goes beyond (X)HTML pages, via the addition of REST or SOAP based Web Services, then you re participating in Web usage dimension 2.0.</p> <p>If you Web presence includes all of the above, with the addition of structured <a href="http://dbpedia.org/resource/Data">data</a> interlinked with structured data across other points of presence on the Web, then you are participating in Web usage dimension 3.0 i.e., "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id14d48d30">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id14d47280">Web</a>" or "Web of Data" or "Data Web".</p> <p>BTW - If you've already done all of the above, and you have started building intelligent agents that exploit the aforementioned structured interlinked data substrate, then you are already in Web usage dimension 4.0.</p> <h3>Related</h3> <ul>Prior posts about <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=web%20evolution&type=text&output=html" id="link-id10e8b978">Web usage pattern evolution</a> </ul>
2008-09-15T13:48:15-04:00
The Trouble with Labels
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-09-12#1438
2008-09-12T01:47:05Z
<p>Unfortunately our fixation with "Labels" and the artificial link that exist between "Labels" and so-called "first mover advantage" continue to impede our progress to clarity about matters such as a fully functional <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> of interlinked <a href="http://dbpedia.org/resource/Data">data</a>.</p> <p> A while back I watched <a href="http://www.ted.com/index.php/talks/kevin_kelly_on_the_next_5_000_days_of_the_web.html" id="link-id14c2c740">Kevin Kelly's 5,000 days presentation at TED</a>. During the presentation, I kept on scratching my head, wondering why phrases like "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id0xb154550">Linked Data</a>", "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0xb5927b8">Semantic Web</a>", "Web of Data", "Data Web" where so unnaturally disconnected from his session narrative.</p> <p>Yesterday I watched <a href="http://www.techcrunch50.com/2008/conference/presenter.php?presenter=63#video" id="link-id14f6e1a8">IMINDI's TechCrunch 50 presentation</a>, and once again I saw the aforementioned pattern repeat itself. This time around, the poor founders of this "<a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0xae767f0">Web</a>" oriented company (which is what they are in reality) took a totally undeserved pasting from a bunch of panelist incapable of seeing beyond today (Web 2.0) and yesterday (initial Web bootstrap).</p> <p>Anyway, thanks to the Web, this post will make a small contribution towards re-connecting the missing phrases to these "Linked Data Web" presentations.</p>
2008-09-16T10:07:49.000015-04:00
Business Value of Linked Data (Enterprise Angle)?
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-09-11#1437
2008-09-11T18:59:24Z
<p>All enterprises run IS/MIS/EIS systems that are supposed to enable optimized exploitation of <a href="http://dbpedia.org/resource/Data">data</a>, <a href="http://dbpedia.org/resource/Information" id="link-id1408bee8">information</a>, and <a href="http://dbpedia.org/resource/Knowledge" id="link-id14c429a8">knowledge</a>. Unfortunately, applications, services (SOAP or REST), database engines, middleware, operating systems, programming languages, development frameworks, network protocols, network topologies, or some other piece of infrastructure, eventually lay claim (possessively) to the data.</p> <p>Courtesy of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10f98db8">Linked Data</a>, we are now able to extend the "document to document" linking mechanism of the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> (Hypertext Linking) to more granular "<a href="http://dbpedia.org/resource/Entity" id="link-id14410810">entity</a> to <a href="http://dbpedia.org/resource/Entity" id="link-id10dbb420">entity</a>" level linking. And in doing so, we have a layer of abstraction that in one swoop alleviates all of the infrastructure oriented data access impediments of yore. I know this sounds simplistic, but be rest assured, imbibing <a href="http://dbpedia.org/resource/Linked_Data" id="link-id14b6af20">Linked Data</a>'s value proposition is really just that simple, once you engage solutions (e.g. <a href="http://virtuoso.openlinksw.com" id="link-id14ce6a20">Virtuoso</a>) that enable you to deploy <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1151c718">Linked Data</a> across your enterprise.</p> <h3>Example: </h3> <p>Microsoft ACCESS, <a href="http://dbpedia.org/resource/SQL" id="link-id14ef3b08">SQL</a> Server, and <a href="http://virtuoso.openlinksw.com" id="link-id10d865b8">Virtuoso</a> all use the Northwind <a href="http://dbpedia.org/resource/SQL" id="link-id10b04250">SQL</a> DB Schema as the basis of the demonstration database shipped with each DBMS product. This schema is comprised of common IS/MIS entities that include: Customers, Contacts, Orders, Products, Employees etc.</p> <p>What we all really want to do as data, <a href="http://dbpedia.org/resource/Information" id="link-id110dd7a0">information</a>, and <a href="http://dbpedia.org/resource/Knowledge" id="link-id11484408">knowledge</a> consumers and/or dispatchers, is be no more than a single "mouse click" away from relevant data/<a href="http://dbpedia.org/resource/Information" id="link-id10c755c8">information</a>/<a href="http://dbpedia.org/resource/Knowledge" id="link-id1464ac88">knowledge</a> data access and/or exploration. Even better (but not always so obvious), we also want anyone in our network (company, division, department, cube-cluster) to inherit these data access efficiencies.</p> <p>In this example, the <a href="http://demo.openlinksw.com/Northwind/Customer/ALFKI" id="link-id14ab8ed0">Web Page about the Customer "ALKI"</a> provides <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id14bdb360">me</a> with a myriad of exploration and data access paths e.g., when I click on the <a href="http://dbpedia.org/resource/Friend_of_a_friend" id="link-id10c388e0">foaf</a>:primarytopic property value link.</p> <p>This simple example, via a single Web Page, should put to rest any doubts about the utility of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0xb042fd8">Linked Data</a>. Of course this is <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=alfki&type=text&output=html" id="link-id10ccccf0">an old demo</a>, but this time around the UI is minimalist as my prior attempts skipped a few steps i.e., starting from within a <a href="http://ode.openlinksw.com" id="link-id10f8a530">Linked Data explorer/browser</a>.</p> <p>Important note: I haven't exported <a href="http://dbpedia.org/resource/SQL" id="link-id0x16dfc2a0">SQL</a> into an RDF data warehouse, I am converting the SQL into RDF <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> on the fly which has two fundamental benefits:</p> <ol> <li>No vulnerability to changes in the source DBMS</li> <li>Superior performance over the RDF warehouse since the source schema is SQL based and I can leverage the optimization of the underlying SQL engine when translating between <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xd9a4030">SPARQL</a> and SQL.</li> </ol> <p>Enjoy!</p> <h3>Related</h3> <ol> <li> <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1434" id="link-id11338a48">Requirements for Relational to RDF Mapping</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1433" id="link-id10d84278">Handling Graph Transitivity in a SQL/RDF Hybrid Engine</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1431" id="link-id10c762e8">How Virtuoso handles the Web Aspects of Linked Data Queries</a>.</li> </ol>
2008-09-11T15:52:48.000050-04:00
Requirements for Relational-to-RDF Mapping
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-09-08#1436
2008-09-08T09:41:25Z
<div> <div style="display:none;">Requirements for Relational-to-RDF Mapping</div> <p>Many of you will know about the W3C relational-to-<a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1e1be0a8">RDF</a> mapping incubator activity. The group is planning to suggest forming a working group for drawing up a specification for relational-to-RDF mapping.</p> <p>To this effect, I recently summarized the group discussions and some of our own experiences around the topic at <<a href="http://esw.w3.org/topic/Rdb2RdfXG/ReqForMappingByOErling" id="link-id146030e8">http://esw.w3.org/topic/Rdb2RdfXG/ReqForMappingByOErling</a>>.</p> <p>I will here discuss this less formally and more in the light of our own experience. A working group goal statement must be neutral vis à vis the following points, even if any working group will unavoidably encounter these issues on the way. A <a href="http://dbpedia.org/resource/Blog" id="link-id0x1e6b3950">blog</a> post on the other hand can be more specific.</p> <p>I gave a talk to the <a href="http://www.w3.org/2005/Incubator/rdb2rdf/" id="link-id0xa0932c68">RDB2RDF XG</a> this spring, with these <a href="http://virtuoso.openlinksw.com/wiki/main/Main/VirtPresentations/Relational2RDF.ppt" id="link-id14572540">slides</a>.</p> <p>The main point is that people would really like to map on-the-fly, if they only could. Making an RDF warehouse is not of value in itself, but it is true that in some cases this cannot be avoided.</p> <p>At first sight, one would think that a mapping specification could be neutral as regards whether one stores the mapped triples as triples or makes them on demand. There is almost no comparison between the complexity of doing non-trivial mappings on-the-fly versus mapping as ETL. Some of this complexity spills over into the requirements for a mapping language. </p> <h2>Eliminating JOINs</h2> <p>We expect to have a situation where one virtual triple can have many possible sources. The mapping is a union of mapped databases. Any integration scenario will have this feature. In such a situation, if we are <code>JOIN</code>ing using such triples, we end up with <code>UNION</code>s of all databases that could produce the triples in question. This is generally not desired. Therefore, in the on-demand mapping case, there must be a lot of type inference logic that is not relevant in the ETL scenario.</p> <p>To make the point clearer, suppose a query like "list the organizations whose representatives have published about <i>xx</i>." Suppose that there are three databases mapped, all of which have a table of organizations, a table of persons with affiliation to organizations, a table of publications by these persons, and finally a table of tags for the publications. Now, we want the laboratories that have published with articles with <a href="http://dbpedia.org/resource/Tag" id="link-id0xa0977bf0">tag</a> <i>XX</i>. It is a matter of common sense in this scenario that a publication will have the author and the author's affiliation in the same database. However, the RDB-to-RDF mapping does not necessarily know this, if all that it is told is that a table makes IRIs of publications by applying a certain pattern to the primary key of the publications table. To infer what needs to be inferred, the system must realize that IRIs from one mapping are disjoint from IRIs from another: A paper in database <i>X</i> will usually not have an author in database <i>Y</i>. The IDs in database <i>Y</i>, even if perchance equal to the IDs in <i>X</i>, do not mean the same thing, and there is no point joining across databases by them.</p> <p>This entire question is a non-issue in the ETL scenario, but is absolutely vital in the real-time mapping. This is also something that must be stated, at least implicitly, in any mapping. If a mapping translates keys of one place to IRIs with one pattern, and keys from another using another pattern, it must be inferable from the patterns whether the sets of IRIs will be disjoint.</p> <p>This is critical. Otherwise we will be joining everything to everything else, and there will be orders of magnitude of penalty compared to hand-crafted <a href="http://dbpedia.org/resource/SQL" id="link-id0xa09490f8">SQL</a> over the same <a href="http://dbpedia.org/resource/Data" id="link-id0xa095efd0">data</a> sources.</p> <h2>Expectations and Limitations on Queries</h2> <p> <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1e360230">SPARQL</a> queries translate quite well to SQL when there is only one table that can produce a triple with a subject of a given class, when there are few columns that can map to a given predicate, and when classes and predicates are literals in the query.</p> <p> <a href="http://virtuoso.openlinksw.com" id="link-id0x1f5edb30">Virtuoso</a> has some SQL extensions for dealing with breaking a wide table into a row per column. This facilitates dealing with predicates that are not known at query compile time. If the table in question is not managed by Virtuoso, Virtuoso's SQL virtualization/federation takes care of the matter. If a mapping system goes directly to third-party SQL, no such tricks can be used.</p> <p>The above example suggests that for supporting on-the-fly mapping without relying on owning the SQL underneath, some subsets of SPARQL may have to be defined. For example, one will probably have to require that all predicates be literals. The alternative is prohibitive run-time cost and complexity.</p> <p>But we must not lose the baby with the bath-water. Aside from offering global identifiers, RDF's attractions include subclasses and sub-predicates. In relational terms, these translate to <code>UNION</code>s and do involve some added cost. A mapping system just has to have means of dealing with this cost, and of recognizing cases where this cost is prohibitive. Some further work is likely to be required for defining well-behaved subsets of SPARQL and mappings.</p> <h2>ETL Ou Ne Pas ETL?</h2> <p>Whether to warehouse or not? If one has hundreds of sources, of which some are not even relational, some ETL would seem necessary. Kashiup Vipul gave a position paper at last year's RDB-to-RDF mapping workshop in Cambridge, Massachusetts, about a system of relational mapping and on-demand RDF-izers of diverse semi-structured biomedical data, e.g., spreadsheets. The issue certainly exists, and any mapping work will likely encounter integration scenarios where one part is fairly neatly mapped from relational stores, and another part comes from a less structured repository of ETLed physical triples.</p> <p>Our take is that if something is a large or very large relational store, then map; else, ETL. With Virtuoso, we can mix mapped and local triples, but this is not a generally available feature of triple stores and standardization will likely have to wait until there are more implementations.</p> <h2>Conclusions</h2> <ul> <li>If you map on demand, watch out for an explosion of <code>UNION</code>s when integrating sources that talk of similar things.</li> <li>If you integrate lots of sources, some ETL is likely unavoidable. Look for ways of dealing with part ETL, part mapping. ETLing everything is not always best or even possible.</li> <li>If you map a single fairly-clean RDB to RDF, mapping will work well, potentially much faster than triple storage. Higher storage density and more data per index lookup on the relational side.</li> <li>If you map on demand, some restrictions to SPARQL may be practically necessary. These have to do with variables in predicate position, variables in class position, etc. Individual implementations may support these, but standardization will likely have to put limits on them.</li> </ul> <p>This was a quick summary, by no means comprehensive, on what an eventual RDB2RDF working group would come across. This is a sort of addendum to the requirements I outlined on the ESW wiki.</p> </div>
2008-09-08T15:03:09-04:00
Transitivity and Graphs for SQL
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-09-08#1435
2008-09-08T09:41:24Z
<div> <div style="display:none;">Transitivity and Graphs for SQL</div> <h2>Background</h2> <p>I have mentioned on a couple of prior occasions that basic graph operations ought to be integrated into the <a href="http://dbpedia.org/resource/SQL" id="link-id0xa1a18c58">SQL</a> query language.</p> <p>The history of databases is by and large about moving from specialized applications toward a generic platform. The introduction of the DBMS itself is the archetypal example. It is all about extracting the common features of applications and making these the features of a platform instead.</p> <p>It is now time to apply this principle to graph traversal.</p> <p>The rationale is that graph operations are somewhat tedious to write in a parallelize-able, latency-tolerant manner. Writing them as one would for memory-based <a href="http://dbpedia.org/resource/Data" id="link-id0xaf8c730">data</a> structures is easier but totally unscalable as soon as there is any latency involved, i.e., disk reads or messages between cluster peers.</p> <p>The ad-hoc nature and very large volume of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xae41ef0">RDF</a> data makes this a timely question. Up until now, the answer to this question has been to materialize any implied facts in RDF stores. If <i>a</i> was part of <i>b</i>, and <i>b</i> part of <i><a href="http://dbpedia.org/resource/C_(programming_language)" id="link-id0xac9d8790">c</a></i>, the implied fact that <i>a</i> is part of <i>c</i> would be inserted explicitly into the database as a pre-query step.</p> <p>This is simple and often efficient, but tends to have the downside that one makes a specialized warehouse for each new type of query. The activity becomes less ad-hoc.</p> <p>Also, this becomes next to impossible when the scale approaches web scale, or if some of the data is liable to be on-and-off included-into or excluded-from the set being analyzed. This is why with <a href="http://virtuoso.openlinksw.com" id="link-id0xb68f9d0">Virtuoso</a> we have tended to favor inference on demand ("backward chaining") and mapping of relational data into RDF without copying.</p> <p>The SQL world has taken steps towards dealing with recursion with the <code>WITH - UNION</code> construct which allows definition of recursive views. The idea there is to define, for example, a tree walk as a <code>UNION</code> of the data of the starting node plus the recursive walk of the starting node's immediate children.</p> <p>The main problem with this is that I do not very well see how a SQL optimizer could effectively rearrange queries involving <code>JOIN</code>s between such recursive views. This model of recursion seems to lose SQL's non-procedural nature. One can no longer easily rearrange <code>JOIN</code>s based on what data is given and what is to be retrieved. If the recursion is written from root to leaf, it is not obvious how to do this from leaf to root. At any rate, queries written in this way are so complex to write, let alone optimize, that I decided to take another approach.</p> <p>Take a question like "list the parts of products of category <i>C</i> which have materials that are classified as toxic." Suppose that the product categories are a tree, the product parts are a tree, and the materials classification is a tree taxonomy where "toxic" has a multilevel substructure.</p> <p>Depending on the count of products and materials, the query can be evaluated as either going from products to parts to materials and then climbing up the materials tree to see if the material is toxic. Or one could do it in reverse, starting with the different toxic materials, looking up the parts containing these, going to the part tree to the product, and up the product hierarchy to see if the product is in the right category. One should be able to evaluate the identical query either way depending on what indices exist, what the cardinalities of the relations are, and so forth — regular cost based optimization.</p> <p>Especially with RDF, there are many problems of this type. In regular SQL, it is a long-standing cultural practice to flatten hierarchies, but this is not the case with RDF.</p> <p>In Virtuoso, we see <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xb3bdcc0">SPARQL</a> as reducing to SQL. Any RDF-oriented database-engine or query-optimization feature is accessed via SQL. Thus, if we address run-time-recursion in the Virtuoso query engine, this becomes, <i>ipso facto</i>, an SQL feature. Besides, we remember that SQL is a much more mature and expressive language than the current SPARQL recommendation.</p> <h2> SQL and Transitivity </h2> <p>We will here look at some simple social network queries. A later article will show how to do more general graph operations. We extend the SQL derived table construct, i.e., <code>SELECT</code> in another <code>SELECT</code>'s <code>FROM</code> clause, with a <code>TRANSITIVE</code> clause.</p> <p>Consider the data:</p> <blockquote> <pre><code>CREATE TABLE "knows" ("p1" INT, "p2" INT, PRIMARY KEY ("p1", "p2") ); ALTER INDEX "knows" ON "knows" PARTITION ("p1" INT); CREATE INDEX "knows2" ON "knows" ("p2", "p1") PARTITION ("p2" INT); </code> </pre></blockquote> <p>We represent a social network with the many-to-many relation "knows". The persons are identified by integers.</p> <blockquote> <pre><code>INSERT INTO "knows" VALUES (1, 2); INSERT INTO "knows" VALUES (1, 3); INSERT INTO "knows" VALUES (2, 4);</code> </pre> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2" FROM "knows" ) "k" WHERE "k"."p1" = 1;</code></pre></blockquote> <p>We obtain the result:</p> <blockquote> <table width="100"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> </tr> <tr> <td align="center">1</td> <td align="center">3</td> </tr> <tr> <td align="center">1</td> <td align="center">2</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> </tr> </table> </blockquote> <p>The operation is reversible:</p> <blockquote> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2" FROM "knows" ) "k" WHERE "k"."p2" = 4; </code> </pre> <table width="100"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> </tr> <tr> <td align="center">2</td> <td align="center">4</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> </tr> </table> </blockquote> <p>Since now we give <i>p2</i>, we traverse from <i>p2</i> towards <i>p1</i>. The result set states that 4 is known by 2 and 2 is known by 1.</p> <p>To see what would happen if <i>x</i> knowing <i>y</i> also meant <i>y</i> knowing <i>x</i>, one could write:</p> <blockquote> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2" FROM (SELECT "p1", "p2" FROM "knows" UNION ALL SELECT "p2", "p1" FROM "knows" ) "k2" ) "k" WHERE "k"."p2" = 4;</code> </pre> <table width="100"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> </tr> <tr> <td align="center">2</td> <td align="center">4</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> </tr> <tr> <td align="center">3</td> <td align="center">4</td> </tr> </table> </blockquote> <p>Now, since we know that 1 and 4 are related, we can ask how they are related.</p> <blockquote> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2", T_STEP (1) AS "via", T_STEP ('step_no') AS "step", T_STEP ('path_id') AS "path" FROM "knows" ) "k" WHERE "p1" = 1 AND "p2" = 4;</code> </pre> <table width="250"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> <th align="center" width="50">via</th> <th align="center" width="50">step</th> <th align="center" width="50">path</th> </tr> <tr> <td align="center">1</td> <td align="center">4</td> <td align="center">1</td> <td align="center">0</td> <td align="center">0</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> <td align="center">2</td> <td align="center">1</td> <td align="center">0</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> <td align="center">4</td> <td align="center">2</td> <td align="center">0</td> </tr> </table> </blockquote> <p>The two first columns are the ends of the path. The next column is the person that is a step on the path. The next one is the number of the step, counting from 0, so that the end of the path that corresponds to the end condition on the column designated as input, i.e., <i>p1</i>, has number 0. Since there can be multiple solutions, the last column is a sequence number allowing distinguishing multiple alternative paths from each other.</p> <p>For LinkedIn users, the friends ordered by distance and descending friend count query, which is at the basis of most LinkedIn search result views can be written as: </p> <blockquote> <pre><code>SELECT p2, dist, (SELECT COUNT (*) FROM "knows" "c" WHERE "c"."p1" = "k"."p2" ) FROM (SELECT TRANSITIVE t_in (1) t_out (2) t_distinct "p1", "p2", t_step ('step_no') AS "dist" FROM "knows" ) "k" WHERE "p1" = 1 ORDER BY "dist", 3 DESC;</code> </pre> <table width="150"> <tr> <th align="center" width="50">p2</th> <th align="center" width="50">dist</th> <th align="center" width="50">aggregate</th> </tr> <tr> <td align="center">2</td> <td align="center">1</td> <td align="center">1</td> </tr> <tr> <td align="center">3</td> <td align="center">1</td> <td align="center">0</td> </tr> <tr> <td align="center">4</td> <td align="center">2</td> <td align="center">0</td> </tr> </table> </blockquote> <h2>How?</h2> <p>The queries shown above work on Virtuoso v6. When running in cluster mode, several thousand graph traversal steps may be proceeding at the same time, meaning that all database access is parallelized and that the algorithm is internally latency-tolerant. By default, all results are produced in a deterministic order, permitting predictable slicing of result sets.</p> <p>Furthermore, for queries where both ends of a path are given, the optimizer may decide to attack the path from both ends simultaneously. So, supposing that every member of a social network has an average of 30 contacts, and we need to find a path between two users that are no more than 6 steps apart, we begin at both ends, expanding each up to 3 levels, and we stop when we find the first intersection. Thus, we reach 2 * 30^3 = 54,000 nodes, and not 30^6 = 729,000,000 nodes.</p> <p>Writing a generic database driven graph traversal framework on the application side, say in Java over <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0xa8a9ef8">JDBC</a>, would easily be over a thousand lines. This is much more work than can be justified just for a one-off, ad-hoc query. Besides, the traversal order in such a case could not be optimized by the DBMS.</p> <h2>Next</h2> <p>In a future <a href="http://dbpedia.org/resource/Blog" id="link-id0xb526a40">blog</a> post I will show how this feature can be used for common graph tasks like critical path, itinerary planning, traveling salesman, the 8 queens chess problem, etc. There are lots of switches for controlling different parameters of the traversal. This is just the beginning. I will also give examples of the use of this in SPARQL.</p> </div>
2008-09-08T15:43:07-04:00
Requirements for Relational-to-RDF Mapping
http://www.openlinksw.com:443/weblog/oerling/?date=2008-09-08#1434
2008-09-08T09:40:06Z
<p>Many of you will know about the W3C relational-to-<a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1d61a948">RDF</a> mapping incubator activity. The group is planning to suggest forming a working group for drawing up a specification for relational-to-RDF mapping.</p> <p>To this effect, I recently summarized the group discussions and some of our own experiences around the topic at <<a href="http://esw.w3.org/topic/Rdb2RdfXG/ReqForMappingByOErling" id="link-id146030e8">http://esw.w3.org/topic/Rdb2RdfXG/ReqForMappingByOErling</a>>.</p> <p>I will here discuss this less formally and more in the light of our own experience. A working group goal statement must be neutral vis à vis the following points, even if any working group will unavoidably encounter these issues on the way. A <a href="http://dbpedia.org/resource/Blog" id="link-id0x21334340">blog</a> post on the other hand can be more specific.</p> <p>I gave a talk to the <a href="http://www.w3.org/2005/Incubator/rdb2rdf/" id="link-id0x1ef85f58">RDB2RDF XG</a> this spring, with these <a href="http://virtuoso.openlinksw.com/wiki/main/Main/VirtPresentations/Relational2RDF.ppt" id="link-id14572540">slides</a>.</p> <p>The main point is that people would really like to map on-the-fly, if they only could. Making an RDF warehouse is not of value in itself, but it is true that in some cases this cannot be avoided.</p> <p>At first sight, one would think that a mapping specification could be neutral as regards whether one stores the mapped triples as triples or makes them on demand. There is almost no comparison between the complexity of doing non-trivial mappings on-the-fly versus mapping as ETL. Some of this complexity spills over into the requirements for a mapping language. </p> <h2>Eliminating JOINs</h2> <p>We expect to have a situation where one virtual triple can have many possible sources. The mapping is a union of mapped databases. Any integration scenario will have this feature. In such a situation, if we are <code>JOIN</code>ing using such triples, we end up with <code>UNION</code>s of all databases that could produce the triples in question. This is generally not desired. Therefore, in the on-demand mapping case, there must be a lot of type inference logic that is not relevant in the ETL scenario.</p> <p>To make the point clearer, suppose a query like "list the organizations whose representatives have published about <i>xx</i>." Suppose that there are three databases mapped, all of which have a table of organizations, a table of persons with affiliation to organizations, a table of publications by these persons, and finally a table of tags for the publications. Now, we want the laboratories that have published with articles with <a href="http://dbpedia.org/resource/Tag" id="link-id0x1d8270d0">tag</a> <i>XX</i>. It is a matter of common sense in this scenario that a publication will have the author and the author's affiliation in the same database. However, the RDB-to-RDF mapping does not necessarily know this, if all that it is told is that a table makes IRIs of publications by applying a certain pattern to the primary key of the publications table. To infer what needs to be inferred, the system must realize that IRIs from one mapping are disjoint from IRIs from another: A paper in database <i>X</i> will usually not have an author in database <i>Y</i>. The IDs in database <i>Y</i>, even if perchance equal to the IDs in <i>X</i>, do not mean the same thing, and there is no point joining across databases by them.</p> <p>This entire question is a non-issue in the ETL scenario, but is absolutely vital in the real-time mapping. This is also something that must be stated, at least implicitly, in any mapping. If a mapping translates keys of one place to IRIs with one pattern, and keys from another using another pattern, it must be inferable from the patterns whether the sets of IRIs will be disjoint.</p> <p>This is critical. Otherwise we will be joining everything to everything else, and there will be orders of magnitude of penalty compared to hand-crafted <a href="http://dbpedia.org/resource/SQL" id="link-id0x1d8b0b40">SQL</a> over the same <a href="http://dbpedia.org/resource/Data" id="link-id0x1ee63530">data</a> sources.</p> <h2>Expectations and Limitations on Queries</h2> <p> <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1dcfd7b0">SPARQL</a> queries translate quite well to SQL when there is only one table that can produce a triple with a subject of a given class, when there are few columns that can map to a given predicate, and when classes and predicates are literals in the query.</p> <p> <a href="http://virtuoso.openlinksw.com" id="link-id0x2006b798">Virtuoso</a> has some SQL extensions for dealing with breaking a wide table into a row per column. This facilitates dealing with predicates that are not known at query compile time. If the table in question is not managed by Virtuoso, Virtuoso's SQL virtualization/federation takes care of the matter. If a mapping system goes directly to third-party SQL, no such tricks can be used.</p> <p>The above example suggests that for supporting on-the-fly mapping without relying on owning the SQL underneath, some subsets of SPARQL may have to be defined. For example, one will probably have to require that all predicates be literals. The alternative is prohibitive run-time cost and complexity.</p> <p>But we must not lose the baby with the bath-water. Aside from offering global identifiers, RDF's attractions include subclasses and sub-predicates. In relational terms, these translate to <code>UNION</code>s and do involve some added cost. A mapping system just has to have means of dealing with this cost, and of recognizing cases where this cost is prohibitive. Some further work is likely to be required for defining well-behaved subsets of SPARQL and mappings.</p> <h2>ETL Ou Ne Pas ETL?</h2> <p>Whether to warehouse or not? If one has hundreds of sources, of which some are not even relational, some ETL would seem necessary. Kashiup Vipul gave a position paper at last year's RDB-to-RDF mapping workshop in Cambridge, Massachusetts, about a system of relational mapping and on-demand RDF-izers of diverse semi-structured biomedical data, e.g., spreadsheets. The issue certainly exists, and any mapping work will likely encounter integration scenarios where one part is fairly neatly mapped from relational stores, and another part comes from a less structured repository of ETLed physical triples.</p> <p>Our take is that if something is a large or very large relational store, then map; else, ETL. With Virtuoso, we can mix mapped and local triples, but this is not a generally available feature of triple stores and standardization will likely have to wait until there are more implementations.</p> <h2>Conclusions</h2> <ul> <li>If you map on demand, watch out for an explosion of <code>UNION</code>s when integrating sources that talk of similar things.</li> <li>If you integrate lots of sources, some ETL is likely unavoidable. Look for ways of dealing with part ETL, part mapping. ETLing everything is not always best or even possible.</li> <li>If you map a single fairly-clean RDB to RDF, mapping will work well, potentially much faster than triple storage. Higher storage density and more data per index lookup on the relational side.</li> <li>If you map on demand, some restrictions to SPARQL may be practically necessary. These have to do with variables in predicate position, variables in class position, etc. Individual implementations may support these, but standardization will likely have to put limits on them.</li> </ul> <p>This was a quick summary, by no means comprehensive, on what an eventual RDB2RDF working group would come across. This is a sort of addendum to the requirements I outlined on the ESW wiki.</p>
2015-08-20T17:16:40.851458-04:00
Transitivity and Graphs for SQL
http://www.openlinksw.com:443/weblog/oerling/?date=2008-09-08#1433
2008-09-08T09:20:11Z
<h2>Background</h2> <p>I have mentioned on a couple of prior occasions that basic graph operations ought to be integrated into the <a href="http://dbpedia.org/resource/SQL" id="link-id0xb1fe830">SQL</a> query language.</p> <p>The history of databases is by and large about moving from specialized applications toward a generic platform. The introduction of the DBMS itself is the archetypal example. It is all about extracting the common features of applications and making these the features of a platform instead.</p> <p>It is now time to apply this principle to graph traversal.</p> <p>The rationale is that graph operations are somewhat tedious to write in a parallelize-able, latency-tolerant manner. Writing them as one would for memory-based <a href="http://dbpedia.org/resource/Data" id="link-id0x1cb37218">data</a> structures is easier but totally unscalable as soon as there is any latency involved, i.e., disk reads or messages between cluster peers.</p> <p>The ad-hoc nature and very large volume of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1e1850a0">RDF</a> data makes this a timely question. Up until now, the answer to this question has been to materialize any implied facts in RDF stores. If <i>a</i> was part of <i>b</i>, and <i>b</i> part of <i><a href="http://dbpedia.org/resource/C_(programming_language)" id="link-id0xa1a08d38">c</a></i>, the implied fact that <i>a</i> is part of <i>c</i> would be inserted explicitly into the database as a pre-query step.</p> <p>This is simple and often efficient, but tends to have the downside that one makes a specialized warehouse for each new type of query. The activity becomes less ad-hoc.</p> <p>Also, this becomes next to impossible when the scale approaches web scale, or if some of the data is liable to be on-and-off included-into or excluded-from the set being analyzed. This is why with <a href="http://virtuoso.openlinksw.com" id="link-id0xa51bd10">Virtuoso</a> we have tended to favor inference on demand ("backward chaining") and mapping of relational data into RDF without copying.</p> <p>The SQL world has taken steps towards dealing with recursion with the <code>WITH - UNION</code> construct which allows definition of recursive views. The idea there is to define, for example, a tree walk as a <code>UNION</code> of the data of the starting node plus the recursive walk of the starting node's immediate children.</p> <p>The main problem with this is that I do not very well see how a SQL optimizer could effectively rearrange queries involving <code>JOIN</code>s between such recursive views. This model of recursion seems to lose SQL's non-procedural nature. One can no longer easily rearrange <code>JOIN</code>s based on what data is given and what is to be retrieved. If the recursion is written from root to leaf, it is not obvious how to do this from leaf to root. At any rate, queries written in this way are so complex to write, let alone optimize, that I decided to take another approach.</p> <p>Take a question like "list the parts of products of category <i>C</i> which have materials that are classified as toxic." Suppose that the product categories are a tree, the product parts are a tree, and the materials classification is a tree taxonomy where "toxic" has a multilevel substructure.</p> <p>Depending on the count of products and materials, the query can be evaluated as either going from products to parts to materials and then climbing up the materials tree to see if the material is toxic. Or one could do it in reverse, starting with the different toxic materials, looking up the parts containing these, going to the part tree to the product, and up the product hierarchy to see if the product is in the right category. One should be able to evaluate the identical query either way depending on what indices exist, what the cardinalities of the relations are, and so forth — regular cost based optimization.</p> <p>Especially with RDF, there are many problems of this type. In regular SQL, it is a long-standing cultural practice to flatten hierarchies, but this is not the case with RDF.</p> <p>In Virtuoso, we see <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xb4b3ce8">SPARQL</a> as reducing to SQL. Any RDF-oriented database-engine or query-optimization feature is accessed via SQL. Thus, if we address run-time-recursion in the Virtuoso query engine, this becomes, <i>ipso facto</i>, an SQL feature. Besides, we remember that SQL is a much more mature and expressive language than the current SPARQL recommendation.</p> <h2> SQL and Transitivity </h2> <p>We will here look at some simple social network queries. A later article will show how to do more general graph operations. We extend the SQL derived table construct, i.e., <code>SELECT</code> in another <code>SELECT</code>'s <code>FROM</code> clause, with a <code>TRANSITIVE</code> clause.</p> <p>Consider the data:</p> <blockquote> <pre><code>CREATE TABLE "knows" ("p1" INT, "p2" INT, PRIMARY KEY ("p1", "p2") ); ALTER INDEX "knows" ON "knows" PARTITION ("p1" INT); CREATE INDEX "knows2" ON "knows" ("p2", "p1") PARTITION ("p2" INT); </code> </pre></blockquote> <p>We represent a social network with the many-to-many relation "knows". The persons are identified by integers.</p> <blockquote> <pre><code>INSERT INTO "knows" VALUES (1, 2); INSERT INTO "knows" VALUES (1, 3); INSERT INTO "knows" VALUES (2, 4);</code> </pre> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2" FROM "knows" ) "k" WHERE "k"."p1" = 1;</code></pre></blockquote> <p>We obtain the result:</p> <blockquote> <table width="100"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> </tr> <tr> <td align="center">1</td> <td align="center">3</td> </tr> <tr> <td align="center">1</td> <td align="center">2</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> </tr> </table> </blockquote> <p>The operation is reversible:</p> <blockquote> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2" FROM "knows" ) "k" WHERE "k"."p2" = 4; </code> </pre> <table width="100"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> </tr> <tr> <td align="center">2</td> <td align="center">4</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> </tr> </table> </blockquote> <p>Since now we give <i>p2</i>, we traverse from <i>p2</i> towards <i>p1</i>. The result set states that 4 is known by 2 and 2 is known by 1.</p> <p>To see what would happen if <i>x</i> knowing <i>y</i> also meant <i>y</i> knowing <i>x</i>, one could write:</p> <blockquote> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2" FROM (SELECT "p1", "p2" FROM "knows" UNION ALL SELECT "p2", "p1" FROM "knows" ) "k2" ) "k" WHERE "k"."p2" = 4;</code> </pre> <table width="100"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> </tr> <tr> <td align="center">2</td> <td align="center">4</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> </tr> <tr> <td align="center">3</td> <td align="center">4</td> </tr> </table> </blockquote> <p>Now, since we know that 1 and 4 are related, we can ask how they are related.</p> <blockquote> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2", T_STEP (1) AS "via", T_STEP ('step_no') AS "step", T_STEP ('path_id') AS "path" FROM "knows" ) "k" WHERE "p1" = 1 AND "p2" = 4;</code> </pre> <table width="250"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> <th align="center" width="50">via</th> <th align="center" width="50">step</th> <th align="center" width="50">path</th> </tr> <tr> <td align="center">1</td> <td align="center">4</td> <td align="center">1</td> <td align="center">0</td> <td align="center">0</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> <td align="center">2</td> <td align="center">1</td> <td align="center">0</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> <td align="center">4</td> <td align="center">2</td> <td align="center">0</td> </tr> </table> </blockquote> <p>The two first columns are the ends of the path. The next column is the person that is a step on the path. The next one is the number of the step, counting from 0, so that the end of the path that corresponds to the end condition on the column designated as input, i.e., <i>p1</i>, has number 0. Since there can be multiple solutions, the last column is a sequence number allowing distinguishing multiple alternative paths from each other.</p> <p>For LinkedIn users, the friends ordered by distance and descending friend count query, which is at the basis of most LinkedIn search result views can be written as: </p> <blockquote> <pre><code>SELECT p2, dist, (SELECT COUNT (*) FROM "knows" "c" WHERE "c"."p1" = "k"."p2" ) FROM (SELECT TRANSITIVE t_in (1) t_out (2) t_distinct "p1", "p2", t_step ('step_no') AS "dist" FROM "knows" ) "k" WHERE "p1" = 1 ORDER BY "dist", 3 DESC;</code> </pre> <table width="150"> <tr> <th align="center" width="50">p2</th> <th align="center" width="50">dist</th> <th align="center" width="50">aggregate</th> </tr> <tr> <td align="center">2</td> <td align="center">1</td> <td align="center">1</td> </tr> <tr> <td align="center">3</td> <td align="center">1</td> <td align="center">0</td> </tr> <tr> <td align="center">4</td> <td align="center">2</td> <td align="center">0</td> </tr> </table> </blockquote> <h2>How?</h2> <p>The queries shown above work on Virtuoso v6. When running in cluster mode, several thousand graph traversal steps may be proceeding at the same time, meaning that all database access is parallelized and that the algorithm is internally latency-tolerant. By default, all results are produced in a deterministic order, permitting predictable slicing of result sets.</p> <p>Furthermore, for queries where both ends of a path are given, the optimizer may decide to attack the path from both ends simultaneously. So, supposing that every member of a social network has an average of 30 contacts, and we need to find a path between two users that are no more than 6 steps apart, we begin at both ends, expanding each up to 3 levels, and we stop when we find the first intersection. Thus, we reach 2 * 30^3 = 54,000 nodes, and not 30^6 = 729,000,000 nodes.</p> <p>Writing a generic database driven graph traversal framework on the application side, say in Java over <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0xb595050">JDBC</a>, would easily be over a thousand lines. This is much more work than can be justified just for a one-off, ad-hoc query. Besides, the traversal order in such a case could not be optimized by the DBMS.</p> <h2>Next</h2> <p>In a future <a href="http://dbpedia.org/resource/Blog" id="link-id0x1e4d4f18">blog</a> post I will show how this feature can be used for common graph tasks like critical path, itinerary planning, traveling salesman, the 8 queens chess problem, etc. There are lots of switches for controlling different parameters of the traversal. This is just the beginning. I will also give examples of the use of this in SPARQL.</p>
2015-08-20T17:17:48.452048-04:00
Epistemology of the Sponger, or How Virtuoso Drives a Web Query
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-09-05#1432
2008-09-05T09:20:56Z
<div> <div style="display:none;">Epistemology of the Sponger, or How Virtuoso Drives a Web Query</div> <p> <a href="http://virtuoso.openlinksw.com" id="link-id0x1ed6cf28">Virtuoso</a> has an extensive collection of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1f8d1f78">RDF</a>-izers called Sponger Cartridges. These take a web resource in one of 30+ formats (so far) and extract RDF from it. The Virtuoso <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id0x1edc90e8">Sponger</a> is a device which evaluates a query and along the way, finds dereferenceable links, dereferences them, and iteratively re-evaluates the query, until either nothing new is found or some limit is reached.</p> <p>We could call this <i>query-driven crawling</i>. The idea is intuitive — what one looks for, determines what one finds.</p> <p>This does however raise certain questions pertaining to the nature and ultimate possibility of <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x1f836b68">knowledge</a>, i.e., epistemology.</p> <p>The process of querying could be said to go from the few to the many, just like the process of harvesting <a href="http://dbpedia.org/resource/Data" id="link-id0x1edb1648">data</a> from the web, the way any search engine does. One follows links or makes joins and thereby increases one's reach.</p> <p>The difference is that a query has no <i>a priori</i> direction. If I ask for the phone numbers of my friends and there are no phone numbers in the database, then it is valid to give an empty result without looking at my friends at all. <a href="http://dbpedia.org/resource/Closed_world_assumption" id="link-id0x1edf1f30">Closed world</a>, as it is said. Never mind that the friends would have had a "see also" link to a retrievable document that did have a phone number.</p> <p>The problem is that a query execution plan determines what possible dereferenceable material the query will encounter during its execution. What is worse, a query plan tends toward the minimal, i.e., toward minimizing the chances of encountering something dereferenceable along the way. Where query and crawl appeared to have a similarity, in fact they have two opposite goals.</p> <p>The user generally has no idea of the execution plan. In the general case, the user <i>cannot</i> have an idea of this plan. There are valid, over 40 year old reasons for leaving the query planning to the database. In exceptional situations the user can read or direct these, but this is really quite tedious and requires understanding that is basically never present.</p> <p>So, given a query, how do we find data that will match it, short of having a pre-loaded database of absolutely everything? This is certainly a desirable goal, and all in the <a href="http://dbpedia.org/resource/Open_world_assumption" id="link-id0x1eb46548">open world</a>, distributed spirit of the web.</p> <p>Let us limit ourselves to queries that have some literals in the object or subject positions. A <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1ed293f8">SPARQL</a> query is basically a graph. Its vertices are variables and literals, and its edges are triple patterns. An edge is labeled by a predicate. For now, we will consider the predicate to always be a literal. From each literal, we can draw a tree, following each edge starting at this literal and descending until we find another literal. Each tree is not always a spanning tree of the graph, but all the trees collectively span the graph.</p> <p>Consider the query </p> <blockquote> <code>{ <john> knows ?x . <mary> knows ?x . ?x label ?l }.</code> </blockquote> The starting points are the literals <code>john</code> and <code>mary</code>. The <code>john</code> tree has one child, <code>?x</code>, which has the children <code>mary</code> and <code>?l</code>. One could notate it as <blockquote> <code>{ <john> knows ?x . {{ <mary> knows ?x} UNION {?x label ?l}}}</code> </blockquote> That is, the head first, and if it has more than one child, a union listing them, recursively. <p>If one composed such queries for each literal in the original pattern and evaluated each as a breadth first walk of the tree, no query optimization tricks, and for each binding of each variable, recorded whether there was something to dereference, one would in a finite time have reached all the directly reachable data. Then one could evaluate the original query, using whatever plan was preferred.</p> <p>The check for dereferenceable data applied to each IRI-valued binding formed in the above evaluation, would consist of looking for "see also", "same as", and other such properties of the IRI. It could also consult text based search engines. Since the evaluation is breadth first, it generates a large number of parallel tasks and is fairly latency tolerant, i.e., it will not die if it must retrieve a few pages from remote sources. We will leave the exact rewrite rules for unions, optionals, aggregates, subqueries, and so on, as an exercise; the general idea should be clear enough.</p> <p>We have here shown a way of transforming SPARQL queries in such a way as to guarantee dereferencing of findable links, without requiring the end user to either explicitly specify or understand query plans.</p> <p>The present Sponger does not work exactly in this manner but it will be developed in this direction. Fortunately, the algorithms outlined above are nothing complicated.</p> </div>
2008-09-05T16:04:28-04:00
Epistemology of the Sponger, or How Virtuoso Drives a Web Query
http://www.openlinksw.com:443/weblog/oerling/?date=2008-09-05#1431
2008-09-05T09:16:20Z
<p> <a href="http://virtuoso.openlinksw.com" id="link-id0x1fabb368">Virtuoso</a> has an extensive collection of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1f8625b8">RDF</a>-izers called Sponger Cartridges. These take a web resource in one of 30+ formats (so far) and extract RDF from it. The Virtuoso <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id0x1f841060">Sponger</a> is a device which evaluates a query and along the way, finds dereferenceable links, dereferences them, and iteratively re-evaluates the query, until either nothing new is found or some limit is reached.</p> <p>We could call this <i>query-driven crawling</i>. The idea is intuitive — what one looks for, determines what one finds.</p> <p>This does however raise certain questions pertaining to the nature and ultimate possibility of <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x1f895b08">knowledge</a>, i.e., epistemology.</p> <p>The process of querying could be said to go from the few to the many, just like the process of harvesting <a href="http://dbpedia.org/resource/Data" id="link-id0x1f879410">data</a> from the web, the way any search engine does. One follows links or makes joins and thereby increases one's reach.</p> <p>The difference is that a query has no <i>a priori</i> direction. If I ask for the phone numbers of my friends and there are no phone numbers in the database, then it is valid to give an empty result without looking at my friends at all. <a href="http://dbpedia.org/resource/Closed_world_assumption" id="link-id0x1f8b0658">Closed world</a>, as it is said. Never mind that the friends would have had a "see also" link to a retrievable document that did have a phone number.</p> <p>The problem is that a query execution plan determines what possible dereferenceable material the query will encounter during its execution. What is worse, a query plan tends toward the minimal, i.e., toward minimizing the chances of encountering something dereferenceable along the way. Where query and crawl appeared to have a similarity, in fact they have two opposite goals.</p> <p>The user generally has no idea of the execution plan. In the general case, the user <i>cannot</i> have an idea of this plan. There are valid, over 40 year old reasons for leaving the query planning to the database. In exceptional situations the user can read or direct these, but this is really quite tedious and requires understanding that is basically never present.</p> <p>So, given a query, how do we find data that will match it, short of having a pre-loaded database of absolutely everything? This is certainly a desirable goal, and all in the <a href="http://dbpedia.org/resource/Open_world_assumption" id="link-id0x1f845188">open world</a>, distributed spirit of the web.</p> <p>Let us limit ourselves to queries that have some literals in the object or subject positions. A <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1f84f348">SPARQL</a> query is basically a graph. Its vertices are variables and literals, and its edges are triple patterns. An edge is labeled by a predicate. For now, we will consider the predicate to always be a literal. From each literal, we can draw a tree, following each edge starting at this literal and descending until we find another literal. Each tree is not always a spanning tree of the graph, but all the trees collectively span the graph.</p> <p>Consider the query </p> <blockquote> <code>{ <john> knows ?x . <mary> knows ?x . ?x label ?l }.</code> </blockquote> The starting points are the literals <code>john</code> and <code>mary</code>. The <code>john</code> tree has one child, <code>?x</code>, which has the children <code>mary</code> and <code>?l</code>. One could notate it as <blockquote> <code>{ <john> knows ?x . {{ <mary> knows ?x} UNION {?x label ?l}}}</code> </blockquote> That is, the head first, and if it has more than one child, a union listing them, recursively. <p>If one composed such queries for each literal in the original pattern and evaluated each as a breadth first walk of the tree, no query optimization tricks, and for each binding of each variable, recorded whether there was something to dereference, one would in a finite time have reached all the directly reachable data. Then one could evaluate the original query, using whatever plan was preferred.</p> <p>The check for dereferenceable data applied to each IRI-valued binding formed in the above evaluation, would consist of looking for "see also", "same as", and other such properties of the IRI. It could also consult text based search engines. Since the evaluation is breadth first, it generates a large number of parallel tasks and is fairly latency tolerant, i.e., it will not die if it must retrieve a few pages from remote sources. We will leave the exact rewrite rules for unions, optionals, aggregates, subqueries, and so on, as an exercise; the general idea should be clear enough.</p> <p>We have here shown a way of transforming SPARQL queries in such a way as to guarantee dereferencing of findable links, without requiring the end user to either explicitly specify or understand query plans.</p> <p>The present Sponger does not work exactly in this manner but it will be developed in this direction. Fortunately, the algorithms outlined above are nothing complicated.</p>
2008-09-05T16:04:26-04:00
Linked Data, Ubiquity Commands, and Resource Descriptions (Update 3)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-09-05#1430
2008-09-05T05:43:00Z
<div> <p> <a href="http://labs.mozilla.com/2008/08/introducing-ubiquity/" id="link-id11258ea0">Ubiquity</a> from <a href="http://labs.mozilla.com/" id="link-id112ebe28">Mozilla Labs</a>, provides an alternative entry point for experiencing the "Controller" aspect of the <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id0xa0d2ccd0">Web</a>'s natural compatibility with the <a href="http://dbpedia.org/resource/Model-view-controller" id="link-id10ec1a08">MVC</a> development pattern. As I've noted (in <a href="http://myopenlink.net/weblog/public/search.vspx?blogid=kidehen-blog-0&q=mvc&type=text&output=html" id="link-id15390f28">various posts</a>) <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> Services, as practiced by the REST oriented Web 2.0 community or SOAP oriented SOA community within the enterprise, is fundamentally about the ("Controller" aspect of <a href="http://dbpedia.org/resource/Model-view-controller" id="link-id13c0d758">MVC</a>. </p> <p>Ubiquity provides a commandline interface for direct invocation of Web Services. For instance, in our case, we can expose the <a href="http://virtuoso.openlinksw.com" id="link-id10b04708">Virtuoso</a>'s in-built <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=rdf%20middleware&type=text&output=html" id="link-id1113ae38">RDF Middleware</a> ("Sponger") and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1457b3b8">Linked Data</a> deployment services via a single command of the form: describe-resource <url> </p> <p>To experience this neat addition to Firefox you need to do the following:</p> <ol> <li> <a href="https://people.mozilla.com/%7Eavarma/ubiquity-0.1.1.xpi" id="link-id13b15e88">Download</a> and install the Ubiquity Extension for Firefox</li> <li> <a href="http://demo.openlinksw.com/ubiq" id="link-id10e85880">Subscribe</a> to the OpenLink Command for Resource Description</li> <li>Click on CTRL+Space (Windows / Linux) or Option+Space (Mac OS X)</li> <li>Type in: describe-resource <a-web-resource-url> </li> </ol> <h3>How to unsubscribe</h3> At the current time, you need to do this if you've installed commands using ubiquity 0.1.0 and seek to use newer versions of the same commands after upgrading to ubiquity 0.1.1. <ol> <li>To unsubscribe use type "about:ubiquity" into browser</li> <li>Click on unsubscribe links associated with you command subscription list</li> </ol> <p>Enjoy!</p> </div>
2008-09-08T09:00:51-04:00
What's Up with Chrome?
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-09-03#1429
2008-09-03T20:33:32Z
<p>Here are a few descriptions of pages covering Google's Chrome browser:</p> <ul> <li> <a href="http://demo.openlinksw.com/proxy/html/http://vanirsystems.com/danielsblog/2008/09/03/google-chrome-my-verdict/" id="link-id14b2ad20">Daniel Lewis</a> - Comparative Analysis</li> <li> <a href="http://demo.openlinksw.com/proxy/html/http://demo.openlinksw.com/proxy/rdf/http://www.crunchbase.com/product/chrome%23this" id="link-id120f35d0">CrunchBase</a> - Product Page</li> <li> <a href="http://demo.openlinksw.com/proxy/html/http://feeds.feedburner.com/~r/OmMalik/~3/381659273/" id="link-id13e28090">GigaOM</a> - Industry Analysis</li> <li> <a href="http://demo.openlinksw.com/proxy/html/http://en.wikipedia.org/wiki/Google_Chrome" id="link-id10db3e48">Wikipedia</a> </li> <li> <a href="http://demo.openlinksw.com/proxy/html/http://news.cnet.com/8301-13860_3-10031661-56.html?part=rss&subj=news&tag=2547-1_3-0-20" id="link-id14a17f78">CNET</a> - Privacy Issues Analysis</li> <li> <a href="http://demo.openlinksw.com/proxy/html/http://feedproxy.google.com/~r/readwriteweb/~3/vF9P80B90XI/security_flaw_in_google_chrome.php" id="link-id14bdf6a8">ReadWriteWeb</a> - Security Issues Analysis</li> <li> <a href="http://demo.openlinksw.com/proxy/html/http://oakleafblog.blogspot.com/2008/09/chromes-evil-terms-of-service.html" id="link-id140af0a8">OakLeaf</a> - SaaS Terms Analysis</li> </ul> <p>As per usual, this is part post and part <a href="http://dbpedia.org/resource/Linked_Data" id="link-id140af0a8">Linked Data</a> demo. This time around, I am showcasing Proxy/Wrapper based dereferencable URIs and a new "Page Description" feature that showcases the capabilities of <a href="http://virtuoso.openlinksw.com" id="link-id11291898">Virtuoso</a>'s in-built RDFization Middleware. Also note, the resource descriptions (RDF) are presented using an HTML page.</p>
2008-09-04T08:39:02.000014-04:00
Programming the Universe
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-09-02#1428
2008-09-02T17:03:59Z
<p>I continue to be intrigued by <a href="http://feeds.feedburner.com/~r/ThinkingSpace" id="link-id14aa0000">Yihong Ding's</a> shared insights as expressed in part 2 of his <a href="http://dbpedia.org/resource/Blog" id="link-id17b07d90">blog</a> series titled: <a href="http://feeds.feedburner.com/~r/ThinkingSpace/~3/381149959/programming-universe-part-two.html" id="link-id10c9f740">Programming the Universe</a>. The <a href="http://dbpedia.org/resource/Blog" id="link-id10fbc0f8">blog</a> series shares Yihong's thoughts and reflections stimulated by the book, also titled: <a href="http://www.amazon.com/Programming-Universe-Quantum-Computer-Scientist/dp/1400033861/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1217201256&sr=1-1" id="link-id14a34190">Programming the Universe</a>.</p> <p> What strikes <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id111dd940">me</a> the most, is how sharing his findings act as serendipitous connectors to related insights and points of view, that ultimately create deeper shared <a href="http://dbpedia.org/resource/Knowledge" id="link-id10d78460">knowledge</a> about the core subject matter, courtesy of the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> hosted Blogosphere.</p> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1390" id="link-id1723bd88">Metcalfe, Einstein, and Linked Data.</a> </li> </ul>
2008-09-03T07:56:50-04:00
Connecting Freebase, Wikipedia, DBpedia, and other Linked Data Spaces (Update 1)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-08-29#1427
2008-08-29T17:53:47Z
<p>Here are some demonstrations of (X)HTML based representations of resource descriptions from <a href="http://www.freebase.com" id="link-id12275470">Freebase</a>, <a href="http://dbpedia.org/resource/DBpedia" id="link-id168abcc0">DBpedia</a>, <a href="http://www.bbc.co.uk/music/artists" id="link-id107c75c8">BBC Music Beta</a>, <a href="http://crunchbase.com" id="link-id1322e9a0">CrunchBase</a>, <a href="http://sw.opencyc.org" id="link-id16e09ea8">OpenCyc</a>, and <a href="http://umbel.org/about/" id="link-id188687c0">UMBEL</a> etc. What is really being demonstrated here is the use of Proxy / Wrapper URIs to expose powerful links across entities distilled from their container documents (or <a href="http://dbpedia.org/resource/Information" id="link-id13e117b0">information</a> resources). Of course, you see exactly the same technique in action whenever you visit <a href="http://dbpedia.org/resource/DBpedia" id="link-id10afe178">DBpedia</a> pages. Again, we are moving the concept of Linking from the document to document level, down to the document-<a href="http://dbpedia.org/resource/Entity" id="link-id16032730">entity</a> to document-<a href="http://dbpedia.org/resource/Entity" id="link-id13eef3d8">entity</a> level. The evolution of network link focal points is illustrated in slides <a href="http://virtuoso.openlinksw.com/presentations/Creating_Deploying_Exploiting_Linked_Data2/Creating_Deploying_Exploiting_Linked_Data2_TimBL_v3.html#(15)" id="link-id183523a8">15</a> to <a href="http://virtuoso.openlinksw.com/presentations/Creating_Deploying_Exploiting_Linked_Data2/Creating_Deploying_Exploiting_Linked_Data2_TimBL_v3.html#(22)" id="link-id18270200">22</a> of my <a href="http://virtuoso.openlinksw.com/presentations/Creating_Deploying_Exploiting_Linked_Data2/Creating_Deploying_Exploiting_Linked_Data2_TimBL_v3.html" id="link-id16f0a7c8">Linked Data Planet presentation</a> remix.</p> <h3>Live Examples</h3> <ol> <li> <a href="http://demo.openlinksw.com/proxy/html/http://www.freebase.com/view/en/abraham_lincoln" id="link-id11cf00b8">Abraham Lincoln</a> - Freebase (note: link from Freebase to <a href="http://dbpedia.org/resource/DBpedia" id="link-id17db1620">DBpedia</a> via Wikipedia)</li> <li> <a href="http://demo.openlinksw.com/proxy/html/http://www.crunchbase.com/company/amazon" id="link-id171d9930">Amazon</a> - CrunchBase (note: links from CruncBase to <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x1ed41510">DBpedia</a>)</li> <li> <a href="http://demo.openlinksw.com/proxy/html/http://www.bbc.co.uk/music/artists/cc197bad-dc9c-440d-a5b5-d52ba2e14234" id="link-id10a01dc0">Cold Play</a> - <a href="http://dbpedia.org/resource/BBC" id="link-id12fa5648">BBC</a> Music Beta (note: links to <a href="http://musicbrainz.org/" id="link-id126f71c8">Musicbrainz</a>)</li> <li> <a href="http://demo.openlinksw.com/proxy/html/http://virtuoso.openlinksw.com/presentations/Creating_Deploying_Exploiting_Linked_Data2/Creating_Deploying_Exploiting_Linked_Data2_TimBL_v3.html" id="link-id1732e820">Linked Data Planet Presentation</a> - Also a Slidy, Bibo Ontology, and <a href="http://dbpedia.org/resource/RDFa" id="link-id104869a8">RDFa</a> usage example</li> <li> <a href="http://demo.openlinksw.com/proxy/html/http://sw.opencyc.org/2008/06/10/concept/en/Music" id="link-id1699d628">Music</a> - <a href="http://dbpedia.org/resource/Cyc" id="link-id126c74f0">OpenCyc</a> Concept which exposes a <a href="http://dbpedia.org/resource/Linked_Data" id="link-id15687380">Hyperdata</a> link to its equivalent <a href="http://demo.openlinksw.com/proxy/html/http://umbel.org/ns/sc/Music" id="link-id13ebbac0">UMBEL Subject Concept</a> and back</li> </ol> <h4> <a href="http://virtuoso.openlinksw.com" id="link-id10537a28">Virtuoso</a>'s RDFization Middleware & <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10869440">Linked Data</a> Deployment Architecture Diagram</h4> <div> <table border="1"> <tr> <td><br /> <br /> <img alt="Image" src="http://virtuoso.openlinksw.com/ldp_presentation/images/linked_data_gen_opts3.png" /> <br /> <br /> </td> </tr> </table> </div> <p>Note: You can substitute my examples using any <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> resource <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id105a3e20">URL</a>. The underlying RDFization and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id18155cd0">Linked Data</a> deployment functionality of the <a href="http://virtuoso.openlinksw.com" id="link-id13f56ed0">Virtuoso</a> demo instance takes care of everything else. Also note that the HTML based resource description page capability is now deployed as part of the <a href="http://virtuoso.openlinksw.com" id="link-id17db1128">Virtuoso</a> <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id100a7630">Sponger</a> component of every <a href="http://virtuoso.openlinksw.com" id="link-id109eed20">Virtuoso</a> installation starting with from version <a href="http://sourceforge.net/project/shownotes.php?group_id=161622&release_id=622380" id="link-id1441f530">5.0.8.</a> </p>
2008-08-29T14:57:02.000001-04:00
Is the Semantic Web necessary (and feasible)?
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-08-29#1426
2008-08-29T15:00:50Z
<p>Here is another "Linked Discourse" effort via a <a href="http://dbpedia.org/resource/Blog" id="link-id13edcda8">blog</a> post that attempts to add perspective to a developing <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> based conversation. In this case, the conversation originates from <a href="http://geekaustin.org" id="link-id15a33728">Juan Sequeda</a>'s recent interview with <a href="http://www.linkedin.com/in/luxzia" id="link-id182a4a80">Jana Thompson</a> titled: <a href="http://geekaustin.org/2008/08/21/juan-sequeda-jana-thompson-necessity-semantic-web/" id="link-id146e1f40">Is the Semantic Web necessary (and feasible)?</a> </p> <blockquote> <cite>Jana: What are the benefits you see to the business community in adopting semantic technology? </cite> </blockquote> <p> <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id1941e3b0">Me</a>: Exposure, exploitation, of untapped treasure trove of interlinked <a href="http://dbpedia.org/resource/Data">data</a>, <a href="http://dbpedia.org/resource/Information" id="link-id13593fc0">information</a>, and <a href="http://dbpedia.org/resource/Knowledge" id="link-id1290c318">knowledge</a> across disparate IT infrastructure via conceptual entry points (<a href="http://dbpedia.org/resource/Entity" id="link-id107bad60">Entity</a> IDs / URIs / Data Source Names) that refer to as "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id15fab9f8">Context</a> Lenses".</p> <br /> <blockquote> <cite> Jana: Do you think these benefits are great enough for businesses to adopt the changes?</cite> </blockquote> <p> <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id0x584ffe0">Me</a>: Yes, infrastructural heterogeneity is a fact of corporate life (growth, mergers, acquisitions etc). Any technology that addresses these challenges is extremely important and valuable. Put differently, the opportunity costs associated with IT infrastructural heterogeneity remains high!</p> <br /> <blockquote> <cite>Jana: How large do you think this impact will actually be?</cite> </blockquote> <p>Me: Huge, enterprise have been aware of their data, <a href="http://dbpedia.org/resource/Information" id="link-id0x1b8057b0">information</a>, and <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x1b3e3760">knowledge</a> treasure troves etc. for eons. Tapping into these via a materialization of the "<a href="http://dbpedia.org/resource/Information">information</a> at your fingertips" vision is something they've simply been waiting to pursue without any platform lock-in, for as long as I've been in this industry.</p> <br /> <blockquote> <cite>Jana: I’ve heard, from contacts in the Bay Area, that they are skeptical of how large this impact of semantic technology will actually be on the web itself, but that the best uses of the technology are for fields such as medical information, or as you mentioned, geo-spatial data.</cite> </blockquote> <p>Me: Unfortunately, those people aren't connecting the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id10a337d8">Semantic Web</a> and open access to heterogeneous data sources, or the intrinsic value of holistic exploration location of <a href="http://dbpedia.org/resource/Entity" id="link-id0xaa58c520">entity</a> based data networks (aka <a href="http://dbpedia.org/resource/Linked_Data" id="link-id188a1910">Linked Data</a>).</p> <br /> <blockquote> <cite> Jana: Are semantic technologies going to be part of the web because of people championing the cause or because it is actually a necessary step?</cite> </blockquote> <p>Me: <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x9eb9aca0">Linked Data</a> technology on the Web is a vital extension of the current Web. Semantic Technology without the "Web" component, or what I refer to as "Semantics Inside only" solutions, simply offer little or no value as Web enhancements based on their incongruence with the essence of the Web i.e., "Open Linkage" and no Silos! A nice looking Silo is still a Silo.</p> <br /> <blockquote> <cite>Jana: In the early days of the web, there was an explosion of new websites, due to the ease of learning HTML, from a business to a person to some crackpot talking about aliens. Even today, CSS and XHTML are not so difficult to learn that a determined person can’t learn them from W3C or other tutorials easily. If OWL becomes the norm for websites, what do you think the effects will be on the web? Do you think it is easy enough to learn that it will be readily adopted as part of the standard toolkit for web developers for businesses?</cite> </blockquote> <p>Me: Correction, learning HTML had nothing to do with the Web's success. The value proposition of the Web simply reached critical mass and you simply couldn't afford to not be part of it. The easiest route to joining the Web juggernaut was a Web Page hosted on a Web Site. The question right now is: what's the equivalent driver for the <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id12e25c98">Web</a> bearing in mind the initial Web bootstrap. My answer is simply this: Open Data Access i.e., getting beyond the data silos that have inadvertently emerged from Web 2.0.</p> <br /> <blockquote> <cite> Jana: Following the same theme, do you think this will lead to an <a href="http://dbpedia.org/resource/Internet" id="link-id17041398">internet</a> full of corporate-controlled websites, with sites only written by developers rather than individuals?</cite> </blockquote> <p> Me: Not at all, we will have an <a href="http://dbpedia.org/resource/Internet" id="link-id0x16a4abe0">Internet</a> owned by it's participants i.e., You and the agents that work on your behalf.</p> <br /> <blockquote> <cite>Jana: So, you are imagining technologies such as <a href="http://dbpedia.org/resource/Drupal" id="link-id107d1d70">Drupal</a> or <a href="http://dbpedia.org/resource/WordPress" id="link-id13f48db8">Wordpress</a>, that allow users to manage sites without a great deal of <a href="http://dbpedia.org/resource/Knowledge">knowledge</a> of the nuts and bolts of current web technologies?</cite> </blockquote> <p>Me: Not at all! I envisage simple forms that provide conduits to powerful meshes of interlinked data spaces associated with Web users.</p> <br /> <blockquote> <cite>Jana: Given all of the buzz, and my own familiarity with ontology, I am just very curious if the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1955d360">semantic web</a> is truly necessary? </cite> </blockquote> <p>Me:This question is no different than saying: I hear the Web is becoming a Database, and I wonder if a Data Dictionary is necessary, or even if access to structured data is necessary. It's also akin to saying: I accept "Search" as my only mechanism for Web interaction even though in reality, I really want to be able to "Find" and "Process" relevant things at a quicker rate than I do today, relative to the amount of information, and information processing time, at my disposal.</p> <br /> <blockquote> <cite>Jana: Will it be worth it to most people to go away from the web in its current form, with keyword searches on sites like Google, to a richer and more interconnected internet with potentially better search technology?</cite> </blockquote> <p>Me: As stated above, we need to add "Find" to the portfolio of functions we seek to perform against the Web. "Finding" and "Searching" are mutually inclusive pursuits at different ends of an activity spectrum.</p> <br /> <blockquote> <cite>Jana: For our more technical readers, I have a few additional questions: If no standardization comes about for mapping relational databases to domain ontologies, how do you see that as influencing the decisions about adoption of semantic technology by businesses? After all, the success of technology often lives or dies on its ease of adoption.</cite> </blockquote> <p>Me: Standardization of<a href="http://esw.w3.org/topic/Rdb2RdfXG/StateOfTheArt" id="link-id10abbc30"> RDBMS to RDF Mapping</a> is not the critical success factor here (of course it would be nice). As stated earlier, the issue of data integration that arises from IT infrastructural heterogeneity has been with decision makers in the enterprise for ever. The problem is now seeping into the broader consumer realm via Web ubiquity. The mistakes made in the enterprise realm are now playing out in the consumer Web realm. In both realms the critical success factors are:</p> <ol> <li> Scalable productivity relative to exponential growth of data generated across Intranets, Extranets, and the Internet</li> <li>Concept based <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id0x114e6888">Context</a> Lenses that transcend logical and physical data heterogeneity by putting dereferencable URIs in front of the Line of Business Application Data and/or Web Data Spaces such as Blogs, Wikis, Discussion Forums etc.).</li> </ol>
2008-08-29T11:08:12.000002-04:00
The Essence of the Matter re. Information Overload
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-08-28#1425
2008-08-28T12:17:55Z
<p>The title of this post is an expression of my gut reaction to the quotes below, which originate from <a href="http://leobard.twoday.net/" id="link-id104b2308">Leo Sauermann</a>'s post about the <a href="http://leobard.twoday.net/stories/5151765/" id="link-id1889d5d8">Nepomuk Semantic Desktop for KDE</a>:</p> <blockquote> <cite><strong>Ansgar Bernardi</strong>, deputy head of the <a href="http://dbpedia.org/resource/Knowledge" id="link-id16d79970">Knowledge</a> Management Department at Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI, or the German Research Center for Artificial Intelligence) and Nepomuk's coordinator, explains, "The basic problem that we all face nowadays is how to handle vast amounts of <a href="http://dbpedia.org/resource/Information" id="link-id13a01b58">information</a> at a sensible rate." According to Bernardi, Nepomuk takes a traditional approach by creating a meta-<a href="http://dbpedia.org/resource/Data">data</a> layer with well-defined elements that services can be built upon to create and manipulate the <a href="http://dbpedia.org/resource/Information" id="link-id102433e8">information</a>.</cite> </blockquote> <p> The comment above echoes my sentiments about the imminence of "<a href="http://dbpedia.org/resource/Information" id="link-id0x10dd6c20">information</a> overload" due to the vast amounts of user generated content on the <a href="http://dbpedia.org/resource/Internet" id="link-id139926b0">Internet</a> as a whole. We are going to need to process more an more data within a fixed 24 hour timeframe, while attempting to balance our professional and personal lives. Be rest assured, this is a very serious issue, and you cannot event begin to address it without a <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id188ebc20">Linked Data</a>.</p> <blockquote> <cite>"The first idea of building the semantic desktop arose from the fact that one of our colleagues could not remember the girlfriends of his friends," Bernard says, more than half-seriously. "Because they kept changing -- you know how it is. The point is, you have a vast amount of <a href="http://dbpedia.org/resource/Information">information</a> on your desktop, hidden in files, hidden in emails, hidden in the names and structures of your folders. Nepomuk gives a standard way to handle such information."</cite> </blockquote> <p>If you get a personal <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id171dd2e0">URI</a> for <a href="http://dbpedia.org/resource/Entity" id="link-id18294318">Entity</a> "You", via a <a href="http://dbpedia.org/resource/Linked_Data" id="link-id188a1b10">Linked Data</a> aware platform (e.g. <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id167ad840">OpenLink Data Spaces</a>) that virtualizes data across your existing Web data spaces (blogs, feed subscriptions, wikis, shared bookmarks, photo galleries, calendars, etc.), you then only have to remember your <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id171c3ef0">URI</a> whenever you need to "Find" something, imagine that!</p> <p>To conclude, "information overload" is the imminent challenge of our time, and the keys to challenge alleviation lie in our ability to construct and maintain (via solutions) few <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id1074ade0">context</a> lenses (URIs) that provide coherent conduits into the dense mesh of structured <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0xd30b090">Linked Data</a> on the Web. </p>
2008-08-28T15:56:20-04:00
Crunchbase & Semantic Web Interview (Remix - Update 1)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-08-27#1424
2008-08-27T18:16:37Z
<p>After reading <a href="http://blog.crunchbase.com/2008/08/26/building-a-semantic-web-interview-with-benjamin-nowack/" id="link-id16b8e0e0">Bengee's interview with CrunchBase</a>, I decided to knock up a quick interview remix as part of my usual attempt to add to the developing discourse.</p> <blockquote> <cite><a href="http://www.crunchbase.com/" id="link-id17c8e7b8">CrunchBase</a>: When we released the <a href="http://www.crunchbase.com/help/api" id="link-id16681f68">CrunchBase API</a>, you were one of the first developers to step up and quickly released a <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com's%20BLOG%20%5B127%5D/1395" id="link-id1016d5f0">CrunchBase Sponger Cartridge</a>. Can you explain what a CrunchBase Sponger Cartridge is?</cite> </blockquote> <blockquote> <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id13243300">Me</a>: A Sponger Cartridge is a <a href="http://dbpedia.org/resource/Data">data</a> access driver for <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> Resources that plugs into our <a href="http://virtuoso.openlinksw.com" id="link-id17042f08">Virtuoso</a> <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id1399b588">Universal Server</a> (DBMS and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id137fd188">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id100b23d8">Web</a> Server combo amongst other things). It uses the internal structure of a resource and/or a web service associated with a resource, to materialize an RDF based <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id10418750">Linked Data graph</a> that essentially describes the resource via its properties (Attributes & Relationships). </blockquote> <br /> <img alt="Image" src="http://virtuoso.openlinksw.com/presentations/Creating_Deploying_Exploiting_Linked_Data2/images/ldp4.png" /> <br /> <br /> <br /> <blockquote> <cite>CrunchBase: And what inspired you to create it?</cite> </blockquote> <blockquote> <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id12fa60c0">Me</a>: Bengee built a new space with your data, and we've built a space on the fly from your data which still resides in your domain. Either solution extols the virtues of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id101a8d28">Linked Data</a> i.e. the ability to explore relationships across data items with high degrees of serendipity (also colloquially known as: following-your-nose pattern in <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id14a3ff30">Semantic Web</a> circles).</blockquote> <blockquote> <a href="http://cb.semsol.org/" id="link-id182a0170">Bengee</a> posted a notice to the <a href="http://esw.w3.org/topic/SweoIG/TaskForces/CommunityProjects/LinkingOpenData" id="link-id131e8d10">Linking Open Data Community</a>'s public <a href="http://lists.w3.org/Archives/Public/public-lod/2008Jul/0110.html" id="link-id11dd0720">mailing list announcing his effort</a>. Bearing in mind the fact that we've been using <a href="http://www.openlinksw.com/blog/~kidehen/?id=1144" id="link-id117cf6e8">middleware to mesh the realms of Web 2.0 and the Linked Data Web</a> for a while, it was a no-brainer to knock something up based on the conceptual similarities between <a href="http://wikicompany.org/wiki/Main_Page" id="link-id13b87a68">Wikicompany</a> and CrunchBase. In a sense, a quadrant of orthogonality is what immediately came to mind re. Wikicompany, CrunchBase, Bengee's RDFization efforts, and ours.</blockquote> <blockquote>Bengee created an RDF based <a href="http://dbpedia.org/resource/Linked_Data" id="link-id133c8fc8">Linked Data</a> warehouse based on the data exposed by your API, which is exposed via the <a href="http://cb.semsol.org/" id="link-id1826f928">Semantic CrunchBase</a> <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id102d8890">data space</a>. In our case we've taken the "RDFization on the fly" approach which produces a transient <a href="http://dbpedia.org/resource/Linked_Data" id="link-id16a0b8d0">Linked Data</a> View of the CrunchBase data exposed by your APIs. Our approach is in line with our world view: all resources on the Web are data sources, and the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1668e6c8">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id188e7da0">Web</a> is about incorporating HTTP into the naming scheme of these data sources so that the conventional <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id13490710">URL</a> based hyperlinking mechanism can be used to access a structured description of a resource, which is then transmitted using a range negotiable representation formats. In addition, based on the fact that we house and publish a lot of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id169aa568">Linked Data</a> on the Web (e.g. <a href="http://dbpedia.org/resource/DBpedia" id="link-id10af10e8">DBpedia</a>, <a href="http://www.pingthesemanticweb.com/about/" id="link-id10a2b710">PingTheSemanticWeb</a>, and others), we've also automatically meshed Crunchbase data with related data in <a href="http://dbpedia.org/resource/DBpedia" id="link-id1403cd40">DBpedia</a> and Wikicompany data.</blockquote> <br /> <blockquote> <cite>CrunchBase: Do you know of any apps that are using CrunchBase Cartridge to enhance their functionality?</cite> </blockquote> <blockquote> <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id177d24c8">Me</a>: Yes, the <a href="http://ode.openlinksw.com" id="link-id10725ca0">OpenLink Data Explorer</a> which provides CrunchBase site visitors with the option to explore the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id17dedea8">Linked Data</a> in the CrunchBase <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id13f02a00">data space</a>. It also allows them to "Mesh" (rather than "Mash") CrunchBase data with other <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11fb3ba0">Linked Data</a> sources on the Web without writing a single line of code. </blockquote> <br /> <blockquote> <cite>CrunchBase: You have been immersed in the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id12e18a00">Semantic Web</a> movement for a while now. How did you first get interested in the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id15132110">Semantic Web</a>?</cite> </blockquote> <blockquote> <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id0xddaa9c8">Me</a>: We saw the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id188b3330">Semantic Web</a> as a vehicle for standardizing conceptual views of heterogeneous data sources via <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id10350978">context</a> lenses (URIs). In 1998 as part of our strategy to expand our business beyond the development and deployment of <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id171d6798">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id138120a0">JDBC</a>, and OLE-DB data providers, we decided to build a <a href="http://dbpedia.org/resource/Virtual_Database" id="link-id13ea6618">Virtual Database</a> Engine (see: <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSHistory" id="link-id11a4fa30">Virtuoso History</a>), and in doing so we sought a standards based mechanism for the conceptual output of the <a href="http://dbpedia.org/resource/Federated_database_system" id="link-id101a1248">data virtualization</a> effort. As of the time of the <a href="http://www.w3.org/DesignIssues/Semantic.html" id="link-id18882cf8">seminal unveiling of the Semantic Web in 1998</a> we were clear about two things, in relation to the effects of the Web and <a href="http://dbpedia.org/resource/Internet" id="link-id12fa2c58">Internet</a> data management infrastructure inflections: 1) Existing DBMS technology had reached it limits 2) Web Servers would ultimately hit their functional limits. These fundamental realities compelled us to develop <a href="http://virtuoso.openlinksw.com" id="link-id102b09a0">Virtuoso</a> with an eye to leveraging the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id11984d98">Semantic Web</a> as a vehicle from completing its technical roadmap.</blockquote> <br /> <blockquote> <cite>CrunchBase: Can you put into layman’s terms exactly what RDF and <a href="http://dbpedia.org/resource/SPARQL" id="link-id1066dcf0">SPARQL</a> are and why they are important? Do they only matter for developers or will they extend past developers at some point and be used by website visitors as well?</cite> </blockquote> <blockquote>Me: RDF (Resource Description Framework) is a Graph based Data Model that facilitates resource description using the <a href="http://www.eslincanada.com/englishlesson2.html" id="link-id178b94a8">Subject, Predicate, and Object principle</a>. Associated with the core data model, as part of the overall framework, are a number of markup languages for expressing your descriptions (just as you express presentation markup semantics in HTML or document structure semantics in XML) that include: <a href="http://dbpedia.org/resource/RDFa" id="link-id188db0a8">RDFa</a> (simple extension of HTML markup for embedding descriptions of things in a page), N3 (a human friendly markup for describing resources), RDF/XML (a machine friendly markup for describing resources).</blockquote> <blockquote> <a href="http://dbpedia.org/resource/SPARQL" id="link-id188c2030">SPARQL</a> is the query language associated with the RDF Data Model, just as <a href="http://dbpedia.org/resource/SQL" id="link-id13f0ffe0">SQL</a> is a query language associated with the Relational Database Model. Thus, when you have RDF based structured and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id166874d0">linked data</a> on the Web, you can query against Web using <a href="http://dbpedia.org/resource/SPARQL" id="link-id1016cc98">SPARQL</a> just as you would against an <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id101c9708">Oracle</a>/<a href="http://dbpedia.org/resource/SQL" id="link-id11cb0b18">SQL</a> Server/<a href="http://dbpedia.org/resource/IBM_DB2" id="link-id10760ec0">DB2</a>/<a href="http://dbpedia.org/resource/IBM_Informix" id="link-id1066c8c0">Informix</a>/<a href="http://dbpedia.org/resource/Ingres" id="link-id18894f40">Ingres</a>/<a href="http://dbpedia.org/resource/MySQL" id="link-iddc9ebb0">MySQL</a>/etc.. DBMS using <a href="http://dbpedia.org/resource/SQL" id="link-id1030d120">SQL</a>. That's it in a nutshell.</blockquote> <br /> <blockquote> <cite>CrunchBase: On your website you wrote that “RDF and <a href="http://dbpedia.org/resource/SPARQL" id="link-id168e9ad0">SPARQL</a> as productivity boosters in everyday web development”. Can you elaborate on why you believe that to be true?</cite> </blockquote> <blockquote>Me: I think the ability to discern a formal description of anything via its discrete properties is of immense value re. productivity, especially when the capability in question results in a graph of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x179f6328">Linked Data</a> that isn't confined to a specific host operating system, database engine, application or service, programming language, or development framework. RDF <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> is about infrastructure for the true materialization of the "<a href="http://dbpedia.org/resource/Information" id="link-id13e475b8">Information</a> at Your Fingertips" vision of yore. Even though it's taken the emergence of RDF Linked Data to make the aforementioned vision tractable, the comprehension of the vision's intrinsic value have been clear for a very long time. Most organizations and/or individuals are quite familiar with the adage: <a href="http://dbpedia.org/resource/Knowledge" id="link-id13e38a30">Knowledge</a> is Power, well there isn't any <a href="http://dbpedia.org/resource/Knowledge" id="link-id188b7348">knowledge</a> without accessible <a href="http://dbpedia.org/resource/Information" id="link-id140415d0">Information</a>, and there isn't any accessible <a href="http://dbpedia.org/resource/Information" id="link-id11a976e8">Information</a> without accessible Data. The Web has always be grounded in accessibility to data (albeit via compound container documents called Web Pages).</blockquote> <blockquote>Bottom line, RDF based Linked Data is about Open <a href="http://dbpedia.org/resource/Reference_(computer_science)" id="link-id1206bfb8">Data access by reference</a> using URIs (HTTP based <a href="http://dbpedia.org/resource/Entity" id="link-idfaa6ce0">Entity</a> IDs / Data Object IDs / Data Source Names), and as I said earlier, the intrinsic value is pretty obvious bearing in mind the costs associated with integrating disparate and heterogeneous data sources -- across intranets, extranets, and the <a href="http://dbpedia.org/resource/Internet" id="link-id188ecc68">Internet</a>.</blockquote> <br /> <blockquote> <cite>CrunchBase: In his definition of Web 3.0, Nova Spivack proposes that the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id12e2d968">Semantic Web</a>, or Semanti<a href="http://dbpedia.org/resource/C_(programming_language)" id="link-id105744c0">c</a> Web technologies, will be force behind much of the innovation that will occur during Web 3.0. Do you agree with Nova Spivack? What role, if any, do you feel the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id13fa4218">Semantic Web</a> will play in Web 3.0?</cite> </blockquote> <blockquote>Me: I agree with Nova. But I see Web 3.0 as a phase within the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id188c9000">Semantic Web</a> innovation continuum. Web 3.0 exists because Web 2.0 exists. Both of these Web versions express usage and technology focus patterns. Web 2.0 is about the use of Open Source technologies to fashion Web Services that are ultimately used to drive proprietary Software as Service (SaaS) style solutions. Web 3.0 is about the use of "Smart Data Access" to fashion a new generation of Linked Data aware Web Services and solutions that exploit the federated nature of the Web to maximum effect; proprietary branding will simply be conveyed via quality of data (cleanliness, <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id188d2ef8">context</a> fidelity, and comprehension of privacy) exposed by URIs.</blockquote> <p>Here are some examples of the CrunchBase Linked Data <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id122756f8">Space</a>, as projected via our CruncBase Sponger Cartridge:</p> <ol> <li> <a href="http://demo.openlinksw.com/rdfbrowser2/?uri=http%3A%2F%2Fwww.crunchbase.com%2Fcompany%2Famazon" id="link-id13e0fd18">Amazon.com</a> </li> <li> <a href="http://demo.openlinksw.com/rdfbrowser2/?uri=http%3A%2F%2Fwww.crunchbase.com%2Fcompany%2Fmicrosoft" id="link-id13eef9e0">Microsoft</a> </li> <li> <a href="http://demo.openlinksw.com/rdfbrowser2/?uri=http%3A%2F%2Fwww.crunchbase.com%2Fcompany%2Fgoogle" id="link-id13fe47a0">Google</a> </li> <li> <a href="http://demo.openlinksw.com/rdfbrowser2/?uri=http%3A%2F%2Fwww.crunchbase.com%2Fcompany%2Fapple" id="link-id170c73b8">Apple</a> </li> </ol>
2008-08-27T20:35:15-04:00
A quick look at SP2B, the SPARQL Performance Benchmark
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-08-27#1423
2008-08-27T16:03:40Z
<div> <div style="display:none;">A quick look at SP2B, the SPARQL Performance Benchmark</div> <p>I finally got around to running the <a href="http://dbis.informatik.uni-freiburg.de/index.php?project=SP2B" id="link-id17bac628">SP<sup>2</sup>B SPARQL Performance Benchmark</a> on the current <a href="http://virtuoso.openlinksw.com" id="link-id0x1dcaaa48">Virtuoso</a> Open Source Edition, v5.0.8.</p> <p>I ran it with the 5M triples scale, which is the highest scale for which the authors give numbers.</p> <p>I got a run time of 25 minutes for the 12 queries, giving an arithmetic mean of the query time of 125 seconds. This is better than the 800 or so seconds that the authors had measured. Also, Q6 of the set had failed for the authors, but we have since fixed this; the fix is in the v5.0.8 cut.</p> <p>I also tried it with a scale of 25M, but this became I/O bound and took a bit longer. I will try this with v6 and v7 cluster later, which are vastly better at anything I/O bound.</p> <p>The machine was a 2GHz Xeon with 8G RAM. The query text was the one from the authors, with an explicit <code>FROM</code> clause added; the client was the command line Interactive <a href="http://dbpedia.org/resource/SQL" id="link-id0x1be2c808">SQL</a> (iSQL).</p> <p>If one does the test with the default index layout without specifying a graph, things will not work very well. Also, returning the million-row results of these queries over the <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id0x1d7ac018">SPARQL protocol</a> is not practical.</p> <p>I will say something more about SP<sup>2</sup>B when I get to have a closer look.</p> </div>
2008-09-02T09:49:57-04:00
A quick look at SP2B, the SPARQL Performance Benchmark
http://www.openlinksw.com:443/weblog/oerling/?date=2008-08-27#1422
2008-08-27T16:00:07Z
<p>I finally got around to running the <a href="http://dbis.informatik.uni-freiburg.de/index.php?project=SP2B" id="link-id17bac628">SP<sup>2</sup>B SPARQL Performance Benchmark</a> on the current <a href="http://virtuoso.openlinksw.com" id="link-id0x1d2a6838">Virtuoso</a> Open Source Edition, v5.0.8.</p> <p>I ran it with the 5M triples scale, which is the highest scale for which the authors give numbers.</p> <p>I got a run time of 25 minutes for the 12 queries, giving an arithmetic mean of the query time of 125 seconds. This is better than the 800 or so seconds that the authors had measured. Also, Q6 of the set had failed for the authors, but we have since fixed this; the fix is in the v5.0.8 cut.</p> <p>I also tried it with a scale of 25M, but this became I/O bound and took a bit longer. I will try this with v6 and v7 cluster later, which are vastly better at anything I/O bound.</p> <p>The machine was a 2GHz Xeon with 8G RAM. The query text was the one from the authors, with an explicit <code>FROM</code> clause added; the client was the command line Interactive <a href="http://dbpedia.org/resource/SQL" id="link-id0x19e74ce0">SQL</a> (iSQL).</p> <p>If one does the test with the default index layout without specifying a graph, things will not work very well. Also, returning the million-row results of these queries over the <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id0x1c4231a0">SPARQL protocol</a> is not practical.</p> <p>I will say something more about SP<sup>2</sup>B when I get to have a closer look.</p>
2008-09-02T09:49:55-04:00
Nice Quote about Information Architecture & World Wide Web
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-08-27#1421
2008-08-27T14:47:12Z
<p>Even with the marginal degrees of serendipitous discovery that the current document oriented <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> offers, it's still possible to stumble across poignant gems such as this statement from <a href="http://www.inspireux.com/" id="link-id12432e10">InspireUX</a> :</p> <img alt="Image" src="http://www.inspireux.com/wp-content/uploads/67.gif" /> <br /> <br /> <p>The statement above resonates with a lot of my fundamental views about the essence of Web. It also drives right at the core of what we are trying to address with the <a href="http://ode.openlinksw.com" id="link-id121118f8">OpenLink Data Explorer</a> (ODE) which simply isn't about <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1a7fcc00">Linked Data</a> visualization, but the combination of visualization, user interaction, and unobtrusive exposure and exploitation of <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> Entities culled from the existing Web of Linked Documents. ODE consumes and processes URIs or URLs. Thus, as long as the (X)HTML container / host document keeps URIs or URLs in "agent view", ODE will give you the option to interact with the-<a href="http://dbpedia.org/resource/Data">data</a>-behind Web <a href="http://dbpedia.org/resource/Information" id="link-id0x177c1dd0">information</a> resources (e.g., Web Pages, Images, Audio etc..)</p> <p>Do remember, "mission-critical" is no longer a corporate / enterprise theme. The lines of demarcation between the individual and enterprise are blurring at warp speed.</p>
2008-08-27T11:03:39-04:00
Virtuoso, Linked Data, and Linq2Rdf (Update 1)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-08-26#1420
2008-08-26T12:36:05Z
<p>There are many challenges that have dogged attempts to mesh the DBMS & Object Technology realms for years, critical issues include:</p> <ol> <li> <a href="http://dbpedia.org/resource/Data">data</a> access & manipulation impedance arising from Model mismatches between Relational Databases and Object Oriented & Object based Languages</li> <li> Record / Data Object Referencing by ID. </li> </ol> <p>The big deal about <a href="http://dbpedia.org/resourcei/Language_Integrated_Query" id="link-id101df2c0">LINQ</a> has been the singular focus on addressing point 1, in particular.</p> <p>I've already written about the Linq2Rdf effort that meshes the best of .NET with the virtues of the "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id10193ae8">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id17143870">Web</a>". </p> <p>Here is an architecture diagram that seeks to illustrate the powerful data access and manipulation options that the combination of Linq2RDF and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10595ce0">Linked Data</a> deliver:</p> <br /> <img alt="Image" src="http://virtuoso.openlinksw.com/Whitepapers/html/linqtordf/linqtordf2.png" /> <p>What may not have been obvious to most in the past, is the fact that Mapping from Object Models to Relational Models wasn't really the solution to the problem at hand. Instead, the mapping should have been the other way around i.e., Relational to Object Model mapping. The emergence of RDF and <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id11e5a240">RDBMS</a> to RDF mapping technology is what makes this age-old headache addressable in very novel ways.</p> <h3>Related</h3> <ol> <li> <a href="http://virtuoso.openlinksw.com/presentations/RDF_Mapping_Presentation_W3C_workshop3/RDF_Mapping_Presentation_W3C_workshop3.html" id="link-id10a9aa08">RDBMS to RDF Mapping</a> - W3C Workshop Presentation </li> <li> <a href="http://virtuoso.openlinksw.com/presentations/Virtuoso_Relational_to_RDF_Mapping/Virtuoso_Relational_to_RDF_Mapping.html" id="link-id16d47330">Virtuoso RDBMS to RDF Mapping</a> - W3C Rdb2Rdf Incubator Group Presentation </li> <li> <a href="http://virtuoso.openlinksw.com/presentations/Virtuoso_RDF_Views/Virtuoso_RDF_Views_1.html" id="link-id1403c4c8">Creating RDF Views over SQL Data Sources</a> - Technology Tutorial</li> </ol>
2008-08-27T07:51:23.000002-04:00
Configuring Virtuoso for Benchmarking
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-08-25#1419
2008-08-25T14:06:11Z
<div> <div style="display:none;">Configuring Virtuoso for Benchmarking</div> <p>I will here summarize what should be known about running benchmarks with <a href="http://virtuoso.openlinksw.com" id="link-id0xc152cf0">Virtuoso</a>.</p> <h2>Physical Memory</h2> <p>For 8G RAM, in the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set —</p> <blockquote> <code> [Parameters]<br /> ...<br /> NumberOfBuffers = 550000 </code> </blockquote> <p>For 16G RAM, double this—</p> <blockquote> <code> [Parameters]<br /> ...<br /> NumberOfBuffers = 1100000 </code> </blockquote> <h2>Transaction Isolation</h2> <p>For most cases, certainly all <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xb7ba270">RDF</a> cases, <i>Read Committed</i> should be the default transaction isolation. In the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set —</p> <blockquote> <code> [Parameters]<br /> ...<br /> DefaultIsolation = 2 </code> </blockquote> <h2>Multiuser Workload</h2> <p>If <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x1a40f308">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x1e003cf8">JDBC</a>, or similarly connected client applications are used, there must be more <code>ServerThreads</code> available than there will be client connections. In the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set —</p> <blockquote> <code> [Parameters]<br /> ...<br /> ServerThreads = 100 </code> </blockquote> <p>With web clients (unlike ODBC, JDBC, or similar clients), it may be justified to have fewer <code>ServerThreads</code> than there are concurrent clients. The <code>MaxKeepAlives</code> should be the maximum number of expected web clients. This can be more than the <code>ServerThreads</code> count. In the <code>[HTTPServer]</code> stanza of <code>virtuoso.ini</code>, set —</p> <blockquote> <code> [HTTPServer]<br /> ...<br /> ServerThreads = 100 <br /> MaxKeepAlives = 1000 <br /> KeepAliveTimeout = 10 </code> </blockquote> <p> <i><b>Note</b> — The <code>[HTTPServer] ServerThreads</code> are taken from the total pool made available by the <code>[Parameters] ServerThreads</code>. Thus, the <code>[Parameters] ServerThreads</code> should always be at least as large as (and is best set greater than) the <code>[HTTPServer] ServerThreads</code>, and if using the closed-source Commercial Version, should not exceed the licensed thread count.</i> </p> <h2>Disk Use</h2> <p>The basic rule is to use one stripe (file) per distinct physical device (not per file system), using no RAID. For example, one might stripe a database over 6 files (6 physical disks), with an initial size of 60000 pages (the files will grow as needed). </p> <p>For the above described example, in the <code>[Database]</code> stanza of <code>virtuoso.ini</code>, set —</p> <blockquote> <code> [Database]<br /> ...<br /> Striping = 1<br /> MaxCheckpointRemap = 2000000 </code> </blockquote> <p>— and in the <code>[Striping]</code> stanza, on one line per <code>SegmentName</code>, set —</p> <blockquote> <code> [Striping]<br /> ...<br /> Segment1 = 60000 , /virtdev/db/virt-seg1.db = q1 , /data1/db/virt-seg1-str2.db = q2 , /data2/db/virt-seg1-str3.db = q3 , /data3/db/virt-seg1-str4.db = q4 , /data4/db/virt-seg1-str5.db = q5 , /data5/db/virt-seg1-str6.db = q6</code> </blockquote> <p>As can be seen here, each file gets a background IO thread (the <code>= q<i>xxx</i></code> clause). It should be noted that all files on the same physical device should have the same <code>q<i>xxx</i></code> value. This is not directly relevant to the benchmarking scenario above, because we have only one file per device, and thus only one file per IO queue.</p> <h2> <a href="http://dbpedia.org/resource/SQL" id="link-id0xc8b97c0">SQL</a> Optimization</h2> <p>If queries have lots of joins but access little <a href="http://dbpedia.org/resource/Data" id="link-id0x193b2fa8">data</a>, as with the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1b283ca0">Berlin SPARQL Benchmark</a>, the SQL compiler must be told not to look for better plans if the best plan so far is quicker than the compilation time expended so far. Thus, in the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set —</p> <blockquote> <code> [Parameters]<br /> ...<br /> StopCompilerWhenXOverRunTime = 1 </code> </blockquote> </div>
2008-08-25T15:29:06.000036-04:00
Configuring Virtuoso for Benchmarking
http://www.openlinksw.com:443/weblog/oerling/?date=2008-08-25#1418
2008-08-25T14:05:46Z
<p>I will here summarize what should be known about running benchmarks with <a href="http://virtuoso.openlinksw.com" id="link-id0xc53af18">Virtuoso</a>.</p> <h2>Physical Memory</h2> <p>For 8G RAM, in the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set —</p> <blockquote> <code> [Parameters]<br /> ...<br /> NumberOfBuffers = 550000 </code> </blockquote> <p>For 16G RAM, double this—</p> <blockquote> <code> [Parameters]<br /> ...<br /> NumberOfBuffers = 1100000 </code> </blockquote> <h2>Transaction Isolation</h2> <p>For most cases, certainly all <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xc2f07a0">RDF</a> cases, <i>Read Committed</i> should be the default transaction isolation. In the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set —</p> <blockquote> <code> [Parameters]<br /> ...<br /> DefaultIsolation = 2 </code> </blockquote> <h2>Multiuser Workload</h2> <p>If <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0xc1c7178">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0xd16fb40">JDBC</a>, or similarly connected client applications are used, there must be more <code>ServerThreads</code> available than there will be client connections. In the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set —</p> <blockquote> <code> [Parameters]<br /> ...<br /> ServerThreads = 100 </code> </blockquote> <p>With web clients (unlike ODBC, JDBC, or similar clients), it may be justified to have fewer <code>ServerThreads</code> than there are concurrent clients. The <code>MaxKeepAlives</code> should be the maximum number of expected web clients. This can be more than the <code>ServerThreads</code> count. In the <code>[HTTPServer]</code> stanza of <code>virtuoso.ini</code>, set —</p> <blockquote> <code> [HTTPServer]<br /> ...<br /> ServerThreads = 100 <br /> MaxKeepAlives = 1000 <br /> KeepAliveTimeout = 10 </code> </blockquote> <p> <i><b>Note</b> — The <code>[HTTPServer] ServerThreads</code> are taken from the total pool made available by the <code>[Parameters] ServerThreads</code>. Thus, the <code>[Parameters] ServerThreads</code> should always be at least as large as (and is best set greater than) the <code>[HTTPServer] ServerThreads</code>, and if using the closed-source Commercial Version, should not exceed the licensed thread count.</i> </p> <h2>Disk Use</h2> <p>The basic rule is to use one stripe (file) per distinct physical device (not per file system), using no RAID. For example, one might stripe a database over 6 files (6 physical disks), with an initial size of 60000 pages (the files will grow as needed). </p> <p>For the above described example, in the <code>[Database]</code> stanza of <code>virtuoso.ini</code>, set —</p> <blockquote> <code> [Database]<br /> ...<br /> Striping = 1<br /> MaxCheckpointRemap = 2000000 </code> </blockquote> <p>— and in the <code>[Striping]</code> stanza, on one line per <code>SegmentName</code>, set —</p> <blockquote> <code> [Striping]<br /> ...<br /> Segment1 = 60000 , /virtdev/db/virt-seg1.db = q1 , /data1/db/virt-seg1-str2.db = q2 , /data2/db/virt-seg1-str3.db = q3 , /data3/db/virt-seg1-str4.db = q4 , /data4/db/virt-seg1-str5.db = q5 , /data5/db/virt-seg1-str6.db = q6</code> </blockquote> <p>As can be seen here, each file gets a background IO thread (the <code>= q<i>xxx</i></code> clause). It should be noted that all files on the same physical device should have the same <code>q<i>xxx</i></code> value. This is not directly relevant to the benchmarking scenario above, because we have only one file per device, and thus only one file per IO queue.</p> <h2> <a href="http://dbpedia.org/resource/SQL" id="link-id0xc9fa298">SQL</a> Optimization</h2> <p>If queries have lots of joins but access little <a href="http://dbpedia.org/resource/Data" id="link-id0xb4e0aa0">data</a>, as with the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0xb2de990">Berlin SPARQL Benchmark</a>, the SQL compiler must be told not to look for better plans if the best plan so far is quicker than the compilation time expended so far. Thus, in the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set —</p> <blockquote> <code> [Parameters]<br /> ...<br /> StopCompilerWhenXOverRunTime = 1 </code> </blockquote>
2008-08-25T15:29:04-04:00
DBpedia Architecture
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-08-22#1416
2008-08-22T02:50:07Z
<p>Here is a pictorial of <a href="http://dbpedia.org/resource/DBpedia">DBpedia</a>'s <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> Deployment & Data Management architecture:</p> <p> <br /> <img alt="Image" style="max-width: 800px;" src="http://wiki.dbpedia.org/files/virtuoso_linked_data_deployment.png" /> </p> <p>Key points:</p> <p> <a href="http://docs.openlinksw.com:80/virtuoso/rdfapiandsql.html" id="link-id13dcfb98">SPASQL</a> (<a href="http://dbpedia.org/resource/SPARQL">SPARQL</a> extension for <a href="http://dbpedia.org/resource/SQL">SQL</a>) enables the intelligent resource representation request handling and <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier">URI</a> dereferencing, that underlies "Linked Data" (i.e., <a href="http://dbpedia.org/resource/Linked_Data">Hyperdata</a> Linking) to occur in-process.<br /> </p>
2008-08-21T22:50:09.000001-04:00
The Future of the Desktop
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-08-21#1415
2008-08-21T15:26:18Z
<p> <a href="http://www.jasonkolb.com/" id="link-id13ba6d90">Jason Kolb</a> (who <a href="http://www.jasonkolb.com/weblog/2008/08/the-future-of-t.html" id="link-id1524e210">initially</a> nudged me to chime in), and then <a href="http://www.readwriteweb.com/archives/future_of_the_desktop.php" id="link-id13a182c0">ReadWriteWeb</a>, and of course <a href="http://www.twine.com/item/11bshgkbr-1k5/the-future-of-the-desktop" id="link-id13f1e1f0">Nova's Twine about the topic</a>, have collectively started an interesting discussion about <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a>.vNext (3.0 and beyond) under the heading: The Future of the Desktop.</p> <p>My contribution to the developing discourse takes the form of a Q&A session. I've taken the questions posed and provided answers that express my particular points of view: </p> <blockquote> <cite>Q: Is the desktop of the future going to just be a web-hosted version of the same old-fashioned desktop metaphors we have today?</cite> </blockquote> <blockquote>A: No, it's going to be a more <a href="http://www.w3.org/TR/webarch/" id="link-id1524d4a0">Web Architecture</a> aware and compliant variant exposed by appropriate metaphors.</blockquote> <br /> <blockquote> <cite>Q: The desktop of the future is going to be a hosted web service</cite> </blockquote> <blockquote>A: A vessel for exploiting the virtues of the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10827ad0">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id155bc698">Web</a>.</blockquote> <br /> <blockquote> <cite>Q: The Browser is Going to Swallow Up the Desktop</cite> </blockquote> <blockquote>A: Literally, of course not! Metaphorically, of course! And then the Browser metaphor will decomposes into function specific bits of Web interaction amenable to orchestration by its users.</blockquote> <br /> <blockquote> <cite>Q: The focus of the desktop will shift from <a href="http://dbpedia.org/resource/Information" id="link-id1667e2e0">information</a> to attention</cite> </blockquote> <blockquote>A: No! <a href="http://dbpedia.org/resource/Knowledge" id="link-id104bb9c8">Knowledge</a>, <a href="http://dbpedia.org/resource/Information" id="link-id1524dd48">Information</a>, and <a href="http://dbpedia.org/resource/Data">Data</a> sharing courtesy of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10723640">Hyperdata</a> & Hypertext Linking.</blockquote> <br /> <blockquote> <cite>Q: Users are going to shift from acting as librarians to acting as daytraders</cite> </blockquote> <blockquote>A: They were Librarians at Web 1.0, Journalist at Web 2.0, and Analysts in Web 3.0 (i.e, analyze structured and interlinked data), and CEOs in Web 4.0 (i.e. get Agents to do stuff intelligently en route to making decisions).</blockquote> <blockquote> <br /> <cite>Q: The Webtop will be more social and will leverage and integrate collective intelligence</cite> </blockquote> <blockquote>A: The <a href="http://dbpedia.org/resource/Linked_Data" id="link-id13a01ed0">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id106343a8">Web</a> vessel will only require you to fill in your profile (once) and then serendipitous discovery and meshing of relevant data will simply happen (the serendipity quotient will grow in line with <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10560050">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id100f4940">Web</a> density).</blockquote> <br /> <blockquote> <cite>Q: The desktop of the future is going to have powerful semantic search and social search capabilities built-in</cite> </blockquote> <blockquote>A: It is going to be able to "Find" rather than "Search" for stuff courtesy of the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10a18a70">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10a976f0">Web</a>.</blockquote> <br /> <blockquote> <cite> Q: Interactive shared spaces will replace folders</cite> </blockquote> <blockquote>A: Data Spaces and their URIs (Data Source Names) replace everything. You simply choose the exploration metaphor that best suits you space interaction needs.</blockquote> <br /> <blockquote> <cite>Q: The Portable Desktop</cite> </blockquote> <blockquote>A: Ubiquitous Desktop i.e. do the same thing (all answers above) on any device connected to the Web.</blockquote> <br /> <blockquote> <cite>Q: The Smart Desktop</cite> </blockquote> <blockquote>A: Vessels with access to Smart Data (<a href="http://dbpedia.org/resource/Linked_Data" id="link-id1666e4e8">Linked Data</a> + Action driven <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id171d1ff0">Context</a> sprinklings).</blockquote> <br /> <blockquote> <cite>Q: Federated, open policies and permissions</cite> </blockquote> <blockquote>A: More federation for sure, XMPP will become a lot more important, and OAuth will enable resurgence of the federated aspects of the Web and <a href="http://dbpedia.org/resource/Internet" id="link-id100a66a8">Internet</a>.</blockquote> <br /> <blockquote> <cite>Q: The personal cloud</cite> </blockquote> <blockquote>A: <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id104ba580">Personal Data Spaces</a> plugged into Clouds (<a href="http://dbpedia.org/resource/Intranet" id="link-id15bbb970">Intranet</a>, <a href="http://dbpedia.org/resource/Extranet" id="link-id1026d6b0">Extranet</a>, <a href="http://dbpedia.org/resource/Internet" id="link-id140508c8">Internet</a>).</blockquote> <br /> <blockquote> <cite>Q: The WebOS</cite> </blockquote> <blockquote>A: An operating system endowed with traditional Database and Host Operating system functionality such as: RDF Data Model, <a href="http://dbpedia.org/resource/SPARQL" id="link-idd86f48">SPARQL</a> Query Language, <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id13f47268">URI</a> based <a href="http://en.wikipedia.org/wiki/Pointer" id="link-id1055bc78">Pointer mechanism</a>, and HTTP based message Bus.</blockquote> <br /> <blockquote> <cite>Q: Who is most likely to own the future desktop?</cite> </blockquote> <blockquote>A: You! And all you need is a <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id106b79e8">URI</a> (an ID or Data Source Name for "<a href="http://dbpedia.org/resource/Entity" id="link-id133c88a0">Entity</a> You") and a Profile Page (a place where "<a href="http://dbpedia.org/resource/Entity" id="link-id15fa8060">Entity</a> You" is Describe by You).</blockquote> <h3>One Last Thing</h3> <p>You can get a feel for the future desktop by <a href="http://ode.openlinksw.com/#Download" id="link-id165ec048">downloading</a> and then installing the <a href="http://ode.openlinksw.com/" id="link-id13baba38">OpenLink Data Explorer</a> plugin for Firefox, which allows you to switch viewing modes between Web Page and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id13f12410">Linked Data</a> behind the page. :-)</p> <h3>Related</h3> <ul> <li> <a href="http://en.wikipedia.org/wiki/OpenLink_Data_Spaces" id="link-id12496e48">OpenLink Data Spaces</a> </li> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/GetAPersonalURIIn5MinutesOrLess" id="link-id1027f060">Get Yourself a URI in 5 Minutes or Less</a> </li> <li> <a href="http://virtuoso.openlinksw.com/presentations/DataPortability_and_DataSpaces/DataPortability_and_DataSpaces.html" id="link-id10890f70">Linked Data Spaces & Data Portability</a> </li> <li> <a href="http://virtuoso.openlinksw.com/presentations/Creating_Deploying_Exploiting_Linked_Data2/Creating_Deploying_Exploiting_Linked_Data2_TimBL_v3.html" id="link-id137efdf8">Linked Data Conference Keynote</a> (<a href="http://dbpedia.org/resource/RDFa" id="link-id1239d300">RDFa</a> based remix edition that includes vital bits from <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id1317a048">TimBL</a>'s <a href="http://www.linkeddataplanet.com/" id="link-id165f57c8">Linked Data Planet presentation</a>).</li> </ul>
2008-08-21T15:59:25.000001-04:00
Yahoo! and the Linked Data Web in a Nutshell
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-08-19#1414
2008-08-19T19:43:00Z
<p>This automated mail from Yahoo! speaks for itself re. <a href="http://dbpedia.org/resource/Linked_Data" id="link-id133de628">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id1404ff40">Web</a> incomprehension!</p> <blockquote> <cite><p> Greetings! </p> <p> This is an automated email from Yahoo! Application Gallery. Please do not reply to this email message. We regret to inform you that your application '<a href="http://dbpedia.org/resource/Blog" id="link-id1056e6e0">Blog</a> <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id170bde38">Data Space</a>' has been rejected. You can view all your applications <a href="http://gallery.yahoo.com/mypage" id="link-id1045f988">here</a>. Moderator Comments: insufficient info </p> <p> Regards, </p> <p> The Yahoo! Application Gallery Team </p> <p> Your use of Yahoo! Application Gallery is subject to http://docs.yahoo.com/info/terms/ </p> </cite> </blockquote> <h3>Message to Yahoo!:</h3> <p> Why bother? You clearly see the Web in a totally different light to the rest of us.</p> <p>If you want to private label the Web, then fine, just don't park your vehicle in the "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id100a8000">Linked Data</a>" or "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id1085c490">Semantic Web</a>" spots.</p> <p>The Web doesn't need any subjectivity bootstraps or booster-shots, it just needs open access to Structured and <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> via URIs.</p> <h3>Kind of Related</h3> <ul> <li> <a href="http://feeds.feedburner.com/~r/ThinkingSpace/~3/369221488/age-of-google-4-future.html" id="link-id128d6fd8">Yihong Ding's Age of Google (4)</a> </li> </ul>
2008-08-19T17:54:38.000008-04:00
Response to: Whole Data Post (Update 3)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-08-15#1413
2008-08-15T13:06:12Z
<p>This post is in response to <a href="http://www.furia.com" id="link-id107907b8">Glenn McDonald</a>'s post titled: <a href="http://www.furia.com/page.cgi?type=log&id=308" id="link-id13dcf2d0">Whole Data</a>, where he highlights a number of issues relating to "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id1016c1f0">Semantic Web</a>" marketing communications and overall messaging, from his perspective.</p> <p> By coincidence, Glenn and I presented at this month's Cambridge <a href="http://dbpedia.org/resource/Semantic_Web" id="link-idd526f48">Semantic Web</a> Gathering.</p> <p>I've provided a dump of Glenn's issues and my responses below:</p> <h3>Issue - RDF</h3> <ul> <li>Ingenious <a href="http://dbpedia.org/resource/Data">data</a> decomposition idea, but: </li> <li>too low-level; the assembly language of data, where we need Java or <a href="http://dbpedia.org/resource/Ruby_programming_language" id="link-id103f3dd0">Ruby</a> </li> <li>"resource" is not the issue; there's no such thing as "metadata", it's all data; "meta" is a perspective </li> <li>lists need to be effortless, not painful and obscure </li> <li>nodes need to be represented, not just implied; they need types and literals in a more pervasive, integrated way. </li> </ul> <h4>Response:</h4> <p>RDF is a Graph based Data Model it stands for Resource Description Framework. The Metadata data angle comes from it's <a href="http://dbpedia.org/resource/Meta_Content_Framework" id="link-id1690df60">Meta Content Framework (MCF)</a> origins. You can express and serialize data based on the RDF Data Model using: Turtle, N3, TriX, N-Triples, and RDF/XML.</p> <h3>Issue - <a href="http://dbpedia.org/resource/SPARQL" id="link-id10234b38">SPARQL</a> (and Freebase's MQL)</h3> <p>These are just appeasement: <br />- old query paradigm: fishing in dark water with superstitiously tied lures; only works well in carefully stocked lakes <br />- we don't ask questions by defining answer shapes and then hoping they're dredged up whole.</p> <h4>Response:</h4> <p> <a href="http://dbpedia.org/resource/SPARQL" id="link-id16e45e50">SPARQL</a>, <a href="http://www.freebase.com/view/freebase/api" id="link-id13e7d468">MQL</a>, and <a href="http://msdn.microsoft.com/en-us/library/bb387145.aspx" id="link-id1516fbd8">Entity-SQL</a> are Graph Model oriented Query Languages. Query Languages always accompany Database Engines. <a href="http://dbpedia.org/resource/SQL" id="link-id13f8c100">SQL</a> is the Relational Model equivalent. </p> <h3>Issue - <a href="http://dbpedia.org/resource/Linked_Data" id="link-id171dee68">Linked Data</a> </h3> <p>Noble attempt to ground the abstract, but: <br />- <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id1576d5f8">URI</a> dereferencing/namespace/<a href="http://dbpedia.org/resource/Open_world_assumption" id="link-id15f50180">open-world</a> issues focus too much technical attention on cross-source cases where the human issues dwarf the technical ones anyway <br />- <a href="http://dbpedia.org/resource/Friend_of_a_friend" id="link-id105df458">FOAF</a> query over the people in this room? forget it. <br />- link asymmetry doesn't scale <br />- identity doesn't scale <br />- generating RDF from non-graph sources: more appeasement, right where the win from actually converting could be biggest! </p> <h4>Response:</h4> <p>Innovative use of HTTP to deliver "<a href="http://dbpedia.org/resource/Reference_%28computer_science%29" id="link-id13eeab20">Data Access by Reference</a>" to the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id13492610">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id105dfc10">Web</a>.</p> <p>When you have a Data Model, Database Engine, and Query Language, the next thing you need is a Data Access mechanism that provides "<a href="http://dbpedia.org/resource/Reference_(computer_science)" id="link-id100ef2c0">Data Access by Reference</a>". <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id16692e88">ODBC</a> and <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id1699b970">JDBC</a> (amongst others) provide "<a href="http://dbpedia.org/resource/Reference_(computer_science)" id="link-id16034b48">Data Access by Reference</a>" via Data Source Names. <a href="http://dbpedia.org/resource/Linked_Data" id="link-id16690118">Linked Data</a> is about the same thing (URIs are Data Source Names) with the following differences:</p> <ul> <li>Naming is scoped to the <a href="http://dbpedia.org/resource/Entity" id="link-id1195dc48">entity</a> level rather than container level</li> <li>HTTP's use within the data source naming scheme expands the referencability of the Named <a href="http://dbpedia.org/resource/Entity" id="link-id10485760">Entity</a> Descriptions beyond traditional confines such as applications, operating systems, and database engines. </li> </ul> <h3> Issue - <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id104684d0">Giant Global Graph</a> </h3> <p>Hugely motivating and powerful idea, worthy of a superhero (Graphius!), but: <br />- giant and global parts are too hard, and starting global makes every problem harder <br />- local projects become unmanageable in global <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id12497088">context</a> (Cyc, Freebase data-modeling lists...). And my thus my plea, again. Forget "semantic" and "<a href="http://dbpedia.org/resource/World_Wide_Web">web</a>", let's fix the database tech first: <br />- node/arc data-model, path-based exploratory query-model <br />- data-graph applications built easily on top of this common model; building them has to be easy, because if it's hard, they'll be bad <br />- given good database tech, good web data-publishing tech will be trivial! <br />- given good tools for graphs, the problems of uniting them will be only as hard as they have to be.</p> <h4>Response:</h4> <p> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id144466d8">Giant Global Graph</a> is just another moniker for a "Web of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id15c2c738">Linked Data</a>" or "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id14e73520">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10aef200">Web</a>".</p> <p>Multi-Model Database technology that meshes the best of the Graph & Relational Models exist. In a nutshell, this is what <a href="http://virtuoso.openlinksw.com" id="link-id13492e10">Virtuoso</a> is all about and it's existed for a very long time :-)</p> <p> <a href="http://virtuoso.openlinksw.com" id="link-id105a4f58">Virtuoso</a> is also a Virtual DBMS engine (so you can see Heterogeneous Relational Data via Graph Model <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id15845110">Context</a> Lenses). Naturally, it is also a <a href="http://dbpedia.org/resource/Linked_Data" id="link-id109e2c78">Linked Data</a> Deployment platform (or <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1086d650">Linked Data</a> Sever). </p> <p>The issue isn't the "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id107f1ba8">Semantic Web</a>" moniker per se., it's about how <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0xba72818">Linked Data</a> (foundation layer of <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id101dbf50">Semantic Web</a>) gets introduced to users. As I said during the MIT Gathering: "The Web is experienced via Web Browsers primarily, so any enhancement to the Web must be exposed via traditional Web Browsers", which is why we've opted to simply add "View <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> Sources" to the existing set of common Browser options that includes:</p> <ol> <li>View page in rendered form (default)</li> <li>View page source (i.e., how you see the markup behind the page)</li> </ol> <p>By exposing the Linked Data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id15a04b70">Web</a> option as described above, you enable the Web user to knowingly transition from the traditional Rendered (X)HTML page view to the Linked Data View (i.e., structured data behind the page). This simple "User Interaction" tweak makes the notion of exploiting a Structured Web becomes somewhat clearer.</p> <p>The Linked Data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10a187d0">Web</a> isn't a panacea. It's just an addition to the existing Web that enrichens the things you can do with the Web. It's predominance, like any application feature, will be subject to the degrees to which it delivers tangible value or matrializes internal and external opportunity costs.</p> <p>Note: The Web isn't ubiquitous today becuase all it's users groked HTML Markup. It's ubquitity is a function of opportunity costs: there simply came a point in the Web boostrap when nobody could afford the opportunity costs associated with being off the Web. The same thing will play out with Linked Data and the broader <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id10a97330">Semantic Web</a> vision.</p> <b>Links:</b> <ol> <li> <a href="http://virtuoso.openlinksw.com/presentations/Creating_Deploying_Exploiting_Linked_Data2/Creating_Deploying_Exploiting_Linked_Data2_TimBL_v3.html(15)" id="link-id137fc560">Linked Data Journey part of my Linked Data Planet Presentation Remix</a>(from slides 15 to 22 - which include bits from <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id1048a968">TimBL</a>'s presentation)</li> <li> <a href="http://ode.openlinksw.com" id="link-id1667df98">OpenLink Data Explorer</a> </li> <li> <a href="http://ode.openlinksw.com/example.html" id="link-id137ee860">OpenLink Data Explorer Screenshots and examples</a>.</li> </ol>
2008-08-15T18:31:48-04:00
DBpedia 3.1 is now Live!
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-08-13#1411
2008-08-13T17:54:31Z
<a href="http://dbpedia.org/About" id="link-id10afcac0">DBpedia</a> 3.1 is now live. The release highlights are as follows:<br /> <ul> <li>116,7 million triples (27% increase over prior release) </li> <li>better <a href="http://www.mpi-inf.mpg.de/%7Esuchanek/downloads/yago/" id="link-id14095ce0">YAGO</a> mapping (instances associated with YAGO classes)</li> <li>Geo extractor code has been improved and is now run for all 14 languages</li> <li>New (X)HTML based Resource/<a href="http://dbpedia.org/resource/Entity" id="link-id1682b988">Entity</a> Description Page (Example:<a href="http://dbpedia.org/resource/Linked_Data" id="link-id1045d8c0"> Linked Data</a>)</li> </ul>Enjoy!<br /> <br />
2008-08-14T08:15:47-04:00
BSBM With Triples and Mapped Relational Data
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-08-06#1410
2008-08-06T19:41:50Z
<div> <div style="display:none;">BSBM With Triples and Mapped Relational Data</div> <p>The special contribution of the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id10039db0">Berlin SPARQL Benchmark</a> (<a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id106b2538">BSBM</a>) to the <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id101a75f8">RDF</a> world is to raise the question of doing OLTP with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xae54170">RDF</a>.</p> <p>Of course, here we immediately hit the question of comparisons with relational databases. To this effect, <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1e847b08">BSBM</a> also specifies a relational schema and can generate the <a href="http://dbpedia.org/resource/Data" id="link-id1206c378">data</a> as either triples or <a href="http://dbpedia.org/resource/SQL" id="link-id1667f040">SQL</a> inserts.</p> <p>The benchmark effectively simulates the case of exposing an existing <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id10a93518">RDBMS</a> as RDF. <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id13e46d80">OpenLink Software</a> calls this <i>RDF Views</i>. <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id12027578">Oracle</a> is beginning to call this <i>semantic covers</i>. The <a href="http://www.w3.org/2005/Incubator/rdb2rdf/" id="link-id161dc678">RDB2RDF XG</a>, a W3C incubator group, has been active in this area since Spring, 2008.</p> <h3>But why an OLTP workload with RDF to begin with?</h3> <p>We believe this is relevant because RDF promises to be the interoperability factor between potentially all of traditional IS. If <a href="http://dbpedia.org/resource/Data" id="link-id0x1e7119d8">data</a> is online for human consumption, it may be online via a <a href="http://dbpedia.org/resource/SPARQL" id="link-id106a8908">SPARQL</a> end-point as well. The economic justification will come from discoverability and from applications integrating multi-source structured data. Online shopping is a fine use case.</p> <p>Warehousing all the world's publishable data as RDF is not our first preference, nor would it be the publisher's. Considerations of duplicate infrastructure and maintenance are reason enough. Consequently, we need to show that mapping can outperform an RDF warehouse, which is what we'll do here.</p> <h3>What We Got </h3> <p>First, we found that <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1400" id="link-id150ea748">making the query plan took much too long</a> in proportion to the run time. With BSBM this is an issue because the queries have lots of joins but access relatively little data. So we made a faster compiler and along the way retouched the cost model a bit.</p> <p>But the really interesting part with BSBM is mapping relational data to RDF. For us, BSBM is a great way of showing that mapping can outperform even the best triple store. A relational row store is as good as unbeatable with the query mix. And when there is a clear mapping, there is no reason the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xae5aff0">SPARQL</a> could not be directly translated.</p> <p>If Chris Bizer et al launched the mapping ship, we will be the ones to pilot it to harbor!</p> <p>We filled two <a href="http://virtuoso.openlinksw.com" id="link-id12dbdc70">Virtuoso</a> instances with a BSBM200000 data set, for 100M triples. One was filled with physical triples; the other was filled with the equivalent relational data plus mapping to triples. Performance figures are given in "query mixes per hour". (An update or follow-on to this post will provide elapsed times for each test run.)</p> <p>With the unmodified benchmark we got:</p> <blockquote> <table> <tr> <td><i>Physical Triples:</i> </td> <td> </td> <td>1297 qmph</td> </tr> <tr> <td><i>Mapped Triples:</i> </td> <td> </td> <td><b>3144 qmph</b> </td> </tr> </table> </blockquote> <p>In both cases, most of the time was spent on Q6, which looks for products with one of three words in the label. We altered Q6 to use text index for the mapping, and altered the databases accordingly. (There is no such thing as an e-commerce site without a text index, so we are amply justified in making this change.)</p> <p>The following were measured on the second run of a 100 query mix series, single test driver, warm cache.</p> <blockquote> <table> <tr> <td><i>Physical Triples:</i> </td> <td> </td> <td> 5746 qmph</td> </tr> <tr> <td><i>Mapped Triples:</i> </td> <td> </td> <td> <b>7525 qmph</b> </td> </tr> </table> </blockquote> <p>We then ran the same with 4 concurrent instances of the test driver. The qmph here is 400 / the longest run time.</p> <blockquote> <table> <tr> <td><i>Physical Triples:</i> </td> <td> </td> <td> 19459 qmph</td> </tr> <tr> <td><i>Mapped Triples:</i> </td> <td> </td> <td> <b>24531 qmph</b> </td> </tr> </table> </blockquote> <p>The system used was 64-bit Linux, 2GHz dual-Xeon 5130 (8 cores) with 8G RAM. The concurrent throughputs are a little under 4 times the single thread throughput, which is normal for SMP due to memory contention. The numbers do not evidence significant overhead from thread synchronization.</p> <p>The query compilation represents about 1/3 of total server side CPU. In an actual online application of this type, queries would be parameterized, so the throughputs would be accordingly higher. We used the <code>StopCompilerWhenXOverRunTime = 1</code> option here to cut needless compiler overhead, the queries being straightforward enough.</p> <p>We also see that the advantage of mapping can be further increased by more compiler optimizations, so we expect in the end mapping will lead RDF warehousing by a factor of 4 or so.</p> <h3>Suggestions for BSBM</h3> <ul> <li> <p> <b>Reporting Rules.</b> The benchmark spec should specify a form for disclosure of test run data, TPC style. This includes things like configuration parameters and exact text of queries. There should be accepted variants of query text, as with the TPC.</p> </li> <li> <p> <b>Multiuser operation.</b> The test driver should get a stream number as parameter, so that each client makes a different query sequence. Also, disk performance in this type of benchmark can only be reasonably assessed with a naturally parallel multiuser workload.</p> </li> <li> <p> <b>Add business intelligence.</b> SPARQL has aggregates now, at least with <a href="http://jena.sourceforge.net/" id="link-id11a25ac0">Jena</a> and <a href="http://virtuoso.openlinksw.com" id="link-id0xb003180">Virtuoso</a>, so let's use these. The BSBM business intelligence metric should be a separate metric off the same data. Adding synthetic sales figures would make more interesting queries possible. For example, producing recommendations like "customers who bought this also bought xxx."</p> </li> <li> <p> <b>For the SPARQL community</b>, BSBM sends the message that one ought to support parameterized queries and stored procedures. This would be a <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id109e2448">SPARQL protocol</a> extension; the SPARUL syntax should also have a way of calling a procedure. Something like <code>select proc (??, ??)</code> would be enough, where <code>??</code> is a parameter marker, like <code>?</code> in <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id13febf48">ODBC</a>/<a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id120416a8">JDBC</a>.</p> </li> <li> <p> <b>Add transactions.</b>Especially if we are contrasting mapping vs. storing triples, having an update flow is relevant. In practice, this could be done by having the test driver send web service requests for order entry and the SUT could implement these as updates to the triples or a mapped relational store. This could use stored procedures or logic in an app server.</p> </li> </ul> <h3>Comments on Query Mix</h3> <p>The time of most queries is less than linear to the scale factor. Q6 is an exception if it is not implemented using a text index. Without the text index, Q6 will inevitably come to dominate query time as the scale is increased, and thus will make the benchmark less relevant at larger scales.</p> <h2>Next</h2> <p>We include the sources of our RDF view definitions and other material for running BSBM with our forthcoming Virtuoso Open Source 5.0.8 release. This also includes all the query optimization work done for BSBM. This will be available in the coming days.</p> </div>
2008-08-06T16:29:44.000003-04:00
BSBM With Triples and Mapped Relational Data
http://www.openlinksw.com:443/weblog/oerling/?date=2008-08-06#1409
2008-08-06T19:35:27Z
<p>The special contribution of the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id10039db0">Berlin SPARQL Benchmark</a> (<a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id106b2538">BSBM</a>) to the <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id101a75f8">RDF</a> world is to raise the question of doing OLTP with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xb230eb0">RDF</a>.</p> <p>Of course, here we immediately hit the question of comparisons with relational databases. To this effect, <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0xa832da8">BSBM</a> also specifies a relational schema and can generate the <a href="http://dbpedia.org/resource/Data" id="link-id1206c378">data</a> as either triples or <a href="http://dbpedia.org/resource/SQL" id="link-id1667f040">SQL</a> inserts.</p> <p>The benchmark effectively simulates the case of exposing an existing <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id10a93518">RDBMS</a> as RDF. <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id13e46d80">OpenLink Software</a> calls this <i>RDF Views</i>. <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id12027578">Oracle</a> is beginning to call this <i>semantic covers</i>. The <a href="http://www.w3.org/2005/Incubator/rdb2rdf/" id="link-id161dc678">RDB2RDF XG</a>, a W3C incubator group, has been active in this area since Spring, 2008.</p> <h3>But why an OLTP workload with RDF to begin with?</h3> <p>We believe this is relevant because RDF promises to be the interoperability factor between potentially all of traditional IS. If <a href="http://dbpedia.org/resource/Data" id="link-id0xabe48a0">data</a> is online for human consumption, it may be online via a <a href="http://dbpedia.org/resource/SPARQL" id="link-id106a8908">SPARQL</a> end-point as well. The economic justification will come from discoverability and from applications integrating multi-source structured data. Online shopping is a fine use case.</p> <p>Warehousing all the world's publishable data as RDF is not our first preference, nor would it be the publisher's. Considerations of duplicate infrastructure and maintenance are reason enough. Consequently, we need to show that mapping can outperform an RDF warehouse, which is what we'll do here.</p> <h3>What We Got </h3> <p>First, we found that <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1400" id="link-id150ea748">making the query plan took much too long</a> in proportion to the run time. With BSBM this is an issue because the queries have lots of joins but access relatively little data. So we made a faster compiler and along the way retouched the cost model a bit.</p> <p>But the really interesting part with BSBM is mapping relational data to RDF. For us, BSBM is a great way of showing that mapping can outperform even the best triple store. A relational row store is as good as unbeatable with the query mix. And when there is a clear mapping, there is no reason the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x96bb5e0">SPARQL</a> could not be directly translated.</p> <p>If Chris Bizer et al launched the mapping ship, we will be the ones to pilot it to harbor!</p> <p>We filled two <a href="http://virtuoso.openlinksw.com" id="link-id12dbdc70">Virtuoso</a> instances with a BSBM200000 data set, for 100M triples. One was filled with physical triples; the other was filled with the equivalent relational data plus mapping to triples. Performance figures are given in "query mixes per hour". (An update or follow-on to this post will provide elapsed times for each test run.)</p> <p>With the unmodified benchmark we got:</p> <blockquote> <table> <tr> <td><i>Physical Triples:</i> </td> <td> </td> <td>1297 qmph</td> </tr> <tr> <td><i>Mapped Triples:</i> </td> <td> </td> <td><b>3144 qmph</b> </td> </tr> </table> </blockquote> <p>In both cases, most of the time was spent on Q6, which looks for products with one of three words in the label. We altered Q6 to use text index for the mapping, and altered the databases accordingly. (There is no such thing as an e-commerce site without a text index, so we are amply justified in making this change.)</p> <p>The following were measured on the second run of a 100 query mix series, single test driver, warm cache.</p> <blockquote> <table> <tr> <td><i>Physical Triples:</i> </td> <td> </td> <td> 5746 qmph</td> </tr> <tr> <td><i>Mapped Triples:</i> </td> <td> </td> <td> <b>7525 qmph</b> </td> </tr> </table> </blockquote> <p>We then ran the same with 4 concurrent instances of the test driver. The qmph here is 400 / the longest run time.</p> <blockquote> <table> <tr> <td><i>Physical Triples:</i> </td> <td> </td> <td> 19459 qmph</td> </tr> <tr> <td><i>Mapped Triples:</i> </td> <td> </td> <td> <b>24531 qmph</b> </td> </tr> </table> </blockquote> <p>The system used was 64-bit Linux, 2GHz dual-Xeon 5130 (8 cores) with 8G RAM. The concurrent throughputs are a little under 4 times the single thread throughput, which is normal for SMP due to memory contention. The numbers do not evidence significant overhead from thread synchronization.</p> <p>The query compilation represents about 1/3 of total server side CPU. In an actual online application of this type, queries would be parameterized, so the throughputs would be accordingly higher. We used the <code>StopCompilerWhenXOverRunTime = 1</code> option here to cut needless compiler overhead, the queries being straightforward enough.</p> <p>We also see that the advantage of mapping can be further increased by more compiler optimizations, so we expect in the end mapping will lead RDF warehousing by a factor of 4 or so.</p> <h3>Suggestions for BSBM</h3> <ul> <li> <p> <b>Reporting Rules.</b> The benchmark spec should specify a form for disclosure of test run data, TPC style. This includes things like configuration parameters and exact text of queries. There should be accepted variants of query text, as with the TPC.</p> </li> <li> <p> <b>Multiuser operation.</b> The test driver should get a stream number as parameter, so that each client makes a different query sequence. Also, disk performance in this type of benchmark can only be reasonably assessed with a naturally parallel multiuser workload.</p> </li> <li> <p> <b>Add business intelligence.</b> SPARQL has aggregates now, at least with <a href="http://jena.sourceforge.net/" id="link-id11a25ac0">Jena</a> and <a href="http://virtuoso.openlinksw.com" id="link-id0xa83f490">Virtuoso</a>, so let's use these. The BSBM business intelligence metric should be a separate metric off the same data. Adding synthetic sales figures would make more interesting queries possible. For example, producing recommendations like "customers who bought this also bought xxx."</p> </li> <li> <p> <b>For the SPARQL community</b>, BSBM sends the message that one ought to support parameterized queries and stored procedures. This would be a <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id109e2448">SPARQL protocol</a> extension; the SPARUL syntax should also have a way of calling a procedure. Something like <code>select proc (??, ??)</code> would be enough, where <code>??</code> is a parameter marker, like <code>?</code> in <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id13febf48">ODBC</a>/<a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id120416a8">JDBC</a>.</p> </li> <li> <p> <b>Add transactions.</b>Especially if we are contrasting mapping vs. storing triples, having an update flow is relevant. In practice, this could be done by having the test driver send web service requests for order entry and the SUT could implement these as updates to the triples or a mapped relational store. This could use stored procedures or logic in an app server.</p> </li> </ul> <h3>Comments on Query Mix</h3> <p>The time of most queries is less than linear to the scale factor. Q6 is an exception if it is not implemented using a text index. Without the text index, Q6 will inevitably come to dominate query time as the scale is increased, and thus will make the benchmark less relevant at larger scales.</p> <h2>Next</h2> <p>We include the sources of our RDF view definitions and other material for running BSBM with our forthcoming Virtuoso Open Source 5.0.8 release. This also includes all the query optimization work done for BSBM. This will be available in the coming days.</p>
2015-08-20T17:15:22.760513-04:00
.NET, LINQ, and RDF based Linked Data (Update 2)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-08-03#1408
2008-08-03T16:07:09Z
<p>At OpenLink, we've been investigating <a href="http://code.google.com/p/linqtordf/" id="link-id1296eb18">LinqToRdf</a>, an exciting project from <a href="http://aabs.wordpress.com/" id="link-id13e860a8">Andrew Matthews</a> that seeks to expose the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id105d84f8">Semantic Web</a> technology space to the large community of .NET developers. </p> <p>The LinqToRdf project is about binding LINQ to RDF. It sits atop <a href="http://razor.occams.info/" id="link-id102e3b10">Joshua Tauberer</a>'s <a href="http://dbpedia.org/resource/C_(programming_language)" id="link-id1471b0d0">C</a># based <a href="http://razor.occams.info/code/semweb/" id="link-id14cb9030">Semantic Web/RDF library</a> which has been out there for a while and works across Microsoft .NET and it's open source variant "Mono".</p> <p>Historically, the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id13ee9f40">Semantic Web</a> realm has been dominated by RDF frameworks such as <a href="http://www.openrdf.org/" id="link-id109f8a68">Sesame</a>, <a href="http://jena.sourceforge.net/" id="link-id144c3210">Jena</a> and <a href="http://librdf.org/" id="link-id10600228">Redland</a>; which by their Open Source orientation, predominantly favor non-Windows platforms (Java and Linux). Conversely, Microsoft's .NET frameworks have sought to offer Conceptualization technology for heterogeneous Logical <a href="http://dbpedia.org/resource/Data">Data</a> Sources via .NET's <a href="http://en.wikipedia.org/wiki/ADO.NET_Entity_Framework" id="link-id10726628">Entity Frameworks</a> and <a href="http://dbpedia.org/resource/ADO.NET" id="link-id13e7edd8">ADO.NET</a>, but without any actual bindings to RDF. </p> <p>Interestingly, believe it or not, .NET already has a data query language that shares a number of similarities with <a href="http://dbpedia.org/resource/SPARQL" id="link-id1042f480">SPARQL</a>, called <a href="http://dbpedia.org/resource/Entity" id="link-id105a46b0">Entity</a>-<a href="http://dbpedia.org/resource/SQL" id="link-id1041d2e8">SQL</a>, and a very innovative programming language called LINQ; that offers a blend of constructs for natural data access and manipulation across relational (<a href="http://dbpedia.org/resource/SQL" id="link-id139f5848">SQL</a>), hierarchical (XML), and graph (Object) models without the traditional object language->database impedance tensions of the past.</p> <p>With regards to all of the above, we've just released a mini white paper that covers the exploitation of <a href="http://virtuoso.openlinksw.com/Whitepapers/html/linqtordf/linqtordf1.htm" id="link-id14b2f138">RDF-based Linked Data using .NET via LINQ</a>. The paper offers a an overview of LinqToRdf, plus enhancements we've contributed to the project (available in <a href="http://aabs.wordpress.com/2008/08/01/announcing-linqtordf-v08/" id="link-id101defa8">LinqToRdf v0.8</a>.). The paper includes real-world examples that tap into a MusicBrainz powered <a href="http://dbpedia.org/resource/Linked_Data" id="link-id101ffd18">Linked Data</a> <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id105cb858">Space</a>, the Music Ontology, the <a href="http://virtuoso.openlinksw.com" id="link-id13f55860">Virtuoso</a> RDF Quad Store, <a href="http://virtuoso.openlinksw.com" id="link-id12826718">Virtuoso</a> <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id1030cb60">Sponger</a> Middleware, and our RDfization Cartridges for Musicbrainz. </p> Enjoy!
2008-08-08T08:54:01.000002-04:00
Virtuoso's Universal Server Architecture (Conceptual & Technical)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-08-03#1406
2008-08-03T13:07:12Z
As they say, a picture speaks a thousand words, so I am exposing two views of <a href="http://virtuoso.openlinksw.com" id="link-id13fe7df8">Virtuoso</a> that have been on the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> for while. <br /> <br />Remember, <a href="http://virtuoso.openlinksw.com" id="link-id13f53ed0">Virtuoso</a> offers <a href="http://dbpedia.org/resource/Data">data</a> management, data access, web <a href="http://dbpedia.org/resource/Application_server" id="link-id109f04b0">application server</a>, enterprise service bus, and virtualization of disparate and heterogeneous data sources, as part of a single, multi threaded, cross-platform server solution; hence it's description as a "<a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id104d2e48">Universal Server</a>".<br /> <br />Conceptual View:<br /> <br /> <img alt="Image" style="max-width: 800px;" src="http://virtuoso.openlinksw.com/images/vconc650.jpg" /> <br /> <br />Technical View (kinda missing <a href="http://dbpedia.org/resource/PHP" id="link-id10660110">PHP</a>, <a href="http://dbpedia.org/resource/Perl" id="link-id1053d9b8">Perl</a>, <a href="http://dbpedia.org/resource/Python_programming_language" id="link-id107bc9c0">Python</a> runtime hosting in the Virtual Application Sever realm):<br /> <br /> <img alt="Image" style="max-width: 800px;" src="http://virtuoso.openlinksw.com/images/virtuoso3arch.gif" /> <br /> <br /> <br /> <a href="http://virtuoso.openlinksw.com" id="link-id0x13cf3798">Virtuoso</a>'s architecture is not a reaction to current trends. The diagrams above are pretty old (with minor touch ups in recent times). At <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id13e194c0">OpenLink Software</a>, we've have a consistent world-view re. standards and the vital role they play when it comes to developing software that enables the construction and exploitation of "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id133c84a8">Context</a> Lenses" that tap into a substrate of Virtualized Logical Data Sources (<a href="http://dbpedia.org/resource/SQL" id="link-id104d1c30">SQL</a>, XML, RDF, Web Services, Full Text etc.).<br /> <br /> <br /> <br /> <br />
2008-08-05T18:07:45-04:00
Time for Context Lenses (Update)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-08-02#1405
2008-08-02T19:06:57Z
As the <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id13dfe618">Linked Data meme</a> continues on it's quest to unravel the mysteries of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id10527b30">Semantic Web</a> vision, it's quite gratifying to see that <a href="http://dbpedia.org/resource/Federated_database_system" id="link-id104f58b0">data virtualization</a> comprehension: creating "Conceptual Views" into logically organized "Disparate & Heterogeneous <a href="http://dbpedia.org/resource/Data">Data</a> Sources" via "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id14a46998">Context</a> Lenses" is taking shape, as illustrated in the "<a href="http://feeds.feedburner.com/%7Er/SemanticBusiness/%7E3/353668031/note-to-self-virtualconceptual-as-wwwsw.html" id="link-id13179dd8">note-to-self</a>" post by <a href="http://www.linkedin.com/in/davidprovost" id="link-id1403dc88">David Provost</a>.<br /> <br /> <br /> <br /> <br />Virtualization of heterogeneous data sources is only achievable if you have a dexterous data model based "Bus" into which the data sources are plugged. RDF has offered such a model for a long time.<br /> <br /> <br /> <br /> <img alt="Image" style="max-width: 800px;" src="http://www.w3.org/DesignIssues/diagrams/sw-clients.png" /> <br /> <br /> <br /> <br />When heterogeneous data sources are plugged into an RDF based integration bus e.g., customer records sourced from a variety of tables, across a plethora of databases, you can only end up with true value if the emergent entities from such an effort are coherently linked and (de)referencable; which is what <a href="http://dbpedia.org/resource/Linked_Data" id="link-id12b06e20">Linked Data</a>'s fundamental preoccupation with dereferencable URIs is all about. Of course, Even when you have all of the above in place, you also need to be able to construct "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id103c2c80">Context</a> Lenses" i.e., <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id1037a260">context</a> driven views of the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id13e48ab8">Linked Data</a> Mesh (or <a href="http://dbpedia.org/resource/Linked_Data" id="link-id101c7718">Linked Data</a> Spaces).<br /> <br /> <br />Additional Diagrams:<br /> <br /> <br />1. <a href="http://www.w3.org/2005/Talks/1110-iswc-tbl/#%2824%29" id="link-id10808cb8">Clients of the RDF Bus</a> <br />2. <a href="http://www.w3.org/2005/Talks/1110-iswc-tbl/#%2825%29" id="link-id11e5a300">RDF Bus Server plugins: Scripts that emit RDF</a> <br />3. <a href="http://www.w3.org/2005/Talks/1110-iswc-tbl/#%2826%29" id="link-id13ea46a0">RDF Bus Servers: RDF Data Managers (Triple or Quad Stores)</a> <br />4. <a href="http://www.w3.org/2005/Talks/1110-iswc-tbl/#%2827%29" id="link-id101d3470">RDF Bus Servers: Relational to RDF Mappers (RDF Views, Semantic Covers etc.)</a> <br />5. <a href="http://www.w3.org/2005/Talks/1110-iswc-tbl/#%2828%29" id="link-id1052c450">RDF Bus Server plugins: XML to RDF Mappers </a> <br />6. <a href="http://www.w3.org/2005/Talks/1110-iswc-tbl/#%2829%29" id="link-id10281ec0">RDF Bus Server plugins: GRDDL based XSLT stylesheets that emit RDF</a> <br />7. <a href="http://www.w3.org/2005/Talks/1110-iswc-tbl/#%2830%29" id="link-id1444faf0">RDF Bus Server plugins: Intelligent RDF Middleware</a> <br /> <br /> <br /> <br /> <br /> <br /> <br />
2008-08-04T11:24:50.000001-04:00
Exploiting the RDF-based Linked Data Web using .NET via LINQ
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-08-01#1403
2008-08-01T17:58:19Z
<div> <div style="display:none;">Exploiting the RDF-based Linked Data Web using .NET via LINQ</div> Recently OpenLink has been investigating <a href="http://code.google.com/p/linqtordf/" id="link-id0x20d8a248">LinqToRdf</a>, an exciting project from <a href="http://aabs.wordpress.com" id="link-id0x21f48218">Andrew Matthews</a> which aims to bring the Semantic Web to .NET. Because of their language bindings and heritage, existing RDF APIs such as Sesame, Jena and Redland predominantly favour non-Windows clients. Conversely Microsoft's ADO.NET Data Services provides a Redmond vision of exposing data on the Web but has no support for RDF. LinqToRdf is, as far as we're aware, the first serious effort to fill this gap and provide a bridge between Windows applications and the Semantic Web.<br /> <br />OpenLink has produced a whitepaper <a href="http://virtuoso.openlinksw.com/Whitepapers/html/linqtordf/linqtordf1.htm" id="link-id0x21f47348">Exploiting the RDF-based Linked Data Web using .NET via LINQ</a> which provides a brief overview of LinqToRdf and an example of its use to retrieve data from the <a href="http://musicbrainz.org" id="link-id0x21f49a88">MusicBrainz</a> music metadatabase via an OpenLink Virtuoso Quad Store. The document also illustrates the use of the <a href="http://virtuoso.openlinksw.com/Whitepapers/pdf/sponger_whitepaper_10102007.pdf" id="link-id0x21f92758">Virtuoso Sponger</a>, an "RDFizer" forming part of the RDF toolset provided with OpenLink Virtuoso Universal Server, to convert the raw MusicBrainz data to RDF on-the-fly. A further aim of the whitepaper is to draw attention to Andrew's excellent effort and hopefully tempt members of the Semantic Web community to contribute.<br /> <br />Andrew was kind enough to incorporate some changes into LinqToRdf in response to OpenLink's testing. These have been included with major improvements of his own in a new release - <a href="http://aabs.wordpress.com/2008/08/01/announcing-linqtordf-v08/" id="link-id0x21f465b8">LinqToRdf v0.8</a>.<br /> <br />Carl Blakeley<br /> </div>
2008-08-01T13:58:19.000004-04:00
Linked Data, Meshups, Twitter, and Friendfeed
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-08-01#1402
2008-08-01T02:11:31Z
<p>Here are some links from my Friendfeed and Twitter <a href="http://dbpedia.org/resource/Data">Data</a> Spaces that expose a number of recent <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0xa91a798">Linked Data</a> "Meshup" examples:</p> <ul> <li> <a href="http://friendfeed.com/kidehen" id="link-id101740a8">Friendfeed</a> </li> <li> <a href="http://twitter.com/kidehen" id="link-id104baa80">Twitter</a> </li> </ul> <p>Enjoy!</p>
2008-07-31T22:17:35-04:00
Virtuoso Optimizations for the Berlin SPARQL Benchmark
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-07-30#1401
2008-07-30T18:52:11Z
<div> <div style="display:none;">Virtuoso Optimizations for the Berlin SPARQL Benchmark </div> <p>We had a look at Chris Bizer's initial results with the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id105c9f78">Berlin SPARQL Benchmark</a> (<a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id102d62b0">BSBM</a>) on <a href="http://virtuoso.openlinksw.com" id="link-id13eb9780">Virtuoso</a>. The first results were rather bad, as nearly all of the run time was spent optimizing the <a href="http://dbpedia.org/resource/SPARQL" id="link-id14a51258">SPARQL</a> statements and under 10% actually running them.</p> <p>So I spent a couple of days on the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xaad28d0">SPARQL</a>/<a href="http://dbpedia.org/resource/SQL" id="link-id108745b0">SQL</a> compiler, to the effect of making it do a better guess of initial execution plan and streamlining some operations. In fact, many of the queries in <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0xaa230b8">BSBM</a> are not particularly sensitive to execution plan, as they access a very small portion of the database. So to close the matter, I put in a flag that makes the <a href="http://dbpedia.org/resource/SQL" id="link-id0x1e9e8e28">SQL</a> compiler give up on devising new plans if the time of the best plan so far is less than the time spent compiling so far.</p> <p>With these changes, available now as a diff on top of 5.0.7, we run quite well, several times better than initially. With the compiler time cut-off in place (ini parameter <code>StopCompilerWhenXOverRunTime = 1</code>), we get the following times, output from the BSBM test driver:</p> <blockquote> <pre> Starting test... 0: 1031.22 ms, total: 1151 ms 1: 982.89 ms, total: 1040 ms 2: 923.27 ms, total: 968 ms 3: 898.37 ms, total: 932 ms 4: 855.70 ms, total: 865 ms Scale factor: 10000 Number of query mix runs: 5 times min/max Query mix runtime: 0.8557 s / 1.0312 s Total runtime: 4.691 seconds QMpH: 3836.77 query mixes per hour CQET: 0.93829 seconds average runtime of query mix CQET (geom.): 0.93625 seconds geometric mean runtime of query mix Metrics for Query 1: Count: 5 times executed in whole run AQET: 0.012212 seconds (arithmetic mean) AQET(geom.): 0.009934 seconds (geometric mean) QPS: 81.89 Queries per second minQET/maxQET: 0.00684000s / 0.03115700s Average result count: 7.0 min/max result count: 3 / 10 Metrics for Query 2: Count: 35 times executed in whole run AQET: 0.030490 seconds (arithmetic mean) AQET(geom.): 0.029776 seconds (geometric mean) QPS: 32.80 Queries per second minQET/maxQET: 0.02467300s / 0.06753000s Average result count: 22.5 min/max result count: 15 / 30 Metrics for Query 3: Count: 5 times executed in whole run AQET: 0.006947 seconds (arithmetic mean) AQET(geom.): 0.006905 seconds (geometric mean) QPS: 143.95 Queries per second minQET/maxQET: 0.00580000s / 0.00795100s Average result count: 4.0 min/max result count: 0 / 10 Metrics for Query 4: Count: 5 times executed in whole run AQET: 0.008858 seconds (arithmetic mean) AQET(geom.): 0.008829 seconds (geometric mean) QPS: 112.89 Queries per second minQET/maxQET: 0.00804400s / 0.01019500s Average result count: 3.4 min/max result count: 0 / 10 Metrics for Query 5: Count: 5 times executed in whole run AQET: 0.087542 seconds (arithmetic mean) AQET(geom.): 0.087327 seconds (geometric mean) QPS: 11.42 Queries per second minQET/maxQET: 0.08165600s / 0.09889200s Average result count: 5.0 min/max result count: 5 / 5 Metrics for Query 6: Count: 5 times executed in whole run AQET: 0.131222 seconds (arithmetic mean) AQET(geom.): 0.131216 seconds (geometric mean) QPS: 7.62 Queries per second minQET/maxQET: 0.12924200s / 0.13298200s Average result count: 3.6 min/max result count: 3 / 5 Metrics for Query 7: Count: 20 times executed in whole run AQET: 0.043601 seconds (arithmetic mean) AQET(geom.): 0.040890 seconds (geometric mean) QPS: 22.94 Queries per second minQET/maxQET: 0.01984400s / 0.06012600s Average result count: 26.4 min/max result count: 5 / 96 Metrics for Query 8: Count: 10 times executed in whole run AQET: 0.018168 seconds (arithmetic mean) AQET(geom.): 0.016205 seconds (geometric mean) QPS: 55.04 Queries per second minQET/maxQET: 0.01097600s / 0.05066900s Average result count: 12.8 min/max result count: 6 / 20 Metrics for Query 9: Count: 20 times executed in whole run AQET: 0.043813 seconds (arithmetic mean) AQET(geom.): 0.043807 seconds (geometric mean) QPS: 22.82 Queries per second minQET/maxQET: 0.04274900s / 0.04504100s Average result count: 0.0 min/max result count: 0 / 0 Metrics for Query 10: Count: 15 times executed in whole run AQET: 0.030697 seconds (arithmetic mean) AQET(geom.): 0.029651 seconds (geometric mean) QPS: 32.58 Queries per second minQET/maxQET: 0.02072000s / 0.03975700s Average result count: 1.1 min/max result count: 0 / 4 real 0 m 5.485 s user 0 m 2.233 s sys 0 m 0.170 s </pre></blockquote> <p>Of the approximately 5.5 seconds of running five query mixes, the test driver spends 2.2 s. The server side processing time is 3.1 s, of which SQL compilation is 1.35 s. The rest is miscellaneous system time. The measurement is on 64-bit Linux, 2GHz dual-Xeon 5130 (8 cores) with 8G RAM. </p> <p>We note that this type of workload would be done with stored procedures or prepared, parameterized queries in the SQL world.</p> <p>There will be some further tuning still but this addresses the bulk of the matter. There will be a separate message about the patch containing these improvements.</p> </div>
2008-08-06T16:29:42-04:00
Virtuoso Optimizations for the Berlin SPARQL Benchmark
http://www.openlinksw.com:443/weblog/oerling/?date=2008-07-30#1400
2008-07-30T18:17:54Z
<p>We had a look at Chris Bizer's initial results with the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id105c9f78">Berlin SPARQL Benchmark</a> (<a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id102d62b0">BSBM</a>) on <a href="http://virtuoso.openlinksw.com" id="link-id13eb9780">Virtuoso</a>. The first results were rather bad, as nearly all of the run time was spent optimizing the <a href="http://dbpedia.org/resource/SPARQL" id="link-id14a51258">SPARQL</a> statements and under 10% actually running them.</p> <p>So I spent a couple of days on the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xa5a8d0e8">SPARQL</a>/<a href="http://dbpedia.org/resource/SQL" id="link-id108745b0">SQL</a> compiler, to the effect of making it do a better guess of initial execution plan and streamlining some operations. In fact, many of the queries in <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0xaf04af8">BSBM</a> are not particularly sensitive to execution plan, as they access a very small portion of the database. So to close the matter, I put in a flag that makes the <a href="http://dbpedia.org/resource/SQL" id="link-id0x1e8d2360">SQL</a> compiler give up on devising new plans if the time of the best plan so far is less than the time spent compiling so far.</p> <p>With these changes, available now as a diff on top of 5.0.7, we run quite well, several times better than initially. With the compiler time cut-off in place (ini parameter <code>StopCompilerWhenXOverRunTime = 1</code>), we get the following times, output from the BSBM test driver:</p> <blockquote> <pre> Starting test... 0: 1031.22 ms, total: 1151 ms 1: 982.89 ms, total: 1040 ms 2: 923.27 ms, total: 968 ms 3: 898.37 ms, total: 932 ms 4: 855.70 ms, total: 865 ms Scale factor: 10000 Number of query mix runs: 5 times min/max Query mix runtime: 0.8557 s / 1.0312 s Total runtime: 4.691 seconds QMpH: 3836.77 query mixes per hour CQET: 0.93829 seconds average runtime of query mix CQET (geom.): 0.93625 seconds geometric mean runtime of query mix Metrics for Query 1: Count: 5 times executed in whole run AQET: 0.012212 seconds (arithmetic mean) AQET(geom.): 0.009934 seconds (geometric mean) QPS: 81.89 Queries per second minQET/maxQET: 0.00684000s / 0.03115700s Average result count: 7.0 min/max result count: 3 / 10 Metrics for Query 2: Count: 35 times executed in whole run AQET: 0.030490 seconds (arithmetic mean) AQET(geom.): 0.029776 seconds (geometric mean) QPS: 32.80 Queries per second minQET/maxQET: 0.02467300s / 0.06753000s Average result count: 22.5 min/max result count: 15 / 30 Metrics for Query 3: Count: 5 times executed in whole run AQET: 0.006947 seconds (arithmetic mean) AQET(geom.): 0.006905 seconds (geometric mean) QPS: 143.95 Queries per second minQET/maxQET: 0.00580000s / 0.00795100s Average result count: 4.0 min/max result count: 0 / 10 Metrics for Query 4: Count: 5 times executed in whole run AQET: 0.008858 seconds (arithmetic mean) AQET(geom.): 0.008829 seconds (geometric mean) QPS: 112.89 Queries per second minQET/maxQET: 0.00804400s / 0.01019500s Average result count: 3.4 min/max result count: 0 / 10 Metrics for Query 5: Count: 5 times executed in whole run AQET: 0.087542 seconds (arithmetic mean) AQET(geom.): 0.087327 seconds (geometric mean) QPS: 11.42 Queries per second minQET/maxQET: 0.08165600s / 0.09889200s Average result count: 5.0 min/max result count: 5 / 5 Metrics for Query 6: Count: 5 times executed in whole run AQET: 0.131222 seconds (arithmetic mean) AQET(geom.): 0.131216 seconds (geometric mean) QPS: 7.62 Queries per second minQET/maxQET: 0.12924200s / 0.13298200s Average result count: 3.6 min/max result count: 3 / 5 Metrics for Query 7: Count: 20 times executed in whole run AQET: 0.043601 seconds (arithmetic mean) AQET(geom.): 0.040890 seconds (geometric mean) QPS: 22.94 Queries per second minQET/maxQET: 0.01984400s / 0.06012600s Average result count: 26.4 min/max result count: 5 / 96 Metrics for Query 8: Count: 10 times executed in whole run AQET: 0.018168 seconds (arithmetic mean) AQET(geom.): 0.016205 seconds (geometric mean) QPS: 55.04 Queries per second minQET/maxQET: 0.01097600s / 0.05066900s Average result count: 12.8 min/max result count: 6 / 20 Metrics for Query 9: Count: 20 times executed in whole run AQET: 0.043813 seconds (arithmetic mean) AQET(geom.): 0.043807 seconds (geometric mean) QPS: 22.82 Queries per second minQET/maxQET: 0.04274900s / 0.04504100s Average result count: 0.0 min/max result count: 0 / 0 Metrics for Query 10: Count: 15 times executed in whole run AQET: 0.030697 seconds (arithmetic mean) AQET(geom.): 0.029651 seconds (geometric mean) QPS: 32.58 Queries per second minQET/maxQET: 0.02072000s / 0.03975700s Average result count: 1.1 min/max result count: 0 / 4 real 0 m 5.485 s user 0 m 2.233 s sys 0 m 0.170 s </pre></blockquote> <p>Of the approximately 5.5 seconds of running five query mixes, the test driver spends 2.2 s. The server side processing time is 3.1 s, of which SQL compilation is 1.35 s. The rest is miscellaneous system time. The measurement is on 64-bit Linux, 2GHz dual-Xeon 5130 (8 cores) with 8G RAM. </p> <p>We note that this type of workload would be done with stored procedures or prepared, parameterized queries in the SQL world.</p> <p>There will be some further tuning still but this addresses the bulk of the matter. There will be a separate message about the patch containing these improvements.</p>
2008-08-06T16:29:37.000003-04:00
WUPnP Cheatsheet
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-07-28#1397
2008-07-29T03:37:55Z
<p> <a href="http://vanirsystems.com/danielsblog/2008/07/28/wupnp-cheatsheet/#comments" id="link-id133b9048">WUPnP Cheatsheet</a>: "</p> <p>The <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id0x1ccdfc68">Web</a> Universal Plug and Play (<acronym title="Web Universal Plug and Play">WUPnP</acronym>) Cheatsheet:</p> <p> <img src="http://vanirsystems.com/images/wupnparch.png" alt="Web Universal Plug and Play (WUPnP) Cheatsheet" width="50%" /> </p> <p>Essentially, if you build an application and use the technologies suggested in the ‘glue section’ then your <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id0x1d626818">web</a> application/service (whether it’s front-end or back-end) will fit into many many other <a href="http://dbpedia.org/resource/World_Wide_Web">web</a> applications/services… and therefore also more manageable for the future! This is WUPnP.</p> <p>Key technologies for making your services/applications as sticky as possible:</p> <ul> <li> <a href="http://en.wikipedia.org/wiki/Dereferenceable_Uniform_Resource_Identifier" id="link-id1016cbd8">Dereferenceable URI’s</a> (which indicate <a href="http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol" id="link-id101d7790">HTTP</a> networking)</li> <li> <a href="http://en.wikipedia.org/wiki/OpenID" id="link-id12ea5e68">OpenID</a> </li> <li> <a href="http://en.wikipedia.org/wiki/OAuth" id="link-id12e1acf0">OAuth</a> </li> <li> <a href="http://en.wikipedia.org/wiki/SPARQL" id="link-id133d34e0">SPARQL</a> </li> <li> <a href="http://en.wikipedia.org/wiki/Linked_Data" id="link-id106a7040">Linked Data</a> <a href="http://en.wikipedia.org/wiki/Resource_Description_Framework" id="link-id1072d890">RDF</a> (or <a href="http://en.wikipedia.org/wiki/RDFa" id="link-id14040c38">RDFa</a>) and <a href="http://en.wikipedia.org/wiki/Web_Ontology_Language" id="link-id1044dda8">OWL</a> </li> </ul> <p>Web-based plug and play fun!</p>" <p>(Via <a href="http://vanirsystems.com/danielsblog" id="link-id10a15838">Daniel Lewis</a>.)</p>
2008-07-29T13:06:40-04:00
CrunchBase gets hooked up with the Linked Data Web!
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-07-25#1395
2008-07-25T14:01:01Z
<p>It's getting really hot in <a href="http://dbpedia.org/resource/Linked_Data" id="link-id15eea8f8">Linked Data</a> land! Two days ago <a href="http://bnode.org/about" id="link-id107e2f70">Benjamin Nowack</a> pinged the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id11b93670">LOD</a> community about his <a href="http://lists.w3.org/Archives/Public/public-lod/2008Jul/0110.html" id="link-id1022a270">RDFization of Crunchbase</a> (sample (X)HTML view: http://cb.semsol.org/company/opera-software) courtesy of Crounchbase releasing an API. As you know, I've always equated <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> Service API to Database CLIs (<a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id16327528">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id1027f410">JDBC</a>, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id10683850">ADO</a>.<a href="http://dbpedia.org/resource/ADO.NET" id="link-id13beb9b8">NET</a> etc.) as both offer code level hooks into <a href="http://dbpedia.org/resource/Data">Data</a> Spaces.</p> <p></p> <p></p> <p>Naturally, we've decided to join the Crunchbase RDFization party, and have just completed a <a href="http://virtuoso.openlinksw.com" id="link-id10282208">Virtuoso</a> <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id10acf0f8">Sponger</a> Cartridge (an RDFizer) for Crouncbase. What we add in our particular cartridge is additional meshing with <a href="http://dbpedia.org/resource/DBpedia" id="link-id115e2a98">DBpedia</a> and Wikicompany <a href="http://dbpedia.org/resource/Linked_Data" id="link-id132f0568">Linked Data</a> Spaces, plus RDFizaton of the Crunchbase (X)HTML pages :-)</p> <p></p> <p></p> <p>As I've postulated for a while, <a href="http://dbpedia.org/resource/Linked_Data" id="link-id13304010">Linked Data</a> is about data "Meshing" and "Meshups". This isn't a buzzword play. I am pointing out an important distinction between "Mashups" and "Meshpus". Which goes as follows: "Mashups" are about code level joining devoid of structured modelling, hence the revelation of code as opposed to data when you look behind a "Mashup". "Meshups" on the other hand, are about joining disparate structured data sources across the Web. And when you look behind a "Meshup" you see structured data (preferably <a href="http://dbpedia.org/resource/Linked_Data" id="link-id143bdb68">Linked Data</a>) that enables further "Meshing".</p> <p></p> <p></p> <p>I truly believe that we are now inches away from critical mass re. <a href="http://dbpedia.org/resource/Linked_Data" id="link-id14829640">Linked Data</a>, and because we are dealing with data, the network-effect will be sky-high! I shudder to think about the state of the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1c41d150">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id133364e8">Web</a> in 12 months time. Yes, I am giving the explosion 12 months (or less). These are very exciting times.</p> <p></p> <p></p> <p>Demo Links:</p> <p></p> <p></p> <ul> <li> <a href="http://demo.openlinksw.com/ode/?uri%5B%5D=http%3A%2F%2Fcb.semsol.org%2Fcompany%2Fopera-software&" id="link-id12fe1dc8">Opera Software via Benjee's Linked Data Space for Cunchbase</a> </li> <li> <a href="http://demo.openlinksw.com/ode/?uri=http%3A%2F%2Fwww.crunchbase.com%2Fcompany%2Fopera-software" id="link-id10739a18">Opera Software via our Linked Data Space for Crunchbas</a> </li> </ul> <p></p> <p></p> <p>For best experience I encourage you to look at the <a href="https://addons.mozilla.org/en-US/firefox/addon/8062" id="link-id1499a0f8">OpenLink Data Explorer extension</a> for Firefox (2.x - 3.x). This enables you to go to Crunchbase (X)HTML pages (and other sites on the Web of course), and then simply use the "View | <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> Sources" main or <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id10051b50">context</a> menu sequence to unveil the Linked Data Sources associated with any Web Page.</p> <p></p> <p>Of course there is much more to come!</p>
2008-07-29T21:43:27-04:00
Twine Opens Up Linked Data Style!
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-07-25#1394
2008-07-25T02:18:00Z
<p>Note to Nova: big time welcome to the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10a424f0">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id13351960">Web</a>!</p> <p>I've just digested <a href="http://www.twine.com/item/1w3ckhq8-997" id="link-id103a38b0">Nova</a>'s post <a href="http://novaspivack.typepad.com/nova_spivacks_weblog/2008/07/subscribe-to-my.html" id="link-id105be838">announcing the opening up of Twine</a>. My test was simple, I opened up his Twine page using Firefox (with the new <a href="https://addons.mozilla.org/en-US/firefox/addon/8062" id="link-id103d2148">OpenLink Data Explorer extension</a> in place), and then simply executed the following browser menu sequence:</p> <ol> <li>View</li> <li> <a href="http://dbpedia.org/resource/Linked_Data" id="link-id102b7880">Linked Data</a> Sources</li> </ol> <p>And voila! The Twine page morphs into a <a href="http://dbpedia.org/resource/Linked_Data" id="link-id137b5cd0">Linked Data</a> <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id10467260">Space</a> where each <a href="http://dbpedia.org/resource/Entity" id="link-id16296328">entity</a> presented is endowed with <a href="http://intranet.usnet.private:8893/RPC2" id="link-id1053b2f0">dereferencable URIs</a>; enabling me to traverse and/or <a href="http://demo.openlinksw.com/ode/?uri%5B%5D=http%3A%2F%2Fwww.twine.com%2Ftwine%2F1p2dqhdx-1jg%2Fnova-spivack-my-public-twine&uri%5B%5D=http%3A%2F%2Fwww.twine.com%2Fitem%2F1w3ckhq8-997&uri%5B%5D=http%3A%2F%2Fmyopenlink.net%2Fdataspace%2Fperson%2Fkidehen&" id="link-id106eb2a0">Mesh</a> his data with other <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> Spaces such as <a href="http://myopenlink.net/dataspace/person/kidehen" id="link-id12e51748">mine</a>, for instance.</p> <p>This is what Linked Data is all about! The fun has only just begun :-)</p>
2008-07-29T21:52:27.000004-04:00
Virtuoso 5.0.7 Release, Now With Jena and Sesame APIs
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-07-17#1393
2008-07-17T17:18:09Z
<div> <div style="display:none;">Virtuoso 5.0.7 Release, Now With Jena and Sesame APIs</div> <h2>Improvements</h2> <ul> <li> <a href="http://docs.openlinksw.com:80/virtuoso/rdfnativestorageproviders.html" id="link-id13e54d98">Full operation</a> with <a href="http://jena.sourceforge.net/" id="link-id0x11a3d360">Jena</a> and <a href="http://sourceforge.net/projects/sesame/" id="link-id0x1108d428">Sesame</a> <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1288aa00">RDF</a> Frameworks. This fully replaces any previous attempts at interop, and introduces samples and test suites.</li> <li>Better support for alternate RDF indexing schemes</li> <li>Parallel operation of the RDF Sponger, importing multiple sources concurrently.</li> <li>New <a href="http://dbpedia.org/resource/Data" id="link-id0x128a9810">data</a> formats supported for on-demand RDF-ization in the Sponger</li> <li>More efficient support for inference of subclass and sub-property; now capable of efficiently handling taxonomies of tens of thousands of classes</li> <li> <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x6af0678">OWL</a> <a href="http://docs.openlinksw.com:80/virtuoso/rdfsparqlrule.html#rdfsparqlruleintro" id="link-id104d58d8">equivalentClass and equivalentProperty</a> support.</li> <li> <a href="http://docs.openlinksw.com:80/virtuoso/rdfdatarepresentation.html#rdfdynamiclocal" id="link-id109606a8">Dynamic IRI host part</a> support for mapped data and for metadata of local resources. Renaming the host or using multiple virtual hosts will accept URIs with the right host part and refer to the same thing, no duplicate storage required.</li> <li> <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x12e0cc38">SPARQL</a> optimizations for <code>LIMIT</code> and <code>OFFSET</code> </li> </ul> <h2>Documentation</h2> <ul> <li> <a href="http://docs.openlinksw.com:80/virtuoso/perfdiag.html#perfdiagqueryplans" id="link-id10a56dd0">How to read query plans and how to use the key performance meters</a> </li> <li> <a href="http://docs.openlinksw.com:80/virtuoso/rdfperformancetuning.html#rdfperfcost" id="link-id106cb5c0">How to diagnose SPARQL queries and how to decide what indexing scheme is right for each RDF use case</a> </li> <li>How to debug RDF views</li> <ul> <li> <a href="http://docs.openlinksw.com:80/virtuoso/sparqldebug.html" id="link-id133b4420">Better documentation of SPARQL extensions and options</a> </li> <li> <a href="http://docs.openlinksw.com:80/virtuoso/rdfviews.html#rdfviewnorthwindexample1" id="link-id1060fdd8">A sample of correct RDF view usage with the Northwind demo data</a> </li> </ul> </ul> <h2>Bug Fixes</h2> <ul> <li>Generally improved safety of built-in functions, better argument checking.</li> <li>Verified UTF8 international character support in all RDF use cases, <a href="http://dbpedia.org/resource/SQL" id="link-id0x12839fd0">SQL</a> client/<a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id0x1288f350">SPARQL protocol</a>/all data formats.</li> </ul> </div>
2008-07-17T15:28:22.000002-04:00
Virtuoso 5.0.7 Release, Now With Jena and Sesame APIs
http://www.openlinksw.com:443/weblog/oerling/?date=2008-07-17#1392
2008-07-17T17:16:19Z
<h2>Improvements</h2> <ul> <li> <a href="http://docs.openlinksw.com:80/virtuoso/rdfnativestorageproviders.html" id="link-id13e54d98">Full operation</a> with <a href="http://jena.sourceforge.net/" id="link-id0x11839970">Jena</a> and <a href="http://sourceforge.net/projects/sesame/" id="link-id0x118521a0">Sesame</a> <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x11e14758">RDF</a> Frameworks. This fully replaces any previous attempts at interop, and introduces samples and test suites.</li> <li>Better support for alternate RDF indexing schemes</li> <li>Parallel operation of the RDF Sponger, importing multiple sources concurrently.</li> <li>New <a href="http://dbpedia.org/resource/Data" id="link-id0x13661868">data</a> formats supported for on-demand RDF-ization in the Sponger</li> <li>More efficient support for inference of subclass and sub-property; now capable of efficiently handling taxonomies of tens of thousands of classes</li> <li> <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x9df079b8">OWL</a> <a href="http://docs.openlinksw.com:80/virtuoso/rdfsparqlrule.html#rdfsparqlruleintro" id="link-id104d58d8">equivalentClass and equivalentProperty</a> support.</li> <li> <a href="http://docs.openlinksw.com:80/virtuoso/rdfdatarepresentation.html#rdfdynamiclocal" id="link-id109606a8">Dynamic IRI host part</a> support for mapped data and for metadata of local resources. Renaming the host or using multiple virtual hosts will accept URIs with the right host part and refer to the same thing, no duplicate storage required.</li> <li> <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x110e7688">SPARQL</a> optimizations for <code>LIMIT</code> and <code>OFFSET</code> </li> </ul> <h2>Documentation</h2> <ul> <li> <a href="http://docs.openlinksw.com:80/virtuoso/perfdiag.html#perfdiagqueryplans" id="link-id10a56dd0">How to read query plans and how to use the key performance meters</a> </li> <li> <a href="http://docs.openlinksw.com:80/virtuoso/rdfperformancetuning.html#rdfperfcost" id="link-id106cb5c0">How to diagnose SPARQL queries and how to decide what indexing scheme is right for each RDF use case</a> </li> <li>How to debug RDF views</li> <ul> <li> <a href="http://docs.openlinksw.com:80/virtuoso/sparqldebug.html" id="link-id133b4420">Better documentation of SPARQL extensions and options</a> </li> <li> <a href="http://docs.openlinksw.com:80/virtuoso/rdfviews.html#rdfviewnorthwindexample1" id="link-id1060fdd8">A sample of correct RDF view usage with the Northwind demo data</a> </li> </ul> </ul> <h2>Bug Fixes</h2> <ul> <li>Generally improved safety of built-in functions, better argument checking.</li> <li>Verified UTF8 international character support in all RDF use cases, <a href="http://dbpedia.org/resource/SQL" id="link-id0x11140c28">SQL</a> client/<a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id0x110947e8">SPARQL protocol</a>/all data formats.</li> </ul>
2008-07-17T15:28:20-04:00
Response to: Where's the Killer Semantic Web Application (Update #2)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-06-26#1391
2008-06-26T18:28:14Z
<p>As is often the case these days, it's much easier to drop a <a href="http://dbpedia.org/resource/Blog" id="link-id13519b98">blog</a> post than it is to make a simple comment in an "old media" style<a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id134e92c8"> data space </a>:-(</p> <blockquote> <p> <cite>My use of "old media" implies: a place that still seeks subscriber data (no OpenID etc..), for the umpteenth time, as the toll fee for discourse development and participation on the Web.</cite> </p> </blockquote> <p>Anyway, here is what I attempted to post as a comment to Dan Grigorovici's post titled: <a href="http://www.semanticweb.com/article.php/12160_3753806_2" id="link-id134dfb80">Where is the Semantic Web Killer App?</a> </p> <p>Dan,</p> <p>An intriguing post to say the least :-) </p> <p>"<a href="http://dbpedia.org/resource/Linked_Data" id="link-id134265c0">Linked Data</a>" and "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id133d7048">Semantic Web</a>" aren't synonymous, they are simply connected, infrastructure DNA-wise. You can have "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id1096cb70">Semantic Web</a>" style graphs (i.e RDF Data) and not have "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id133f0f48">Linked Data</a>" as per <a href="http://dbpedia.org/resource/Linked_Data" id="link-id134fc7c0">Linked Data</a> deployment tenets and best practices, a very important point.</p> <p>I've stated repeatedly, the "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id134f03e8">Linked Data</a>" emphasis has more to do with focusing on a point of crystallization within the larger "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id134104f0">Semantic Web</a>" vision, so here is a quick recap:</p> <h3>What is <a href="http://dbpedia.org/resource/Linked_Data" id="link-id133decd0">Linked Data</a>?</h3> <p>A term coined by <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id1340dd28">TimBL</a> that describes an application of HTTP to the time-tested process of "<a href="http://dbpedia.org/resource/Reference_(computer_science)">Data Access by Reference</a>". "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id10695c20">Linked Data</a>" adds vital items to the "Data Access by Reference" pattern that have been erstwhile unattainable:</p> <ul> <li> The use of a Data Source Naming scoped to Database / Data Container Records as opposed to Tables, Views, Stored Procedures, Databases, and other Record Container tuple collections. Example: in <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-idd9c8af8">ODBC</a> / <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id130b0df0">JDBC</a>, a Data Source Name's scope stops at the Table / View level. In the <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> realm you get an added layer of granularity due to record level name scope</li> <li>Incorporation of HTTP into the Data Source Naming scheme, which injects the expanse of the Web into the Data Access Range of the Data Source Name (i.e. a Named Record); so you can reference a record's description directly via HTTP which is simply a major deal (to put things mildly).</li> </ul> <p>So we have HTTP based URIs as the Data Sources Names for a "Linked Data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id1044a2d8">Web</a>" i.e a Web of inter-connected Data Source Names that de-emphasize the importance of their host containers (Compound Documents / <a href="http://dbpedia.org/resource/Information" id="link-id134e0d38">Information</a> Resources).</p> <p>The business case or value proposition of "Linked Data" is synonymous with the value proposition of data access technologies such as <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id13400500">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id134f0250">JDBC</a>. <a href="http://dbpedia.org/resource/ADO.NET" id="link-id10923840">ADO</a>.<a href="http://dbpedia.org/resource/ADO.NET" id="link-id134ef878">NET</a>, OLE-DB, XMLA, and others (enterprise or consumer) in relation to the Individual and Enterprise pursuit of agility; in a realm where data is growing exponentially, and the maximum processing time in a single day remains 24 hrs. Data Access & Data Integration are timeless challenges due to the following constants:</p> <ul> <li> Structured Data Schema Heterogeneity - we will always model the same things differently</li> <li>Dirtiness of Data within Structured Data Containers - we are error prone due to laziness / sloppiness, time constraints, and the inherent limitation of our DNA based CPUs when dealing with large volumes of data.</li> </ul> <p> Note: The line between the Enterprise & Individuals continue to blur by the second, this is something I covered during my <a href="http://virtuoso.openlinksw.com/presentations/Creating_Deploying_Exploiting_Linked_Data2/Creating_Deploying_Exploiting_Linked_Data2.html" id="link-id13479488">Linked Data Planet keynote</a>, which is like most things I put on the Web (via this <a href="http://dbpedia.org/resource/Blog" id="link-id130ac870">blog</a> <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id10923ba8">data space</a>), is a live and practical demonstration of the virtues of Linked Data courtesy of <a href="http://dbpedia.org/resource/RDFa" id="link-id133fd270">RDFa</a>, the <a href="http://bibliontology.com/" id="link-id134248d8">Bibliographic Ontology</a>, and dereferencable URIs (i.e. HTTP based Data Source Names for Documents and the Entities they host).</p> <h3>Related</h3> <ul> <li> <a href="http://zitgist.com/labs/linked_data.html" id="link-id104778a8">Linked Data FAQ</a> - by <a href="http://community.linkeddata.org/dataspace/person/mkbergman#this" id="link-idd8e71b0">Mike Bergman</a> </li> <li> <a href="http://www.wired.com/special_multimedia/2008/st_infoporn_1607" id="link-id134c1e80">The Planetary Computer</a> from <a href="http://www.wired.com" id="link-id13416518">Wired Magazine</a> - which is basically the effect of Linked Data under a different label (note to Wired: "Tired of old Media repetitive Registrations" when seeking to make comments in the OpenID era!).</li> </ul>
2008-07-19T15:50:46-04:00
Metcalfe, Einstein, and Linked Data
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-06-23#1390
2008-06-23T20:48:30Z
<p>Metcalfe’s law states that the value of a telecommunications network is proportional to the square of the number of users of the system (n²), where the linkages between users (nodes) exist by definition. For <a href="http://dbpedia.org/resource/Information" id="link-id106b0c10">information</a> bases, the <a href="http://dbpedia.org/resource/Data">data</a> objects are the nodes. <a href="http://dbpedia.org/resource/Linked_Data" id="link-id13fc5940">Linked Data</a> works to add the connections between the nodes.</p> <p>I would tweak of the law modification expressed in <a href="http://community.linkeddata.org/dataspace/person/mkbergman#this" id="link-id1401cce0">Mike Bergman</a>'s <a href="http://feeds.feedburner.com/%7Er/AI3_AdaptiveInformation/%7E3/318146056/" id="link-id104fc870">post</a> which states:</p> <blockquote> <cite>the value of a <a href="http://dbpedia.org/resource/Linked_Data" id="link-id14b74530">Linked Data</a> network is proportional to the square of the number of links between the data objects.</cite> </blockquote> By simply injecting "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id10ca1eb8">Context</a>" which is what a high fidelity <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x9c3d088">linked data</a> mesh facilitates i.e. a mesh of weighted links endowed with specifically typed links (as opposed to a single ambiguous type unspecific link), you end up with an even more insight into the power of a <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10cebfc8">Web</a>. <h3>Channeling Einstein</h3> <p>How about <a href="http://dbpedia.org/resource/Albert_Einstein" id="link-id1115dd38">Einstein</a>'s famous equaton: <a href="http://en.wikipedia.org/wiki/Mass-energy_equivalence" id="link-id1a783ab8">E=mc<sup>2</sup></a>? I am talking Energy (vitality) and Mass equivalence, where "E" is for Energy, "m" for Network Mesh base Mass ( where each <a href="http://dbpedia.org/resource/Entity" id="link-id0x97482d0">entity</a> network node contains sub-particles that are themselves dense network meshes all endowed with typed links and weightings), and "<a href="http://dbpedia.org/resource/C_%28programming_language%29" id="link-id10b632c0">c</a>" is for computer processing speed (processing speed is growing exponentially!). When you beam queries down a <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id14398d00">context</a> rich mesh (a <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10acf280">giant global graph</a> comprised of named and dereferencable data sources), especially a mesh to which we are all connected, what do you get? Infrastructure for generating an unbelievable amount of intellectual energy (the result of exploding the sub-data-graphs within graph nodes) that is much better equipped to handle current and future challenges. Even better, we end up making constructive use of Einstein's findings (remember, we built a bomb the first time around!). <a href="http://www.w3.org/People/Berners-Lee/" id="link-id10ece0b8">TimBL</a> articulates this fundamental value of the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> in slightly different language, but at the core, this is the essence of the Web as I believe he envisioned; the ability to connect us all in such a way that we exploit our collective manpower and <a href="http://dbpedia.org/resource/Knowledge" id="link-id1300bf58">knowledge</a> constructively and unobtrusively, en route to making the world a much better place :-)</p> <p>Note: None of this in incongruent with being compensated (i.e. making money) for contributing tangible value into, or around, the Mesh we know as the Web :-)</p> <h3>Related</h3> <ul> <li> <a href="http://dbpedia.org/resource/Business_@_the_Speed_of_Thought" id="link-id1095d330">Business at the Speed of Thought</a> - by <a href="http://dbpedia.org/resource/Bill_Gates" id="link-id14043c50">Bill Gates</a> </li> <li> <a href="http://www.gladwell.com/blink/index.html" id="link-id1043a4d0">Blink</a> - by <a href="http://dbpedia.org/resource/Malcolm_Gladwell" id="link-id13825918">Malcolm Gladwell</a> </li> </ul>
2008-09-02T13:03:01-04:00
A Simple Linked Data Guide for the Enterprise
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-06-23#1389
2008-06-23T19:29:16Z
<p> <a href="http://community.linkeddata.org/dataspace/person/mkbergman#this" id="link-id12dd9d88">Mike Bergman</a> has just published a nice <a href="http://feeds.feedburner.com/~r/AI3_AdaptiveInformation/~3/318146056/" id="link-id10b96a98">Linked Data FAQ</a> aimed at Enterprise audiences. His post draws on a collection of questions collated from a plethora of interactions with Enterprise oriented folks during last week's <a href="http://www.linkeddataplanet.com" id="link-id12276c00">Linked Data Planet conference</a>.</p> <p>Enjoy!</p>
2008-06-23T16:54:29-04:00
What do people have against URLs or URIs? (Updated)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-06-22#1388
2008-06-22T22:36:14Z
<p>Stumbled across a nice post titled: <a href="http://derivadow.com/2008/06/22/what-do-people-have-against-urls" id="link-id10c035c8">What do people have against URLs</a>?. My answer: Everything, if they don't understand the inherent power of URLs when incorporated into the "<a href="http://dbpedia.org/resource/Data">Data</a> Source Naming" mechanism of the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> called: URIs :-)</p> <p>URIs are simple to use i.e you simply click on them via a user agents UI. However, URLs when incorporated into Data Source Naming en route to constructing HTTP based Identifiers, that deliver HTTP based pointers to the location / address of a Resource Descriptions, another matter.</p> <p>I touched on this issue in my <a href="http://virtuoso.openlinksw.com/presentations/Creating_Deploying_Exploiting_Linked_Data2/Creating_Deploying_Exploiting_Linked_Data2.html" id="link-id1076e998">Linked Data Planet keynote</a> last week, and I must say, it did set off a light.</p> <p>I believe, we can only get the broader Web community to comprehend the utility of URIs (Web Data Source Names) by exposing said utility via the Web's Universal Client (Web Browser). For instance, how do URN based Identity / Naming schemes help in a world dominated by Web Browsers that only grok "http://"? From my vantage point, the practical solution is for data providers who already have "doi", "lsid" and other Handle based Identifiers in place, to embark upon http-to-native-naming-scheme-proxying.</p> <p>In my usual "dog-fooding" and "practice what you preach" fashion, this is exactly what we do in the new <a href="http://myopenlink.net:8890/~kidehen/Public/rdfb.xpi" id="link-id13038bb0">Linked Data Web extension</a> that we've decided to reveal to the public (albeit late beta). Thus, when you use an existing browser to view pages with "lsid" or "doi" URNs, you still enjoy the utility of getting at the "Raw <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1090f2a0">Linked Data</a> Sources" that these names expose.</p>
2008-06-23T09:37:57.000003-04:00
My Linked Data Planet Keynote (Updated with missing link)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-06-19#1387
2008-06-19T05:25:00Z
<p>I've finally found a second to drop a note about my keynote.</p> <p>The keynote: <a href="http://virtuoso.openlinksw.com/presentations/Creating_Deploying_Exploiting_Linked_Data2/Creating_Deploying_Exploiting_Linked_Data2.html" id="link-id103acfb8">Creating, Deploying, and Exploiting Linked Data</a>, sought to achieve the fundamental goal of: Demystify the concept of "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id107134e8">Linked Data</a>" using anecdotal material that resonates with enterprise decision makers.</p> <p>To my pleasure, 90% of the audience members confirmed familiarization with the "<a href="http://dbpedia.org/resource/Data">Data</a> Source Name" concept of <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id106d97a8">Open Database Connectivity</a> (<a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id10956268">ODBC</a>). Thus, all I had to do was map "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id10a55728">Linked Data</a>" to <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id10e77210">ODBC</a>, and then unveil the fundamental add-ons that "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id10d1d290">Linked Data</a>" delivers:</p> <ul> <li>The ability to give database records names (Identifiers)</li> <li>The use of HTTP in the database record naming mechanism - which expands a named database record's reference scope via the expanse of the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> (i.e HTTP based Identifiers called URIs).</li> </ul> <p>I believe a majority of attendees came to realize that the combination above injects a new Web interaction dynamic: access to "<a href="http://dbpedia.org/resource/Topic" id="link-id110978d0">Subject matter Concepts</a>" and <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id10ea5160">Named Entities</a> contained within a page via HTTP base Data Source Names (URIs).</p> <p>BTW - My presentation is a <a href="http://dbpedia.org/resource/Linked_Data" id="link-id109e5e30">Linked Data Space</a> in it's own right courtesy of the <a href="http://bibliontology.com/" id="link-id10e76d90">Bibliographic Ontology</a> (which provides slide show modeling) and <a href="http://dbpedia.org/resource/RDFa" id="link-id10d48e40">RDFa</a> that allows me to embed annotations into my <a href="http://www.w3.org/Talks/Tools/Slidy/" id="link-id104be488">Slidy</a> based presentation :-)</p> <h3>Related</h3> <ul> <li> <a href="http://virtuoso.openlinksw.com/presentations/Creating_Deploying_Exploiting_Linked_Data2/Linked_Data_2008_keynote.ppt" id="link-id10a63640">PowerPoint</a> version of Presentation</li> <li> <a href="http://www.slideshare.net/rumito/linked-data-planet-key-note/" id="link-id103aaff8">Slideshare hosted </a>version</li> <li> <a href="http://www.authorstream.com/Presentation/rumito-72460-linked-data-planet-key-note-2008-keynote-science-technology-ppt-powerpoint/" id="link-id10b97c68">Authorstream hosted </a>version</li> <li> <a href="http://docs.google.com/Presentation?id=dc7jvc6m_1061gz888hdb" id="link-id10e01640">Google Docs hosted </a>version</li> </ul>
2008-06-19T09:48:14-04:00
Missing Bits from semanticweb.com Interview
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-06-13#1386
2008-06-13T02:02:56Z
<p>Yikes! I've just discovered that the final part of the semanticweb.com's interview with <a href="http://www.cs.umd.edu/~hendler/2003/foaf.rdf#jhendler" id="link-id10483c28">Jim Hendler</a> and I, includes critical paragraphs that omit my example links :-( As you can imagine, this is a quite excruciating, bearing in mind that "Literals" are of marginal value in a <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1418a578">Linked Data</a> world.</p> <p>Anyway, thanks to the Blogosphere, I can attempt to fix this problem myself -- via this post :-)</p> <p> <strong>Q. If you wanted to provide a bewildered but still curious novice a public example of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id107e67d0">Linked Data</a> at work in their everyday life, what would it be?</strong> </p> <p> <strong><a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id13851f80">Kingsley Idehen</a>:</strong> Any one of the following:</p> <p> <a href="http://community.linkeddata.org/dataspace/person/kidehen2" id="link-id144c84b0">My Linking Open Data community Profile Page</a> - the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10339910">Linked Data</a> integration is exposed via the "Explore <a href="http://dbpedia.org/resource/Data">Data</a>" Tab <a href="http://demo.openlinksw.com/rdfbrowser2/?uri%5B%5D=http%3A%2F%2Fmyopenlink.net%2Fdataspace%2Fkidehen&" id="link-id12116d10">My Linked Data Space</a> - viewed via OpenLink's AJAR (Asynchronous Javascript and RDF) based <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10728ed0">Linked Data</a> Brower <a href="http://demo.openlinksw.com/rdfbrowser2/?uri=http%3A%2F%2Fmyopenlink.net%2Fdataspace%2Fkidehen%2Fcalendar%2FKingsley%2527s%2520Calendar" id="link-id144ef138">My Events Calendar Tag Cloud</a> - a <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0xb8fdf10">Linked Data</a> view of my Calendar Space using an RDF-aware browser In all cases, you have the ability to explore my data spaces by simply clicking on the links, which on the surface appear to be standard hypertext links, although in reality you are dealing with <a href="http://dbpedia.org/resource/Linked_Data" id="link-id142827a8">hyperdata</a> links (i.e., links to entities that result in the generation of <a href="http://dbpedia.org/resource/Entity" id="link-id144c8438">entity</a> description pages that expose <a href="http://dbpedia.org/resource/Entity" id="link-id149c9cf8">entity</a> properties via <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10551628">hyperdata</a> links). Thus, you have a single page that describes me in a very rich way since it encompasses all data associated with me, covering: personal profile, <a href="http://dbpedia.org/resource/Blog" id="link-id10ac5148">blog</a> posts, bookmarks, <a href="http://dbpedia.org/resource/Tag" id="link-id122ff4b0">tag</a> clouds, social networks etc.</p> <p> <strong>Q. What would you show the CEO or CTO of a company outside the tech industry?</strong> </p> <p> <strong><a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id106143a8">Kingsley Idehen</a>:</strong> A link to the <a href="http://demo.openlinksw.com/Northwind/Customer/ALFKI#this" id="link-id106144e0">Entity ALFKI</a>, from the popular Northwind Database associated with Microsoft Access and <a href="http://dbpedia.org/resource/SQL" id="link-id10ca6f68">SQL</a> Server database installations. This particular link exposes a typical enterprise <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id14202348">data space</a> (orders, customers, employees, suppliers ...) in a single page. The <a href="http://dbpedia.org/resource/Linked_Data" id="link-id144e9070">hyperdata</a> links represent intricate data relationships common to most business systems that will ultimately seek to repurpose existing legacy data sources and SOA services as <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a>. Alternatively, I would show the same links via the <a href="http://dataviewer.zitgist.com/?uri=http%3A//demo.openlinksw.com/Northwind/Customer/ALFKI" id="link-id10e2e1d0">Zitgist Data Viewer</a> (another Linked Data-aware browser). In both cases, I am exploiting direct access to entities via HTTP due to the protocols incorporation into the Data Source Naming scheme.</p>
2008-06-13T09:01:40.000003-04:00
Internet.com Interviews Jim Hendler & I
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-06-12#1385
2008-06-12T00:40:19Z
<p>The build up to <a href="http://www.linkeddataplanet.com" id="link-id110a2350">Linked Data Planet</a> continues... Here is <a href="http://www.semanticweb.com" id="link-id11083a68">semanticweb.com</a>'s interview with <a href="http://www.cs.rpi.edu/~hendler/" id="link-id10c4e560">Jim Hendler</a> and *<a href="http://community.linkeddata.org/dataspace/person/kidehen2#this" id="link-id10e71dc8">I</a>* titled: <a href="http://www.semanticweb.com//article.php/3751731" id="link-id1071c688">Linked Data Leaders - The Semantic Web is Here</a>.</p>
2008-06-11T20:55:15-04:00
Linked Data in Action: Library of Congress
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-06-11#1384
2008-06-11T16:36:40Z
<p>As I start my countdown to the upcoming <a href="http://www.linkeddataplanet.com" id="link-id106a81b8">Linked Data Planet conference</a>, here is the first of a series of posts geared towards showcasing practical use of the burgeoning <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id109470d0">Linked Data Web</a>.</p> <p>First up, the Library of Congress, take a look at the following pages which are "Human" and machine based "User Agent" friendly:</p> <ul> <li> <a href="http://lcsh.info/sh85118553#concept" id="link-id102927f8">Science</a> </li> <li> <a href="http://lcsh.info/sh85062913#concept" id="link-id10f13820">Humanities</a> </li> <li> <a href="http://lcsh.info/sh85082139#concept" id="link-id10ca5c58">Mathematics</a> </li> <li> <a href="http://lcsh.info/sh85020816#concept" id="link-id1230aef8">Cataloging</a> </li> <li> <a href="http://lcsh.info/sh95000541#concept" id="link-id1110e140">World Wide Web</a> </li> </ul> <p>Key point: The pages above are served up in line with <a href="http://dbpedia.org/resource/Linked_Data" id="link-id102f96a8">Linked Data</a> deployment and publishing tenets espoused by the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id10685ed8">Linking Open Data Community</a> (<a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id103915b0">LOD</a>) which include (in my preferred terminology):</p> <ul> <li>Giving "Names" to things you observe (aka <a href="http://dbpedia.org/resource/Data">Data</a> Source Names or "DSNs" for short)</li> <li>Use HTTP URLs in your data source naming scheme so that "access by reference" to your data sources exploits the expanse of the HTTP driven <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> i.e make your DSNs "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id1071cb88">Linked Data Source Names</a>" (LDNS)</li> <li>Remember that Documents / Pages are compound in nature, and they aren't the only data sources we would want to name; a document's LDSN must be distinct from the LDSNs used for the <a href="http://dbpedia.org/resource/Topic" id="link-id10c020d0">subject matter concepts</a> and/or <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-ide7a0a58">named entities</a> associated with a document </li> <li> Use the RDF Data Model to express structure within your data source(s)</li> <li>Use LDSNs when constructing statements/claims/assertions/records (triples) inside your structured data sources</li> <li> When publishing Web Pages related to your data sources; use at least one of the following to methods to guide user agents to data sources associated with your published page; the HTML <a href="http://blogs.usnet.private:8893/RPC2" id="link-id12326c48">LINK tag</a>, <a href="http://dbpedia.org/resource/RDFa" id="link-id10751788">RDFa</a>, <a href="http://dbpedia.org/resource/GRDDL" id="link-id1050e290">GRDDL</a>, or <a href="http://dbpedia.org/resource/Content_negotiation" id="link-id12e930b0">Content Negotiation</a>. </li> </ul> <p>The items above are features that users and decision makers should start to hone into when seeking, and evaluating, platforms that facilitate cost-effective exploitation of the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x9dde928">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x18c3b1c0">Web</a>.</p>
2008-06-11T13:16:31.000010-04:00
De Paradigmata and The Foundational Issues
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-06-09#1383
2008-06-09T14:02:21Z
<div> <div style="display:none;">De Paradigmata and The Foundational Issues</div> <p>I thought that we had talked ourselves to exhaustion and beyond over the issue of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1dd07c68">semantic web</a> layer cake. Apparently not. There was a paper called <i>Functional Architecture for the Semantic Web</i> by <a href="http://gerberaj.googlepages.com/" id="link-id106b8130">Aurona Gerber</a> et al at <a href="http://www.eswc2008.org/" id="link-id0x17137300">ESWC2008</a>.</p> <p>The thrust of the matter was that for newcomers the layer cake was confusing and did not clearly indicate the architecture. Why, sure. My point is that no rearranging of the boxes will cut it for the general case.</p> <p>Any diagram containing the boxes of the layer cake (i.e., <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x1a9138c0">URI</a>, <a href="http://dbpedia.org/resource/XML" id="link-id0x1cc4a8d8">XML</a>, <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xa21c1308">SPARQL</a>, <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x1aa28050">OWL</a>, <a href="http://dbpedia.org/resource/Rule_Interchange_Format" id="link-id0x137268d0">RIF</a>, Crypto, etc., etc.) in whatever order or arrangement can at best be a sort of overview of how these standards reference each other.</p> <p>Such diagrams are a little like saying that a car combines the combustion properties of fuel/air mixes with the tension and compression resistance properties of metals and composites for producing motion and secondly links to Newton's laws of motion and to aerodynamics.</p> <p>Not false. But it does not say that a car is good for economical commute or showing off at the strip or any number of niches that a mature industry has grown to serve.</p> <p>Now, talking of software engineering, modules and interfaces are good and even necessary. The trick is to know where to put the interface.</p> <p>Such a thing cannot possibly be inferred from the standards' inter-reference picture. APIs, especially if these are Web service APIs, should go where there is low <a href="http://dbpedia.org/resource/Data" id="link-id0x196fcba0">data</a> volume and tolerance for latency. For example, either inference is a preprocessing step or it is embedded right inside a SPARQL engine. Such a thing cannot be seen from the picture. Same for trust. Trust is not an after-thought at the top of the picture, except maybe in the sense of referring to the other parts.</p> <p>We hear it over and over. Scale and speed are critical. Arrange the blocks of any real system as makes sense for data flow; do not confuse literature references with control or data structure.</p> <p>The even-more foundational issue is the promotion of the general concept of a Web of Data.</p> <p>The core idea that the Web would be a query-able collection of data with meaningful reference between data of different provenance cannot be inferred from the picture, even though this should be its primary message. Or it is better to say that the first picture shown should stress this idea and then one could leave the layer cake, in whatever version, for explaining the standards' order of evolution or inter-reference.</p> <p>So, the value proposition:</p> <p>Why? Explosion of data volume, increased need of keeping up-to-date, increasing opportunity cost of not keeping in real time.</p> <p>What? An architecture that is designed for unanticipated joining and evolution of data across heterogeneous sources, either at Web or enterprise scale.</p> <p>How? URI everything and everything is cool, or, give things global names. Use <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x13700d00">RDF</a>. Reuse names or ontologies where can. (An ontology is a set of classes and property names plus some more.) Map relational data on the fly or store as RDF, whichever works. Query with SPARQL, easier than <a href="http://dbpedia.org/resource/SQL" id="link-id0x17865208">SQL</a>.</p> <p>So, my challenge for the graphics people would be to make an illustration of the above. Forget the alphabet soup. Show the layer cake as a historical reference or literature guide. Do not imply that this proliferation of boxes equates to an equal proliferation of Web services, for example.</p> </div>
2008-06-11T15:54:49-04:00
voiD, or Will the LOD Cloud Bring Rain?
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-06-09#1382
2008-06-09T14:02:20Z
<div> <div style="display:none;">voiD, or Will the LOD Cloud Bring Rain?</div> <p>At <a href="http://www.eswc2008.org/" id="link-id0x1c3bec48">ESWC2008</a>, we saw the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1f0db270">Linked Open Data</a> Cloud condense its first drops of precipitation.</p> <p> <a href="http://community.linkeddata.org/MediaWiki/index.php?MetaLOD#Kick-off_meeting_at_ESWC08" id="link-id106ee858">voiD, Vocabulary of Interlinked Datasets</a>, is an idea whose time has clearly come. By the end of the conference, many speakers had already adopted the <a href="http://dbpedia.org/resource/Meme" id="link-id0x16c99ad0">meme</a>.</p> <p>The point is to describe what is inside the <a href="http://dbpedia.org/resource/Data" id="link-id0x1c540958">data</a> sets. People may know this from having worked with the sets or from putting them together but to an outsider this is not evident.</p> <p>The Semantic Sitemap says where there are files or end points for access. But it does not say what is inside these. Also for federation, it is important to be able to determine whether it makes sense to send a particular query to a particular end point.</p> <p>If we play this right, this is what voiD will provide. I have to think of Dan Simmons' flamboyant Hyperion sci-fi series where the "void which binds" was a sort of hyperspace containing the thoughts of entities, past and present and even provided teleportation.</p> <p>So what does the voiD hold, aside infinite potentialities?</p> <p>The obvious part is DC-like provenance, version, authorship, license and such data set wide <a href="http://dbpedia.org/resource/Information" id="link-id0x16c05280">information</a>. Also the subject matter could be classified by reference to <a href="http://umbel.org/about/" id="link-id0x1abf1558">UMBEL</a> or the <a href="http://www.mpi-inf.mpg.de/~suchanek/downloads/yago/" id="link-id0x1b49ee78">Yago</a> classification of <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x184dea28">DBpedia</a>.</p> <p>More is needed, though. The simple part is listing the ontologies, if any. Also a set of namespaces would be an idea but this could be very large.</p> <p>So let us look at what we'd like to be able to answer with the voiD set.</p> <p>The below could be a sample of voiD questions?</p> <ul> <li> <p> <i>What subjects are in the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1bbac318">LOD</a> cloud?</i> </p> </li> <li> <p> <i>Given this <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x1f74c7e8">URI</a>, what set in the LOD cloud can tell me more?</i> This is divided into asking a text index like <a href="http://sindice.org/" id="link-id0x1d57a8f8">Sindice</a> for the location, getting the namespace or data set and then querying voiD.</p> </li> <li> <p> <i>What need I federate/load in order to combine all that is reachable from a given vocabulary?</i> There could be for example a graph showing the data sets and edges between them, edges being qualified by a set of same as assertions, itself a voiD described set, if translations were needed.</p> </li> <li> <p> <i>What sets are from the same or equally trusted publisher as this one?</i> </p> </li> </ul> <p>These things are roughly divided into description of the set and then some details on how it is stored on a given end point.</p> <ul> <li> <p> <i>Given this set, in which other sets will I find use of the same URIs?</i> For example, if I have language version x, I wish to know that language version y will have the same URIs insofar the things meant are the same.</p> </li> <li> <p> <i>Given this set, which sets of same as assertions will I have for mapping to which other sets?</i> For example, if I have <a href="http://www.geonames.org/" id="link-id0x1b372140">Geonames</a>, I wish to know that set x will map at least some of the URIs in Geonames to DBpedia URIs.</p> </li> </ul> <p>Let me further point out that it is increasingly clear to the community that universal sameAs is dubious, hence sameAs assertions ought to be kept separate and included or excluded depending on the usage context.</p> <ul> <li> <p> <i>Given this set, what are the interesting queries I can do?</i> This is a sort of advertisement for human consumption. This is not a list of queries for crashing the end point. Denial of service can be done in <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1b25dea8">SPARQL</a> without knowing the end point content anyhow, so this is not an added risk exposer.</p> </li> <li> <p> <i>Vocabularies used.</i> This is a reference to the OWL or RDFS resources giving the applicable ontologies, if present. Also, a complete list of classes whose direct instances actually occur in the set is useful.</p> </li> <li> <p> <i>Ballpark cardinality.</i> Something like a <a href="http://darq.sourceforge.net/" id="link-id0x1ed8f580">DARQ</a> optimization profile would be a good idea. I would say that there should be a possibility of just including a DARQ description file as is. This is a sort of baseline and since it already exist, we are spared the committee trouble of figuring out what it ought to contain and what not. If we start defining this from scratch, it will take long. Further, let this be optional. Quite Independently of this, query processors may make optimization related queries to remote end points insofar the specific end point supports these. This will come in time. For now, just the basics.</p> </li> </ul> <p>Along with this, LOD SPARQL end points could adopt a couple of basic conventions. The simplest would be to agree that each would host a graph with a given URI that would contain the voiD descriptions of the data sets contained, along with the graph URI used for each set, if different from the publisher's URI for the graph. There is a point to this since an end point may load multiple data sets into one graph.</p> <p>We hope to have a good idea of the matter in a couple of weeks, certainly a general statement of direction to be published at <a href="http://www.linkeddataplanet.com/" id="link-id0x1b049830">Linked Data Planet</a> in a couple of weeks.</p> </div>
2008-06-11T15:15:21-04:00
The DARQ Matter of Federation
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-06-09#1381
2008-06-09T14:02:19Z
<div> <div style="display:none;">The DARQ Matter of Federation</div> <p>Astronomers propose that the universe is held together, so to speak, by the gravity of invisible "dark matter" spread in interstellar and intergalactic space.</p> <p>For the <a href="http://dbpedia.org/resource/Data" id="link-id0x19dbf410">data</a> web, it will be held together by federation, also an invisible factor. As in Minkowski space, so in <a href="http://dbpedia.org/resource/Cyberspace" id="link-id0x9fc13ff8">cyberspace</a>.</p> <p>To take the astronomical analogy further, putting too much visible stuff in one place makes a black hole, whose chief properties are that it is very heavy, can only get heavier and that nothing comes out.</p> <p> <a href="http://darq.sourceforge.net/" id="link-id0x1d06bd88">DARQ</a> is Bastian Quilitz's federated extension of the <a href="http://jena.sourceforge.net/" id="link-id0x1cf28f70">Jena</a> <a href="http://jena.sourceforge.net/ARQ/" id="link-id0x1cba22c8">ARQ</a> <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x171c7dc8">SPARQL</a> processor. It has existed for a while and was also presented at <a href="http://www.eswc2008.org/" id="link-id0x1ed53cd0">ESWC2008</a>. There is also SPARQL FED from Andy Seaborne, an explicit means of specifying which end point will process which fragment of a distributed SPARQL query. Still, for federation to deliver in an open, decentralized world, it must be transparent. For a specific application, with a predictable workload, it is of course OK to partition queries explicitly.</p> <p>Bastian had split <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x1ce846c0">DBpedia</a> among five <a href="http://virtuoso.openlinksw.com" id="link-id0x1cad0640">Virtuoso</a> servers and was querying this set with DARQ. The end result was that there was a rather frightful cost of federation as opposed to all the data residing in a single Virtuoso. The other result was that if selectivity of predicates was not correctly guessed by the federation engine, the proposition was a non-starter. With correct join order it worked, though.</p> <p>Yet, we really want federation. Looking further down the road, we simply must make federation work. This is just as necessary as running on a server cluster for mid-size workloads.</p> <p>Since we are convinced of the cause, let's talk about the means.</p> <p>For DARQ as it now stands, there's probably an order of magnitude or even more to gain from a couple of simple tricks. If going to a SPARQL end point that is not the outermost in the loop join sequence, batch the requests together in one <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x19a48280">HTTP</a>/1.1 message. So, if the query is "get me my friends living in cities of over a million people," there will be the fragment "get city where x lives" and later "ask if population of x greater than 1000000". If I have 100 friends, I send the 100 requests in a batch to each eligible server.</p> <p>Further, if running against a server of known brand, use a client-server connection and prepared statements with array parameters. This can well improve the processing speed at the remote end point by another order of magnitude. This gain may however not be as great as the latency savings from message batching. We will provide a sample of how to do this with Virtuoso over <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x1cf18278">JDBC</a> so Bastian can try this if interested.</p> <p>These simple things will give a lot of mileage and may even decide whether federation is an option in specific applications. For the open web however, these measures will not yet win the day.</p> <p>When federating <a href="http://dbpedia.org/resource/SQL" id="link-id0x1cf7d0e8">SQL</a>, colocation of data is sort of explicit. If two tables are joined and they are in the same source, then the join can go to the source. For SPARQL this is also so but with a twist:</p> <p>If a foaf:Person is found on a given server, this does not mean that the Person's geek code or email hash will be on the same server. Thus <code>{?p name "Johnny" . ?p geekCode ?g . ?p emailHash ?h }</code> does not necessarily denote a colocated join if many servers serve items of the vocabulary.</p> <p>However, in most practical cases, for obtaining a rapid answer, treating this as a colocated fragment will be appropriate. Thus, it may be necessary to be able to declare that geek codes will be assumed colocated with names. This will save a lot of message passing and offer decent, if not theoretically total recall. For search style applications, starting with such assumptions will make sense. If nothing is found, then we can partition each join step separately for the unlikely case that there were a server that gave geek codes but not names.</p> <p>For Virtuoso, we find that a federated query's asynchronous, parallel evaluation model is not so different from that on a local cluster. So the cluster version could have the option of federated query. The difference is that a cluster is local and tightly coupled and predictably partitioned but a federated setting is none of these.</p> <p>For description, we would take DARQ's description model and maybe extend it a little where needed. Also we would enhance the protocol to allow just asking for the query cost estimate given a query with literals specified. We will do this eventually.</p> <p>We would like to talk to Bastian about large improvements to DARQ, specially when working with Virtuoso. We'll see.</p> <p>Of course, one mode of federating is the crawl-as-you-go approach of the Virtuoso <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id0x1e163140">Sponger</a>. This will bring in fragments following seeAlso or sameAs declarations or other references. This will however not have the recall of a warehouse or federation over well described SPARQL end-points. But up to a certain volume it has the speed of local storage.</p> <p>The emergence of voiD (Vocabulary of Interlinked Data) is a step in the direction of making federation a reality. There is <a href="http://www.openlinksw.com/weblog/oerling/?id=1377" id="link-id1109a4c8">a separate post</a> about this.</p> </div>
2008-06-11T15:15:14-04:00
Aspects of RDF to RDF Mapping
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-06-09#1380
2008-06-09T14:02:18Z
<div> <div style="display:none;">Aspects of RDF to RDF Mapping</div> <p>The W3C has recently launched an <a href="http://www.w3.org/2005/Incubator/rdb2rdf/" id="link-idd763f48">incubator group about mapping relational data to RDF</a>.</p> <p>From participating in the group for the few initial sessions, I get the following impressions.</p> <p>There is a segment of users, for example from the biomedical community, who do heavy duty <a href="http://dbpedia.org/resource/Data" id="link-id0x17f9e6f8">data</a> integration and look to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x17eabf48">RDF</a> for managing complexity. Unifying heterogeneous data under OWL ontologies, reasoning, and data integrity, are points of interest.</p> <p>There is another segment that is concerned with semantifying the document web, which topic includes initiatives such as <a href="http://triplify.org/" id="link-id0x1a25cd28">Triplify</a> and <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x182c41e8">semantic web</a> search such as <a href="http://sindice.org/" id="link-id0x1a29c5e8">Sindice</a>. The emphasis there is on minimizing entry cost and creating critical mass. The next one to come will clean up the semantics, if these need be cleaned up at all.</p> <p>(Some cleanup is taking place with <a href="http://www.mpi-inf.mpg.de/~suchanek/downloads/yago/" id="link-id0x17fd2b70">Yago</a> and <a href="http://zitgist.com/about/" id="link-id0x17e6ab88">Zitgist</a>, but this is a matter for a different post.)</p> <p>Thus, technically speaking, the mapping landscape is diverse, but ETL (extract-transform-load) seems to predominate. The biomedical people make data warehouses for answering specific questions. The web people are interested in putting data out in the expectation that the next player will warehouse it and allow running complex meshups against the whole of the RDF-ized web.</p> <p>As one would expect, these groups see different issues and needs. Roughly speaking, one is about quality and structure and the other is about volume.</p> <p>Where do we stand?</p> <p>We are with the research data warehousers in saying that the mapping question is very complex and that it would indeed be nice to bypass ETL and go to the source <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x182acd68">RDBMS</a>(s) on demand. Projects in this direction are ongoing.</p> <p>We are with the web people in building large RDF stores with scalable query answering for arbitrary RDF, for example, hosting a lot of the Linking Open Data sets, and working with Zitgist.</p> <p>These things are somewhat different.</p> <p>At present, both the research warehousers and the web scalers predominantly go for ETL.</p> <p>This is fine by us as we definitely are in the large RDF store race.</p> <p>Still, mapping has its point. A relational store will perform quite a bit faster than a quad store if it has the right covering indices or application-specific compressed columnar layout. Thus, there is nothing to block us from querying analytics in <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x16c91438">SPARQL</a>, once the obviously necessary extensions of sub-query, expressions and aggregation are in place.</p> <p>To cite an example, the Ordnance Survey of the UK has a GIS system running on <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x17ee37c8">Oracle</a> with an entry pretty much for each mailbox, lamp post, and hedgerow in the country. According to Ordnance Survey, this would be 1 petatriple, 1e15 triples. "Such a big server farm that we'd have to put it on our map," as Jenny Harding put it at <a href="http://www.eswc2008.org/" id="link-id0x1cab6330">ESWC2008</a>. I'd add that an even bigger map entry would be the power plant needed to run the 100,000 or so PCs this would take. This is counting 10 gigatriples per PC, which would not even give very good working sets.</p> <p>So, on-the-fly RDBMS-to-RDF mapping in some cases is simply necessary. Still, the benefits of RDF for integration can be preserved if the translation middleware is smart enough. Specifically, this entails knowing what tables can be joined with what other tables and pushing maximum processing to the RDBMS(s) involved in the query.</p> <p>You can download the slide set I used for the <a href="http://virtuoso.openlinksw.com" id="link-id0xa1fb7e8">Virtuoso</a> presentation for the RDB to RDF mapping incubator group (<a href="http://virtuoso.openlinksw.com/wiki/main/Main/VirtPresentations/Relational2RDF.ppt" id="link-id106f9e88">PPT</a>; <a href="http://virtuoso.openlinksw.com/wiki/main/Main/VirtPresentations" id="link-id10a8dc90">other formats</a> coming soon). The main point is that real integration is hard and needs smart query splitting and optimization, as well as real understanding of the databases and subject matter from the <a href="http://dbpedia.org/resource/Information" id="link-id0x17ee38a0">information</a> architect. Sometimes in the web space it can suffice to put data out there with trivial RDF translation and hope that a search engine or such will figure out how to join this with something else. For the enterprise, things are not so. Benefits are clear if one can navigate between disjoint silos but making this accurate enough for deriving business conclusions, as well as efficient enough for production, is a soluble and non-trivial question.</p> <p>We will show the basics of this with the <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x1844d718">TPC-H</a> mapping, and by joining this with physical triples. We will also make a set of TPC-H format table sets, make mappings between keys in one to keys in the other, and show joins between the two. The SPARQL querying of one such data store is a done deal, including the SPARQL extensions for this. There is even a demo paper, Business Intelligence Extensions for SPARQL (<a href="http://virtuoso.openlinksw.com/wiki/main/Main/VirtPresentations/RDFAndMapped_BI.pdf" id="link-id12ea4b18">PDF</a>; <a href="http://virtuoso.openlinksw.com/wiki/main/Main/VirtPresentations" id="link-id106e1810">other formats</a> coming soon), by us on the subject in the ESWC 2008 proceedings. If there is an issue left, it is just the technicality of always producing <a href="http://dbpedia.org/resource/SQL" id="link-id0x17fc8d60">SQL</a> that looks hand-crafted and hence is better understood by the target RDBMS(s). For example, Oracle works better if one uses an <code>IN</code> sub-query instead of the equivalent existence test.</p> <p>Follow this <a href="http://dbpedia.org/resource/Blog" id="link-id0xa9bcef8">blog</a> for more on the topic; published papers are always a limited view on the matter.</p> </div>
2008-06-11T13:15:39-04:00
ESWC 2008
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-06-09#1379
2008-06-09T14:02:16Z
<div> <div style="display:none;">ESWC 2008</div> <p>Yrjänä Rankka and I attended <a href="http://www.eswc2008.org/" id="link-id10b7a038">ESWC2008</a> on behalf of OpenLink.</p> <p>We were invited at the last minute to give a <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id105df758">Linked Open Data</a> talk at Paolo Bouquet's Identity and Reference workshop. We also had a demo of <a href="http://dbpedia.org/resource/SPARQL" id="link-id12eacca0">SPARQL</a> BI (<a href="http://virtuoso.openlinksw.com/wiki/main/Main/VirtPresentations/ESWC2008%20SPARQL%20BI%20OpenLink.ppt" id="link-id10b43e58">PPT</a>); <a href="http://virtuoso.openlinksw.com/wiki/main/Main/VirtPresentations" id="link-id1116d8f0">other formats coming soon</a>), our business intelligence extensions to <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x16c9bfc8">SPARQL</a> as well as joining between relational <a href="http://dbpedia.org/resource/Data" id="link-id10badc40">data</a> mapped to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id108edaf8">RDF</a> and native <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x181a5ed8">RDF</a> <a href="http://dbpedia.org/resource/Data" id="link-id0x17e69910">data</a>. i was also speaking at the social networks panel chaired by Harry Halpin.</p> <p>I have gathered a few impressions that I will share in the next few posts (<a href="http://www.openlinksw.com/weblog/oerling/?id=1375" id="link-id107298e0">1 - RDF Mapping</a>, <a href="http://www.openlinksw.com/weblog/oerling/?id=1376" id="link-id10b3a530">2 - DARQ</a>, <a href="http://www.openlinksw.com/weblog/oerling/?id=1377" id="link-id107290e0">3 - voiD</a>, <a href="http://www.openlinksw.com/weblog/oerling/?id=1378" id="link-id1071a950">4 - Paradigmata</a>). <i>Caveat: This is not meant to be complete or impartial press coverage of the event but rather some quick comments on issues of personal/OpenLink interest. The fact that I do not mention something does not mean that it is unimportant.</i> </p> <h2>The voiD Graph</h2> <p> <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1a87f110">Linked Open Data</a> was well represented, with Chris Bizer, Tom Heath, ourselves and many others. The great advance for <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id108f3c48">LOD</a> this time around is <a href="http://community.linkeddata.org/MediaWiki/index.php?MetaLOD#Kick-off_meeting_at_ESWC08" id="link-id10df9830">voiD, the Vocabulary of Interlinked Datasets</a>, a means to describe what in fact is inside the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1a089980">LOD</a> cloud, how to join it with what and so forth. Big time important if there is to be a <a href="http://www.openlinksw.com/weblog/oerling/?id=1377" id="link-iddf74578">web of federatable data sources</a>, feeding directly into what we have been saying for a while about SPARQL end-point self-description and discovery. There is reasonable hope of having something by the date of <a href="http://www.linkeddataplanet.com/" id="link-id10dd0848">Linked Data Planet</a> in a couple of weeks.</p> <h2>Federating</h2> <p>Bastian Quilitz gave a talk about his <a href="http://darq.sourceforge.net/" id="link-id108746e8">DARQ</a>, a federated version of Jena's ARQ.</p> <p>Something like <a href="http://darq.sourceforge.net/" id="link-id0x1a2d9860">DARQ</a>'s optimization statistics should make their way into the <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id10992348">SPARQL protocol</a> as well as the voiD data set description.</p> <p>We really need federation but more on this in <a href="http://www.openlinksw.com/weblog/oerling/?id=1376" id="link-id1059d688">a separate post</a>.</p> <h2> <a href="http://xsparql.deri.ie/" id="link-id10314308">XSPARQL</a> </h2> <p>Axel Polleres et al had a paper about <a href="http://xsparql.deri.ie/" id="link-id0x1ad77490">XSPARQL</a>, a merge of <a href="http://dbpedia.org/resource/XQuery" id="link-id10b98e90">XQuery</a> and SPARQL. While visiting DERI a couple of weeks back and again at the conference, we talked about OpenLink implementing the spec. It is evident that the engines must be in the same process and not communicate via the <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id0x17e75190">SPARQL protocol</a> for this to be practical. We could do this. We'll have to see when.</p> <p>Politically, using <a href="http://dbpedia.org/resource/XQuery" id="link-id0x18a9bf10">XQuery</a> to give expressions and XML synthesis to SPARQL would be fitting. These things are needed anyhow, as surely as aggregation and sub-queries but the latter would not so readily come from XQuery. Some rapprochement between RDF and XML folks is desirable anyhow.</p> <h2>Panel: Will the Sem Web Rise to the Challenge of the Social Web?</h2> <p>The social web panel presented the question of whether the sem web was ready for prime time with data portability.</p> <p>The main thrust was expressed in Harry Halpin's rousing closing words: "Men will fight in a battle and lose a battle for a cause they believe in. Even if the battle is lost, the cause may come back and prevail, this time changed and under a different name. Thus, there may well come to be something like our <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id122f4da0">semantic web</a>, but it may not be the one we have worked all these years to build if we do not rise to the occasion before us right now."</p> <p>So, how to do this? Dan Brickley asked the audience how many supported, or were aware of, the latest Web 2.0 things, such as <a href="http://dbpedia.org/page/OAuth" id="link-idf300bc0">OAuth</a> and <a href="http://dbpedia.org/page/OpenID" id="link-id10ce7a40">OpenID</a>. A few were. The general idea was that research (after all, this was a research event) should be more integrated and open to the world at large, not living at the "outdated pace" of a 3 year funding cycle. Stefan Decker of DERI acquiesced in principle. Of course there is impedance mismatch between specialization and interfacing with everything.</p> <p>I said that triples and vocabularies existed, that OpenLink had <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id1210dbf8">ODS</a> (<a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id11076be8">OpenLink Data Spaces</a>, <a href="http://community.linkeddata.org/" id="link-id10d46710">Community LinkedData</a>) for managing one's data-web presence, but that scale would be the next thing. Rather large scale even, with 100 gigatriples (Gtriples) reached before one even noticed. It takes a lot of PCs to host this, maybe $400K worth at today's prices, without replication. Count 16G ram and a few cores per Gtriple so that one is not waiting for disk all the time.</p> <p>The tricks that Web 2.0 silos do with app-specific data structures and app-specific partitioning do not really work for RDF without compromising the whole point of smooth schema evolution and tolerance of ragged data.</p> <p>So, simple vocabularies, minimal inference, minimal blank nodes. Besides, note that the inference will have to be done at run time, not forward-chained at load time, if only because users will not agree on what sameAs and other declarations they want for their queries. Not to mention spam or malicious sameAs declarations!</p> <p>As always, there was the question of business models for the open data web and for semantic technologies in general. As we see it, <a href="http://dbpedia.org/resource/Information" id="link-id108b7688">information</a> overload is the factor driving the demand. Better contextuality will justify semantic technologies. Due to the large volumes and complex processing, a data-as-service model will arise. The data may be open, but its query infrastructure, cleaning, and keeping up-to-date, can be monetized as services.</p> <h2>Identity and Reference</h2> <p>For the identity and reference workshop, the ultimate question is metaphysical and has no single universal answer, even though people, ever since the dawn of time and earlier, have occupied themselves with the issue. Consequently, I started with the Genesis quote where Adam called things by <i>nominibus suis</i>, off-hand implying that things would have some intrinsic ontologically-due names. This would be among the older references to the question, at least in widely known sources.</p> <p>For present purposes, the consensus seemed to be that what would be considered the same as something else depended entirely on the application. What was similar enough to warrant a sameAs for cooking purposes might not warrant a sameAs for chemistry. In fact, complete and exact sameness for URIs would be very rare. So, instead of making generic weak similarity assertions like similarTo or seeAlso, one would choose a set of strong sameAs assertions and have these in effect for query answering if they were appropriate to the granularity demanded by the application.</p> <p>Therefore sameAs is our permanent companion, and there will in time be malicious and spam sameAs. So, nothing much should be materialized on the basis of sameAs assertions in an <a href="http://dbpedia.org/resource/Open_world_assumption" id="link-id10c4dfd0">open world</a>. For an app-specific warehouse, sameAs can be resolved at load time.</p> <p>There was naturally some apparent tension between the Occam camp of <a href="http://dbpedia.org/resource/Entity" id="link-id105fd240">entity</a> name services and the LOD camp. I would say that the issue is more a perceived polarity than a real one. People will, inevitably, continue giving things names regardless of any centralized authority. Just look at natural language. But having a dictionary that is commonly accepted for established domains of discourse is immensely helpful.</p> <h2>CYC and NLP</h2> <p>The semantic search workshop was interesting, especially CYC's presentation. CYC is, as it were, the grand old man of <a href="http://dbpedia.org/resource/Knowledge" id="link-id10568158">knowledge</a> representation. Over the long term, I would have support of the CYC inference language inside a database query processor. This would mostly be for repurposing the huge <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x1acff9d0">knowledge</a> base for helping in search type queries. If it is for transactions or financial reporting, then queries will be <a href="http://dbpedia.org/resource/SQL" id="link-id130a0a80">SQL</a> and make little or no use of any sort of inference. If it is for summarization or finding things, the opposite holds. For scaling, the issue is just making correct cardinality guesses for query planning, which is harder when inference is involved. We'll see.</p> <p>I will also have a closer look at natural language one of these days, quite inevitably, since <a href="http://zitgist.com/about/" id="link-id10795828">Zitgist</a> (for example) is into <a href="http://dbpedia.org/resource/Entity" id="link-id0x18a12918">entity</a> disambiguation.</p> <h2>Scale</h2> <p>Garlic gave a talk about their Data Patrol and QDOS. We agree that storing the data for these as triples instead of 1000 or so constantly changing relational tables could well make the difference between next-to-unmanageable and efficiently adaptive.</p> <p>Garlic probably has the largest triple collection in constant online use to date. We will soon join them with our hosting of the whole LOD cloud and <a href="http://sindice.org/" id="link-id0x17f18a38">Sindice</a>/<a href="http://zitgist.com/about/" id="link-id0x184e9e90">Zitgist</a> as triples.</p> <h2>Conclusions</h2> <p>There is a mood to deliver applications. Consequently, scale remains a central, even the principal topic. So for now we make bigger centrally-managed databases. At the next turn around the corner we will have to turn to federation. The point here is that a planetary-scale, centrally-managed, online system can be made when the workload is uniform and anticipatable, but if it is free-form queries and complex analysis, we have a problem. So we move in the direction of federating and charging based on usage whenever the workload is more complex than making simple lookups now and then.</p> <p>For the <a href="http://virtuoso.openlinksw.com" id="link-id1026ac28">Virtuoso</a> roadmap, this changes little. Next we make data sets available on Amazon EC2, as widely promised at ESWC. With big scale also comes rescaling and repartitioning, so this gets additional weight, as does further parallelizing of single user workloads. As it happens, the same medicine helps for both. At <a href="http://www.linkeddataplanet.com/" id="link-id0x17ff5c20">Linked Data Planet</a>, we will make more announcements.</p> </div>
2008-06-11T13:15:33-04:00
De Paradigmata and The Foundational Issues
http://www.openlinksw.com:443/weblog/oerling/?date=2008-06-09#1378
2008-06-09T14:00:41Z
<p>I thought that we had talked ourselves to exhaustion and beyond over the issue of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0xa1684d90">semantic web</a> layer cake. Apparently not. There was a paper called <i>Functional Architecture for the Semantic Web</i> by <a href="http://gerberaj.googlepages.com/" id="link-id106b8130">Aurona Gerber</a> et al at <a href="http://www.eswc2008.org/" id="link-id0x1ce7f490">ESWC2008</a>.</p> <p>The thrust of the matter was that for newcomers the layer cake was confusing and did not clearly indicate the architecture. Why, sure. My point is that no rearranging of the boxes will cut it for the general case.</p> <p>Any diagram containing the boxes of the layer cake (i.e., <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x19d3bdf8">URI</a>, <a href="http://dbpedia.org/resource/XML" id="link-id0x19a89358">XML</a>, <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1de67920">SPARQL</a>, <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0xa50c0e40">OWL</a>, <a href="http://dbpedia.org/resource/Rule_Interchange_Format" id="link-id0x171be3b0">RIF</a>, Crypto, etc., etc.) in whatever order or arrangement can at best be a sort of overview of how these standards reference each other.</p> <p>Such diagrams are a little like saying that a car combines the combustion properties of fuel/air mixes with the tension and compression resistance properties of metals and composites for producing motion and secondly links to Newton's laws of motion and to aerodynamics.</p> <p>Not false. But it does not say that a car is good for economical commute or showing off at the strip or any number of niches that a mature industry has grown to serve.</p> <p>Now, talking of software engineering, modules and interfaces are good and even necessary. The trick is to know where to put the interface.</p> <p>Such a thing cannot possibly be inferred from the standards' inter-reference picture. APIs, especially if these are Web service APIs, should go where there is low <a href="http://dbpedia.org/resource/Data" id="link-id0x1aad9288">data</a> volume and tolerance for latency. For example, either inference is a preprocessing step or it is embedded right inside a SPARQL engine. Such a thing cannot be seen from the picture. Same for trust. Trust is not an after-thought at the top of the picture, except maybe in the sense of referring to the other parts.</p> <p>We hear it over and over. Scale and speed are critical. Arrange the blocks of any real system as makes sense for data flow; do not confuse literature references with control or data structure.</p> <p>The even-more foundational issue is the promotion of the general concept of a Web of Data.</p> <p>The core idea that the Web would be a query-able collection of data with meaningful reference between data of different provenance cannot be inferred from the picture, even though this should be its primary message. Or it is better to say that the first picture shown should stress this idea and then one could leave the layer cake, in whatever version, for explaining the standards' order of evolution or inter-reference.</p> <p>So, the value proposition:</p> <p>Why? Explosion of data volume, increased need of keeping up-to-date, increasing opportunity cost of not keeping in real time.</p> <p>What? An architecture that is designed for unanticipated joining and evolution of data across heterogeneous sources, either at Web or enterprise scale.</p> <p>How? URI everything and everything is cool, or, give things global names. Use <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1ae09b88">RDF</a>. Reuse names or ontologies where can. (An ontology is a set of classes and property names plus some more.) Map relational data on the fly or store as RDF, whichever works. Query with SPARQL, easier than <a href="http://dbpedia.org/resource/SQL" id="link-id0x1ab0b5e0">SQL</a>.</p> <p>So, my challenge for the graphics people would be to make an illustration of the above. Forget the alphabet soup. Show the layer cake as a historical reference or literature guide. Do not imply that this proliferation of boxes equates to an equal proliferation of Web services, for example.</p>
2008-06-11T15:54:42-04:00
voiD, or Will the LOD Cloud Bring Rain?
http://www.openlinksw.com:443/weblog/oerling/?date=2008-06-09#1377
2008-06-09T13:58:41Z
<p>At <a href="http://www.eswc2008.org/" id="link-id0x1d07a490">ESWC2008</a>, we saw the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0xa288f548">Linked Open Data</a> Cloud condense its first drops of precipitation.</p> <p> <a href="http://community.linkeddata.org/MediaWiki/index.php?MetaLOD#Kick-off_meeting_at_ESWC08" id="link-id106ee858">voiD, Vocabulary of Interlinked Datasets</a>, is an idea whose time has clearly come. By the end of the conference, many speakers had already adopted the <a href="http://dbpedia.org/resource/Meme" id="link-id0x171cb518">meme</a>.</p> <p>The point is to describe what is inside the <a href="http://dbpedia.org/resource/Data" id="link-id0x1ce412a0">data</a> sets. People may know this from having worked with the sets or from putting them together but to an outsider this is not evident.</p> <p>The Semantic Sitemap says where there are files or end points for access. But it does not say what is inside these. Also for federation, it is important to be able to determine whether it makes sense to send a particular query to a particular end point.</p> <p>If we play this right, this is what voiD will provide. I have to think of Dan Simmons' flamboyant Hyperion sci-fi series where the "void which binds" was a sort of hyperspace containing the thoughts of entities, past and present and even provided teleportation.</p> <p>So what does the voiD hold, aside infinite potentialities?</p> <p>The obvious part is DC-like provenance, version, authorship, license and such data set wide <a href="http://dbpedia.org/resource/Information" id="link-id0xa04eb368">information</a>. Also the subject matter could be classified by reference to <a href="http://umbel.org/about/" id="link-id0x19a96a00">UMBEL</a> or the <a href="http://www.mpi-inf.mpg.de/~suchanek/downloads/yago/" id="link-id0x1ce30fe8">Yago</a> classification of <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x1ced78a0">DBpedia</a>.</p> <p>More is needed, though. The simple part is listing the ontologies, if any. Also a set of namespaces would be an idea but this could be very large.</p> <p>So let us look at what we'd like to be able to answer with the voiD set.</p> <p>The below could be a sample of voiD questions?</p> <ul> <li> <p> <i>What subjects are in the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1a623bd8">LOD</a> cloud?</i> </p> </li> <li> <p> <i>Given this <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x1712a638">URI</a>, what set in the LOD cloud can tell me more?</i> This is divided into asking a text index like <a href="http://sindice.org/" id="link-id0x9f815570">Sindice</a> for the location, getting the namespace or data set and then querying voiD.</p> </li> <li> <p> <i>What need I federate/load in order to combine all that is reachable from a given vocabulary?</i> There could be for example a graph showing the data sets and edges between them, edges being qualified by a set of same as assertions, itself a voiD described set, if translations were needed.</p> </li> <li> <p> <i>What sets are from the same or equally trusted publisher as this one?</i> </p> </li> </ul> <p>These things are roughly divided into description of the set and then some details on how it is stored on a given end point.</p> <ul> <li> <p> <i>Given this set, in which other sets will I find use of the same URIs?</i> For example, if I have language version x, I wish to know that language version y will have the same URIs insofar the things meant are the same.</p> </li> <li> <p> <i>Given this set, which sets of same as assertions will I have for mapping to which other sets?</i> For example, if I have <a href="http://www.geonames.org/" id="link-id0x1ce04288">Geonames</a>, I wish to know that set x will map at least some of the URIs in Geonames to DBpedia URIs.</p> </li> </ul> <p>Let me further point out that it is increasingly clear to the community that universal sameAs is dubious, hence sameAs assertions ought to be kept separate and included or excluded depending on the usage context.</p> <ul> <li> <p> <i>Given this set, what are the interesting queries I can do?</i> This is a sort of advertisement for human consumption. This is not a list of queries for crashing the end point. Denial of service can be done in <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x19a887f8">SPARQL</a> without knowing the end point content anyhow, so this is not an added risk exposer.</p> </li> <li> <p> <i>Vocabularies used.</i> This is a reference to the OWL or RDFS resources giving the applicable ontologies, if present. Also, a complete list of classes whose direct instances actually occur in the set is useful.</p> </li> <li> <p> <i>Ballpark cardinality.</i> Something like a <a href="http://darq.sourceforge.net/" id="link-id0x19a1fa10">DARQ</a> optimization profile would be a good idea. I would say that there should be a possibility of just including a DARQ description file as is. This is a sort of baseline and since it already exist, we are spared the committee trouble of figuring out what it ought to contain and what not. If we start defining this from scratch, it will take long. Further, let this be optional. Quite Independently of this, query processors may make optimization related queries to remote end points insofar the specific end point supports these. This will come in time. For now, just the basics.</p> </li> </ul> <p>Along with this, LOD SPARQL end points could adopt a couple of basic conventions. The simplest would be to agree that each would host a graph with a given URI that would contain the voiD descriptions of the data sets contained, along with the graph URI used for each set, if different from the publisher's URI for the graph. There is a point to this since an end point may load multiple data sets into one graph.</p> <p>We hope to have a good idea of the matter in a couple of weeks, certainly a general statement of direction to be published at <a href="http://www.linkeddataplanet.com/" id="link-id0x1a8f7d40">Linked Data Planet</a> in a couple of weeks.</p>
2015-08-20T17:14:17.124682-04:00
The DARQ Matter of Federation
http://www.openlinksw.com:443/weblog/oerling/?date=2008-06-09#1376
2008-06-09T13:57:30Z
<p>Astronomers propose that the universe is held together, so to speak, by the gravity of invisible "dark matter" spread in interstellar and intergalactic space.</p> <p>For the <a href="http://dbpedia.org/resource/Data" id="link-id0x19bbd830">data</a> web, it will be held together by federation, also an invisible factor. As in Minkowski space, so in <a href="http://dbpedia.org/resource/Cyberspace" id="link-id0x19af2488">cyberspace</a>.</p> <p>To take the astronomical analogy further, putting too much visible stuff in one place makes a black hole, whose chief properties are that it is very heavy, can only get heavier and that nothing comes out.</p> <p> <a href="http://darq.sourceforge.net/" id="link-id0x19b7a9c8">DARQ</a> is Bastian Quilitz's federated extension of the <a href="http://jena.sourceforge.net/" id="link-id0x19ce3da0">Jena</a> <a href="http://jena.sourceforge.net/ARQ/" id="link-id0xa569a258">ARQ</a> <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1a8d2270">SPARQL</a> processor. It has existed for a while and was also presented at <a href="http://www.eswc2008.org/" id="link-id0x1aad1d00">ESWC2008</a>. There is also SPARQL FED from Andy Seaborne, an explicit means of specifying which end point will process which fragment of a distributed SPARQL query. Still, for federation to deliver in an open, decentralized world, it must be transparent. For a specific application, with a predictable workload, it is of course OK to partition queries explicitly.</p> <p>Bastian had split <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x1a8ac770">DBpedia</a> among five <a href="http://virtuoso.openlinksw.com" id="link-id0x19601d30">Virtuoso</a> servers and was querying this set with DARQ. The end result was that there was a rather frightful cost of federation as opposed to all the data residing in a single Virtuoso. The other result was that if selectivity of predicates was not correctly guessed by the federation engine, the proposition was a non-starter. With correct join order it worked, though.</p> <p>Yet, we really want federation. Looking further down the road, we simply must make federation work. This is just as necessary as running on a server cluster for mid-size workloads.</p> <p>Since we are convinced of the cause, let's talk about the means.</p> <p>For DARQ as it now stands, there's probably an order of magnitude or even more to gain from a couple of simple tricks. If going to a SPARQL end point that is not the outermost in the loop join sequence, batch the requests together in one <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x19b94818">HTTP</a>/1.1 message. So, if the query is "get me my friends living in cities of over a million people," there will be the fragment "get city where x lives" and later "ask if population of x greater than 1000000". If I have 100 friends, I send the 100 requests in a batch to each eligible server.</p> <p>Further, if running against a server of known brand, use a client-server connection and prepared statements with array parameters. This can well improve the processing speed at the remote end point by another order of magnitude. This gain may however not be as great as the latency savings from message batching. We will provide a sample of how to do this with Virtuoso over <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x17822258">JDBC</a> so Bastian can try this if interested.</p> <p>These simple things will give a lot of mileage and may even decide whether federation is an option in specific applications. For the open web however, these measures will not yet win the day.</p> <p>When federating <a href="http://dbpedia.org/resource/SQL" id="link-id0x1a651628">SQL</a>, colocation of data is sort of explicit. If two tables are joined and they are in the same source, then the join can go to the source. For SPARQL this is also so but with a twist:</p> <p>If a foaf:Person is found on a given server, this does not mean that the Person's geek code or email hash will be on the same server. Thus <code>{?p name "Johnny" . ?p geekCode ?g . ?p emailHash ?h }</code> does not necessarily denote a colocated join if many servers serve items of the vocabulary.</p> <p>However, in most practical cases, for obtaining a rapid answer, treating this as a colocated fragment will be appropriate. Thus, it may be necessary to be able to declare that geek codes will be assumed colocated with names. This will save a lot of message passing and offer decent, if not theoretically total recall. For search style applications, starting with such assumptions will make sense. If nothing is found, then we can partition each join step separately for the unlikely case that there were a server that gave geek codes but not names.</p> <p>For Virtuoso, we find that a federated query's asynchronous, parallel evaluation model is not so different from that on a local cluster. So the cluster version could have the option of federated query. The difference is that a cluster is local and tightly coupled and predictably partitioned but a federated setting is none of these.</p> <p>For description, we would take DARQ's description model and maybe extend it a little where needed. Also we would enhance the protocol to allow just asking for the query cost estimate given a query with literals specified. We will do this eventually.</p> <p>We would like to talk to Bastian about large improvements to DARQ, specially when working with Virtuoso. We'll see.</p> <p>Of course, one mode of federating is the crawl-as-you-go approach of the Virtuoso <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id0x1dddce48">Sponger</a>. This will bring in fragments following seeAlso or sameAs declarations or other references. This will however not have the recall of a warehouse or federation over well described SPARQL end-points. But up to a certain volume it has the speed of local storage.</p> <p>The emergence of voiD (Vocabulary of Interlinked Data) is a step in the direction of making federation a reality. There is <a href="http://www.openlinksw.com/weblog/oerling/?id=1377" id="link-id1109a4c8">a separate post</a> about this.</p>
2008-06-11T15:15:00-04:00
Aspects of RDF to RDF Mapping
http://www.openlinksw.com:443/weblog/oerling/?date=2008-06-09#1375
2008-06-09T13:52:20Z
<p>The W3C has recently launched an <a href="http://www.w3.org/2005/Incubator/rdb2rdf/" id="link-idd763f48">incubator group about mapping relational data to RDF</a>.</p> <p>From participating in the group for the few initial sessions, I get the following impressions.</p> <p>There is a segment of users, for example from the biomedical community, who do heavy duty <a href="http://dbpedia.org/resource/Data" id="link-id0x1b388bf0">data</a> integration and look to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1a24b198">RDF</a> for managing complexity. Unifying heterogeneous data under OWL ontologies, reasoning, and data integrity, are points of interest.</p> <p>There is another segment that is concerned with semantifying the document web, which topic includes initiatives such as <a href="http://triplify.org/" id="link-id0x16cb5c48">Triplify</a> and <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1adcd2b8">semantic web</a> search such as <a href="http://sindice.org/" id="link-id0x1a462ee0">Sindice</a>. The emphasis there is on minimizing entry cost and creating critical mass. The next one to come will clean up the semantics, if these need be cleaned up at all.</p> <p>(Some cleanup is taking place with <a href="http://www.mpi-inf.mpg.de/~suchanek/downloads/yago/" id="link-id0x17faa940">Yago</a> and <a href="http://zitgist.com/about/" id="link-id0x1acd23f0">Zitgist</a>, but this is a matter for a different post.)</p> <p>Thus, technically speaking, the mapping landscape is diverse, but ETL (extract-transform-load) seems to predominate. The biomedical people make data warehouses for answering specific questions. The web people are interested in putting data out in the expectation that the next player will warehouse it and allow running complex meshups against the whole of the RDF-ized web.</p> <p>As one would expect, these groups see different issues and needs. Roughly speaking, one is about quality and structure and the other is about volume.</p> <p>Where do we stand?</p> <p>We are with the research data warehousers in saying that the mapping question is very complex and that it would indeed be nice to bypass ETL and go to the source <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x17f28d60">RDBMS</a>(s) on demand. Projects in this direction are ongoing.</p> <p>We are with the web people in building large RDF stores with scalable query answering for arbitrary RDF, for example, hosting a lot of the Linking Open Data sets, and working with Zitgist.</p> <p>These things are somewhat different.</p> <p>At present, both the research warehousers and the web scalers predominantly go for ETL.</p> <p>This is fine by us as we definitely are in the large RDF store race.</p> <p>Still, mapping has its point. A relational store will perform quite a bit faster than a quad store if it has the right covering indices or application-specific compressed columnar layout. Thus, there is nothing to block us from querying analytics in <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1a2c81c8">SPARQL</a>, once the obviously necessary extensions of sub-query, expressions and aggregation are in place.</p> <p>To cite an example, the Ordnance Survey of the UK has a GIS system running on <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x18a82010">Oracle</a> with an entry pretty much for each mailbox, lamp post, and hedgerow in the country. According to Ordnance Survey, this would be 1 petatriple, 1e15 triples. "Such a big server farm that we'd have to put it on our map," as Jenny Harding put it at <a href="http://www.eswc2008.org/" id="link-id0x16533418">ESWC2008</a>. I'd add that an even bigger map entry would be the power plant needed to run the 100,000 or so PCs this would take. This is counting 10 gigatriples per PC, which would not even give very good working sets.</p> <p>So, on-the-fly RDBMS-to-RDF mapping in some cases is simply necessary. Still, the benefits of RDF for integration can be preserved if the translation middleware is smart enough. Specifically, this entails knowing what tables can be joined with what other tables and pushing maximum processing to the RDBMS(s) involved in the query.</p> <p>You can download the slide set I used for the <a href="http://virtuoso.openlinksw.com" id="link-id0x16c57ed0">Virtuoso</a> presentation for the RDB to RDF mapping incubator group (<a href="http://virtuoso.openlinksw.com/wiki/main/Main/VirtPresentations/Relational2RDF.ppt" id="link-id106f9e88">PPT</a>; <a href="http://virtuoso.openlinksw.com/wiki/main/Main/VirtPresentations" id="link-id10a8dc90">other formats</a> coming soon). The main point is that real integration is hard and needs smart query splitting and optimization, as well as real understanding of the databases and subject matter from the <a href="http://dbpedia.org/resource/Information" id="link-id0x1b132910">information</a> architect. Sometimes in the web space it can suffice to put data out there with trivial RDF translation and hope that a search engine or such will figure out how to join this with something else. For the enterprise, things are not so. Benefits are clear if one can navigate between disjoint silos but making this accurate enough for deriving business conclusions, as well as efficient enough for production, is a soluble and non-trivial question.</p> <p>We will show the basics of this with the <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x17fc7b58">TPC-H</a> mapping, and by joining this with physical triples. We will also make a set of TPC-H format table sets, make mappings between keys in one to keys in the other, and show joins between the two. The SPARQL querying of one such data store is a done deal, including the SPARQL extensions for this. There is even a demo paper, Business Intelligence Extensions for SPARQL (<a href="http://virtuoso.openlinksw.com/wiki/main/Main/VirtPresentations/RDFAndMapped_BI.pdf" id="link-id12ea4b18">PDF</a>; <a href="http://virtuoso.openlinksw.com/wiki/main/Main/VirtPresentations" id="link-id106e1810">other formats</a> coming soon), by us on the subject in the ESWC 2008 proceedings. If there is an issue left, it is just the technicality of always producing <a href="http://dbpedia.org/resource/SQL" id="link-id0x18439b70">SQL</a> that looks hand-crafted and hence is better understood by the target RDBMS(s). For example, Oracle works better if one uses an <code>IN</code> sub-query instead of the equivalent existence test.</p> <p>Follow this <a href="http://dbpedia.org/resource/Blog" id="link-id0x16c29ea0">blog</a> for more on the topic; published papers are always a limited view on the matter.</p>
2008-06-11T13:15:19.000010-04:00
ESWC 2008
http://www.openlinksw.com:443/weblog/oerling/?date=2008-06-09#1374
2008-06-09T13:49:15Z
<p>Yrjänä Rankka and I attended <a href="http://www.eswc2008.org/" id="link-id10b7a038">ESWC2008</a> on behalf of OpenLink.</p> <p>We were invited at the last minute to give a <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id105df758">Linked Open Data</a> talk at Paolo Bouquet's Identity and Reference workshop. We also had a demo of <a href="http://dbpedia.org/resource/SPARQL" id="link-id12eacca0">SPARQL</a> BI (<a href="http://virtuoso.openlinksw.com/wiki/main/Main/VirtPresentations/ESWC2008%20SPARQL%20BI%20OpenLink.ppt" id="link-id10b43e58">PPT</a>); <a href="http://virtuoso.openlinksw.com/wiki/main/Main/VirtPresentations" id="link-id1116d8f0">other formats coming soon</a>), our business intelligence extensions to <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1843a368">SPARQL</a> as well as joining between relational <a href="http://dbpedia.org/resource/Data" id="link-id10badc40">data</a> mapped to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id108edaf8">RDF</a> and native <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1843a3b0">RDF</a> <a href="http://dbpedia.org/resource/Data" id="link-id0x1843a3c8">data</a>. i was also speaking at the social networks panel chaired by Harry Halpin.</p> <p>I have gathered a few impressions that I will share in the next few posts (<a href="http://www.openlinksw.com/weblog/oerling/?id=1375" id="link-id107298e0">1 - RDF Mapping</a>, <a href="http://www.openlinksw.com/weblog/oerling/?id=1376" id="link-id10b3a530">2 - DARQ</a>, <a href="http://www.openlinksw.com/weblog/oerling/?id=1377" id="link-id107290e0">3 - voiD</a>, <a href="http://www.openlinksw.com/weblog/oerling/?id=1378" id="link-id1071a950">4 - Paradigmata</a>). <i>Caveat: This is not meant to be complete or impartial press coverage of the event but rather some quick comments on issues of personal/OpenLink interest. The fact that I do not mention something does not mean that it is unimportant.</i> </p> <h2>The voiD Graph</h2> <p> <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x16c781e0">Linked Open Data</a> was well represented, with Chris Bizer, Tom Heath, ourselves and many others. The great advance for <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id108f3c48">LOD</a> this time around is <a href="http://community.linkeddata.org/MediaWiki/index.php?MetaLOD#Kick-off_meeting_at_ESWC08" id="link-id10df9830">voiD, the Vocabulary of Interlinked Datasets</a>, a means to describe what in fact is inside the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x16c78228">LOD</a> cloud, how to join it with what and so forth. Big time important if there is to be a <a href="http://www.openlinksw.com/weblog/oerling/?id=1377" id="link-iddf74578">web of federatable data sources</a>, feeding directly into what we have been saying for a while about SPARQL end-point self-description and discovery. There is reasonable hope of having something by the date of <a href="http://www.linkeddataplanet.com/" id="link-id10dd0848">Linked Data Planet</a> in a couple of weeks.</p> <h2>Federating</h2> <p>Bastian Quilitz gave a talk about his <a href="http://darq.sourceforge.net/" id="link-id108746e8">DARQ</a>, a federated version of Jena's ARQ.</p> <p>Something like <a href="http://darq.sourceforge.net/" id="link-id0x16c782e8">DARQ</a>'s optimization statistics should make their way into the <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id10992348">SPARQL protocol</a> as well as the voiD data set description.</p> <p>We really need federation but more on this in <a href="http://www.openlinksw.com/weblog/oerling/?id=1376" id="link-id1059d688">a separate post</a>.</p> <h2> <a href="http://xsparql.deri.ie/" id="link-id10314308">XSPARQL</a> </h2> <p>Axel Polleres et al had a paper about <a href="http://xsparql.deri.ie/" id="link-id0x1a2d8458">XSPARQL</a>, a merge of <a href="http://dbpedia.org/resource/XQuery" id="link-id10b98e90">XQuery</a> and SPARQL. While visiting DERI a couple of weeks back and again at the conference, we talked about OpenLink implementing the spec. It is evident that the engines must be in the same process and not communicate via the <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id0x1d99c1d0">SPARQL protocol</a> for this to be practical. We could do this. We'll have to see when.</p> <p>Politically, using <a href="http://dbpedia.org/resource/XQuery" id="link-id0x1acae1f0">XQuery</a> to give expressions and XML synthesis to SPARQL would be fitting. These things are needed anyhow, as surely as aggregation and sub-queries but the latter would not so readily come from XQuery. Some rapprochement between RDF and XML folks is desirable anyhow.</p> <h2>Panel: Will the Sem Web Rise to the Challenge of the Social Web?</h2> <p>The social web panel presented the question of whether the sem web was ready for prime time with data portability.</p> <p>The main thrust was expressed in Harry Halpin's rousing closing words: "Men will fight in a battle and lose a battle for a cause they believe in. Even if the battle is lost, the cause may come back and prevail, this time changed and under a different name. Thus, there may well come to be something like our <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id122f4da0">semantic web</a>, but it may not be the one we have worked all these years to build if we do not rise to the occasion before us right now."</p> <p>So, how to do this? Dan Brickley asked the audience how many supported, or were aware of, the latest Web 2.0 things, such as <a href="http://dbpedia.org/page/OAuth" id="link-idf300bc0">OAuth</a> and <a href="http://dbpedia.org/page/OpenID" id="link-id10ce7a40">OpenID</a>. A few were. The general idea was that research (after all, this was a research event) should be more integrated and open to the world at large, not living at the "outdated pace" of a 3 year funding cycle. Stefan Decker of DERI acquiesced in principle. Of course there is impedance mismatch between specialization and interfacing with everything.</p> <p>I said that triples and vocabularies existed, that OpenLink had <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id1210dbf8">ODS</a> (<a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id11076be8">OpenLink Data Spaces</a>, <a href="http://community.linkeddata.org/" id="link-id10d46710">Community LinkedData</a>) for managing one's data-web presence, but that scale would be the next thing. Rather large scale even, with 100 gigatriples (Gtriples) reached before one even noticed. It takes a lot of PCs to host this, maybe $400K worth at today's prices, without replication. Count 16G ram and a few cores per Gtriple so that one is not waiting for disk all the time.</p> <p>The tricks that Web 2.0 silos do with app-specific data structures and app-specific partitioning do not really work for RDF without compromising the whole point of smooth schema evolution and tolerance of ragged data.</p> <p>So, simple vocabularies, minimal inference, minimal blank nodes. Besides, note that the inference will have to be done at run time, not forward-chained at load time, if only because users will not agree on what sameAs and other declarations they want for their queries. Not to mention spam or malicious sameAs declarations!</p> <p>As always, there was the question of business models for the open data web and for semantic technologies in general. As we see it, <a href="http://dbpedia.org/resource/Information" id="link-id108b7688">information</a> overload is the factor driving the demand. Better contextuality will justify semantic technologies. Due to the large volumes and complex processing, a data-as-service model will arise. The data may be open, but its query infrastructure, cleaning, and keeping up-to-date, can be monetized as services.</p> <h2>Identity and Reference</h2> <p>For the identity and reference workshop, the ultimate question is metaphysical and has no single universal answer, even though people, ever since the dawn of time and earlier, have occupied themselves with the issue. Consequently, I started with the Genesis quote where Adam called things by <i>nominibus suis</i>, off-hand implying that things would have some intrinsic ontologically-due names. This would be among the older references to the question, at least in widely known sources.</p> <p>For present purposes, the consensus seemed to be that what would be considered the same as something else depended entirely on the application. What was similar enough to warrant a sameAs for cooking purposes might not warrant a sameAs for chemistry. In fact, complete and exact sameness for URIs would be very rare. So, instead of making generic weak similarity assertions like similarTo or seeAlso, one would choose a set of strong sameAs assertions and have these in effect for query answering if they were appropriate to the granularity demanded by the application.</p> <p>Therefore sameAs is our permanent companion, and there will in time be malicious and spam sameAs. So, nothing much should be materialized on the basis of sameAs assertions in an <a href="http://dbpedia.org/resource/Open_world_assumption" id="link-id10c4dfd0">open world</a>. For an app-specific warehouse, sameAs can be resolved at load time.</p> <p>There was naturally some apparent tension between the Occam camp of <a href="http://dbpedia.org/resource/Entity" id="link-id105fd240">entity</a> name services and the LOD camp. I would say that the issue is more a perceived polarity than a real one. People will, inevitably, continue giving things names regardless of any centralized authority. Just look at natural language. But having a dictionary that is commonly accepted for established domains of discourse is immensely helpful.</p> <h2>CYC and NLP</h2> <p>The semantic search workshop was interesting, especially CYC's presentation. CYC is, as it were, the grand old man of <a href="http://dbpedia.org/resource/Knowledge" id="link-id10568158">knowledge</a> representation. Over the long term, I would have support of the CYC inference language inside a database query processor. This would mostly be for repurposing the huge <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x17f7dd40">knowledge</a> base for helping in search type queries. If it is for transactions or financial reporting, then queries will be <a href="http://dbpedia.org/resource/SQL" id="link-id130a0a80">SQL</a> and make little or no use of any sort of inference. If it is for summarization or finding things, the opposite holds. For scaling, the issue is just making correct cardinality guesses for query planning, which is harder when inference is involved. We'll see.</p> <p>I will also have a closer look at natural language one of these days, quite inevitably, since <a href="http://zitgist.com/about/" id="link-id10795828">Zitgist</a> (for example) is into <a href="http://dbpedia.org/resource/Entity" id="link-id0x1a2c8bd0">entity</a> disambiguation.</p> <h2>Scale</h2> <p>Garlic gave a talk about their Data Patrol and QDOS. We agree that storing the data for these as triples instead of 1000 or so constantly changing relational tables could well make the difference between next-to-unmanageable and efficiently adaptive.</p> <p>Garlic probably has the largest triple collection in constant online use to date. We will soon join them with our hosting of the whole LOD cloud and <a href="http://sindice.org/" id="link-id0x1b383720">Sindice</a>/<a href="http://zitgist.com/about/" id="link-id0x1b383738">Zitgist</a> as triples.</p> <h2>Conclusions</h2> <p>There is a mood to deliver applications. Consequently, scale remains a central, even the principal topic. So for now we make bigger centrally-managed databases. At the next turn around the corner we will have to turn to federation. The point here is that a planetary-scale, centrally-managed, online system can be made when the workload is uniform and anticipatable, but if it is free-form queries and complex analysis, we have a problem. So we move in the direction of federating and charging based on usage whenever the workload is more complex than making simple lookups now and then.</p> <p>For the <a href="http://virtuoso.openlinksw.com" id="link-id1026ac28">Virtuoso</a> roadmap, this changes little. Next we make data sets available on Amazon EC2, as widely promised at ESWC. With big scale also comes rescaling and repartitioning, so this gets additional weight, as does further parallelizing of single user workloads. As it happens, the same medicine helps for both. At <a href="http://www.linkeddataplanet.com/" id="link-id0x1a2c7eb0">Linked Data Planet</a>, we will make more announcements.</p>
2008-06-11T13:15:11.000008-04:00
Reasoning Matters Contd
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-06-06#1373
2008-06-06T18:29:02Z
<p>I just stumbled across a post titled: <a href="http://clarkparsia.com/weblog/2008/06/06/why-reasoning-matters-consistency-checking/" id="link-id11003f00">Why Reasoning Matters: Consistency Checking</a> from <a href="http://clarkparsia.com/about" id="link-id137e8bc0">Clark and Parsia</a> </p> <p>As you can see from my recent post about how we've started the process of <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1372" id="link-id100b7d20">inoculating DBpedia against the potential dangers of "contextual incoherence"</a>, we are entering a newer era in the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id106c35e0">Semantic Web</a>'s evolution. My post and the one from Clark & Parsia both touch different aspects of the "<a href="http://dbpedia.org/resource/Data">Data</a> Dictionary" for the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x9d80080">Semantic Web</a> issue.</p> <p>Note: in my universe of discourse, a Data Dictionary manifests when the constraints and class hierarchies defined in an ontology (e.g. a <a href="http://dbpedia.org/resource/World_Wide_Web">web</a> accessible shared ontology) are functionally bound to a data manager. Interestingly the binding can take the following forms:</p> <ul> <li>Engine Hosted - which is what you get with <a href="http://docs.openlinksw.com:80/virtuoso/rdfsparqlrule.html#rdfsparqlruleintro" id="link-id105c4408">Virtuoso's in-built Inference Engine</a> </li> <li>External - which is what you get when the Inference Engine is a distinct component from the data manager (example: <a href="http://pellet.owldl.org/owlgres" id="link-id13fa37f8">Owlgres</a> which can sit in front of 3rd party <a href="http://dbpedia.org/resource/SPARQL" id="link-id107127e8">SPARQL</a> endpoints via ARQ)</li> </ul> <p>The classification terminology I use above is very much off-the-cuff, its sole purpose is architectural distinction.</p> <p>Anyway, it's really nice to see that we are entering an era re. the <a href="http://dbpedia.org/resource/Semantic_Web">Semantic Web</a> vision, where the virtues of reasoning are getting simpler to demonstrate and articulate.</p> <p>In a nutshell, the point-point data integration era is coming to an end! The era of intelligent ontology based enterprise data integration is nigh!</p> <p>Of course, there is much more to come on the practical utility front, so stay tuned as we work our way through the <a href="http://dbpedia.org/resource/DBpedia" id="link-id10424078">DBpedia</a> inoculation program.</p>
2008-06-06T14:38:54-04:00
DBpedia receives shot #1 of CLASSiness vaccine
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-06-05#1372
2008-06-05T17:11:34Z
<p> The current live instance of <a href="http://dbpedia.org/resource/DBpedia" id="link-id107c7b60">DBpedia</a> has just received dose #1 of a series of planned "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id10d3ec78">Context</a>" oriented booster shots. These shots seek to to protect <a href="http://dbpedia.org/resource/DBpedia" id="link-id143648f0">DBpedia</a> from contextual incoherence as it grows in <a href="http://dbpedia.org/resource/Data">data</a> set expanse and popularity. Dose #1 (vaccine label: <a href="http://www.mpi-inf.mpg.de/~suchanek/downloads/yago/" id="link-id16d497d0">Yago</a>) equips <a href="http://dbpedia.org/resource/DBpedia" id="link-id13f90120">DBpedia</a> with a functional (albeit non exclusive) Data Dictionary component courtesy of the <a href="http://www.mpi-inf.mpg.de/~suchanek/downloads/yago/" id="link-id10509a08">Yago</a> Class Hierarchy .</p> <p> When the <a href="http://dbpedia.org/resource/DBpedia" id="link-id10a1b378">DBpedia</a> & <a href="http://www.mpi-inf.mpg.de/~suchanek/downloads/yago/" id="link-id10934068">Yago</a> integration took place last year (around WWW2007, Banff) there was a little, but costly omission that occurred: nobody sought to load the <a href="http://www.mpi-inf.mpg.de/~suchanek/downloads/yago/" id="link-id106e47f0">Yago</a> Class Hierarchy into the <a href="http://virtuoso.openlinksw.com" id="link-id13f90890">Virtuoso</a>'s Inference Engine :-(</p> <p> Anyway, the Class Hierarchy has now been loaded into the <a href="http://virtuoso.openlinksw.com" id="link-id100004f8">Virtuoso</a>'s inference engine (as <a href="http://virtuoso.openlinksw.com" id="link-id139900e8">Virtuoso</a> Inference Rules) and the following queries are now feasible using the live <a href="http://virtuoso.openlinksw.com" id="link-id109b02c8">Virtuoso</a> based <a href="http://dbpedia.org/resource/DBpedia" id="link-id143624d8">DBpedia</a> instance hosted by <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id12f918c0">OpenLink Software</a>:</p> <p> -- Find all Fiction Books associated with a property "dbpedia:name" that has literal value: "The Lord of the Rings" .</p> <p> <span _fck_bookmark="1" style="display: none; "> </span> </p> <p> <span style="font-size:16px;"><span style="font-family:courier new,courier,monospace;">DEFINE input:inference "http://dbpedia.org/resource/inference/rules/yago#"<br /> </span></span> </p> <p> <span style="font-size:16px;"> <span style="font-family:courier new,courier,monospace;"><span class="Apple-style-span">PREFIX rdf: &lt;http://www.w3.org/1999/02/22-rdf-syntax-ns#&gt;</span> </span></span> </p> <p> <span style="font-size:16px;"><span style="font-family:courier new,courier,monospace;">PREFIX dbpedia: &lt;http://dbpedia.org/property&gt;</span></span> </p> <p> <span style="font-size:16px;"><span style="font-family:courier new,courier,monospace;">PREFIX yago: &lt;http://dbpedia.org/class/yago&gt;&nbsp;</span></span> </p> <div> <span style="font-size:16px;"><span style="font-family:courier new,courier,monospace;"><br /> </span></span> </div> <div> <span style="font-size:16px;"><span style="font-family:courier new,courier,monospace;">SELECT DISTINCT ?s</span></span> </div> <div> <span style="font-size:16px;"><span style="font-family:courier new,courier,monospace;">FROM < xmlns="http" dbpedia.org="dbpedia.org">//dbpedia.org></span></span> </div> <div> <span style="font-size:16px;"><span style="font-family:courier new,courier,monospace;">WHERE {</span></span> </div> <div> <span style="font-size:16px;"><span style="font-family:courier new,courier,monospace;">?s a yago:Fiction106367107 .</span></span> </div> <div> <span style="font-size:16px;"><span style="font-family:courier new,courier,monospace;">?s dbpedia:name "The Lord of the Rings"@en .</span></span> </div> <div> <span style="font-size:16px;"><span style="font-family:courier new,courier,monospace;">}</span></span> </div> <p> -- Variant of query with <a href="http://virtuoso.openlinksw.com" id="link-id10870920">Virtuoso</a>'s Full Text Index extension via the bif:contains function/magic predicate</p> <p> <span style="font-size:14px;"><span style="font-family:courier new,courier,monospace;">DEFINE input:inference "http://dbpedia.org/resource/inference/rules/yago#"</span></span> </p> <p> <span style="font-size:14px;"><span style="font-family:courier new,courier,monospace;">PREFIX rdf: &lt;http://www.w3.org/1999/02/22-rdf-syntax-ns#&gt;</span></span> </p> <p> <span style="font-size:16px;"><span style="font-family:courier new,courier,monospace;">PREFIX dbpedia: &lt;http://dbpedia.org/property&gt;</span></span> </p> <p> <span style="font-size:16px;"><span style="font-family:courier new,courier,monospace;">PREFIX yago: &lt;http://dbpedia.org/class/yago&gt;&nbsp;</span></span> </p> <p> <span style="font-size:16px;"><span style="font-family:courier new,courier,monospace;">SELECT DISTINCT ?s ?n</span></span> </p> <p> <span style="font-size:16px;"><span style="font-family:courier new,courier,monospace;">FROM < xmlns="http" dbpedia.org="dbpedia.org">//dbpedia.org></span></span> </p> <p> <span style="font-size:16px;"><span style="font-family:courier new,courier,monospace;">WHERE {</span></span> </p> <p> <span style="font-size:16px;"><span style="font-family:courier new,courier,monospace;">?s a yago:Fiction106367107 .</span></span> </p> <p> <span style="font-size:16px;"><span style="font-family:courier new,courier,monospace;">?s dbpedia:name ?n .</span></span> </p> <p> <span style="font-size:16px;"><span style="font-family:courier new,courier,monospace;">?n bif:contains 'Lord and Rings'</span></span> </p> <p> <span style="font-size:16px;"><span style="font-family:courier new,courier,monospace;">}</span></span> </p> <p> -- Retrieve all individuals instances of Fiction Class which should include all Books.</p> <p> <span style="font-family:courier new,courier,monospace;"><span style="font-size:16px;">DEFINE input:inference "http://dbpedia.org/resource/inference/rules/yago#"<br /> </span></span> </p> <p> <span style="font-family:courier new,courier,monospace;"> <span style="font-size:16px;"><span class="Apple-style-span">PREFIX rdf: &lt;http://www.w3.org/1999/02/22-rdf-syntax-ns#&gt;</span> </span></span> </p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"> <span style="font-family:courier new,courier,monospace;"><span style="font-size:16px;">PREFIX dbpedia: &lt;http://dbpedia.org/property&gt;<br /> </span></span> </p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"> <span style="font-family:courier new,courier,monospace;"><br /> </span> </p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"> <span style="font-family:courier new,courier,monospace;"><span style="font-size:16px;">PREFIX yago: &lt;http://dbpedia.org/class/yago&gt;&nbsp;</span></span> </p> <div> <span style="font-family:courier new,courier,monospace;"><br /> </span> </div> <div> <span style="font-family:courier new,courier,monospace;">SELECT DISTINCT ?s</span> </div> <div> <span style="font-family:courier new,courier,monospace;">FROM < xmlns="http" dbpedia.org="dbpedia.org">//dbpedia.org></span> </div> <div> <span style="font-family:courier new,courier,monospace;">WHERE {</span> </div> <div> <span style="font-family:courier new,courier,monospace;">?s a yago:Fiction106367107 .</span> </div> <div> <span style="font-family:courier new,courier,monospace;">} LIMIT 50</span> </div> <p> Note: you can also move the inference pragmas to the <a href="http://virtuoso.openlinksw.com" id="link-id13dd0d20">Virtuoso</a> Sever side i.e place the inference rules in a server instance config file, thereby negating the need to place "define input:inference 'http://dbpedia.org/resource/inference/rules/yago#'" pragmas directly in your <a href="http://dbpedia.org/resource/SPARQL" id="link-id10dddd08">SPARQL</a> queries.</p> <h3> Related</h3> <ul> <li> Mike's <a href="http://www.mkbergman.com/?p=431" id="link-id13f2f318">UMBEL: Making Linked Data Classy</a>post</li> <li> Fred's announcement about the <a href="http://fgiasson.com/blog/index.php/2007/12/20/second-version-of-yago-more-facts-and-entities/" id="link-id10a1b178">Yago revamp en route to UMBEL</a> </li> <li> <uo> <a href="http://fgiasson.com/blog/index.php/2008/04/20/exploding-the-domain-umbel-web-services-by-zitgist/" id="link-id14363358">Expanding Data Object Domains via UMBEL</a> </uo> </li> <li> My <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=umbel&type=text&output=html" id="link-id1101ca98">Prior posts about UMBEL</a> </li> </ul>
2010-07-13T10:45:40-04:00
1995
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-06-04#1371
2008-06-04T21:05:17Z
<p> <a href="http://vanirsystems.com/danielsblog/2008/06/04/1995/#comments" id="link-id10422580">1995</a>: "</p> <p>1995 (and the early 90’s) must have been a visionaries time of dreaming… most of their dreams are happening today.</p> <p>Watch <a href="http://dbpedia.org/resource/Steve_Jobs" id="link-id102d3868">Steve Jobs</a> (then of <a href="http://dbpedia.org/resource/NeXT" id="link-id13fa5140">NeXT</a>) discuss what he thinks will be popular in 1996 and beyond at <a href="http://dbpedia.org/resource/OpenStep" id="link-id10df20e0">OpenStep</a> Days 1995:</p> <ul> <li> <a href="http://uk.youtube.com/watch?v=odqojmB6C_Y" id="link-id103534a0">‘The Future of Objects, 3/5″ by Steve Jobs (YouTube Video)</a> </li> <li> <a href="http://uk.youtube.com/watch?v=j7WpcRReDlo" id="link-id13f31910">‘The Future of Objects, 4/5″ by Steve Jobs (YouTube Video)</a> </li> </ul> <p>Heres a spoiler:</p> <ul> <li>There is static <a href="http://dbpedia.org/resource/World_Wide_Web">web</a> document publishing</li> <li>There is dynamic web document publishing</li> <li>People will want to buy things off the web: e-commerce</li> </ul> <p>The thing that OpenStep propose is:</p> <ul> <li> <a href="http://dbpedia.org/resource/WebObjects" id="link-id10762ed8">WebObjects</a>: an <a href="http://dbpedia.org/resource/Object-oriented_programming" id="link-id1107f680">Object Oriented</a> representation of <a href="http://dbpedia.org/resource/Data">Data</a> available in distributed form over the web</li> </ul> <p>What Steve was suggesting was one of the beginnings of the <a href="http://dbpedia.org/resource/Data_Web" id="link-id1047b568">Data Web</a>! Yep, <a href="http://dbpedia.org/resource/Portable_Distributed_Objects" id="link-id105c5330">Portable Distributed Objects</a> and <a href="http://dbpedia.org/resource/Enterprise_Objects_Framework" id="link-id1006c850">Enterprise Objects Framework</a> was one of the influences of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id143cf598">Semantic Web</a> / <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1075c898">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x1e9ade30">Web</a>…. not surprising as <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id10b56c80">Tim Berners-Lee</a> designed the initial web stack on a <a href="http://dbpedia.org/resource/NeXT" id="link-id105edcb0">NeXT</a> computer!</p> <p>I’m going to spend a little time this evening figuring out how much ‘distributed objects’ stuff has been taken from the OpenStep stuff into the Objective-<a href="http://dbpedia.org/resource/C_(programming_language)" id="link-id0x19fe21b8">C</a> + Cocoa environment. (<- I guess I must be quite geeky ;-))</p>" <p>(Via <a href="http://vanirsystems.com/danielsblog" id="link-id1092ed90">Daniel Lewis</a>.)</p>
2008-06-06T07:54:33.000010-04:00
Nice Presentation about Semantic Web by Nova Spivack
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-06-03#1370
2008-06-03T18:53:47Z
<p> <a href="http://thenextweb.org/2008/06/03/video-nova-spivack-making-sense-of-the-semantic-web/" id="link-id1046bfc8">Nova Spivack delivers a nice Semantic Web presentation</a> at <a href="http://2008.thenextweb.org/" id="link-idff1b7d8">NextWeb</a>. My only differences with Nova are:</p> <ul>-- Timeframe - I believe we are closer to 2020 than he envisages</ul> <ul>-- Business models - there are going to be new business models courtesy of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1f71c1b0">Semantic Web</a> effect.</ul> <p>Anyway, enjoy!</p> <p>BTW - The only reason why the <a href="http://dbpedia.org/resource/Semantic_Web">Semantic Web</a> is perceived as complex relative to the original Document <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> is simple this: The Semantic Web was designed in public view by the W3C and many collaborators, whereas the Document Web simply came into public view and consciousness as a somewhat finished solution.</p>
2008-06-03T15:06:33.000004-04:00
Virtuoso Cluster Paper
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-05-30#1369
2008-05-30T10:02:04Z
<div> <div style="display:none;">Virtuoso Cluster Paper</div> <div> <div>We have a new article on <a href="http://virtuoso.openlinksw.com" id="link-id10424890">Virtuoso</a> cluster, submitted to ISWC 2008.</div> <div> Right now we are working on hosting the billion triples challenge <a href="http://dbpedia.org/resource/Data" id="link-id1077f800">data</a> set at Amazon EC2 using <a href="http://virtuoso.openlinksw.com" id="link-id102117f0">Virtuoso</a> Cluster. This will be the first publicly available instance of <a href="http://virtuoso.openlinksw.com" id="link-id0x20387e80">Virtuoso</a> Cluster and all interested may then instantiate their own copy on the EC2 infrastructure. </div> <br /> <div> <a href="http://www.openlinksw.com/weblog/oerling/2008iswc_webscale_rdf.pdf" id="link-id10af2f30">Towards Web Scale RDF</a> <br /> <a href="http://www.openlinksw.com/weblog/oerling/RDFAndMapped_BI.pdf" id="link-idfedf9f0">Integrating Open Sources and Relational Data with SPARQL</a> <br /> <a href="http://www.openlinksw.com/weblog/oerling/bisparql2.pdf" id="link-id106e5418">Business Intelligence Extensions for SPARQL</a> <br /> </div> <br /> <div> Look for a separate announcement in the near future. </div> </div> </div>
2008-05-30T06:02:05-04:00
Virtuoso Cluster Paper
http://www.openlinksw.com:443/weblog/oerling/?date=2008-05-30#1368
2008-05-30T09:57:47Z
<p>We have a new article on <a href="http://virtuoso.openlinksw.com">Virtuoso</a> Cluster, submitted to ISWC 2008.</p> <p>Right now we are working on hosting the billion triples challenge <a href="http://dbpedia.org/resource/Data">data</a> set on Amazon EC2 using Virtuoso Cluster. This will be the first publicly available instance of Virtuoso Cluster and all interested may then instantiate their own copy on the EC2 infrastructure.</p> <ul> <li> <p> <a href="http://www.openlinksw.com/weblog/oerling/2008iswc_webscale_rdf.pdf" id="link-id1062bc98">Towards Web Scale RDF</a> (PDF)</p> </li> <li> <p> <a href="http://www.openlinksw.com/weblog/oerling/RDFAndMapped_BI.pdf" id="link-id1063efa8">Integrating Open Sources and Relational Data with SPARQL</a> (PDF)</p> </li> <li> <p> <a href="http://www.openlinksw.com/weblog/oerling/bisparql2.pdf" id="link-id10c37708">Business Intelligence Extensions for SPARQL</a> (PDF)</p> </li> </ul> <p>Look for a separate announcement in the near future.</p> <a href="index.vspx?tag=database" rel="tag" style="display:none;">database</a><a href="index.vspx?tag=databases" rel="tag" style="display:none;">databases</a><a href="index.vspx?tag=rdf" rel="tag" style="display:none;">rdf</a><a href="index.vspx?tag=semanticweb" rel="tag" style="display:none;">semanticweb</a><a href="index.vspx?tag=sparql" rel="tag" style="display:none;">sparql</a><a href="index.vspx?tag=virtuoso" rel="tag" style="display:none;">virtuoso</a>
2008-05-30T10:36:31-04:00
Web Evolution
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-05-26#1367
2008-05-26T14:51:59Z
<a href="http://yihongs-research.blogspot.com" id="link-id109b5500">Yihong Ding</a> has posted an interesting series of posts under the banner: <a href="http://feeds.feedburner.com/~r/ThinkingSpace/~3/297992608/revision-of-web-evolution-series.html" id="link-id130cb6e8">Web Evolution</a>. Post number 4 in the series covers: <a href="http://yihongs-research.blogspot.com/2007/08/mapping-between-web-evolution-and-human.html" id="link-id10275ec0">Web Evolution and Human Growth</a>. This particular post is orthogonal (related but independent) to some of my <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=data%20web%20evolution&type=text&output=html" id="link-id10b1cf18">earlier posts about Web Evolution</a>.
2008-05-27T07:45:51.000003-04:00
State of the Semantic Web Presentation
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-05-22#1365
2008-05-22T20:38:28Z
<p>Unfortunately a number of Linking Open <a href="http://dbpedia.org/resource/Data">Data</a> (<a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-idffe3680">LOD</a>) community / <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1017b420">Linked Data</a> tribe members (myself included) aren't at the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id10035c28">Semantic Web</a> Technologies conference in San Jose (we are in a busy period for <a href="http://idehen.net/dataspace/kidehen/calendar/MyCalendar" id="link-id10228c50">Semantic Web Technology related Conferences</a>). But all isn't lost as <a href="http://www.w3.org/People/Ivan/" id="link-id100be140">Ivan Herman</a> (W3C <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id10181b78">Semantic Web</a> Activity Lead) , <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id1004a7e8">LOD</a> member, and SWEO colleague has carried the banner with aplomb.</p> <p>Ivan's presentation titled: <a href="http://www.w3.org/2008/Talks/0518-SanJose-IH/HTML/Overview.html" id="link-id11011990">State of the Semantic Web</a>, is a must view for those who need a quick update on where things are re. the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id101797b0">Semantic Web</a> in general.</p> <p>I also liked the fact that in proper "Lead by example" manner, his presentation isn't PDF or PPT based, it's a <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> Document :-)</p> <p>Hint: as per usual, this post contains a <a href="http://dbpedia.org/resource/Linked_Data" id="link-id100bdc28">Linked Data</a> demo nugget. This time around, it's in the form of a shared calendar covering a large number of <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id1002dd00">Semantic Web</a> Technology events. All I had to do was subscribe to a number of WebDAV accessible iCal files from my <a href="http://idehen.net/dataspace/kidehen/calendar/MyCalendar" id="link-id10f90900">Calendar Data Space</a> and the platform did the rest i.e. produce <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10044188">Linked Data</a> Objects for events associated with a plethora of conferences.</p> <p>If you assimilate Ivan's presentation properly, you will note I've just generated, and shared, a large number of URIs covering a range of conference events. Thus, you can extend my contributions (thereby enriching the <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id1018ea80">GGG</a>) by simply associating additional data from your <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10180538">Linked</a> <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id10103330">Data Space</a> with mine. All you have to do is use my calendar data objects URIs in your statements.</p>
2008-05-23T06:53:08-04:00
Context, Tagging, Semantic Web, and Linked Data (Updated)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-05-22#1366
2008-05-22T17:23:02Z
<p>Courtesy of <a href="http://novaspivack.typepad.com/about.html" id="link-id101d8750">Nova Spivack</a>'s post titled: <a href="http://feeds.feedburner.com/~r/typepad/MindingThePlanet/~3/295624567/tagging-and-the.html" id="link-id11067248">Tagging and the Semantic Web: Tags as Objects</a>, I stumbled across a related post by <a href="http://www.designmills.com/" id="link-idffb9a38">John Clarke</a> titled: <a href="http://feeds.feedburner.com/~r/DesignMills/~3/294554634/" id="link-id101d6138">Tagging and the Semantic Web</a>. Both of these posts use the common practice of tagging to shed light on the increasing realization that "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id11011f98"><a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1356" id="link-id1003f248">The Pursuit of Context</a></a>" is the fusion point between the current <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> and its evolution into a structured Web of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id101d6788">Linked Data</a>.</p> <h3>How Semantic Tagging Works (from a 1000 feet)</h3> <p>When tagging a document, the semantic tagging service passes the content of a target document through a processing pipeline (a distillation process of sorts) that results in automagic extraction of the following:</p> <ul> -- <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id1015fdd0">Named Entities</a> </ul> <ul>-- <a href="http://dbpedia.org/resource/Topic" id="link-id100ccff8">Subject matter Entities</a> (<a href="http://dbpedia.org/resource/Topic" id="link-idfe9a898">Subject matter Concepts</a> reflecting topics covered by the document</ul> <p>Once the extraction phase is completed, a user is presented with a list of "suggested tags" using a variety of user interaction techniques. The literal values of elected Tags are then associated with one or more <a href="http://dbpedia.org/resource/Tag" id="link-idfed5eb0">Tag</a> and <a href="http://dbpedia.org/resource/Tag" id="link-id101ae0c8">Tag</a> Meaning <a href="http://dbpedia.org/resource/Data">Data</a> Objects, with each Object type endowed with a unique Identifier.</p> <h3>Issues to Note</h3> <p>Broad acceptance that: "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id100b9010">Context</a> is king", is gradually taking shape. That said, "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id101d2670">Context</a>" landlocked within Literal values offers little over what we have right now (e.g. at <a href="http://del.icio.us" id="link-id1004be08">Del.icio.us</a> or <a href="http://www.technorati.com" id="link-id100421c8">Technorati</a>), long term. By this I mean: if the end product of semantically enhanced tagging leaves us with: Literal <a href="http://dbpedia.org/resource/Tag" id="link-id101e5730">Tag</a> values only, Tags associated with <a href="http://dbpedia.org/resource/Tag" id="link-id1004a890">Tag</a> Data Objects endowed with platform specific Identifiers, or <a href="http://dbpedia.org/resource/Tag" id="link-id100364f8">Tag</a> Data Objects with any other Identity scheme that excludes <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id101e6630">HTTP</a>, the ability of Web users to discern or derive multiple perspectives from the base <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id10180868">Context</a> (exposed by semantically enhanced Tags) will be lost, or severely impeded at best.</p> <p>The shape, form, and quality of the lookup substrate that underlies semantic tagging services, ultimately affects "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id10160f28">context</a> fidelity" matters such as <a href="http://dbpedia.org/resource/Entity" id="link-id100f2618">Entity</a> Disambiguation. The importance of quality lookup infrastructure on the burgeoning <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10044b10">Linked Data Web</a> is the reason why <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id10102360">OpenLink Software</a> is intimately involved with the <a href="http://dbpedia.org/resource/DBpedia" id="link-id110760f8">DBpedia</a> and <a href="http://umbel.org/about/" id="link-id1015fc68">UMBEL</a> projects. </p> <h3>Conclusions</h3> <p>I am immensely happy to see that the Web 2.0 and <a href="http://dbpedia.org/resource/Semantic_Web" id="link-idffb8ca8">Semantic Web</a> communities are beginning to coalesce around the issue of "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id101656b0">Context</a>". This was the case at the <a href="http://events.linkeddata.org/ldow2008/" id="link-id1017b878">WWW2008 Linked Data Workshop</a>, I am feeling a similar vibe emerging from the <a href="http://www.semantic-conference.com/" id="link-idffb9978">Semantic Web Technologies</a> conference currently nearing completion in San Jose. Of course, I will be talking about, and demonstrating practical utility of all of this, at the upcoming <a href="http://www.linkeddataplanet.com" id="link-id10042168">Linked Data Planet</a> conference.</p> <h3>Related</h3> <ul> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/tagcloud" id="link-id147a1848">My Data Space Tag Cloud</a> (*a <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x24756e98">Linked Data</a> <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id0x24c2bd20">Space</a>*) </ul> <ul> <a href="http://www.faviki.com/" id="link-id101ac668">Faviki</a> (note: this service needs to expose <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1042cdc0">Linked Data</a> compliant <a href="http://dbpedia.org/resource/Tag" id="link-id1038c2e0">Tag</a> URIs) </ul> <ul> <a href="http://moat-project.org/ontology" id="link-id10199770">MOAT Ontology</a> </ul>
2008-05-27T18:36:37-04:00
ODBC & WODBC Comparison
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-05-20#1364
2008-05-20T19:37:53Z
<p> <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id100eb550">ODBC</a> delivers open <a href="http://dbpedia.org/resource/Data" id="link-idffd2338">data</a> access (by reference) to a broad range of enterprise databases via a '<a href="http://dbpedia.org/resource/C_(programming_language)" id="link-id104fd1d8">C</a>' based API. Thanks to the <a href="http://www.iodbc.org" id="link-id104721b0">iODBC</a> and <a href="http://www.unixodbc.org" id="link-id10954990">unixODBC</a> projects, <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id10494670">ODBC</a> is available across broad range of platforms beyond Windows.</p> <p> <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0xc900928">ODBC</a> identifies <a href="http://dbpedia.org/resource/Data" id="link-id10f82200">data</a> sources using <a href="http://dbpedia.org/resource/Data" id="link-id0xcaad080">Data</a> Source Names (DSNs). </p> <p> WODBC (<a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> Open Database Connectivity) delivers open <a href="http://dbpedia.org/resource/Data">data</a> access to Web Databases / Data Spaces. The Data Source Naming scheme: <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id1009ce40">URI</a> or IRI, is <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id101fc1b0">HTTP</a> based thereby enabling data access by reference via the Web. </p> <p> <a href="http://dbpedia.org/resource/Open_Database_Connectivity">ODBC</a> DSNs bind ODBC client applications to Tables, Views, Stored Procedures. </p> <p>WODBC DSNs bind you to a Data <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id10182a88">Space</a> (e.g. my <a href="http://myopenlink.net/dataspace/person/kidehen" id="link-id105a7858">FOAF based Profile Page</a> where you can use the "Explore Data Tab" to look around if you are a human visitor) or a specific <a href="http://dbpedia.org/resource/Entity" id="link-id10bd8578">Entity</a> within a Data <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id10780dc0">Space</a> (i.e <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id10848e08">Person Entity Me</a>).</p> <p>ODBC Drivers are built using APIs (DBMS Call Level Interfaces) provided by DBMS vendors. Thus, a DBMS vendor can chose not to release an API, or do so selectivity, for competitive advantage or market disruption purposes (it's happened!).</p> <p>WODBC Drivers are also built using APIs (Web Services associated with a Web Data <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id0xcbe6348">Space</a>). These drivers are also referred to as <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=rdf%20middleware&type=text&output=html" id="link-id16564058">RDF Middleware</a> or RDFizers. The "Web" component of WODBC ensures openness, you publish Data with URIs from your <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1064a768">Linked Data</a> Server and that's it; your data <a href="http://en.wikipedia.org/wiki/Data_Spaces">space</a> or specific data entities are live and accessible (by reference) over the Web!</p> <p>So we have come full circle (or cycle), the Web is becoming more of a structured database everyday! What's new is old, and what's old is new! </p> <p>Data Access is everything, without "Data" there is no <a href="http://dbpedia.org/resource/Information" id="link-id100a9de8">information</a> or <a href="http://dbpedia.org/resource/Knowledge" id="link-id10bb67e8">knowledge</a>. Without "Data" there's not notion of vitality, purpose, or value.</p> <p>URIs make or break everything in the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10a71638">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10494400">Web</a> just as ODBC DSNs do within the enterprise. </p> <p>I've deliberately left <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id10a05280">JDBC</a>, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id104e4a70">ADO</a>.<a href="http://dbpedia.org/resource/ADO.NET" id="link-id10215668">NET</a>, and OLE-DB out of this piece due to their respective programming languages and frameworks specificity. None of these mechanisms match the platform availability breadth of ODBC.</p> <p>The Web as a true <a href="http://dbpedia.org/resource/Model-view-controller" id="link-id108ee598">M</a>-<a href="http://dbpedia.org/resource/Model-view-controller" id="link-id0xcda5e90">V</a>-C pattern is now crystalizing. The "M" (Model) component of M-V-C is finally rising to the realm of broad attention courtesy of the "<a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id1024ff08">Linked Data" meme</a> and "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id1831b418">Semantic Web</a>" vision.</p> <p>By the way, M-V-C lines up nicely with Web 1.0 (Web Forms / Pages), Web 2.0 (Web Services based APIs), and Web 3.0 (Data Web, Web of Data, or <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0xb6d0e90">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0xb22a158">Web</a>) :-)</p>
2008-05-20T15:46:11-04:00
Commercializing the Semantic Web
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-05-16#1363
2008-05-16T22:04:01Z
<p>Unfortunately, I could only spend 4 days at the recent <a href="http://www2008.org/" id="link-id196acf60">WWW2008</a> event in <a href="http://dbpedia.org/resource/Beijing" id="link-id1974fe28">Beijing</a> (I departed the morning following the <a href="http://events.linkeddata.org/ldow2008/" id="link-id1863f858">Linked Data Workshop</a>), so I couldn't take my slot on the "Commercializing the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id18990f90">Semantic Web</a> panel" etc.. Anyway, thanks to the <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id0x18f29310">Web</a> I can still inject my points of view in the broad <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> based discourse. Well so I hoped, when I attempted to post a comment to Paul Miller's ZDNet domain hosted <a href="http://dbpedia.org/resource/Blog" id="link-id180d6750">blog</a> thread titled: <a href="http://blogs.zdnet.com/semantic-web/?p=132" id="link-id12d206c0">Commercialising the Semantic Web</a>.</p> <p>Unfortunately, the cost of completing ZDNet's unwieldy signup process simply exceeded the benefits of dropping my comments in their particular space :-( Thus, I'll settle for a trackback ping instead.</p> <p>What follows is the cut and paste of my intended comment contributions to Paul's post.</p> <p>Paul,</p> <p> As discussed earlier this week during <a href="http://blogs.talis.com/nodalities/2008/05/kingsley-idehen-talks-about-openlink-software-linked-data-and-the-semantic-web.php" id="link-id1332fb48">our podcast session</a>, commercialization of <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id17382338">Semantic Web</a> technology shouldn't be a mercurial matter at this stage in the game :-) It's all about looking at how it provides value :-)</p> <p>From the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10d4f4a8">Linked Data</a> angle, the ability to produce, dispatch, and exploit "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id13bed160">Context</a>" across an array of "Perspectives" from a plethora of disparate <a href="http://dbpedia.org/resource/Data" id="link-id1731e5f0">data</a> sources on the Web and/or behind corporate firewalls, offers immense commercial value.</p> <p> <a href="http://developer.yahoo.com/searchmonkey/" id="link-id1975d248">Yahoo's Searchmonkey</a> effort will certainly bring clarity to some of the points I made during the podcast re. the role of URIs as "value consumption tickets" (<a href="http://dbpedia.org/resource/Data" id="link-id173eb7b0">Data</a> Services are exposed via URIs). There has to be a trigger (in user space) that compels Web users to seek broader, or simply varied, perspectives as a response to <a href="http://dbpedia.org/resource/Data" id="link-id0x1c7e7f60">data</a> encountered on the Web. Yahoo! is about to put this light on in a big way (imho).</p> <p>The "self annotating" nature of the Web is what ultimately drives the manifestation of the long awaited <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0xa18a83e8">Semantic Web</a>. I believe I postulated about <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=self%20annotation&type=text&output=html" id="link-id173d7458">"Self Annotation & the Semantic Web" in a number of prior posts</a> which, by the way, should be <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&type=text&kwds=self%20annotation&OpenSearch" id="link-id10b12208">DataRSS compatible right now</a> due to Yahoo's support of OpenSearch <a href="http://dbpedia.org/resource/Data" id="link-id0x1b8412e8">Data</a> Providers (which this <a href="http://dbpedia.org/resource/Blog" id="link-id170b8df8">Blog</a> Space has been for eons).</p> <p>Today, have many communities adding strucuture to the Web (via their respective tools of preference) without explicitly realizing what they are contributing. Every RSS/Atom feed, <a href="http://dbpedia.org/resource/Tag" id="link-id183d5178">Tag</a>, Weblog, Shared Bookmark, <a href="http://dbpedia.org/resource/WikiWord" id="link-id10c5e758">Wikiword</a>, Microformat, Microformat++ (<a href="http://dbpedia.org/resource/Embedded_RDF" id="link-id16d8ee40">eRDF</a> or <a href="http://dbpedia.org/resource/RDFa" id="link-id1059a688">RDFa</a>), <a href="http://dbpedia.org/resource/GRDDL" id="link-id1090ae10">GRDDL</a> stylesheet, and RDFizer etc.. is a piece of structured <a href="http://dbpedia.org/resource/Data">data</a>.</p> <p>Finally, the different communities are all finding ways to work together (thank heavens!) and the results are going to be cataclysmic when it all plays out :-)</p> <p>Data, Structure, and Extraction are the keys to the Semantic Life! First you get the Data in a container (<a href="http://dbpedia.org/resource/Information" id="link-id180e5648">information</a> resource), and then you add Structure to the <a href="http://dbpedia.org/resource/Information" id="link-id103801e0">information</a> resource (RSS, Atom, <a href="http://dbpedia.org/resource/Microformats" id="link-id17825e40">microformats</a>, <a href="http://dbpedia.org/resource/RDFa" id="link-id189a8738">RDFa</a>, <a href="http://dbpedia.org/resource/Embedded_RDF" id="link-id1933d5c0">eRDF</a>, SIOC, FOAF, etc.), once you have Structure RDFization (i.e. transformation to <a href="http://dbpedia.org/resource/Linked_Data" id="link-id19744878">Linked Data</a>) is a synch thanks to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id180dde30">RDF</a> Middleware (as per <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&type=text&kwds=self%20annotation&OpenSearch" id="link-id16dc3130">earlier RDF middleware posts</a>).</p>
2008-05-18T10:58:26.000003-04:00
Commercializing the Semantic Web
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-05-16#1362
2008-05-16T20:02:45Z
<p>Unfortunately, I could only spend 4 days at the recent <a href="http://www2008.org/" id="link-id196acf60">WWW2008</a> event in <a href="http://dbpedia.org/resource/Beijing" id="link-id1974fe28">Beijing</a> (I departed the morning following the <a href="http://events.linkeddata.org/ldow2008/" id="link-id1863f858">Linked Data Workshop</a>), so I couldn't take my slot on the "Commercializing the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id18990f90">Semantic Web</a> panel" etc.. Anyway, thanks to the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> I can still inject my points of view in the broad Web based discourse. Well so I hoped, when I attempted to post a comment to Paul Miller's ZDNet domain hosted <a href="http://dbpedia.org/resource/Blog" id="link-id180d6750">blog</a> thread titled: <a href="http://blogs.zdnet.com/semantic-web/?p=132" id="link-id12d206c0">Commercialising the Semantic Web</a>.</p> <p>Unfortunately, the cost of completing ZDNet's unwieldy signup process simply exceeded the benefits of dropping my comments in their particular space :-( Thus, I'll settle for a trackback ping instead.</p> <p>What follows is the cut and paste of my intended comment contributions to Paul's post.</p> <p>Paul,</p> <p> As discussed earlier this week during <a href="http://blogs.talis.com/nodalities/2008/05/kingsley-idehen-talks-about-openlink-software-linked-data-and-the-semantic-web.php" id="link-id1332fb48">our podcast session</a>, commercialization of <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id17382338">Semantic Web</a> technology shouldn't be a mercurial matter at this stage in the game :-) It's all about looking at how it provides value :-)</p> <p>From the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10d4f4a8">Linked Data</a> angle, the ability to produce, dispatch, and exploit "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id13bed160">Context</a>" across an array of "Perspectives" from a plethora of disparate <a href="http://dbpedia.org/resource/Data" id="link-id1731e5f0">data</a> sources on the Web and/or behind corporate firewalls, offers immense commercial value.</p> <p> <a href="http://developer.yahoo.com/searchmonkey/" id="link-id1975d248">Yahoo's Searchmonkey</a> effort will certainly bring clarity to some of the points I made during the podcast re. the role of URIs as "value consumption tickets" (<a href="http://dbpedia.org/resource/Data" id="link-id173eb7b0">Data</a> Services are exposed via URIs). There has to be a trigger (in user space) that compels Web users to seek broader, or simply varied, perspectives as a response to <a href="http://dbpedia.org/resource/Data" id="link-id0x1c7e7f60">data</a> encountered on the Web. Yahoo! is about to put this light on in a big way (imho).</p> <p>The "self annotating" nature of the Web is what ultimately drives the manifestation of the long awaited <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0xa18a83e8">Semantic Web</a>. I believe I postulated about <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=self%20annotation&type=text&output=html" id="link-id173d7458">"Self Annotation & the Semantic Web" in a number of prior posts</a> which, by the way, should be <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&type=text&kwds=self%20annotation&OpenSearch" id="link-id10b12208">DataRSS compatible right now</a> due to Yahoo's support of OpenSearch <a href="http://dbpedia.org/resource/Data">Data</a> Providers (which this <a href="http://dbpedia.org/resource/Blog" id="link-id170b8df8">Blog</a> Space has been for eons).</p> <p>Today, have many communities adding strucuture to the Web (via their respective tools of preference) without explicitly realizing what they are contributing. Every RSS/Atom feed, <a href="http://dbpedia.org/resource/Tag" id="link-id183d5178">Tag</a>, Weblog, Shared Bookmark, <a href="http://dbpedia.org/resource/WikiWord" id="link-id10c5e758">Wikiword</a>, Microformat, Microformat++ (<a href="http://dbpedia.org/resource/Embedded_RDF" id="link-id16d8ee40">eRDF</a> or <a href="http://dbpedia.org/resource/RDFa" id="link-id1059a688">RDFa</a>), <a href="http://dbpedia.org/resource/GRDDL" id="link-id1090ae10">GRDDL</a> stylesheet, and RDFizer etc.. is a piece of structured data.</p> <p>Finally, the different communities are all finding ways to work together (thank heavens!) and the results are going to be cataclysmic when it all plays out :-)</p> <p>Data, Structure, and Extraction are the keys to the Semantic Life! First you get the Data in a container (<a href="http://dbpedia.org/resource/Information" id="link-id180e5648">information</a> resource), and then you add Structure to the <a href="http://dbpedia.org/resource/Information" id="link-id103801e0">information</a> resource (RSS, Atom, <a href="http://dbpedia.org/resource/Microformats" id="link-id17825e40">microformats</a>, <a href="http://dbpedia.org/resource/RDFa" id="link-id189a8738">RDFa</a>, <a href="http://dbpedia.org/resource/Embedded_RDF" id="link-id1933d5c0">eRDF</a>, SIOC, FOAF, etc.), once you have Structure RDFization (i.e. transformation to <a href="http://dbpedia.org/resource/Linked_Data" id="link-id19744878">Linked Data</a>) is a synch thanks to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id180dde30">RDF</a> Middleware (as per <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&type=text&kwds=self%20annotation&OpenSearch" id="link-id16dc3130">earlier RDF middleware posts</a>).</p>
2008-05-16T16:15:29.000001-04:00
My Talis Podcast re. Semantic Web, Linked Data, and OpenLink Software
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-05-16#1361
2008-05-16T00:10:23Z
<p> <a href="http://blogs.talis.com/nodalities/2008/05/kingsley-idehen-talks-about-openlink-software-linked-data-and-the-semantic-web.php" id="link-id1036b118">My podcast interview</a> with <a href="http://www.linkedin.com/in/pau1mi11er" id="link-id1026ed10">Paul Miller</a> of <a href="http://www.talis.com" id="link-id12d210d8">Talis</a> is out. As I listened to the podcast (naturally awkward affair) I got a first hand sense of Paul's mastery of the art of interviewing, even when dealing with a fast talking <a href="http://dbpedia.org/resource/Data" id="link-id180e1208">data</a> blitzers like me. Personally, I think I still talk a little too fast (the Nigerian in me), especially when the subject matter hones right into the epicenter of my professional passions: Open <a href="http://dbpedia.org/resource/Data" id="link-id1737a258">Data</a> Access and Heterogeneous <a href="http://dbpedia.org/resource/Data" id="link-id180f0668">Data</a> Integration (aka. <a href="http://dbpedia.org/resource/Virtual_Database" id="link-id10c62348">Virtual Database</a> Technology) -- so you may need to rewind every now and then during the interview :-)</p> <p>During this particular podcast interview, I deliberately wanted to have an conversation about the practical value of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id180c9f88">Linked Data</a>, rather than the technical innards. The fundamental utility of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id17387618">Linked Data</a> remains somewhat mercurial, and I am certainly hoping to do my bit at the upcoming <a href="http://dbpedia.org/resource/Linked_Data" id="link-id183ec288">Linked Data</a> Planet conference re. demonstrating and articulating <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1401f250">linked data</a> value across the blurring realms of "the individual" and "the enterprise".</p> <p> <strong>Note to my old schoolmates on Facebook</strong>: when you listen to this podcast you will at least reconcile "Uyi Idehen" with "<a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id180a7060">Kingsley Idehen</a>". Unfortunately, Facebook refuses to let me Identify myself in the manner I choose. Ideally, I would like to have the name: "Kingsley (Uyi) Idehen" associated with my Facebook ID since this is the Identifier known to my personal network of friends, family, and old schoolmates. This Identity predicament is a long running Identity case study in the making.</p>
2008-05-16T12:53:49.000002-04:00
On "Semantic", "Semantic Web", and "Linked Data Web"
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-05-15#1360
2008-05-15T14:11:13Z
<p> <a href="http://novaspivack.typepad.com/" id="link-id102f4e00">Nova Spivack</a> has just penned a post titled: <a href="http://novaspivack.typepad.com/nova_spivacks_weblog/2008/05/on-the-differen.html" id="link-id101a2300">On the Difference Between "Semantic" and "Semantic Web</a>", where he covers the fundamental difference between "Semantic" (what I call "Semantics Inside") and "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id11dd0578">Semantic Web</a>" applications. I would like to extend the distinctions further by adding the "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id10b54ca0">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id106f73d0">Web</a>" distinctions to the developing discourse. </p> <p>The <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1089ff48">Linked Data Web</a> (aka. <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10653828">Linked Data</a>) describes <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id134abfb0">RDF</a> <a href="http://dbpedia.org/resource/Data" id="link-id140283a8">data</a> injected into the Web, where the <a href="http://dbpedia.org/resource/Identity_(object-oriented_programming)" id="link-id1029ebf0">Data Object Identifiers</a> (URIs) in an <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id1011b180">RDF</a> graph (collection of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id103a4960">RDF</a> triples) are endowed with <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id104362d8">HTTP</a> based URIs. The net effect of this approach to <a href="http://dbpedia.org/resource/Data" id="link-id107963a0">Data</a> Object Identity is that it facilitates "Open <a href="http://dbpedia.org/resource/Data" id="link-id1331f640">Data</a> Access by Reference" on the Web (aka <a href="http://dbpedia.org/resource/Data" id="link-id10a3c608">data</a> dereferencing).</p> <p>If you recall pre Web ubiquity, in the enterprise realm for instance, Open Database Connectivity (<a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id12c6dd40">ODBC</a>) emerged as a mechanism for separating <a href="http://dbpedia.org/resource/Data" id="link-id13d6a5b0">Data</a> Access and <a href="http://dbpedia.org/resource/Data" id="link-id10b29488">Data</a> Management in the database oriented Client-Sever model. Although <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id106a8bd8">ODBC</a> gave you access to <a href="http://dbpedia.org/resource/Data">data</a>, the data access entry point took the form of a data access specific naming mechanism called a "Data Source Name" (DSN). <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id106eef18">ODBC</a> DSNs typically exposed Tables or Views. The same thing applies to <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id12c6dfe8">JDBC</a> where a non <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id104cb620">HTTP</a> based URN scheme applies.</p> <p>Zip forward to where we are today on the Web; the Web is evolving from a Document centric Database to a Distributed <a href="http://dbpedia.org/resource/Object_database" id="link-id12d15268">Object Database</a>, and you should see that in <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10716bb8">Linked Data</a> we are now truly looking at the best of all worlds: Web Open Database Connectivity (WODBC) with the following advantages:</p> <ul>- direct Access to a single Record (an <a href="http://dbpedia.org/resource/Entity" id="link-id1037d530">Entity</a>) or Record Sets (<a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id10d48e98">RDF</a> based <a href="http://dbpedia.org/resource/Entity" id="link-id1402c8f0">Entity</a> Sets) by reference over <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id10bae7a8">HTTP</a> across disparate Data Spaces on the Web</ul> <ul>- the ability to mesh disparate data sources without being impeded by back-end DBMS engine model, vendor, host operating development frameworks, or host operating system specificity</ul> <ul>- an opportunity to learn from the enterprise DBMS market and Client-Server markets of yore with regards to the shape and form of next generation <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10fe4558">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10153c98">Web</a> oriented solutions.</ul> <p>To conclude, we now have "Semantics Inside" (<a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id109d1280">RDF</a> or non <a href="http://dbpedia.org/resource/Resource_Description_Framework">RDF</a>), "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id106741a8">Semantic Web</a>" (RDF graphs with Object Identifiers that may or may not be <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id1011cc28">HTTP</a> based), and "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id10793f70">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id149ecc10">Web</a>" (RDF graphs with Object Identifiers that must be <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id10a3b860">HTTP</a> based and dereferencable) oriented applications, in the emerging landscape associated with the "Semantics" moniker.</p> <p>As per usual, this post is a record in my <a href="http://dbpedia.org/resource/Blog" id="link-id1020e240">Blog</a> oriented <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id105cbf90">Data Space</a> on the Web. The permalink of this post is a <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id10ce53a8">URI</a> constructed with <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id1082f0f8">Giant Global Graph</a> enrichment in mind :-)</p>
2008-05-15T14:31:38.000004-04:00
DBpedia Benchmark Revisited
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-05-09#1359
2008-05-09T19:33:42Z
<div> <div style="display:none;">DBpedia Benchmark Revisited</div> <p>We ran the <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x1cd6d0c8">DBpedia</a> benchmark queries again with different configurations of <a href="http://virtuoso.openlinksw.com" id="link-id0x1bf01048">Virtuoso</a>. I had not studied the details of the matter previously but now did have a closer look at the queries.</p> <p>Comparing numbers given by different parties is a constant problem. In the case reported here, we loaded the full DBpedia 3, all languages, with about 198M triples, onto Virtuoso v5 and Virtuoso Cluster v6, all on the same 4 core 2GHz Xeon with 8G RAM. All databases were striped on 6 disks. The Cluster configuration was with 4 processes in the same box.</p> <p>We ran the queries in two variants:</p> <ul> <li>With graph specified in the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1b9d3ca0">SPARQL</a> <code>FROM</code> clause, using the default indices.</li> <li>With no graph specified anywhere, using an alternate indexing scheme.</li> </ul> <p>The times below are for the sequence of 5 queries; individual query times are not reported. I did not do a line-by-line review of the execution plans since they seem to run well enough. We could get some extra mileage from cost model tweaks, especially for the numeric range conditions, but we will do this when somebody comes up with better times.</p> <p>First, about Virtuoso v5: Because there is a query in the set that specifies no condition on S or O and only P, this simply cannot be done with the default indices. With Virtuoso Cluster v6 it sort-of can, because v6 is more space efficient.</p> <p>So we added the index:</p> <blockquote> <code> create bitmap index <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1c364a58">rdf</a>_quad_pogs on rdf_quad (p, o, g, s); </code> </blockquote> <table> <tr> <td> </td> <td align="center"><b>Virtuoso v5 with<br /> gspo, ogps, pogs</b> </td> <td align="center"><b>Virtuoso Cluster v6 with <br />gspo, ogps</b> </td> <td align="center"><b>Virtuoso Cluster v6 with <br />gspo, ogps, pogs</b> </td> </tr> <tr> <td><b>cold</b> </td> <td align="center">210 s</td> <td align="center">136 s</td> <td align="center">33.4 s</td> </tr> <tr> <td><b>warm</b> </td> <td align="center">0.600 s</td> <td align="center">4.01 s</td> <td align="center">0.628 s</td> </tr> </table> <p>OK, so now let us do it without a graph being specified. For all platforms, we drop any existing indices, and --</p> <blockquote> <code> create table r2 (g iri_id_8, s, iri_id_8, p iri_id_8, o any, primary key (s, p, o, g)) <br /> alter index R2 on R2 partition (s int (0hexffff00)); <br /> <br /> log_enable (2); <br /> insert into r2 (g, s, p, o) select g, s, p, o from rdf_quad; <br /> <br /> drop table rdf_quad; <br /> alter table r2 rename RDF_QUAD; <br /> create bitmap index rdf_quad_opgs on rdf_quad (o, p, g, s) partition (o varchar (-1, 0hexffff)); <br /> create bitmap index rdf_quad_pogs on rdf_quad (p, o, g, s) partition (o varchar (-1, 0hexffff)); <br /> create bitmap index rdf_quad_gpos on rdf_quad (g, p, o, s) partition (o varchar (-1, 0hexffff)); </code> </blockquote> <p>The code is identical for v5 and v6, except that with v5 we use <code>iri_id (32 bit)</code> for the type, not <code>iri_id_8 (64 bit)</code>. We note that we run out of IDs with v5 around a few billion triples, so with v6 we have double the ID length and still manage to be vastly more space efficient.</p> <p>With the above 4 indices, we can query the <a href="http://dbpedia.org/resource/Data" id="link-id0x1bae4cd8">data</a> pretty much in any combination without hitting a full scan of any index. We note that all indices that do not begin with s end with s as a bitmap. This takes about 60% of the space of a non-bitmap index for data such as DBpedia.</p> <p>If you intend to do completely arbitrary RDF queries in Virtuoso, then chances are you are best off with the above index scheme.</p> <table> <tr> <td> </td> <td align="center"><b> Virtuoso v5 with<br /> gspo, ogps, pogs</b> </td> <td align="center"><b> Virtuoso Cluster v6 with <br /> spog, pogs, opgs, gpos </b> </td> </tr> <tr> <td><b>warm</b> </td> <td align="center">0.595 s</td> <td align="center">0.617 s</td> </tr> </table> <p>The cold times were about the same as above, so not reproduced.</p> <h3>Graph or No Graph?</h3> <p>It is in the SPARQL spirit to specify a graph and for pretty much any application, there are entirely sensible ways of keeping the data in graphs and specifying which ones are concerned by queries. This is why Virtuoso is set up for this by default.</p> <p>On the other hand, for the open web scenario, dealing with an unknown large number of graphs, enumerating graphs is not possible and questions like which graph of which source asserts x become relevant. We have two distinct use cases which warrant different setups of the database, simple as that.</p> <p>The latter use case is not really within the SPARQL spec, so implementations may or may not support this. For example <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x1cd2db78">Oracle</a> or Vertica would not do this well since they partition data according to graph or predicate, respectively. On the other hand, stores that work with one quad table, which is most of the ones out there, should do it maybe with some configuring, as shown above.</p> <p>Frameworks like Jena are not to my <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x1b300390">knowledge</a> geared towards having a wildcard for graph, although I would suppose this can be arranged by adding some "super-graph" object, a graph of all graphs. I don't think this is directly supported and besides most apps would not need it.</p> <p>Once the indices are right, there is no difference between specifying a graph and not specifying a graph with the queries considered. With more complex queries, specifying a graph or set of graphs does allow some optimizations that cannot be done with no graph specified. For example, bitmap intersections are possible only when all leading key parts are given.</p> <h3>Conclusions</h3> <p>The best warm cache time is with v5; the five queries run under 600 ms after the first go. This is noted to show that all-in-memory with a single thread of execution is hard to beat.</p> <p>Cluster v6 performs the same queries in 623 ms. What is gained in parallelism is lost in latency if all operations complete in microseconds. On the other hand, Cluster v6 leaves v5 in the dust in any situation that has less than 100% hit rate. This is due to actual benefit from parallelism if operations take longer than a few microseconds, such as in the case of disk reads. Cluster v6 has substantially better data layout on disk, as well as fewer pages to load for the same content.</p> <p>This makes it possible to run the queries without the pogs index on Cluster v6 even when v5 takes prohibitively long.</p> <p>The morale of the story is to have a lot of RAM and space-efficient data representation.</p> <p>The DBpedia benchmark does not specify any random access pattern that would give a measure of sustained throughput under load, so we are left with the extremes of cold and warm cache of which neither is quite realistic.</p> <p>Chris Bizer and I have talked on and off about benchmarks and I have made suggestions that we will see incorporated into the Berlin SPARQL benchmark, which will, I believe, be much more informative.</p> <h3>Appendix: Query Text</h3> <p>For reference, the query texts specifying the graph are below. To run without specifying the graph, just drop the <code>FROM <<a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1c371db0">http</a>://dbpedia.org></code> from each query. The returned row counts are indicated below each query's text.</p> <blockquote> <code><pre> sparql SELECT ?p ?o FROM <http://dbpedia.org> WHERE { <http://dbpedia.org/resource/Metropolitan_Museum_of_Art> ?p ?o }; -- 1337 rows sparql PREFIX p: <http://dbpedia.org/property/> SELECT ?film1 ?actor1 ?film2 ?actor2 FROM <http://dbpedia.org> WHERE { ?film1 p:starring <http://dbpedia.org/resource/Kevin_Bacon> . ?film1 p:starring ?actor1 . ?film2 p:starring ?actor1 . ?film2 p:starring ?actor2 . }; -- 23910 rows sparql PREFIX p: <http://dbpedia.org/property/> SELECT ?artist ?artwork ?museum ?director FROM <http://dbpedia.org> WHERE { ?artwork p:artist ?artist . ?artwork p:museum ?museum . ?museum p:director ?director }; -- 303 rows sparql PREFIX geo: <http://www.w3.org/2003/01/geo/wgs84_pos#> PREFIX foaf: <http://xmlns.com/foaf/0.1/> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> SELECT ?s ?homepage FROM <http://dbpedia.org> WHERE { <http://dbpedia.org/resource/Berlin> geo:lat ?berlinLat . <http://dbpedia.org/resource/Berlin> geo:long ?berlinLong . ?s geo:lat ?lat . ?s geo:long ?long . ?s foaf:homepage ?homepage . FILTER ( ?lat <= ?berlinLat + 0.03190235436 && ?long >= ?berlinLong - 0.08679199218 && ?lat >= ?berlinLat - 0.03190235436 && ?long <= ?berlinLong + 0.08679199218) }; -- 56 rows sparql PREFIX geo: <http://www.w3.org/2003/01/geo/wgs84_pos#> PREFIX foaf: <http://xmlns.com/foaf/0.1/> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> PREFIX p: <http://dbpedia.org/property/> SELECT ?s ?a ?homepage FROM <http://dbpedia.org> WHERE { <http://dbpedia.org/resource/New_York_City> geo:lat ?nyLat . <http://dbpedia.org/resource/New_York_City> geo:long ?nyLong . ?s geo:lat ?lat . ?s geo:long ?long . ?s p:architect ?a . ?a foaf:homepage ?homepage . FILTER ( ?lat <= ?nyLat + 0.3190235436 && ?long >= ?nyLong - 0.8679199218 && ?lat >= ?nyLat - 0.3190235436 && ?long <= ?nyLong + 0.8679199218) }; -- 13 rows </pre> </code> </blockquote> </div>
2008-05-12T11:24:43-04:00
DBpedia Benchmark Revisited
http://www.openlinksw.com:443/weblog/oerling/?date=2008-05-09#1358
2008-05-09T19:27:00Z
<p>We ran the <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x1b7f9688">DBpedia</a> benchmark queries again with different configurations of <a href="http://virtuoso.openlinksw.com" id="link-id0x1cca2e00">Virtuoso</a>. I had not studied the details of the matter previously but now did have a closer look at the queries.</p> <p>Comparing numbers given by different parties is a constant problem. In the case reported here, we loaded the full DBpedia 3, all languages, with about 198M triples, onto Virtuoso v5 and Virtuoso Cluster v6, all on the same 4 core 2GHz Xeon with 8G RAM. All databases were striped on 6 disks. The Cluster configuration was with 4 processes in the same box.</p> <p>We ran the queries in two variants:</p> <ul> <li>With graph specified in the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1b77f758">SPARQL</a> <code>FROM</code> clause, using the default indices.</li> <li>With no graph specified anywhere, using an alternate indexing scheme.</li> </ul> <p>The times below are for the sequence of 5 queries; individual query times are not reported. I did not do a line-by-line review of the execution plans since they seem to run well enough. We could get some extra mileage from cost model tweaks, especially for the numeric range conditions, but we will do this when somebody comes up with better times.</p> <p>First, about Virtuoso v5: Because there is a query in the set that specifies no condition on S or O and only P, this simply cannot be done with the default indices. With Virtuoso Cluster v6 it sort-of can, because v6 is more space efficient.</p> <p>So we added the index:</p> <blockquote> <code> create bitmap index <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1cb0b180">rdf</a>_quad_pogs on rdf_quad (p, o, g, s); </code> </blockquote> <table> <tr> <td> </td> <td align="center"><b>Virtuoso v5 with<br /> gspo, ogps, pogs</b> </td> <td align="center"><b>Virtuoso Cluster v6 with <br />gspo, ogps</b> </td> <td align="center"><b>Virtuoso Cluster v6 with <br />gspo, ogps, pogs</b> </td> </tr> <tr> <td><b>cold</b> </td> <td align="center">210 s</td> <td align="center">136 s</td> <td align="center">33.4 s</td> </tr> <tr> <td><b>warm</b> </td> <td align="center">0.600 s</td> <td align="center">4.01 s</td> <td align="center">0.628 s</td> </tr> </table> <p>OK, so now let us do it without a graph being specified. For all platforms, we drop any existing indices, and --</p> <blockquote> <code> create table r2 (g iri_id_8, s, iri_id_8, p iri_id_8, o any, primary key (s, p, o, g)) <br /> alter index R2 on R2 partition (s int (0hexffff00)); <br /> <br /> log_enable (2); <br /> insert into r2 (g, s, p, o) select g, s, p, o from rdf_quad; <br /> <br /> drop table rdf_quad; <br /> alter table r2 rename RDF_QUAD; <br /> create bitmap index rdf_quad_opgs on rdf_quad (o, p, g, s) partition (o varchar (-1, 0hexffff)); <br /> create bitmap index rdf_quad_pogs on rdf_quad (p, o, g, s) partition (o varchar (-1, 0hexffff)); <br /> create bitmap index rdf_quad_gpos on rdf_quad (g, p, o, s) partition (o varchar (-1, 0hexffff)); </code> </blockquote> <p>The code is identical for v5 and v6, except that with v5 we use <code>iri_id (32 bit)</code> for the type, not <code>iri_id_8 (64 bit)</code>. We note that we run out of IDs with v5 around a few billion triples, so with v6 we have double the ID length and still manage to be vastly more space efficient.</p> <p>With the above 4 indices, we can query the <a href="http://dbpedia.org/resource/Data" id="link-id0x6339b80">data</a> pretty much in any combination without hitting a full scan of any index. We note that all indices that do not begin with s end with s as a bitmap. This takes about 60% of the space of a non-bitmap index for data such as DBpedia.</p> <p>If you intend to do completely arbitrary RDF queries in Virtuoso, then chances are you are best off with the above index scheme.</p> <table> <tr> <td> </td> <td align="center"><b> Virtuoso v5 with<br /> gspo, ogps, pogs</b> </td> <td align="center"><b> Virtuoso Cluster v6 with <br /> spog, pogs, opgs, gpos </b> </td> </tr> <tr> <td><b>warm</b> </td> <td align="center">0.595 s</td> <td align="center">0.617 s</td> </tr> </table> <p>The cold times were about the same as above, so not reproduced.</p> <h3>Graph or No Graph?</h3> <p>It is in the SPARQL spirit to specify a graph and for pretty much any application, there are entirely sensible ways of keeping the data in graphs and specifying which ones are concerned by queries. This is why Virtuoso is set up for this by default.</p> <p>On the other hand, for the open web scenario, dealing with an unknown large number of graphs, enumerating graphs is not possible and questions like which graph of which source asserts x become relevant. We have two distinct use cases which warrant different setups of the database, simple as that.</p> <p>The latter use case is not really within the SPARQL spec, so implementations may or may not support this. For example <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x11ed7028">Oracle</a> or Vertica would not do this well since they partition data according to graph or predicate, respectively. On the other hand, stores that work with one quad table, which is most of the ones out there, should do it maybe with some configuring, as shown above.</p> <p>Frameworks like Jena are not to my <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x1a49ded0">knowledge</a> geared towards having a wildcard for graph, although I would suppose this can be arranged by adding some "super-graph" object, a graph of all graphs. I don't think this is directly supported and besides most apps would not need it.</p> <p>Once the indices are right, there is no difference between specifying a graph and not specifying a graph with the queries considered. With more complex queries, specifying a graph or set of graphs does allow some optimizations that cannot be done with no graph specified. For example, bitmap intersections are possible only when all leading key parts are given.</p> <h3>Conclusions</h3> <p>The best warm cache time is with v5; the five queries run under 600 ms after the first go. This is noted to show that all-in-memory with a single thread of execution is hard to beat.</p> <p>Cluster v6 performs the same queries in 623 ms. What is gained in parallelism is lost in latency if all operations complete in microseconds. On the other hand, Cluster v6 leaves v5 in the dust in any situation that has less than 100% hit rate. This is due to actual benefit from parallelism if operations take longer than a few microseconds, such as in the case of disk reads. Cluster v6 has substantially better data layout on disk, as well as fewer pages to load for the same content.</p> <p>This makes it possible to run the queries without the pogs index on Cluster v6 even when v5 takes prohibitively long.</p> <p>The morale of the story is to have a lot of RAM and space-efficient data representation.</p> <p>The DBpedia benchmark does not specify any random access pattern that would give a measure of sustained throughput under load, so we are left with the extremes of cold and warm cache of which neither is quite realistic.</p> <p>Chris Bizer and I have talked on and off about benchmarks and I have made suggestions that we will see incorporated into the Berlin SPARQL benchmark, which will, I believe, be much more informative.</p> <h3>Appendix: Query Text</h3> <p>For reference, the query texts specifying the graph are below. To run without specifying the graph, just drop the <code>FROM <<a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1905bfd0">http</a>://dbpedia.org></code> from each query. The returned row counts are indicated below each query's text.</p> <blockquote> <code><pre> sparql SELECT ?p ?o FROM <http://dbpedia.org> WHERE { <http://dbpedia.org/resource/Metropolitan_Museum_of_Art> ?p ?o }; -- 1337 rows sparql PREFIX p: <http://dbpedia.org/property/> SELECT ?film1 ?actor1 ?film2 ?actor2 FROM <http://dbpedia.org> WHERE { ?film1 p:starring <http://dbpedia.org/resource/Kevin_Bacon> . ?film1 p:starring ?actor1 . ?film2 p:starring ?actor1 . ?film2 p:starring ?actor2 . }; -- 23910 rows sparql PREFIX p: <http://dbpedia.org/property/> SELECT ?artist ?artwork ?museum ?director FROM <http://dbpedia.org> WHERE { ?artwork p:artist ?artist . ?artwork p:museum ?museum . ?museum p:director ?director }; -- 303 rows sparql PREFIX geo: <http://www.w3.org/2003/01/geo/wgs84_pos#> PREFIX foaf: <http://xmlns.com/foaf/0.1/> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> SELECT ?s ?homepage FROM <http://dbpedia.org> WHERE { <http://dbpedia.org/resource/Berlin> geo:lat ?berlinLat . <http://dbpedia.org/resource/Berlin> geo:long ?berlinLong . ?s geo:lat ?lat . ?s geo:long ?long . ?s foaf:homepage ?homepage . FILTER ( ?lat <= ?berlinLat + 0.03190235436 && ?long >= ?berlinLong - 0.08679199218 && ?lat >= ?berlinLat - 0.03190235436 && ?long <= ?berlinLong + 0.08679199218) }; -- 56 rows sparql PREFIX geo: <http://www.w3.org/2003/01/geo/wgs84_pos#> PREFIX foaf: <http://xmlns.com/foaf/0.1/> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> PREFIX p: <http://dbpedia.org/property/> SELECT ?s ?a ?homepage FROM <http://dbpedia.org> WHERE { <http://dbpedia.org/resource/New_York_City> geo:lat ?nyLat . <http://dbpedia.org/resource/New_York_City> geo:long ?nyLong . ?s geo:lat ?lat . ?s geo:long ?long . ?s p:architect ?a . ?a foaf:homepage ?homepage . FILTER ( ?lat <= ?nyLat + 0.3190235436 && ?long >= ?nyLong - 0.8679199218 && ?lat >= ?nyLat - 0.3190235436 && ?long <= ?nyLong + 0.8679199218) }; -- 13 rows </pre> </code> </blockquote>
2008-05-12T11:24:36-04:00
Comments about recent Semantic Gang Podcast
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-05-02#1357
2008-05-02T21:44:31Z
<p>After listening to the <a href="http://semanticgang.talis.com/2008/05/02/april-2008-the-semantic-web-gang-discuss-a-wikipedia-for-data/" id="link-id1089e218">latest Semantic Web Gang podcast</a>, I found myself agreeing with some of the points made by <a href="http://www.linkedin.com/in/iskold" id="link-id10b91e58">Alex Iskold</a>, specifically: </p> <ul>-- <a href="http://dbpedia.org/resource/Linked_Data" id="link-id106e24e0">Linked Data</a> does not implicitly imply making all your <a href="http://dbpedia.org/resource/Data" id="link-id17ab3d48">data</a> public</ul> <ul>-- <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11fdcef0">Linked Data</a> principles benefit <a href="http://dbpedia.org/resource/Intranet" id="link-id109756e8">Intranet</a> and <a href="http://dbpedia.org/resource/Extranet" id="link-id1099cfd8">Extranet</a> style <a href="http://dbpedia.org/resource/Data" id="link-id10cd25b0">data</a> integration (trumps alternative <a href="http://dbpedia.org/resource/federated_database_system" id="link-id14f29940">distributed database</a> integration approaches any day)</ul> <ul>-- Business exploitation of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0xca51940">Linked Data</a> on the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> will certainly be driven by the correlation of opportunity costs (which is more than likely what Alex meant by "use cases") associated with the lack of URIs originating from the domain of a given business (Tom Heath: also effectively alluded to this via his <a href="http://dbpedia.org/resource/BBC" id="link-id16f33348">BBC</a> and <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id10decf38">URI</a> land grab anecdotes; same applies Georgi's examples)</ul> <ul>-- History is a great tutor, answers to many of today's problems always lie somewhere in plain sight of the past.</ul> <p>Of course, I also believe that <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> serves Web <a href="http://dbpedia.org/resource/Data" id="link-id0x1afebd58">Data</a> Integration across the <a href="http://dbpedia.org/resource/Internet" id="link-id10aa5668">Internet</a> very well too, and the fact that it will be beneficial to businesses in a big way. No individual or organization is an island, I think the <a href="http://dbpedia.org/resource/Internet" id="link-id0xb25fbd0">Internet</a> and Web have done a good job of demonstrating that thus far :-) We're all <a href="http://dbpedia.org/resource/Data">data</a> nodes in a <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id5d8a3a8">Giant Global Graph</a>.</p> <p> <a href="http://myopenlink.net/dataspace/person/danieljohnlewis#this" id="link-id17cac8a0">Daniel lewis</a> did shed light on the read-write aspects of the Linked Data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10be8590">Web</a>, which is actually very close to the callout for a Wikipedia for Data. <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id10a810c0">TimBL</a> has been working on this via <a href="http://dig.csail.mit.edu/2005/ajar/release/tabulator/0.8/tab.html" id="link-id184b7108">Tabulator</a> (see <a href="http://dig.csail.mit.edu/2007/tab/tutorial/editing.mov" id="link-id1416f1e8">Tabulator Editing Screencast</a>), <a href="http://bnode.org/about" id="link-id17e33750">Bengamin Nowack</a> also added <a href="http://arc.semsol.org/download/plugins/data_wiki" id="link-id1688cc40">similar functionality to ARC</a>, and of course we support the same <a href="http://dbpedia.org/resource/SPARQL" id="link-id10bff7c8">SPARQL</a> UPDATE into an <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id168ace08">RDF</a> <a href="http://dbpedia.org/resource/Information" id="link-id10641878">information</a> resource via the <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xddb5240">RDF</a> Sink feature of our WebDAV and <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/OdsBriefcase" id="link-id0x11199310">ODS</a>-Briefcase implementations.</p>
2008-05-05T20:06:42.000004-04:00
In Perpetual Pursuit of Context
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-05-02#1356
2008-05-02T19:18:33Z
<p>I've always been of the opinion that concise value proposition articulation shouldn't be the achilles of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id158efe90">Semantic Web</a>. As the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id13a2db40">Linked Data</a> wave climbs up the "value Appreciation and Comprehension chain", it's getting clearer by the second that "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id109316f0">Context</a>" is a point of confluence for <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id108daa60">Semantic Web</a> Technologies and easy to comprehend value, from the perspectives of those outside the core community.</p> <p>In today's primarily Document centric <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a>, the pursuit of <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id14edadd0">Context</a> is akin to pursuing a mirage in a desert of user generated content. The quest is labor intensive, and you ultimaely end up without water at the end of the pursuit :-)</p> <p>Listening to the <a href="http://blogs.talis.com/nodalities/2008/05/christine-connors-talks-about-semantic-technologies-at-dow-jones.php" id="link-id12d5e1c0">Christine Connor's podcast interview with Talis</a> simply reinforces my strong belief that "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id0x1ec69518">Context</a>, Context, Context" is the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0xa279438">Semantic Web</a>'s equivalent of Real Estate's "Location, Location, Location" (ignore the <a href="http://dbpedia.org/resource/Subprime_lending" id="link-id140b8098">subprime</a> loans mess for now). The critical thing to note is that you cannot unravel "Context" from existing Web content without incorporating powerful disambiguation technology into an "<a href="http://dbpedia.org/resource/Entity" id="link-id15a2f380">Entity</a> Extraction" process. Of course, you cannot even consider seriously pursing any <a href="http://dbpedia.org/resource/Entity" id="link-id10868a18">entity</a> extraction and disambiguation endeavor without a lookup backbone that exposes "<a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id168dc230">Named Entities</a>" and their relationships to "<a href="http://dbpedia.org/resource/Topic" id="link-id17cb1950">Subject matter Concepts</a>" (BTW - this is what <a href="http://umbel.org/about/" id="link-id14f406a0">UMBEL</a> is all about). Thus, when looking at the broad subject of the <a href="http://dbpedia.org/resource/Semantic_Web">Semantic Web</a>, we can also look at "Context" as the vital point of confluence for the <a href="http://dbpedia.org/resource/Data" id="link-id12d67e38">Data</a> oriented (<a href="http://dbpedia.org/resource/Linked_Data" id="link-id14f8daf0">Linked Data</a>) and the "Linguistic Meaning" oriented perspectives.</p> <p>I am even inclined to state publicly that "Context" may ultimately be the foundation for <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=dimension%20web%204.0%20&type=text&output=html" id="link-id17cb0708">4th "Web Interaction Dimension"</a> where practical use of <a href="http://dbpedia.org/resource/Artificial_intelligence" id="link-id10b15088">AI</a> leverages a <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1ebf9310">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10b27018">Web</a> substrate en route to exposing new kinds of value :-)</p> <p>"Context" may also be the focal point of concise value proposition articulation to <a href="http://dbpedia.org/resource/Venture_Capital" id="link-id10837578">VCs</a> as in: "My solution offers the ability to discover and exploit "Context" iteratively, at the rate of $X.XX per iteration, across a variety of market segments :-)</p>
2008-05-03T15:07:32-04:00
XTech Talks covering Linked Data
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-05-02#1355
2008-05-02T14:53:08Z
<p>Courtesy a post by <a href="http://community.linkeddata.org/dataspace/person/bizer#this" id="link-id10868548">Chris Bizer</a> to the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id15739748">LOD</a> community <a href="http://lists.w3.org/Archives/Public/public-lod/" id="link-id10fae0f8">mailing list</a>, here is a list of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id140a0880">Linked Data</a> oriented talks at the upcoming <a href="http://2008.xtech.org" id="link-id12801f00">XTech</a> 2008 event (also see the <a href="http://2008.xtech.org/public/schedule/grid" id="link-id10f65940">XTech 2008 Schedule</a> which is <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1258a4c8">Linked Data</a> friendly). Of course, I am posting this to my <a href="http://dbpedia.org/resource/Blog" id="link-id140a29c0">Blog</a> <a href="http://dbpedia.org/resource/Data" id="link-id12d5a640">Data</a> <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id10979b80">Space</a> with the sole purpose of adding <a href="http://dbpedia.org/resource/Data" id="link-id176be078">data</a> to the rapidly growing <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id1099aec8">Giant Global Graph</a> of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10d72d88">Linked Data</a>, basically adding to my collection of live <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11014000">Linked Data</a> utility demos :-)</p> <p>Here is the list:</p> <ol> <li> <a href="http://2008.xtech.org/public/schedule/detail/561" id="link-id17df4d78">Linked Data Deployment</a> (<a href="http://myopenlink.net/dataspace/person/danieljohnlewis#this" id="link-id17c47d28">Daniel Lewis</a>, <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id108fce00">OpenLink Software</a>)</li> <li> <a href="http://2008.xtech.org/public/schedule/detail/524" id="link-id1068c0e0">The Programmes Ontology</a> (Tom Scott, <a href="http://dbpedia.org/resource/BBC" id="link-id1566da50">BBC</a> and all) </li> <li> <a href="http://2008.xtech.org/public/schedule/detail/528" id="link-id1072be40">SemWebbing the London Gazette</a> (Jeni Tennison, The Stationery Office) </li> <li> <a href="http://2008.xtech.org/public/schedule/detail/583" id="link-id1099e4e0">Searching, publishing and remixing a Web of Semantic Data</a> (<a href="http://community.linkeddata.org/dataspace/person/cygri#this" id="link-id17e25b78">Richard Cyganiak</a>, DERI Galway) </li> <li> <a href="http://2008.xtech.org/public/schedule/detail/477" id="link-idf9764c8">Building a Semantic Web Search Engine: Challenges and Solutions</a> (Aidan Hogan, DERI Galway) </li> <li>'<a href="http://2008.xtech.org/public/schedule/detail/550" id="link-id140a3c50">That's not what you said yesterday!</a>' - evolving your <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> API (<a href="http://iandavis.com/id/me" id="link-id14f8d498">Ian Davis</a>, Talis) </li> <li> <a href="http://2008.xtech.org/public/schedule/detail/527" id="link-id10c5a9c8">Representing, indexing and mining scientific data using XML and RDF: Golem and CrystalEye</a> (<a href="http://wwmm.ch.cam.ac.uk/blogs/walkingshaw/" id="link-id108c5e28">Andrew Walkingshaw</a>, <a href="http://dbpedia.org/resource/University_of_Cambridge" id="link-id10891560">University of Cambridge</a>)</li> </ol> <p>For the time challenged (i.e. those unable to view this post using it's permalink / <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id10db39f0">URI</a> as a <a href="http://dbpedia.org/resource/Data" id="link-id10f29bb8">data</a> source via the <a href="http://demo.openlinksw.com/rdfbrowser" id="link-id10f72778">OpenLink RDF Browser</a>, <a href="http://zitgist.com/about/" id="link-id107b73b0">Zitgist</a> <a href="http://dataviewer.zitgist.com" id="link-id1686d528">Data Viewer</a>, <a href="http://www4.wiwiss.fu-berlin.de/rdf_browser" id="link-id110479e8">DISCO Hyperdata Browser</a>, or <a href="http://dig.csail.mit.edu/2005/ajar/release/tabulator/0.8/tab.html" id="link-id140ba0e8">Tabulator</a>), the benefits of this post are as follows:</p> <ul> <li>automatic <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id172d2fc8">URI</a> generation for all linked items in this post</li> <li>automatic propagation of tags to <a href="http://del.icio.us" id="link-id10547380">del</a>.<a href="http://del.icio.us" id="link-id1093cc10">icio</a>.<a href="http://del.icio.us" id="link-id168ce3a0">us</a>, <a href="http://www.technorati.com" id="link-id17aa8af0">Technorati</a>, and <a href="http://www.pingthesemanticweb.com/about/" id="link-id10868ad8">PingTheSemanticWeb</a> </li> <li>automatic association of formal meanings to my Tags using the <a href="http://moat-project.org/ontology" id="link-id10c98608">MOAT Ontology</a> </li> <li>automatic collation and generation of statistical <a href="http://dbpedia.org/resource/Data" id="link-id10a4d1d8">data</a> about my tags using the SCOT Ontology (*missing link is a callout to SCOT <a href="http://dbpedia.org/resource/Tag" id="link-id168b7c10">Tag</a> Ontology folks to sort the project's home page <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id11fd4118">URL</a> at the very least*) </li> <li>explicit typing of my Tags as <a href="http://dbpedia.org/resource/SKOS" id="link-id10940eb8">SKOS</a> Concepts. </li> </ul> <p>Put differently, I cost-effectively contribute to the <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10a081a8">GGG</a> across all <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=web%20dimensions&type=text&output=html" id="link-id10597530">Web interaction dimensions</a> (1.0, 2.0, 3.0) :-)</p>
2008-05-05T17:07:17-04:00
SPARQL at WWW 2008
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-04-30#1354
2008-04-30T16:28:10Z
<div> <div style="display:none;">SPARQL at WWW 2008</div> <p>Andy Seaborne and Eric Prud'hommeaux, editors of the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x183b13a8">SPARQL</a> recommendation, convened a SPARQL birds of a feather session at <a href="http://www2008.org/" id="link-id0xd31c2d0">WWW 2008</a>. The administrative outcome was that implementors could now experiment with extensions, hopefully keeping each other current about their efforts and that towards the end of 2008, a new W3C working group might begin formalizing the experiences into a new SPARQL spec.</p> <p>The session drew a good crowd, including many users and developers. The wishes were largely as expected, with a few new ones added. Many of the wishes already had diverse implementations, however most often without interop. I will below give some comments on the main issues discussed.</p> </div> <li> <p> <b>SPARQL Update</b> - This is likely the most universally agreed upon extension. Implementations exist, largely along the lines of Andy Seaborne's SPARUL spec, which is also likely material for a W3C member submission. The issue is without much controversy; transactions fall outside the scope, which is reasonable enough. With triple stores, we can define things as combinations of inserts and deletes, and isolation we just leave aside. If anything, operating on a transactional platform such as <a href="http://virtuoso.openlinksw.com" id="link-id0x1d442cd8">Virtuoso</a>, one wishes to disable transactions for any operations such as bulk loads and long-running inserts and deletes. Transactionality has pretty much no overhead for a few hundred rows, but for a few hundred million rows the cost of locking and rollback is prohibitive. With Virtuoso, we have a row auto-commit mode which we recommend for use with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xec62c58">RDF</a>: It commits by itself now and then, optionally keeping a roll forward log, and is transactional enough not to leave half triples around, i.e., inserted in one index but not another.</p> <p>As far as we are concerned, updating physical triples along the SPARUL lines is pretty much a done deal.</p> <p>The matter of updating relational <a href="http://dbpedia.org/resource/Data" id="link-id0x19f995c8">data</a> mapped to RDF is a whole other kettle of fish. On this, I should say that RDF has no special virtues for expressing transactions but rather has a special genius for integration. Updating is best left to web service interfaces that use <a href="http://dbpedia.org/resource/SQL" id="link-id0x125b2b98">SQL</a> on the inside. Anyway, updating union views, which most mappings will be, is complicated. Besides, for transactions, one usually knows exactly what one wishes to update.</p> </li> <li> <p> <b>Full Text</b> - Many people expressed a desire for full text access. Here we run into a deplorable confusion with regexps. The closest SPARQL has to full text in its native form is regexps, but these are not really mappable to full text except in rare special cases and I would despair of explaining to an end user what exactly these cases are. So, in principle, some regexps are equivalent to full text but in practice I find it much preferable to keep these entirely separate.</p> <p>It was noted that what the users want is a text box for search words. This is a front end to the CONTAINS predicate of most SQL implementations. Ours is MS SQL Server compatible and has a SPARQL version called <code>bif:contains</code>. One must still declare which triples one wants indexed for full text, though. This admin overhead seems inevitable, as text indexing is a large overhead and not needed by all applications.</p> <p>Also, text hits are not boolean; usually they come with a hit score. Thus, a SPARQL extension for this could look like </p> <blockquote> <code>select * where { ?thing has_description ?d . ?d ftcontains "gizmo" ftand "widget" score ?score . }</code> </blockquote> <p>This would return all the subjects, descriptions, and scores, from subjects with a has_description property containing widget and gizmo. Extending the basic pattern is better than having the match in a filter, since the match binds a variable.</p> <p>The <a href="http://dbpedia.org/resource/XQuery" id="link-id0xfec6788">XQuery</a>/<a href="http://dbpedia.org/resource/XPath" id="link-id0x1a789e38">XPath</a> groups have recently come up with a full-text spec, so I used their style of syntax above. We already have a full-text extension, as do some others. but for standardization, it is probably most appropriate to take the XQuery work as a basis. The XQuery full-text spec is quite complex, but I would expect most uses to get by with a small subset, and the structure seems better thought out, at first glance, than the more ad-hoc implementations in diverse SQLs.</p> <p>Again, declaring any text index to support the search, as well as its timeliness or transactionality, are best left to implementations.</p> </li> <li> <p> <b>Federation</b> - This is a tricky matter. <a href="http://jena.sourceforge.net/ARQ/" id="link-id0xba487f0">ARQ</a> has a SPARQL extension for sending a nested set of triple patterns to a specific end-point. The <a href="http://darq.sourceforge.net/" id="link-id0xf8a5ab0">DARQ</a> project has something more, including a selectivity model for SPARQL.</p> <p>With federated SQL, life is simpler since after the views are expanded, we have a query where each table is at a known server and has more or less known statistics. Generally, execution plans where as much work as possible is pushed to the remote servers are preferred, and modeling the latencies is not overly hard. With SPARQL, each triple pattern could in principle come from any of the federated servers. Associating a specific end-point to a fragment of the query just passes the problem to the user. It is my guess that this is the best we can do without getting very elaborate, and possibly buggy, end-point content descriptions for routing federated queries.</p> <p>Having said this, there remains the problem of join order. I suggested that we enhance the protocol by allowing asking an end-point for the query cost for a given SPARQL query. Since they all must have a cost model for optimization, this should not be an impossible request. A time cost and estimated cardinality would be enough. Making statistics available <i>à la</i> DARQ was also discussed. Being able to declare cardinalities expected of a remote end-point is probably necessary anyway, since not all will implement the cost model interface. For standardization, agreeing on what is a proper description of content and cardinality and how fine grained this must be will be so difficult that I would not wait for it. A cost model interface would nicely hide this within the end-point itself.</p> <p>With Virtuoso, we do not have a federated SPARQL scheme but we could have the ARQ-like service construct. We'd use our own cost model with explicit declarations of cardinalities of the remote data for guessing a join order. Still, this is a bit of work. We'll see.</p> <p>For practicality, the service construct coupled with join order hints is the best short term bet. Making this pretty enough for standardization is not self-evident, as it requires end-point description and/or cost model hooks for things to stay declarative.</p> </li> <li> <p> <b>End-point description</b> - This question has been around for a while; I have <a href="http://www.openlinksw.com/weblog/oerling/?id=1085" id="link-id101d3440">blogged about it earlier</a>, but we are not really at a point where there would be even rough consensus about an end-point ontology. We should probably do something on our own to demonstrate some application of this, as we host lots of <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x162d0de8">linked open data</a> sets.</p> </li> <li> <p> <b>SQL equivalence</b> - There were many requests for aggregation, some for subqueries and nesting, expressions in select, negation, existence and so on. I would call these all SQL equivalence. One use case was taking all the teams in the database and for all with over 5 members, add the big_team class and a property for member count.</p> <p>With Virtuoso, we could write this as -- </p> <blockquote> <code>construct { ?team a big_team . ?team member_count ?ct } from ... where {?team a team . { select ?team2 count (*) as ?ct where { ?m member_of ?team2 } . filter (?team = ?team2 and ? ct > 5) }}</code> </blockquote> <p>We have pretty much all the SQL equivalence features, as we have been working for some time at translating the <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x11c870c8">TPC-H</a> workload into SPARQL.</p> <p>The usefulness of these things is uncontested but standardization could be hard as there are subtle questions about variable scope and the like.</p> </li> <li> <p> <b>Inference</b> - The SPARQL spec does not deal with transitivity or such matters because it is assumed that these are handled by an underlying inference layer. This is however most often not so. There was interest in more fine grained control of inference, for example declaring that just one property in a query would be transitive or that subclasses should be taken into account in only one triple pattern. As far as I am concerned, this is very reasonable, and we even offer extensions for this sort of thing in Virtuoso's SPARQL. This however only makes sense if the inference is done at query time and pattern by pattern. For instance, if forward chaining is used, this no longer makes sense. Specifying that some forward chaining ought to be done at query time is impractical, as the operation can be very large and time consuming and it is the DBA's task to determine what should be stored and for how long, how changes should be propagated, and so on. All these are application dependent and standardizing will be difficult.</p> <p>Support for RDF features like lists and bags would all fall into the functions an underlying inference layer should perform. These things are of special interest when querying <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x156f3830">OWL</a> models, for example.</p> </li> <li> <p> <b>Path expressions</b> - Path expressions were requested by a few people. We have implemented some, as in </p> <blockquote> <code>?product+?has_supplier+>s_name = "Gizmos, Inc.".</code> </blockquote> This means that one supplier of product has name "Gizmo, Inc.". This is a nice shorthand but we run into problems if we start supporting repetitive steps, optional steps, and the like.</li> <p>In conclusion, update, full text, and basic counting and grouping would seem straightforward at this point. Nesting queries, value subqueries, views, and the like should not be too hard if an agreement is reached on scope rules. Inference and federation will probably need more experimentation but a lot can be had already with very simple fine grained control of backward chaining, if such applies, or with explicit end-point references and explicit join order. These are practical but not pretty enough for committee consensus, would be my guess. Anyway, it will be a few months before anything formal will happen.</p>
2008-08-28T11:26:06.000004-04:00
SPARQL at WWW 2008
http://www.openlinksw.com:443/weblog/oerling/?date=2008-04-30#1353
2008-04-30T15:59:15Z
<p>Andy Seaborne and Eric Prud'hommeaux, editors of the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xddcd418">SPARQL</a> recommendation, convened a SPARQL birds of a feather session at <a href="http://www2008.org/" id="link-id0xf6c0510">WWW 2008</a>. The administrative outcome was that implementors could now experiment with extensions, hopefully keeping each other current about their efforts and that towards the end of 2008, a new W3C working group might begin formalizing the experiences into a new SPARQL spec.</p> <p>The session drew a good crowd, including many users and developers. The wishes were largely as expected, with a few new ones added. Many of the wishes already had diverse implementations, however most often without interop. I will below give some comments on the main issues discussed.</p> <li> <p> <b>SPARQL Update</b> - This is likely the most universally agreed upon extension. Implementations exist, largely along the lines of Andy Seaborne's SPARUL spec, which is also likely material for a W3C member submission. The issue is without much controversy; transactions fall outside the scope, which is reasonable enough. With triple stores, we can define things as combinations of inserts and deletes, and isolation we just leave aside. If anything, operating on a transactional platform such as <a href="http://virtuoso.openlinksw.com" id="link-id0xf9e8d18">Virtuoso</a>, one wishes to disable transactions for any operations such as bulk loads and long-running inserts and deletes. Transactionality has pretty much no overhead for a few hundred rows, but for a few hundred million rows the cost of locking and rollback is prohibitive. With Virtuoso, we have a row auto-commit mode which we recommend for use with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xe098028">RDF</a>: It commits by itself now and then, optionally keeping a roll forward log, and is transactional enough not to leave half triples around, i.e., inserted in one index but not another.</p> <p>As far as we are concerned, updating physical triples along the SPARUL lines is pretty much a done deal.</p> <p>The matter of updating relational <a href="http://dbpedia.org/resource/Data" id="link-id0xf55ec50">data</a> mapped to RDF is a whole other kettle of fish. On this, I should say that RDF has no special virtues for expressing transactions but rather has a special genius for integration. Updating is best left to web service interfaces that use <a href="http://dbpedia.org/resource/SQL" id="link-id0xff00f38">SQL</a> on the inside. Anyway, updating union views, which most mappings will be, is complicated. Besides, for transactions, one usually knows exactly what one wishes to update.</p> </li> <li> <p> <b>Full Text</b> - Many people expressed a desire for full text access. Here we run into a deplorable confusion with regexps. The closest SPARQL has to full text in its native form is regexps, but these are not really mappable to full text except in rare special cases and I would despair of explaining to an end user what exactly these cases are. So, in principle, some regexps are equivalent to full text but in practice I find it much preferable to keep these entirely separate.</p> <p>It was noted that what the users want is a text box for search words. This is a front end to the CONTAINS predicate of most SQL implementations. Ours is MS SQL Server compatible and has a SPARQL version called <code>bif:contains</code>. One must still declare which triples one wants indexed for full text, though. This admin overhead seems inevitable, as text indexing is a large overhead and not needed by all applications.</p> <p>Also, text hits are not boolean; usually they come with a hit score. Thus, a SPARQL extension for this could look like </p> <blockquote> <code>select * where { ?thing has_description ?d . ?d ftcontains "gizmo" ftand "widget" score ?score . }</code> </blockquote> <p>This would return all the subjects, descriptions, and scores, from subjects with a has_description property containing widget and gizmo. Extending the basic pattern is better than having the match in a filter, since the match binds a variable.</p> <p>The <a href="http://dbpedia.org/resource/XQuery" id="link-id0xee42f50">XQuery</a>/<a href="http://dbpedia.org/resource/XPath" id="link-id0xd169630">XPath</a> groups have recently come up with a full-text spec, so I used their style of syntax above. We already have a full-text extension, as do some others. but for standardization, it is probably most appropriate to take the XQuery work as a basis. The XQuery full-text spec is quite complex, but I would expect most uses to get by with a small subset, and the structure seems better thought out, at first glance, than the more ad-hoc implementations in diverse SQLs.</p> <p>Again, declaring any text index to support the search, as well as its timeliness or transactionality, are best left to implementations.</p> </li> <li> <p> <b>Federation</b> - This is a tricky matter. <a href="http://jena.sourceforge.net/ARQ/" id="link-id0xcbe1e18">ARQ</a> has a SPARQL extension for sending a nested set of triple patterns to a specific end-point. The <a href="http://darq.sourceforge.net/" id="link-id0xe7c0088">DARQ</a> project has something more, including a selectivity model for SPARQL.</p> <p>With federated SQL, life is simpler since after the views are expanded, we have a query where each table is at a known server and has more or less known statistics. Generally, execution plans where as much work as possible is pushed to the remote servers are preferred, and modeling the latencies is not overly hard. With SPARQL, each triple pattern could in principle come from any of the federated servers. Associating a specific end-point to a fragment of the query just passes the problem to the user. It is my guess that this is the best we can do without getting very elaborate, and possibly buggy, end-point content descriptions for routing federated queries.</p> <p>Having said this, there remains the problem of join order. I suggested that we enhance the protocol by allowing asking an end-point for the query cost for a given SPARQL query. Since they all must have a cost model for optimization, this should not be an impossible request. A time cost and estimated cardinality would be enough. Making statistics available <i>à la</i> DARQ was also discussed. Being able to declare cardinalities expected of a remote end-point is probably necessary anyway, since not all will implement the cost model interface. For standardization, agreeing on what is a proper description of content and cardinality and how fine grained this must be will be so difficult that I would not wait for it. A cost model interface would nicely hide this within the end-point itself.</p> <p>With Virtuoso, we do not have a federated SPARQL scheme but we could have the ARQ-like service construct. We'd use our own cost model with explicit declarations of cardinalities of the remote data for guessing a join order. Still, this is a bit of work. We'll see.</p> <p>For practicality, the service construct coupled with join order hints is the best short term bet. Making this pretty enough for standardization is not self-evident, as it requires end-point description and/or cost model hooks for things to stay declarative.</p> </li> <li> <p> <b>End-point description</b> - This question has been around for a while; I have <a href="http://www.openlinksw.com/weblog/oerling/?id=1085" id="link-id101d3440">blogged about it earlier</a>, but we are not really at a point where there would be even rough consensus about an end-point ontology. We should probably do something on our own to demonstrate some application of this, as we host lots of <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0xfaec780">linked open data</a> sets.</p> </li> <li> <p> <b>SQL equivalence</b> - There were many requests for aggregation, some for subqueries and nesting, expressions in select, negation, existence and so on. I would call these all SQL equivalence. One use case was taking all the teams in the database and for all with over 5 members, add the big_team class and a property for member count.</p> <p>With Virtuoso, we could write this as -- </p> <blockquote> <code>construct { ?team a big_team . ?team member_count ?ct } from ... where {?team a team . { select ?team2 count (*) as ?ct where { ?m member_of ?team2 } . filter (?team = ?team2 and ? ct > 5) }}</code> </blockquote> <p>We have pretty much all the SQL equivalence features, as we have been working for some time at translating the <a href="http://dbpedia.org/resource/TPC-H" id="link-id0xe2392e8">TPC-H</a> workload into SPARQL.</p> <p>The usefulness of these things is uncontested but standardization could be hard as there are subtle questions about variable scope and the like.</p> </li> <li> <p> <b>Inference</b> - The SPARQL spec does not deal with transitivity or such matters because it is assumed that these are handled by an underlying inference layer. This is however most often not so. There was interest in more fine grained control of inference, for example declaring that just one property in a query would be transitive or that subclasses should be taken into account in only one triple pattern. As far as I am concerned, this is very reasonable, and we even offer extensions for this sort of thing in Virtuoso's SPARQL. This however only makes sense if the inference is done at query time and pattern by pattern. For instance, if forward chaining is used, this no longer makes sense. Specifying that some forward chaining ought to be done at query time is impractical, as the operation can be very large and time consuming and it is the DBA's task to determine what should be stored and for how long, how changes should be propagated, and so on. All these are application dependent and standardizing will be difficult.</p> <p>Support for RDF features like lists and bags would all fall into the functions an underlying inference layer should perform. These things are of special interest when querying <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0xfd87f78">OWL</a> models, for example.</p> </li> <li> <p> <b>Path expressions</b> - Path expressions were requested by a few people. We have implemented some, as in </p> <blockquote> <code>?product+?has_supplier+>s_name = "Gizmos, Inc.".</code> </blockquote> This means that one supplier of product has name "Gizmo, Inc.". This is a nice shorthand but we run into problems if we start supporting repetitive steps, optional steps, and the like.</li> <p>In conclusion, update, full text, and basic counting and grouping would seem straightforward at this point. Nesting queries, value subqueries, views, and the like should not be too hard if an agreement is reached on scope rules. Inference and federation will probably need more experimentation but a lot can be had already with very simple fine grained control of backward chaining, if such applies, or with explicit end-point references and explicit join order. These are practical but not pretty enough for committee consensus, would be my guess. Anyway, it will be a few months before anything formal will happen.</p>
2008-08-28T11:26:00-04:00
Clearing Up RDF misrepresentation once again!
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-04-30#1352
2008-04-30T15:51:17Z
<p> <a href="http://myopenlink.net/dataspace/person/danieljohnlewis#this" id="link-id12d57690">Daniel Lewis</a> has penned a post titled: <a href="http://vanirsystems.com/danielsblog/2008/04/30/clearing-up-some-misconceptions-again/" id="link-id10c99f18">Clearing up some misconceptions..again</a>, in response to <a href="http://elgg.org/bwerdmuller/foaf#elgg2" id="link-id14fe1bc8">Ben Werdmuller</a>'s post titled: <a href="http://blogs.zdnet.com/social/?p=477" id="link-id141cee58">Introducing the Open Data Definition</a>. </p> <p>The great thing about the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id105991a8">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10a6ec78">Web</a> is that it's much easier to discovery and respond to these points of view before the ink dries :-) Ben certainly needs to take a look at the <a href="http://www.w3.org/RDF/FAQ" id="link-id10f78958">Semantic Web FAQ</a> pre or post assimilation of Daniel's response.</p>
2008-04-30T12:07:58.000001-04:00
Linked Data enters state of Evoluation
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-04-29#1351
2008-04-29T19:56:14Z
<p>During a brief chat with <a href="http://community.linkeddata.org/dataspace/person/mhausenblas#this" id="link-idfeb0100">Michael Hausenblas</a> about a new <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1049feb0">Linked Data</a> project he is championing called: <a href="http://esw.w3.org/topic/SweoIG/TaskForces/CommunityProjects/LinkingOpenData/LForum" id="link-id16a857d8">LForum</a>, I made a freudian slip, in the form of the typo: <strong>Evoluation</strong>, which at the time was supposed to have been: <strong>Evolution</strong>. Anyway, we had a chuckle and realized we were on to something, so I proceeded to formalize the definition: </p> <blockquote> <cite>Evoluation is evolution devoid of the randomness of mutation. A state of being in which it is possible to evaluate and choose evolutionary paths.</cite> </blockquote> <p> <strong>Evoluation</strong> actually describes where we are today in relation to the <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id105c1518">World Wide Web</a>; to the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id103f9d00">Linking Open Data community</a> (<a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id1048c210">LOD</a>), it's taking the path towards becoming a <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id104c3a20">Giant Global Graph</a> of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id104968e0">Linked Data</a>; to the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> 2.0 community, it's simply a collection of Web Services and associated APIs; and to many others, it remains an opaque collection of interlinked documents.</p> <p>The great thing about the Web is that it allows netizens to explore a plethora of paths without adversely affecting the paths of others. That said, controlling one's path may take mutation out of evolution, but we are still left with the requirement to adapt and eventually survive in a competitive environment. Thus, although we can evaluate and choose from the many paths the Web's evolution offers us, the path that delivers the most benefits ultimately dominates. :-) </p>
2008-04-29T16:25:47-04:00
Linked Data and Information Architecture
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-04-29#1350
2008-04-29T14:37:22Z
<div> <div style="display:none;">Linked Data and Information Architecture</div> <p>We had a workshop on <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1437ac70">Linked Open Data</a> (<a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1315f788">LOD</a>) last week in <a href="http://www2008.org/" id="link-id0x13737468">Beijing</a>. You can see the papers in <a href="http://events.linkeddata.org/ldow2008/#program" id="link-id10651ab8">the program</a>. The event was a success with plenty of good talks and animated conversation. I will not go into every paper here but will comment a little on the conversation and draw some technology requirements going forward.</p> <p>Tim Berners-Lee showed a read-write version of <a href="http://dig.csail.mit.edu/2005/ajar/release/tabulator/0.8/tab.html" id="link-id0x15633520">Tabulator</a>. This raises the question of updating on the <a href="http://dbpedia.org/resource/Data" id="link-id0x1350a178">Data</a> Web. The consensus was that one could assert what one wanted in one's own space but that others' spaces would be read-only. What spaces one considered relevant would be the user's or developer's business, as in the document web.</p> <p>It seems to me that a significant use case of LOD is an open-web situation where the user picks a broad read-only "data wallpaper" or backdrop of assertions, and then uses this combined with a much smaller, local, writable data set. This is certainly the case when editing data for publishing, as in Tim's demo. This will also be the case when developing mesh-ups combining multiple distinct data sets bound together by sets of SameAs assertions, for example. Questions like, "What is the minimum subset of n data sets needed for deriving the result?" will be common. This will also be the case in applications using proprietary data combined with open data.</p> <p>This means that databases will have to deal with queries that specify large lists of included graphs, all graphs in the store or all graphs with an exclusion list. All this is quite possible but again should be considered when architecting systems for an open <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0xa27bae8">linked data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x155c3f18">web</a>.</p> <p>"There is data but what can we really do with it? How far can we trust it, and what can we confidently decide based on it?"</p> <p>As an answer to this question, <a href="http://zitgist.com/about/" id="link-id0xd447580">Zitgist</a> has compiled the <a href="http://umbel.org/about/" id="link-id0x14735008">UMBEL</a> taxonomy using <a href="http://dbpedia.org/resource/SKOS" id="link-id0x15ab1c48">SKOS</a>. This draws on Wikipedia, Open CYC, Wordnet, and <a href="http://www.mpi-inf.mpg.de/~suchanek/downloads/yago/" id="link-id0x15d5aa88">YAGO</a>, hence the acronym WOWY. UMBEL is both a taxononmy and a set of instance data, containing a large set of <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id0x9fe45d98">named entities</a>, including persons, organizations, geopolitical entities, and so forth. By extracting references to this set of named entities from documents and correlating this to the taxonomy, one gets a good idea of what a document (or part thereof) is about.</p> <p>Kingsley presented this in the Zitgist demo. This is our answer to the criticism about <a href="http://dbpedia.org/resource/DBpedia" id="link-id0xa1920800">DBpedia</a> having errors in classification. DBpedia, as a bootstrap stage, is about giving names to all things. Subsequent efforts like UMBEL are about refining the relationships.</p> <p>"Should there be a global <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x12cd5290">URI</a> dictionary?"</p> <p>There was a talk by Paolo Bouquet about <a href="http://dbpedia.org/resource/Entity" id="link-id0x12d03400">Entity</a> Name System, a a sort of data DNS, with the purpose of associating some description and rough classification to URIs. This would allow discovering URIs for reuse. I'd say that this is good if it can cut down on the SameAs proliferation and if this can be widely distributed and replicated for resilience, <i>à la</i> DNS. On the other hand, it was pointed out that this was not quite in the LOD spirit, where parties would mint their own dereferenceable URIs, in their own domains. We'll see.</p> <p>"What to do when identity expires?"</p> <p>Giovanni of Sindice said that a document should be removed from search if it was no longer available. Kingsley pointed out that resilience of reference requires some way to recover data. The data web cannot be less resilient than the document web, and there is a point to having access to history. He recommended hooking up with the <a href="http://dbpedia.org/resource/Internet" id="link-id0x143e4130">Internet</a> Archive, since they make long term persistence their business. In this way, if an application depends on data, and the URIs on which it depends are no longer dereferenceable or or provide content from a new owner of the domain, those who need the old version can still get it and host it themselves.</p> <p>It is increasingly clear that OWL SameAs is both the blessing and bane of linked data. We can easily have tens of URIs for the same thing, especially with people. Still, these should be considered the same.</p> <p>Returning every synonym in a query answer hardly makes sense but accepting them as input seems almost necessary. This is what we do with <a href="http://virtuoso.openlinksw.com" id="link-id0x15a2a930">Virtuoso</a>'s SameAs support. Even so, this can easily double query times even when there are no synonyms.</p> <p>Be that as it may, SameAs is here to stay; just consider the mapping of DBpedia to Geonames, for example.</p> <p>Also, making aberrant SameAs statements can completely poison a data set and lead to absurd query results. Hence choosing which SameAs assertions from which source will be considered seems necessary. In an open web scenario, this leads inevitably to multi-graph queries that can be complex to write with regular <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x12bb8ce8">SPARQL</a>. By extension, it seems that a good query would also include the graphs actually used for deriving each result row. This is of course possible but has some implications on how databases should be organized.</p> <p>Yves Raymond gave a talk about deriving identity between Musicbrainz and Jamendo. I see the issue as a core question of linked data in general. The algorithm Yves presented started with attribute value similarities and then followed related entities. Artists would be the same if they had similar names and similar names of albums with similar song titles, for example. We can find the same basic question in any analysis, for example, looking at how news reporting differs between media, supposing there is adequate entity extraction.</p> <p>There is basic graph diffing in <a href="http://data.semanticweb.org/conference/iswc-aswc/2007/tracks/research/papers/533/html" id="link-id0x153c1fa8">RDFSync</a>, for example. But here we are expanding the context significantly. We will traverse references to some depth, allow similarity matches, SameAs, and so forth. Having presumed identity of two URIs, we can then look at the difference in their environment to produce a human readable summary. This could then be evaluated for purposes of analysis or of combining content.</p> <p>At first sight, these algorithms seem well parallelizable, as long as all threads have access to all data. For scaling, this means a probably message-bound distributed algorithm. This is something to look into for the next stage of linked data.</p> <p>Some inference is needed, but if everybody has their own choice of data sets to query, then everybody would also have their own entailed triples. This will make for an explosion of entailed graphs if forward chaining is used. Forward chaining is very nice because it keeps queries simple and easy to optimize. With Virtuoso, we still favor backward chaining since we expect a great diversity of graph combinations and near infinite volume in the open web scenario. With private repositories of slowly changing data put together for a special application, the situation is different.</p> <p>In conclusion, we have a real LOD movement with actual momentum and a good idea of what to do next. The next step is promoting this to the broader community, starting with <a href="http://www.linkeddataplanet.com/" id="link-id0x155d1d00">Linked Data Planet</a> in New York in June.</p> </div>
2015-08-21T14:32:10.859283-04:00
On Sem Web Search
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-04-29#1349
2008-04-29T14:37:21Z
<div> <div style="display:none;">On Sem Web Search</div> <p> <i>"I give the search keywords and you give me a <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1a603f18">SPARQL</a> end-point and a query that will get the <a href="http://dbpedia.org/resource/Data" id="link-id0x1bda5c00">data</a>."</i> </p> <p>Thus did one SPARQL user describe the task of a semantic/data web search engine.</p> <p>In <a href="http://www.openlinksw.com/weblog/oerling/?id=1336" id="link-idff98750">a previous post</a>, I suggested that if the data web were the size of the document web, we'd be looking at two orders of magnitude more search complexity. It just might be so.</p> <p>In the conversation, I pointed out that a search engine might have a copy of everything and even a capability to do SPARQL and full text on it all, yet still the users would be better off doing the queries against the SPARQL end-points of the data publishers. It is a bit like the fact that not all web browsing runs off Google's cache. With the data web, the point is even more pronounced, as serving a hit from Google's cache is a small operation but a complex query might be a very large one.</p> <p>Yet, the data web is about ad-hoc joining between data sets of different origins. Thus a search engine of the data web ought to be capable of joining also, even if large queries ought to be run against individual publishers' end-points or the user's own data warehouse.</p> <p>For ranking, the general consensus was that no single hit-ranking would be good for the data web. Thus word frequency-based hit-scores are OK for text hits but more is not obvious. I would think that some link analysis could apply but this will take some more experimentation.</p> <p>For search summaries, if we have splitting of data sets into small fragments <i>à la</i> <a href="http://sindice.com/" id="link-id0x1d2b7288">Sindice</a>, search summaries are pretty much the same as with just text search. If we store triples, then we can give text style summaries of text hits in literals and Fresnel lens views of the structured data around the literal. For showing a page of hits, the lenses must abbreviate heavily but this is still feasible. The engine would know about the most common ontologies and summarize instance data accordingly.</p> <p>Chris Bizer pointed out that trust and provenance are critical, especially if an answer is arrived at by joining multiple data sets. The trust of the conclusion is no greater than that of the weakest participating document. Different users will have different trusted sources.</p> <p>A mature data web search engine would combine a provenance/trust specification, a search condition consisting of SPARQL or full text or both, and a specification for hit rank. Again, most searches would use defaults, but these three components should in principle be orthogonally specifiable.</p> <p>Many places may host the same data set either for download or SPARQL access. The <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x1b2317d0">URI</a> of the data set is not its <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id0x1c55dd68">URL</a>. Different places may further host multiple data sets on one end-point. Thus the search engine ought to return all end-points where the set is to be found. The end-points themselves ought to be able to say what data sets they contain, under what graph IRIs. Since there is no consensus about end-point self description, this too would be left to the search engine. In practice, this could be accomplished by extending Sindice's semantic site map specification. A possible query would be to find an end-point containing a set of named data sets. If none were found, the search engine itself could run a query joining all the sets since it at least would hold them all.</p> <p>Since many places will host sets like Wordnet or Uniprot, indexing these once for each copy hardly makes sense. Thus a site should identify its data by the data set's URI and not the copy's URL.</p> <p>It came up in the discussion that search engines should share a ping format so that a single message format would be enough to notify any engine about data being updated. This is already partly the case with Sindice and <a href="http://www.pingthesemanticweb.com/" id="link-id0xa405ebd0">PTSW</a> (<a href="http://www.pingthesemanticweb.com/" id="link-id0x1c051a00">PingTheSemanticWeb</a>) sharing a ping format. </p> <p>Further, since it is no trouble to publish a copy of the 45G Uniprot file but a fair amount of work to index it, search engines should be smart about processing requests to index things, since these can amount to a denial of service attack. </p> <p>Probably very large data sets should be indexed only in the form supplied by their publisher, and others hosting copies would just state that they hold a copy. If the claim to the copy proved false, users could complain and the search engine administrator would remove the listing. It seems that some manual curating cannot be avoided here. </p> <h2>On Data Web Search Business Model</h2> <p>It seems there can be an overlap between the data web search and the data web hosting businesses. For example, Talis rents space for hosting <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1a60c7e0">RDF</a> data with SPARQL access. A search engine should offer basic indexing of everything for free, but could charge either data publishers or end users for running SPARQL queries across data sets. These do not have the nicely anticipatable and fairly uniform resource consumption of text lookups. In this manner, a search provider could cost-justify the capacity for allowing arbitrary queries. </p> <p>The value of the data web consists of unexpected joining. Such joining takes place most efficiently if the sources are at least in some proximity, for example in the same data center. Thus the search provider could monetize functioning as the database provider for mesh-ups. In the document web, publishing pages is very simple and there is no great benefit from co-locating search and pages, rather the opposite. For the data web, the hosting with SPARQL and all is more complex and resembles providing search. Thus providing search can combine with providing SPARQL hosting, once we accept in principle that search should have arbitrary inter-document joining, even if it is at an extra premium.</p> <p>The present search business model is advertising. If the data web is to be accessed by automated agents such as mesh-up code, display of ads is not self-evident. This is quite separate from the fact that semantics can lead to better ad targeting.</p> <p>One model would be to do text lookups for free from a regular web page but show ads, just a la Google search ads. Using the service via web services for text or SPARQL would have a cost paid by the searching or publishing party and would not be financed by advertising.</p> <p>In the case of data used in value-add data products (mesh-ups) that have financial value to their users, the original publisher of the data could even be paid for keeping the data up-to-date. This would hold for any time-sensitive feeds like news or financial feeds. Thus the hosting/search provider would be a broker of data-use fees and the data producer would be in the position of an AdSense inventory owner, i.e., a web site which shows AdSense ads. Organizing this under a hub providing back-office functions similar to an ad network could make sense even if the actual processing were divided among many sites.</p> <p>Kingsley has repeatedly formulated the core value proposition of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x3728a2f8">semantic web</a> in terms of dealing with <a href="http://dbpedia.org/resource/Information" id="link-id0x1bbcbeb8">information</a> overload: There is the real-time enterprise and the real-time individual and both are beasts of perception. Their image is won and lost in the <a href="http://dbpedia.org/resource/Internet" id="link-id0x1843b020">Internet</a> online conversation space. We know that allegations, even if later proven false, will stick if left unchallenged. The function of semantics on the web is to allow one to track and manage where one stands. In fact, Garlik has made a business of just this, but now from a privacy and security angle. The <a href="http://www.garlik.com/" id="link-id0x1aa76ab0">Garlik DataPatrol</a> harvests data from diverse sources and allows assessing vulnerability to identity theft, for example.</p> <p>If one is in the business of collating all the structured data in the world, as a data web search engine is, then providing custom alerts for both security or public image management is quite natural. This can be a very valuable service if it works well.</p> <p>At OpenLink, we will now experiment with the Sindice/<a href="http://zitgist.com/about/" id="link-id0x18800228">Zitgist</a>/PingTheSemanticWeb content. This is a regular part of the productization of <a href="http://virtuoso.openlinksw.com" id="link-id0x1adf39c8">Virtuoso</a>'s cluster edition. We expect to release some results in the next 4 weeks.</p> </div>
2015-08-21T14:31:03.220528-04:00
WWW 2008
http://www.openlinksw.com:443/blog/vdb/blog/?date=2008-04-29#1348
2008-04-29T14:37:20Z
<div> <div style="display:none;">WWW 2008</div> <p>Following my return from WWW 2008 in <a href="http://www2008.org/" id="link-id0x9ff7d5d0">Beijing</a>, I will write a series of <a href="http://dbpedia.org/resource/Blog" id="link-id0x9e4a7650">blog</a> posts discussing diverse topics that were brought up in presentations and conversations during the week.</p> <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x9e7ae398">Linked data</a> was our main interest in the conference and there was a one day workshop on this, unfortunately overlapping with a day of W3C Advisory Committee meetings. Hence Tim Berners-Lee, one of the chairs of the workshop, could not attend for most of the day. Still, he was present to say that "<a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0xa287d38">Linked open data</a> is the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x15372940">semantic web</a> and the web done as it ought to be done." <p>For my part, I will draw some architecture conclusions from the different talks and extrapolate about the requirements on database platforms for linked data.</p> <p>Chris Bizer predicted that 2008 would be the year of <a href="http://dbpedia.org/resource/Data" id="link-id0xa1454c58">data</a> web search, if 2007 was the year of <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xa0f73c50">SPARQL</a>. This may be the case, as linked data is now pretty much a reality and the questions of discovery become prevalent. There was a birds-of-a-feather session on this and I will make some comments on what we intend to explore in bridging between the text index based semantic web search engines and SPARQL.</p> <p>Andy Seaborne convened a birds-of-a-feather session on the future of SPARQL. Many of the already anticipated and implemented requirements were confirmed and a few were introduced. A separate blog post will discuss these further.</p> <p>From the various discussions held throughout the conference, we conclude that plug-and-play operation with the major semantic web frameworks of Jena, Sesame, and Redland, is our major immediate-term deliverable. Our efforts in this direction thus far are insufficient and we will next have these done with the right supervision and proper interop testing. The issues are fortunately simple but doing things totally right require some small server side support and some <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0xa5d4d5b8">JDBC</a>/<a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x9dc28d10">ODBC</a> tweaks, so to the interested, we advise to wait for an update to be published on this blog.</p> <p>I further had a conversation with Andy Seaborne about using Jena reasoning capabilities with <a href="http://virtuoso.openlinksw.com" id="link-id0xa2754050">Virtuoso</a> and generally the issues of "impedance mismatch" between reasoning and typical database workloads. More on this later. </p> </div>
2008-04-29T13:35:23-04:00
Linked Data and Information Architecture
http://www.openlinksw.com:443/weblog/oerling/?date=2008-04-29#1347
2008-04-29T12:08:24Z
<p>We had a workshop on <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x9e7e5098">Linked Open Data</a> (<a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0xb9e86c8">LOD</a>) last week in <a href="http://www2008.org/" id="link-id0x131a72a0">Beijing</a>. You can see the papers in <a href="http://events.linkeddata.org/ldow2008/#program" id="link-id10651ab8">the program</a>. The event was a success with plenty of good talks and animated conversation. I will not go into every paper here but will comment a little on the conversation and draw some technology requirements going forward.</p> <p>Tim Berners-Lee showed a read-write version of <a href="http://dig.csail.mit.edu/2005/ajar/release/tabulator/0.8/tab.html" id="link-id0x9da015a0">Tabulator</a>. This raises the question of updating on the <a href="http://dbpedia.org/resource/Data" id="link-id0x14813578">Data</a> Web. The consensus was that one could assert what one wanted in one's own space but that others' spaces would be read-only. What spaces one considered relevant would be the user's or developer's business, as in the document web.</p> <p>It seems to me that a significant use case of LOD is an open-web situation where the user picks a broad read-only "data wallpaper" or backdrop of assertions, and then uses this combined with a much smaller, local, writable data set. This is certainly the case when editing data for publishing, as in Tim's demo. This will also be the case when developing mesh-ups combining multiple distinct data sets bound together by sets of SameAs assertions, for example. Questions like, "What is the minimum subset of n data sets needed for deriving the result?" will be common. This will also be the case in applications using proprietary data combined with open data.</p> <p>This means that databases will have to deal with queries that specify large lists of included graphs, all graphs in the store or all graphs with an exclusion list. All this is quite possible but again should be considered when architecting systems for an open <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x9f1a1d8">linked data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0xa5a3e1b0">web</a>.</p> <p>"There is data but what can we really do with it? How far can we trust it, and what can we confidently decide based on it?"</p> <p>As an answer to this question, <a href="http://zitgist.com/about/" id="link-id0xad44dc0">Zitgist</a> has compiled the <a href="http://umbel.org/about/" id="link-id0x9ebde358">UMBEL</a> taxonomy using <a href="http://dbpedia.org/resource/SKOS" id="link-id0xa04a85c0">SKOS</a>. This draws on Wikipedia, Open CYC, Wordnet, and <a href="http://www.mpi-inf.mpg.de/~suchanek/downloads/yago/" id="link-id0x9fdd9018">YAGO</a>, hence the acronym WOWY. UMBEL is both a taxononmy and a set of instance data, containing a large set of <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id0xa0b9cb70">named entities</a>, including persons, organizations, geopolitical entities, and so forth. By extracting references to this set of named entities from documents and correlating this to the taxonomy, one gets a good idea of what a document (or part thereof) is about.</p> <p>Kingsley presented this in the Zitgist demo. This is our answer to the criticism about <a href="http://dbpedia.org/resource/DBpedia" id="link-id0xdc5d940">DBpedia</a> having errors in classification. DBpedia, as a bootstrap stage, is about giving names to all things. Subsequent efforts like UMBEL are about refining the relationships.</p> <p>"Should there be a global <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0xa0aa2cc0">URI</a> dictionary?"</p> <p>There was a talk by Paolo Bouquet about <a href="http://dbpedia.org/resource/Entity" id="link-id0x9e6f4b28">Entity</a> Name System, a a sort of data DNS, with the purpose of associating some description and rough classification to URIs. This would allow discovering URIs for reuse. I'd say that this is good if it can cut down on the SameAs proliferation and if this can be widely distributed and replicated for resilience, <i>à la</i> DNS. On the other hand, it was pointed out that this was not quite in the LOD spirit, where parties would mint their own dereferenceable URIs, in their own domains. We'll see.</p> <p>"What to do when identity expires?"</p> <p>Giovanni of Sindice said that a document should be removed from search if it was no longer available. Kingsley pointed out that resilience of reference requires some way to recover data. The data web cannot be less resilient than the document web, and there is a point to having access to history. He recommended hooking up with the <a href="http://dbpedia.org/resource/Internet" id="link-id0xcc203f8">Internet</a> Archive, since they make long term persistence their business. In this way, if an application depends on data, and the URIs on which it depends are no longer dereferenceable or or provide content from a new owner of the domain, those who need the old version can still get it and host it themselves.</p> <p>It is increasingly clear that OWL SameAs is both the blessing and bane of linked data. We can easily have tens of URIs for the same thing, especially with people. Still, these should be considered the same.</p> <p>Returning every synonym in a query answer hardly makes sense but accepting them as input seems almost necessary. This is what we do with <a href="http://virtuoso.openlinksw.com" id="link-id0xa0d28c98">Virtuoso</a>'s SameAs support. Even so, this can easily double query times even when there are no synonyms.</p> <p>Be that as it may, SameAs is here to stay; just consider the mapping of DBpedia to Geonames, for example.</p> <p>Also, making aberrant SameAs statements can completely poison a data set and lead to absurd query results. Hence choosing which SameAs assertions from which source will be considered seems necessary. In an open web scenario, this leads inevitably to multi-graph queries that can be complex to write with regular <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xc89a768">SPARQL</a>. By extension, it seems that a good query would also include the graphs actually used for deriving each result row. This is of course possible but has some implications on how databases should be organized.</p> <p>Yves Raymond gave a talk about deriving identity between Musicbrainz and Jamendo. I see the issue as a core question of linked data in general. The algorithm Yves presented started with attribute value similarities and then followed related entities. Artists would be the same if they had similar names and similar names of albums with similar song titles, for example. We can find the same basic question in any analysis, for example, looking at how news reporting differs between media, supposing there is adequate entity extraction.</p> <p>There is basic graph diffing in <a href="http://data.semanticweb.org/conference/iswc-aswc/2007/tracks/research/papers/533/html" id="link-id0x9fe62620">RDFSync</a>, for example. But here we are expanding the context significantly. We will traverse references to some depth, allow similarity matches, SameAs, and so forth. Having presumed identity of two URIs, we can then look at the difference in their environment to produce a human readable summary. This could then be evaluated for purposes of analysis or of combining content.</p> <p>At first sight, these algorithms seem well parallelizable, as long as all threads have access to all data. For scaling, this means a probably message-bound distributed algorithm. This is something to look into for the next stage of linked data.</p> <p>Some inference is needed, but if everybody has their own choice of data sets to query, then everybody would also have their own entailed triples. This will make for an explosion of entailed graphs if forward chaining is used. Forward chaining is very nice because it keeps queries simple and easy to optimize. With Virtuoso, we still favor backward chaining since we expect a great diversity of graph combinations and near infinite volume in the open web scenario. With private repositories of slowly changing data put together for a special application, the situation is different.</p> <p>In conclusion, we have a real LOD movement with actual momentum and a good idea of what to do next. The next step is promoting this to the broader community, starting with <a href="http://www.linkeddataplanet.com/" id="link-id0xa16319a8">Linked Data Planet</a> in New York in June.</p>
2008-04-29T17:18:15.000009-04:00
On Sem Web Search
http://www.openlinksw.com:443/weblog/oerling/?date=2008-04-29#1346
2008-04-29T12:03:47Z
<p> <i>"I give the search keywords and you give me a <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1a474e80">SPARQL</a> end-point and a query that will get the <a href="http://dbpedia.org/resource/Data" id="link-id0x18e93100">data</a>."</i> </p> <p>Thus did one SPARQL user describe the task of a semantic/data web search engine.</p> <p>In <a href="http://www.openlinksw.com/weblog/oerling/?id=1336" id="link-idff98750">a previous post</a>, I suggested that if the data web were the size of the document web, we'd be looking at two orders of magnitude more search complexity. It just might be so.</p> <p>In the conversation, I pointed out that a search engine might have a copy of everything and even a capability to do SPARQL and full text on it all, yet still the users would be better off doing the queries against the SPARQL end-points of the data publishers. It is a bit like the fact that not all web browsing runs off Google's cache. With the data web, the point is even more pronounced, as serving a hit from Google's cache is a small operation but a complex query might be a very large one.</p> <p>Yet, the data web is about ad-hoc joining between data sets of different origins. Thus a search engine of the data web ought to be capable of joining also, even if large queries ought to be run against individual publishers' end-points or the user's own data warehouse.</p> <p>For ranking, the general consensus was that no single hit-ranking would be good for the data web. Thus word frequency-based hit-scores are OK for text hits but more is not obvious. I would think that some link analysis could apply but this will take some more experimentation.</p> <p>For search summaries, if we have splitting of data sets into small fragments <i>à la</i> <a href="http://sindice.com/" id="link-id0x19f64110">Sindice</a>, search summaries are pretty much the same as with just text search. If we store triples, then we can give text style summaries of text hits in literals and Fresnel lens views of the structured data around the literal. For showing a page of hits, the lenses must abbreviate heavily but this is still feasible. The engine would know about the most common ontologies and summarize instance data accordingly.</p> <p>Chris Bizer pointed out that trust and provenance are critical, especially if an answer is arrived at by joining multiple data sets. The trust of the conclusion is no greater than that of the weakest participating document. Different users will have different trusted sources.</p> <p>A mature data web search engine would combine a provenance/trust specification, a search condition consisting of SPARQL or full text or both, and a specification for hit rank. Again, most searches would use defaults, but these three components should in principle be orthogonally specifiable.</p> <p>Many places may host the same data set either for download or SPARQL access. The <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x19f34660">URI</a> of the data set is not its <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id0x1afbb680">URL</a>. Different places may further host multiple data sets on one end-point. Thus the search engine ought to return all end-points where the set is to be found. The end-points themselves ought to be able to say what data sets they contain, under what graph IRIs. Since there is no consensus about end-point self description, this too would be left to the search engine. In practice, this could be accomplished by extending Sindice's semantic site map specification. A possible query would be to find an end-point containing a set of named data sets. If none were found, the search engine itself could run a query joining all the sets since it at least would hold them all.</p> <p>Since many places will host sets like Wordnet or Uniprot, indexing these once for each copy hardly makes sense. Thus a site should identify its data by the data set's URI and not the copy's URL.</p> <p>It came up in the discussion that search engines should share a ping format so that a single message format would be enough to notify any engine about data being updated. This is already partly the case with Sindice and <a href="http://www.pingthesemanticweb.com/" id="link-id0x17e36628">PTSW</a> (<a href="http://www.pingthesemanticweb.com/" id="link-id0x1a2a9060">PingTheSemanticWeb</a>) sharing a ping format. </p> <p>Further, since it is no trouble to publish a copy of the 45G Uniprot file but a fair amount of work to index it, search engines should be smart about processing requests to index things, since these can amount to a denial of service attack. </p> <p>Probably very large data sets should be indexed only in the form supplied by their publisher, and others hosting copies would just state that they hold a copy. If the claim to the copy proved false, users could complain and the search engine administrator would remove the listing. It seems that some manual curating cannot be avoided here. </p> <h2>On Data Web Search Business Model</h2> <p>It seems there can be an overlap between the data web search and the data web hosting businesses. For example, Talis rents space for hosting <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x192732f8">RDF</a> data with SPARQL access. A search engine should offer basic indexing of everything for free, but could charge either data publishers or end users for running SPARQL queries across data sets. These do not have the nicely anticipatable and fairly uniform resource consumption of text lookups. In this manner, a search provider could cost-justify the capacity for allowing arbitrary queries. </p> <p>The value of the data web consists of unexpected joining. Such joining takes place most efficiently if the sources are at least in some proximity, for example in the same data center. Thus the search provider could monetize functioning as the database provider for mesh-ups. In the document web, publishing pages is very simple and there is no great benefit from co-locating search and pages, rather the opposite. For the data web, the hosting with SPARQL and all is more complex and resembles providing search. Thus providing search can combine with providing SPARQL hosting, once we accept in principle that search should have arbitrary inter-document joining, even if it is at an extra premium.</p> <p>The present search business model is advertising. If the data web is to be accessed by automated agents such as mesh-up code, display of ads is not self-evident. This is quite separate from the fact that semantics can lead to better ad targeting.</p> <p>One model would be to do text lookups for free from a regular web page but show ads, just a la Google search ads. Using the service via web services for text or SPARQL would have a cost paid by the searching or publishing party and would not be financed by advertising.</p> <p>In the case of data used in value-add data products (mesh-ups) that have financial value to their users, the original publisher of the data could even be paid for keeping the data up-to-date. This would hold for any time-sensitive feeds like news or financial feeds. Thus the hosting/search provider would be a broker of data-use fees and the data producer would be in the position of an AdSense inventory owner, i.e., a web site which shows AdSense ads. Organizing this under a hub providing back-office functions similar to an ad network could make sense even if the actual processing were divided among many sites.</p> <p>Kingsley has repeatedly formulated the core value proposition of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x192611e0">semantic web</a> in terms of dealing with <a href="http://dbpedia.org/resource/Information" id="link-id0x1b0abcf0">information</a> overload: There is the real-time enterprise and the real-time individual and both are beasts of perception. Their image is won and lost in the <a href="http://dbpedia.org/resource/Internet" id="link-id0x1b2d4078">Internet</a> online conversation space. We know that allegations, even if later proven false, will stick if left unchallenged. The function of semantics on the web is to allow one to track and manage where one stands. In fact, Garlik has made a business of just this, but now from a privacy and security angle. The <a href="http://www.garlik.com/" id="link-id0x1a563b50">Garlik DataPatrol</a> harvests data from diverse sources and allows assessing vulnerability to identity theft, for example.</p> <p>If one is in the business of collating all the structured data in the world, as a data web search engine is, then providing custom alerts for both security or public image management is quite natural. This can be a very valuable service if it works well.</p> <p>At OpenLink, we will now experiment with the Sindice/<a href="http://zitgist.com/about/" id="link-id0x1bb17320">Zitgist</a>/PingTheSemanticWeb content. This is a regular part of the productization of <a href="http://virtuoso.openlinksw.com" id="link-id0x1b20cfc8">Virtuoso</a>'s cluster edition. We expect to release some results in the next 4 weeks.</p>
2008-10-02T11:37:07.000002-04:00
WWW 2008
http://www.openlinksw.com:443/weblog/oerling/?date=2008-04-29#1345
2008-04-29T11:59:16Z
<p>Following my return from WWW 2008 in <a href="http://www2008.org/" id="link-id0xa1901330">Beijing</a>, I will write a series of <a href="http://dbpedia.org/resource/Blog" id="link-id0xd356660">blog</a> posts discussing diverse topics that were brought up in presentations and conversations during the week.</p> <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1290cb78">Linked data</a> was our main interest in the conference and there was a one day workshop on this, unfortunately overlapping with a day of W3C Advisory Committee meetings. Hence Tim Berners-Lee, one of the chairs of the workshop, could not attend for most of the day. Still, he was present to say that "<a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0xa2754050">Linked open data</a> is the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0xa099dea8">semantic web</a> and the web done as it ought to be done." <p>For my part, I will draw some architecture conclusions from the different talks and extrapolate about the requirements on database platforms for linked data.</p> <p>Chris Bizer predicted that 2008 would be the year of <a href="http://dbpedia.org/resource/Data" id="link-id0x9e0f0a98">data</a> web search, if 2007 was the year of <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xd9e64a0">SPARQL</a>. This may be the case, as linked data is now pretty much a reality and the questions of discovery become prevalent. There was a birds-of-a-feather session on this and I will make some comments on what we intend to explore in bridging between the text index based semantic web search engines and SPARQL.</p> <p>Andy Seaborne convened a birds-of-a-feather session on the future of SPARQL. Many of the already anticipated and implemented requirements were confirmed and a few were introduced. A separate blog post will discuss these further.</p> <p>From the various discussions held throughout the conference, we conclude that plug-and-play operation with the major semantic web frameworks of Jena, Sesame, and Redland, is our major immediate-term deliverable. Our efforts in this direction thus far are insufficient and we will next have these done with the right supervision and proper interop testing. The issues are fortunately simple but doing things totally right require some small server side support and some <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0xacc8c7e8">JDBC</a>/<a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x9e48d258">ODBC</a> tweaks, so to the interested, we advise to wait for an update to be published on this blog.</p> <p>I further had a conversation with Andy Seaborne about using Jena reasoning capabilities with <a href="http://virtuoso.openlinksw.com" id="link-id0x134b8e98">Virtuoso</a> and generally the issues of "impedance mismatch" between reasoning and typical database workloads. More on this later. </p>
2008-04-29T13:35:17-04:00
Linked Data Trip Report - Part 1 (Update 2)
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-04-28#1343
2008-04-28T18:48:33Z
<p>Typo cleansed edition :-)</p> <h2>Objectives</h2> <ul> <li>Meet <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id12c52e40">LOD</a> Community Members</li> <li>Participate in Workshop </li> </ul> <h3>Meeting <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id10456058">LOD</a> Community Members</h3> <p>Although the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> continues to shrink the planet by removing the restrictions of geopgrahic location, meeting people face-to-face remains invaluable (*priceless in Mastercard AD speak*). Naturally, meeting and chatting with as many <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id1382d4f8">LOD</a> community members as possible was high up on my agenda.</p> <h3>Participate in Workshop </h3> <p>As one of the co-chairs of the Linking Open <a href="http://dbpedia.org/resource/Data" id="link-id143a8c30">Data</a> Workshop (<a href="http://events.linkeddata.org/ldow2008/" id="link-id10621d70">LODW</a>), I had a 5 minute workshop opening slot during which I spoke about the following:</p> <h4>Where we are today: </h4> <p>We have <a href="http://dbpedia.org/resource/DBpedia" id="link-id109b40a0">DBpedia</a> as a major hub on the burgeoning <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1074f248">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id101ed948">Web</a>. When OpenLink offered to host <a href="http://dbpedia.org/resource/DBpedia" id="link-id10cd1b20">DBpedia</a> (a combination of <a href="http://virtuoso.openlinksw.com" id="link-id13d19880">Virtuoso</a> DBMS Software and sizable backend Hardware infrastructure), it did so knowing that such an effort would emphatically address the "chicken and egg" conundrum that, prior to this undertaking, stifled the ability to demonstrate practical utility of <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id13835980">HTTP</a> based <a href="http://dbpedia.org/resource/Linked_Data" id="link-id14a89c28">Linked Data</a>.<br /> <br /> Today, the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10aa9fe0">Linked Data</a> bootstrap mission has been accomplished.</p> <h4>Where we go next:</h4> <p>Although <a href="http://dbpedia.org/resource/DBpedia" id="link-id12fe5d98">DBpedia</a> is a hub (ground zero of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id14aca9b8">Linked Data</a>), we have to put it into perspective in relation to a new set of needs and expectations moving forward. Today, <a href="http://dbpedia.org/resource/DBpedia" id="link-id1027f098">DBpedia</a> is a Sun at the heart of a Solar System within the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id14a6adf8">Linked Data</a> Galaxy. But unlike Space as we know it, in <a href="http://dbpedia.org/resource/Cyberspace" id="link-id14a80398">Cyberspace</a> we can have connectivity and collaboration across Solar Systems -- life exists elsewhere and we are part of a collaborative collective unimpeded by constraints of space travel etc. Thus, expect to see the emergence of other Solar Systems accessible to <a href="http://dbpedia.org/resource/DBpedia" id="link-id14904998">DBpedia</a> and its collections of planets (see. <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id1097d200">LOD</a> <a href="http://richard.cyganiak.de/2007/10/lod/" id="link-id14acecf0">diagram</a>). Examples underway include <a href="http://umbel.org/about/" id="link-idfe92c08">UMBEL</a> which will serve the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id107fac40">Linked Data</a> planets from <a href="http://dbpedia.org/resource/Cyc" id="link-id12fb9e88">OpenCyc</a> (Subject Matter Concepts), <a href="http://www.mpi-inf.mpg.de/~suchanek/downloads/yago/" id="link-id147ea790">Yago</a> (<a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id14ae83a8">Named Entities</a>), and <a href="http://www.bio2rdf.org/" id="link-id10890640">Bio2RDF</a> (which provides powerful Bio Informatics based <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1439a858">Linked Data</a> planet).</p> <p>I urged the community to veer more aggressively towards developing and demonstrating practical <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11f8e188">Linked Data</a> driven solutions that are aligned to well known problems. Of course, I encouraged all presenters to make this an integral part of their presentations :-) </p> <h4>Workshop Summary: </h4> <p>The workshop was well attended and I found all the presentations engaging and full of enthusiasm. </p> <p>As the sessions progressed, it became clear during a number of accompanying Q&A sessions that a new <a href="http://dbpedia.org/resource/Linked_Data" id="link-idff249b0">Linked Data</a> exploitation frontier is emerging. The frontier in question takes the form of a <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10f73b50">Linked Data</a> substrate capable of addressing the taxonomic needs of solutions aimed at automated <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id10d16e00">Named Entity Extraction</a>, Disambiguation, Subject matter Concept alignment, transparently integrated with existing Web Content. Thus, we are moving beyond the minting and deployment of of dereferencable URIs and <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id10419210">RDF</a> <a href="http://dbpedia.org/resource/Data" id="link-id151520d0">data</a> sets to automagically associating existing Web Content with <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id105573b0">Named Entities</a> (People, Organizations, Places, Events etc..) and Subject matter Concepts (Politics, Music, Sports, and others) while remaining true to the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id14579108">Linking Open Data Community</a> creed i.e. ensuring the Named <a href="http://dbpedia.org/resource/Entity" id="link-id147d31e8">Entity</a> and Subject matter Concept URIs are available to user agents or users seeking to produce alternative <a href="http://dbpedia.org/resource/Data" id="link-id14ae41f0">data</a> views (i.e. Mesh-ups).</p> <p>I will get to part 2 of this report once the actual workshop sessions slides go live (*these are different from the <a href="http://events.linkeddata.org/ldow2008/" id="link-id1034eee8">pre-event PDFs links</a>*).</p>
2008-04-29T11:07:43.000002-04:00
Linked Data Illustrated and a Virtuoso Functionality Reminder
http://www.openlinksw.com:443/blog/kidehen@openlinksw.com/blog/?date=2008-04-28#1342
2008-04-28T17:32:47Z
<a href="http://myopenlink.net/dataspace/person/danieljohnlewis#this" id="link-id156ceb30">Daniel Lewis</a> has put together a nice <a href="http://vanirsystems.com/danielsblog/2008/04/27/linked-data-the-role-of-the-data-server/" id="link-id10456040">collection of Linked Data related posts</a> that illustrate the fundamentals of the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1033f6f0">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id106fa168">Web</a> and the vital role that <a href="http://virtuoso.openlinksw.com" id="link-id10141c20">Virtuoso</a> plays as a deployment platform. Remember, <a href="http://virtuoso.openlinksw.com" id="link-id10301e38">Virtuoso</a> was architected in 1998 (see <a href="http://virtuoso.openlinksw.com/wiki/main/Main/VOSHistory" id="link-id10c44088">Virtuoso History</a>) in anticipation of the eventual <a href="http://dbpedia.org/resource/Internet" id="link-id1383a1e8">Internet</a>, <a href="http://dbpedia.org/resource/Intranet" id="link-id1028e770">Int