http://www.openlinksw.com/blog/vdb/blog/
OpenLink Virtuoso (Product Blog)
A great place to track Virtuoso's rapid evolution.
kidehen@openlinksw.com
kidehen@openlinksw.com
2009-11-23T11:53:28Z
Virtuoso Universal Server 05.12.3041
http://www.openlinksw.com/weblog/public/images/vbloglogo.gif
Some Interesting VLDB 2009 Papers (2 of 5)
http://www.openlinksw.com/blog/vdb/blog/?date=2009-09-01#1579
2009-09-01T15:46:14Z
2009-09-01T17:32:45.000003-04:00
<h3> <a href="http://dbpedia.org/resource/Intel_Corporation" id="link-id0x449c5e0">Intel</a> on <a href="http://dbpedia.org/resource/Hash_join" id="link-id0x4e82430">Hash Join</a> </h3> <p>Intel and <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x10bae5e8">Oracle</a> had measured hash and sort merge joins on Intel Core i7. The result was that hash join with both tables partitioned to match <a href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x3827798">CPU</a> <a href="http://dbpedia.org/resource/Cache" id="link-id0x2545b978">cache</a> was still the best but that sort/merge would catch up with more <a href="http://dbpedia.org/resource/SIMD" id="link-id0x32f4e40">SIMD</a> instructions in the future.</p> <p>We should probably experiment with this but the most important partitioning of hash joins is still between cluster nodes. Within the process, we will see. The tradeoff of doing all in cache-sized partitions is larger intermediate results which in turn will impact the working set of disk pages in RAM. For one-off queries this is OK; for online use this has an effect.</p> <h3>1000 TABLE Queries</h3> <p> <a href="http://dbpedia.org/resource/SAP_AG" id="link-id0x4ed7710">SAP</a> presented a paper about <a href="http://dbpedia.org/resource/Federated_database_system" id="link-id0x26827fd8">federating relational databases</a>. Queries would be expressed against VIEWs defined over remote TABLEs, UNIONed together and so forth. Traditional methods of <a href="http://dbpedia.org/resource/Program_optimization" id="link-id0x3838888">optimization</a> would run out of memory; a single 1000 TABLE plan is already a big thing. Enumerating multiple variations of such is not possible in practice. So the solution was to plan in two stages â first arrange the subqueries and derived TABLEs, and then do the JOIN orders locally. Further, local JOIN orders could even be adjusted at run time based on the actual <a href="http://dbpedia.org/resource/Data" id="link-id0x26033030">data</a>. Nice.</p> <h3>Oracle Subqueries and New Implementation of LOBs</h3> <p>Oracle presented some new <a href="http://dbpedia.org/resource/SQL" id="link-id0x23a0eb48">SQL</a> optimizations, combining and inlining subqueries and derived TABLEs. We do fairly similar things and might extend the repertoire of tricks in the direction outlined by Oracle as and when the need presents itself. This further confirms that SQL and other query optimization is really an incremental collection of specially recognized patterns. We still have not found any other way of doing it.</p> <p>Another interesting piece by Oracle was about their re-implementation of large object support, where they compared LOB loading to file system and raw device speeds.</p> <h3> <a href="http://dbpedia.org/resource/Amadeus_CRS" id="link-id0x1566d470">Amadeus CRS</a> booking system, steady query time for arbitrary single table queries</h3> <p>There was a paper about a memory-resident database that could give steady time for any kind of single-table scan query. The innovation was to not use indices, but to have one partition of the table per processor core, all in memory. Then each core would have exactly two cursors â one reading, the other writing. The write cursor should keep ahead of the read cursor. Like this, there would be no read/write contention on pages, no locking, no multiple threads splitting a tree at different points, none of the complexity of a multithreaded database engine. Then, when the cursor would hit a row, it would look at the set of queries or updates and add the result to the output if there was a result. The data indexes the queries, not the other way around. We have done something similar for detecting changes in a full text corpus but never thought of doing queries this way.</p> <p>Well, we are all about JOINs so this is not for us, but it deserves a mention for being original and clever. And indeed, anything one can ask about a table will likely be served with great predictability.</p> <h3> <a href="http://dbpedia.org/resource/Greenplum" id="link-id0x196b0538">Greenplum</a> </h3> <p> <a href="http://dbpedia.org/resource/Google" id="link-id0x108f8878">Google</a>'s chief economist said that the winning career choice would be to pick a scarce skill that made value from something that was plentiful. For the 2010s this career is that of the statistician/data analyst. We've said it before â the next web is analytics for all. The Greenplum talk was divided between the Fox use case, with 200TB of data about ads, web site traffic, and other things, growing 5TB a day. The message was that cubes and drill down are passé, that it is about complex statistical methods that have to run in the database, that the new kind of geek is the data geek, whose vocation it is to consume and spit out data, discover things in it, and so forth.</p> <p>The technical part was about Greenplum, a SQL database running on a cluster with a <a href="http://dbpedia.org/resource/PostgreSQL" id="link-id0x3106d00">PostgreSQL</a> back-end. The interesting points were embedding <a href="http://dbpedia.org/resource/MapReduce" id="link-id0x17968370">MapReduce</a> into SQL, and using relational tables for arrays and complex data types â pretty much what we also do. Greenplum emphasized scale-out and found column orientation more like a nice-to-have.</p> <h3> <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x119f7948">MonetDB</a>, optimizing database for CPU cache</h3> <p>The MonetDB people from <a href="http://dbpedia.org/resource/National_Research_Institute_for_Mathematics_and_Computer_Science" id="link-id0x3617658">CWI</a> in Amsterdam gave a 10 year best paper award talk about optimizing database for CPU cache. The key point was that if data is stored as columns, it ought also to be transferred as columns inside the execution engine. Materialize big chunks of state to cut down on interpretation overhead and use cache to best effect. They vector for CPU cache; we vector for scale-out, since the only way to ship operations is to ship many at a time. So we might as well vector also in single servers. This could be worth an experiment. Also we regularly visit the topic of <a href="http://dbpedia.org/resource/Column-oriented_DBMS" id="link-id0x38d43d8">column storage</a>. But we are not yet convinced that it would be better than row-style covering indices for <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x17e25760">RDF</a> quads. But something could certainly be tried, given time.</p>
Comparison of Open Source Databases with TPC D Queries
http://www.openlinksw.com/blog/vdb/blog/?date=2007-02-05#1132
2007-02-05T11:44:34Z
2008-04-17T21:04:40-04:00
<div> <div style="display:none;">Comparison of Open Source Databases with TPC D Queries</div> <p> <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1116" id="link-id10598cc0">Last time</a> we talked about database engine and transactions. Now we have come to the realm of query processing in our revisiting of the DBMS side of <a href="http://virtuoso.openlinksw.com" id="link-id0xd3ecd30">Virtuoso</a>.</p> <p>Now the well established, respectable standard benchmark for the basics of query processing is TPC D with its derivatives H and R. So we have, for testing how different <a href="http://dbpedia.org/resource/SQL" id="link-id0x15cf6938">SQL</a> optimizers manage the 22 queries, run a mini version of the D queries with a 1% scale database, some 30M of <a href="http://dbpedia.org/resource/Data" id="link-id0x15ce3928">data</a>, all in memory. This basically catches whether SQL implementations miss some of the expected tricks and how efficient in memory loop and hash joins and aggregation are.</p> <p>When we get to our next stop, high volume I/O, we will run the same with D databases in the 10G ballpark.</p> <p>The databases were tested on the same machine, with warm cache, taking the best run of 3. All had full statistics and were running with read committed isolation, where applicable. The data was generated using the procedures from the Virtuoso test suite. The Virtuoso version tested was 5.0, to be released shortly. The <a href="http://dbpedia.org/resource/MySQL" id="link-id0xc952f58">MySQL</a> was 5.0.27, the PostgreSQL was 8.1.6. </p> <table style="width: 334px; height: 556px; " border="1"> <tbody> <tr> <th rowspan="2">Query</th> <th colspan="4">Query Times in Milliseconds</th> </tr> <tr> <th> Virtuoso </th> <th> PostgreSQL </th> <th> MySQL </th> <th> MySQL with InnoDB </th> </tr> <tr> <td> Q1 </td> <td align="right"> <b>206</b> </td> <td align="right"> 763 </td> <td align="right"> 312 </td> <td align="right"> 198 </td> </tr> <tr> <td> Q2 </td> <td align="right"> 4 </td> <td align="right"> 6 </td> <td align="right"> <b>3</b> </td> <td align="right"> <b>3</b> </td> </tr> <tr> <td> Q3 </td> <td align="right"> <b>13</b> </td> <td align="right"> 51 </td> <td align="right"> 254 </td> <td align="right"> 64 </td> </tr> <tr> <td> Q4 </td> <td align="right"> <b>4</b> </td> <td align="right"> 16 </td> <td align="right"> 24 </td> <td align="right"> 60 </td> </tr> <tr> <td> Q5 </td> <td align="right"> <b>15</b> </td> <td align="right"> 22 </td> <td align="right"> 64 </td> <td align="right"> 68 </td> </tr> <tr> <td> Q6 </td> <td align="right"> <b>9</b> </td> <td align="right"> 70 </td> <td align="right"> 189 </td> <td align="right"> 65 </td> </tr> <tr> <td> Q7 </td> <td align="right"> <b>52</b> </td> <td align="right"> 143 </td> <td align="right"> 211 </td> <td align="right"> 84 </td> </tr> <tr> <td> Q8 </td> <td align="right"> 29 </td> <td align="right"> 31 </td> <td align="right"> 13 </td> <td align="right"> <b>11</b> </td> </tr> <tr> <td> Q9 </td> <td align="right"> <b>36</b> </td> <td align="right"> 114 </td> <td align="right"> 97 </td> <td align="right"> 61 </td> </tr> <tr> <td> Q10 </td> <td align="right"> <b>32</b> </td> <td align="right"> 51 </td> <td align="right"> 117 </td> <td align="right"> 57 </td> </tr> <tr> <td> Q11 </td> <td align="right"> 16 </td> <td align="right"> <b>9</b> </td> <td align="right"> 12 </td> <td align="right"> 10 </td> </tr> <tr> <td> Q12 </td> <td align="right"> <b>8</b> </td> <td align="right"> 21 </td> <td align="right"> 18 </td> <td align="right"> 130 </td> </tr> <tr> <td> Q13 </td> <td align="right"> <b>18</b> </td> <td align="right"> 74 </td> <td align="right"> - </td> <td align="right"> - </td> </tr> <tr> <td> Q14 </td> <td align="right"> <b>7</b> </td> <td align="right"> 21 </td> <td align="right"> 418 </td> <td align="right"> 1425 </td> </tr> <tr> <td> Q15 </td> <td align="right"> <b>14</b> </td> <td align="right"> 43 </td> <td align="right"> 389 </td> <td align="right"> 122 </td> </tr> <tr> <td> Q16 </td> <td align="right"> <b>16</b> </td> <td align="right"> 22 </td> <td align="right"> 18 </td> <td align="right"> 25 </td> </tr> <tr> <td> Q17 </td> <td align="right"> <b>1</b> </td> <td align="right"> 54 </td> <td align="right"> 26 </td> <td align="right"> 10 </td> </tr> <tr> <td> Q18 </td> <td align="right"> <b>82</b> </td> <td align="right"> 120 </td> <td align="right"> - </td> <td align="right"> - </td> </tr> <tr> <td> Q19 </td> <td align="right"> 19 </td> <td align="right"> 8 </td> <td align="right"> <b>2</b> </td> <td align="right"> 17 </td> </tr> <tr> <td> Q20 </td> <td align="right"> <b>7<b> </b></b></td> <td align="right"> 15 </td> <td align="right"> 66 </td> <td align="right"> 52 </td> </tr> <tr> <td> Q21 </td> <td align="right"> <b>34</b> </td> <td align="right"> 86 </td> <td align="right"> 524 </td> <td align="right"> 278 </td> </tr> <tr> <td> Q22 </td> <td align="right"> <b>4</b> </td> <td align="right"> 323 </td> <td align="right"> 3311</td> <td align="right"> 805 </td> </tr> <tr> <td>Total (msec)</td> <td align="right"><b>626</b></td> <td align="right">2063</td> <td align="right">6068</td> <td align="right">3545</td> </tr> </tbody> </table> <p>We lead by a fair margin but MySQL is hampered by obviously getting some execution plans wrong and not doing Q13 and Q18 at all, at least not under several tens of seconds; so we left these out of the table in the interest of having comparable totals.</p> <p>As usual, we also ran the workload on <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x167807c8">Oracle</a> 10g R2. Since Oracle does not like their numbers being published without explicit approval, we will just say that we are even with them within the parameters described above. Oracle has a more efficient decimal type so it wins where that is central, as on Q1. Also it seems to notice that the <code>GROUP BY</code>s of Q18 are produced in order of grouping columns, so it needs no intermediate table for storing the aggregates. If we addressed these matters, we'd lead by some 15% whereas now we are even. A faster decimal arithmetic implementation may be in the release after next.</p> <p>In the next posts, we will look at IO and disk allocation, and also return to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xbb63c20">RDF</a> and LUBM.</p> </div>