http://www.openlinksw.com/weblog/oerling/
Orri Erling's Weblog
oerling@openlinksw.com
oerling@openlinksw.com
2009-11-23T10:51:19Z
Virtuoso Universal Server 05.12.3041
http://www.openlinksw.com/weblog/public/images/vbloglogo.gif
European Commission and the Data Overflow
http://www.openlinksw.com/weblog/oerling/?date=2009-10-27#1585
2009-10-27T18:29:51Z
2009-10-27T14:57:28.000002-04:00
<p>The European Commission recently circulated a questionnaire to selected experts on what could be done for the future of big <a href="http://dbpedia.org/resource/Data" id="link-id0x79cfe58">data</a>.</p> <p>Since the <a href="http://cordis.europa.eu/fp7/ict/content-knowledge/consultation_en.html" id="link-id1191c0f8">questionnaire is public</a>, I am publishing my answers below.</p> <ol type="1" start="1"> <li> <p> <b>Data and data types</b> </p> <ol type="a" start="1"> <li> <p> <b>What volumes of data are we dealing with today? What is the growth rate? Where can we expect to be in 2015? </b> </p> <p>Private data warehouses of corporations have more than doubled yearly for the past years; hundreds of TB is not exceptional. This will continue. The real shift is in structured data being published in increasing quantities with a minimum level of integrate-ability through use of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x7d7e7a0">RDF</a> and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x7f2a788">linked data</a> principles. There are rewards for use of standard vocabularies and identifiers through search engines recognizing such data. There is convergence around <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x7dfbca8">DBpedia</a> identifiers for real-world entities, e.g., most things that would be in the news.</p> <p>This also means that internal data processes and silos may be enriched with this content. There is consequent pressure for accommodating more diversity of data, with more flexible <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x7babaf8">schema</a>.</p> <p>Ultimately, all content presently stored in RDBs and presented in public accessible dynamic web pages will end up on the web of linked data. Examples are product catalogs, price lists, event schedules and the like.</p> <p>The volume of the well known linked data sets is around 10 billion statements. With the above mentioned trends, growth by two or three orders of magnitude by 2015 seems reasonable, This is so especially if explicit semantics are extracted from the document web and if there is some further progress in the precision/recall of such extraction.</p> <p>Relevant sections of this mass of data are a potential addition to any present or future analytics application.</p> <p>Since arbitrary analytics over the database which is the web cannot be economically provided by a centralized search engine, a cloud model may be used for on-demand selection of relevant data and mixing it with private data. This will drive database innovation for the next years even more than the continued classical warehouse growth.</p> <p>Science data is another driver of the data overflow. For example, faster gene sequencing, more accurate measurements in high energy physics, better imaging, and remote sensing will produce large volumes of data. This data has highly regular structure but labeling this data with source and lineage calls for a flexible, schema-last, self-describing model, such as RDF and linked data. Data and <a href="http://dbpedia.org/resource/Metadata" id="link-id0x96ce60">metadata</a> should travel together but may have different data models.</p> <p>By and large, the metadata of science data will be another stream to the web of linked data, at least to the degree it is publicly accessible. Restricted circles can and likely will implement similar ideas.</p> </li> <li> <p> <b>What types of data can we deal with intelligently due to their inherent structure (geospatial, temporal, social or <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x7e8e248">knowledge</a> graphs, 3D, sensor streams...)?</b> </p> <p>All the above types should be supported inside one DBMS so as to allow efficient querying combining conditions on all these types of data, e.g., <i>photos of sunsets taken last summer in Ibiza, with over 20 megapixels, by people I know.</i> </p> <p>Note that the test for being a sunset is an operation on the image blob that should be taken to the data; the images cannot be economically transferred.</p> <p>Interleaving of all database functions and types becomes increasingly important.</p> </li> </ol> </li> <li> <p> <b>Industries, communities</b> </p> <ol type="a" start="1"> <li> <p> <b>Who is producing these data and why? Could they do it better? How?</b> </p> <p>Right now, projects such as <a href="http://www.bio2rdf.org/" id="link-id0x43bd098">Bio2RDF</a>, <a href="http://neurocommons.org/page/Main_Page" id="link-id0x5c074b0">Neurocommons</a>, and DBPedia produce this data. The processes are in place and are reasonable. Incremental improvement is to be expected. These processes, along with the <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id0x72131d0">linked data meme</a> generally taking off, drive demand for better <a href="http://dbpedia.org/resource/Natural_language_processing" id="link-id0x71e7798">NLP</a> (<a href="http://dbpedia.org/resource/Natural_language_processing" id="link-id0x7e0e2f0">Natural Language Processing</a>), e.g., <a href="http://dbpedia.org/resource/Entity" id="link-id0x71ab500">entity</a> and relationship extraction, especially extraction that can produce instance data in given ontologies (e.g., events) using common identifiers (e.g., DBPedia URIs).</p> <p>Mapping of RDBs to RDF is possible, and a W3C working group is developing standards for this. The required baseline level has been reached; the rest is a matter of automating deployment. Within the enterprise, there are advantages to be gained for <a href="http://dbpedia.org/resource/Information" id="link-id0x7a8e9a8">information</a> integration; e.g., all entities in the CRM space can be integrated with all email and support tickets through giving everything a <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x599f630">URI</a>. Some of this information may even be published on an <a href="http://dbpedia.org/resource/Extranet" id="link-id0x2a28f98">extranet</a> for self-service and web-service interfaces. This has been done at small scales and the rest is a matter of spreading adoption and lowering the entry barrier. Incremental progress will take place, eventually resulting in qualitatively better integration along the value chain when adoption is sufficiently widespread.</p> </li> <li> <p> <b>Who is consuming these data and why? Could they do it better? How?</b> </p> <p>Consumers are various. The greatest need is for tools that summarize complex data and allow getting a bird's eye view of what data is in the first instance available. Consuming the data is hindered by the user not even necessarily knowing what data there is. This is somewhat new, as traditionally the business analyst did know the schema of the warehouse and was proficient with <a href="http://dbpedia.org/resource/SQL" id="link-id0x5999558">SQL</a> report generators and statistics packages.</p> <p>Where Web 2.0 made the <i>citizen journalist</i>, the web of linked data will make the <i>citizen analyst</i>. For this to happen, with benefits for individuals, enterprises, and governments alike, more work in user interfaces, knowledge discovery, and query composition will be useful. We may envision a "meshup economy" where data is plentiful, but the unit of value and exchange is the smart report that crystallizes actionable value from this ocean.</p> </li> <li> <p> <b>What industrial sectors in Europe could become more competitive if they became much better at managing data?</b> </p> <p>Any sector could benefit. Early adopters are seen in the biomedical field and to an extent in media. </p> </li> <li> <p> <b>Is the regulation landscape imposing constraints (privacy, compliance ...) that don't have today good tool support?</b> </p> <p>The regulation landscape drives database demand through data retention requirements and the like.</p> <p>With data integration, especially with privacy-sensitive data (as in medicine), there are issues of whether one dares put otherwise-shareable information online. Regulation is needed to protect individuals, but integration should still be possible for science.</p> <p>For this, we see a need for progress in applying policy-based approaches (e.g., row level security) to relatively schema-last data such as RDF. This is possible but needs some more work. Also, creating on-the-fly-anonymizing views on data might help.</p> <p>More research is needed for reconciling the need for security with the advantages of broad-based <i>ad hoc</i> integration. Ideally, data should be intelligent, aware of its origins and classification and cautious of whom it interacts with, all of this supported under the covers so that the user could ask anything but the data might refuse to answer or might restrict answers according to the user's profile. This is a tall order and implementing something of the sort is an open question.</p> </li> <li> <p> <b>What are the main practical problem identified for individuals and organizations? Please give examples and tell us about the main obstacles and barriers.</b> </p> <p>We have come across the following:</p> <ul> <li>Knowing that the data exists in the first place.</li> <li>If the data is found, figuring out the provenance, units and precision of measurement, identifiers, and the like.</li> <li>Compatible subject matter but incompatible representation: For example, one has numbers on a map with different maps for different points in time; another has time series of instrument data with geo-location for the instrument. It is only to be expected that the time interval between measurements is not the same. So there is need for a lot of one-off programming to align data.</li> </ul> <p>Other problems have to do with sheer volume, i.e., transfer of data even in a local area network is too slow, let alone over a wide area network. Computation needs to go to the data, and databases need to support this.</p> </li> </ol> </li> <li> <p> <b>Services, software stacks, protocols, standards, benchmarks</b> </p> <ol type="a" start="1"> <li> <p> <b>What combinations of components are needed to deal with these problems?</b> </p> <p>Recent times have seen a proliferation of special purpose databases. Since the data needs of the future are about combining data with maximum agility and minimum performance hit, there is need to gather the currently-separate functionality into an integrated system with sufficient flexibility. We see some of this in integration of map-reduce and scale-out databases. The former antagonists have become partners. Vertica, <a href="http://dbpedia.org/resource/Greenplum" id="link-id0x45ecfa0">Greenplum</a>, and OpenLink <a href="http://virtuoso.openlinksw.com" id="link-id0x7f73fc8">Virtuoso</a> are example of DBMS featuring work in this direction.</p> <p>Interoperability and at least <i>de facto</i> standards in ways of doing this will emerge.</p> </li> <li> <p> <b>What data exchange and processing mechanisms will be needed to work across platforms and programming languages?</b> </p> <p> <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x776a1a0">HTTP</a>, <a href="http://dbpedia.org/resource/XML" id="link-id0x2a4e8d0">XML</a>, and RDF are in fact very verbose, yet these are the formats and models that have uptake. Thus, these will continue to be used even though one might think binary formats to be more efficient.</p> <p>There are of course science data set standards that are more compressed and these will continue, hopefully adding a practice of rich metadata in RDF.</p> <p>For internals of systems, MPI and TCP/IP with proprietary optimized wire formats will continue. Inter-system communication will likely continue to be HTTP, XML, and RDF as appropriate.</p> </li> <li> <p> <b>What data environments are today so wastefully messy that they would benefit from the development of standards?</b> </p> <p>RDF and <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x2a35960">OWL</a> are not messy but they could use some more performance; we are working on this. <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x12362e8">SPARQL</a> is finally acquiring the capabilities of a serious query language, so things are slowly coming together.</p> <p>Community process for developing application domain specific vocabularies works quite well, even though one could argue it is <i>ad hoc</i> and not up to what a modeling purist might wish.</p> <p>Top-down imposition of standards has a mixed history, with long and expensive development and sometimes no or little uptake, consider some WS* standards for example.</p> </li> <li> <p> <b>What kind of performance is expected or required of these systems? Who will measure it reliably? How?</b> </p> <p>Relational databases have a history of substantial investment in <a href="http://dbpedia.org/resource/Program_optimization" id="link-id0x7b2d7c8">optimization</a> and some of them are very good for what they do, e.g., the newer generation of analytics databases.</p> <p>The very large schema-last, no-SQL, sometimes eventually consistent key-value stores have a somewhat shorter history but do fill a real need.</p> <p>These trends will merge: Extreme scale, schema-last, complex queries, even more complex inference, custom code for in-database machine learning and other bulk processing.</p> <p>We find RDF augmented with some binary types at this crossroads. This point of the design space will have to provide performance roughly on the level of today's best relational solution for workloads that fit the relational model. The added cost of schema-last and inference must come down. We are working on this. Research work such as carried out with <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x794ee48">MonetDB</a> gives clues as to how these aims can be reached.</p> <p>The separation of query language and inference is artificial. After the concepts are mature, these functions will merge and execute close to the data; there are clear evolutionary pressures in this direction.</p> <p>Benchmarks are key. Some gain can be had even from repurposing standard relational benchmarks like <a href="http://www.tpc.org/" id="link-id0x7d45c58">TPC</a>-<a href="http://dbpedia.org/resource/TPC-H" id="link-id0x45b0198">H</a>. But the TPC-H rules do not allow official reporting of such.</p> <p>Development of benchmarks for RDF, complex queries, and inference is needed. A bold challenge to the community, it should be rooted in real-life integration needs and involve high heterogeneity. A key-value store benchmark might also be conceived. A transaction benchmark like TPC-<a href="http://dbpedia.org/resource/C%2B%2B" id="link-id0x7e32178">C</a> might be the basis, maybe augmented with massive user-generated content like reviews and blogs.</p> <p>If benchmarks exist and are not too easy nor inaccessibly difficult nor too expensive to run â think of the high end TPC-C results â then TPC-style rules and processes would be quite adequate. The threshold to publish should be lowered: Everybody runs the TPC workloads internally but few publish.</p> <p>Some EC initiative for benchmarking could make sense, similar to the TREC initiative of the US government. Industry should be consulted for the specific content; possibly the answers to the present questionnaire can provide an approximate direction.</p> <p>Benchmarks should be run by software vendors on their own systems, tuned by themselves. But there should be a process of disclosure and auditing; the TPC rules give an example. Compliance should not be too expensive or time consuming. Some community development for automating these things would be a worthwhile target for EC funding.</p> </li> </ol> </li> <li> <p> <b>Usability and training</b> </p> <ol type="a" start="1"> <li> <p> <b>How difficult will it be for a developer of average competence to deploy components whose core is based on rather deep computer science? Do we all need to understand Monads and Continuations? What can be done to make it ever easier?</b> </p> <p>In the database world, huge advances in technology have taken place behind a relatively simple and stable interface: SQL. For the linked data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x7e01618">web</a>, the same will take place behind SPARQL.</p> <p>Beyond these, for example, programming with MPI with good utilization of a cluster platform for an arbitrary algorithm, is quite difficult. The casual amateur is hereby warned.</p> <p>There is no single solution. For automatic parallelization, since explicit, programmatic parallelization of things with MPI for example is very unscalable in terms of required skill, we should favor declarative and/or functional approaches.</p> <p>Developing a debugger and explanation engine for rule-based and description-logics-based inference would be an idea.</p> <p>For procedural workloads, things like Erlang may be good in cases and are not overly difficult in principle, especially if there are good debugging facilities.</p> <p>For shipping functions in a cluster or cloud, the <a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id0x43665a8">BOOM</a> (<a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id0x7718f00">Berkeley Orders Of Magnitude</a>) approach or logic programming with explicit specification of compute location seem promising, surely more flexible than map-reduce. The question is whether a <a href="http://dbpedia.org/resource/PHP" id="link-id0x7d64f68">PHP</a> developer can be made to do logic programming.</p> <p>This bridge will be crossed only with actual need and even then reluctantly. We may look at the Web 2.0 practice of sharding <a href="http://dbpedia.org/resource/MySQL" id="link-id0xbab1ae98">MySQL</a>, inconvenient as this may be, for an example. There is inertia and thus re-architecting is a constant process that is generally in reaction to facts, <i>post hoc</i>, often a point solution. One could argue that planning ahead would be smarter but by and large the world does not work so.</p> <p>One part of the answer is an infinitely-scalable SQL database that expands and shrinks in the clouds, with the usual semantics, maybe optional eventual consistency and built-in map reduce. If such a thing is inexpensive enough and syntax-level-compatible with present installed base, many developers do not have to learn very much more.</p> <p>This is maybe good for the bread-and-butter IT, but European competitiveness should not rest on this. Therefore we wish to go for bold new application types for which the client-server database application is not the model. Data-centric languages like BOOM, if they can be made very efficient and have good debugging support, are attractive there. These do require more intellectual investment but that is not a problem since the less-inquisitive part of the developer community is served by the first part of the answer.</p> </li> <li> <p> <b>How is a developer of average skills going to learn about these new advanced tools? How can we plan for excellent documentation and training, community mentoring, exchange of good practices, etc... across all EU countries?</b> </p> <p>For the most part, developers do not learn things for the sake of learning. When they have learned something and it is adequate, they stay with it for the most part and are even reluctant to engage in cross-camps interaction. The research world is often similarly insular. A new inflection in the application landscape is needed to drive learning. This inflection is provided by the <a href="https://wiki.mozilla.org/Labs/Ubiquity" id="link-id0x770df38">ubiquity</a> of mobile devices, sensor data, explicit semantics, NLP concept extraction, web of linked data, and such factors.</p> <p>RDFa is a good example of a new technique piggybacking on something everybody uses, namely HTML. These new things should, within possibility, be deployed in the usual technology stack, <a href="http://en.wikipedia.org/wiki/LAMP_%28software_bundle%29" id="link-id0x55596a8">LAMP</a> or Java. Of course these do not have to be LAMP or Java or HTML or HTTP themselves but they must manifest through these.</p> <p>A lot of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x3d5378">semantic web</a> potential can be realized within the client-server database application model, thus no fundamental re-architecting, just some new data types and queries.</p> <p>For data- or processing-intensive tasks, an on-demand hookup to cloud-based servers with Erlang and/or BOOM for programming model would be easy enough to learn and utilize.</p> <p>The question is one of providing challenges. Addressing actual challenges with these techniques will lead to maturity, documentation, examples, and training. With virtual, Europe-wide distributed teams a reality in many places, Europe-wide dissemination is no longer insurmountable.</p> <p>As the data overflow proceeds, its victims will multiply and create demand for solutions. The EC could here encourage research project use cases gaining an extended life past the end of research projects, possibly being maintained and multiplied and spun off.</p> <p>If such things could be mutated into self-sustaining service businesses with pay-per-use revenue, say through a cloud SaaS business model, still primarily leveraging an open source technology stack, we could have self-propagating and self-supporting models for exploiting advanced IT. This would create interest, and interest would drive training and dissemination.</p> <p>The problem is creating the pull.</p> </li> </ol> </li> <li> <p> <b>Challenges</b> </p> <ol type="a" start="1"> <li> <p> <b>What should be, in this domain, the equivalent of the Netflix challenge, Ansari X Prize, <a href="http://dbpedia.org/resource/Google" id="link-id0x6a6c2b0">Google</a> Lunar X Prize, etc. ... ?</b> </p> <p>The EC itself no doubt suffers from data overflow in one function or another. Unless security/secrecy prohibits, simply publishing a large data set and a description of what operations should be done on it would be a start. The more real the data, the better â reality is consistently more complex and surprising than imagination. Since many interesting problems touch on fraud detection and law enforcement, there may be some security obstacles for using these application domains as subject matters of open challenges.</p> <p>Once there is a good benchmark, as discussed above, there can be some prize money allocated for the winners, specially if the race is tight.</p> <p>The Semantic Web Challenge and the Billion Triples Challenge exist and are useful as such, but do not seem to have any huge impact.</p> <p>The incentives should be sufficient and part of the expenses arising from running for such challenges could be funded. Otherwise investing in existing business development will be more interesting to industry. Some industry participation seems necessary; we would wish academia and industry to work closer. Also, having industry supply the baseline guarantees that academia actually does further the state of the art. This is not always certain.</p> <p>If challenges are based on actual problems, whether of the EC, its member governments, or private entities, and winning the challenge may lead to a contract for supplying an actual solution, these will naturally become more interesting for consortia involving integrators, specialist software vendors, and academia. Such a model would build actual capacity to deploy leading edge technologies in production, which is sorely needed.</p> </li> <li> <p> <b>What should one do to set up such a challenge, administer, and monitor it?</b> </p> <p>The EC should probably circulate a call for actual problem scenarios involving big data. If the matter of the overflow is as dire as represented, cases should be easy to find. A few should be selected and then anonymized if needed.</p> <p>The party with the use case would benefit by having hopefully the best work on it. The contestants would benefit from having real world needs guide R&D. The EC would not have to do very much, except possibly use some money for funding the best proposals. The winner would possibly get a large account and related sales and service income. The contestants would have to be teams possibly involving many organizations; for example, development and first-line services and support could come from different companies along a systems integrator model such as is widely used in the US.</p> <p>There may be a good benchmark at the time, possibly resulting from FP7 itself. In such a case, the EC could offer a prize for winners. Details would have to be worked out case by case. Such a challenge could be repeated a few times, as benchmark-driven progress in databases or TREC for example have taken some years to reach a point of slowdown in progress.</p> <p>Administrating such an activity should not be prohibitive, as most of the expertise can be found with the stakeholders.</p> </li> </ol> </li> </ol>
VLDB 2009 TPC Workshop (3 of 5)
http://www.openlinksw.com/weblog/oerling/?date=2009-09-01#1576
2009-09-01T15:51:09Z
2009-09-01T17:32:30-04:00
<p>Michael <a href="http://dbpedia.org/resource/Michael_Stonebraker" id="link-id0x15e5efe0">Stonebraker</a> gave the keynote at the <a href="http://www.tpc.org/" id="link-id0x18cee5f0">TPC</a> workshop. His message was that the TPC, at the venerable age of 21, was already a decade late in reinventing itself. From the height of relevance at the time of the debit/credit benchmark twenty years back, it was slipping into the sunset of irrelevance unless it paid attention.</p> <p>Now we are great fans of the TPC and while we have not published results by the TPC book, we have extensively used TPC material for guiding <a href="http://dbpedia.org/resource/Program_optimization" id="link-id0x4e55368">optimization</a>, as has pretty much everybody else.</p> <p>It is true that the rules encourage unrealistic configurations. The emphasis on random access from disk that is built into the rules leads to disk configurations that are very improbable in practice, such as 1PB of disks for 3TB of <a href="http://dbpedia.org/resource/Data" id="link-id0x191cd880">data</a>, just so there are enough disk arms in parallel. Stonebraker also pointed out that replication and failover were ubiquitous in real life and that roll forward from logs was unrealistic as a recovery model since it took so long. Benchmarks should therefore include replication.</p> <p>Further, Stonebraker challenged the TPC to go for the new frontier, which he described as the huge data sets in science and on big web sites. Scientists, the ones who would save our planet from the diverse ills confronting it, do not like relational databases. They avoid them when can. They want arrays for physics, and graphs for biology and chemistry. <a href="http://dbpedia.org/resource/MapReduce" id="link-id0x53f6040">MapReduce</a> is eating database's lunch; what will you do about this?</p> <p>I later suggested incorporating an <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x18902070">RDF</a> <a href="http://dbpedia.org/resource/Metadata" id="link-id0x3990af8">metadata</a> benchmark into the TPC suite. We'll see about this; we'll first have to come up with a suitable one. There is a great deal of pressure for making good RDF benchmarks but this is not yet in the center of the mainstream that TPC tends to cover.</p> <p>TPC's own talk was about the life cycle of benchmarks. A benchmark begins a bit ahead of the mainstream, with a problem that is difficult but not so difficult as to be uncommon. When the solution to this problem becomes commonplace, the benchmark's relevance gradually drops.</p> <p>There was a talk on robustness of query plans which was well to the point. Indeed, there are performance cliffs at certain points; for example, when passing from memory-only to disk-pageable data structures, or when switching from indexed access to table scans, or from loop to hash joins. Quite so. The analysis I really would have liked to see would have been one of what happens when passing from single server to a cluster, and from local joins to cross-partition ones. Also contrasting of <a href="http://dbpedia.org/resource/Cache" id="link-id0x1942aca8">cache</a> fusion and partitioning. We have our own data and experience but we find we don't have time to measure all the other systems.</p> <p>Anyway it is good to raise the question of smooth and predictable performance.</p>
Updated hardware improves LUBM 8000 load rate in Virtuoso 6
http://www.openlinksw.com/weblog/oerling/?date=2009-08-14#1568
2009-08-14T19:01:30Z
2009-08-15T15:27:25-04:00
<p>We repeated the <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1562" id="link-id173d3068">earlier LUBM 8000 experiment</a> on a newer machine, with 2 x Xeon 5520 and 72G 1333MHz memory, and once again with the 2 machines as a networked cluster. Otherwise the settings were the same.</p> <p>The load rate is now 160,739 triples-per-second.</p> <table> <tr> <th></th> <td> </td> <th align="center"><a href="http://virtuoso.openlinksw.com" id="link-id0x199b9740">Virtuoso</a> 6 <br /> (previous run)</th> <td> </td> <th align="center">Virtuoso 6 <br /> (new run)</th> <td> </td> <th align="center">Virtuoso 6 <br /> (newest run)</th> </tr> <tr> <td align="left">blades</td> <td> </td> <td align="center">1 </td> <td> </td> <td align="center">1 </td> <td> </td> <td align="center">2</td> </tr> <tr> <td align="left">processors</td> <td> </td> <td align="center">2 x Xeon 5410</td> <td> </td> <td align="center">2 x Xeon 5520</td> <td> </td> <td align="center"> 2 x Xeon 5520 <br />+ <br />2 x Xeon 5410 <br />with 1x1GigE <br />interconnect </td> </tr> <tr> <td align="left">memory</td> <td> </td> <td align="center"> 16G 667 MHz</td> <td> </td> <td align="center">72G 1333 MHz</td> <td> </td> <td align="center">72G 1333 MHz <br />+ <br /> 16G 667 MHz <br /> respectively</td> </tr> <tr> <td align="left">reported load rate<br />triples-per-second</td> <td> </td> <td align="center"> 110,532 </td> <td> </td> <td align="center"> 160,739 </td> <td> </td> <td align="center"> 214,188 </td> </tr> </table> <p>Again, if others talk about loading LUBM, so must we. Otherwise, this metric is rather uninteresting.</p>
Single Virtuoso host loads 110,500 triples-per-second on LUBM 8000
http://www.openlinksw.com/weblog/oerling/?date=2009-06-29#1562
2009-06-29T16:12:34Z
2009-08-15T16:06:42.000001-04:00
<p>LUBM load speed still seems to be a metric that is quoted in comparisons of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id142df6e8">RDF</a> stores. Consequently, we too measured the load time of LUBM 8000, 1,068-million triples, on the newest <a href="http://virtuoso.openlinksw.com" id="link-id1389dfa0">Virtuoso</a>.</p> <p>The real time for the load was 161m 3s. The rate was 110,532 triples-per-second. The hardware was one machine with 2 x Xeon 5410 (quad core, 2.33 GHz) and 16G 667 MHz RAM. The software was Virtuoso 6 Cluster, configured into 8 partitions (processes) — one partition per CPU core. Each partition had its database striped over 6 disks total; the 6 disks on the system were shared between the 8 database processes.</p> <p>The load was done on 8 streams, one per server process. At the beginning of the load, the CPU usage was 740% with no disk; at the end, it was around 700% with 25% disk wait. 100% counts here for one CPU core or one disk being constantly busy.</p> <p>The RDF store was configured with the default two indices over quads, these being GSPO and OGPS. Text indexing of literals was not enabled. No materialization of entailed triples was made.</p> <p>We think that LUBM loading is not a realistic benchmark for the world but since other people publish such numbers, so do we.</p>
Single Virtuoso host loads 110,500 triples-per-second on LUBM 8000
http://www.openlinksw.com/weblog/oerling/?date=2009-06-29#1562
2009-06-29T16:12:34Z
2009-08-15T16:06:42.000001-04:00
<p>LUBM load speed still seems to be a metric that is quoted in comparisons of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id142df6e8">RDF</a> stores. Consequently, we too measured the load time of LUBM 8000, 1,068-million triples, on the newest <a href="http://virtuoso.openlinksw.com" id="link-id1389dfa0">Virtuoso</a>.</p> <p>The real time for the load was 161m 3s. The rate was 110,532 triples-per-second. The hardware was one machine with 2 x Xeon 5410 (quad core, 2.33 GHz) and 16G 667 MHz RAM. The software was Virtuoso 6 Cluster, configured into 8 partitions (processes) — one partition per CPU core. Each partition had its database striped over 6 disks total; the 6 disks on the system were shared between the 8 database processes.</p> <p>The load was done on 8 streams, one per server process. At the beginning of the load, the CPU usage was 740% with no disk; at the end, it was around 700% with 25% disk wait. 100% counts here for one CPU core or one disk being constantly busy.</p> <p>The RDF store was configured with the default two indices over quads, these being GSPO and OGPS. Text indexing of literals was not enabled. No materialization of entailed triples was made.</p> <p>We think that LUBM loading is not a realistic benchmark for the world but since other people publish such numbers, so do we.</p>
Virtuoso RDF: A Getting Started Guide for the Developer
http://www.openlinksw.com/weblog/oerling/?date=2008-12-17#1504
2008-12-17T12:31:34Z
2008-12-17T12:41:21.000001-05:00
<p>It is a long standing promise of mine to dispel the false impression that using <a href="http://virtuoso.openlinksw.com/" id="link-id113506d0">Virtuoso</a> to work with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id115d9528">RDF</a> is complicated.</p> <p>The purpose of this presentation is to show a programmer how to put RDF into Virtuoso and how to query it. This is done programmatically, with no confusing user interfaces.</p> <p>You should have a Virtuoso Open Source tree built and installed. We will look at the LUBM benchmark demo that comes with the package. All you need is a Unix shell. Running the shell under emacs (<code>m-x shell</code>) is the best. But the open source <code>isql</code> utility should have command line editing also. The emacs shell is however convenient for cutting and pasting things between shell and files.</p> <p>To get started, cd into <code>binsrc/tests/lubm</code>.</p> <p>To verify that this works, you can do </p> <blockquote> <pre>./test_server.sh virtuoso-t</pre></blockquote> <p>This will test the server with the LUBM queries. This should report 45 tests passed. After this we will do the tests step-by-step.</p> <h2>Loading the <a href="http://dbpedia.org/resource/Data" id="link-id10f7bd90">Data</a> </h2> <p>The file <code>lubm-load.sql</code> contains the commands for loading the LUBM single university qualification database.</p> <p>The data files themselves are in <code>lubm_8000</code>, 15 files in RDFXML.</p> <p>There is also a little ontology called <code>inf.nt</code>. This declares the subclass and subproperty relations used in the benchmark.</p> <p>So now let's go through this procedure.</p> <p>Start the server:</p> <blockquote> <pre>$ virtuoso-t -f & </pre></blockquote> <p>This starts the server in foreground mode, and puts it in the background of the shell.</p> <p>Now we connect to it with the isql utility.</p> <blockquote> <pre>$ isql 1111 dba dba </pre></blockquote> <p>This gives a <code>SQL></code> prompt. The default username and password are both <code>dba</code>.</p> <p>When a command is <a href="http://dbpedia.org/resource/SQL" id="link-id1176ce70">SQL</a>, it is entered directly. If it is <a href="http://dbpedia.org/resource/SPARQL" id="link-id156df468">SPARQL</a>, it is prefixed with the keyword <code>sparql</code>. This is how all the SQL clients work. Any SQL client, such as any <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id152d0a00">ODBC</a> or <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id157ad6a0">JDBC</a> application, can use SPARQL if the SQL string starts with this keyword.</p> <p>The <code>lubm-load.sql</code> file is quite self-explanatory. It begins with defining an SQL procedure that calls the RDF/XML load function, <code>DB..RDF_LOAD_RDFXML</code>, for each file in a directory.</p> <p>Next it calls this function for the <code>lubm_8000</code> directory under the server's working directory.</p> <blockquote> <pre>sparql CLEAR GRAPH <lubm>; sparql CLEAR GRAPH <inf>; load_lubm ( server_root() || '/lubm_8000/' ); </pre></blockquote> <p>Then it verifies that the right number of triples is found in the <lubm> graph.</p> <blockquote> <pre>sparql SELECT COUNT(*) FROM <lubm> WHERE { ?x ?y ?z } ; </pre></blockquote> <p>The echo commands below this are interpreted by the isql utility, and produce output to show whether the test was passed. They can be ignored for now.</p> <p>Then it adds some implied <code>subOrganizationOf</code> triples. This is part of setting up the LUBM test database.</p> <blockquote> <pre>sparql PREFIX ub: <http://www.lehigh.edu/~zhp2/2004/0401/univ-bench.owl#> INSERT INTO GRAPH <lubm> { ?x ub:subOrganizationOf ?z } FROM <lubm> WHERE { ?x ub:subOrganizationOf ?y . ?y ub:subOrganizationOf ?z . }; </pre></blockquote> <p>Then it loads the ontology file, <code>inf.nt</code>, using the Turtle load function, <code>DB.DBA.TTLP</code>. The arguments of the function are the text to load, the default namespace prefix, and the <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id15835550">URI</a> of the target graph.</p> <blockquote> <pre>DB.DBA.TTLP ( file_to_string ( 'inf.nt' ), 'http://www.lehigh.edu/~zhp2/2004/0401/univ-bench.owl', 'inf' ) ; sparql SELECT COUNT(*) FROM <inf> WHERE { ?x ?y ?z } ; </pre></blockquote> <p>Then we declare that the triples in the <code><inf></code> graph can be used for inference at run time. To enable this, a SPARQL query will declare that it uses the <code>'inft'</code> rule set. Otherwise this has no effect.</p> <blockquote> <pre>rdfs_rule_set ('inft', 'inf'); </pre></blockquote> <p>This is just a log checkpoint to finalize the work and truncate the transaction log. The server would also eventually do this in its own time.</p> <blockquote> <pre>checkpoint; </pre></blockquote> <p>Now we are ready for querying.</p> <h2>Querying the Data</h2> <p>The queries are given in 3 different versions: The first file, <code>lubm.sql</code>, has the queries with most inference open coded as <code>UNIONs</code>. The second file, <code>lubm-inf.sql</code>, has the inference performed at run time using the ontology <a href="http://dbpedia.org/resource/Information" id="link-id1109faf0">information</a> in the <code><inf></code> graph we just loaded. The last, <code>lubm-phys.sql</code>, relies on having the entailed triples physically present in the <code><lubm></code> graph. These entailed triples are inserted by the SPARUL commands in the <code>lubm-cp.sql</code> file.</p> <p>If you wish to run all the commands in a SQL file, you can type <code>load <filename>;</code> (e.g., <code>load lubm-cp.sql;</code>) at the <code>SQL></code> prompt. If you wish to try individual statements, you can paste them to the command line.</p> <p>For example: </p> <blockquote> <pre>SQL> sparql PREFIX ub: <http://www.lehigh.edu/~zhp2/2004/0401/univ-bench.owl#> SELECT * FROM <lubm> WHERE { ?x a ub:Publication . ?x ub:publicationAuthor <http://www.Department0.University0.edu/AssistantProfessor0> }; VARCHAR _______________________________________________________________________ http://www.Department0.University0.edu/AssistantProfessor0/Publication0 http://www.Department0.University0.edu/AssistantProfessor0/Publication1 http://www.Department0.University0.edu/AssistantProfessor0/Publication2 http://www.Department0.University0.edu/AssistantProfessor0/Publication3 http://www.Department0.University0.edu/AssistantProfessor0/Publication4 http://www.Department0.University0.edu/AssistantProfessor0/Publication5 6 Rows. -- 4 msec. </pre></blockquote> <p>To stop the server, simply type <code>shutdown;</code> at the <code>SQL></code> prompt.</p> <p>If you wish to use a <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id11384668">SPARQL protocol</a> end point, just enable the HTTP listener. This is done by adding a stanza like â</p> <blockquote> <pre>[HTTPServer] ServerPort = 8421 ServerRoot = . ServerThreads = 2 </pre></blockquote> <p>â to the end of the <code>virtuoso.ini</code> file in the <code>lubm</code> directory. Then shutdown and restart (type <code>shutdown;</code> at the <code>SQL></code> prompt and then <code>virtuoso-t -f &</code> at the shell prompt).</p> <p>Now you can connect to the end point with a web browser. The <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id113d02d8">URL</a> is <code>http://localhost:8421/sparql</code>. Without parameters, this will show a human readable form. With parameters, this will execute SPARQL.</p> <p>We have shown how to load and query RDF with Virtuoso using the most basic SQL tools. Next you can access RDF from, for example, <a href="http://dbpedia.org/resource/PHP" id="link-id142d0ba0">PHP</a>, using the PHP ODBC interface.</p> <p>To see how to use <a href="http://jena.sourceforge.net/" id="link-id117074f0">Jena</a> or <a href="http://sourceforge.net/projects/sesame/" id="link-id1103c9b0">Sesame</a> with Virtuoso, look at <a href="http://docs.openlinksw.com/virtuoso/rdfnativestorageproviders.html" id="link-id15488ce8">Native RDF Storage Providers</a>. To see how RDF data types are supported, see <a href="http://docs.openlinksw.com/virtuoso/VirtuosoDriverJDBC.html#jdbcrdf" id="link-id15784a40">Extension datatype for RDF</a> </p> <p>To work with large volumes of data, you must add memory to the configuration file and use the row-autocommit mode, i.e., do <code>log_enable (2);</code> before the load command. Otherwise Virtuoso will do the entire load as a single transaction, and will run out of rollback space. See <a href="http://docs.openlinksw.com/virtuoso/" id="link-id111410f0">documentation</a> for more.</p>
Virtuoso RDF: A Getting Started Guide for the Developer
http://www.openlinksw.com/weblog/oerling/?date=2008-12-17#1504
2008-12-17T12:31:34Z
2008-12-17T12:41:21.000001-05:00
<p>It is a long standing promise of mine to dispel the false impression that using <a href="http://virtuoso.openlinksw.com/" id="link-id113506d0">Virtuoso</a> to work with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id115d9528">RDF</a> is complicated.</p> <p>The purpose of this presentation is to show a programmer how to put RDF into Virtuoso and how to query it. This is done programmatically, with no confusing user interfaces.</p> <p>You should have a Virtuoso Open Source tree built and installed. We will look at the LUBM benchmark demo that comes with the package. All you need is a Unix shell. Running the shell under emacs (<code>m-x shell</code>) is the best. But the open source <code>isql</code> utility should have command line editing also. The emacs shell is however convenient for cutting and pasting things between shell and files.</p> <p>To get started, cd into <code>binsrc/tests/lubm</code>.</p> <p>To verify that this works, you can do </p> <blockquote> <pre>./test_server.sh virtuoso-t</pre></blockquote> <p>This will test the server with the LUBM queries. This should report 45 tests passed. After this we will do the tests step-by-step.</p> <h2>Loading the <a href="http://dbpedia.org/resource/Data" id="link-id10f7bd90">Data</a> </h2> <p>The file <code>lubm-load.sql</code> contains the commands for loading the LUBM single university qualification database.</p> <p>The data files themselves are in <code>lubm_8000</code>, 15 files in RDFXML.</p> <p>There is also a little ontology called <code>inf.nt</code>. This declares the subclass and subproperty relations used in the benchmark.</p> <p>So now let's go through this procedure.</p> <p>Start the server:</p> <blockquote> <pre>$ virtuoso-t -f & </pre></blockquote> <p>This starts the server in foreground mode, and puts it in the background of the shell.</p> <p>Now we connect to it with the isql utility.</p> <blockquote> <pre>$ isql 1111 dba dba </pre></blockquote> <p>This gives a <code>SQL></code> prompt. The default username and password are both <code>dba</code>.</p> <p>When a command is <a href="http://dbpedia.org/resource/SQL" id="link-id1176ce70">SQL</a>, it is entered directly. If it is <a href="http://dbpedia.org/resource/SPARQL" id="link-id156df468">SPARQL</a>, it is prefixed with the keyword <code>sparql</code>. This is how all the SQL clients work. Any SQL client, such as any <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id152d0a00">ODBC</a> or <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id157ad6a0">JDBC</a> application, can use SPARQL if the SQL string starts with this keyword.</p> <p>The <code>lubm-load.sql</code> file is quite self-explanatory. It begins with defining an SQL procedure that calls the RDF/XML load function, <code>DB..RDF_LOAD_RDFXML</code>, for each file in a directory.</p> <p>Next it calls this function for the <code>lubm_8000</code> directory under the server's working directory.</p> <blockquote> <pre>sparql CLEAR GRAPH <lubm>; sparql CLEAR GRAPH <inf>; load_lubm ( server_root() || '/lubm_8000/' ); </pre></blockquote> <p>Then it verifies that the right number of triples is found in the <lubm> graph.</p> <blockquote> <pre>sparql SELECT COUNT(*) FROM <lubm> WHERE { ?x ?y ?z } ; </pre></blockquote> <p>The echo commands below this are interpreted by the isql utility, and produce output to show whether the test was passed. They can be ignored for now.</p> <p>Then it adds some implied <code>subOrganizationOf</code> triples. This is part of setting up the LUBM test database.</p> <blockquote> <pre>sparql PREFIX ub: <http://www.lehigh.edu/~zhp2/2004/0401/univ-bench.owl#> INSERT INTO GRAPH <lubm> { ?x ub:subOrganizationOf ?z } FROM <lubm> WHERE { ?x ub:subOrganizationOf ?y . ?y ub:subOrganizationOf ?z . }; </pre></blockquote> <p>Then it loads the ontology file, <code>inf.nt</code>, using the Turtle load function, <code>DB.DBA.TTLP</code>. The arguments of the function are the text to load, the default namespace prefix, and the <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id15835550">URI</a> of the target graph.</p> <blockquote> <pre>DB.DBA.TTLP ( file_to_string ( 'inf.nt' ), 'http://www.lehigh.edu/~zhp2/2004/0401/univ-bench.owl', 'inf' ) ; sparql SELECT COUNT(*) FROM <inf> WHERE { ?x ?y ?z } ; </pre></blockquote> <p>Then we declare that the triples in the <code><inf></code> graph can be used for inference at run time. To enable this, a SPARQL query will declare that it uses the <code>'inft'</code> rule set. Otherwise this has no effect.</p> <blockquote> <pre>rdfs_rule_set ('inft', 'inf'); </pre></blockquote> <p>This is just a log checkpoint to finalize the work and truncate the transaction log. The server would also eventually do this in its own time.</p> <blockquote> <pre>checkpoint; </pre></blockquote> <p>Now we are ready for querying.</p> <h2>Querying the Data</h2> <p>The queries are given in 3 different versions: The first file, <code>lubm.sql</code>, has the queries with most inference open coded as <code>UNIONs</code>. The second file, <code>lubm-inf.sql</code>, has the inference performed at run time using the ontology <a href="http://dbpedia.org/resource/Information" id="link-id1109faf0">information</a> in the <code><inf></code> graph we just loaded. The last, <code>lubm-phys.sql</code>, relies on having the entailed triples physically present in the <code><lubm></code> graph. These entailed triples are inserted by the SPARUL commands in the <code>lubm-cp.sql</code> file.</p> <p>If you wish to run all the commands in a SQL file, you can type <code>load <filename>;</code> (e.g., <code>load lubm-cp.sql;</code>) at the <code>SQL></code> prompt. If you wish to try individual statements, you can paste them to the command line.</p> <p>For example: </p> <blockquote> <pre>SQL> sparql PREFIX ub: <http://www.lehigh.edu/~zhp2/2004/0401/univ-bench.owl#> SELECT * FROM <lubm> WHERE { ?x a ub:Publication . ?x ub:publicationAuthor <http://www.Department0.University0.edu/AssistantProfessor0> }; VARCHAR _______________________________________________________________________ http://www.Department0.University0.edu/AssistantProfessor0/Publication0 http://www.Department0.University0.edu/AssistantProfessor0/Publication1 http://www.Department0.University0.edu/AssistantProfessor0/Publication2 http://www.Department0.University0.edu/AssistantProfessor0/Publication3 http://www.Department0.University0.edu/AssistantProfessor0/Publication4 http://www.Department0.University0.edu/AssistantProfessor0/Publication5 6 Rows. -- 4 msec. </pre></blockquote> <p>To stop the server, simply type <code>shutdown;</code> at the <code>SQL></code> prompt.</p> <p>If you wish to use a <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id11384668">SPARQL protocol</a> end point, just enable the HTTP listener. This is done by adding a stanza like â</p> <blockquote> <pre>[HTTPServer] ServerPort = 8421 ServerRoot = . ServerThreads = 2 </pre></blockquote> <p>â to the end of the <code>virtuoso.ini</code> file in the <code>lubm</code> directory. Then shutdown and restart (type <code>shutdown;</code> at the <code>SQL></code> prompt and then <code>virtuoso-t -f &</code> at the shell prompt).</p> <p>Now you can connect to the end point with a web browser. The <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id113d02d8">URL</a> is <code>http://localhost:8421/sparql</code>. Without parameters, this will show a human readable form. With parameters, this will execute SPARQL.</p> <p>We have shown how to load and query RDF with Virtuoso using the most basic SQL tools. Next you can access RDF from, for example, <a href="http://dbpedia.org/resource/PHP" id="link-id142d0ba0">PHP</a>, using the PHP ODBC interface.</p> <p>To see how to use <a href="http://jena.sourceforge.net/" id="link-id117074f0">Jena</a> or <a href="http://sourceforge.net/projects/sesame/" id="link-id1103c9b0">Sesame</a> with Virtuoso, look at <a href="http://docs.openlinksw.com/virtuoso/rdfnativestorageproviders.html" id="link-id15488ce8">Native RDF Storage Providers</a>. To see how RDF data types are supported, see <a href="http://docs.openlinksw.com/virtuoso/VirtuosoDriverJDBC.html#jdbcrdf" id="link-id15784a40">Extension datatype for RDF</a> </p> <p>To work with large volumes of data, you must add memory to the configuration file and use the row-autocommit mode, i.e., do <code>log_enable (2);</code> before the load command. Otherwise Virtuoso will do the entire load as a single transaction, and will run out of rollback space. See <a href="http://docs.openlinksw.com/virtuoso/" id="link-id111410f0">documentation</a> for more.</p>
Virtuoso Vs. MySQL: Setting the Berlin Record Straight (update 2)
http://www.openlinksw.com/weblog/oerling/?date=2008-11-20#1484
2008-11-20T11:06:11Z
2008-11-24T10:15:05-05:00
<p>In the context of the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0xa5314d8">Berlin SPARQL Benchmark</a>, I have repeatedly written about measurement procedures and steady state. The point is that the numbers at larger scales are unreliable due to cache behavior if one is not careful about measurement and does not have adequate warmup. Thus it came to pass that one cut of the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x18482c20">BSBM</a> paper had 3 seconds for <a href="http://dbpedia.org/resource/MySQL" id="link-id0xb8c54de8">MySQL</a> and 100 for <a href="http://virtuoso.openlinksw.com" id="link-id0x189b2210">Virtuoso</a>, basically through ignoring cache effects.</p> <p>So we decided to do it ourselves.</p> <p>The score is (updated with revised <code>innodb_buffer_pool_size</code> setting, based on advice noted down below):</p> <table border="1" cellspacing="2" cellpadding="5"> <tr> <th>n-clients</th> <th>Virtuoso</th> <th>MySQL <br /> (with increased buffer pool size)</th> <th>MySQL <br /> (with default buffer poll size)</th> </tr> <tr align="right"> <td>1</td> <td> 41,161.33</td> <td> 27,023.11 </td> <td> 12,171.41</td> </tr> <tr align="right"> <td>4</td> <td> 127,918.30</td> <td> (pending) </td> <td> 37,566.82</td> </tr> <tr align="right"> <td>8</td> <td> 218,162.29 </td> <td> 105,524.23 </td> <td> 51,104.39 </td> </tr> <tr align="right"> <td>16</td> <td> 214,763.58 </td> <td> 98,852.42 </td> <td> 47,589.18 </td> </tr> </table> <p>The metric is the query mixes per hour from the BSBM test driver output. For the interested, the complete output is <a href="http://www.openlinksw.com/weblog/oerling/texts/bsbmres.txt" id="link-id1119f770">here</a>.</p> <p>The benchmark is pure <a href="http://dbpedia.org/resource/SQL" id="link-id0x5257718">SQL</a>, nothing to do with <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xb8c463e0">SPARQL</a> or <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x16e68d50">RDF</a>.</p> <p>The hardware is 2 x Xeon 5345 (2 x quad core, 2.33 GHz), 16 G RAM. The OS is 64-bit Debian Linux.</p> <p>The benchmark was run at a scale of 200,000. Each run had 2000 warm-up query mixes and 500 measured query mixes, which gives steady state, eliminating any effects of OS disk cache and the like. Both databases were configured to use 8G for disk cache. The test effectively runs from memory. We ran an analyze table on each MySQL table but noticed that this had no effect. Virtuoso does the stats sampling on the go; possibly MySQL also since the explicit stats did not make any difference. The MySQL tables were served by the InnoDB engine. MySQL appears to cache results of queries in some cases. This was not apparent in the tests.</p> <p>The versions are 5.09 for Virtuoso and 5.1.29 for MySQL. You can download and examine --</p> <ul> <li> <a href="http://www.openlinksw.com/weblog/oerling/texts/virtuoso.ini" id="link-id14fe17f0">Virtuoso configuration file</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/texts/my.cnf" id="link-id116fe490">MySQL configuration file</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/texts/create_tables_and_rdf_view.sql" id="link-id14ce9268">Table definitions & RDF views</a> </li> <li> <a href="http://www.openlinksw.com/weblog/oerling/texts/mysqlinx.sql" id="link-id1535e298">Indexes on MySQL tables</a> </li> </ul> <p> <strike>MySQL ought to do better. We suspect that here, just as in the TPC-D experiment we made way back, the query plans are not quite right. Also we rarely saw over 300% CPU utilization for MySQL. It is possible there is a config parameter that affects this. The public is invited to tell us about such.</strike> </p> <p> <b>Update:</b> </p> <p>Andreas Schultz of the BSBM team advised us to increase the <code>innodb_buffer_pool_size</code> setting in the MySQL config. We did and it produced some improvement. Indeed, this is more like it, as we now see CPU utilization around 700% instead of the 300% in the previously published run, which rendered it suspect. Also, our experiments with TPC-D led us to expect better. We ran these things a few times so as to have warm cache.</p> <p>On the first run, we noticed that the Innodb warm up time was somewhere well in excess of 2000 query mixes. Another time, we should make a graph of throughput as a function of time for both MySQL and Virtuoso. We recently made a greedy prefetch hack that should give us some mileage there. For the next BSBM, all we can advise is to run larger scale system for half an hour first and then measure and then measure again. If the second measurement is the same as the first then it is good.</p> <p>As always, since MySQL is not our specialty, we confidently invite the public to tell us how to make it run faster. So, unless something more turns up, our next trial is a revisit of <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x122eaa00">TPC-H</a>.</p>
ISWC 2008: Some Questions
http://www.openlinksw.com/weblog/oerling/?date=2008-11-04#1479
2008-11-04T15:54:42Z
2008-11-04T14:36:50.000010-05:00
<h2>Inference: Is it always forward chaining?</h2> <p>We got a number of questions about <a href="http://virtuoso.openlinksw.com" id="link-id0x131604a8">Virtuoso</a>'s inference support. It seems that we are the odd one out, as we do not take it for granted that inference ought to consist of materializing entailment.</p> <p>Firstly, of course one can materialize all one wants with Virtuoso. The simplest way to do this is using SPARUL. With the recent transitivity extensions to <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1422f910">SPARQL</a>, it is also easy to materialize implications of transitivity with a single statement. Our point is that for trivial entailment such as subclass, sub-property, single transitive property, and <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x145894a8">owl</a>:sameAs, we do not require materialization, as we can resolve these at run time also, with backward-chaining built into the engine.</p> <p>For more complex situations, one needs to materialize the entailment. At the present time, we know how to generalize our transitive feature to run arbitrary backward-chaining rules, including recursive ones. We could have a sort of Datalog backward-chaining embedded in our <a href="http://dbpedia.org/resource/SQL" id="link-id0x1458a288">SQL</a>/SPARQL and could run this with good parallelism, as the transitive feature already works with clustering and partitioning without dying of message latency. Exactly when and how we do this will be seen. Even if users want entailment to be materialized, such a rule system could be used for producing the materialization at good speed.</p> <p>We had a word with <a href="http://web.comlab.ox.ac.uk/people/Ian.Horrocks/" id="link-id117c99d0">Ian Horrocks</a> on the question. He noted that it is often naive on behalf of the community to tend to equate description of semantics with description of algorithm. The <a href="http://dbpedia.org/resource/Data" id="link-id0x14cf0b18">data</a> need not always be blown up.</p> <p>The advantage of not always materializing is that the working set stays better. Once the working set is no longer in memory, response times jump disproportionately. Also, if the data changes or is retracted or is unreliable, one can end up doing a lot of extra work with materialization. Consider the effect of one malicious sameAs statement. This can lead to a lot of effects that are hard to retract. On the other hand, if running in memory with static data such as the LUBM benchmark, the queries run some 20% faster if entailment subclasses and sub-properties are materialized rather than done at run time.</p> <h2>Genetic Algorithms for SPARQL?</h2> <p>Our compliments for the wildest idea of the conference go to <a href="http://www.eyaloren.org/" id="link-id1a203af8">Eyal Oren</a>, <a href="http://www.few.vu.nl/~cgueret/" id="link-id16208758">Christophe Guéret</a>, and <a href="http://www.few.vu.nl/~schlobac/" id="link-id111923e0">Stefan Schlobach</a>, <i>et al</i>, for their <a href="http://www.informatik.uni-trier.de/~ley/db/conf/semweb/iswc2008.html#OrenGS08" id="link-id11793540">paper on using genetic algorithms for guessing how variables in a SPARQL query ought to be instantiated</a>. Prisoners of our "conventional wisdom" as we are, this might never have occurred to us.</p> <h2>Schema Last?</h2> <p>It is interesting to see how the industry comes to the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1154c1b0">semantic web</a> conferences talking about schema last while at the same time the traditional semantic web people stress enforcing schema constraints and making more predictably performing and database friendlier logics. So do the extremes converge.</p> <p>There is a point to schema last. <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x14c6a930">RDF</a> is very good for getting a view of ad hoc or unknown data. One can just load and look at what there is. Also, additions of unforeseen optional properties or relations to the schema are easy and efficient. However, it seems that a really high traffic online application would always benefit from having some application specific data structures. Such could also save considerably in hardware.</p> <p>It is not a sharp divide between RDF and relational application oriented representation. We have the capabilities in our RDB to RDF mapping. We just need to show this and have SPARUL and data loading</p>
ISWC 2008: Some Questions
http://www.openlinksw.com/weblog/oerling/?date=2008-11-04#1479
2008-11-04T15:54:42Z
2008-11-04T14:36:50.000010-05:00
<h2>Inference: Is it always forward chaining?</h2> <p>We got a number of questions about <a href="http://virtuoso.openlinksw.com" id="link-id0x131604a8">Virtuoso</a>'s inference support. It seems that we are the odd one out, as we do not take it for granted that inference ought to consist of materializing entailment.</p> <p>Firstly, of course one can materialize all one wants with Virtuoso. The simplest way to do this is using SPARUL. With the recent transitivity extensions to <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1422f910">SPARQL</a>, it is also easy to materialize implications of transitivity with a single statement. Our point is that for trivial entailment such as subclass, sub-property, single transitive property, and <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x145894a8">owl</a>:sameAs, we do not require materialization, as we can resolve these at run time also, with backward-chaining built into the engine.</p> <p>For more complex situations, one needs to materialize the entailment. At the present time, we know how to generalize our transitive feature to run arbitrary backward-chaining rules, including recursive ones. We could have a sort of Datalog backward-chaining embedded in our <a href="http://dbpedia.org/resource/SQL" id="link-id0x1458a288">SQL</a>/SPARQL and could run this with good parallelism, as the transitive feature already works with clustering and partitioning without dying of message latency. Exactly when and how we do this will be seen. Even if users want entailment to be materialized, such a rule system could be used for producing the materialization at good speed.</p> <p>We had a word with <a href="http://web.comlab.ox.ac.uk/people/Ian.Horrocks/" id="link-id117c99d0">Ian Horrocks</a> on the question. He noted that it is often naive on behalf of the community to tend to equate description of semantics with description of algorithm. The <a href="http://dbpedia.org/resource/Data" id="link-id0x14cf0b18">data</a> need not always be blown up.</p> <p>The advantage of not always materializing is that the working set stays better. Once the working set is no longer in memory, response times jump disproportionately. Also, if the data changes or is retracted or is unreliable, one can end up doing a lot of extra work with materialization. Consider the effect of one malicious sameAs statement. This can lead to a lot of effects that are hard to retract. On the other hand, if running in memory with static data such as the LUBM benchmark, the queries run some 20% faster if entailment subclasses and sub-properties are materialized rather than done at run time.</p> <h2>Genetic Algorithms for SPARQL?</h2> <p>Our compliments for the wildest idea of the conference go to <a href="http://www.eyaloren.org/" id="link-id1a203af8">Eyal Oren</a>, <a href="http://www.few.vu.nl/~cgueret/" id="link-id16208758">Christophe Guéret</a>, and <a href="http://www.few.vu.nl/~schlobac/" id="link-id111923e0">Stefan Schlobach</a>, <i>et al</i>, for their <a href="http://www.informatik.uni-trier.de/~ley/db/conf/semweb/iswc2008.html#OrenGS08" id="link-id11793540">paper on using genetic algorithms for guessing how variables in a SPARQL query ought to be instantiated</a>. Prisoners of our "conventional wisdom" as we are, this might never have occurred to us.</p> <h2>Schema Last?</h2> <p>It is interesting to see how the industry comes to the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1154c1b0">semantic web</a> conferences talking about schema last while at the same time the traditional semantic web people stress enforcing schema constraints and making more predictably performing and database friendlier logics. So do the extremes converge.</p> <p>There is a point to schema last. <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x14c6a930">RDF</a> is very good for getting a view of ad hoc or unknown data. One can just load and look at what there is. Also, additions of unforeseen optional properties or relations to the schema are easy and efficient. However, it seems that a really high traffic online application would always benefit from having some application specific data structures. Such could also save considerably in hardware.</p> <p>It is not a sharp divide between RDF and relational application oriented representation. We have the capabilities in our RDB to RDF mapping. We just need to show this and have SPARUL and data loading</p>
ISWC 2008: The Scalable Knowledge Systems Workshop
http://www.openlinksw.com/weblog/oerling/?date=2008-11-03#1471
2008-11-03T13:16:47Z
2008-11-03T12:33:49-05:00
<p>Mike Dean of <a href="http://dbpedia.org/resource/BBN_Technologies" id="link-id0x21d04768">BBN Technologies</a> opened the Scalable <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x22348c58">Knowledge</a> Systems Workshop with an invited talk. He reminded us of the facts of nature as concern the cost of distributed computing and running out of space for the working set. Developers in the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x22570328">semantic web</a> field deplorably often ignore these facts, or alternatively recognize them and admit that they are unbeatable, that one just can't join across partitions.</p> <p>I gave a talk about the <a href="http://virtuoso.openlinksw.com" id="link-id0x23f313f0">Virtuoso</a> Cluster edition, wherein I repeated essentially the same ground facts as Mike and outlined how we (in spite of these) profit from distributed memory multiprocessing. To those not intimate with these questions, let me affirm that deriving benefit from threading in a symmetric multiprocessor box, let alone a cluster connected by a network, totally depends on having many relatively long running things going at a time and blocking as seldom as possible.</p> <p>Further, Mike Dean talked about <a href="http://www.asio.bbn.com/" id="link-id0x1d74c108">ASIO</a>, the BBN suite of semantic web tools. His most challenging statement was about the storage engine, a network-database-inspired triple-store using memory-mapped files. </p> <p>Will the <a href="http://dbpedia.org/resource/CODASYL" id="link-id0x1f8ee860">CODASYL</a> days come back, and will the linked list on disk be the way to store triples/quads? I would say that this will have, especially with a memory-mapped file, probably a better best-case as a B-tree but that this also will be less predictable with fragmentation. With Virtuoso, using a B-tree index, we see about 20-30% of CPU time spent on index lookup when running LUBM queries. With a disk-based memory-mapped linked-list storage, we would see some improvements in this while getting hit probably worse than now in the case of fragmentation. Plus compaction on the fly would not be nearly as easy and surely far less local, if there were pointers between pages. So it is my intuition that trees are a safer bet with varying workloads while linked lists can be faster in a query-dominated in-memory situation.</p> <p>Chris Bizer presented the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1d670da0">Berlin SPARQL Benchmark</a> (<a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x21928808">BSBM</a>), which has already been discussed here in some detail. He did acknowledge that the next round of the race must have a real steady-state rule. This just means that the benchmark must be run long enough for the system under test to reach a state where the cache is full and the performance remains indefinitely at the same level. Reaching steady state can take 20-30 minutes in some cases.</p> <p>Regardless of steady state, BSBM has two generally valid conclusions: </p> <ol> <li>mapping relational to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xab811020">RDF</a>, where possible, is faster than triple storage; and </li> <li>the equivalent relational solution can be some 10x faster than the pure triples representation.</li> </ol> <p>Mike Dean asked whether BSBM was a case of a setup to have triple stores fail. Not necessarily, I would say; we should understand that one motivation of BSBM is testing mapping technologies. Therefore it must have a workload where mapping makes sense. Of course there are workloads where triples are unchallenged â take the <a href="http://challenge.semanticweb.org/" id="link-id0x2538c3b8">Billion Triples Challenge</a> <a href="http://dbpedia.org/resource/Data" id="link-id0x1d673760">data</a> set for one.</p> <p>Also, with BSBM, once should note that the query optimization time plays a fairly large role since most queries touch relatively little data. Also, even if the scale is large, the working set is not nearly the size of the database. This in fact penalizes mapping technologies against native <a href="http://dbpedia.org/resource/SQL" id="link-id0xac16cc10">SQL</a> since the difference there is compiling the query, especially since parameters are not used. So, Chris, since we both like to map, let's make a benchmark that shows mapping closer to native SQL.</p> <h2>Bridging the 10x Gap?</h2> <p>When we run Virtuoso relational against Virtuoso triple store with the <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x1d7dc518">TPC-H</a> workload, we see that the relational case is significantly faster. These are long queries, thus query optimization time is negligible; we are here comparing memory-based access times. Why is this? The answer is that a single index lookup gives multiple column values with almost no penalty for the extra column. Also, since the number of total joins is lower, the overhead coming from moving from join to next join is likewise lower. This is just a meter of count of executed instructions.</p> <p>A column store joins in principle just as much as a triple store. However, since the BI workload often consists of scanning over large tables, the joins tend to be local, the needed lookup can often use the previous location as a starting point. A triple store can do the same if queries have high locality. We do this in some SQL situations and can try this with triples also. The RDF workload is typically more random in its access pattern, though. The other factor is the length of control path. A column store has a simpler control flow if it knows that the column will have exactly one value per row. With RDF, this is not a given. Also, the column store's row is identified by a single number and not a multipart key. These two factors give the column store running with a fixed schema some edge over the more generic RDF quad store.</p> <p>There was some discussion on how much closer a triple store could come to a relational one. Some gains are undoubtedly possible. We will see. For the ideal row store workload, the <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x22e5b6f8">RDBMS</a> will continue to have some edge. Large online systems typically have a large part of the workload that is simple and repetitive. There is nothing to prevent one having special indices for supporting such workload, even while retaining the possibility of arbitrary triples elsewhere. Some degree of application-specific data structure does make sense. We just need to show how this is done. In this way, we have a continuum and not an either/or choice of triples vs. tables.</p> <h2>Scale, Where Next?</h2> <p>Concerning the future direction of the workshop, there were a few directions suggested. One of the more interesting ones was Mike Dean's suggestion about dealing with a large volume of same-as assertions, specifically a volume where materializing all the entailed triples was no longer practical. Of course, there is the question of scale. This time, we were the only ones focusing on a parallel database with no restrictions on joining.</p>
ISWC 2008: The Scalable Knowledge Systems Workshop
http://www.openlinksw.com/weblog/oerling/?date=2008-11-03#1471
2008-11-03T13:16:47Z
2008-11-03T12:33:49-05:00
<p>Mike Dean of <a href="http://dbpedia.org/resource/BBN_Technologies" id="link-id0x21d04768">BBN Technologies</a> opened the Scalable <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x22348c58">Knowledge</a> Systems Workshop with an invited talk. He reminded us of the facts of nature as concern the cost of distributed computing and running out of space for the working set. Developers in the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x22570328">semantic web</a> field deplorably often ignore these facts, or alternatively recognize them and admit that they are unbeatable, that one just can't join across partitions.</p> <p>I gave a talk about the <a href="http://virtuoso.openlinksw.com" id="link-id0x23f313f0">Virtuoso</a> Cluster edition, wherein I repeated essentially the same ground facts as Mike and outlined how we (in spite of these) profit from distributed memory multiprocessing. To those not intimate with these questions, let me affirm that deriving benefit from threading in a symmetric multiprocessor box, let alone a cluster connected by a network, totally depends on having many relatively long running things going at a time and blocking as seldom as possible.</p> <p>Further, Mike Dean talked about <a href="http://www.asio.bbn.com/" id="link-id0x1d74c108">ASIO</a>, the BBN suite of semantic web tools. His most challenging statement was about the storage engine, a network-database-inspired triple-store using memory-mapped files. </p> <p>Will the <a href="http://dbpedia.org/resource/CODASYL" id="link-id0x1f8ee860">CODASYL</a> days come back, and will the linked list on disk be the way to store triples/quads? I would say that this will have, especially with a memory-mapped file, probably a better best-case as a B-tree but that this also will be less predictable with fragmentation. With Virtuoso, using a B-tree index, we see about 20-30% of CPU time spent on index lookup when running LUBM queries. With a disk-based memory-mapped linked-list storage, we would see some improvements in this while getting hit probably worse than now in the case of fragmentation. Plus compaction on the fly would not be nearly as easy and surely far less local, if there were pointers between pages. So it is my intuition that trees are a safer bet with varying workloads while linked lists can be faster in a query-dominated in-memory situation.</p> <p>Chris Bizer presented the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1d670da0">Berlin SPARQL Benchmark</a> (<a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x21928808">BSBM</a>), which has already been discussed here in some detail. He did acknowledge that the next round of the race must have a real steady-state rule. This just means that the benchmark must be run long enough for the system under test to reach a state where the cache is full and the performance remains indefinitely at the same level. Reaching steady state can take 20-30 minutes in some cases.</p> <p>Regardless of steady state, BSBM has two generally valid conclusions: </p> <ol> <li>mapping relational to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xab811020">RDF</a>, where possible, is faster than triple storage; and </li> <li>the equivalent relational solution can be some 10x faster than the pure triples representation.</li> </ol> <p>Mike Dean asked whether BSBM was a case of a setup to have triple stores fail. Not necessarily, I would say; we should understand that one motivation of BSBM is testing mapping technologies. Therefore it must have a workload where mapping makes sense. Of course there are workloads where triples are unchallenged â take the <a href="http://challenge.semanticweb.org/" id="link-id0x2538c3b8">Billion Triples Challenge</a> <a href="http://dbpedia.org/resource/Data" id="link-id0x1d673760">data</a> set for one.</p> <p>Also, with BSBM, once should note that the query optimization time plays a fairly large role since most queries touch relatively little data. Also, even if the scale is large, the working set is not nearly the size of the database. This in fact penalizes mapping technologies against native <a href="http://dbpedia.org/resource/SQL" id="link-id0xac16cc10">SQL</a> since the difference there is compiling the query, especially since parameters are not used. So, Chris, since we both like to map, let's make a benchmark that shows mapping closer to native SQL.</p> <h2>Bridging the 10x Gap?</h2> <p>When we run Virtuoso relational against Virtuoso triple store with the <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x1d7dc518">TPC-H</a> workload, we see that the relational case is significantly faster. These are long queries, thus query optimization time is negligible; we are here comparing memory-based access times. Why is this? The answer is that a single index lookup gives multiple column values with almost no penalty for the extra column. Also, since the number of total joins is lower, the overhead coming from moving from join to next join is likewise lower. This is just a meter of count of executed instructions.</p> <p>A column store joins in principle just as much as a triple store. However, since the BI workload often consists of scanning over large tables, the joins tend to be local, the needed lookup can often use the previous location as a starting point. A triple store can do the same if queries have high locality. We do this in some SQL situations and can try this with triples also. The RDF workload is typically more random in its access pattern, though. The other factor is the length of control path. A column store has a simpler control flow if it knows that the column will have exactly one value per row. With RDF, this is not a given. Also, the column store's row is identified by a single number and not a multipart key. These two factors give the column store running with a fixed schema some edge over the more generic RDF quad store.</p> <p>There was some discussion on how much closer a triple store could come to a relational one. Some gains are undoubtedly possible. We will see. For the ideal row store workload, the <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x22e5b6f8">RDBMS</a> will continue to have some edge. Large online systems typically have a large part of the workload that is simple and repetitive. There is nothing to prevent one having special indices for supporting such workload, even while retaining the possibility of arbitrary triples elsewhere. Some degree of application-specific data structure does make sense. We just need to show how this is done. In this way, we have a continuum and not an either/or choice of triples vs. tables.</p> <h2>Scale, Where Next?</h2> <p>Concerning the future direction of the workshop, there were a few directions suggested. One of the more interesting ones was Mike Dean's suggestion about dealing with a large volume of same-as assertions, specifically a volume where materializing all the entailed triples was no longer practical. Of course, there is the question of scale. This time, we were the only ones focusing on a parallel database with no restrictions on joining.</p>
Virtuoso - Are We Too Clever for Our Own Good? (updated)
http://www.openlinksw.com/weblog/oerling/?date=2008-10-26#1465
2008-10-26T12:15:35Z
2008-10-27T12:07:52-04:00
<p>"Physician, heal thyself," it is said. We profess to say what the messaging of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1fa3da18">semantic web</a> ought to be, but is our own perfect?</p> <p>I will here engage in some critical introspection as well as amplify on some answers given to <a href="http://virtuoso.openlinksw.com" id="link-id0x1e1eecf0">Virtuoso</a>-related questions in recent times.</p> <p>I use some conversations from the <a href="http://dbpedia.org/resource/Vienna" id="link-id0x1ec0b2e0">Vienna</a> <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x2045ac10">Linked Data</a> Practitioners meeting as a starting point. These views are mine and are limited to the Virtuoso server. These do not apply to the <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id0x2045ac38">ODS</a> (<a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id0x14f63c58">OpenLink Data Spaces</a>) applications line, <a href="http://oat.openlinksw.com/" id="link-id0x14f63c80">OAT</a> (<a href="http://oat.openlinksw.com/" id="link-id0x1e536928">OpenLink Ajax Toolkit</a>), or <a href="http://ode.openlinksw.com/" id="link-id0x1eaed7f8">ODE</a> (<a href="http://ode.openlinksw.com/" id="link-id0x1edfff88">OpenLink Data Explorer</a>).</p> <h3>"It is not always clear what the main thrust is, we get the impression that you are spread too thin," said <a href="http://www.informatik.uni-leipzig.de/~auer/foaf.rdf#me" id="link-id0x1b8a9580">Sören Auer</a>.</h3> <p>Well, personally, I am all for core competence. This is why I do not participate in all the online conversations and groups as much as I could, for example. Time and energy are critical resources and must be invested where they make a difference. In this case, the real core competence is running in the database race. This in itself, come to think of it, is a pretty broad concept.</p> <p>This is why we put a lot of emphasis on Linked Data and the <a href="http://dbpedia.org/resource/Data" id="link-id0x1b85fa38">Data</a> Web for now, as this is the emerging game. This is a deliberate choice, not an outside imperative or built-in limitation. More specifically, this means exposing any pre-existing relational data as linked data plus being the definitive <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1f5b4468">RDF</a> store.</p> <p>We can do this because we own our database and <a href="http://dbpedia.org/resource/SQL" id="link-id0x20076468">SQL</a> and data access middleware and have a history of connecting to any <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1ffd6f98">RDBMS</a> out there.</p> <p>The principal message we have been hearing from the RDF field is the call for scale of triple storage. This is even louder than the call for relational mapping. We believe that in time mapping will exceed triple storage as such, once we get some real production strength mappings deployed, enough to outperform RDF warehousing.</p> <p>There are also RDF middleware things like RDF-ization and demand-driven web harvesting (i.e, the so-called Sponger). These are <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1316f720">SPARQL</a> options, thus accessed via standard interfaces. We have little desire to create our own languages or APIs, or to tell people how to program. This is why we recently introduced <a href="http://sourceforge.net/projects/sesame/" id="link-id0x20756a68">Sesame</a>- and <a href="http://jena.sourceforge.net/" id="link-id0x1ec01ac0">Jena</a>-compatible APIs to our RDF store. From what we hear, these work. On the other hand, we do not hesitate to move beyond the standards when there is obvious value or necessity. This is why we brought SPARQL up to and beyond SQL expressivity. It is not a case of E3 (Embrace, Extend, Extinguish).</p> <p>Now, this message could be better reflected in our material on the web. This <a href="http://dbpedia.org/resource/Blog" id="link-id0x2027b410">blog</a> is a rather informal step in this direction; more is to come. For now we concentrate on delivering.</p> <p>The conventional communications wisdom is to split the message by target audience. For this, we should split the RDF, relational, and web services messages from each other. We believe that a challenger, like the semantic web technology stack, must have a compelling message to tell for it to be interesting. This is not a question of research prototypes. The new technology cannot lack something the installed technology takes for granted.</p> <p>This is why we do not tend to show things like how to insert and query a few triples: No business out there will insert and query triples for the sake of triples. There must be a more compelling story â for example, turning the whole world into a database. This is why our examples start with things like turning the <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x2051ff98">TPC-H</a> database into RDF, queries and all. Anything less is not interesting. Why would an enterprise that has business intelligence and integration issues way more complex than the rather stereotypical TPC-H even look at a technology that pretends to be all for integration and all for expressivity of queries, yet cannot answer the first question of the entry exam?</p> <p>The world out there is complex. But maybe we ought to make some simple tutorials? So, as a call to the people out there, tell us what a good tutorial would be. The question is more about figuring out what is out there and adapting these and making a sort of compatibility list. Jena and Sesame stuff ought to run as is. We could offer a webinar to all the data web luminaries showing how to promote the data web message with Virtuoso. After all, why not show it on the best platform?</p> <h3>"You are arrogant. When I read your papers or documentation, the impression I get is that you say you are smart and the reader is stupid."</h3> <p>We should answer in multiple parts.</p> <p>For general collateral, like web sites and documentation:</p> <p>The web site gives a confused product image. For the Virtuoso product, we should divide at the top into</p> <ul> <li> Data web and RDF - Host linked data, expose relational assets as linked data;</li> <li> Relational Database - Full function, high performance, open source, Federated/Virtual Relational DBMS, expose heterogeneous RDB assets through one point of contact for integration;</li> <li> Web Services - access all the above over standard protocols, dynamic web pages, web hosting.</li> </ul> <p>For each point, one simple statement. We all know what the above things mean?</p> <p>Then we add a new point about scalability that impacts all the above, namely the Virtuoso version 6 Cluster, meaning that you can do all these things at 10 to 1000 times the scale. This means this much more data or in some cases this much more requests per second. This too is clear.</p> <p>Far as I am concerned, hosting Java or .<a href="http://dbpedia.org/resource/.NET_Framework" id="link-id0x1f297540">NET</a> does not have to be on the front page. Also, we have no great interest in going against <a href="http://dbpedia.org/resource/Apache" id="link-id0x1ea29578">Apache</a> when it comes to a web server only situation. The fact that we have a web listener is important for some things but our claim to fame does not rest on this.</p> <p>Then for documentation and training materials: The documentation should be better. Specifically it should have more of a how-to dimension since nobody reads the whole thing anyhow. About online tutorials, the order of presentation should be different. They do not really reflect what is important at the present moment either.</p> <p>Now for conference papers: Since taking the data web as a focus area, we have submitted some papers and had some rejected because these do not have enough references and do not explain what is obvious to ourselves.</p> <p>I think that the communications failure in this case is that we want to talk about end to end solutions and the reviewers expect research. For us, the solution is interesting and exists only if there is an adequate functionality mix for addressing a specific use case. This is why we do not make a paper about query cost model alone because the cost model, while indispensable, is a thing that is taken for granted where we come from. So we mention RDF adaptations to cost model, as these are important to the whole but do not find these to be the justification for a whole paper. If we made papers on this basis, we would have to make five times as many. Maybe we ought to.</p> <h3>"Virtuoso is very big and very difficult"</h3> <p>One thing that is not obvious from the Virtuoso packaging is that the minimum installation is an executable under 10MB and a config file. Two files.</p> <p>This gives you SQL and SPARQL out of the box. Adding <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x20a2e7d0">ODBC</a> and <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x1e4cceb8">JDBC</a> clients is as simple as it gets. After this, there is basic database functionality. Tuning is a matter of a few parameters that are explained on this blog and elsewhere. Also, the full scale installation is available as an Amazon EC2 image, so no installation required.</p> <p>Now for the difficult side:</p> <p>Use SQL and SPARQL; use stored procedures whenever there is server side business logic. For some time critical web pages, use VSP. Do not use VSPX. Otherwise, use whatever you are used to â <a href="http://dbpedia.org/resource/PHP" id="link-id0x20b03f08">PHP</a> or Java or anything else. For web services, simple is best. Stick to basics. "The engineer is one who can invent a simple thing." Use SQL statements rather than admin UI.</p> <p>Know that you can start a server with no database file and you get an initial database with nothing extra. The demo database, the way it is produced by installers is cluttered.</p> <p>We should put this into a couple of use case oriented how-tos.</p> <p>Also, we should create a network of "friendly local virtuoso geeks" for providing basic training and services so we do not have to explain these things all the time. To all you data-web-ers out there â please sign up and we will provide instructions, etc. Contact Yrjänä Rankka (ghard[at-sign]openlinksw.com), or go through the mailing lists; do not contact me directly.</p> <h3>"OK, we understand that you may be good at the large end of the spectrum but how do you reconcile this with the lightweight or embedded end, like the semantic desktop?"</h3> <p>Now, what is good for one end is usually good for the other. Namely, a database, no matter the scale, needs to have space efficient storage, fast index lookup, and correct query plans. Then there are things that occur only at the high-end, like clustering, but these are separate things. For embedding, the initial memory footprint needs to be small. With Virtuoso, this is accomplished by leaving out some 200 built-in tables and 100,000 lines of SQL procedures that are normally in by default, supporting things such as DAV and diverse other protocols. After all, if SPARQL is all one wants these are not needed.</p> <p>If one really wants to do one's server logic (like web listener and thread dispatching) oneself, this is not impossible but requires some advice from us. On the other hand, if one wants to have logic for security close to the data, then using stored procedures is recommended; these execute right next to the data, and support inline SPARQL and SQL. Depending on the license status of the other code, some special licensing arrangements may apply.</p> <p>We are talking about such things with different parties at present.</p> <h3>"How webby are you? What is webby?"</h3> <p>"Webby means distributed, heterogeneous, open; not monolithic consolidation of everything."</p> <p>We are philosophically webby. We come from open standards; we are after all called OpenLink; our history consists of connecting things. We believe in choice â the user should be able to pick the best of breed for components and have them work together. We cannot and do not wish to force replacement of existing assets. Transforming data on the fly and connecting systems, leaving data where it originally resides, is the first preference. For the data web, the first preference is a federation of independent SPARQL end points. When there is harvesting, we prefer to do it on demand, as with our Sponger. With the immense amount of data out there we believe in finding what is relevant <i>when</i> it is relevant, preferably close at hand, leveraging things like social networks. With a data web, many things which are now siloized, such as marketplaces and social networks, will return to the open.</p> <p>Google-style crawling of everything becomes less practical if one needs to run complex <i>ad hoc</i> queries against the mass of data. For these types of scenarios, if one needs to warehouse, the data cloud will offer solutions where one pays for database on demand. While we believe in loosely coupled federation where possible, we have serious work on the scalability side for the data center and the compute-on-demand cloud.</p> <h3>"How does OpenLink see the next five years unfolding?"</h3> <p>Personally, I think we have the basics for the birth of a new inflection in the <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x2018bd98">knowledge</a> economy. The <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x1ec110d8">URI</a> is the unit of exchange; its value and competitive edge lie in the data it links you with. A name without context is worth little, but as a name gets more use, more <a href="http://dbpedia.org/resource/Information" id="link-id0x1ecfba08">information</a> can be found through that name. This is anything from financial statistics, to legal precedents, to news reporting or government data. Right now, if the SEC just added one line of markup to the XBRL template, this would instantaneously make all SEC-mandated reporting into linked data via GRDDL.</p> <p>The URI is a carrier of brand. An information brand gets traffic and references, and this can be monetized in diverse ways. The key word is <i>context</i>. Information overload is here to stay, and only better context offers the needed increase in productivity to stay ahead of the flood.</p> <p>Semantic technologies on the whole can help with this. Why these should be semantic web or data web technologies as opposed to just semantic is the linked data value proposition. Even smart islands are still islands. Agility, scale, and scope, depend on the possibility of combining things. Therefore common terminologies and dereferenceability and discoverability are important. Without these, we are at best dealing with closed systems even if they were smart. The expert systems of the 1980s are a case in point.</p> <p>Ever since the .com era, the <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id0x1c4c9248">URL</a> has been a brand. Now it becomes a URI. Thus, entirely hiding the URI from the user experience is not always desirable. The URI is a sort of handle on the provenance and where more can be found; besides, people are already used to these.</p> <p>With linked data, information value-add products become easy to build and deploy. They can be basically just canned SPARQL queries combining data in a useful and insightful manner. And where there is traffic there can be monetization, whether by advertizing, subscription, or other means. Such possibilities are a natural adjunct to the blogosphere. To publish analysis, one no longer needs to be a think tank or media company. We could call this scenario the birth of a meshup economy.</p> <p>For OpenLink itself, this is our roadmap. The immediate future is about getting our high end offerings like clustered RDF storage generally available, both on the cloud and for private data centers. Ourselves, we will offer the whole <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x20791bf0">Linked Open Data</a> cloud as a database. The single feature to come in version 2 of this is fully automatic partitioning and repartitioning for on-demand scale; now, you have to choose how many partitions you have.</p> <p>This makes some things possible that were hard thus far.</p> <p>On the mapping front, we go for real-scale data integration scenarios where we can show that SPARQL can unify terms and concepts across databases, yet bring no added cost for complex queries. Enterprises can use their existing warehouses and have an added level of abstraction, the possibility of cross systems interlinking, the advantages of using the same taxonomies and ontologies across systems, and so forth.</p> <p>Then there will be developments in the direction of smarter web harvesting on demand with the Virtuoso <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id0x1f27e6d8">Sponger</a>, and federation of heterogeneous SPARQL end points. The federation is not so unlike clustering, except the time scales are 2 orders of magnitude longer. The work on SPARQL end point statistics and data set description and discovery is a good development in the community.</p> <p>Then there will be NLP integration, as exemplified by the Open Calais linked data wrapper and more.</p> <p>Can we pull this off or is this being spread too thin? We know from experience that all this can be accomplished. Scale is already here; we show it with the billion triples set. Mapping is here; we showed it last in the Berlin Benchmark. We will also show some TPC-H results after we get a little quiet after the ISWC event. Then there is ongoing maintenance but with this we have shown a steady turnaround and quick time to fix for pretty much anything.</p>
Virtuoso Update, Billion Triples and Outlook
http://www.openlinksw.com/weblog/oerling/?date=2008-10-02#1448
2008-10-02T09:31:17Z
2008-10-02T12:47:02.000002-04:00
<p>I will say a few things about what we have been doing and where we can go.</p> <p>Firstly, we have a fairly scalable platform with <a href="http://virtuoso.openlinksw.com" id="link-id0xa412e450">Virtuoso</a> 6 Cluster. It was most recently tested with the workload discussed in the previous <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1445" id="link-id1638a5b8">Billion Triples post</a>.</p> <p>There is an updated version of <a href="http://www.openlinksw.com/weblog/oerling/2008webscale_rdf.pdf" id="link-id16280a68">the paper about this</a>. This will be presented at the web scale workshop of ISWC 2008 in Karlsruhe.</p> <p>Right now, we are polishing some things in Virtuoso 6 -- some optimizations for smarter balancing of interconnect traffic over multiple network interfaces, and some more <a href="http://dbpedia.org/resource/SQL" id="link-id0x1c1c5f48">SQL</a> optimizations specific to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1bcb6108">RDF</a>. The must-have basics, like parallel running of sub-queries and aggregates, and all-around unrolling of loops of every kind into large partitioned batches, is all there and proven to work.</p> <p>We spent a lot of time around the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x3a4e17c8">Berlin SPARQL Benchmark</a> story, so we got to the more advanced stuff like the <a href="http://challenge.semanticweb.org/" id="link-id0x1a66c568">Billion Triples Challenge</a> rather late. We did along the way also run <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x188c2608">BSBM</a> with an <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x1aa97f98">Oracle</a> back-end, with Virtuoso mapping <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1abd87a0">SPARQL</a> to SQL. This merits its own analysis in the near future. This will be the basic how-to of mapping OLTP systems to RDF. Depending on the case, one can use this for lookups in real-time or ETL.</p> <p>RDF will deliver value in complex situations. An example of a complex relational mapping use case came from Ordnance Survey, presented at the <a href="http://www.w3.org/2005/Incubator/rdb2rdf/" id="link-id0x1a941678">RDB2RDF XG</a>. Examples of complex warehouses include the <a href="http://neurocommons.org/page/Main_Page" id="link-id0x1aa5a9f8">Neurocommons</a> database, the Billion Triples Challenge, and the <a href="http://www.garlik.com/" id="link-id0x372df7b0">Garlik DataPatrol</a>.</p> <p>In comparison, the Berlin workload is really simple and one where RDF is not at its best, as amply discussed on the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1a671cf0">Linked Data</a> forum. BSBM's primary value is as a demonstrator for the basic mapping tasks that will be repeated over and over for pretty much any online system when presence on the <a href="http://dbpedia.org/resource/Data" id="link-id0x1ab83dd0">data</a> web becomes as indispensable as presence on the HTML web.</p> <p>I will now talk about the complex warehouse/web-harvesting side. I will come to the mapping in another post.</p> <p>Now, all the things shown in the <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1445" id="link-id14de1d18">Billion Triples post</a> can be done with a relational system specially built for each purpose. Since we are a general purpose <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x340d3470">RDBMS</a>, we use this capability where it makes sense. For example, storing statistics about which tags or interests occur with which other tags or interests as RDF blank nodes makes no sense. We do not even make the experiment; we know ahead of time that the result is at least an order of magnitude in favor of the relational row-oriented solution in both space and time.</p> <p>Whenever there is a data structure specially made for answering one specific question, like joint occurrence of tags, RDB and mapping is the way to go. With Virtuoso, this can fully-well coexist with physical triples, and can still be accessed in SPARQL and mixed with triples. This is territory that we have not extensively covered yet, but we will be giving some examples about this later.</p> <p>The real value of RDF is in agility. When there is no time to design and load a new warehouse for every new question, RDF is unparalleled. Also SPARQL, once it has the necessary extensions of aggregating and sub-queries, is nicer than SQL, especially when we have sub-classes and sub-properties, transitivity, and "same as" enabled. These things have some run time cost and if there is a report one is hitting absolutely all the time, then chances are that resolving terms and identity at load-time and using materialized views in SQL is the reasonable thing. If one is inventing a new report every time, then RDF has a lot more convenience and flexibility.</p> <p>We are just beginning to explore what we can do with data sets such as the online conversation space, linked data, and the open ontologies of <a href="http://umbel.org/about/" id="link-id0x19cabf38">UMBEL</a> and <a href="http://dbpedia.org/resource/Cyc" id="link-id0x19cecd10">OpenCyc</a>. It is safe to say that we can run with real world scale without loss of query expressivity. There is an incremental cost for performance but this is not prohibitive. Serving the whole billion triples set from memory would cost about $32K in hardware. $8K will do if one can wait for disk part of the time. One can use these numbers as a basis for costing larger systems. For online search applications, one will note that running the indexes pretty much from memory is necessary for flat response time. For back office analytics this is not necessarily as critical. It all depends on the use case.</p> <p>We expect to be able to combine geography, social proximity, subject matter, and <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id0x1a8202e8">named entities</a>, with hierarchical taxonomies and traditional full text, and to present this through a simple user interface.</p> <p>We expect to do this with online response times if we have a limited set of starting points and do not navigate more than 2 or 3 steps from each starting point. An example would be to have a full text pattern and news group, and get the cloud of interests from the authors of matching posts. Another would be to make a faceted view of the properties of the 1000 people most closely connected to one person.</p> <p>Queries like finding the fastest online responders to questions about romance across the global board-scape, or finding the person who initiates the most long running conversations about crime, take a bit longer but are entirely possible.</p> <p>The genius of RDF is to be able to do these things within a general purpose database, ad hoc, in a single query language, mostly without materializing intermediate results. Any of these things could be done with arbitrary efficiency in a custom built system. But what is special now is that the cost of access to this type of <a href="http://dbpedia.org/resource/Information" id="link-id0x1ab0a918">information</a> and far beyond drops dramatically as we can do these things in a far less labor intensive way, with a general purpose system, with no redesigning and reloading of warehouses at every turn. The query becomes a commodity.</p> <p>Still, one must know what to ask. In this respect, the self-describing nature of RDF is unmatched. A query like <i>list the top 10 attributes with the most distinct values for all persons</i> cannot be done in SQL. SQL simply does not allow the columns to be variable.</p> <p>Further, we can accept queries as text, the way people are used to supplying them, and use structure for drill-down or result-relevance, and also recognize named entities and subject matter concepts in query text. Very simple NLP will go a long way towards keeping SPARQL out of the user experience.</p> <p>The other way of keeping query complexity hidden is to publish hand-written SPARQL as parameter-fed canned reports.</p> <p>Between now and ISWC 2008, the last week of October, we will put out demos showing some of these things. Stay tuned.</p>
A quick look at SP2B, the SPARQL Performance Benchmark
http://www.openlinksw.com/weblog/oerling/?date=2008-08-27#1422
2008-08-27T16:00:07Z
2008-09-02T09:49:55-04:00
<p>I finally got around to running the <a href="http://dbis.informatik.uni-freiburg.de/index.php?project=SP2B" id="link-id17bac628">SP<sup>2</sup>B SPARQL Performance Benchmark</a> on the current <a href="http://virtuoso.openlinksw.com" id="link-id0x1d2a6838">Virtuoso</a> Open Source Edition, v5.0.8.</p> <p>I ran it with the 5M triples scale, which is the highest scale for which the authors give numbers.</p> <p>I got a run time of 25 minutes for the 12 queries, giving an arithmetic mean of the query time of 125 seconds. This is better than the 800 or so seconds that the authors had measured. Also, Q6 of the set had failed for the authors, but we have since fixed this; the fix is in the v5.0.8 cut.</p> <p>I also tried it with a scale of 25M, but this became I/O bound and took a bit longer. I will try this with v6 and v7 cluster later, which are vastly better at anything I/O bound.</p> <p>The machine was a 2GHz Xeon with 8G RAM. The query text was the one from the authors, with an explicit <code>FROM</code> clause added; the client was the command line Interactive <a href="http://dbpedia.org/resource/SQL" id="link-id0x19e74ce0">SQL</a> (iSQL).</p> <p>If one does the test with the default index layout without specifying a graph, things will not work very well. Also, returning the million-row results of these queries over the <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id0x1c4231a0">SPARQL protocol</a> is not practical.</p> <p>I will say something more about SP<sup>2</sup>B when I get to have a closer look.</p>
Configuring Virtuoso for Benchmarking
http://www.openlinksw.com/weblog/oerling/?date=2008-08-25#1418
2008-08-25T14:05:46Z
2008-08-25T15:29:04-04:00
<p>I will here summarize what should be known about running benchmarks with <a href="http://virtuoso.openlinksw.com" id="link-id0xc53af18">Virtuoso</a>.</p> <h2>Physical Memory</h2> <p>For 8G RAM, in the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set â</p> <blockquote> <code> [Parameters]<br /> ...<br /> NumberOfBuffers = 550000 </code> </blockquote> <p>For 16G RAM, double thisâ</p> <blockquote> <code> [Parameters]<br /> ...<br /> NumberOfBuffers = 1100000 </code> </blockquote> <h2>Transaction Isolation</h2> <p>For most cases, certainly all <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xc2f07a0">RDF</a> cases, <i>Read Committed</i> should be the default transaction isolation. In the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set â</p> <blockquote> <code> [Parameters]<br /> ...<br /> DefaultIsolation = 2 </code> </blockquote> <h2>Multiuser Workload</h2> <p>If <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0xc1c7178">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0xd16fb40">JDBC</a>, or similarly connected client applications are used, there must be more <code>ServerThreads</code> available than there will be client connections. In the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set â</p> <blockquote> <code> [Parameters]<br /> ...<br /> ServerThreads = 100 </code> </blockquote> <p>With web clients (unlike ODBC, JDBC, or similar clients), it may be justified to have fewer <code>ServerThreads</code> than there are concurrent clients. The <code>MaxKeepAlives</code> should be the maximum number of expected web clients. This can be more than the <code>ServerThreads</code> count. In the <code>[HTTPServer]</code> stanza of <code>virtuoso.ini</code>, set â</p> <blockquote> <code> [HTTPServer]<br /> ...<br /> ServerThreads = 100 <br /> MaxKeepAlives = 1000 <br /> KeepAliveTimeout = 10 </code> </blockquote> <p> <i><b>Note</b> â The <code>[HTTPServer] ServerThreads</code> are taken from the total pool made available by the <code>[Parameters] ServerThreads</code>. Thus, the <code>[Parameters] ServerThreads</code> should always be at least as large as (and is best set greater than) the <code>[HTTPServer] ServerThreads</code>, and if using the closed-source Commercial Version, should not exceed the licensed thread count.</i> </p> <h2>Disk Use</h2> <p>The basic rule is to use one stripe (file) per distinct physical device (not per file system), using no RAID. For example, one might stripe a database over 6 files (6 physical disks), with an initial size of 60000 pages (the files will grow as needed). </p> <p>For the above described example, in the <code>[Database]</code> stanza of <code>virtuoso.ini</code>, set â</p> <blockquote> <code> [Database]<br /> ...<br /> Striping = 1<br /> MaxCheckpointRemap = 2000000 </code> </blockquote> <p>â and in the <code>[Striping]</code> stanza, on one line per <code>SegmentName</code>, set â</p> <blockquote> <code> [Striping]<br /> ...<br /> Segment1 = 60000 , /virtdev/db/virt-seg1.db = q1 , /data1/db/virt-seg1-str2.db = q2 , /data2/db/virt-seg1-str3.db = q3 , /data3/db/virt-seg1-str4.db = q4 , /data4/db/virt-seg1-str5.db = q5 , /data5/db/virt-seg1-str6.db = q6</code> </blockquote> <p>As can be seen here, each file gets a background IO thread (the <code>= q<i>xxx</i></code> clause). It should be noted that all files on the same physical device should have the same <code>q<i>xxx</i></code> value. This is not directly relevant to the benchmarking scenario above, because we have only one file per device, and thus only one file per IO queue.</p> <h2> <a href="http://dbpedia.org/resource/SQL" id="link-id0xc9fa298">SQL</a> Optimization</h2> <p>If queries have lots of joins but access little <a href="http://dbpedia.org/resource/Data" id="link-id0xb4e0aa0">data</a>, as with the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0xb2de990">Berlin SPARQL Benchmark</a>, the SQL compiler must be told not to look for better plans if the best plan so far is quicker than the compilation time expended so far. Thus, in the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set â</p> <blockquote> <code> [Parameters]<br /> ...<br /> StopCompilerWhenXOverRunTime = 1 </code> </blockquote>
BSBM With Triples and Mapped Relational Data
http://www.openlinksw.com/weblog/oerling/?date=2008-08-06#1409
2008-08-06T19:35:27Z
2008-08-06T16:29:40-04:00
<p>The special contribution of the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id10039db0">Berlin SPARQL Benchmark</a> (<a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id106b2538">BSBM</a>) to the <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id101a75f8">RDF</a> world is to raise the question of doing OLTP with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xb230eb0">RDF</a>.</p> <p>Of course, here we immediately hit the question of comparisons with relational databases. To this effect, <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0xa832da8">BSBM</a> also specifies a relational schema and can generate the <a href="http://dbpedia.org/resource/Data" id="link-id1206c378">data</a> as either triples or <a href="http://dbpedia.org/resource/SQL" id="link-id1667f040">SQL</a> inserts.</p> <p>The benchmark effectively simulates the case of exposing an existing <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id10a93518">RDBMS</a> as RDF. <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id13e46d80">OpenLink Software</a> calls this <i>RDF Views</i>. <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id12027578">Oracle</a> is beginning to call this <i>semantic covers</i>. The <a href="http://www.w3.org/2005/Incubator/rdb2rdf/" id="link-id161dc678">RDB2RDF XG</a>, a W3C incubator group, has been active in this area since Spring, 2008.</p> <h3>But why an OLTP workload with RDF to begin with?</h3> <p>We believe this is relevant because RDF promises to be the interoperability factor between potentially all of traditional IS. If <a href="http://dbpedia.org/resource/Data" id="link-id0xabe48a0">data</a> is online for human consumption, it may be online via a <a href="http://dbpedia.org/resource/SPARQL" id="link-id106a8908">SPARQL</a> end-point as well. The economic justification will come from discoverability and from applications integrating multi-source structured data. Online shopping is a fine use case.</p> <p>Warehousing all the world's publishable data as RDF is not our first preference, nor would it be the publisher's. Considerations of duplicate infrastructure and maintenance are reason enough. Consequently, we need to show that mapping can outperform an RDF warehouse, which is what we'll do here.</p> <h3>What We Got </h3> <p>First, we found that <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1400" id="link-id150ea748">making the query plan took much too long</a> in proportion to the run time. With BSBM this is an issue because the queries have lots of joins but access relatively little data. So we made a faster compiler and along the way retouched the cost model a bit.</p> <p>But the really interesting part with BSBM is mapping relational data to RDF. For us, BSBM is a great way of showing that mapping can outperform even the best triple store. A relational row store is as good as unbeatable with the query mix. And when there is a clear mapping, there is no reason the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x96bb5e0">SPARQL</a> could not be directly translated.</p> <p>If Chris Bizer et al launched the mapping ship, we will be the ones to pilot it to harbor!</p> <p>We filled two <a href="http://virtuoso.openlinksw.com" id="link-id12dbdc70">Virtuoso</a> instances with a BSBM200000 data set, for 100M triples. One was filled with physical triples; the other was filled with the equivalent relational data plus mapping to triples. Performance figures are given in "query mixes per hour". (An update or follow-on to this post will provide elapsed times for each test run.)</p> <p>With the unmodified benchmark we got:</p> <blockquote> <table> <tr> <td><i>Physical Triples:</i> </td> <td>Â Â </td> <td>1297 qmph</td> </tr> <tr> <td><i>Mapped Triples:</i> </td> <td>Â Â </td> <td><b>3144 qmph</b> </td> </tr> </table> </blockquote> <p>In both cases, most of the time was spent on Q6, which looks for products with one of three words in the label. We altered Q6 to use text index for the mapping, and altered the databases accordingly. (There is no such thing as an e-commerce site without a text index, so we are amply justified in making this change.)</p> <p>The following were measured on the second run of a 100 query mix series, single test driver, warm cache.</p> <blockquote> <table> <tr> <td><i>Physical Triples:</i> </td> <td>Â Â </td> <td> 5746 qmph</td> </tr> <tr> <td><i>Mapped Triples:</i> </td> <td>Â Â </td> <td> <b>7525 qmph</b> </td> </tr> </table> </blockquote> <p>We then ran the same with 4 concurrent instances of the test driver. The qmph here is 400 / the longest run time.</p> <blockquote> <table> <tr> <td><i>Physical Triples:</i> </td> <td>Â Â </td> <td> 19459 qmph</td> </tr> <tr> <td><i>Mapped Triples:</i> </td> <td>Â Â </td> <td> <b>24531 qmph</b> </td> </tr> </table> </blockquote> <p>The system used was 64-bit Linux, 2GHz dual-Xeon 5130 (8 cores) with 8G RAM. The concurrent throughputs are a little under 4 times the single thread throughput, which is normal for SMP due to memory contention. The numbers do not evidence significant overhead from thread synchronization.</p> <p>The query compilation represents about 1/3 of total server side CPU. In an actual online application of this type, queries would be parameterized, so the throughputs would be accordingly higher. We used the <code>StopCompilerWhenXOverRunTime = 1</code> option here to cut needless compiler overhead, the queries being straightforward enough.</p> <p>We also see that the advantage of mapping can be further increased by more compiler optimizations, so we expect in the end mapping will lead RDF warehousing by a factor of 4 or so.</p> <h3>Suggestions for BSBM</h3> <ul> <li> <p> <b>Reporting Rules.</b> The benchmark spec should specify a form for disclosure of test run data, TPC style. This includes things like configuration parameters and exact text of queries. There should be accepted variants of query text, as with the TPC.</p> </li> <li> <p> <b>Multiuser operation.</b> The test driver should get a stream number as parameter, so that each client makes a different query sequence. Also, disk performance in this type of benchmark can only be reasonably assessed with a naturally parallel multiuser workload.</p> </li> <li> <p> <b>Add business intelligence.</b> SPARQL has aggregates now, at least with <a href="http://jena.sourceforge.net/" id="link-id11a25ac0">Jena</a> and <a href="http://virtuoso.openlinksw.com" id="link-id0xa83f490">Virtuoso</a>, so let's use these. The BSBM business intelligence metric should be a separate metric off the same data. Adding synthetic sales figures would make more interesting queries possible. For example, producing recommendations like "customers who bought this also bought xxx."</p> </li> <li> <p> <b>For the SPARQL community</b>, BSBM sends the message that one ought to support parameterized queries and stored procedures. This would be a <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id109e2448">SPARQL protocol</a> extension; the SPARUL syntax should also have a way of calling a procedure. Something like <code>select proc (??, ??)</code> would be enough, where <code>??</code> is a parameter marker, like <code>?</code> in <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id13febf48">ODBC</a>/<a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id120416a8">JDBC</a>.</p> </li> <li> <p> <b>Add transactions.</b>Especially if we are contrasting mapping vs. storing triples, having an update flow is relevant. In practice, this could be done by having the test driver send web service requests for order entry and the SUT could implement these as updates to the triples or a mapped relational store. This could use stored procedures or logic in an app server.</p> </li> </ul> <h3>Comments on Query Mix</h3> <p>The time of most queries is less than linear to the scale factor. Q6 is an exception if it is not implemented using a text index. Without the text index, Q6 will inevitably come to dominate query time as the scale is increased, and thus will make the benchmark less relevant at larger scales.</p> <h2>Next</h2> <p>We include the sources of our RDF view definitions and other material for running BSBM with our forthcoming Virtuoso Open Source 5.0.8 release. This also includes all the query optimization work done for BSBM. This will be available in the coming days.</p>
Virtuoso Optimizations for the Berlin SPARQL Benchmark
http://www.openlinksw.com/weblog/oerling/?date=2008-07-30#1400
2008-07-30T18:17:54Z
2008-08-06T16:29:37.000003-04:00
<p>We had a look at Chris Bizer's initial results with the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id105c9f78">Berlin SPARQL Benchmark</a> (<a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id102d62b0">BSBM</a>) on <a href="http://virtuoso.openlinksw.com" id="link-id13eb9780">Virtuoso</a>. The first results were rather bad, as nearly all of the run time was spent optimizing the <a href="http://dbpedia.org/resource/SPARQL" id="link-id14a51258">SPARQL</a> statements and under 10% actually running them.</p> <p>So I spent a couple of days on the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xa5a8d0e8">SPARQL</a>/<a href="http://dbpedia.org/resource/SQL" id="link-id108745b0">SQL</a> compiler, to the effect of making it do a better guess of initial execution plan and streamlining some operations. In fact, many of the queries in <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0xaf04af8">BSBM</a> are not particularly sensitive to execution plan, as they access a very small portion of the database. So to close the matter, I put in a flag that makes the <a href="http://dbpedia.org/resource/SQL" id="link-id0x1e8d2360">SQL</a> compiler give up on devising new plans if the time of the best plan so far is less than the time spent compiling so far.</p> <p>With these changes, available now as a diff on top of 5.0.7, we run quite well, several times better than initially. With the compiler time cut-off in place (ini parameter <code>StopCompilerWhenXOverRunTime = 1</code>), we get the following times, output from the BSBM test driver:</p> <blockquote> <pre> Starting test... 0: 1031.22 ms, total: 1151 ms 1: 982.89 ms, total: 1040 ms 2: 923.27 ms, total: 968 ms 3: 898.37 ms, total: 932 ms 4: 855.70 ms, total: 865 ms Scale factor: 10000 Number of query mix runs: 5 times min/max Query mix runtime: 0.8557 s / 1.0312 s Total runtime: 4.691 seconds QMpH: 3836.77 query mixes per hour CQET: 0.93829 seconds average runtime of query mix CQET (geom.): 0.93625 seconds geometric mean runtime of query mix Metrics for Query 1: Count: 5 times executed in whole run AQET: 0.012212 seconds (arithmetic mean) AQET(geom.): 0.009934 seconds (geometric mean) QPS: 81.89 Queries per second minQET/maxQET: 0.00684000s / 0.03115700s Average result count: 7.0 min/max result count: 3 / 10 Metrics for Query 2: Count: 35 times executed in whole run AQET: 0.030490 seconds (arithmetic mean) AQET(geom.): 0.029776 seconds (geometric mean) QPS: 32.80 Queries per second minQET/maxQET: 0.02467300s / 0.06753000s Average result count: 22.5 min/max result count: 15 / 30 Metrics for Query 3: Count: 5 times executed in whole run AQET: 0.006947 seconds (arithmetic mean) AQET(geom.): 0.006905 seconds (geometric mean) QPS: 143.95 Queries per second minQET/maxQET: 0.00580000s / 0.00795100s Average result count: 4.0 min/max result count: 0 / 10 Metrics for Query 4: Count: 5 times executed in whole run AQET: 0.008858 seconds (arithmetic mean) AQET(geom.): 0.008829 seconds (geometric mean) QPS: 112.89 Queries per second minQET/maxQET: 0.00804400s / 0.01019500s Average result count: 3.4 min/max result count: 0 / 10 Metrics for Query 5: Count: 5 times executed in whole run AQET: 0.087542 seconds (arithmetic mean) AQET(geom.): 0.087327 seconds (geometric mean) QPS: 11.42 Queries per second minQET/maxQET: 0.08165600s / 0.09889200s Average result count: 5.0 min/max result count: 5 / 5 Metrics for Query 6: Count: 5 times executed in whole run AQET: 0.131222 seconds (arithmetic mean) AQET(geom.): 0.131216 seconds (geometric mean) QPS: 7.62 Queries per second minQET/maxQET: 0.12924200s / 0.13298200s Average result count: 3.6 min/max result count: 3 / 5 Metrics for Query 7: Count: 20 times executed in whole run AQET: 0.043601 seconds (arithmetic mean) AQET(geom.): 0.040890 seconds (geometric mean) QPS: 22.94 Queries per second minQET/maxQET: 0.01984400s / 0.06012600s Average result count: 26.4 min/max result count: 5 / 96 Metrics for Query 8: Count: 10 times executed in whole run AQET: 0.018168 seconds (arithmetic mean) AQET(geom.): 0.016205 seconds (geometric mean) QPS: 55.04 Queries per second minQET/maxQET: 0.01097600s / 0.05066900s Average result count: 12.8 min/max result count: 6 / 20 Metrics for Query 9: Count: 20 times executed in whole run AQET: 0.043813 seconds (arithmetic mean) AQET(geom.): 0.043807 seconds (geometric mean) QPS: 22.82 Queries per second minQET/maxQET: 0.04274900s / 0.04504100s Average result count: 0.0 min/max result count: 0 / 0 Metrics for Query 10: Count: 15 times executed in whole run AQET: 0.030697 seconds (arithmetic mean) AQET(geom.): 0.029651 seconds (geometric mean) QPS: 32.58 Queries per second minQET/maxQET: 0.02072000s / 0.03975700s Average result count: 1.1 min/max result count: 0 / 4 real 0 m 5.485 s user 0 m 2.233 s sys 0 m 0.170 s </pre></blockquote> <p>Of the approximately 5.5 seconds of running five query mixes, the test driver spends 2.2 s. The server side processing time is 3.1 s, of which SQL compilation is 1.35 s. The rest is miscellaneous system time. The measurement is on 64-bit Linux, 2GHz dual-Xeon 5130 (8 cores) with 8G RAM. </p> <p>We note that this type of workload would be done with stored procedures or prepared, parameterized queries in the SQL world.</p> <p>There will be some further tuning still but this addresses the bulk of the matter. There will be a separate message about the patch containing these improvements.</p>
DBpedia Benchmark Revisited
http://www.openlinksw.com/weblog/oerling/?date=2008-05-09#1358
2008-05-09T19:27:00Z
2008-05-12T11:24:36-04:00
<p>We ran the <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x1b7f9688">DBpedia</a> benchmark queries again with different configurations of <a href="http://virtuoso.openlinksw.com" id="link-id0x1cca2e00">Virtuoso</a>. I had not studied the details of the matter previously but now did have a closer look at the queries.</p> <p>Comparing numbers given by different parties is a constant problem. In the case reported here, we loaded the full DBpedia 3, all languages, with about 198M triples, onto Virtuoso v5 and Virtuoso Cluster v6, all on the same 4 core 2GHz Xeon with 8G RAM. All databases were striped on 6 disks. The Cluster configuration was with 4 processes in the same box.</p> <p>We ran the queries in two variants:</p> <ul> <li>With graph specified in the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1b77f758">SPARQL</a> <code>FROM</code> clause, using the default indices.</li> <li>With no graph specified anywhere, using an alternate indexing scheme.</li> </ul> <p>The times below are for the sequence of 5 queries; individual query times are not reported. I did not do a line-by-line review of the execution plans since they seem to run well enough. We could get some extra mileage from cost model tweaks, especially for the numeric range conditions, but we will do this when somebody comes up with better times.</p> <p>First, about Virtuoso v5: Because there is a query in the set that specifies no condition on S or O and only P, this simply cannot be done with the default indices. With Virtuoso Cluster v6 it sort-of can, because v6 is more space efficient.</p> <p>So we added the index:</p> <blockquote> <code> create bitmap index <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1cb0b180">rdf</a>_quad_pogs on rdf_quad (p, o, g, s); </code> </blockquote> <table> <tr> <td>Â </td> <td align="center"><b>Virtuoso v5 with<br /> gspo, ogps, pogs</b> </td> <td align="center"><b>Virtuoso Cluster v6 with <br />gspo, ogps</b> </td> <td align="center"><b>Virtuoso Cluster v6 with <br />gspo, ogps, pogs</b> </td> </tr> <tr> <td><b>cold</b> </td> <td align="center">210 s</td> <td align="center">136 s</td> <td align="center">33.4 s</td> </tr> <tr> <td><b>warm</b> </td> <td align="center">0.600 s</td> <td align="center">4.01 s</td> <td align="center">0.628 s</td> </tr> </table> <p>OK, so now let us do it without a graph being specified. For all platforms, we drop any existing indices, and --</p> <blockquote> <code> create table r2 (g iri_id_8, s, iri_id_8, p iri_id_8, o any, primary key (s, p, o, g)) <br /> alter index R2 on R2 partition (s int (0hexffff00)); <br /> <br /> log_enable (2); <br /> insert into r2 (g, s, p, o) select g, s, p, o from rdf_quad; <br /> <br /> drop table rdf_quad; <br /> alter table r2 rename RDF_QUAD; <br /> create bitmap index rdf_quad_opgs on rdf_quad (o, p, g, s) partition (o varchar (-1, 0hexffff)); <br /> create bitmap index rdf_quad_pogs on rdf_quad (p, o, g, s) partition (o varchar (-1, 0hexffff)); <br /> create bitmap index rdf_quad_gpos on rdf_quad (g, p, o, s) partition (o varchar (-1, 0hexffff)); </code> </blockquote> <p>The code is identical for v5 and v6, except that with v5 we use <code>iri_id (32 bit)</code> for the type, not <code>iri_id_8 (64 bit)</code>. We note that we run out of IDs with v5 around a few billion triples, so with v6 we have double the ID length and still manage to be vastly more space efficient.</p> <p>With the above 4 indices, we can query the <a href="http://dbpedia.org/resource/Data" id="link-id0x6339b80">data</a> pretty much in any combination without hitting a full scan of any index. We note that all indices that do not begin with s end with s as a bitmap. This takes about 60% of the space of a non-bitmap index for data such as DBpedia.</p> <p>If you intend to do completely arbitrary RDF queries in Virtuoso, then chances are you are best off with the above index scheme.</p> <table> <tr> <td>Â </td> <td align="center"><b> Virtuoso v5 with<br /> gspo, ogps, pogs</b> </td> <td align="center"><b> Virtuoso Cluster v6 with <br /> spog, pogs, opgs, gpos </b> </td> </tr> <tr> <td><b>warm</b> </td> <td align="center">0.595 s</td> <td align="center">0.617 s</td> </tr> </table> <p>The cold times were about the same as above, so not reproduced.</p> <h3>Graph or No Graph?</h3> <p>It is in the SPARQL spirit to specify a graph and for pretty much any application, there are entirely sensible ways of keeping the data in graphs and specifying which ones are concerned by queries. This is why Virtuoso is set up for this by default.</p> <p>On the other hand, for the open web scenario, dealing with an unknown large number of graphs, enumerating graphs is not possible and questions like which graph of which source asserts x become relevant. We have two distinct use cases which warrant different setups of the database, simple as that.</p> <p>The latter use case is not really within the SPARQL spec, so implementations may or may not support this. For example <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x11ed7028">Oracle</a> or Vertica would not do this well since they partition data according to graph or predicate, respectively. On the other hand, stores that work with one quad table, which is most of the ones out there, should do it maybe with some configuring, as shown above.</p> <p>Frameworks like Jena are not to my <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x1a49ded0">knowledge</a> geared towards having a wildcard for graph, although I would suppose this can be arranged by adding some "super-graph" object, a graph of all graphs. I don't think this is directly supported and besides most apps would not need it.</p> <p>Once the indices are right, there is no difference between specifying a graph and not specifying a graph with the queries considered. With more complex queries, specifying a graph or set of graphs does allow some optimizations that cannot be done with no graph specified. For example, bitmap intersections are possible only when all leading key parts are given.</p> <h3>Conclusions</h3> <p>The best warm cache time is with v5; the five queries run under 600 ms after the first go. This is noted to show that all-in-memory with a single thread of execution is hard to beat.</p> <p>Cluster v6 performs the same queries in 623 ms. What is gained in parallelism is lost in latency if all operations complete in microseconds. On the other hand, Cluster v6 leaves v5 in the dust in any situation that has less than 100% hit rate. This is due to actual benefit from parallelism if operations take longer than a few microseconds, such as in the case of disk reads. Cluster v6 has substantially better data layout on disk, as well as fewer pages to load for the same content.</p> <p>This makes it possible to run the queries without the pogs index on Cluster v6 even when v5 takes prohibitively long.</p> <p>The morale of the story is to have a lot of RAM and space-efficient data representation.</p> <p>The DBpedia benchmark does not specify any random access pattern that would give a measure of sustained throughput under load, so we are left with the extremes of cold and warm cache of which neither is quite realistic.</p> <p>Chris Bizer and I have talked on and off about benchmarks and I have made suggestions that we will see incorporated into the Berlin SPARQL benchmark, which will, I believe, be much more informative.</p> <h3>Appendix: Query Text</h3> <p>For reference, the query texts specifying the graph are below. To run without specifying the graph, just drop the <code>FROM <<a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1905bfd0">http</a>://dbpedia.org></code> from each query. The returned row counts are indicated below each query's text.</p> <blockquote> <code><pre> sparql SELECT ?p ?o FROM <http://dbpedia.org> WHERE { <http://dbpedia.org/resource/Metropolitan_Museum_of_Art> ?p ?o }; -- 1337 rows sparql PREFIX p: <http://dbpedia.org/property/> SELECT ?film1 ?actor1 ?film2 ?actor2 FROM <http://dbpedia.org> WHERE { ?film1 p:starring <http://dbpedia.org/resource/Kevin_Bacon> . ?film1 p:starring ?actor1 . ?film2 p:starring ?actor1 . ?film2 p:starring ?actor2 . }; -- 23910 rows sparql PREFIX p: <http://dbpedia.org/property/> SELECT ?artist ?artwork ?museum ?director FROM <http://dbpedia.org> WHERE { ?artwork p:artist ?artist . ?artwork p:museum ?museum . ?museum p:director ?director }; -- 303 rows sparql PREFIX geo: <http://www.w3.org/2003/01/geo/wgs84_pos#> PREFIX foaf: <http://xmlns.com/foaf/0.1/> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> SELECT ?s ?homepage FROM <http://dbpedia.org> WHERE { <http://dbpedia.org/resource/Berlin> geo:lat ?berlinLat . <http://dbpedia.org/resource/Berlin> geo:long ?berlinLong . ?s geo:lat ?lat . ?s geo:long ?long . ?s foaf:homepage ?homepage . FILTER ( ?lat <= ?berlinLat + 0.03190235436 && ?long >= ?berlinLong - 0.08679199218 && ?lat >= ?berlinLat - 0.03190235436 && ?long <= ?berlinLong + 0.08679199218) }; -- 56 rows sparql PREFIX geo: <http://www.w3.org/2003/01/geo/wgs84_pos#> PREFIX foaf: <http://xmlns.com/foaf/0.1/> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> PREFIX p: <http://dbpedia.org/property/> SELECT ?s ?a ?homepage FROM <http://dbpedia.org> WHERE { <http://dbpedia.org/resource/New_York_City> geo:lat ?nyLat . <http://dbpedia.org/resource/New_York_City> geo:long ?nyLong . ?s geo:lat ?lat . ?s geo:long ?long . ?s p:architect ?a . ?a foaf:homepage ?homepage . FILTER ( ?lat <= ?nyLat + 0.3190235436 && ?long >= ?nyLong - 0.8679199218 && ?lat >= ?nyLat - 0.3190235436 && ?long <= ?nyLong + 0.8679199218) }; -- 13 rows </pre> </code> </blockquote>
Virtuoso 5.0.6 Updates
http://www.openlinksw.com/weblog/oerling/?date=2008-03-25#1326
2008-03-25T16:59:08Z
2008-03-26T11:59:02-04:00
<p>I will here summarize the developments since the last <a href="http://data.openlinksw.com/oplweb/product_family/virtuoso#this" id="link-id15843368">Virtuoso</a> 5 Open Source release.</p> <p>On the <a href="http://dbpedia.org/resource/RDF" id="link-id101cae58">RDF</a> side, the bitmap intersection join has been improved quite a bit so that it is now almost always more than 2x more efficient than the equivalent nested loop join.</p> <p>XML trees in the object position in <a href="http://dbpedia.org/resource/RDF" id="link-id1172c108">RDF</a> quads were in some cases incorrectly indexed, leading to failure to retrieve quads. This is fixed and should problems occur in existing databases, they can be corrected by simply dropping and re-creating an index.</p> <p>Also the cost model has been further tuned. We have run the <a href="http://dbpedia.org/resource/TPC-H" id="link-idd65a998">TPC</a>-<a href="http://dbpedia.org/resource/TPC-H" id="link-id11a2bf48">H</a> queries with larger databases and have profiled it extensively. There are improvements to locking, especially for concurrency of transactions with large shared lock sets, as is the case in the <a href="http://dbpedia.org/resource/TPC-H" id="link-id12cfd690">TPC</a>-<a href="http://dbpedia.org/resource/TPC-H" id="link-id15891ae0">H</a> queries. The rules stipulate that these have to be run with repeatable read. There are also optimizations for decimal floating point.</p> <p>A sampling of <a href="http://dbpedia.org/resource/TPC-H" id="link-id15b12eb0">TPC</a>-<a href="http://dbpedia.org/resource/TPC-H" id="link-id1172c740">H</a> queries translated into <a href="http://dbpedia.org/resource/SPARQL" id="link-id15533aa8">SPARQL</a> comes with the new demo database. These show a live sample of the <a href="http://dbpedia.org/resource/TPC-H" id="link-id15b82cd8">TPC</a>-<a href="http://dbpedia.org/resource/TPC-H" id="link-id15521d50">H</a> schema translated into linked data, complete with <a href="http://dbpedia.org/resource/SPARQL" id="link-id15ae14d8">SPARQL</a> translations of the original queries. Some work is still ongoing there but the relational to <a href="http://dbpedia.org/resource/RDF" id="link-id101b1240">RDF</a> mapping is mature enough for real business intelligence applications now.</p> <p>On the closed source side, we have some adjustments to the virtual database. When using <a href="http://data.openlinksw.com/oplweb/product_family/virtuoso#this" id="link-id10888df0">Virtuoso</a> as a front end to Oracle, using the <a href="http://dbpedia.org/resource/TPC-H" id="link-id134a9378">TPC</a>-<a href="http://dbpedia.org/resource/TPC-H" id="link-id10366320">H</a> queries as a metric, the virtual database overhead is minimal. Previously, we had some overhead because some queries were rewritten in a way that Oracle would not optimize as well as the original <a href="http://dbpedia.org/resource/TPC-H" id="link-id15536320">TPC</a>-<a href="http://dbpedia.org/resource/TPC-H" id="link-idfd4e278">H</a> text. Specifically, turning an IN sub-query predicate into an equivalent EXISTS did not sit well with Oracle.</p>
TPC H as Linked Data (Updated 2)
http://www.openlinksw.com/weblog/oerling/?date=2008-03-06#1321
2008-03-06T16:22:03Z
2008-08-28T11:25:55-04:00
<p>We have a new demo online at <a href="http://demo.openlinksw.com/tpc-h" id="link-id1829c9a0">http://demo.openlinksw.com/tpc-h</a>. This takes the industry standard <a href="http://dbpedia.org/resource/TPC-H" id="link-id0xeb7e460">TPC-H</a> benchmark <a href="http://dbpedia.org/resource/Data" id="link-id0xb40fcb8">data</a> and presents it as <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x9edbd128">linked data</a> with a <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xf566a50">SPARQL</a> end point and dereferenceable URIs. </p> <p>This is an example of using <a href="http://virtuoso.openlinksw.com" id="link-id0x11e59f80">Virtuoso</a>'s relational-to-<a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xfc93c70">RDF</a> mapping for publishing business data, for browsing using the linked data principles and opening it to analytics queries in SPARQL.</p> <p> As noted before, we have extended SPARQL with aggregation and nested queries, thus making it a viable <a href="http://dbpedia.org/resource/SQL" id="link-id0xffe4520">SQL</a> substitute for decision support queries. </p> <p>The article at <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSTPCHLinkedData" id="link-id10799d10">http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSTPCHLinkedData</a> gives details and the source code for the implementation.</p> <p> We are still working on some aspects of the more complex TPC-H queries, thus the demo is not complete with all the 22 queries. This is however enough to see a representative sample of how analytics queries work with SPARQL and Virtuoso's SQL-to-RDF mapping. The demo will be part of the next Virtuoso Open Source download, probably out next week.</p>
What's Wrong With LUBM?
http://www.openlinksw.com/weblog/oerling/?date=2008-02-05#1312
2008-02-05T11:47:11Z
2008-03-25T14:43:37-04:00
<p>In the interest of participating in a community benchmark development process, I will here outline some desiderata and explain how we could improve on LUBM. I will also touch on the message such an effort ought to convey.</p> <p>A blow-by-blow analysis of the performance of a complex system such as a DBMS is more than fits within the scope of human attention at one go. This is why this all must be abbreviated into a single metric. Only when thus abbreviated, can this information be used in context. The metric's practical value is relative to how well it predicts the performance of the system in some real task. This means a task not likely to be addressed by an alternative technology, unless the challenger clearly beats the incumbent.</p> <p>A benchmark is promotional material, both well as for the technology being benchmarked as a whole. This is why the benchmark, whatever it does, should do something that the technology does well, surely better than any alternative technology. A case in point is that one ought not to take a pure relational workload and <a href="http://dbpedia.org/resource/RDF" id="link-id0x189e2b18">RDF</a>-ize it, for then the relational variant is likely to come out on top.</p> <p>In this regard LUBM is not so bad because its reliance on class and property hierarchies and the occasional transitivity or inference rule makes the workload typically <a href="http://dbpedia.org/resource/RDF" id="link-id0x17fe8588">RDF</a>, a little ways apart from a purely relational implementation of the task.</p> <p> <a href="http://dbpedia.org/resource/RDF" id="link-id0x1802e258">RDF</a>'s claim to fame is linked data. This means giving things globally unique names and thereby making anything joinable with anything else, insofar there is agreement on the names. <a href="http://dbpedia.org/resource/RDF" id="link-id0xa04d16c0">RDF</a> is a key to a new class of problems, call it web scale database. Web scale here refers first to heterogeneity and multiplicity of independent sources and secondly to volume of data.</p> <p>Now there are plenty of relational applications with very large volumes of data. On the non-relational side, there are even larger applications, such as web search engines. All these have a set schema and a specific workload they are meant to address. <a href="http://dbpedia.org/resource/RDF" id="link-id0x1971d010">RDF</a> versions of such are conceivable but hold no intrinsic advantage if considered in the specific niche alone.</p> <p>The claim to fame of <a href="http://dbpedia.org/resource/RDF" id="link-id0x18d9ace0">RDF</a> is not to outperform these on their home turf but to open another turf altogether, allowing agile joining and composing of all these resources.</p> <p>This is why a benchmark, i.e., an an advertisement for the <a href="http://dbpedia.org/resource/RDF" id="link-id0x1be2af60">RDF</a> value proposition, should not just take a relational workload and <a href="http://dbpedia.org/resource/RDF" id="link-id0x18041730">RDF</a>-ize it. The benchmark should carry some of the web in it.</p> <p>If we just intend to measure how well an <a href="http://dbpedia.org/resource/RDF" id="link-id0x1cb49658">RDF</a> store joins triples to other triples, LUBM is almost good enough. If it defined a query mix with different frequencies for short and long queries and a concurrent query metric, it would be pretty much there. Our adaptation of it is adequate for counting joins per second. But joins per second is not a value proposition.</p> <p>So we have two questions: </p> <ol> <li> <p>If we just take the <a href="http://dbpedia.org/resource/RDF" id="link-id0x19cec640">RDF</a> model and <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1998cc70">SPARQL</a>, how do we make a benchmark that fills in what LUBM does not cover?</p> </li> <li>How do we make a benchmark that displays <a href="http://dbpedia.org/resource/RDF" id="link-id0xbd74610">RDF</a>'s strengths against a comparable relational solution? A priori, by going somewhere where SQL has trouble reaching.</li> </ol> <p>The answers to the first are not very complex:</p> <ul> <li> <p>Add some optionals. Have different frequencies of occurrence for some properties.</p> </li> <li> <p>Add different graphs. Make queries joining between graphs and drawing on different graphs. Querying against all graphs of the store is not a part of the language. Still this would be useful but leave it out for now.</p> </li> <li> <p>Add some filters and arithmetic. Not much can be done there, though because expressions cannot be returned and there is no aggregation or grouping.</p> </li> <li> <p>Split the workload into short and long queries. The short should be typical for online use and the long ones for analysis. Different execution frequencies for different queries is a must. Analysis is limited by lack of grouping, expressions or aggregation. Still, something can be contrived by looking for a pattern that does not exist or occurs extremely rarely. Producing result sets of millions of rows is not realistic.</p> </li> <li> <p>Many of the LUBM queries return thousands of rows, even when scoped to a single university. This is not very realistic. No user interface displays that sort of quantity. Of course, the intermediate results can be large as you please but the output must be somehow ranked. <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xd623d40">SPARQL</a> has order by and limit, so these will have to be used. <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x18032180">TPC</a> <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x1a130188">H</a> for example has almost always a group by/order by combination and sometimes a result rows limit.</p> </li> <li> <p>The degree of inference in LUBM is about right, mostly sub-classes and sub-properties, nothing complex. We certainly regard this as a database benchmark more than a knowledge representation or rule system one. </p> </li> <li> <p>LUBM does an OK job of defining a scale factor. I think that a concurrent query metric can just be so many queries per time at a given scale. The number of clients, I would say, can be decided by the test sponsor, taking whatever works best. A load balancer or web server can always be tuned to enforce some limit on concurrency. I don't think that a scale rule like in TPC C, where it says that only so many transactions per minute are allowed per warehouse is needed here. The effect of this is that when reporting a higher throughput, one has to automatically have a bigger database.</p> </li> </ul> <p>There is nothing to prevent these improvements from being put into a subsequent version of LUBM.</p> <p>Building something that shows <a href="http://dbpedia.org/resource/RDF" id="link-id0x18034830">RDF</a> at its best is a slightly different proposition. For this, we cannot be limited to the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x18d990a0">SPARQL</a> recommendation and must allow custom application code and language extensions. Examples would be scripting similar to SQL stored procedures and extensions such as we have made for sub-queries and aggregation, explained a couple of posts back. </p> <p>Maybe the Billion Triples challenge produces some material that we can use for this. We need to go for spaces that are not easily reached with SQL, have distributed computing, federation, discovery, demand driven import of data and such like. </p> <p>I'll write more about ways of making <a href="http://dbpedia.org/resource/RDF" id="link-id0x19c403c8">RDF</a> shine in some future post. </p> <p>There are two kinds of workloads: online and offline. Online is what must be performed in an interactive situation, without significant human perceptible delay, i.e. within 500 ms. Anything else is offline. </p> <p>Because this is how any online system is designed, this should be reflected in the benchmark. Ideally we would make two benchmarks.</p>
LUBM results with Virtuoso 6.0
http://www.openlinksw.com/weblog/oerling/?date=2008-02-04#1308
2008-02-04T09:58:03Z
2008-08-28T12:06:04-04:00
<p>We have now run the LUBM benchmark on <a href="http://virtuoso.openlinksw.com" id="link-id0x1a6cb3c8">Virtuoso</a> v6, with the same configuration <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1302" id="link-id107f0238">as discussed last Friday</a>.</p> <p>We had a database of 8000 universities, and we ran 8 clients on slices of 100, 1000 and 8000 universities — same <a href="http://dbpedia.org/resource/Data" id="link-id0x12ac6cc8">data</a> but different sizes of working set.</p> <blockquote> <pre> 100 universities: 35.3 qps 1000 universities: 26.3 qps 8000 universities: 13.1 qps</pre></blockquote> <p>The 100 universities slice is about the same as with v5.0.5 (35.3 vs 33.1 qps). <br />The 8000 universities set is almost 3x better (13.1 vs. 4.8 qps).</p> <p>This comes from the fact that the v6 database takes half of the space of the v5.0.5 one. Further, this is with 64-bit IDs for everything. If the 5.5 database were with 64-bit IDs, we'd have a difference of over 3x. This is worth something if it lets you get by with only 1 terabyte of RAM for the 100 billion triple application, instead of 3 TB.</p> <p> <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1358" id="link-id15fb4d38">In a few more days</a>, we'll give the results for Virtuoso v6 Cluster.</p>
LUBM results with Virtuoso 6.0
http://www.openlinksw.com/weblog/oerling/?date=2008-02-04#1308
2008-02-04T09:58:03Z
2008-08-28T12:06:04-04:00
<p>We have now run the LUBM benchmark on <a href="http://virtuoso.openlinksw.com" id="link-id0x1a6cb3c8">Virtuoso</a> v6, with the same configuration <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1302" id="link-id107f0238">as discussed last Friday</a>.</p> <p>We had a database of 8000 universities, and we ran 8 clients on slices of 100, 1000 and 8000 universities — same <a href="http://dbpedia.org/resource/Data" id="link-id0x12ac6cc8">data</a> but different sizes of working set.</p> <blockquote> <pre> 100 universities: 35.3 qps 1000 universities: 26.3 qps 8000 universities: 13.1 qps</pre></blockquote> <p>The 100 universities slice is about the same as with v5.0.5 (35.3 vs 33.1 qps). <br />The 8000 universities set is almost 3x better (13.1 vs. 4.8 qps).</p> <p>This comes from the fact that the v6 database takes half of the space of the v5.0.5 one. Further, this is with 64-bit IDs for everything. If the 5.5 database were with 64-bit IDs, we'd have a difference of over 3x. This is worth something if it lets you get by with only 1 terabyte of RAM for the 100 billion triple application, instead of 3 TB.</p> <p> <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1358" id="link-id15fb4d38">In a few more days</a>, we'll give the results for Virtuoso v6 Cluster.</p>
Latest LUBM Benchmark results for Virtuoso
http://www.openlinksw.com/weblog/oerling/?date=2008-02-01#1304
2008-02-01T14:39:04Z
2008-08-28T12:06:01-04:00
<p>We have now taken a close look at the query side of the LUBM benchmark, <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1296" id="link-id10a98120">as promised a couple of blog posts ago.</a> </p> <p>We load 8000 universities and run a query mix consisting of the 14 LUBM queries with different numbers of clients against different portions of the database.</p> <p>When it is all in memory, we get 33 queries per second with 8 concurrent clients; when it is so I/O bound that 7.7 of 8 threads wait for disk, we get 5 qps. This was run in 8G RAM with 2 Xeon 5130.</p> <p>We adapted some of the queries so that they do not run over the whole database. In terms of retrieving triples per second, this would be about 330000 for the rate of 33 qps, with 4 cores at 2GHz. This is a combination of random access and linear scans and bitmap merge intersections; lookups for non-found triples are not counted. The rate of random lookups alone based on known G, S, P, O, without any query logic, is about 250000 random lookups per core per second.</p> <p>The article <a href="http://virtuoso.openlinksw.com/wiki/main/Main/VOSArticleLUBMBenchmark" id="link-id10237708">LUBM and Virtuoso</a> gives the details.</p> <p>In the process of going through the workload we made some cost model adjustments and optimized the bitmap intersection join. In this way we can quickly determine which subjects are, for example, professors holding a degree from a given university. So the benchmark served us well in that it provided an incentive to further optimize some things.</p> <p>Now, what has been said about <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x104257c0">RDF</a> benchmarking previously still holds. What does it mean to do so many LUBM queries per second? What does this say about the capacity to run an online site off RDF <a href="http://dbpedia.org/resource/Data" id="link-id0x7376478">data</a>? Or about <a href="http://dbpedia.org/resource/Information" id="link-id0x13fd3f30">information</a> integration? Not very much. But then this was not the aim of the authors either.</p> <p>So we still need to make a benchmark for online queries and search, and another for E-science and business intelligence. But we are getting there.</p> <p>In the immediate future, we have the general availability of <a href="http://virtuoso.openlinksw.com" id="link-id0x193509e8">Virtuoso</a> Open Source 5.0.5 early next week. This comes with a LUBM test driver and a test suite running against the LUBM qualification database.</p> <p>After this we will give some numbers for the cluster edition with LUBM and <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x1b8d1348">TPC-H</a>.</p>
Latest LUBM Benchmark results for Virtuoso
http://www.openlinksw.com/weblog/oerling/?date=2008-02-01#1304
2008-02-01T14:39:04Z
2008-08-28T12:06:01-04:00
<p>We have now taken a close look at the query side of the LUBM benchmark, <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1296" id="link-id10a98120">as promised a couple of blog posts ago.</a> </p> <p>We load 8000 universities and run a query mix consisting of the 14 LUBM queries with different numbers of clients against different portions of the database.</p> <p>When it is all in memory, we get 33 queries per second with 8 concurrent clients; when it is so I/O bound that 7.7 of 8 threads wait for disk, we get 5 qps. This was run in 8G RAM with 2 Xeon 5130.</p> <p>We adapted some of the queries so that they do not run over the whole database. In terms of retrieving triples per second, this would be about 330000 for the rate of 33 qps, with 4 cores at 2GHz. This is a combination of random access and linear scans and bitmap merge intersections; lookups for non-found triples are not counted. The rate of random lookups alone based on known G, S, P, O, without any query logic, is about 250000 random lookups per core per second.</p> <p>The article <a href="http://virtuoso.openlinksw.com/wiki/main/Main/VOSArticleLUBMBenchmark" id="link-id10237708">LUBM and Virtuoso</a> gives the details.</p> <p>In the process of going through the workload we made some cost model adjustments and optimized the bitmap intersection join. In this way we can quickly determine which subjects are, for example, professors holding a degree from a given university. So the benchmark served us well in that it provided an incentive to further optimize some things.</p> <p>Now, what has been said about <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x104257c0">RDF</a> benchmarking previously still holds. What does it mean to do so many LUBM queries per second? What does this say about the capacity to run an online site off RDF <a href="http://dbpedia.org/resource/Data" id="link-id0x7376478">data</a>? Or about <a href="http://dbpedia.org/resource/Information" id="link-id0x13fd3f30">information</a> integration? Not very much. But then this was not the aim of the authors either.</p> <p>So we still need to make a benchmark for online queries and search, and another for E-science and business intelligence. But we are getting there.</p> <p>In the immediate future, we have the general availability of <a href="http://virtuoso.openlinksw.com" id="link-id0x193509e8">Virtuoso</a> Open Source 5.0.5 early next week. This comes with a LUBM test driver and a test suite running against the LUBM qualification database.</p> <p>After this we will give some numbers for the cluster edition with LUBM and <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x1b8d1348">TPC-H</a>.</p>
LUBM and Virtuoso 5.5
http://www.openlinksw.com/weblog/oerling/?date=2008-02-01#1302
2008-02-01T12:37:53Z
2008-03-25T14:43:30.000004-04:00
<p>We have now taken a close look at the query side of the LUBM benchmark, as promised a couple of blog posts ago.</p> <p>We load 8000 universities and run a query mix consisting of the 14 LUBM queries with different numbers of clients against different portions of the database.</p> <p>When it is all in memory, we get 33 queries per second with 8 concurrent clients; when it is so I/O bound that 7.7 of 8 threads wait for disk, we get 5 qps. This was run in 8G RAM with 2 Xeon 5130.</p> <p>We adapted some of the queries so that they do not run over the whole database. In terms of retrieving triples per second, this would be about 330000 for the rate of 33 qps, with 4 cores at 2GHz. This is a combination of random access and linear scans and bitmap merge intersections; lookups for non-found triples are not counted. The rate of random lookups alone based on known G, S, P, O, without any query logic, is about 250000 random lookups per core per second.</p> <p>The article <a href="http://virtuoso.openlinksw.com/wiki/main/Main/VOSArticleLUBMBenchmark" id="link-id121f54d8">LUBM and Virtuoso</a> gives the details.</p> <p>In the process of going through the workload, we made some cost model adjustments and optimized the bitmap intersection join. In this way we can quickly determine which subjects are, for example, professors holding a degree from a given university. So the benchmark served us well in that it provided an incentive to further optimize some things.</p> <p>Now, what has been said about <a href="http://dbpedia.org/resource/RDF" id="link-id0x186bcec0">RDF</a> benchmarking previously still holds. What does it mean to do so many LUBM queries per second? What does this say about the capacity to run an online site off <a href="http://dbpedia.org/resource/RDF" id="link-id0xa1e11918">RDF</a> data? Or about information integration? Not very much. But then this was not the aim of the authors either.</p> <p>So we still need to make a benchmark for online queries and search, and another for E-science and business intelligence. But we are getting there.</p> <p>In the immediate future, we have the general availability of <a href="http://data.openlinksw.com/oplweb/product_family/virtuoso#this" id="link-id0x189b11b8">Virtuoso</a> Open Source 5.0.5 early next week. This comes with a LUBM test driver and a test suite running against the LUBM qualification database.</p> <p>After this we will give some numbers for the cluster edition with LUBM and <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x1b948dd8">TPC</a> <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x189e58a8">H</a>.</p>
SPARQL Extensions for Subqueries
http://www.openlinksw.com/weblog/oerling/?date=2008-01-16#1296
2008-01-16T15:11:00Z
2008-03-25T14:43:28.000001-04:00
<p>Last time I said we had extended <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1c668bb0">SPARQL</a> for sub-queries. As a preview of the new functionality, let us look at a query from <a href="http://dbpedia.org/resource/TPC-H" id="link-id0xd469258">TPC</a> <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x1c68fe58">H</a>.</p> <p>Below is the <a href="http://data.openlinksw.com/oplweb/product_family/virtuoso#this" id="link-id0xcabe1f0">Virtuoso</a> <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1d3ba3d0">SPARQL</a> version of Q2.</p> <blockquote> <pre> <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xd45b8c0">sparql</a> define sql:signal-void-variables 1 prefix tpcd: <http://www.openlinksw.com/schemas/tpcd#> prefix oplsioc: <http://www.openlinksw.com/schemas/oplsioc#> prefix sioc: <http://rdfs.org/sioc/ns#> prefix foaf: <http://xmlns.com/foaf/0.1/> select ?supp+>tpcd:acctbal, ?supp+>tpcd:name, ?supp+>tpcd:has_nation+>tpcd:name as ?nation_name, ?part+>tpcd:partkey, ?part+>tpcd:mfgr, ?supp+>tpcd:address, ?supp+>tpcd:phone, ?supp+>tpcd:comment from <http://example.com/tpcd> where { ?ps a tpcd:partsupp ; tpcd:has_supplier ?supp ; tpcd:has_part ?part . ?supp+>tpcd:has_nation+>tpcd:has_region tpcd:name 'EUROPE' . ?part tpcd:size 15 . ?ps tpcd:supplycost ?minsc . { select ?p min(?ps+>tpcd:supplycost) as ?minsc where { ?ps a tpcd:partsupp ; tpcd:has_part ?p ; tpcd:has_supplier ?ms . ?ms+>tpcd:has_nation+>tpcd:has_region tpcd:name 'EUROPE' . } } filter (?part+>tpcd:type like '%BRASS') } order by desc (?supp+>tpcd:acctbal) ?supp+>tpcd:has_nation+>tpcd:name ?supp+>tpcd:name ?part+>tpcd:partkey ; </pre></blockquote> <p>Note the pattern <code>{ ?ms+>tpcd:has_nation+>tpcd:has_region tpcd:name 'EUROPE' }</code> which is a shorthand for <code>{ ?ms tpcd:has_nation ?t1 . ?t1 tpcd:has-region ?t2 . ?t2 tpcd:has_region ?t3 . ?t3 tpcd:name "EUROPE" } </code> </p> <p>Also note a sub-query is used for determining the lowest supply cost for a part. </p> <p>The SQL text of the query can be found in the <a href="http://dbpedia.org/resource/TPC-H" id="link-id0xb945ed8">TPC</a> <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x1c832038">H</a> benchmark specification, reproduced below: </p> <blockquote> <pre>select s_acctbal, s_name, n_name, p_partkey, p_mfgr, s_address, s_phone, s_comment from part, supplier, partsupp, nation, region where p_partkey = ps_partkey and s_suppkey = ps_suppkey and p_size = 15 and p_type like '%BRASS' and s_nationkey = n_nationkey and n_regionkey = r_regionkey and r_name = 'EUROPE' and ps_supplycost = ( select min(ps_supplycost) from partsupp, supplier, nation, region where p_partkey = ps_partkey and s_suppkey = ps_suppkey and s_nationkey = n_nationkey and n_regionkey = r_regionkey and r_name = 'EUROPE') order by s_acctbal desc, n_name, s_name, p_partkey; </pre></blockquote> <p> For brevity we have omitted the declarations for mapping the <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x1d365d90">TPC</a> <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x17b6f668">H</a> schema to its <a href="http://dbpedia.org/resource/RDF" id="link-id0x1898eaf0">RDF</a> equivalent. The mapping is straightforward, with each column mapping to a predicate and each table to a class.</p> <p> This is now part of the next <a href="http://data.openlinksw.com/oplweb/product_family/virtuoso#this" id="link-id0x1aaaa3f0">Virtuoso</a> Open Source cut, due around next week. </p> <p> As of this writing we are going through the <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x1950f1c0">TPC</a> <a href="http://dbpedia.org/resource/TPC-H" id="link-id0xd0d8bc0">H</a> query by query and testing with mapping going to <a href="http://data.openlinksw.com/oplweb/product_family/virtuoso#this" id="link-id0x18b88f08">Virtuoso</a> and Oracle databases. </p> <p> Also we have been busy measuring <a href="http://data.openlinksw.com/oplweb/product_family/virtuoso#this" id="link-id0x17b3fbb0">Virtuoso</a> 6. Even after switching from 32-bit to 64-bit IDs for IRIs and objects, the new databases are about half the size of the same <a href="http://data.openlinksw.com/oplweb/product_family/virtuoso#this" id="link-id0xbe2ec80">Virtuoso</a> 5.0.2 databases. This does not include any stream compression like gzip for disk pages. The load and query speeds are higher because of better working set. For all in memory, they are about even with 5.0.2. So now on an 8G box, we load 1067 million LUBM triples at 39.7 Kt/s instead of 29 Kt/s with 5.0.2. Right now we experimenting with clusters at Amazon EC2. We'll write about that in a bit. </p>
Retrospective and Outlook for 2008
http://www.openlinksw.com/weblog/oerling/?date=2007-12-18#1286
2007-12-18T10:53:40Z
2008-04-14T14:02:31.000004-04:00
<p>At this close of the year, I'll give a little recap of the past year in terms of <a href="http://virtuoso.openlinksw.com/" id="link-idfe5ba58">Virtuoso</a> development, and take a look at where we are headed for 2008.</p> <p>A year ago, I was in the middle of redoing the <a href="http://data.openlinksw.com/oplweb/product_family/virtuoso#this">Virtuoso</a> database engine for better SMP performance. We redid the way traversal of index structures and cache buffers was serialized for SMP, and generally compared Virtuoso and Oracle engines function by function. We had just returned from the ISWC 2006 in Athens, Georgia, and the Virtuoso database was becoming a usable triple store.</p> <p>Soon thereafter, we confirmed that all this worked when we put out the first cut of <a href="http://dbpedia.org/" id="link-id149dcff8">Dbpedia</a> with Chris Bizer, et al, and were working with Alan Ruttenberg on what would become <a href="http://esw.w3.org/topic/HCLS/Banff2007Demo" id="link-id10c25b50">the Banff health care and life sciences demo</a>.</p> <p>The <a href="http://www2007.org/" id="link-idfdbd0e0">WWW 2007 conference in Banff</a>, Canada, was a sort of kick-off for the <a href="http://esw.w3.org/topic/SweoIG/TaskForces/CommunityProjects/LinkingOpenData" id="link-id10e54940">Linking Open Data</a> movement, which started as a community project under <a href="http://www.w3.org/2001/sw/sweo/" id="link-idfd99988">SWEO</a>, the W3C interest group for Semantic Web Education and Outreach, and has gained a life of its own since.</p> <p>Right after WWW 2007, the Virtuoso development effort split onto two tracks: one for enhancing the then new 5.0 release; and one for building a new generation of Virtuoso, notably featuring clustering and double storage density for <a href="http://dbpedia.org/resource/RDF">RDF</a>.</p> <p>The first track produced constant improvements to the relational-to-RDF mapping functionality, <a href="http://dbpedia.org/resource/SPARQL">SPARQL</a> enhancements, and Redland-, Jena- and Sesame-compatible client libraries with Virtuoso as as a triple store. These things have been out with testers for a while and are all generally available as of this writing.</p> <p>The second track started with adding key compression to the storage engine, specifically with regard to RDF, even though there are some gains in relational applications as well. With RDF, the space consumption drops to about half, all without recourse to any non-random access compatible compression like gzip. Since the start of August, we turned to clustering and are now code complete, pretty much with all the tricks one would expect, of course full function SQL and taking advantage of co-located joins and doing aggregation and generally all possible processing where the data is. I have covered details of this along the way in previous posts. The key point is that now the thing is written and works with test cases. </p> <p>In late October, we were at the W3C workshop for mapping relational data to RDF. For us, this confirmed the importance of mapping and scalability in general. Ivan Herman proposed forming a W3C incubator group on benchmarking. Also a W3C incubator group of relational to RDF mapping is being formed. </p> <p>Now, scalability has two sides. One is dealing with volume, and the other is dealing with complexity. Volume alone will not help if interesting queries cannot be formulated. Hence, we recently extended SPARQL with sub-queries so that we can now express at least any SQL workloads, which was previously not the case. It is sort of a contradiction in terms to say that SPARQL is the universal language for information integration while not being able to express, for example, the <a href="http://dbpedia.org/resource/TPC-H">TPC-H</a> queries. Well, we fixed this. A separate post will highlight how. The W3C process will eventually follow, as the necessity of these things is undeniable, on the unimpeachable authority of the whole SQL world. Anyway, for now, SPARQL as it is ought to become a recommendation and extensions can be addressed later.</p> <p>For now, the only RDF benchmark that seems to be out there is the loading part of the LUBM. We did a couple of enhancements of our own for that just recently, but much bigger things are on the way. Also, the billion triples challenge is an interesting initiative in the area. We all recognize that loading any number of triples is a finite problem with known solutions. The challenge is running interesting queries on large volumes. </p> <p>Our present emphasis is demonstrating both RDF data warehousing and RDF mapping with complex queries and large data. We start with the TPC-H benchmark and doing the queries both through mapping to SQL against any RDBMS â Oracle, DB2, Virtuoso or other â and by querying the physical RDF rendition of the data in Virtuoso. From there, we move to querying a collection of RDBMS hosting similar data. </p> <p>Doing this with performance at the level of direct SQL in the case of mapping and not very much slower with physical triples is an important milestone on the way to a real world enterprise data web. Real life has harder and more unexpected issues than a benchmark, but at any rate doing the benchmark without breaking a sweat is a step on the way. We sent a paper to ESWC 2008 about that but it was rather incomplete. By the time of the VLDB submissions deadline in March we'll have more meat.</p> <p>Another tack soon to start is a re-architecting of <a href="http://zitgist.com/" id="link-id10e8b9f0">Zitgist</a> around clustered Virtuoso. Aside matters of scale, we will make a number of qualitatively new things possible. Again, more will be released in the first quarter of 2008.</p> <p>Beyond these short and mid-term goals we have the introduction of entirely dynamic and demand driven partitioning, <i>a la</i> Google Bigtable or Amazon Dynamo. Now, regular partitioning will do for a while yet but this is the future when we move the the vision of linked data everywhere.</p> <p>In conclusion, this year we have built the basis and the next year is about deployment. The bulk of really new development is behind us and now we start applying. Also, the community will find adoption easier due to our recent support of the common RDF APIs. </p> <a href="index.vspx?tag=database" rel="tag" style="display:none;">database</a><a href="index.vspx?tag=databases" rel="tag" style="display:none;">databases</a><a href="index.vspx?tag=lubm" rel="tag" style="display:none;">lubm</a><a href="index.vspx?tag=benchmarking" rel="tag" style="display:none;">benchmarking</a><a href="index.vspx?tag=semanticweb" rel="tag" style="display:none;">semanticweb</a><a href="index.vspx?tag=sparql" rel="tag" style="display:none;">sparql</a>
Virtuoso LUBM Load Update
http://www.openlinksw.com/weblog/oerling/?date=2007-12-06#1284
2007-12-06T13:35:21Z
2008-04-14T14:02:31-04:00
<p>As part of the recent conversation on benchmarking <a href="http://dbpedia.org/resource/RDF" id="link-id0x17cd8cf8">RDF</a> stores, we re-ran the LUBM 8000 load test (1067 million triples) with the current <a href="http://data.openlinksw.com/oplweb/product_family/virtuoso#this" id="link-id0x17581ae0">Virtuoso</a>.</p> <p>We did it on two different machines, one with 2 Xeon 5130 2Ghz and 8G RAM and one with 2 Xeon 5330 2GHZ and 16G RAM. Both had 6 x 7800 rpm SATA-2 drives. The load rate on the 16G configuration was 36.8 Ktriples per second. The load rate on the 8G configuration was 29.7 Ktriples per second. Both loads were made using 6 concurrent load streams. Some small changes to the numbers may be released later as a result of changing tuning.</p> <p>The Virtuoso version was 5.0, in the update to be released on the week of Dec 10, 2007. This is an incremental release of Virtuoso 5.0 and has the same engine as the prior 5.0s, with some optimizations for RDF loading and diverse bug fixes, notably in RDF mapping of relational data. This release will be further described in a separate post.</p> <p>The load does not include forward chaining but then Virtuoso supports sub-class and sub-property without materializing the entailed triples.</p> <p>Most of the LUBM entailed triples represent sub-classes and sub-properties. The LUBM query and forward chaining side deserves a separate treatment but this is for another time.</p> <p>Most recent posts on this blog refer to Virtuoso 6, which is presently under development. We will publish results with the 6.0 engine later. Also, further enhancements to triple store performance will take place on the Virtuoso 6 platform.</p>
Storage News
http://www.openlinksw.com/weblog/oerling/?date=2007-07-12#1225
2007-07-12T14:16:40Z
2008-04-24T13:22:32-04:00
<p>I have been away from the world for a few weeks, concentrating on technology.</p> <p>We have now implemented an entirely new storage layout. With <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x9fd979d0">RDF</a> <a href="http://dbpedia.org/resource/Data" id="link-id0x1ccee438">data</a>, we have now successfully doubled the working set.</p> <p>This means that the number of triples that will fit in memory is doubled for any configuration. For any database in the hundreds of millions of triples, this is very significant. For LUBM data, we go from 75b to 35b per triple with the default indices.</p> <p>This is obtained without using gzip or some other stream compression. Thus no decompression is needed at read time. Random access speeds are within 5% of those of <a href="http://virtuoso.openlinksw.com" id="link-id0x1e135ff0">Virtuoso</a> v5.0.1, but the space requirement is halved and you can still locate a random triple in cache in a few microseconds.</p> <p>What is better still, when using 8-byte IDs for IRIs instead of 4-byte ones, the space consumption stays almost the same since unique values are stored only once per page.</p> <p>When applying gzip to the new storage layout, we usually get 3x compression. This means that 99% of 8K pages fit in 3K after compression. This is no real surprise since an index is repetitive pretty much by definition, even if the repeated sections are now shorter than in v5.0.1.</p> <p>Gzip applied to pages does nothing for the working set since a page must remain random accessible for fast search but will cut disk usage to between half and a third. We will make this an option later. There are other tricks to be done with compression, like using a separate dictionary for non key text columns in relational applications. This would improve the working set in <a href="http://dbpedia.org/resource/TPC-C" id="link-id0x9fbf44c0">TPC-C</a> and TPC-D quite a bit so we may do this also while on the subject.</p> <p>Right now we are writing the clustering support, revising all internal APIs to run with batches of rows instead of single rows. We will most likely release clustering and the new storage layout together, towards the end of summer, at least in internal deployments.</p> <p>I will <a href="http://dbpedia.org/resource/Blog" id="link-id0xba13470">blog</a> about results as and when they are obtained, over the next few weeks.</p>
Comparison of Open Source Databases with TPC D Queries
http://www.openlinksw.com/weblog/oerling/?date=2007-02-05#1131
2007-02-05T11:45:32Z
2008-04-17T21:04:29-04:00
<p> <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1116" id="link-id10598cc0">Last time</a> we talked about database engine and transactions. Now we have come to the realm of query processing in our revisiting of the DBMS side of <a href="http://virtuoso.openlinksw.com" id="link-id0x1a4279e8">Virtuoso</a>.</p> <p>Now the well established, respectable standard benchmark for the basics of query processing is TPC D with its derivatives H and R. So we have, for testing how different <a href="http://dbpedia.org/resource/SQL" id="link-id0x17ce3a18">SQL</a> optimizers manage the 22 queries, run a mini version of the D queries with a 1% scale database, some 30M of <a href="http://dbpedia.org/resource/Data" id="link-id0x17370eb0">data</a>, all in memory. This basically catches whether SQL implementations miss some of the expected tricks and how efficient in memory loop and hash joins and aggregation are.</p> <p>When we get to our next stop, high volume I/O, we will run the same with D databases in the 10G ballpark.</p> <p>The databases were tested on the same machine, with warm cache, taking the best run of 3. All had full statistics and were running with read committed isolation, where applicable. The data was generated using the procedures from the Virtuoso test suite. The Virtuoso version tested was 5.0, to be released shortly. The <a href="http://dbpedia.org/resource/MySQL" id="link-id0xe435ad8">MySQL</a> was 5.0.27, the PostgreSQL was 8.1.6. </p> <table style="width: 334px; height: 556px; " border="1"> <tbody> <tr> <th rowspan="2">Query</th> <th colspan="4">Query Times in Milliseconds</th> </tr> <tr> <th> Virtuoso </th> <th> PostgreSQL </th> <th> MySQL </th> <th> MySQL with InnoDB </th> </tr> <tr> <td> Q1 </td> <td align="right"> <b>206</b> </td> <td align="right"> 763 </td> <td align="right"> 312 </td> <td align="right"> 198 </td> </tr> <tr> <td> Q2 </td> <td align="right"> 4 </td> <td align="right"> 6 </td> <td align="right"> <b>3</b> </td> <td align="right"> <b>3</b> </td> </tr> <tr> <td> Q3 </td> <td align="right"> <b>13</b> </td> <td align="right"> 51 </td> <td align="right"> 254 </td> <td align="right"> 64 </td> </tr> <tr> <td> Q4 </td> <td align="right"> <b>4</b> </td> <td align="right"> 16 </td> <td align="right"> 24 </td> <td align="right"> 60 </td> </tr> <tr> <td> Q5 </td> <td align="right"> <b>15</b> </td> <td align="right"> 22 </td> <td align="right"> 64 </td> <td align="right"> 68 </td> </tr> <tr> <td> Q6 </td> <td align="right"> <b>9</b> </td> <td align="right"> 70 </td> <td align="right"> 189 </td> <td align="right"> 65 </td> </tr> <tr> <td> Q7 </td> <td align="right"> <b>52</b> </td> <td align="right"> 143 </td> <td align="right"> 211 </td> <td align="right"> 84 </td> </tr> <tr> <td> Q8 </td> <td align="right"> 29 </td> <td align="right"> 31 </td> <td align="right"> 13 </td> <td align="right"> <b>11</b> </td> </tr> <tr> <td> Q9 </td> <td align="right"> <b>36</b> </td> <td align="right"> 114 </td> <td align="right"> 97 </td> <td align="right"> 61 </td> </tr> <tr> <td> Q10 </td> <td align="right"> <b>32</b> </td> <td align="right"> 51 </td> <td align="right"> 117 </td> <td align="right"> 57 </td> </tr> <tr> <td> Q11 </td> <td align="right"> 16 </td> <td align="right"> <b>9</b> </td> <td align="right"> 12 </td> <td align="right"> 10 </td> </tr> <tr> <td> Q12 </td> <td align="right"> <b>8</b> </td> <td align="right"> 21 </td> <td align="right"> 18 </td> <td align="right"> 130 </td> </tr> <tr> <td> Q13 </td> <td align="right"> <b>18</b> </td> <td align="right"> 74 </td> <td align="right"> - </td> <td align="right"> - </td> </tr> <tr> <td> Q14 </td> <td align="right"> <b>7</b> </td> <td align="right"> 21 </td> <td align="right"> 418 </td> <td align="right"> 1425 </td> </tr> <tr> <td> Q15 </td> <td align="right"> <b>14</b> </td> <td align="right"> 43 </td> <td align="right"> 389 </td> <td align="right"> 122 </td> </tr> <tr> <td> Q16 </td> <td align="right"> <b>16</b> </td> <td align="right"> 22 </td> <td align="right"> 18 </td> <td align="right"> 25 </td> </tr> <tr> <td> Q17 </td> <td align="right"> <b>1</b> </td> <td align="right"> 54 </td> <td align="right"> 26 </td> <td align="right"> 10 </td> </tr> <tr> <td> Q18 </td> <td align="right"> <b>82</b> </td> <td align="right"> 120 </td> <td align="right"> - </td> <td align="right"> - </td> </tr> <tr> <td> Q19 </td> <td align="right"> 19 </td> <td align="right"> 8 </td> <td align="right"> <b>2</b> </td> <td align="right"> 17 </td> </tr> <tr> <td> Q20 </td> <td align="right"> <b>7<b> </b></b></td> <td align="right"> 15 </td> <td align="right"> 66 </td> <td align="right"> 52 </td> </tr> <tr> <td> Q21 </td> <td align="right"> <b>34</b> </td> <td align="right"> 86 </td> <td align="right"> 524 </td> <td align="right"> 278 </td> </tr> <tr> <td> Q22 </td> <td align="right"> <b>4</b> </td> <td align="right"> 323 </td> <td align="right"> 3311</td> <td align="right"> 805 </td> </tr> <tr> <td>Total (msec)</td> <td align="right"><b>626</b></td> <td align="right">2063</td> <td align="right">6068</td> <td align="right">3545</td> </tr> </tbody> </table> <p>We lead by a fair margin but MySQL is hampered by obviously getting some execution plans wrong and not doing Q13 and Q18 at all, at least not under several tens of seconds; so we left these out of the table in the interest of having comparable totals.</p> <p>As usual, we also ran the workload on <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0xe957b80">Oracle</a> 10g R2. Since Oracle does not like their numbers being published without explicit approval, we will just say that we are even with them within the parameters described above. Oracle has a more efficient decimal type so it wins where that is central, as on Q1. Also it seems to notice that the <code>GROUP BY</code>s of Q18 are produced in order of grouping columns, so it needs no intermediate table for storing the aggregates. If we addressed these matters, we'd lead by some 15% whereas now we are even. A faster decimal arithmetic implementation may be in the release after next.</p> <p>In the next posts, we will look at IO and disk allocation, and also return to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xe074e40">RDF</a> and LUBM.</p>
Virtuoso 5.0 Preview
http://www.openlinksw.com/weblog/oerling/?date=2007-01-10#1116
2007-01-10T15:08:43Z
2008-04-17T21:04:24-04:00
<p>As <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1108" id="link-id10c66e68">previously said</a>, we have a <a href="http://virtuoso.openlinksw.com" id="link-id0x1a5caeb8">Virtuoso</a> with brand new engine multithreading. It is now complete and passes its regular test suite. This is the basis for Virtuoso 5.0, to be available as the open source and commercial cuts as before.</p> <p>As one benchmark, we used the <a href="http://dbpedia.org/resource/TPC-C" id="link-id0x15f8cbd8">TPC-C</a> test driver that has always been bundled with Virtuoso. We ran 100000 new orders worth of the TPC-C transaction mix first with one client and then with 4 clients, each client going to its own warehouse, so there was not much lock contention. We did this on a 4 core Intel, the working set in RAM. With the old one, 1 client took 1m43 and 4 clients took 3m47. With the new one, one client took 1m30 and 4 clients took 2m37. So, 400000 new orders in 2m37, for 152820 new orders per minute as opposed to 105720 per minute previously. Do not confuse with the official tpmC metric, that one involves a whole bunch of further rules.</p> <p>TPC-C has activity spread over a few different tables. With tests dealing with fewer tables, improvements in parallelism are far greater.</p> <p>Aside from better parallelism, we have other features. One of them is a change in the read committed isolation, so that we now return the previous committed state for uncommitted changed rows instead of waiting for the updating transaction to terminate. This is similar to what <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x18184c08">Oracle</a> does for read committed. Also we now do log checkpoints without having to abort pending write transactions.</p> <p>When we have faster inserts, we actually see the <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xde6fca0">RDF</a> bulk loader run slower. This is really backwards. The reason is that while one thread parses, other threads insert and if the inserting threads are done they go to wait on a semaphore and this whole business of context switching absolutely kills performance. With slower inserts, the parser keeps ahead so there is less context switching, hence better overall throughput. I still do not get it how the OS can spend between 1.5 and 6 microseconds, several thousand instructions, deciding what to do next when there are only 3-4 eligible threads and all the rest is background which goes with a few dozen slices per second. Solaris is a little better than Linux at this but not dramatically so. Mac OS X is way worse.</p> <p>As said, we use Oracle 10G2 on the same platform (Linux FC5 64 bit) for sparring. It is really a very good piece of software. We have written the TPC C transactions in <a href="http://dbpedia.org/resource/SQL" id="link-id0x15b33600">SQL</a>/PL. What is surprising is that these procedures run amazingly slowly, even with a single client. Otherwise the Oracle engine is very fast. Well, as I recall, the official TPC C runs with Oracle use an OCI client and no stored procedures. Strange. While Virtuoso for example fills the initial TPC C state a little faster than Oracle, the procedures run 5-10 times slower with Oracle than with Virtuoso, all <a href="http://dbpedia.org/resource/Data" id="link-id0xd9d1150">data</a> in warm cache and a single client. While some parts of Oracle are really well optimized, all basic joins and aggregates etc, we are surprised at how they could have neglected such a central piece as the PL.</p> <p>Also, we have looked at transaction semantics. Serializable is mostly serializable with Oracle but does not always keep a steady count. Also it does not prevent inserts into a space that has been found empty by a serializable transaction. True, it will not show these inserts to the serializable transaction, so in this it follows the rules. Also, to make a read really repeatable, it seems that the read has to be FOR UPDATE. Otherwise one can not implement a reliable resource transaction, like changing the balance of an account.</p> <p>Anyway, the Virtuoso engine overhaul is now mostly complete. This is of course an open ended topic but the present batch is nearing completion. We have gone through as many as 3 implementations of hash joins, some things have yet to be finished there. Oracle has very good hash joins. The only way we could match that was to do it all in memory, dropping any persistent storage of the hash. This is of course OK if the hash is not very large and anyway hash joins go sour if the hash does not fit in working set.</p> <p>As next topics, we have more RDF and the LUBM benchmark to finish. Also we should revisit TPC-D.</p> <p>Databases are really quite complicated and extensive pieces of software. Much more so than the casual observer might think.</p>
Ideas on RDF Store Benchmarking
http://www.openlinksw.com/weblog/oerling/?date=2006-11-21#1084
2006-11-21T14:09:21Z
2008-04-16T16:53:22-04:00
<p>This post presents some ideas and use cases for <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x18a21d88">RDF</a> store benchmarking.</p> <h4>Use Cases</h4> <ul> <li>Basic triple storage and retrieval. The LUBM benchmark captures many aspects of this.</li> <li>Recursive rule application. The simpler cases of this are things like transitive closure.</li> <li>Mapping of relational <a href="http://dbpedia.org/resource/Data" id="link-id0xdd0f4b0">data</a> to RDF. Since relational benchmarks are well established, as in the TPC benchmarks, the schemas and test data generation can come from there. The problem is that the D/H/R benchmarks consist of aggregates and grouping exclusively but <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xdd8b060">SPARQL</a> does not have these. </li> </ul> <h4>Benchmarking Triple Stores</h4> <p>An RDF benchmark suite should meet the following criteria:</p> <ul> <li>Have a single scale factor.</li> <li>Produce a single metric, queries per unit of time, for example. The metric should be concisely expressible, for example 10 qpsR at 100M, options 1, 2, 3. Due to the heterogeneous nature of the systems under test, the result's short form likely needs to specify the metric, scale and options included in the test.</li> <li>Have optional parts, such as different degrees of inferencing and maybe language extensions such as full text, as this is a likely component of any social software.</li> <li>Have a specification for a full disclosure report, TPC style, even though we can skip the auditing part in the interest of making it easy for vendors to publish results and be listed.</li> <li>Have a subject domain where real data are readily available and which is broadly understood by the community. For example, SIOC data about on-line communities seems appropriate. Typical degree of connectedness, number of triples per person etc can be measured from real files .</li> <li>Have a diverse enough workload. This should include initial bulk load of data, some adding of triples during the run and continuous query load.</li> </ul> <p>The query load should illustrate the following types of operations:</p> <ul> <li>Basic lookups, such as would be made for filling in a person's home page in a social networks app. List data of user plus names and emails of friends. Relatively short joins, unions, and optionals.</li> <li>Graph operations like shortest path from individual to individual in a social network.</li> <li>Selecting data with drill down, as in faceted browsing. For example, start with articles having <a href="http://dbpedia.org/resource/Tag" id="link-id0x1a11f478">tag</a> t, see distinct tags of articles with tag t, select another tag t2 to see the distinct tags of articles with both t and t2 and so forth.</li> <li>Retrieving all closely related nodes, as in composing a SIOC snapshot over a person's post in different communities, the recent activity report for a forum etc. These will be construct or describe queries. The coverage of describe is unclear, hence construct may be better.</li> </ul> <p>If we take an application like LinkedIn as a model, we can get a reasonable estimate of the relative frequency of different queries. For the queries per second metric, we can define the mix similarly to <a href="http://dbpedia.org/resource/TPC-C" id="link-id0x19ba4830">TPC C</a>. We count executions of the main query and divide by running time. Within this time, for every 10 executions of the main query there are varying numbers of executions of secondary queries, typically more complex ones.</p> <h4>Full Disclosure Report</h4> <p>The report contains basic TPC-like items such as:</p> <ul> <li>Metric qps/scale/options</li> <li>Software used, DBMS, RDF toolkit if separate</li> <li>Hardware. Number, clock and type of CPUs per machine, number of machines in cluster, RAM per machine, disks per machine, manufacturer, price of hardware/software</li> </ul> <p>These can go into a summary spreadsheet that is just like the TPC ones.</p> <p>Additionally, the full report should include:</p> <ul> <li>Configuration files for DBMS, web server, other components.</li> <li>Parameters for test driver, i.e., number of clicks, how many concurrent clicks. The tester determines the degree of parallelism that gets the best throughput and should indicate this in the report. Making a graph of throughput as function of concurrent clients is a lot of work and maybe not necessary here.</li> <li>Duration in real time. Since for any large database with a few G of working set the warm up time is easily 30 minutes, the warm up time should be mentioned but not included in the metric. The measured interval should not be less than 1h in duration and should reflect a "steady state," as defined in the TPC rules.</li> <li>Source code of server side application logic. This can be inference rules, stored procedures, dynamic web pages or any other server side software-like thing that exists or is modified for the purpose of the test.</li> <li>Specification of test driver. If there is a commonly used test driver, its type, parameters and version. If the test driver is custom, reference to its source code.</li> <li>Database sizes. For a preallocated database of n G, how much was free after the initial load, how much after the test run? How many bytes per triple.</li> <li>CPU/IO. This may not always be readily measurable but is interesting still. Maybe a realistic spec is listing the sum of CPU minutes across all server machines and server processes. For IO, maybe the system totals from iostat before and after the full run, including load and warm-up. If the DBMS and RDF toolkits are separate, it is interesting to know the division of CPU time between them. </li> </ul> <h4>Test Drivers</h4> <p>OpenLink has a multithreaded C program that simulates n web users multiplexed over m threads. For example, 10000 users with 100 threads, each user with its own state, so that they carry out their respective usage patterns independently, getting served as soon as the server is available, still having no more than m requests going at any time. The usage pattern is something like go check the mail, browse the catalogue, add to shopping cart etc. This can be modified to browse a social network database and produce the desired query mix. This generates <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x19f1f538">HTTP</a> requests, hence would work against a SPARQL end point or any set of dynamic web pages.</p> <p>The program produces a running report of the clicks per second rate and statistics at the end, listing the min/avg/max times per operation.</p> <p>This can be packaged as a separate open source download once the test spec is agreed upon.</p> <p>For generating test data, a modification of the LUBM generator is probably the most convenient choice.</p> <h4>Benchmarking Relational to RDF Mapping</h4> <p>This area is somewhat more complex than triple storage.</p> <p>At least the following factors enter into the evaluation: </p> <ul> <li>Degree of SPARQL compliance. For example, can one have a variable as predicate? Are there limits on optionals and unions?</li> <li>Are the data being queried split over multiple <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1aea8f70">RDBMS</a> and joined between them?</li> <li>Type of use case. Is this about navigational lookups or about statistics? OLTP or OLAP? It would be the former, as SPARQL does not really have aggregation. Still, many of the interesting queries are about comparing large data sets.</li> </ul> <p>The rationale for mapping relational data to RDF is often data integration. Even in simple cases like the <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id0xd69ed58">OpenLink Data Spaces</a> applications, a single SPARQL query will often result in a union of queries over distinct relational schemas, each somewhat similar but different in its details.</p> <p>A test for mapping should represent this aspect. Of course, translating a column into a predicate is easy and useful, specially when copying data. Still, the full power of mapping seems to involve a single query over disparate sources with disparate schemas.</p> <p>A real world case is OpenLink's ongoing work for mapping <a href="http://dbpedia.org/resource/WordPress" id="link-id0x19c5c5c0">WordPress</a>, <a href="http://dbpedia.org/resource/MediaWiki" id="link-id0x19a0c5d0">Mediawiki</a>, phpBB, <a href="http://dbpedia.org/resource/Drupal" id="link-id0xd106ef8">Drupal</a>, and possibly other popular web applications into SIOC.</p> <p>Using this as a benchmark might make sense because the source schemas are widely known, there is a lot of real world data in these systems, and the test driver might even be the same as with the above proposed triple store benchmark. The query mix might have to be somewhat tailored.</p> <p>Another "enterprise style" scenario might be to take the TPC C and TPC D databases â after all both have products, customers and orders â and map them into a common ontology. Then there could be queries sometimes running on only one, sometimes joining both.</p> <p>Considering the times and the audience, the WordPress/Mediawiki scenario might be culturally more interesting and more fun to demo.</p> <p>The test has two aspects: Throughput and coverage. I think these should be measured separately.</p> <p>The throughput can be measured with queries that are generally sensible, such as "get articles by an author that I know with tags t1 and t2."</p> <p>Then there are various pathological queries that work specially poorly with mapping. For example, if the types of subjects are not given, if the predicate is known at run time only, if the graph is not given, we get a union of everything joined with another union of everything and many of the joins between the terms of the different unions are identically empty but the software may not know this.</p> <p>In a real world case, I would simply forbid such queries. In the benchmarking case, these may be of some interest. If the mapping is clever enough, it may survive cases like "list all predicates and objects of everything called gizmo where the predicate is in the product ontology".</p> <p>It may be good to divide the test into a set of straightforward mappings and special cases and measure them separately. The former will be queries that a reasonably written application would do for producing user reports.</p>
More RDF scalability tests
http://www.openlinksw.com/weblog/oerling/?date=2006-11-01#1074
2006-11-01T19:26:40Z
2008-04-16T16:53:18-04:00
<p>We have lately been busy with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x17524ab8">RDF</a> scalability. We work with the 8000 university LUBM <a href="http://dbpedia.org/resource/Data" id="link-id0xd4ba910">data</a> set, a little over a billion triples. We can load it in 23h 46m on a box with 8G RAM. With 16G we probably could get it in 16h.</p> <p>The resulting database is 75G, 74 bytes per triple which is not bad. It will shrink a little more if explicitly compacted by merging adjacent partly filled pages. See <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSBitmapIndexing" id="link-id105e5cf8">Advances in Virtuoso RDF Triple Storage</a> for an in-depth treatment of the subject.</p> <p>The real question of RDF scalability is finding a way of having more than one CPU on the same index tree without them hitting the prohibitive penalty of waiting for a mutex. The sure solution is partitioning, would probably have to be by range of the whole key. but before we go to so much trouble, we'll look at dropping a couple of critical sections from index random access. Also some kernel parameters may be adjustable, like a spin count before calling the scheduler when trying to get an occupied mutex. Still we should not waste too much time on platform specifics. We'll see.</p> <p>We just updated the <a href="http://virtuoso.openlinksw.com" id="link-id0x189d64b8">Virtuoso</a> Open Source cut. The latest RDF refinements are not in, so maybe the cut will have to be refreshed shortly.</p> <p>We are also now applying the relational to RDF mapping discussed in <a href="http://virtuoso.openlinksw.com/wiki/main/Main/VOSSQLRDF" id="link-id10677bb8">Declarative SQL Schema to RDF Ontology Mapping</a> to the <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id0xa0f5fde0">ODS</a> applications.</p> <p>There is a form of the mapping in the VOS cut on the net but it is not quite ready yet. We must first finish testing it through mapping all the relational schemas of the ODS apps before we can really recommend it. This is another reason for a VOS update in the near future.</p> <p>We will be looking at the query side of LUBM after the ISWC 2006 conference. So far, we find queries compile OK for many SIOC use cases with the cost model that there is now. A more systematic review of the cost model for <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x19b96630">SPARQL</a> will come when we get to the queries.</p> <p>We put some ideas about inferencing in the Advances in Triple Storage paper. The question is whether we should forward chain such things as class subsumption and subproperties. If we build these into the <a href="http://dbpedia.org/resource/SQL" id="link-id0x19bbd098">SQL</a> engine used for running SPARQL, we probably can do these as unions at run time with good performance and better working set due to not storing trivial entailed triples. Some more thought and experimentation needs to go into this.</p>
RDF Bulk Loading Revisited
http://www.openlinksw.com/weblog/oerling/?date=2006-09-28#1058
2006-09-28T11:39:52Z
2008-04-16T16:53:17.000007-04:00
<p>We have made new benchmarks with loading the 47 million triples of the Wikipedia links <a href="http://dbpedia.org/resource/Data" id="link-id0x1713ab70">data</a> set. So far, our best result is 40 minutes with a dual core Xeon with 8G memory. This comes to about 18000 triples per second with between 1.2 and 2 CPU cores busy, slightly depending on configuration parameters. Our previous best result was with a dual 1.6GHz SPARC with 7700 triples per second on loading the 2M triple Wordnet data set.</p> <p>These are memory based speeds. We have implemented an automatic background compaction for database tables and have tried the Wikipedia load with and without. The CPU cost of the compaction was about 10% with a slight gain in real time due to less IO.</p> <p>But the real deal remains IO. With the compaction on, we got 91 bytes per triple, all included, i.e., two indices on the triples table, dictionaries from IRI IDs to URIs, etc. The compaction is rather simple — it just detects adjacent dirty pages about to be written to disk and sees if the set of contiguous dirty pages would fit on fewer pages than they now take. If so, it rewrites the pages and frees the ones left over. It does not touch clean pages. With some more logic it could also compact clean pages, provided the result did not have more dirty pages than the initial situation. With more aggressive compaction we will get about 75 bytes per triple. We will try this.</p> <p>But the real gains will come from index compression with bitmaps. For the Wikipedia data set, this will cut one of the indices to about a third of its current size. This is also the index with the more random access, so the benefit is compounded in terms of working set. At that point we will be looking at about 50 bytes per triple. We will see next week how this works with the LUBM <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x19af50b0">RDF</a> benchmark.</p>