OpenLink Community Blog

New Semantic Publishing Benchmark Record

Mon, 11 Jan 2016 15:22:22 GMT

There is a new SPB (Semantic Publishing Benchmark) 256 Mtriple record with Virtuoso.

As before, the result has been measured with the feature/analytics branch of the v7fasttrack open source distribution, and it will soon be available as a preconfigured Amazon EC2 image. The updated benchmarks AMI with this version of the software will be out there within the next week, to be announced on this blog.

On the Cost of RDF Query Optimization

RDF query optimization is harder than the relational equivalent; first, because there are more joins, hence an NP complete explosion of plan search space, and second, because cardinality estimation is harder and usually less reliable. The work on characteristic sets, pioneered by Thomas Neumann in RDF3X, uses regularities in structure for treating properties usually occurring in the same subject as columns of a table. The same idea is applied for tuning physical representation in the joint Virtuoso / MonetDB work published at WWW 2015.

The Virtuoso results discussed here, however, are all based on a single RDF quad table with Virtuoso's default index configuration.

Introducing query plan caching raises the Virtuoso score from 80 qps to 144 qps at the 256 Mtriple scale. The SPB queries are not extremely complex; lookups with many more triple patterns exist in actual workloads, e.g., Open PHACTS. In such applications, query optimization indeed dominates execution times. In SPB, data volumes touched by queries grow near linearly with data scale. At the 256 Mtriple scale, nearly half of CPU cycles are spent deciding a query plan. Below are the CPU cycles for execution and compilation per query type, sorted by descending sum of the times, scaled to milliseconds per execution. These are taken from a one minute sample of running at full throughput.

Test system is the same used before in the TPC-H series: dual Xeon E5-2630 Sandy Bridge, 2 x 6 cores x 2 threads, 2.3GHz, 192 GB RAM.

We measure the compile and execute times, with and without using hash join. When considering hash join, the throughput is 80 qps. When not considering hash join, the throughput is 110 qps. With query plan caching, the throughput is 145 qps whether or not hash join is considered. Using hash join is not significant for the workload but considering its use in query optimization leads to significant extra work.

With hash join

Compile	Execute	Total	Query
`3156Â ms`	`1181Â ms`	`4337Â ms`	`Total`
`1327Â ms`	`28Â ms`	`1355Â ms`	`query 01`
`444Â ms`	`460Â ms`	`904Â ms`	`query 08`
`466Â ms`	`54Â ms`	`520Â ms`	`query 06`
`123Â ms`	`268Â ms`	`391Â ms`	`query 05`
`257Â ms`	`5Â ms`	`262Â ms`	`query 11`
`191Â ms`	`59Â ms`	`250Â ms`	`query 10`
`9Â ms`	`179Â ms`	`188Â ms`	`query 04`
`114Â ms`	`26Â ms`	`140Â ms`	`query 07`
`46Â ms`	`62Â ms`	`108Â ms`	`query 09`
`71Â ms`	`25Â ms`	`96Â ms`	`query 12`
`61Â ms`	`13Â ms`	`74Â ms`	`query 03`
`47Â ms`	`2Â ms`	`49Â ms`	`query 02`

Â Â Â Â

Without hash join

Compile	Execute	Total	Query
`1816Â ms`	`1019Â ms`	`2835Â ms`	`Total`
`197Â ms`	`466Â ms`	`663Â ms`	`query 08`
`609Â ms`	`32Â ms`	`641Â ms`	`query 01`
`188Â ms`	`293Â ms`	`481Â ms`	`query 05`
`275Â ms`	`61Â ms`	`336Â ms`	`query 09`
`163Â ms`	`10Â ms`	`173Â ms`	`query 03`
`128Â ms`	`38Â ms`	`166Â ms`	`query 10`
`102Â ms`	`5Â ms`	`107Â ms`	`query 11`
`63Â ms`	`27Â ms`	`90Â ms`	`query 12`
`24Â ms`	`57Â ms`	`81Â ms`	`query 06`
`47Â ms`	`1Â ms`	`48Â ms`	`query 02`
`15Â ms`	`24Â ms`	`39Â ms`	`query 07`
`5Â ms`	`5Â ms`	`10Â ms`	`query 04`

Considering hash join always slows down compilation, and sometimes improves and sometimes worsens execution. Some improvement in cost-model and plan-space traversal-order is possible, but altogether removing compilation via caching is better still. The results are as expected, since a lookup workload such as SPB has little use for hash join by nature.

The rationale for considering hash join in the first place is that analytical workloads rely heavily on this. A good TPC-H score is simply unfeasible without this as previously discussed on this blog. If RDF is to be a serious contender beyond serving lookups, then hash join is indispensable. The decision for using this however depends on accurate cardinality estimates on either side of the join.

Previous work (e.g., papers from FORTH around MonetDB) advocates doing away with a cost model altogether, since one is hard and unreliable with RDF anyway. The idea is not without its attraction but will lead to missing out of analytics or to relying on query hints for hash join.

The present Virtuoso thinking is that going to rule based optimization is not the preferred solution, but rather using characteristic sets for reducing triples into wider tables, which also cuts down on plan search space and increases reliability of cost estimation.

When looking at execution alone, we see that actual database operations are low in the profile, with memory management taking the top 19%. This is due to CONSTRUCT queries allocating small blocks for returning graphs, which is entirely avoidable.

New Semantic Publishing Benchmark Record

Mon, 11 Jan 2016 15:22:09 GMT

There is a new SPB (Semantic Publishing Benchmark) 256 Mtriple record with Virtuoso.

On the Cost of RDF Query Optimization

The Virtuoso results discussed here, however, are all based on a single RDF quad table with Virtuoso's default index configuration.

Test system is the same used before in the TPC-H series: dual Xeon E5-2630 Sandy Bridge, 2 x 6 cores x 2 threads, 2.3GHz, 192 GB RAM.

With hash join

Compile	Execute	Total	Query
`3156Â ms`	`1181Â ms`	`4337Â ms`	`Total`
`1327Â ms`	`28Â ms`	`1355Â ms`	`query 01`
`444Â ms`	`460Â ms`	`904Â ms`	`query 08`
`466Â ms`	`54Â ms`	`520Â ms`	`query 06`
`123Â ms`	`268Â ms`	`391Â ms`	`query 05`
`257Â ms`	`5Â ms`	`262Â ms`	`query 11`
`191Â ms`	`59Â ms`	`250Â ms`	`query 10`
`9Â ms`	`179Â ms`	`188Â ms`	`query 04`
`114Â ms`	`26Â ms`	`140Â ms`	`query 07`
`46Â ms`	`62Â ms`	`108Â ms`	`query 09`
`71Â ms`	`25Â ms`	`96Â ms`	`query 12`
`61Â ms`	`13Â ms`	`74Â ms`	`query 03`
`47Â ms`	`2Â ms`	`49Â ms`	`query 02`

Â Â Â Â

Without hash join

Compile	Execute	Total	Query
`1816Â ms`	`1019Â ms`	`2835Â ms`	`Total`
`197Â ms`	`466Â ms`	`663Â ms`	`query 08`
`609Â ms`	`32Â ms`	`641Â ms`	`query 01`
`188Â ms`	`293Â ms`	`481Â ms`	`query 05`
`275Â ms`	`61Â ms`	`336Â ms`	`query 09`
`163Â ms`	`10Â ms`	`173Â ms`	`query 03`
`128Â ms`	`38Â ms`	`166Â ms`	`query 10`
`102Â ms`	`5Â ms`	`107Â ms`	`query 11`
`63Â ms`	`27Â ms`	`90Â ms`	`query 12`
`24Â ms`	`57Â ms`	`81Â ms`	`query 06`
`47Â ms`	`1Â ms`	`48Â ms`	`query 02`
`15Â ms`	`24Â ms`	`39Â ms`	`query 07`
`5Â ms`	`5Â ms`	`10Â ms`	`query 04`

Life after sun.jdbc.odbc.JdbcOdbcDriver

Thu, 04 Jun 2015 21:12:19 GMT

In the beginning...

When Sun originally released Java 1.0, there were no JDBC drivers -- there wasn't even a JDBC.

Data access came in Java 2.0, as JDBC 1.0, but there were very few JDBC drivers from any source, as would be expected with any new technology -- but the ODBC ecosystem (itself then at only v2.0) was going strong.

Sun recognized that Java wouldn't have as much uptake without a functional data access solution -- so they produced and bundled the original Type 1 JDBC-ODBC Bridge Driver, sun.jdbc.odbc.JdbcOdbcDriver, but from the very beginning, they warned that users "should use the JDBC-ODBC Bridge only for experimental prototyping or when you have no other driver available."

That bundled JDBC-ODBC Bridge was (and always remained) single-threaded, and though it received some other updates along the way, it only ever supported a subset of JDBC 2.0 and later. Sun (and later Oracle) recommended that users employ "a pure Java JDBC technology-enabled driver, type 3 or 4, in order to get all of the benefits of the Java programming language and the JDBC API."

Where does OpenLink Software come in?

Even in the early days of JDBC, we saw that there would not always be an available JDBC driver for a given target data source -- but the numbers of ODBC drivers were rapidly increasing, supporting every major and many minor DBMS and other data sources. We saw a need for an enterprise-grade, non-experimental Bridge solution, with full support for the JDBC API.

We delivered this first as our Type 3 Multi-Tier solution, bridging from JDBC in one environment (typically a UNIX-like OS) to ODBC in another (most often, Microsoft Windows).

Type 3 Enterprise Edition (Multi-Tier) Architecture Diagram
(click to enlarge)

Soon afterward, we released our first Type 1 Single-Tier solution, first for Windows, but quickly also supporting Linux, Mac OS X, Solaris, AIX, HP-UX, and other UNIX-like OS, through our own iODBC driver manager -- which was itself helping to expand the ODBC ecosystem far beyond its birthplace on Windows.

Type 1 Lite Edition (Single-Tier) Architecture Diagram
(click to enlarge)

Why is Java 8 so special?

Sun long warned that the JRE-bundled Bridge was transitional, and Oracle confirmed immediately upon acquisition that it would "be removed in JDK 8. In addition, Oracle does not support the JDBC-ODBC Bridge." Java 8 is now in full release, and indeed, the venerable sun.jdbc.odbc.JdbcOdbcDriver is no longer present, as evidenced by the scary looking error --

java.lang.ClassNotFoundException: sun.jdbc.odbc.JdbcOdbcDriver
    at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:30
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:259)

Any Java users or applications relying on ODBC connections and also needing the security and other improvements found in Java 8 are left high and dry... Or would be, but for OpenLink Software.

Our JDBC-to-ODBC Bridge, in both Type 1 and Type 3 forms, has been available and regularly updated since its original release for JDBC 1. Fully multi-threaded since Java Runtime Environments (JREs) could handle such, we have also kept pace with the JDBC API -- now at JDBC 4.2, in 2015's Java 8 a/k/a JDK/JVM/JRE 1.8 -- and maintained compatibility with the also-evolving ODBC API, now at 3.8.

Especially important for the modern world, our solutions support both 64-bit and 32-bit environments, including both 64-bit JVMs and ODBC drivers, and our Type 3 solutions can even bridge between these, whether you have a 64-bit JVM and need to connect to a 32-bit ODBC driver, or you have a 32-bit JVM and need to connect to a 64-bit ODBC driver.

As always, our solutions are available for immediate download, with a free two-week trial license provided alongside. We encourage pre-purchase installation, configuration, and testing, with support provided through our web-based Support Forums and even free up-and-running Support Cases. Once you've confirmed the driver works for you, entry level and special offer licenses may be purchased online or through our Sales Team; these as well as custom license configurations or partnership (IBP, ISV, VAR, OEM, etc.) arrangements are always available by direct contact.

ICDE 2012 (post 6 of 6) - Science Data Panel

Tue, 17 Apr 2012 19:38:20 GMT

Michael Stonebraker chaired a panel on the future of science data at ICDE 2012 last week. Other participants were Jeremy Kepner from MIT Lincoln Labs, Anastasia Ailamaki from EPFL, and Alex Szalay from Johns Hopkins University.

This is the thrust of what was said, noted from memory. My comments follow after the synopsis.

Jeremy Kepner: When Java was new we saw it as the coming thing and figured that in HPC we should find space for this. When MapReduce and Hadoop came along, we saw this as a sea change in parallel programming models. This was so simple literally anybody could make parallel algorithms whereas this was not so with MPI. Even parallel distributed arrays are harder. So MapReduce was a game changer, together with the cloud where anybody can get a cluster. Hardly a week passes without me having to explain to somebody in government what MapReduce and Hadoop are about.

We have a lot of arrays and a custom database for them. But the arrays are sparse so this is in fact a triple store. Our users like to work in MATLAB, and any data management must run together with that.

Of course, MapReduce is not a real scheduler, and Hadoop is not a real file system. For deployment, we must integrate real schedulers and make HDFS look like a file system to applications. The abstraction of a file system is something people like. Being able to skip a time-consuming data-ingestion process with a database is an advantage with file-based paradigms like Hadoop. If this is enhanced with the right scheduling features, this can be a good component in the HPC toolbox.

Michael Stonebraker: Users of the data use math packages like R, MATLAB, SAS, SPSS, or similar. If business intelligence is about AVG, MIN, MAX, COUNT, and GROUP BY, science applications are much more diverse in their analytics. All science algorithms have an inner loop that resembles linear algebra operations like matrix multiplication. Data is more often than not a large array. There are some graphs in biology and chemistry, but the world is primarily rectangular. Relational databases can emulate sparse arrays but are 20x slower than a custom-made array database for dense arrays. And I will not finish without picking on MapReduce: I know of 2000-node MapReduce clusters. The work they do is maybe that of a 100-node parallel database. So if 2000 nodes is what you want to operate, be my guest.

Science database is a zero billion dollar business. We do not expect to make money from the science market with SciDB, which by now works and has commercial services supplied by Paradigm 4, while the code itself is open source, which is a must for the science community. The real business opportunity is in the analytics needed by insurance and financial services in general, which are next to identical with the science use cases SciDB tackles. This makes the vendors pay attention.

Alex Szalay: The way astronomy is done today is through surveys: a telescope scans through the sky and produces data. We have now for 10 years operated the Sloane Sky Survey and kept the data online. We have all the data, and complete query logs, available for anyone interested. When we set out to do this with Jim Gray, everybody found this a crazy idea, but it has worked out.

Anastasia Ailamaki: We do not use SciDB. We find a lot of spatial use cases. Researchers need access to simulation results which are usually over a spatial model, like in earthquake simulations and the brain. Off-the-shelf techniques like R trees do not work -- the objects overlap too much -- so we have made our own spatial indexing. We make custom software when it is necessary, and are not tied to vendors. In geospatial applications, we can create meshes of different shapes -- like tetrahedral or cubes for earthquakes, and cylinders for the brain -- and index these in a geospatial index. But since an R tree is inefficient when objects overlap too much, as these do, we just find one; and then because there is reachability from an object to neighboring ones, we use this to get all the objects in the area of interest.

* * *

This is obviously a diverse field. Probably the message that we can synthesize out of this is that flexibility and parallel programming models are what we need to pay attention to. There is a need to go beyond what one can do in SQL while continuing to stay close to the data. Also, allowing for plug-in data types and index structures may be useful; we sometimes get requests for such anyway.

The continuing argument around MapReduce and Hadoop is a lasting feature of the landscape. A parallel DB will beat MapReduce any day at joining across partitions; the problem is to overcome the mindset that sees Hadoop as the always-first answer to anything parallel. People will likely have to fail with this before they do anything else. For us, the matter is about having database-resident logic for extract-transform-load (ETL) that can do data-integration type-transformations and maybe iterative graph algorithms that constantly join across partitions, better than a MapReduce job, while still allowing application logic to be written in Java. Teaching sem-web-heads to write SQL procedures and to know about join order, join type, and partition locality, has proven to be difficult. People do not understand latency, whether in client-server or cluster settings. This is why they do not see the point of stored procedures or of shipping functions to data. This sounds like a terrible indictment, like saying that people do not understand why rivers flow downhill. Yet, it is true. This is also why MapReduce is maybe the only parallel programming paradigm that can be successfully deployed in the absence of this understanding, since it is actually quite latency-tolerant, not having any synchronous cross-partition operations except for the succession of the map and reduce steps themselves.

Maybe it is so that the database guys see MapReduce as an insult to their intelligence and the rest of the world sees it as the only understandable way of running grep and sed (Unix commands for string search/replace) in parallel, with the super bonus of letting you reshuffle the outputs so that you can compare everything to everything else, which grep alone never let you do.

* * *

Making a database that does not need data loading seems a nice idea, and CWI has actually done something in this direction in "Here are my Data Files. Here are my Queries. Where are my Results?"] However, there is another product called Algebra Data that claims to take in data without loading and to optimize storage based on access. We do not have immediate plans in this direction. Bulk load is already quite fast (take 100G TPC-H in 70 minutes or so), but faster is always possible.

LOD2 Plenary and Review: Semanticist, Think Database!

Fri, 30 Sep 2011 21:02:01 GMT

Last week the LOD2 FP7 project had its first review, preceded by its third plenary meeting.

Before this, we did, as promised, get the column store and vectored execution capabilities of Virtuoso 7 Single-Server Edition extended to Virtuoso 7 Cluster Edition. More interesting still, we decoupled storage from the database server process, so now database files can migrate between server processes. This means that clusters are now elastic, i.e., new servers can be added to a cluster and the load can be redistributed without reloading the data.

These things were long planned, but now are done. Measurements will be published in some weeks, as part of CWI's continued running of RDF store benchmarks, per the LOD2 plan.

Doing the column store and elastic cluster is work enough, so I do not in general participate in support or consultancy or the like. This has some pros and cons. On the plus side, there is a relative lack of noise and a very clear idea of focus. Of course, this work is most highly applied, thus always informed by use cases, thus forgetting what ought to be done out there is not the problem. Rather, the problem is forgetting how things in fact are done as opposed to how they could or should be done.

To cut a long story short, it has become clear to me that the DBMS must tell the application developer what to do. Of course, the application developer could also look at performance metrics, but they do not, and explaining these metrics is too much work and yields no lasting benefit. Developers will produce all kinds of performance diagnostic traces if requested, but going through this song and dance can also be avoided by the right automation.

So, I will introduce two new product features called Wazzup? and Saywhat?

Wazzup? is answered by a mood line, like "Heavily disk bound: 100G more memory will give 10x speedup" or "Network bound: Processing in larger batches will give 5x more throughput" and Saywhat? is answered by some commentary on the user's last action, for example "there is no ?order with o_totalprice < 0" or "there is no property O_misspelledtotallprrice."

Wazzup? is about overall system state, and Saywhat? is about the user session, specifically query plans. But an explanation of a query plan is not understandable, so this will just point out some salient facts, like the reason why the answer comes out empty.

The other thing that came to my attention is the fact that a user has no instinctive feel for ETL. A database person takes it for a self-evident truth that data is loaded in bulk, but the application developer does not think of that. Likewise, the line between warehousing and federating is not instinctively felt; actually the question is not even posed in these terms. So one will find Web protocols and end-points and glue code on the app server when one ought to have ETL and adequate hardware for running the consolidated database.

Further, under-provisioning of equipment is endemic with semanticists. The Semantic Web gets a needlessly bad rap just because we find too much data on too little equipment. For example, I was surprised to learn that the Linked Geodata demo ran on only 16 GB RAM and 6 processor cores with 2 billion triples and 350 million points in a geo index. Now, even with our greatest space efficiency advances, there is no way this will run from memory.

It is not that the Web 2.0 stack is necessarily efficient (we hear the wildest stories of lack of database understanding from that side too), but at least there is a culture of running with enough equipment. Surely when the web-scale data gear (e.g. Google Bigtable, Yahoo PNUTS, Amazon Dynamo) was new, by the operators' own admission there was no way for this to be particularly efficient, database-wise. Not if your eventual consistency is a client application to a shared MySQL back-end. For a lookup or single-record-update workload, who cares when there is enough hardware? For analytics, there is the de facto impossibility of doing big joins, but map reduce is for that, all offline. The big web houses have always known how to deal with data; it is the smaller Web 2.0 guys who patch systems together with duct tape and memcache. Even so, the online experience gets created.

Semanticism has no part of this outlook, except maybe for Freebase, but then they are from California and now have been inside Google for a while.

We quite understand that when one needs to get big data online, one makes a key-value store as a point solution, because this way one owns what one operates, and the time to market is a lot shorter than if one tried building all this inside a general-purpose DBMS. Besides, the people who can in fact do this almost do not exist, and even if one had a whole army of this rare breed, development is not very scalable in a tightly-integrated system like a high-performance DBMS. Still further, to even start, one needs to own the DBMS, meaning that the initial platform must be known through and through. This is an issue even though open source platforms exist.

The graph data, semdata, schema-last, RDF, linked data enterprise -- whatever one calls it -- makes the bold proposition of bringing complex-query-at-scale to heterogeneous data. This is a database claim.

In the meantime, test deployments are made in defiance of database best practices. This is a bit like test driving a race car in reverse gear and steering by looking in the rear-view mirror.

There is also no short-term scalable way to educate people. At the LOD2 review, one comment was that an integrated project ought to clearly indicate how to set up the tool chain for good performance, specially as concerns interfaces between the tools. This is very true. Experience shows that developers of tools cannot accurately anticipate what usage patterns will emerge in the field. Therefore, we propose to do better than just documentation; we will make the server recognize the common sources of inefficiency and point the user to the right action.

Provisioning and usage patterns: The DBMS ought to know best.

Imagine the following conversation:

DBMS: Your application does single-triple INSERTs over client-server protocol all day, from a single client. 57% of real time goes in client server latency, 40% in cluster interconnect latency, 2% in compiling the statements, and 1% in doing the work. Use array parameters or bulk load from a file.

Operator: My developers use industry-standard Java class libraries with a service-oriented architecture and strictly enforced interfaces. This is called software engineering. Watch out ere you raise your voice against the canon.

[Some weeks later, after the load job has gone on for 10 days and gotten a third of the way, developers have discovered that JDBC has array parameters and are trying these.]

DBMS: 60% of real time goes into waiting for locks. 10% of transactions get aborted for deadlock. Transactions consist of an average of 10 client-server operations. Use stored procedures; acquire locks in predictable order; do SELECT FOR UPDATE. Throughput will be 4x higher if client-server operations are merged into a single operation. The transactions only INSERT; hence consider bulk load instead.

Operator: We are using an enterprise-class three-tier architecture. It has "enterprise" in the name and all the big guys are using it, so it must be scalable. Besides, it is distributed transactions, and distributed computing is the wave of the future. You are a cluster yourself, so the pot's got no business calling the kettle black.

[After a while, the data gets loaded with bulk load, but now on a single stream.]

DBMS: CPU is at 400% for an INSERT workload; adding more parallel threads will get 4.5x better throughput.

[Some time has elapsed and there are Ajax client apps out there trying to use the data.]

DBMS: Will you really not give me another 140 GB RAM and 16 more cores?

Operator: No, on general principles I will not, shut up.

DBMS: Do you know that your page impression takes 3 seconds and anything over 0.25 seconds is visibly slow? 300 GB worth of distinct pages have been accessed in the last 24 hours for 160 GB of RAM. Latency will drop 10x by using SSD; 50x by increasing RAM.

Operator: No dice, bucket. Shut up, besides, when I scroll through the data I always use for testing, I get it fast enough, you are just doing this out of greed and self-importance. You are a server among many, just like the mail server; you databases are just pretentious.

Currently addressing any of the above sorts of issues takes a long time and involves mostly-avoidable support communication. Questions of this sort do occur. We can probably produce commentary like the above based on logging some 50 numbers, and making some 15 regularly-run reports over these. The patterns to watch out for are well known. No, we will not make a Zippy the Pinhead office assistant; a computer should not try to be cute. This one will talk only in terms of gains from adjusting the deployment or usage patterns.

Now, suppose the operator said yes to the request for more cores and memory; then it would be up to the DBMS to deliver. This entails a capacity to redistribute itself automatically, and to give a quantitative report on the success of this measure. This means usage-based repartitioning of the data to equalize load over a cluster. The relevant metric in the above case is the drop in response time. On the other hand, the DBMS should also notice if there is clearly unused capacity.

This all will be presented as a line in the status report, so there is no extra wizard or workload analyzer that one must remember to run. For programmatic use there are SQL views for the relevant reports.

As for ETL, even if the DBMS can detect that it is not being done right, this does not mean that the user will know what to do. Therefore, for all the Web harvesting we support, as well as any import from local file system or Web services, with some RDF-ization, we will simply implement a proper ETL utility that will do things right. Wazzup? can just point the user to that if the workload looks like loading. This will have its own status report giving a load and transform rate and will point out what takes the longest, after everything is duly parallelized and made asynchronous.

Beyond these lessons, there is more to say about the review and plenary, we will get to that a bit later. We did promise a new edition of the LOD cache in a couple of months, now on the clustered column-store platform. Look for advances in data discoverability.

Virtuoso 6.2 brings New Features!

Wed, 22 Sep 2010 21:08:24 GMT

Virtuoso 6.2 introduces a major number of enhancements to areas including...

Linked Data Deployment
Linked Data Middleware
Data Virtualization
Dynamic Data Exchange & Data Replication
Security

Linked Data Deployment

Feature	Description	Benefit
Automatic Deployment	Linked Data Pages are now automatically published for every Virtuoso Data Object; users need only load their data into the RDF Quad Store.	Handcrafted URL-Rewrite Rules are no longer necessary.
HTTP Metadata Enhancements	HTTP `Link:` header is used to transfer vital metadata (e.g., relationships between a Descriptor Resource and its Subject) from HTTP Servers to User Agents.	Enables HTTP-oriented tools to work with such relationships and other metadata.
HTML Metadata Embedding	HTML resource `<head />` and `<link />` elements and their `@rel` attributes are used to transfer vital metadata (e.g., relationships between a Descriptor Resource and its Subject) from HTTP Servers to User Agents.	Enables HTML-oriented tools to work with such relationships and other metadata.
Hammer Stack Auto-Discovery Patterns	HTML resource `<head />` section and `<link />` elements, the HTTP `Link:` header, and XRD-based `"host-meta"` resources collectively provide structured metadata about Virtuoso hosts, associated Linked Data Spaces, and specific Data Items (Entities).	Enables humans and machines to easily distinguish between Descriptor Resources and their Subjects, irrespective of URI scheme.

Linked Data Middleware

Feature	Description	Benefit
New Sponger Cartridges	New cartridges (data access and transformation drivers) for Twitter, Facebook, Amazon, eBay, LinkedIn, and others.	Enable users and user agents to deal with the Sponged data spaces as though they were named graphs in a quad store, or tables in an RDBMS.
New Descriptor Pages	HTML-based descriptor pages are automatically generated.	Descriptor subjects, and the constellation of navigable attribute-and-value pairs that constitute their descriptive representation, are clearly identified.
Automatic Subject Identifier Generation	De-referenceable data object identifiers are automatically created.	Removes tedium and risk of error associated with nuance-laced manual construction of identifiers.
Support for OData, JSON, RDFa	Additional data representation and serialization formats associated with Linked Data.	Increases flexibility and interoperability.

Data Virtualization

Feature	Description	Benefit
Materialized RDF Views	RDF Views over ODBC/JDBC Data Sources can now (optionally) keep the Quad Store in sync with the RDBMS data source.	Enables high-performance Faceted Browsing while remaining sensitive to changes in the RDBMS data sources.
CSV-to-RDF Transformation	Wizard-based generation of RDF Linked Data from CSV files.	Speeds deployment of data which may only exist in CSV form as Linked Data.
Transparent Data Access Binding	SPASQL (SPARQL Query Language integrated into SQL) is usable over ODBC, JDBC, ADO.NET, OLEDB, or XMLA connections.	Enables Desktop Productivity Tools to transparently work with any blend of RDBMS and RDF data sources.

Dynamic Data Exchange & Data Replication

Feature	Description	Benefit
Quad Store to Quad Store Replication	High-fidelity graph-data replication between one or more database instances.	Enables a wide variety of deployment topologies.
Delta Engine	Automated generation of deltas at the named-graph-level, matches transactional replication offered by the Virtuoso SQL engine.	Brings RDF replication on par with SQL replication.
PubSubHubbub Support	Deep integration within Quad Store as an optional mechanism for shipping deltas.	Enables push-based data replication across a variety of topologies.

Security

Feature	Description	Benefit
WebID support at the DBMS core	Use WebID protocol for low-level ACL-based protection of database objects (RDF or Relational) and Web Services.	Enables application of sophisticated security and data access policies to Web Services (e.g., SPARQL endpoint) and actual DBMS objects.
Webfinger	Supports using `mailto:` and `acct:` URIs in the context of WebID and other mechanisms, when domain holders have published necessary XRDS resources.	Enables more intuitive identification of people and organizations.
Fingerpoint	Similar to Webfinger but does not require XRDS resources; instea,d it works directly with SPARQL endpoints exposed using auto-discovery patterns in the `<head />` section of HTML documents.	Enables more intuitive identification of people and organizations.

Linked Data and Information Architecture

Tue, 29 Apr 2008 14:37:22 GMT

We had a workshop on Linked Open Data (LOD) last week in Beijing. You can see the papers in the program. The event was a success with plenty of good talks and animated conversation. I will not go into every paper here but will comment a little on the conversation and draw some technology requirements going forward.

Tim Berners-Lee showed a read-write version of Tabulator. This raises the question of updating on the Data Web. The consensus was that one could assert what one wanted in one's own space but that others' spaces would be read-only. What spaces one considered relevant would be the user's or developer's business, as in the document web.

It seems to me that a significant use case of LOD is an open-web situation where the user picks a broad read-only "data wallpaper" or backdrop of assertions, and then uses this combined with a much smaller, local, writable data set. This is certainly the case when editing data for publishing, as in Tim's demo. This will also be the case when developing mesh-ups combining multiple distinct data sets bound together by sets of SameAs assertions, for example. Questions like, "What is the minimum subset of n data sets needed for deriving the result?" will be common. This will also be the case in applications using proprietary data combined with open data.

This means that databases will have to deal with queries that specify large lists of included graphs, all graphs in the store or all graphs with an exclusion list. All this is quite possible but again should be considered when architecting systems for an open linked data web.

"There is data but what can we really do with it? How far can we trust it, and what can we confidently decide based on it?"

As an answer to this question, Zitgist has compiled the UMBEL taxonomy using SKOS. This draws on Wikipedia, Open CYC, Wordnet, and YAGO, hence the acronym WOWY. UMBEL is both a taxononmy and a set of instance data, containing a large set of named entities, including persons, organizations, geopolitical entities, and so forth. By extracting references to this set of named entities from documents and correlating this to the taxonomy, one gets a good idea of what a document (or part thereof) is about.

Kingsley presented this in the Zitgist demo. This is our answer to the criticism about DBpedia having errors in classification. DBpedia, as a bootstrap stage, is about giving names to all things. Subsequent efforts like UMBEL are about refining the relationships.

"Should there be a global URI dictionary?"

There was a talk by Paolo Bouquet about Entity Name System, a a sort of data DNS, with the purpose of associating some description and rough classification to URIs. This would allow discovering URIs for reuse. I'd say that this is good if it can cut down on the SameAs proliferation and if this can be widely distributed and replicated for resilience, Ã la DNS. On the other hand, it was pointed out that this was not quite in the LOD spirit, where parties would mint their own dereferenceable URIs, in their own domains. We'll see.

"What to do when identity expires?"

Giovanni of Sindice said that a document should be removed from search if it was no longer available. Kingsley pointed out that resilience of reference requires some way to recover data. The data web cannot be less resilient than the document web, and there is a point to having access to history. He recommended hooking up with the Internet Archive, since they make long term persistence their business. In this way, if an application depends on data, and the URIs on which it depends are no longer dereferenceable or or provide content from a new owner of the domain, those who need the old version can still get it and host it themselves.

It is increasingly clear that OWL SameAs is both the blessing and bane of linked data. We can easily have tens of URIs for the same thing, especially with people. Still, these should be considered the same.

Returning every synonym in a query answer hardly makes sense but accepting them as input seems almost necessary. This is what we do with Virtuoso's SameAs support. Even so, this can easily double query times even when there are no synonyms.

Be that as it may, SameAs is here to stay; just consider the mapping of DBpedia to Geonames, for example.

Also, making aberrant SameAs statements can completely poison a data set and lead to absurd query results. Hence choosing which SameAs assertions from which source will be considered seems necessary. In an open web scenario, this leads inevitably to multi-graph queries that can be complex to write with regular SPARQL. By extension, it seems that a good query would also include the graphs actually used for deriving each result row. This is of course possible but has some implications on how databases should be organized.

Yves Raymond gave a talk about deriving identity between Musicbrainz and Jamendo. I see the issue as a core question of linked data in general. The algorithm Yves presented started with attribute value similarities and then followed related entities. Artists would be the same if they had similar names and similar names of albums with similar song titles, for example. We can find the same basic question in any analysis, for example, looking at how news reporting differs between media, supposing there is adequate entity extraction.

There is basic graph diffing in RDFSync, for example. But here we are expanding the context significantly. We will traverse references to some depth, allow similarity matches, SameAs, and so forth. Having presumed identity of two URIs, we can then look at the difference in their environment to produce a human readable summary. This could then be evaluated for purposes of analysis or of combining content.

At first sight, these algorithms seem well parallelizable, as long as all threads have access to all data. For scaling, this means a probably message-bound distributed algorithm. This is something to look into for the next stage of linked data.

Some inference is needed, but if everybody has their own choice of data sets to query, then everybody would also have their own entailed triples. This will make for an explosion of entailed graphs if forward chaining is used. Forward chaining is very nice because it keeps queries simple and easy to optimize. With Virtuoso, we still favor backward chaining since we expect a great diversity of graph combinations and near infinite volume in the open web scenario. With private repositories of slowly changing data put together for a special application, the situation is different.

In conclusion, we have a real LOD movement with actual momentum and a good idea of what to do next. The next step is promoting this to the broader community, starting with Linked Data Planet in New York in June.

On Sem Web Search

Tue, 29 Apr 2008 14:37:21 GMT

"I give the search keywords and you give me a SPARQL end-point and a query that will get the data."

Thus did one SPARQL user describe the task of a semantic/data web search engine.

In a previous post, I suggested that if the data web were the size of the document web, we'd be looking at two orders of magnitude more search complexity. It just might be so.

In the conversation, I pointed out that a search engine might have a copy of everything and even a capability to do SPARQL and full text on it all, yet still the users would be better off doing the queries against the SPARQL end-points of the data publishers. It is a bit like the fact that not all web browsing runs off Google's cache. With the data web, the point is even more pronounced, as serving a hit from Google's cache is a small operation but a complex query might be a very large one.

Yet, the data web is about ad-hoc joining between data sets of different origins. Thus a search engine of the data web ought to be capable of joining also, even if large queries ought to be run against individual publishers' end-points or the user's own data warehouse.

For ranking, the general consensus was that no single hit-ranking would be good for the data web. Thus word frequency-based hit-scores are OK for text hits but more is not obvious. I would think that some link analysis could apply but this will take some more experimentation.

For search summaries, if we have splitting of data sets into small fragments Ã la Sindice, search summaries are pretty much the same as with just text search. If we store triples, then we can give text style summaries of text hits in literals and Fresnel lens views of the structured data around the literal. For showing a page of hits, the lenses must abbreviate heavily but this is still feasible. The engine would know about the most common ontologies and summarize instance data accordingly.

Chris Bizer pointed out that trust and provenance are critical, especially if an answer is arrived at by joining multiple data sets. The trust of the conclusion is no greater than that of the weakest participating document. Different users will have different trusted sources.

A mature data web search engine would combine a provenance/trust specification, a search condition consisting of SPARQL or full text or both, and a specification for hit rank. Again, most searches would use defaults, but these three components should in principle be orthogonally specifiable.

Many places may host the same data set either for download or SPARQL access. The URI of the data set is not its URL. Different places may further host multiple data sets on one end-point. Thus the search engine ought to return all end-points where the set is to be found. The end-points themselves ought to be able to say what data sets they contain, under what graph IRIs. Since there is no consensus about end-point self description, this too would be left to the search engine. In practice, this could be accomplished by extending Sindice's semantic site map specification. A possible query would be to find an end-point containing a set of named data sets. If none were found, the search engine itself could run a query joining all the sets since it at least would hold them all.

Since many places will host sets like Wordnet or Uniprot, indexing these once for each copy hardly makes sense. Thus a site should identify its data by the data set's URI and not the copy's URL.

It came up in the discussion that search engines should share a ping format so that a single message format would be enough to notify any engine about data being updated. This is already partly the case with Sindice and PTSW (PingTheSemanticWeb) sharing a ping format.

Further, since it is no trouble to publish a copy of the 45G Uniprot file but a fair amount of work to index it, search engines should be smart about processing requests to index things, since these can amount to a denial of service attack.

Probably very large data sets should be indexed only in the form supplied by their publisher, and others hosting copies would just state that they hold a copy. If the claim to the copy proved false, users could complain and the search engine administrator would remove the listing. It seems that some manual curating cannot be avoided here.

On Data Web Search Business Model

It seems there can be an overlap between the data web search and the data web hosting businesses. For example, Talis rents space for hosting RDF data with SPARQL access. A search engine should offer basic indexing of everything for free, but could charge either data publishers or end users for running SPARQL queries across data sets. These do not have the nicely anticipatable and fairly uniform resource consumption of text lookups. In this manner, a search provider could cost-justify the capacity for allowing arbitrary queries.

The value of the data web consists of unexpected joining. Such joining takes place most efficiently if the sources are at least in some proximity, for example in the same data center. Thus the search provider could monetize functioning as the database provider for mesh-ups. In the document web, publishing pages is very simple and there is no great benefit from co-locating search and pages, rather the opposite. For the data web, the hosting with SPARQL and all is more complex and resembles providing search. Thus providing search can combine with providing SPARQL hosting, once we accept in principle that search should have arbitrary inter-document joining, even if it is at an extra premium.

The present search business model is advertising. If the data web is to be accessed by automated agents such as mesh-up code, display of ads is not self-evident. This is quite separate from the fact that semantics can lead to better ad targeting.

One model would be to do text lookups for free from a regular web page but show ads, just a la Google search ads. Using the service via web services for text or SPARQL would have a cost paid by the searching or publishing party and would not be financed by advertising.

In the case of data used in value-add data products (mesh-ups) that have financial value to their users, the original publisher of the data could even be paid for keeping the data up-to-date. This would hold for any time-sensitive feeds like news or financial feeds. Thus the hosting/search provider would be a broker of data-use fees and the data producer would be in the position of an AdSense inventory owner, i.e., a web site which shows AdSense ads. Organizing this under a hub providing back-office functions similar to an ad network could make sense even if the actual processing were divided among many sites.

Kingsley has repeatedly formulated the core value proposition of the semantic web in terms of dealing with information overload: There is the real-time enterprise and the real-time individual and both are beasts of perception. Their image is won and lost in the Internet online conversation space. We know that allegations, even if later proven false, will stick if left unchallenged. The function of semantics on the web is to allow one to track and manage where one stands. In fact, Garlik has made a business of just this, but now from a privacy and security angle. The Garlik DataPatrol harvests data from diverse sources and allows assessing vulnerability to identity theft, for example.

If one is in the business of collating all the structured data in the world, as a data web search engine is, then providing custom alerts for both security or public image management is quite natural. This can be a very valuable service if it works well.

At OpenLink, we will now experiment with the Sindice/Zitgist/PingTheSemanticWeb content. This is a regular part of the productization of Virtuoso's cluster edition. We expect to release some results in the next 4 weeks.

Virtuoso Cluster

Wed, 23 May 2007 14:09:37 GMT

We often get questions on clustering support, especially around RDF, where databases quickly get rather large. So we will answer them here.

But first on some support technology. We have an entire new disk allocation and IO system. It is basically operational but needs some further tuning. It offers much better locality and much better sequential access speeds.

Specially for dealing with large RDF databases, we will introduce data compression. We have over the years looked at different key compression possibilities but have never been very excited by them since thy complicate random access to index pages and make for longer execution paths, require scraping data for one logical thing from many places, and so on. Anyway, now we will compress pages before writing them to disk, so the cache is in machine byte order and alignment and disk is compressed. Since multiple processors are commonplace on servers, they can well be used for compression, that being such a nicely local operation, all in cache and requiring no serialization with other things.

Of course, what was fixed length now becomes variable length, but if the compression ratio is fairly constant, we reserve space for the expected compressed size, and deal with the rare overflows separately. So no complicated shifting data around when something grows.

Once we are done with this, this could well be a separate intermediate release.

Now about clusters. We have for a long time had various plans for clusters but have not seen the immediate need for execution. With the rapid growth in the Linking Open Data movement and questions on web scale knowledge systems, it is time to get going.

How will it work? Virtuoso remains a generic DBMS, thus the clustering support is an across the board feature, not something for RDF only. So we can join Oracle, IBM DB2, and others at the multi-terabyte TPC races.

We introduce hash partitioning at the index level and allow for redundancy, where multiple nodes can serve the same partition, allowing for load balancing read and replacement of failing nodes and growth of cluster without interruption of service.

The SQL compiler, SPARQL, and database engine all stay the same. There is a little change in the SQL run time, not so different from what we do with remote databases at present in the context of our virtual database federation. There is a little extra complexity for distributed deadlock detection and sometimes multiple threads per transaction. We remember that one RPC round trip Is 3-4 index lookups, so we pipeline things so as to move requests in batches, a few dozen at a time.

The cluster support will be in the same executable and will be enabled by configuration file settings. Administration is limited to one node, but Web and SQL clients can connect to any node and see the same data. There is no balancing between storage and control nodes because clients can simply be allocated round robin for statistically even usage. In relational applications, as exemplified by TPC-C, if one partitions by fields with an application meaning (such as warehouse ID), and if clients have an affinity to a particular chunk of data, they will of course preferentially connect to nodes hosting this data. With RDF, such affinity is unlikely, so nodes are basically interchangeable.

In practice, we develop in June and July. Then we can rent a supercomputer maybe from Amazon EC2 and experiment away.

We should just come up with a name for this. Maybe something astronomical, like star cluster. Big, bright but in this case not far away.

OAT and JS Hijacking

Wed, 04 Apr 2007 04:18:00 GMT

OAT and JS Hijacking

There has been some recent buzz regarding security issues of Web 2.0 sites. I decided to thoroughly study and analyze how this works, manifests and influences JS toolkits.

The best approach here is to study whitepaper located at http://comparitech.net/websecuritytools . One can find a typical attack scenario, some defense suggestions and security considerations. While the paper is (in my opinion) oversized, telling rather small amount of facts on large amount of pages, it contains some interesting information. Let me share how all this relates to OAT.

First of all, I must conclude that none of OAT-based apps is vulnerable to mentioned attacks, for two reasons:

attacks are only appliable in JSON documents, fetched via GET method. Our OAT apps use XML format, requested with POST (via XML/A - SOAP).
attacks are only appliable when security tokens (i.e. user name, password) are passed via browser cookie. This is not our case, since we pass these in POST request's body.

However, we might take into consideration the fact that OAT can be used for building different applications. So I added some low-level support for security countermeasures mentioned in the PDF document. Specifically (as a client-side only toolkit), the following two methods:

a) cookie-based secret, passed as URL parameter in GET requests. So every OAT.AJAX.GET invocation will

set a cookie called 'oatSecurityCookie' to some random value
add 'oatSecurityCookie=xxx' to URL string, where xxx equals to the random value mentioned above.

b) enhanced JSON parser with smarter text analysis, effectively filtering out

comments (at the beginning and end of JSONified text)
the 'while(1);' trap, situated at the beginning of received text

Both methods are now mentioned in the OAT documentation. Of course, applying OAT eqipped with these countermeasures doesn't make (yet) your app secure: one must explicitely use one (or both) methods on server side, to complement OAT's security features.

To sum this up - OAT is now providing a maximum support for securing GETting JSONified data. It is up to user whether he wants to take advantage of this. We are - as (nearly) always - ready :)