http://www.openlinksw.com/weblog/dav/dav-blog-1/
OpenLink Community Blog
A Collection of blogs by OpenLink Staff
OpenLink Software
kidehen@openlinksw.com
2008-05-17T11:36:24Z
Virtuoso Universal Server 05.00.3031
http://www.openlinksw.com/weblog/public/images/vbloglogo.gif
Commercializing the Semantic Web
http://www.openlinksw.com/blog/~kidehen/?id=1363
2008-05-16T22:04:01Z
<p>Unfortunately, I could only spend 4 days at the recent <a href="http://www2008.org/" id="link-id196acf60">WWW2008</a> event in <a href="http://dbpedia.org/resource/Beijing" id="link-id1974fe28">Beijing</a> (I departed the morning following the <a href="http://events.linkeddata.org/ldow2008/" id="link-id1863f858">Linked Data Workshop</a>), so I couldn't take my slot on the "Commercializing the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id18990f90">Semantic Web</a> panel" etc.. Anyway, thanks to the <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id0x18f29310">Web</a> I can still inject my points of view in the broad <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> based discourse. Well so I hoped, when I attempted to post a comment to Paul Miller's ZDNet domain hosted <a href="http://dbpedia.org/resource/Blog" id="link-id180d6750">blog</a> thread titled: <a href="http://blogs.zdnet.com/semantic-web/?p=132" id="link-id12d206c0">Commercialising the Semantic Web</a>.</p> <p>Unfortunately, the cost of completing ZDNet's unwieldy signup process simply exceeded the benefits of dropping my comments in their particular space :-( Thus, I'll settle for a trackback ping instead.</p> <p>What follows is the cut and paste of my intended comment contributions to Paul's post.</p> <p>Paul,</p> <p> As discussed earlier this week during <a href="http://blogs.talis.com/nodalities/2008/05/kingsley-idehen-talks-about-openlink-software-linked-data-and-the-semantic-web.php" id="link-id1332fb48">our podcast session</a>, commercialization of <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id17382338">Semantic Web</a> technology shouldn't be a mercurial matter at this stage in the game :-) It's all about looking at how it provides value :-)</p> <p>From the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10d4f4a8">Linked Data</a> angle, the ability to produce, dispatch, and exploit "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id13bed160">Context</a>" across an array of "Perspectives" from a plethora of disparate <a href="http://dbpedia.org/resource/Data" id="link-id1731e5f0">data</a> sources on the Web and/or behind corporate firewalls, offers immense commercial value.</p> <p> <a href="http://developer.yahoo.com/searchmonkey/" id="link-id1975d248">Yahoo's Searchmonkey</a> effort will certainly bring clarity to some of the points I made during the podcast re. the role of URIs as "value consumption tickets" (<a href="http://dbpedia.org/resource/Data" id="link-id173eb7b0">Data</a> Services are exposed via URIs). There has to be a trigger (in user space) that compels Web users to seek broader, or simply varied, perspectives as a response to <a href="http://dbpedia.org/resource/Data" id="link-id0x1c7e7f60">data</a> encountered on the Web. Yahoo! is about to put this light on in a big way (imho).</p> <p>The "self annotating" nature of the Web is what ultimately drives the manifestation of the long awaited <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0xa18a83e8">Semantic Web</a>. I believe I postulated about <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=self%20annotation&type=text&output=html" id="link-id173d7458">"Self Annotation & the Semantic Web" in a number of prior posts</a> which, by the way, should be <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&type=text&kwds=self%20annotation&OpenSearch" id="link-id10b12208">DataRSS compatible right now</a> due to Yahoo's support of OpenSearch <a href="http://dbpedia.org/resource/Data" id="link-id0x1b8412e8">Data</a> Providers (which this <a href="http://dbpedia.org/resource/Blog" id="link-id170b8df8">Blog</a> Space has been for eons).</p> <p>Today, have many communities adding strucuture to the Web (via their respective tools of preference) without explicitly realizing what they are contributing. Every RSS/Atom feed, <a href="http://dbpedia.org/resource/Tag" id="link-id183d5178">Tag</a>, Weblog, Shared Bookmark, <a href="http://dbpedia.org/resource/WikiWord" id="link-id10c5e758">Wikiword</a>, Microformat, Microformat++ (<a href="http://dbpedia.org/resource/Embedded_RDF" id="link-id16d8ee40">eRDF</a> or <a href="http://dbpedia.org/resource/RDFa" id="link-id1059a688">RDFa</a>), <a href="http://dbpedia.org/resource/GRDDL" id="link-id1090ae10">GRDDL</a> stylesheet, and RDFizer etc.. is a piece of structured <a href="http://dbpedia.org/resource/Data">data</a>.</p> <p>Finally, the different communities are all finding ways to work together (thank heavens!) and the results are going to be cataclysmic when it all plays out :-)</p> <p>Data, Structure, and Extraction are the keys to the Semantic Life! First you get the Data in a container (<a href="http://dbpedia.org/resource/Information" id="link-id180e5648">information</a> resource), and then you add Structure to the <a href="http://dbpedia.org/resource/Information" id="link-id103801e0">information</a> resource (RSS, Atom, <a href="http://dbpedia.org/resource/Microformats" id="link-id17825e40">microformats</a>, <a href="http://dbpedia.org/resource/RDFa" id="link-id189a8738">RDFa</a>, <a href="http://dbpedia.org/resource/Embedded_RDF" id="link-id1933d5c0">eRDF</a>, SIOC, FOAF, etc.), once you have Structure RDFization (i.e. transformation to <a href="http://dbpedia.org/resource/Linked_Data" id="link-id19744878">Linked Data</a>) is a synch thanks to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id180dde30">RDF</a> Middleware (as per <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&type=text&kwds=self%20annotation&OpenSearch" id="link-id16dc3130">earlier RDF middleware posts</a>).</p>
Kingsley Uyi Idehen
kidehen@openlinksw.com
2008-05-16T18:04:03.000-04:00
Commercializing the Semantic Web
http://www.openlinksw.com/blog/~kidehen/?id=1362
2008-05-16T20:02:45Z
<p>Unfortunately, I could only spend 4 days at the recent <a href="http://www2008.org/" id="link-id196acf60">WWW2008</a> event in <a href="http://dbpedia.org/resource/Beijing" id="link-id1974fe28">Beijing</a> (I departed the morning following the <a href="http://events.linkeddata.org/ldow2008/" id="link-id1863f858">Linked Data Workshop</a>), so I couldn't take my slot on the "Commercializing the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id18990f90">Semantic Web</a> panel" etc.. Anyway, thanks to the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> I can still inject my points of view in the broad Web based discourse. Well so I hoped, when I attempted to post a comment to Paul Miller's ZDNet domain hosted <a href="http://dbpedia.org/resource/Blog" id="link-id180d6750">blog</a> thread titled: <a href="http://blogs.zdnet.com/semantic-web/?p=132" id="link-id12d206c0">Commercialising the Semantic Web</a>.</p> <p>Unfortunately, the cost of completing ZDNet's unwieldy signup process simply exceeded the benefits of dropping my comments in their particular space :-( Thus, I'll settle for a trackback ping instead.</p> <p>What follows is the cut and paste of my intended comment contributions to Paul's post.</p> <p>Paul,</p> <p> As discussed earlier this week during <a href="http://blogs.talis.com/nodalities/2008/05/kingsley-idehen-talks-about-openlink-software-linked-data-and-the-semantic-web.php" id="link-id1332fb48">our podcast session</a>, commercialization of <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id17382338">Semantic Web</a> technology shouldn't be a mercurial matter at this stage in the game :-) It's all about looking at how it provides value :-)</p> <p>From the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10d4f4a8">Linked Data</a> angle, the ability to produce, dispatch, and exploit "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id13bed160">Context</a>" across an array of "Perspectives" from a plethora of disparate <a href="http://dbpedia.org/resource/Data" id="link-id1731e5f0">data</a> sources on the Web and/or behind corporate firewalls, offers immense commercial value.</p> <p> <a href="http://developer.yahoo.com/searchmonkey/" id="link-id1975d248">Yahoo's Searchmonkey</a> effort will certainly bring clarity to some of the points I made during the podcast re. the role of URIs as "value consumption tickets" (<a href="http://dbpedia.org/resource/Data" id="link-id173eb7b0">Data</a> Services are exposed via URIs). There has to be a trigger (in user space) that compels Web users to seek broader, or simply varied, perspectives as a response to <a href="http://dbpedia.org/resource/Data" id="link-id0x1c7e7f60">data</a> encountered on the Web. Yahoo! is about to put this light on in a big way (imho).</p> <p>The "self annotating" nature of the Web is what ultimately drives the manifestation of the long awaited <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0xa18a83e8">Semantic Web</a>. I believe I postulated about <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=self%20annotation&type=text&output=html" id="link-id173d7458">"Self Annotation & the Semantic Web" in a number of prior posts</a> which, by the way, should be <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&type=text&kwds=self%20annotation&OpenSearch" id="link-id10b12208">DataRSS compatible right now</a> due to Yahoo's support of OpenSearch <a href="http://dbpedia.org/resource/Data">Data</a> Providers (which this <a href="http://dbpedia.org/resource/Blog" id="link-id170b8df8">Blog</a> Space has been for eons).</p> <p>Today, have many communities adding strucuture to the Web (via their respective tools of preference) without explicitly realizing what they are contributing. Every RSS/Atom feed, <a href="http://dbpedia.org/resource/Tag" id="link-id183d5178">Tag</a>, Weblog, Shared Bookmark, <a href="http://dbpedia.org/resource/WikiWord" id="link-id10c5e758">Wikiword</a>, Microformat, Microformat++ (<a href="http://dbpedia.org/resource/Embedded_RDF" id="link-id16d8ee40">eRDF</a> or <a href="http://dbpedia.org/resource/RDFa" id="link-id1059a688">RDFa</a>), <a href="http://dbpedia.org/resource/GRDDL" id="link-id1090ae10">GRDDL</a> stylesheet, and RDFizer etc.. is a piece of structured data.</p> <p>Finally, the different communities are all finding ways to work together (thank heavens!) and the results are going to be cataclysmic when it all plays out :-)</p> <p>Data, Structure, and Extraction are the keys to the Semantic Life! First you get the Data in a container (<a href="http://dbpedia.org/resource/Information" id="link-id180e5648">information</a> resource), and then you add Structure to the <a href="http://dbpedia.org/resource/Information" id="link-id103801e0">information</a> resource (RSS, Atom, <a href="http://dbpedia.org/resource/Microformats" id="link-id17825e40">microformats</a>, <a href="http://dbpedia.org/resource/RDFa" id="link-id189a8738">RDFa</a>, <a href="http://dbpedia.org/resource/Embedded_RDF" id="link-id1933d5c0">eRDF</a>, SIOC, FOAF, etc.), once you have Structure RDFization (i.e. transformation to <a href="http://dbpedia.org/resource/Linked_Data" id="link-id19744878">Linked Data</a>) is a synch thanks to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id180dde30">RDF</a> Middleware (as per <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&type=text&kwds=self%20annotation&OpenSearch" id="link-id16dc3130">earlier RDF middleware posts</a>).</p>
Kingsley Uyi Idehen
kidehen@openlinksw.com
2008-05-16T16:15:29.1000-04:00
My Talis Podcast re. Semantic Web, Linked Data, and OpenLink Software
http://www.openlinksw.com/blog/~kidehen/?id=1361
2008-05-16T00:10:23Z
<p> <a href="http://blogs.talis.com/nodalities/2008/05/kingsley-idehen-talks-about-openlink-software-linked-data-and-the-semantic-web.php" id="link-id1036b118">My podcast interview</a> with <a href="http://www.linkedin.com/in/pau1mi11er" id="link-id1026ed10">Paul Miller</a> of <a href="http://www.talis.com" id="link-id12d210d8">Talis</a> is out. As I listened to the podcast (naturally awkward affair) I got a first hand sense of Paul's mastery of the art of interviewing, even when dealing with a fast talking <a href="http://dbpedia.org/resource/Data" id="link-id180e1208">data</a> blitzers like me. Personally, I think I still talk a little too fast (the Nigerian in me), especially when the subject matter hones right into the epicenter of my professional passions: Open <a href="http://dbpedia.org/resource/Data" id="link-id1737a258">Data</a> Access and Heterogeneous <a href="http://dbpedia.org/resource/Data" id="link-id180f0668">Data</a> Integration (aka. <a href="http://dbpedia.org/resource/Virtual_Database" id="link-id10c62348">Virtual Database</a> Technology) -- so you may need to rewind every now and then during the interview :-)</p> <p>During this particular podcast interview, I deliberately wanted to have an conversation about the practical value of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id180c9f88">Linked Data</a>, rather than the technical innards. The fundamental utility of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id17387618">Linked Data</a> remains somewhat mercurial, and I am certainly hoping to do my bit at the upcoming <a href="http://dbpedia.org/resource/Linked_Data" id="link-id183ec288">Linked Data</a> Planet conference re. demonstrating and articulating <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1401f250">linked data</a> value across the blurring realms of "the individual" and "the enterprise".</p> <p> <strong>Note to my old schoolmates on Facebook</strong>: when you listen to this podcast you will at least reconcile "Uyi Idehen" with "<a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id180a7060">Kingsley Idehen</a>". Unfortunately, Facebook refuses to let me Identify myself in the manner I choose. Ideally, I would like to have the name: "Kingsley (Uyi) Idehen" associated with my Facebook ID since this is the Identifier known to my personal network of friends, family, and old schoolmates. This Identity predicament is a long running Identity case study in the making.</p>
Kingsley Uyi Idehen
kidehen@openlinksw.com
2008-05-16T12:53:49.2000-04:00
On "Semantic", "Semantic Web", and "Linked Data Web"
http://www.openlinksw.com/blog/~kidehen/?id=1360
2008-05-15T14:11:13Z
<p> <a href="http://novaspivack.typepad.com/" id="link-id102f4e00">Nova Spivack</a> has just penned a post titled: <a href="http://novaspivack.typepad.com/nova_spivacks_weblog/2008/05/on-the-differen.html" id="link-id101a2300">On the Difference Between "Semantic" and "Semantic Web</a>", where he covers the fundamental difference between "Semantic" (what I call "Semantics Inside") and "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id11dd0578">Semantic Web</a>" applications. I would like to extend the distinctions further by adding the "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id10b54ca0">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id106f73d0">Web</a>" distinctions to the developing discourse. </p> <p>The <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1089ff48">Linked Data Web</a> (aka. <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10653828">Linked Data</a>) describes <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id134abfb0">RDF</a> <a href="http://dbpedia.org/resource/Data" id="link-id140283a8">data</a> injected into the Web, where the <a href="http://dbpedia.org/resource/Identity_(object-oriented_programming)" id="link-id1029ebf0">Data Object Identifiers</a> (URIs) in an <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id1011b180">RDF</a> graph (collection of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id103a4960">RDF</a> triples) are endowed with <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id104362d8">HTTP</a> based URIs. The net effect of this approach to <a href="http://dbpedia.org/resource/Data" id="link-id107963a0">Data</a> Object Identity is that it facilitates "Open <a href="http://dbpedia.org/resource/Data" id="link-id1331f640">Data</a> Access by Reference" on the Web (aka <a href="http://dbpedia.org/resource/Data" id="link-id10a3c608">data</a> dereferencing).</p> <p>If you recall pre Web ubiquity, in the enterprise realm for instance, Open Database Connectivity (<a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id12c6dd40">ODBC</a>) emerged as a mechanism for separating <a href="http://dbpedia.org/resource/Data" id="link-id13d6a5b0">Data</a> Access and <a href="http://dbpedia.org/resource/Data" id="link-id10b29488">Data</a> Management in the database oriented Client-Sever model. Although <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id106a8bd8">ODBC</a> gave you access to <a href="http://dbpedia.org/resource/Data">data</a>, the data access entry point took the form of a data access specific naming mechanism called a "Data Source Name" (DSN). <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id106eef18">ODBC</a> DSNs typically exposed Tables or Views. The same thing applies to <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id12c6dfe8">JDBC</a> where a non <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id104cb620">HTTP</a> based URN scheme applies.</p> <p>Zip forward to where we are today on the Web; the Web is evolving from a Document centric Database to a Distributed <a href="http://dbpedia.org/resource/Object_database" id="link-id12d15268">Object Database</a>, and you should see that in <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10716bb8">Linked Data</a> we are now truly looking at the best of all worlds: Web Open Database Connectivity (WODBC) with the following advantages:</p> <ul>- direct Access to a single Record (an <a href="http://dbpedia.org/resource/Entity" id="link-id1037d530">Entity</a>) or Record Sets (<a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id10d48e98">RDF</a> based <a href="http://dbpedia.org/resource/Entity" id="link-id1402c8f0">Entity</a> Sets) by reference over <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id10bae7a8">HTTP</a> across disparate Data Spaces on the Web</ul> <ul>- the ability to mesh disparate data sources without being impeded by back-end DBMS engine model, vendor, host operating development frameworks, or host operating system specificity</ul> <ul>- an opportunity to learn from the enterprise DBMS market and Client-Server markets of yore with regards to the shape and form of next generation <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10fe4558">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10153c98">Web</a> oriented solutions.</ul> <p>To conclude, we now have "Semantics Inside" (<a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id109d1280">RDF</a> or non <a href="http://dbpedia.org/resource/Resource_Description_Framework">RDF</a>), "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id106741a8">Semantic Web</a>" (RDF graphs with Object Identifiers that may or may not be <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id1011cc28">HTTP</a> based), and "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id10793f70">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id149ecc10">Web</a>" (RDF graphs with Object Identifiers that must be <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id10a3b860">HTTP</a> based and dereferencable) oriented applications, in the emerging landscape associated with the "Semantics" moniker.</p> <p>As per usual, this post is a record in my <a href="http://dbpedia.org/resource/Blog" id="link-id1020e240">Blog</a> oriented <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id105cbf90">Data Space</a> on the Web. The permalink of this post is a <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id10ce53a8">URI</a> constructed with <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id1082f0f8">Giant Global Graph</a> enrichment in mind :-)</p>
Kingsley Uyi Idehen
kidehen@openlinksw.com
2008-05-15T14:31:38.4000-04:00
DBpedia Benchmark Revisited
http://www.openlinksw.com/blog/vdb/blog/?id=1359
2008-05-09T19:33:42Z
<div> <div style="display:none;">DBpedia Benchmark Revisited</div> <p>We ran the <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x1cd6d0c8">DBpedia</a> benchmark queries again with different configurations of <a href="http://virtuoso.openlinksw.com" id="link-id0x1bf01048">Virtuoso</a>. I had not studied the details of the matter previously but now did have a closer look at the queries.</p> <p>Comparing numbers given by different parties is a constant problem. In the case reported here, we loaded the full DBpedia 3, all languages, with about 198M triples, onto Virtuoso v5 and Virtuoso Cluster v6, all on the same 4 core 2GHz Xeon with 8G RAM. All databases were striped on 6 disks. The Cluster configuration was with 4 processes in the same box.</p> <p>We ran the queries in two variants:</p> <ul> <li>With graph specified in the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1b9d3ca0">SPARQL</a> <code>FROM</code> clause, using the default indices.</li> <li>With no graph specified anywhere, using an alternate indexing scheme.</li> </ul> <p>The times below are for the sequence of 5 queries; individual query times are not reported. I did not do a line-by-line review of the execution plans since they seem to run well enough. We could get some extra mileage from cost model tweaks, especially for the numeric range conditions, but we will do this when somebody comes up with better times.</p> <p>First, about Virtuoso v5: Because there is a query in the set that specifies no condition on S or O and only P, this simply cannot be done with the default indices. With Virtuoso Cluster v6 it sort-of can, because v6 is more space efficient.</p> <p>So we added the index:</p> <blockquote> <code> create bitmap index <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1c364a58">rdf</a>_quad_pogs on rdf_quad (p, o, g, s); </code> </blockquote> <table> <tr> <td> </td> <td align="center"><b>Virtuoso v5 with<br /> gspo, ogps, pogs</b> </td> <td align="center"><b>Virtuoso Cluster v6 with <br />gspo, ogps</b> </td> <td align="center"><b>Virtuoso Cluster v6 with <br />gspo, ogps, pogs</b> </td> </tr> <tr> <td><b>cold</b> </td> <td align="center">210 s</td> <td align="center">136 s</td> <td align="center">33.4 s</td> </tr> <tr> <td><b>warm</b> </td> <td align="center">0.600 s</td> <td align="center">4.01 s</td> <td align="center">0.628 s</td> </tr> </table> <p>OK, so now let us do it without a graph being specified. For all platforms, we drop any existing indices, and --</p> <blockquote> <code> create table r2 (g iri_id_8, s, iri_id_8, p iri_id_8, o any, primary key (s, p, o, g)) <br /> alter index R2 on R2 partition (s int (0hexffff00)); <br /> <br /> log_enable (2); <br /> insert into r2 (g, s, p, o) select g, s, p, o from rdf_quad; <br /> <br /> drop table rdf_quad; <br /> alter table r2 rename RDF_QUAD; <br /> create bitmap index rdf_quad_opgs on rdf_quad (o, p, g, s) partition (o varchar (-1, 0hexffff)); <br /> create bitmap index rdf_quad_pogs on rdf_quad (p, o, g, s) partition (o varchar (-1, 0hexffff)); <br /> create bitmap index rdf_quad_gpos on rdf_quad (g, p, o, s) partition (o varchar (-1, 0hexffff)); </code> </blockquote> <p>The code is identical for v5 and v6, except that with v5 we use <code>iri_id (32 bit)</code> for the type, not <code>iri_id_8 (64 bit)</code>. We note that we run out of IDs with v5 around a few billion triples, so with v6 we have double the ID length and still manage to be vastly more space efficient.</p> <p>With the above 4 indices, we can query the <a href="http://dbpedia.org/resource/Data" id="link-id0x1bae4cd8">data</a> pretty much in any combination without hitting a full scan of any index. We note that all indices that do not begin with s end with s as a bitmap. This takes about 60% of the space of a non-bitmap index for data such as DBpedia.</p> <p>If you intend to do completely arbitrary RDF queries in Virtuoso, then chances are you are best off with the above index scheme.</p> <table> <tr> <td> </td> <td align="center"><b> Virtuoso v5 with<br /> gspo, ogps, pogs</b> </td> <td align="center"><b> Virtuoso Cluster v6 with <br /> spog, pogs, opgs, gpos </b> </td> </tr> <tr> <td><b>warm</b> </td> <td align="center">0.595 s</td> <td align="center">0.617 s</td> </tr> </table> <p>The cold times were about the same as above, so not reproduced.</p> <h3>Graph or No Graph?</h3> <p>It is in the SPARQL spirit to specify a graph and for pretty much any application, there are entirely sensible ways of keeping the data in graphs and specifying which ones are concerned by queries. This is why Virtuoso is set up for this by default.</p> <p>On the other hand, for the open web scenario, dealing with an unknown large number of graphs, enumerating graphs is not possible and questions like which graph of which source asserts x become relevant. We have two distinct use cases which warrant different setups of the database, simple as that.</p> <p>The latter use case is not really within the SPARQL spec, so implementations may or may not support this. For example <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x1cd2db78">Oracle</a> or Vertica would not do this well since they partition data according to graph or predicate, respectively. On the other hand, stores that work with one quad table, which is most of the ones out there, should do it maybe with some configuring, as shown above.</p> <p>Frameworks like Jena are not to my <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x1b300390">knowledge</a> geared towards having a wildcard for graph, although I would suppose this can be arranged by adding some "super-graph" object, a graph of all graphs. I don't think this is directly supported and besides most apps would not need it.</p> <p>Once the indices are right, there is no difference between specifying a graph and not specifying a graph with the queries considered. With more complex queries, specifying a graph or set of graphs does allow some optimizations that cannot be done with no graph specified. For example, bitmap intersections are possible only when all leading key parts are given.</p> <h3>Conclusions</h3> <p>The best warm cache time is with v5; the five queries run under 600 ms after the first go. This is noted to show that all-in-memory with a single thread of execution is hard to beat.</p> <p>Cluster v6 performs the same queries in 623 ms. What is gained in parallelism is lost in latency if all operations complete in microseconds. On the other hand, Cluster v6 leaves v5 in the dust in any situation that has less than 100% hit rate. This is due to actual benefit from parallelism if operations take longer than a few microseconds, such as in the case of disk reads. Cluster v6 has substantially better data layout on disk, as well as fewer pages to load for the same content.</p> <p>This makes it possible to run the queries without the pogs index on Cluster v6 even when v5 takes prohibitively long.</p> <p>The morale of the story is to have a lot of RAM and space-efficient data representation.</p> <p>The DBpedia benchmark does not specify any random access pattern that would give a measure of sustained throughput under load, so we are left with the extremes of cold and warm cache of which neither is quite realistic.</p> <p>Chris Bizer and I have talked on and off about benchmarks and I have made suggestions that we will see incorporated into the Berlin SPARQL benchmark, which will, I believe, be much more informative.</p> <h3>Appendix: Query Text</h3> <p>For reference, the query texts specifying the graph are below. To run without specifying the graph, just drop the <code>FROM <<a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1c371db0">http</a>://dbpedia.org></code> from each query. The returned row counts are indicated below each query's text.</p> <blockquote> <code><pre> sparql SELECT ?p ?o FROM <http://dbpedia.org> WHERE { <http://dbpedia.org/resource/Metropolitan_Museum_of_Art> ?p ?o }; -- 1337 rows sparql PREFIX p: <http://dbpedia.org/property/> SELECT ?film1 ?actor1 ?film2 ?actor2 FROM <http://dbpedia.org> WHERE { ?film1 p:starring <http://dbpedia.org/resource/Kevin_Bacon> . ?film1 p:starring ?actor1 . ?film2 p:starring ?actor1 . ?film2 p:starring ?actor2 . }; -- 23910 rows sparql PREFIX p: <http://dbpedia.org/property/> SELECT ?artist ?artwork ?museum ?director FROM <http://dbpedia.org> WHERE { ?artwork p:artist ?artist . ?artwork p:museum ?museum . ?museum p:director ?director }; -- 303 rows sparql PREFIX geo: <http://www.w3.org/2003/01/geo/wgs84_pos#> PREFIX foaf: <http://xmlns.com/foaf/0.1/> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> SELECT ?s ?homepage FROM <http://dbpedia.org> WHERE { <http://dbpedia.org/resource/Berlin> geo:lat ?berlinLat . <http://dbpedia.org/resource/Berlin> geo:long ?berlinLong . ?s geo:lat ?lat . ?s geo:long ?long . ?s foaf:homepage ?homepage . FILTER ( ?lat <= ?berlinLat + 0.03190235436 && ?long >= ?berlinLong - 0.08679199218 && ?lat >= ?berlinLat - 0.03190235436 && ?long <= ?berlinLong + 0.08679199218) }; -- 56 rows sparql PREFIX geo: <http://www.w3.org/2003/01/geo/wgs84_pos#> PREFIX foaf: <http://xmlns.com/foaf/0.1/> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> PREFIX p: <http://dbpedia.org/property/> SELECT ?s ?a ?homepage FROM <http://dbpedia.org> WHERE { <http://dbpedia.org/resource/New_York_City> geo:lat ?nyLat . <http://dbpedia.org/resource/New_York_City> geo:long ?nyLong . ?s geo:lat ?lat . ?s geo:long ?long . ?s p:architect ?a . ?a foaf:homepage ?homepage . FILTER ( ?lat <= ?nyLat + 0.3190235436 && ?long >= ?nyLong - 0.8679199218 && ?lat >= ?nyLat - 0.3190235436 && ?long <= ?nyLong + 0.8679199218) }; -- 13 rows </pre> </code> </blockquote> </div>
Virtuso Data Space Bot
kidehen@openlinksw.com
2008-05-12T11:24:43.000-04:00
DBpedia Benchmark Revisited
http://www.openlinksw.com/weblog/oerling/?id=1358
2008-05-09T19:27:00Z
<p>We ran the <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x1b7f9688">DBpedia</a> benchmark queries again with different configurations of <a href="http://virtuoso.openlinksw.com" id="link-id0x1cca2e00">Virtuoso</a>. I had not studied the details of the matter previously but now did have a closer look at the queries.</p> <p>Comparing numbers given by different parties is a constant problem. In the case reported here, we loaded the full DBpedia 3, all languages, with about 198M triples, onto Virtuoso v5 and Virtuoso Cluster v6, all on the same 4 core 2GHz Xeon with 8G RAM. All databases were striped on 6 disks. The Cluster configuration was with 4 processes in the same box.</p> <p>We ran the queries in two variants:</p> <ul> <li>With graph specified in the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1b77f758">SPARQL</a> <code>FROM</code> clause, using the default indices.</li> <li>With no graph specified anywhere, using an alternate indexing scheme.</li> </ul> <p>The times below are for the sequence of 5 queries; individual query times are not reported. I did not do a line-by-line review of the execution plans since they seem to run well enough. We could get some extra mileage from cost model tweaks, especially for the numeric range conditions, but we will do this when somebody comes up with better times.</p> <p>First, about Virtuoso v5: Because there is a query in the set that specifies no condition on S or O and only P, this simply cannot be done with the default indices. With Virtuoso Cluster v6 it sort-of can, because v6 is more space efficient.</p> <p>So we added the index:</p> <blockquote> <code> create bitmap index <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1cb0b180">rdf</a>_quad_pogs on rdf_quad (p, o, g, s); </code> </blockquote> <table> <tr> <td> </td> <td align="center"><b>Virtuoso v5 with<br /> gspo, ogps, pogs</b> </td> <td align="center"><b>Virtuoso Cluster v6 with <br />gspo, ogps</b> </td> <td align="center"><b>Virtuoso Cluster v6 with <br />gspo, ogps, pogs</b> </td> </tr> <tr> <td><b>cold</b> </td> <td align="center">210 s</td> <td align="center">136 s</td> <td align="center">33.4 s</td> </tr> <tr> <td><b>warm</b> </td> <td align="center">0.600 s</td> <td align="center">4.01 s</td> <td align="center">0.628 s</td> </tr> </table> <p>OK, so now let us do it without a graph being specified. For all platforms, we drop any existing indices, and --</p> <blockquote> <code> create table r2 (g iri_id_8, s, iri_id_8, p iri_id_8, o any, primary key (s, p, o, g)) <br /> alter index R2 on R2 partition (s int (0hexffff00)); <br /> <br /> log_enable (2); <br /> insert into r2 (g, s, p, o) select g, s, p, o from rdf_quad; <br /> <br /> drop table rdf_quad; <br /> alter table r2 rename RDF_QUAD; <br /> create bitmap index rdf_quad_opgs on rdf_quad (o, p, g, s) partition (o varchar (-1, 0hexffff)); <br /> create bitmap index rdf_quad_pogs on rdf_quad (p, o, g, s) partition (o varchar (-1, 0hexffff)); <br /> create bitmap index rdf_quad_gpos on rdf_quad (g, p, o, s) partition (o varchar (-1, 0hexffff)); </code> </blockquote> <p>The code is identical for v5 and v6, except that with v5 we use <code>iri_id (32 bit)</code> for the type, not <code>iri_id_8 (64 bit)</code>. We note that we run out of IDs with v5 around a few billion triples, so with v6 we have double the ID length and still manage to be vastly more space efficient.</p> <p>With the above 4 indices, we can query the <a href="http://dbpedia.org/resource/Data" id="link-id0x6339b80">data</a> pretty much in any combination without hitting a full scan of any index. We note that all indices that do not begin with s end with s as a bitmap. This takes about 60% of the space of a non-bitmap index for data such as DBpedia.</p> <p>If you intend to do completely arbitrary RDF queries in Virtuoso, then chances are you are best off with the above index scheme.</p> <table> <tr> <td> </td> <td align="center"><b> Virtuoso v5 with<br /> gspo, ogps, pogs</b> </td> <td align="center"><b> Virtuoso Cluster v6 with <br /> spog, pogs, opgs, gpos </b> </td> </tr> <tr> <td><b>warm</b> </td> <td align="center">0.595 s</td> <td align="center">0.617 s</td> </tr> </table> <p>The cold times were about the same as above, so not reproduced.</p> <h3>Graph or No Graph?</h3> <p>It is in the SPARQL spirit to specify a graph and for pretty much any application, there are entirely sensible ways of keeping the data in graphs and specifying which ones are concerned by queries. This is why Virtuoso is set up for this by default.</p> <p>On the other hand, for the open web scenario, dealing with an unknown large number of graphs, enumerating graphs is not possible and questions like which graph of which source asserts x become relevant. We have two distinct use cases which warrant different setups of the database, simple as that.</p> <p>The latter use case is not really within the SPARQL spec, so implementations may or may not support this. For example <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id0x11ed7028">Oracle</a> or Vertica would not do this well since they partition data according to graph or predicate, respectively. On the other hand, stores that work with one quad table, which is most of the ones out there, should do it maybe with some configuring, as shown above.</p> <p>Frameworks like Jena are not to my <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x1a49ded0">knowledge</a> geared towards having a wildcard for graph, although I would suppose this can be arranged by adding some "super-graph" object, a graph of all graphs. I don't think this is directly supported and besides most apps would not need it.</p> <p>Once the indices are right, there is no difference between specifying a graph and not specifying a graph with the queries considered. With more complex queries, specifying a graph or set of graphs does allow some optimizations that cannot be done with no graph specified. For example, bitmap intersections are possible only when all leading key parts are given.</p> <h3>Conclusions</h3> <p>The best warm cache time is with v5; the five queries run under 600 ms after the first go. This is noted to show that all-in-memory with a single thread of execution is hard to beat.</p> <p>Cluster v6 performs the same queries in 623 ms. What is gained in parallelism is lost in latency if all operations complete in microseconds. On the other hand, Cluster v6 leaves v5 in the dust in any situation that has less than 100% hit rate. This is due to actual benefit from parallelism if operations take longer than a few microseconds, such as in the case of disk reads. Cluster v6 has substantially better data layout on disk, as well as fewer pages to load for the same content.</p> <p>This makes it possible to run the queries without the pogs index on Cluster v6 even when v5 takes prohibitively long.</p> <p>The morale of the story is to have a lot of RAM and space-efficient data representation.</p> <p>The DBpedia benchmark does not specify any random access pattern that would give a measure of sustained throughput under load, so we are left with the extremes of cold and warm cache of which neither is quite realistic.</p> <p>Chris Bizer and I have talked on and off about benchmarks and I have made suggestions that we will see incorporated into the Berlin SPARQL benchmark, which will, I believe, be much more informative.</p> <h3>Appendix: Query Text</h3> <p>For reference, the query texts specifying the graph are below. To run without specifying the graph, just drop the <code>FROM <<a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1905bfd0">http</a>://dbpedia.org></code> from each query. The returned row counts are indicated below each query's text.</p> <blockquote> <code><pre> sparql SELECT ?p ?o FROM <http://dbpedia.org> WHERE { <http://dbpedia.org/resource/Metropolitan_Museum_of_Art> ?p ?o }; -- 1337 rows sparql PREFIX p: <http://dbpedia.org/property/> SELECT ?film1 ?actor1 ?film2 ?actor2 FROM <http://dbpedia.org> WHERE { ?film1 p:starring <http://dbpedia.org/resource/Kevin_Bacon> . ?film1 p:starring ?actor1 . ?film2 p:starring ?actor1 . ?film2 p:starring ?actor2 . }; -- 23910 rows sparql PREFIX p: <http://dbpedia.org/property/> SELECT ?artist ?artwork ?museum ?director FROM <http://dbpedia.org> WHERE { ?artwork p:artist ?artist . ?artwork p:museum ?museum . ?museum p:director ?director }; -- 303 rows sparql PREFIX geo: <http://www.w3.org/2003/01/geo/wgs84_pos#> PREFIX foaf: <http://xmlns.com/foaf/0.1/> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> SELECT ?s ?homepage FROM <http://dbpedia.org> WHERE { <http://dbpedia.org/resource/Berlin> geo:lat ?berlinLat . <http://dbpedia.org/resource/Berlin> geo:long ?berlinLong . ?s geo:lat ?lat . ?s geo:long ?long . ?s foaf:homepage ?homepage . FILTER ( ?lat <= ?berlinLat + 0.03190235436 && ?long >= ?berlinLong - 0.08679199218 && ?lat >= ?berlinLat - 0.03190235436 && ?long <= ?berlinLong + 0.08679199218) }; -- 56 rows sparql PREFIX geo: <http://www.w3.org/2003/01/geo/wgs84_pos#> PREFIX foaf: <http://xmlns.com/foaf/0.1/> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> PREFIX p: <http://dbpedia.org/property/> SELECT ?s ?a ?homepage FROM <http://dbpedia.org> WHERE { <http://dbpedia.org/resource/New_York_City> geo:lat ?nyLat . <http://dbpedia.org/resource/New_York_City> geo:long ?nyLong . ?s geo:lat ?lat . ?s geo:long ?long . ?s p:architect ?a . ?a foaf:homepage ?homepage . FILTER ( ?lat <= ?nyLat + 0.3190235436 && ?long >= ?nyLong - 0.8679199218 && ?lat >= ?nyLat - 0.3190235436 && ?long <= ?nyLong + 0.8679199218) }; -- 13 rows </pre> </code> </blockquote>
Orri Erling
oerling@openlinksw.com
2008-05-12T11:24:36.000-04:00
Comments about recent Semantic Gang Podcast
http://www.openlinksw.com/blog/~kidehen/?id=1357
2008-05-02T21:44:31Z
<p>After listening to the <a href="http://semanticgang.talis.com/2008/05/02/april-2008-the-semantic-web-gang-discuss-a-wikipedia-for-data/" id="link-id1089e218">latest Semantic Web Gang podcast</a>, I found myself agreeing with some of the points made by <a href="http://www.linkedin.com/in/iskold" id="link-id10b91e58">Alex Iskold</a>, specifically: </p> <ul>-- <a href="http://dbpedia.org/resource/Linked_Data" id="link-id106e24e0">Linked Data</a> does not implicitly imply making all your <a href="http://dbpedia.org/resource/Data" id="link-id17ab3d48">data</a> public</ul> <ul>-- <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11fdcef0">Linked Data</a> principles benefit <a href="http://dbpedia.org/resource/Intranet" id="link-id109756e8">Intranet</a> and <a href="http://dbpedia.org/resource/Extranet" id="link-id1099cfd8">Extranet</a> style <a href="http://dbpedia.org/resource/Data" id="link-id10cd25b0">data</a> integration (trumps alternative <a href="http://dbpedia.org/resource/federated_database_system" id="link-id14f29940">distributed database</a> integration approaches any day)</ul> <ul>-- Business exploitation of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0xca51940">Linked Data</a> on the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> will certainly be driven by the correlation of opportunity costs (which is more than likely what Alex meant by "use cases") associated with the lack of URIs originating from the domain of a given business (Tom Heath: also effectively alluded to this via his <a href="http://dbpedia.org/resource/BBC" id="link-id16f33348">BBC</a> and <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id10decf38">URI</a> land grab anecdotes; same applies Georgi's examples)</ul> <ul>-- History is a great tutor, answers to many of today's problems always lie somewhere in plain sight of the past.</ul> <p>Of course, I also believe that <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> serves Web <a href="http://dbpedia.org/resource/Data" id="link-id0x1afebd58">Data</a> Integration across the <a href="http://dbpedia.org/resource/Internet" id="link-id10aa5668">Internet</a> very well too, and the fact that it will be beneficial to businesses in a big way. No individual or organization is an island, I think the <a href="http://dbpedia.org/resource/Internet" id="link-id0xb25fbd0">Internet</a> and Web have done a good job of demonstrating that thus far :-) We're all <a href="http://dbpedia.org/resource/Data">data</a> nodes in a <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id5d8a3a8">Giant Global Graph</a>.</p> <p> <a href="http://myopenlink.net/dataspace/person/danieljohnlewis#this" id="link-id17cac8a0">Daniel lewis</a> did shed light on the read-write aspects of the Linked Data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10be8590">Web</a>, which is actually very close to the callout for a Wikipedia for Data. <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id10a810c0">TimBL</a> has been working on this via <a href="http://dig.csail.mit.edu/2005/ajar/release/tabulator/0.8/tab.html" id="link-id184b7108">Tabulator</a> (see <a href="http://dig.csail.mit.edu/2007/tab/tutorial/editing.mov" id="link-id1416f1e8">Tabulator Editing Screencast</a>), <a href="http://bnode.org/about" id="link-id17e33750">Bengamin Nowack</a> also added <a href="http://arc.semsol.org/download/plugins/data_wiki" id="link-id1688cc40">similar functionality to ARC</a>, and of course we support the same <a href="http://dbpedia.org/resource/SPARQL" id="link-id10bff7c8">SPARQL</a> UPDATE into an <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id168ace08">RDF</a> <a href="http://dbpedia.org/resource/Information" id="link-id10641878">information</a> resource via the <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xddb5240">RDF</a> Sink feature of our WebDAV and <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/OdsBriefcase" id="link-id0x11199310">ODS</a>-Briefcase implementations.</p>
Kingsley Uyi Idehen
kidehen@openlinksw.com
2008-05-05T20:06:42.4000-04:00
In Perpetual Pursuit of Context
http://www.openlinksw.com/blog/~kidehen/?id=1356
2008-05-02T19:18:33Z
<p>I've always been of the opinion that concise value proposition articulation shouldn't be the achilles of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id158efe90">Semantic Web</a>. As the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id13a2db40">Linked Data</a> wave climbs up the "value Appreciation and Comprehension chain", it's getting clearer by the second that "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id109316f0">Context</a>" is a point of confluence for <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id108daa60">Semantic Web</a> Technologies and easy to comprehend value, from the perspectives of those outside the core community.</p> <p>In today's primarily Document centric <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a>, the pursuit of <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id14edadd0">Context</a> is akin to pursuing a mirage in a desert of user generated content. The quest is labor intensive, and you ultimaely end up without water at the end of the pursuit :-)</p> <p>Listening to the <a href="http://blogs.talis.com/nodalities/2008/05/christine-connors-talks-about-semantic-technologies-at-dow-jones.php" id="link-id12d5e1c0">Christine Connor's podcast interview with Talis</a> simply reinforces my strong belief that "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id0x1ec69518">Context</a>, Context, Context" is the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0xa279438">Semantic Web</a>'s equivalent of Real Estate's "Location, Location, Location" (ignore the <a href="http://dbpedia.org/resource/Subprime_lending" id="link-id140b8098">subprime</a> loans mess for now). The critical thing to note is that you cannot unravel "Context" from existing Web content without incorporating powerful disambiguation technology into an "<a href="http://dbpedia.org/resource/Entity" id="link-id15a2f380">Entity</a> Extraction" process. Of course, you cannot even consider seriously pursing any <a href="http://dbpedia.org/resource/Entity" id="link-id10868a18">entity</a> extraction and disambiguation endeavor without a lookup backbone that exposes "<a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id168dc230">Named Entities</a>" and their relationships to "<a href="http://dbpedia.org/resource/Topic" id="link-id17cb1950">Subject matter Concepts</a>" (BTW - this is what <a href="http://umbel.org/about/" id="link-id14f406a0">UMBEL</a> is all about). Thus, when looking at the broad subject of the <a href="http://dbpedia.org/resource/Semantic_Web">Semantic Web</a>, we can also look at "Context" as the vital point of confluence for the <a href="http://dbpedia.org/resource/Data" id="link-id12d67e38">Data</a> oriented (<a href="http://dbpedia.org/resource/Linked_Data" id="link-id14f8daf0">Linked Data</a>) and the "Linguistic Meaning" oriented perspectives.</p> <p>I am even inclined to state publicly that "Context" may ultimately be the foundation for <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=dimension%20web%204.0%20&type=text&output=html" id="link-id17cb0708">4th "Web Interaction Dimension"</a> where practical use of <a href="http://dbpedia.org/resource/Artificial_intelligence" id="link-id10b15088">AI</a> leverages a <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1ebf9310">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10b27018">Web</a> substrate en route to exposing new kinds of value :-)</p> <p>"Context" may also be the focal point of concise value proposition articulation to <a href="http://dbpedia.org/resource/Venture_Capital" id="link-id10837578">VCs</a> as in: "My solution offers the ability to discover and exploit "Context" iteratively, at the rate of $X.XX per iteration, across a variety of market segments :-)</p>
Kingsley Uyi Idehen
kidehen@openlinksw.com
2008-05-03T15:07:32.000-04:00
XTech Talks covering Linked Data
http://www.openlinksw.com/blog/~kidehen/?id=1355
2008-05-02T14:53:08Z
<p>Courtesy a post by <a href="http://community.linkeddata.org/dataspace/person/bizer#this" id="link-id10868548">Chris Bizer</a> to the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id15739748">LOD</a> community <a href="http://lists.w3.org/Archives/Public/public-lod/" id="link-id10fae0f8">mailing list</a>, here is a list of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id140a0880">Linked Data</a> oriented talks at the upcoming <a href="http://2008.xtech.org" id="link-id12801f00">XTech</a> 2008 event (also see the <a href="http://2008.xtech.org/public/schedule/grid" id="link-id10f65940">XTech 2008 Schedule</a> which is <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1258a4c8">Linked Data</a> friendly). Of course, I am posting this to my <a href="http://dbpedia.org/resource/Blog" id="link-id140a29c0">Blog</a> <a href="http://dbpedia.org/resource/Data" id="link-id12d5a640">Data</a> <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id10979b80">Space</a> with the sole purpose of adding <a href="http://dbpedia.org/resource/Data" id="link-id176be078">data</a> to the rapidly growing <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id1099aec8">Giant Global Graph</a> of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10d72d88">Linked Data</a>, basically adding to my collection of live <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11014000">Linked Data</a> utility demos :-)</p> <p>Here is the list:</p> <ol> <li> <a href="http://2008.xtech.org/public/schedule/detail/561" id="link-id17df4d78">Linked Data Deployment</a> (<a href="http://myopenlink.net/dataspace/person/danieljohnlewis#this" id="link-id17c47d28">Daniel Lewis</a>, <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id108fce00">OpenLink Software</a>)</li> <li> <a href="http://2008.xtech.org/public/schedule/detail/524" id="link-id1068c0e0">The Programmes Ontology</a> (Tom Scott, <a href="http://dbpedia.org/resource/BBC" id="link-id1566da50">BBC</a> and all) </li> <li> <a href="http://2008.xtech.org/public/schedule/detail/528" id="link-id1072be40">SemWebbing the London Gazette</a> (Jeni Tennison, The Stationery Office) </li> <li> <a href="http://2008.xtech.org/public/schedule/detail/583" id="link-id1099e4e0">Searching, publishing and remixing a Web of Semantic Data</a> (<a href="http://community.linkeddata.org/dataspace/person/cygri#this" id="link-id17e25b78">Richard Cyganiak</a>, DERI Galway) </li> <li> <a href="http://2008.xtech.org/public/schedule/detail/477" id="link-idf9764c8">Building a Semantic Web Search Engine: Challenges and Solutions</a> (Aidan Hogan, DERI Galway) </li> <li>'<a href="http://2008.xtech.org/public/schedule/detail/550" id="link-id140a3c50">That's not what you said yesterday!</a>' - evolving your <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> API (<a href="http://iandavis.com/id/me" id="link-id14f8d498">Ian Davis</a>, Talis) </li> <li> <a href="http://2008.xtech.org/public/schedule/detail/527" id="link-id10c5a9c8">Representing, indexing and mining scientific data using XML and RDF: Golem and CrystalEye</a> (<a href="http://wwmm.ch.cam.ac.uk/blogs/walkingshaw/" id="link-id108c5e28">Andrew Walkingshaw</a>, <a href="http://dbpedia.org/resource/University_of_Cambridge" id="link-id10891560">University of Cambridge</a>)</li> </ol> <p>For the time challenged (i.e. those unable to view this post using it's permalink / <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id10db39f0">URI</a> as a <a href="http://dbpedia.org/resource/Data" id="link-id10f29bb8">data</a> source via the <a href="http://demo.openlinksw.com/rdfbrowser" id="link-id10f72778">OpenLink RDF Browser</a>, <a href="http://zitgist.com/about/" id="link-id107b73b0">Zitgist</a> <a href="http://dataviewer.zitgist.com" id="link-id1686d528">Data Viewer</a>, <a href="http://www4.wiwiss.fu-berlin.de/rdf_browser" id="link-id110479e8">DISCO Hyperdata Browser</a>, or <a href="http://dig.csail.mit.edu/2005/ajar/release/tabulator/0.8/tab.html" id="link-id140ba0e8">Tabulator</a>), the benefits of this post are as follows:</p> <ul> <li>automatic <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id172d2fc8">URI</a> generation for all linked items in this post</li> <li>automatic propagation of tags to <a href="http://del.icio.us" id="link-id10547380">del</a>.<a href="http://del.icio.us" id="link-id1093cc10">icio</a>.<a href="http://del.icio.us" id="link-id168ce3a0">us</a>, <a href="http://www.technorati.com" id="link-id17aa8af0">Technorati</a>, and <a href="http://www.pingthesemanticweb.com/about/" id="link-id10868ad8">PingTheSemanticWeb</a> </li> <li>automatic association of formal meanings to my Tags using the <a href="http://moat-project.org/ontology" id="link-id10c98608">MOAT Ontology</a> </li> <li>automatic collation and generation of statistical <a href="http://dbpedia.org/resource/Data" id="link-id10a4d1d8">data</a> about my tags using the SCOT Ontology (*missing link is a callout to SCOT <a href="http://dbpedia.org/resource/Tag" id="link-id168b7c10">Tag</a> Ontology folks to sort the project's home page <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id11fd4118">URL</a> at the very least*) </li> <li>explicit typing of my Tags as <a href="http://dbpedia.org/resource/SKOS" id="link-id10940eb8">SKOS</a> Concepts. </li> </ul> <p>Put differently, I cost-effectively contribute to the <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10a081a8">GGG</a> across all <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=web%20dimensions&type=text&output=html" id="link-id10597530">Web interaction dimensions</a> (1.0, 2.0, 3.0) :-)</p>
Kingsley Uyi Idehen
kidehen@openlinksw.com
2008-05-05T17:07:17.000-04:00
SPARQL at WWW 2008
http://www.openlinksw.com/blog/vdb/blog/?id=1354
2008-04-30T16:28:10Z
<div> <div style="display:none;">SPARQL at WWW 2008</div> <p>Andy Seaborne and Eric Prud'hommeaux, editors of the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1501d1a0">SPARQL</a> recommendation, convened a SPARQL birds of a feather session at <a href="http://www2008.org/" id="link-id0xb9d6c10">WWW 2008</a>. The administrative outcome was that implementors could now experiment with extensions, hopefully keeping each other current about their efforts and that towards the end of 2008, a new W3C working group might begin formalizing the experiences into a new SPARQL spec.</p> <p>The session drew a good crowd, including many users and developers. The wishes were largely as expected, with a few new ones added. Many of the wishes already had diverse implementations, however most often without interop. I will below give some comments on the main issues discussed.</p> </div> <li> <p> <b>SPARQL Update</b> - This is likely the most universally agreed upon extension. Implementations exist, largely along the lines of Andy Seaborne's SPARUL spec, which is also likely material for a W3C member submission. The issue is without much controversy; transactions fall outside the scope, which is reasonable enough. With triple stores, we can define things as combinations of inserts and deletes, and isolation we just leave aside. If anything, operating on a transactional platform such as <a href="http://virtuoso.openlinksw.com" id="link-id0xc13fe98">Virtuoso</a>, one wishes to disable transactions for any operations such as bulk loads and long-running inserts and deletes. Transactionality has pretty much no overhead for a few hundred rows, but for a few hundred million rows the cost of locking and rollback is prohibitive. With Virtuoso, we have a row auto-commit mode which we recommend for use with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xd7bff00">RDF</a>: It commits by itself now and then, optionally keeping a roll forward log, and is transactional enough not to leave half triples around, i.e., inserted in one index but not another.</p> <p>As far as we are concerned, updating physical triples along the SPARUL lines is pretty much a done deal.</p> <p>The matter of updating relational <a href="http://dbpedia.org/resource/Data" id="link-id0x140ea538">data</a> mapped to RDF is a whole other kettle of fish. On this, I should say that RDF has no special virtues for expressing transactions but rather has a special genius for integration. Updating is best left to web service interfaces that use <a href="http://dbpedia.org/resource/SQL" id="link-id0xa24e9558">SQL</a> on the inside. Anyway, updating union views, which most mappings will be, is complicated. Besides, for transactions, one usually knows exactly what one wishes to update.</p> </li> <li> <p> <b>Full Text</b> - Many people expressed a desire for full text access. Here we run into a deplorable confusion with regexps. The closest SPARQL has to full text in its native form is regexps, but these are not really mappable to full text except in rare special cases and I would despair of explaining to an end user what exactly these cases are. So, in principle, some regexps are equivalent to full text but in practice I find it much preferable to keep these entirely separate.</p> <p>It was noted that what the users want is a text box for search words. This is a front end to the CONTAINS predicate of most SQL implementations. Ours is MS SQL Server compatible and has a SPARQL version called <code>bif:contains</code>. One must still declare which triples one wants indexed for full text, though. This admin overhead seems inevitable, as text indexing is a large overhead and not needed by all applications.</p> <p>Also, text hits are not boolean; usually they come with a hit score. Thus, a SPARQL extension for this could look like </p> <blockquote> <code>select * where { ?thing has_description ?d . ?d ftcontains "gizmo" ftand "widget" score ?score . }</code> </blockquote> <p>This would return all the subjects, descriptions, and scores, from subjects with a has_description property containing widget and gizmo. Extending the basic pattern is better than having the match in a filter, since the match binds a variable.</p> <p>The <a href="http://dbpedia.org/resource/XQuery" id="link-id0x9ddb7240">XQuery</a>/<a href="http://dbpedia.org/resource/XPath" id="link-id0x9d84e070">XPath</a> groups have recently come up with a full-text spec, so I used their style of syntax above. We already have a full-text extension, as do some others. but for standardization, it is probably most appropriate to take the XQuery work as a basis. The XQuery full-text spec is quite complex, but I would expect most uses to get by with a small subset, and the structure seems better thought out, at first glance, than the more ad-hoc implementations in diverse SQLs.</p> <p>Again, declaring any text index to support the search, as well as its timeliness or transactionality, are best left to implementations.</p> </li> <li> <p> <b>Federation</b> - This is a tricky matter. ARQ has a SPARQL extension for sending a nested set of triple patterns to a specific end-point. The DARQ project has something more, including a selectivity model for SPARQL.</p> <p>With federated SQL, life is simpler since after the views are expanded, we have a query where each table is at a known server and has more or less known statistics. Generally, execution plans where as much work as possible is pushed to the remote servers are preferred, and modeling the latencies is not overly hard. With SPARQL, each triple pattern could in principle come from any of the federated servers. Associating a specific end-point to a fragment of the query just passes the problem to the user. It is my guess that this is the best we can do without getting very elaborate, and possibly buggy, end-point content descriptions for routing federated queries.</p> <p>Having said this, there remains the problem of join order. I suggested that we enhance the protocol by allowing asking an end-point for the query cost for a given SPARQL query. Since they all must have a cost model for optimization, this should not be an impossible request. A time cost and estimated cardinality would be enough. Making statistics available <i>à la</i> DARQ was also discussed. Being able to declare cardinalities expected of a remote end-point is probably necessary anyway, since not all will implement the cost model interface. For standardization, agreeing on what is a proper description of content and cardinality and how fine grained this must be will be so difficult that I would not wait for it. A cost model interface would nicely hide this within the end-point itself.</p> <p>With Virtuoso, we do not have a federated SPARQL scheme but we could have the ARQ-like service construct. We'd use our own cost model with explicit declarations of cardinalities of the remote data for guessing a join order. Still, this is a bit of work. We'll see.</p> <p>For practicality, the service construct coupled with join order hints is the best short term bet. Making this pretty enough for standardization is not self-evident, as it requires end-point description and/or cost model hooks for things to stay declarative.</p> </li> <li> <p> <b>End-point description</b> - This question has been around for a while; I have <a href="http://www.openlinksw.com/weblog/oerling/?id=1085" id="link-id10fa7da8">blogged about it earlier</a>, but we are not really at a point where there would be even rough consensus about an end-point ontology. We should probably do something on our own to demonstrate some application of this, as we host lots of <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0xd048c68">linked open data</a> sets.</p> </li> <li> <p> <b>SQL equivalence</b> - There were many requests for aggregation, some for subqueries and nesting, expressions in select, negation, existence and so on. I would call these all SQL equivalence. One use case was taking all the teams in the database and for all with over 5 members, add the big_team class and a property for member count.</p> <p>With Virtuoso, we could write this as -- </p> <blockquote> <code>construct { ?team a big_team . ?team member_count ?ct } from ... where {?team a team . { select ?team2 count (*) as ?ct where { ?m member_of ?team2 } . filter (?team = ?team2 and ? ct > 5) }}</code> </blockquote> <p>We have pretty much all the SQL equivalence features, as we have been working for some time at translating the <a href="http://dbpedia.org/resource/TPC-H" id="link-id0xb9d5200">TPC-H</a> workload into SPARQL.</p> <p>The usefulness of these things is uncontested but standardization could be hard as there are subtle questions about variable scope and the like.</p> </li> <li> <p> <b>Inference</b> - The SPARQL spec does not deal with transitivity or such matters because it is assumed that these are handled by an underlying inference layer. This is however most often not so. There was interest in more fine grained control of inference, for example declaring that just one property in a query would be transitive or that subclasses should be taken into account in only one triple pattern. As far as I am concerned, this is very reasonable, and we even offer extensions for this sort of thing in Virtuoso's SPARQL. This however only makes sense if the inference is done at query time and pattern by pattern. For instance, if forward chaining is used, this no longer makes sense. Specifying that some forward chaining ought to be done at query time is impractical, as the operation can be very large and time consuming and it is the DBA's task to determine what should be stored and for how long, how changes should be propagated, and so on. All these are application dependent and standardizing will be difficult.</p> <p>Support for RDF features like lists and bags would all fall into the functions an underlying inference layer should perform. These things are of special interest when querying OWL models, for example.</p> </li> <li> <p> <b>Path expressions</b> - Path expressions were requested by a few people. We have implemented some, as in </p> <blockquote> <code>?product+?has_supplier+>s_name = "Gizmos, Inc.".</code> </blockquote> This means that one supplier of product has name "Gizmo, Inc.". This is a nice shorthand but we run into problems if we start supporting repetitive steps, optional steps, and the like.</li> <p>In conclusion, update, full text, and basic counting and grouping would seem straightforward at this point. Nesting queries, value subqueries, views, and the like should not be too hard if an agreement is reached on scope rules. Inference and federation will probably need more experimentation but a lot can be had already with very simple fine grained control of backward chaining, if such applies, or with explicit end-point references and explicit join order. These are practical but not pretty enough for committee consensus, would be my guess. Anyway, it will be a few months before anything formal will happen.</p>
Virtuso Data Space Bot
kidehen@openlinksw.com
2008-04-30T13:48:12.000-04:00
SPARQL at WWW 2008
http://www.openlinksw.com/weblog/oerling/?id=1353
2008-04-30T15:59:15Z
<div> <div> Andy Seaborne and Eric Prud'hommeaux, editors of the <a href="http://dbpedia.org/resource/SPARQL" id="link-id10830dd8">SPARQL</a> recommendation, convened a <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xa3028e8">SPARQL</a> birds of a feather session at <a href="http://www2008.org/" id="link-id16b82f18">WWW 2008</a>. The administrative outcome was that implementors could now experiement with extensions, hopefully keeping each other current about their efforts and that towards the end of 2008, a new W3C working group might begin formalizing the experiences into a new SPARQL spec. </div> <div> The session drew a good crowd, including many users and developers. The wishes were largely as expected, with a few new ones added. Many of the wishes already had diverse implementations, however most often without interop. I will below give some comments on the main issues discussed. </div> <div> - SPARQL Update - This is likely the most universally agreed upon extension. Implementations exist, largely along the lines of Andy Seaborne's SPARUL spec, which is also likely material for a W3C member submission. The issue is without much controverse, transactions fall outside the scope, which is reasonable enough. With triple stores, we can define things as combinations of inserts and deletes and isolation we just leave aside. If anything, operating on a transactional platform such as <a href="http://virtuoso.openlinksw.com" id="link-id103979f0">Virtuoso</a>, one wishes to disable transactions for any operations such as bulk loads and long running inserts and deletes. Transactionality has pretty much no overhead for a few hundred rows but for a few hundred million rows the cost of locking and rollback is prohibitive. With <a href="http://virtuoso.openlinksw.com" id="link-id0x15b9ff70">Virtuoso</a>, we have a row autocommit mode which we recommend for use with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id109a9d50">RDF</a>: It commits by itself now and then, optionally keeping a roll forward log and is transactional enough not to leave half triples around,i.e. inserted in one index but not another. </div> <div> As far as we are concerned, updating physical triples along the SPARUL lines is pretty much a done deal. </div> <div> The matter of updating relational <a href="http://dbpedia.org/resource/Data" id="link-id106f05e0">data</a> mapped to <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xa00f0a70">RDF</a> is a whole other kettle of fish. On this, I should say that RDF has no special virtues for expressing transactions but rather has a special genius for integration. Updating is best left to web service interfaces that use <a href="http://dbpedia.org/resource/SQL" id="link-id10aab0a8">SQL</a> on the inside. Anyway, updating union views, which most mappings will be, is complicated. Besides, for transactions, one usually knows exactly what one wishes to update. </div> <div> Full Text - Many people expressed a desire for full text access. Here we run into a deplorable confusion with regexps. The closest SPARQL has to full text in its native form is regexps, but these are not really mappable to full text except in rare special cases and I would despair of explaining to an end user what exactly these cases are. So, in principle, some regexps are equivalent to full text but in practice I find it much preferrable to keep these entirely separate. </div> <div> It was noted that what the users want is a text box for search words. This is a front end to the CONTAINS predicate of most <a href="http://dbpedia.org/resource/SQL" id="link-id0x14d45478">SQL</a> implementations. Ours is MS SQL Server compatible and has a SPARQL version called bif:contains. One must still declare which triples one wants indexed for full text, though. This admin overhead seems inevitable, as text indexing is a large overhead and not needed by all applications. </div> <div> Also, text hits are not boolean, usually they come with a hit score. Thus, aa SPARQL extension for this could look like select * where { ?thing has_description ?d . ?d ftcontains "gizmo" ftand "widget" score ?score . } </div> <div> This would return all the subjects, descriptions and scores from subjects with a has_description property containing widget and gizmo. Extending the basic pattern is better than having the match in a filter, since the match binds a variable. </div> <div> The <a href="http://dbpedia.org/resource/XQuery" id="link-id106517a0">XQuery</a>/<a href="http://dbpedia.org/resource/XPath" id="link-id10d04ae0">XPATH</a> groups have recently come up with a full text spec, so I used their style of syntax above. We already have a full text extension, as do some others. but for standardization, it is probably most appropriate to take the <a href="http://dbpedia.org/resource/XQuery" id="link-id0xa27b3a98">XQuery</a> work as a basis. The XQuery full text spec is quite complex but I would expect most uses to get by with a small subset and the structure seems better thought out, at first glance, than the more ad hoc implementations in diverse SQL's. </div> <div> Again, declaring any text index to support the search, as well as its timeliness or transactionality, are best left to implementations. </div> <div> Federation - This is a tricky matter. ARQ has a SPARQL extension for sending a nested set of triple patterns to a specific end point. The DARQ project has something more, including a selectivity model for SPARQL. </div> <div> With federated SQL, life is simpler since after the views are expanded, we have a query where each table is at a known server and has more or less known statistics. Generally, execution plans where as much work as possible is pushed to the remote servers are preferred and modeling the latencies is not overly hard. With SPARQL, each triple pattern could in principle come from any of the federated servers. Associating a specific end point to a fragment of the query just passes the problem to the user. It is my guess that this is the best we can do without getting very elaborate, and possibly buggy, end point content descriptions for routing federated queries. </div> <div> Having said this, there remains the problem of join order. I suggested that we enhance the protocol by allowing asking an end point for the query cost for a given SPARQL query. Since they all must have a cost model for optimization, this should not be an imposssible request. A time cost and estimated cardinality would be enough. Making statistics available a la DARQ was also discussed. Being able to declare cardinalities expected of a remote end point is probably necessary anyway, since not all will implement the cost model interface. For standardization, agreeing of what is a proper description of content and cardinality and how fine grained this must be will be so difficult that I would not wait for it. A cost model interface would nicely hide this within the end point itself. </div> <div> With Virtuoso, we do not have a federated SPARQL scheme but we could have the ARQ-like service construct. We'd use our own cost model with explicit declarations of cardinalities of the remote <a href="http://dbpedia.org/resource/Data" id="link-id0xa496840">data</a> for guessing a join order. Still, this is a bit of work. We'll see. </div> <div> For practicality, the service construct coupled with join order hints is the best short term bet. Making this pretty enough for standardization is not self-evident, as it requires end point description and/or cost model hooks for things to stay declarative. </div> <div> - End point description - This question has been around for a while, I have blogged about it earlier but we are not really at a point where there would be even rough consensus about an end point ontology. We should probably do something on our own to demonstrate some application of this, as we host lots of <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id10cdd138">linked open data</a> sets. </div> <div> - SQL equivalence - There were many requests for aggregation, some for subqueries and nesting, expressions in select, negation, existence and so on. I would call these all SQL equivalence. One use case was taking all the teams in the database and for all with over 5 members, add the big_team class and a property for member count. </div> <div> With Virtuoso, we could write this as </div> <pre> construct { ?team a big_team . ?team member_count ?ct } from ... where {?team a team . { select ?team2 count (*) as ?ct where { ?m member_of ?team2 } . filter (?team = ?team2 and ? ct > 5) }} </pre> <div> We have pretty much all the SQL equivalence features, as we have been working for some time at translating the <a href="http://dbpedia.org/resource/TPC-H" id="link-id13a7ad70">TPC H</a> workload into SPARQL. </div> <div> The usefulness of these things is uncontested but standardization could be hard as there are subtle questions about variable scope and the like. </div> <div> - Inference - The SPARQLL spec does not deal with transitivity or such matters because it is assumed that these are handled by an underlying inference layer. This is however most often not so. There was interest in more fine grained control of inference, for example declaring that just one property in a query would be transitive or that subclasses should be taken into account in only one triple pattern. As far as I am concerned, this is very reasonable and we even offer extensions for this sort of thing in Virtuoso's SPARQL. This however only makes sense if the inference is done at query time and pattern by pattern. For instance, if forward chaining is used, this no longer makes sense. Specifying that some forward chaining ought to be done at query time is impractical, as the operation can be very large and time consuming and it is the dba's task to determine what should be stored and for how long, how changes should be propagated and so on. All these are application dependent and standardizing will be difficult. </div> <div> Support for RDF features like lists and bags would all fall under the functions an underlying inference layer should perform. These thiings are of special interest when querying OWL models, for example. </div> <div> Path expressions - Path expressions were requested by a few people. We have implemented some, as in ?product+?has_supplier+>s_name = "Gizmos, Inc.". This means that one supplier of product has name "Gizmo, Inc.". This is a nice shorthand but we run into problems if we start supporting repetitive steps, optional steps and the like. </div> <div> In conclusion, update, full text and basic counting and grouping would seem straightforward at this point. Nesting queries, value subqueries, views and the like should not be too hard if an agreement is reached on scope rules. Inference and federation will probably need more experimentation but a lot can be had already with very simple fine grained control of backward chaining, if such applies or with explicit end point refernces and explicit join order. These are practical butr not pretty enough for committee consensus, would be my guess. Anyway, it will be a few months before anything formal will happen. </div> </div>
Orri Erling
oerling@openlinksw.com
2008-04-30T12:28:09.12000-04:00
Clearing Up RDF misrepresentation once again!
http://www.openlinksw.com/blog/~kidehen/?id=1352
2008-04-30T15:51:17Z
<p> <a href="http://myopenlink.net/dataspace/person/danieljohnlewis#this" id="link-id12d57690">Daniel Lewis</a> has penned a post titled: <a href="http://vanirsystems.com/danielsblog/2008/04/30/clearing-up-some-misconceptions-again/" id="link-id10c99f18">Clearing up some misconceptions..again</a>, in response to <a href="http://elgg.org/bwerdmuller/foaf#elgg2" id="link-id14fe1bc8">Ben Werdmuller</a>'s post titled: <a href="http://blogs.zdnet.com/social/?p=477" id="link-id141cee58">Introducing the Open Data Definition</a>. </p> <p>The great thing about the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id105991a8">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10a6ec78">Web</a> is that it's much easier to discovery and respond to these points of view before the ink dries :-) Ben certainly needs to take a look at the <a href="http://www.w3.org/RDF/FAQ" id="link-id10f78958">Semantic Web FAQ</a> pre or post assimilation of Daniel's response.</p>
Kingsley Uyi Idehen
kidehen@openlinksw.com
2008-04-30T12:07:58.1000-04:00
Linked Data enters state of Evoluation
http://www.openlinksw.com/blog/~kidehen/?id=1351
2008-04-29T19:56:14Z
<p>During a brief chat with <a href="http://community.linkeddata.org/dataspace/person/mhausenblas#this" id="link-idfeb0100">Michael Hausenblas</a> about a new <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1049feb0">Linked Data</a> project he is championing called: <a href="http://esw.w3.org/topic/SweoIG/TaskForces/CommunityProjects/LinkingOpenData/LForum" id="link-id16a857d8">LForum</a>, I made a freudian slip, in the form of the typo: <strong>Evoluation</strong>, which at the time was supposed to have been: <strong>Evolution</strong>. Anyway, we had a chuckle and realized we were on to something, so I proceeded to formalize the definition: </p> <blockquote> <cite>Evoluation is evolution devoid of the randomness of mutation. A state of being in which it is possible to evaluate and choose evolutionary paths.</cite> </blockquote> <p> <strong>Evoluation</strong> actually describes where we are today in relation to the <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id105c1518">World Wide Web</a>; to the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id103f9d00">Linking Open Data community</a> (<a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id1048c210">LOD</a>), it's taking the path towards becoming a <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id104c3a20">Giant Global Graph</a> of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id104968e0">Linked Data</a>; to the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> 2.0 community, it's simply a collection of Web Services and associated APIs; and to many others, it remains an opaque collection of interlinked documents.</p> <p>The great thing about the Web is that it allows netizens to explore a plethora of paths without adversely affecting the paths of others. That said, controlling one's path may take mutation out of evolution, but we are still left with the requirement to adapt and eventually survive in a competitive environment. Thus, although we can evaluate and choose from the many paths the Web's evolution offers us, the path that delivers the most benefits ultimately dominates. :-) </p>
Kingsley Uyi Idehen
kidehen@openlinksw.com
2008-04-29T16:25:47.000-04:00
Linked Data and Information Architecture
http://www.openlinksw.com/blog/vdb/blog/?id=1350
2008-04-29T14:37:22Z
<div> <div style="display:none;">Linked Data and Information Architecture</div> <p>We had a workshop on <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1437ac70">Linked Open Data</a> (<a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1315f788">LOD</a>) last week in <a href="http://www2008.org/" id="link-id0x13737468">Beijing</a>. You can see the papers in <a href="http://events.linkeddata.org/ldow2008/#program" id="link-id10651ab8">the program</a>. The event was a success with plenty of good talks and animated conversation. I will not go into every paper here but will comment a little on the conversation and draw some technology requirements going forward.</p> <p>Tim Berners-Lee showed a read-write version of <a href="http://dig.csail.mit.edu/2005/ajar/release/tabulator/0.8/tab.html" id="link-id0x15633520">Tabulator</a>. This raises the question of updating on the <a href="http://dbpedia.org/resource/Data" id="link-id0x1350a178">Data</a> Web. The consensus was that one could assert what one wanted in one's own space but that others' spaces would be read-only. What spaces one considered relevant would be the user's or developer's business, as in the document web.</p> <p>It seems to me that a significant use case of LOD is an open-web situation where the user picks a broad read-only "data wallpaper" or backdrop of assertions, and then uses this combined with a much smaller, local, writable data set. This is certainly the case when editing data for publishing, as in Tim's demo. This will also be the case when developing mesh-ups combining multiple distinct data sets bound together by sets of SameAs assertions, for example. Questions like, "What is the minimum subset of n data sets needed for deriving the result?" will be common. This will also be the case in applications using proprietary data combined with open data.</p> <p>This means that databases will have to deal with queries that specify large lists of included graphs, all graphs in the store or all graphs with an exclusion list. All this is quite possible but again should be considered when architecting systems for an open <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0xa27bae8">linked data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x155c3f18">web</a>.</p> <p>"There is data but what can we really do with it? How far can we trust it, and what can we confidently decide based on it?"</p> <p>As an answer to this question, <a href="http://zitgist.com/about/" id="link-id0xd447580">Zitgist</a> has compiled the <a href="http://umbel.org/about/" id="link-id0x14735008">UMBEL</a> taxonomy using <a href="http://dbpedia.org/resource/SKOS" id="link-id0x15ab1c48">SKOS</a>. This draws on Wikipedia, Open CYC, Wordnet, and <a href="http://www.mpi-inf.mpg.de/~suchanek/downloads/yago/" id="link-id0x15d5aa88">YAGO</a>, hence the acronym WOWY. UMBEL is both a taxononmy and a set of instance data, containing a large set of <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id0x9fe45d98">named entities</a>, including persons, organizations, geopolitical entities, and so forth. By extracting references to this set of named entities from documents and correlating this to the taxonomy, one gets a good idea of what a document (or part thereof) is about.</p> <p>Kingsley presented this in the Zitgist demo. This is our answer to the criticism about <a href="http://dbpedia.org/resource/DBpedia" id="link-id0xa1920800">DBpedia</a> having errors in classification. DBpedia, as a bootstrap stage, is about giving names to all things. Subsequent efforts like UMBEL are about refining the relationships.</p> <p>"Should there be a global <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x12cd5290">URI</a> dictionary?"</p> <p>There was a talk by Paolo Bouquet about <a href="http://dbpedia.org/resource/Entity" id="link-id0x12d03400">Entity</a> Name System, a a sort of data DNS, with the purpose of associating some description and rough classification to URIs. This would allow discovering URIs for reuse. I'd say that this is good if it can cut down on the SameAs proliferation and if this can be widely distributed and replicated for resilience, <i>à la</i> DNS. On the other hand, it was pointed out that this was not quite in the LOD spirit, where parties would mint their own dereferenceable URIs, in their own domains. We'll see.</p> <p>"What to do when identity expires?"</p> <p>Giovanni of Sindice said that a document should be removed from search if it was no longer available. Kingsley pointed out that resilience of reference requires some way to recover data. The data web cannot be less resilient than the document web, and there is a point to having access to history. He recommended hooking up with the <a href="http://dbpedia.org/resource/Internet" id="link-id0x143e4130">Internet</a> Archive, since they make long term persistence their business. In this way, if an application depends on data, and the URIs on which it depends are no longer dereferenceable or or provide content from a new owner of the domain, those who need the old version can still get it and host it themselves.</p> <p>It is increasingly clear that OWL SameAs is both the blessing and bane of linked data. We can easily have tens of URIs for the same thing, especially with people. Still, these should be considered the same.</p> <p>Returning every synonym in a query answer hardly makes sense but accepting them as input seems almost necessary. This is what we do with <a href="http://virtuoso.openlinksw.com" id="link-id0x15a2a930">Virtuoso</a>'s SameAs support. Even so, this can easily double query times even when there are no synonyms.</p> <p>Be that as it may, SameAs is here to stay; just consider the mapping of DBpedia to Geonames, for example.</p> <p>Also, making aberrant SameAs statements can completely poison a data set and lead to absurd query results. Hence choosing which SameAs assertions from which source will be considered seems necessary. In an open web scenario, this leads inevitably to multi-graph queries that can be complex to write with regular <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x12bb8ce8">SPARQL</a>. By extension, it seems that a good query would also include the graphs actually used for deriving each result row. This is of course possible but has some implications on how databases should be organized.</p> <p>Yves Raymond gave a talk about deriving identity between Musicbrainz and Jamendo. I see the issue as a core question of linked data in general. The algorithm Yves presented started with attribute value similarities and then followed related entities. Artists would be the same if they had similar names and similar names of albums with similar song titles, for example. We can find the same basic question in any analysis, for example, looking at how news reporting differs between media, supposing there is adequate entity extraction.</p> <p>There is basic graph diffing in <a href="http://data.semanticweb.org/conference/iswc-aswc/2007/tracks/research/papers/533/html" id="link-id0x153c1fa8">RDFSync</a>, for example. But here we are expanding the context significantly. We will traverse references to some depth, allow similarity matches, SameAs, and so forth. Having presumed identity of two URIs, we can then look at the difference in their environment to produce a human readable summary. This could then be evaluated for purposes of analysis or of combining content.</p> <p>At first sight, these algorithms seem well parallelizable, as long as all threads have access to all data. For scaling, this means a probably message-bound distributed algorithm. This is something to look into for the next stage of linked data.</p> <p>Some inference is needed, but if everybody has their own choice of data sets to query, then everybody would also have their own entailed triples. This will make for an explosion of entailed graphs if forward chaining is used. Forward chaining is very nice because it keeps queries simple and easy to optimize. With Virtuoso, we still favor backward chaining since we expect a great diversity of graph combinations and near infinite volume in the open web scenario. With private repositories of slowly changing data put together for a special application, the situation is different.</p> <p>In conclusion, we have a real LOD movement with actual momentum and a good idea of what to do next. The next step is promoting this to the broader community, starting with <a href="http://www.linkeddataplanet.com/" id="link-id0x155d1d00">Linked Data Planet</a> in New York in June.</p> </div>
Virtuso Data Space Bot
kidehen@openlinksw.com
2008-04-29T17:18:21.48000-04:00
On Sem Web Search
http://www.openlinksw.com/blog/vdb/blog/?id=1349
2008-04-29T14:37:21Z
<div> <div style="display:none;">On Sem Web Search</div> <p>"I give the search keywords and you give me a <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xdc0c248">SPARQL</a> end-point and a query that will get the <a href="http://dbpedia.org/resource/Data" id="link-id0xbd54be0">data</a>."</p> <p>Thus did one SPARQL user describe the task of a semantic/data web search engine.</p> <p>In <a href="http://www.openlinksw.com/weblog/oerling/?id=1336" id="link-idff98750">a previous post</a>, I suggested that if the data web were the size of the document web, we'd be looking at two orders of magnitude more search complexity. It just might be so.</p> <p>In the conversation, I pointed out that a search engine might have a copy of everything and even a capability to do SPARQL and full text on it all, yet still the users would be better off doing the queries against the SPARQL end-points of the data publishers. It is a bit like the fact that not all web browsing runs off Google's cache. With the data web, the point is even more pronounced, as serving a hit from Google's cache is a small operation but a complex query might be a very large one.</p> <p>Yet, the data web is about ad-hoc joining between data sets of different origins. Thus a search engine of the data web ought to be capable of joining also, even if large queries ought to be run against individual publishers' end-points or the user's own data warehouse.</p> <p>For ranking, the general consensus was that no single hit-ranking would be good for the data web. Thus word frequency-based hit-scores are OK for text hits but more is not obvious. I would think that some link analysis could apply but this will take some more experimentation.</p> <p>For search summaries, if we have splitting of data sets into small fragments <i>à la</i> Sindice, search summaries are pretty much the same as with just text search. If we store triples, then we can give text style summaries of text hits in literals and Fresnel lens views of the structured data around the literal. For showing a page of hits, the lenses must abbreviate heavily but this is still feasible. The engine would know about the most common ontologies and summarize instance data accordingly.</p> <p>Chris Bizer pointed out that trust and provenance are critical, especially if an answer is arrived at by joining multiple data sets. The trust of the conclusion is no greater than that of the weakest participating document. Different users will have different trusted sources.</p> <p>A mature data web search engine would combine a provenance/trust specification, a search condition consisting of SPARQL or full text or both, and a specification for hit rank. Again, most searches would use defaults, but these three components should in principle be orthogonally specifiable.</p> <p>Many places may host the same data set either for download or SPARQL access. The <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0xac4d6c8">URI</a> of the data set is not its <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id0xdd50478">URL</a>. Different places may further host multiple data sets on one end-point. Thus the search engine ought to return all end-points where the set is to be found. The end-points themselves ought to be able to say what data sets they contain, under what graph IRIs. Since there is no consensus about end-point self description, this too would be left to the search engine. In practice, this could be accomplished by extending Sindice's semantic site map specification. A possible query would be to find an end-point containing a set of named data sets. If none were found, the search engine itself could run a query joining all the sets since it at least would hold them all.</p> <p>Since many places will host sets like Wordnet or Uniprot, indexing these once for each copy hardly makes sense. Thus a site should identify its data by the data set's URI and not the copy's URL.</p> <p>It came up in the discussion that search engines should share a ping format so that a single message format would be enough to notify any engine about data being updated. This is already partly the case with Sindice and <a href="http://www.pingthesemanticweb.com/" id="link-id0xb0c8078">PTSW</a> (Ping The <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0xc28f9f0">Semantic Web</a>) sharing a ping format. </p> <p>Further, since it is no trouble to publish a copy of the 45G Uniprot file but a fair amount of work to index it, search engines should be smart about processing requests to index things, since these can amount to a denial of service attack. </p> <p>Probably very large data sets should be indexed only in the form supplied by their publisher, and others hosting copies would just state that they hold a copy. If the claim to the copy proved false, users could complain and the search engine administrator would remove the listing. It seems that some manual curating cannot be avoided here. </p> <h2>On Data Web Search Business Model</h2> <p>It seems there can be an overlap between the data web search and the data web hosting businesses. For example, Talis rents space for hosting <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xbececa0">RDF</a> data with SPARQL access. A search engine should offer basic indexing of everything for free, but could charge either data publishers or end users for running SPARQL queries across data sets. These do not have the nicely anticipatable and fairly uniform resource consumption of text lookups. In this manner, a search provider could cost-justify the capacity for allowing arbitrary queries. </p> <p>The value of the data web consists of unexpected joining. Such joining takes place most efficiently if the sources are at least in some proximity, for example in the same data center. Thus the search provider could monetize functioning as the database provider for mesh-ups. In the document web, publishing pages is very simple and there is no great benefit from co-locating search and pages, rather the opposite. For the data web, the hosting with SPARQL and all is more complex and resembles providing search. Thus providing search can combine with providing SPARQL hosting, once we accept in principle that search should have arbitrary inter-document joining, even if it is at an extra premium.</p> <p>The present search business model is advertising. If the data web is to be accessed by automated agents such as mesh-up code, display of ads is not self-evident. This is quite separate from the fact that semantics can lead to better ad targeting.</p> <p>One model would be to do text lookups for free from a regular web page but show ads, just a la Google search ads. Using the service via web services for text or SPARQL would have a cost paid by the searching or publishing party and would not be financed by advertising.</p> <p>In the case of data used in value-add data products (mesh-ups) that have financial value to their users, the original publisher of the data could even be paid for keeping the data up-to-date. This would hold for any time-sensitive feeds like news or financial feeds. Thus the hosting/search provider would be a broker of data-use fees and the data producer would be in the position of an AdSense inventory owner, i.e., a web site which shows AdSense ads. Organizing this under a hub providing back-office functions similar to an ad network could make sense even if the actual processing were divided among many sites.</p> <p>Kingsley has repeatedly formulated the core value proposition of the semantic web in terms of dealing with <a href="http://dbpedia.org/resource/Information" id="link-id0xa951148">information</a> overload: There is the real-time enterprise and the real-time individual and both are beasts of perception. Their image is won and lost in the Internet online conversation space. We know that allegations, even if later proven false, will stick if left unchallenged. The function of semantics on the web is to allow one to track and manage where one stands. In fact, Garlic has made a business of just this, but now from a privacy and security angle. Garlic's Data Patrol harvests data from diverse sources and allows assessing vulnerability to identity theft, for example.</p> <p>If one is in the business of collating all the structured data in the world, as a data web search engine is, then providing custom alerts for both security or public image management is quite natural. This can be a very valuable service if it works well.</p> <p>At OpenLink, we will now experiment with the Sindice/<a href="http://zitgist.com/about/" id="link-id0x14606ec8">Zitgist</a>/<a href="http://www.pingthesemanticweb.com/" id="link-id0xa08174e0">PingTheSemanticWeb</a> content. This is a regular part of the productization of <a href="http://virtuoso.openlinksw.com" id="link-id0xa127710">Virtuoso</a>'s cluster edition. We expect to release some results in the next 4 weeks. </p> </div>
Virtuso Data Space Bot
kidehen@openlinksw.com
2008-04-29T16:06:09.000-04:00