http://www.openlinksw.com/weblog/dav/dav-blog-1/
OpenLink Community Blog
A Collection of blogs by OpenLink Staff
kidehen@openlinksw.com
kidehen@openlinksw.com
2009-11-23T14:05:11Z
Virtuoso Universal Server 05.12.3041
http://www.openlinksw.com/weblog/public/images/vbloglogo.gif
5 Very Important Things to Note about HTTP based Linked Data
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2009-11-19#1591
2009-11-19T19:49:09Z
2009-11-23T07:04:19.000003-05:00
<ol> <li> It isn't <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id115dfd68">World Wide Web</a> Specific (HTTP != World Wide Web)</li> <li> It isn't Open <a href="http://dbpedia.org/resource/Data">Data</a> Specific </li> <li> It isn't about "Free" (Beer or Speech) </li> <li> It isn't about Markup (so don't expect to grok it via "markup first" approach) </li> <li>It's about <a href="http://dbpedia.org/resource/Linked_Data" id="link-id13a6aa98">Hyperdata</a> - the use of HTTP and REST to deliver a powerful platform agnostic mechanism for Data Reference, Access, and Integration.</li> </ol> <p> When trying to understand HTTP based <a href="http://dbpedia.org/resource/Linked_Data" id="link-id18aa1490">Linked Data</a>, especially if you're well versed in DBMS technology use (User, Power User, Architect, Analyst, DBA, or Programmer) think:</p> <ul> <li> Open Database Connectivity (<a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id1428fba0">ODBC</a>) without operating system, data model, or wire-protocol specificity or lock-in potential </li> <li> Java Database Connectivity (<a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id18d3c2a8">JDBC</a>) without programming language specificity </li> <li> <a href="http://dbpedia.org/resource/ADO.NET" id="link-id125725b8">ADO</a>.NET without .NET runtime specificity and .NET bound language specificity </li> <li> OLE-DB without Windows operating system & programming language specificity </li> <li> XMLA without XML format specificity - with Tabular and Multidimensional results formats expressible in a variety of data representation formats. </li> <li>All of the above scoped to the Record rather than Container level, with Generic HTTP scheme URIs associated with each Record, Field, and Field value (optionally) </li> </ul> <p>Remember the need for Data Access & Integration technology is the by product of the following realities:</p> <ol> <li> Human curated data is ultimately dirty, because: <ul> <li>our thick thumbs, inattention, distractions, and general discomfort with typing, make typos prevalent</li> <li>database engines exist for a variety of data models - Graph, Relational, Hierarchical;</li> <li>within databases you have different record container/partition names e.g. Table Names;</li> <li>within a database record container you have records that are really aspects of the same thing (different keys exist in a plethora of operational / line of business systems that expose aspects of the same <a href="http://dbpedia.org/resource/Entity" id="link-id13378338">entity</a> e.g., customer data that spans Accounts, CRM, ERP application databases);</li> <li>different field names (one database has "EMP" while another has "Employee") for the same record</li>.</ul> </li> <li>Units of measurement is driven by locale, the UK office wants to see sales in Pounds Sterling while the French office prefers Euros etc.</li> <li>All of the above is subject to <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id17e46398">context</a> halos which can be quite granular re. sensitivity e.g. staff travel between locations that alter locales and their roles; basically, profiles matters a lot.</li> </ol> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1364" id="link-id128f0fe8">ODBC and WODBC (Web Open Database Connectivity) Comparison</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1364" id="link-id1367cd18">Creating, Deploying, and Exploiting Linked Data Presentation</a> </li> <li> <a href="http://www.odata.org/" id="link-id122ab708">Open Data Protocol Project</a> </li> </ul>
Short Recap of Virtuoso Basics (#3 of 5)
http://www.openlinksw.com/blog/vdb/blog/?date=2009-04-30#1552
2009-04-30T15:49:53Z
2009-04-30T12:11:45-04:00
<p>(Third of five posts related to the <a href="http://www2009.org/" id="link-id0x14b582b8">WWW 2009</a> conference, held the week of April 20, 2009.) </p> <p>There are some points that came up in conversation at WWW 2009 that I will reiterate here. We find there is still some lack of clarity in the product image, so I will here condense it.</p> <p> <a href="http://virtuoso.openlinksw.com" id="link-id0x14bf48b8">Virtuoso</a> is a DBMS. We pitch it primarily to the <a href="http://dbpedia.org/resource/Data" id="link-id0x16bc4490">data</a> web space because this is where we see the emerging frontier. Virtuoso does both <a href="http://dbpedia.org/resource/SQL" id="link-id0x1223dc30">SQL</a> and <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x170eec88">SPARQL</a> and can do both at large scale and high performance. The popular perception of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x15a05fc0">RDF</a> and Relational models as mutually exclusive and antagonistic poles is based on the poor scalability of early RDF implementations. What we do is to have all the RDF specifics, like IRIs and typed literals as native SQL types, and to have a cost based optimizer that knows about this all.</p> <p>If you want application-specific data structures as opposed to a schema-agnostic quad-store model (triple + graph-name), then Virtuoso can give you this too. <a href="http://docs.openlinksw.com/virtuoso/rdfsparqlintegrationmiddleware.html#rdfviews" id="link-id14ddc7c8">Rendering application specific data structures as RDF</a> applies equally to relational data in non-Virtuoso databases because Virtuoso SQL can <a href="http://docs.openlinksw.com/virtuoso/qsvdbsrv.html" id="link-id14aaea70">federate tables from heterogenous DBMS</a>.</p> <p>On top of this, there is a <a href="http://docs.openlinksw.com/virtuoso/qswebserver.html" id="link-id16fcde60">web server built in</a>, so that no extra server is needed for web services, web pages, and the like.</p> <p>Installation is simple, just one exe and one config file. There is a huge amount of code in <a href="http://docs.openlinksw.com/virtuoso/installation.html" id="link-id16767b40">installers</a> â application code and test suites and such â but none of this is needed when you deploy. Scale goes from a 25MB memory footprint on the desktop to hundreds of gigabytes of RAM and endless terabytes of disk on shared-nothing clusters.</p> <p>Clusters (coming in Release 6) and SQL federation are <a href="http://download.openlinksw.com/download/product_matrix.vsp?p=l_os&c=39&df=16" id="link-id16722550">commercial only</a>; the rest can be had <a href="http://sourceforge.net/project/showfiles.php?group_id=161622" id="link-id131080a8">under GPL</a>.</p> <p>To condense further:</p> <ul> <li>Scalable Delivery of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1060ad98">Linked Data</a> </li> <li>SPARQL and SQL <ul> <li>Arbitrary RDF Data + Relational</li> <li>Also From 3rd Party <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x16bbce60">RDBMS</a> </li> </ul> </li> <li>Easy Deployment </li> <li>Standard Interfaces <ul> <li> <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x12e284d8">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0xb5e1400">JDBC</a>, OLE DB, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id0x15a55db8">ADO</a>.<a href="http://dbpedia.org/resource/.NET_Framework" id="link-id0x16beb070">NET</a>, XMLA</li> <li> <a href="http://jena.sourceforge.net/" id="link-id0x122b5008">Jena</a>, <a href="http://sourceforge.net/projects/sesame/" id="link-id0x148d4078">Sesame</a>, etc.</li> <li>All Web Protocols </li> </ul> </li> </ul>
Short Recap of Virtuoso Basics (#3 of 5)
http://www.openlinksw.com/weblog/oerling/?date=2009-04-30#1550
2009-04-30T15:49:53Z
2009-04-30T12:11:43.000001-04:00
<p>(Third of five posts related to the <a href="http://www2009.org/" id="link-id0x1081fe40">WWW 2009</a> conference, held the week of April 20, 2009.) </p> <p>There are some points that came up in conversation at WWW 2009 that I will reiterate here. We find there is still some lack of clarity in the product image, so I will here condense it.</p> <p> <a href="http://virtuoso.openlinksw.com" id="link-id0xd0e85f0">Virtuoso</a> is a DBMS. We pitch it primarily to the <a href="http://dbpedia.org/resource/Data" id="link-id0x14a294d8">data</a> web space because this is where we see the emerging frontier. Virtuoso does both <a href="http://dbpedia.org/resource/SQL" id="link-id0x108042f8">SQL</a> and <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x10889878">SPARQL</a> and can do both at large scale and high performance. The popular perception of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x107d3b40">RDF</a> and Relational models as mutually exclusive and antagonistic poles is based on the poor scalability of early RDF implementations. What we do is to have all the RDF specifics, like IRIs and typed literals as native SQL types, and to have a cost based optimizer that knows about this all.</p> <p>If you want application-specific data structures as opposed to a schema-agnostic quad-store model (triple + graph-name), then Virtuoso can give you this too. <a href="http://docs.openlinksw.com/virtuoso/rdfsparqlintegrationmiddleware.html#rdfviews" id="link-id14ddc7c8">Rendering application specific data structures as RDF</a> applies equally to relational data in non-Virtuoso databases because Virtuoso SQL can <a href="http://docs.openlinksw.com/virtuoso/qsvdbsrv.html" id="link-id14aaea70">federate tables from heterogenous DBMS</a>.</p> <p>On top of this, there is a <a href="http://docs.openlinksw.com/virtuoso/qswebserver.html" id="link-id16fcde60">web server built in</a>, so that no extra server is needed for web services, web pages, and the like.</p> <p>Installation is simple, just one exe and one config file. There is a huge amount of code in <a href="http://docs.openlinksw.com/virtuoso/installation.html" id="link-id16767b40">installers</a> â application code and test suites and such â but none of this is needed when you deploy. Scale goes from a 25MB memory footprint on the desktop to hundreds of gigabytes of RAM and endless terabytes of disk on shared-nothing clusters.</p> <p>Clusters (coming in Release 6) and SQL federation are <a href="http://download.openlinksw.com/download/product_matrix.vsp?p=l_os&c=39&df=16" id="link-id16722550">commercial only</a>; the rest can be had <a href="http://sourceforge.net/project/showfiles.php?group_id=161622" id="link-id131080a8">under GPL</a>.</p> <p>To condense further:</p> <ul> <li>Scalable Delivery of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x12211da8">Linked Data</a> </li> <li>SPARQL and SQL <ul> <li>Arbitrary RDF Data + Relational</li> <li>Also From 3rd Party <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x168db0e0">RDBMS</a> </li> </ul> </li> <li>Easy Deployment </li> <li>Standard Interfaces <ul> <li> <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x10473bf0">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x12187f58">JDBC</a>, OLE DB, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id0x10354e48">ADO</a>.<a href="http://dbpedia.org/resource/.NET_Framework" id="link-id0x16eeadd0">NET</a>, XMLA</li> <li> <a href="http://jena.sourceforge.net/" id="link-id0x12e3fe08">Jena</a>, <a href="http://sourceforge.net/projects/sesame/" id="link-id0x15e62470">Sesame</a>, etc.</li> <li>All Web Protocols </li> </ul> </li> </ul>
Time for RDBMS Primacy Downgrade is Nigh! (No Embedded Images Edition - Update 1)
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2009-01-27#1520
2009-01-27T19:19:44Z
2009-03-17T11:50:58-04:00
<p> As the world works it way through a "once in a generation" economic crisis, the long overdue downgrade of the <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id15750540">RDBMS</a>, from its pivotal position at the apex of the <a href="http://dbpedia.org/resource/Data" id="link-id0x24ea3650">data</a> access and data management pyramid is nigh.</p> <h3>What is the Data Access, and Data Management Value Pyramid?</h3> <p> As depicted below, a top-down view of the data access and data management value chain. The term: apex, simply indicates value primacy, which takes the form of a data access API based entry point into a DBMS realm -- aligned to an underlying data model. Examples of data access APIs include: Native Call Level Interfaces (CLIs), <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id11c254c0">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id149b16a8">JDBC</a>, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id11451eb0">ADO</a>.NET, <a href="http://dbpedia.org/resource/OLE_DB" id="link-id15b02478">OLE-DB</a>, <a href="http://dbpedia.org/resource/XML_for_Analysis" id="link-id1181fa10">XMLA</a>, and <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id0x1f8394a8">Web</a> Services.</p> See: <a href="http://virtuoso.openlinksw.com/images/Agility_Value_Factors_Pyramid.png" id="link-id146cadd8"> AVF Pyramid Diagram.</a> <p> The degree to which ad-hoc views of data managed by a DBMS can be produced and dispatched to relevant data consumers (e.g. people), without compromising concurrency, data durability, and security, collectively determine the "Agility Value Factor" (AVF) of a given DBMS. Remember, agility as the cornerstone of environmental adaptation is as old as the concept of evolution, and intrinsic to all pursuits of primacy. </p> <p>In simpler business oriented terms, look at AVF as the degree to which DBMS technology affects the ability to effectively implement "Market Leadership Discipline" along the following pathways: innovation, operation excellence, or customer intimacy. </p> <h3>Why has RDBMS Primacy has Endured?</h3> <p> Historically, at least since the late '80s, the RDBMS genre of DBMS has consistently offered the highest AVF relative to other DBMS genres en route to primacy within the value pyramid. The desire to improve on paper reports and spreadsheets is basically what DBMS technology has fundamentally addressed to date, even though conceptual level interaction with data has never been its forte.</p> See: <a href="http://virtuoso.openlinksw.com/images/Old_RDBMS_Primacy_Pyramid.png" id="link-id134dab90"> RDBMS Primacy Diagram.</a> <p> For more then 10 years -- at the very least -- limitations of the traditional RDBMS in the realm of conceptual level interaction with data across diverse data sources and schemas (enterprise, Web, and <a href="http://dbpedia.org/resource/Internet" id="link-id116001c0">Internet</a>) has been crystal clear to many RDBMS technology practitioners, as indicated by some of the quotes excerpted below:</p> <blockquote> <cite> <p> "Future of Database Research is excellent, but what is the future of data?" </p> "..it is hard for <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id14932398">me</a> to disagree with the conclusions in this report. It captures exactly the right thoughts, and should be a must read for everyone involved in the area of databases and database research in particular." <p>-- <a href="http://jhingran.typepad.com/anant_jhingrans_musings/" id="link-id11334c50">Dr. Anant Jingran</a>, CTO, IBM <a href="http://dbpedia.org/resource/Information" id="link-id150c7970">Information</a> Management Systems, commenting on the <a href="http://db.cs.berkeley.edu/claremont/" id="link-id11c3b408">2007 RDBMS technology retreat</a> attended by a number of key DBMS technology pioneers and researchers.</p> </cite> </blockquote> <blockquote> <cite> <p> "<a href="http://www.databasecolumn.com/2007/09/one-size-fits-all.html" id="link-id15c14f08">One size fits all: A concept whose time has come and gone</a> </p> <p> </p> <ol> <li> They are direct descendants of System R and <a href="http://dbpedia.org/resource/Ingres" id="link-id146da780">Ingres</a> and were architected more than 25 years ago</li> <li> They are advocating "one size fits all"; i.e. a single engine that solves all DBMS needs. </li> </ol> <p>-- Prof. <a href="http://en.wikipedia.org/wiki/Michael_Stonebraker" id="link-id145c4e28">Michael Stonebreaker</a>, one of the founding fathers of the RDBMS industry.</p> </cite> </blockquote> <p>Until this point in time, the requisite confluence of "circumstantial pain" and "open standards" based technology required to enable an objective "compare and contrast" of RDBMS engine virtues and viable alternatives hasn't occurred. Thus, the RDBMS has endured it position of primacy albeit on a "one size fits all basis". </p> <h4>Circumstantial Pain</h4> <p> As mentioned earlier, we are in the midst of an economic crisis that is ultimately about a consistent inability to connect dots across a substrate of interlinked data sources that transcend traditional data access boundaries with high doses of schematic heterogeneity. Ironically, in a era of the dot-com, we haven't been able to make meaningful connections between relevant "real-world things" that extend beyond primitive data hosted database tables and content management style document containers; we've struggled to achieve this in the most basic sense, let alone evolve our ability to connect inline with the <a href="http://www.vldb2007.org/program/slides/s1161-brodie.pdf" id="link-id11a0dcf0">exponential rate at which the Internet & Web are spawning "universes of discourse" (data spaces) that emanate from user activity</a> (within the enterprise and across the Internet & Web). In a nutshell, we haven't been able to upgrade our interaction with data such that "conceptual models" and resulting "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id12da4b00">context</a> lenses" (or facets) become concrete; by this I mean: real-world <a href="http://dbpedia.org/resource/Entity" id="link-id146a48a8">entity</a> interaction making its way into the computer realm as opposed to the impedance we all suffer today when we transition from conceptual model interaction (real-world) to logical model interaction (when dealing with RDBMS based data access and data management). </p> <p>Here are some simple examples of what I can only best describe as: "critical dots unconnected", resulting from an inability to interact with data conceptually:</p> <strong>Government (Globally) -</strong> <p> Financial regulatory bodies couldn't effectively discern that a <a href="http://dbpedia.org/resource/Credit_default_swap" id="link-id115ba0e0">Credit Default Swap</a> is an Insurance policy in all but literal name. And in not doing so the cost of an unregulated <a href="http://dbpedia.org/resource/Insurance" id="link-id158d4960">insurance policy</a> laid the foundation for exacerbating the toxicity of fatally flawed mortgage backed securities. Put simply: a flawed insurance policy was the fallback on a toxic security that financiers found exotic based on superficial packaging.</p> <strong>Enterprises - </strong> <p> Banks still don't understand that capital really does exists in tangible and intangible forms; with the intangible being the variant that is inherently dynamic. For example, a tech companies intellectual capital far exceeds the value of fixture, fittings, and buildings, but you be amazed to find that in most cases this vital asset has not significant value when banks get down to the nitty gritty of debt collateral; instead, a buffer of flawed securitization has occurred atop a borderline static asset class covering the aforementioned buildings, fixtures, and fittings. </p> <p> In the general enterprise arena, IT executives continued to "rip and replace" existing technology without ever effectively addressing the timeless inability to connect data across disparate data silos generated by internal enterprise applications, let alone the broader need to mesh data from the inside with external data sources. No correlations made between the growth of buzzwords and the compounding nature of data integration challenges. It's 2009 and only a miniscule number of executives dare fantasize about being anywhere within distance of the: relevant information at your fingertips vision. </p> <p> Looking more holistically at data interaction in general, whether you interact with data in the enterprise space (i.e., at work) or on the Internet or Web, you ultimately are delving into a mishmash of disparate computer systems, applications, service (Web or SOA), and databases (of the RDBMS variety in a majority of cases) associated with a plethora of disparate schemas. Yes, but even today "rip and replace" is still the norm pushed by most vendors; pitting one mono culture against another as exemplified by irrelevances such as: FOSS/LAMP vs Commercial or Web vs. Enterprise, when none of this matters if the data access and integration issues are recognized let alone addressed (see: <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1497?sid=0df0294caee8b37925c6a888bbbca136&realm=wa" id="link-id15c27300">Applications are Like Fish and Data Like Wine</a>). </p> <p> Like the current credit-crunch, exponential growth of data originating from disparate application databases and associated schemas, within shrinking processing time frames, has triggered a rethinking of what defines data access and data management value today en route to an inevitable RDBMS downgrade within the value pyramid.</p> <h3>Technology</h3> <p>There have been many attempts to address real-world modeling requirements across the broader DBMS community from Object Databases to Object-Relational Databases, and more recently the emergence of simple <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id1128dad0">Entity</a>-Attribute-Value model DBMS engines. In all cases failure has come down to the existence of one or more of the following deficiencies, across each potential alternative:</p> <ol> <li>Query language standardization - nothing close to <a href="http://dbpedia.org/resource/SQL" id="link-id16002d60">SQL</a> standardization</li> <li>Data Access API standardization - nothing close to ODBC, JDBC, OLE-DB, or ADO.NET</li> <li>Wire protocol standardization - nothing close to HTTP</li> <li>Distributed Identity infrastructure - nothing close to the non-repudiatable digital Identity that <a href="http://dbpedia.org/resource/Friend_of_a_friend" id="link-id14926b18">foaf</a>+ssl accords</li> <li>Use of Identifiers as network based pointers to data sources - nothing close to RDF based <a href="http://dbpedia.org/resource/Linked_Data" id="link-id16180a28">Linked Data</a> </li> <li>Negotiable data representation - nothing close to Mime and HTTP based Content Negotiation</li> <li>Scalability especially in the era of Internet & Web scale.</li> </ol> <h4>Entity-Attribute-Value with Classes & Relationships (<a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id13e741b8">EAV</a>/CR) data models</h4> <p>A common characteristic shared by all post-relational DBMS management systems (from Object Relational to pure Object) is an orientation towards variations of EAV/CR based data models. Unfortunately, all efforts in the EAV/CR realm have typically suffered from at least one of the deficiencies listed above. In addition, the same "one DBMS model fits all" approach that lies at the heart of the RDBMS downgrade also exists in the EAV/CR realm.</p> <h3>What Comes Next?</h3> <p>The RDBMS is not going away (ever), but its era of primacy -- by virtue of its placement at the apex of the data access and data management value pyramid -- is over! I make this bold claim for the following reasons: </p> <ol> <li> The Internet aided "Global Village" has brought "<a href="http://en.wikipedia.org/wiki/Open_World_Assumption" id="link-id1148e560">Open World</a>" vs "<a href="http://en.wikipedia.org/wiki/Closed_World_Assumption" id="link-id11967cd0">Closed World</a>" assumption issues to the fore e.g., the current global economic crisis remains centered on the inability to connect dots across "Open World" and "Closed World" data frontiers </li> <li> Entity-Attribute-Value with Classes & Relationships (EAV/CR) based DBMS models are more effective when dealing with disparate data associated with disparate schemas, across disparate DBMS engines, host operating systems, and networks. </li> </ol> <p>Based on the above, it is crystal clear that a different kind of DBMS -- one with higher AVF relative to the RDBMS -- needs to sit atop today's data access and data management value pyramid. The characteristics of this DBMS must include the following:</p> <ol> <li> Every item of data (Datum/Entity/Object/Resource) has Identity</li> <li> Identity is achieved via Identifiers that aren't locked at the DBMS, OS, Network, or Application levels</li> <li> Object Identifiers and Object values are independent (extricably linked by association)</li> <li> Object values should be de-referencable via Object Identifier</li> <li> Representation of de-referenced value graph (entity, attributes, and values mesh) must be negotiable (i.e. content negotiation)</li> <li>Structured query language must provide mechanism for Creation, Deletion, Updates, and Querying of data objects</li> <li> Performance & Scalability across "Closed World" (enterprise) and "Open World" (Internet & Web) realms.</li> </ol> <p>Quick recap, I am not saying that RDBMS engine technology is dead or obsolete. I am simply stating that the era of RDBMS primacy within the data access and data management value pyramid is over. </p> <p>The problem domain (conceptual model views over heterogeneous data sources) at the apex of the aforementioned pyramid has simply evolved beyond the natural capabilities of the RDBMS which is rooted in "Closed World" assumptions re., data definition, access, and management. The need to maintain domain based conceptual interaction with data is now palpable at every echelon within our "Global Village" - Internet, Web, Enterprise, Government etc.</p> <p>It is my personal view that an EAV/CR model based DBMS, with support for the seven items enumerated above, can trigger the long anticipated RDBMS downgrade. Such a DBMS would be inherently multi-model because you would need to the best of RDBMS and EAV/CR model engines in a single product, with in-built support for HTTP and other Internet protocols in order to effectively address data representation and serialization issues.</p> <h4>EAV/CR Oriented Data Access & Management Technology</h4> <p>Examples of contemporary EAV/CR frameworks that provide concrete conceptual layers for data access and data management currently include:</p> <ul> <li> <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id115d1cb0"> Resource Description Framework</a> (RDF) - an EAV/CR based framework</li> <li> <a href="http://dbpedia.org/resource/Linked_Data" id="link-id116cf810">RDF Linked Data </a>- EAV/CR based framework that mandates de-referencable HTTP based Identifiers</li> <li> <a href="http://dbpedia.org/resource/ADO.NET_Entity_Framework" id="link-id13daa160">ADO.NET Entity Frameworks</a> - Microsoft .NET based EAV/CR framework</li> <li> <a href="http://dbpedia.org/page/Core_Data" id="link-id11111838">Core Data Services </a>- Mac OS X based EAV/CR framework that evolved from NeXT's <a href="http://dbpedia.org/resource/Enterprise_Objects_Framework" id="link-id15c27df0">Enterprise Object Frameworks</a> (EOF).</li> </ul> <p>The frameworks above provide the basis for a revised AVF pyramid, as depicted below, that reflects today's data access and management realities i.e., an Internet & Web driven global village comprised of interlinked distributed data objects, compatible with "Open World" assumptions.</p> See: <a href="http://virtuoso.openlinksw.com/images/New_EAV_RDBMS_Pyramid.png" id="link-id158e0760">New EAV/CR Primacy Diagram.</a> <h3>Related</h3> <ul> <li> <a href="http://dynamicorange.com/2009/01/22/blueblog-how-and-why-glue-is-using-amazon-simpledb-instead-of-a-relational-database/" id="link-id15e07c10">How & Why Glue is Using Amazon SimpleDB</a> </li> <li> <a href="http://www.cs.cmu.edu/afs/cs.cmu.edu/user/clamen/OODBMS/Manifesto/htManifesto/node4.html#SECTION00022000000000000000" id="link-id116cf450">Object Database Manifesto (Identity excerpt)</a> </li> <li> <a href="http://www.unixspace.com/context/databases.html" id="link-id150b2c20">Database Models Overview</a> </li> <li> <a href="http://www.youtube.com/watch?v=WEj9vqVvHPc&feature=related" id="link-id0x1135d978">Ted Nelson Explaining Irregularity and Idiosyncrasy of Data Structures</a> - ZigZag Demo </li> </ul>
The Time for RDBMS Primacy Downgrade is Nigh!
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2009-01-24#1519
2009-01-25T00:04:00Z
2009-06-03T18:09:58.000001-04:00
<p> As the world works it way through a "once in a generation" economic crisis, the long overdue downgrade of the <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id15750540">RDBMS</a>, from its pivotal position at the apex of the <a href="http://dbpedia.org/resource/Data" id="link-id0x66a74b8">data</a> access and data management pyramid is nigh.</p> <h3>What is the Data Access, and Data Management Value Pyramid?</h3> <p> As depicted below, a top-down view of the data access and data management value chain. The term: apex, simply indicates value primacy, which takes the form of a data access API based entry point into a DBMS realm -- aligned to an underlying data model. Examples of data access APIs include: Native Call Level Interfaces (CLIs), <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id11c254c0">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id149b16a8">JDBC</a>, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id11451eb0">ADO</a>.NET, <a href="http://dbpedia.org/resource/OLE_DB" id="link-id15b02478">OLE-DB</a>, <a href="http://dbpedia.org/resource/XML_for_Analysis" id="link-id1181fa10">XMLA</a>, and <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id0x2fef498">Web</a> Services.</p> <div> <img alt="Image" src="http://virtuoso.openlinksw.com/images/Agility_Value_Factors_Pyramid.png" /> </div> <p> The degree to which ad-hoc views of data managed by a DBMS can be produced and dispatched to relevant data consumers (e.g. people), without compromising concurrency, data durability, and security, collectively determine the "Agility Value Factor" (AVF) of a given DBMS. Remember, agility as the cornerstone of environmental adaptation is as old as the concept of evolution, and intrinsic to all pursuits of primacy. </p> <p>In simpler business oriented terms, look at AVF as the degree to which DBMS technology affects the ability to effectively implement "Market Leadership Discipline" along the following pathways: innovation, operation excellence, or customer intimacy. </p> <h3>Why has RDBMS Primacy has Endured?</h3> <p> Historically, at least since the late '80s, the RDBMS genre of DBMS has consistently offered the highest AVF relative to other DBMS genres en route to primacy within the value pyramid. The desire to improve on paper reports and spreadsheets is basically what DBMS technology has fundamentally addressed to date, even though conceptual level interaction with data has never been its forte.</p> <div> <img alt="Image" src="http://virtuoso.openlinksw.com/images/Old_RDBMS_Primacy_Pyramid.png" /> </div> <p> For more then 10 years -- at the very least -- limitations of the traditional RDBMS in the realm of conceptual level interaction with data across diverse data sources and schemas (enterprise, Web, and <a href="http://dbpedia.org/resource/Internet" id="link-id116001c0">Internet</a>) has been crystal clear to many RDBMS technology practitioners, as indicated by some of the quotes excerpted below:</p> <blockquote> <cite> <p> "Future of Database Research is excellent, but what is the future of data?" </p> "..it is hard for <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id14932398">me</a> to disagree with the conclusions in this report. It captures exactly the right thoughts, and should be a must read for everyone involved in the area of databases and database research in particular." <p>-- <a href="http://jhingran.typepad.com/anant_jhingrans_musings/" id="link-id11334c50">Dr. Anant Jingran</a>, CTO, IBM <a href="http://dbpedia.org/resource/Information" id="link-id150c7970">Information</a> Management Systems, commenting on the <a href="http://db.cs.berkeley.edu/claremont/" id="link-id11c3b408">2007 RDBMS technology retreat</a> attended by a number of key DBMS technology pioneers and researchers.</p> </cite> </blockquote> <blockquote> <cite> <p> "<a href="http://www.databasecolumn.com/2007/09/one-size-fits-all.html" id="link-id15c14f08">One size fits all: A concept whose time has come and gone</a> </p> <p> </p> <ol> <li> They are direct descendants of System R and <a href="http://dbpedia.org/resource/Ingres" id="link-id146da780">Ingres</a> and were architected more than 25 years ago</li> <li> They are advocating "one size fits all"; i.e. a single engine that solves all DBMS needs. </li> </ol> <p>-- Prof. <a href="http://en.wikipedia.org/wiki/Michael_Stonebraker" id="link-id145c4e28">Michael Stonebreaker</a>, one of the founding fathers of the RDBMS industry.</p> </cite> </blockquote> <p>Until this point in time, the requisite confluence of "circumstantial pain" and "open standards" based technology required to enable an objective "compare and contrast" of RDBMS engine virtues and viable alternatives hasn't occurred. Thus, the RDBMS has endured it position of primacy albeit on a "one size fits all basis". </p> <h4>Circumstantial Pain</h4> <p> As mentioned earlier, we are in the midst of an economic crisis that is ultimately about a consistent inability to connect dots across a substrate of interlinked data sources that transcend traditional data access boundaries with high doses of schematic heterogeneity. Ironically, in a era of the dot-com, we haven't been able to make meaningful connections between relevant "real-world things" that extend beyond primitive data hosted database tables and content management style document containers; we've struggled to achieve this in the most basic sense, let alone evolve our ability to connect inline with the <a href="http://www.vldb2007.org/program/slides/s1161-brodie.pdf" id="link-id11a0dcf0">exponential rate at which the Internet & Web are spawning "universes of discourse" (data spaces) that emanate from user activity</a> (within the enterprise and across the Internet & Web). In a nutshell, we haven't been able to upgrade our interaction with data such that "conceptual models" and resulting "<a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id12da4b00">context</a> lenses" (or facets) become concrete; by this I mean: real-world <a href="http://dbpedia.org/resource/Entity" id="link-id146a48a8">entity</a> interaction making its way into the computer realm as opposed to the impedance we all suffer today when we transition from conceptual model interaction (real-world) to logical model interaction (when dealing with RDBMS based data access and data management). </p> <p>Here are some simple examples of what I can only best describe as: "critical dots unconnected", resulting from an inability to interact with data conceptually:</p> <strong>Government (Globally) -</strong> <p> Financial regulatory bodies couldn't effectively discern that a <a href="http://dbpedia.org/resource/Credit_default_swap" id="link-id115ba0e0">Credit Default Swap</a> is an Insurance policy in all but literal name. And in not doing so the cost of an unregulated <a href="http://dbpedia.org/resource/Insurance" id="link-id158d4960">insurance policy</a> laid the foundation for exacerbating the toxicity of fatally flawed mortgage backed securities. Put simply: a flawed insurance policy was the fallback on a toxic security that financiers found exotic based on superficial packaging.</p> <strong>Enterprises - </strong> <p> Banks still don't understand that capital really does exists in tangible and intangible forms; with the intangible being the variant that is inherently dynamic. For example, a tech companies intellectual capital far exceeds the value of fixture, fittings, and buildings, but you be amazed to find that in most cases this vital asset has not significant value when banks get down to the nitty gritty of debt collateral; instead, a buffer of flawed securitization has occurred atop a borderline static asset class covering the aforementioned buildings, fixtures, and fittings. </p> <p> In the general enterprise arena, IT executives continued to "rip and replace" existing technology without ever effectively addressing the timeless inability to connect data across disparate data silos generated by internal enterprise applications, let alone the broader need to mesh data from the inside with external data sources. No correlations made between the growth of buzzwords and the compounding nature of data integration challenges. It's 2009 and only a miniscule number of executives dare fantasize about being anywhere within distance of the: relevant information at your fingertips vision. </p> <p> Looking more holistically at data interaction in general, whether you interact with data in the enterprise space (i.e., at work) or on the Internet or Web, you ultimately are delving into a mishmash of disparate computer systems, applications, service (Web or SOA), and databases (of the RDBMS variety in a majority of cases) associated with a plethora of disparate schemas. Yes, but even today "rip and replace" is still the norm pushed by most vendors; pitting one mono culture against another as exemplified by irrelevances such as: FOSS/LAMP vs Commercial or Web vs. Enterprise, when none of this matters if the data access and integration issues are recognized let alone addressed (see: <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1497?sid=0df0294caee8b37925c6a888bbbca136&realm=wa" id="link-id15c27300">Applications are Like Fish and Data Like Wine</a>). </p> <p> Like the current credit-crunch, exponential growth of data originating from disparate application databases and associated schemas, within shrinking processing time frames, has triggered a rethinking of what defines data access and data management value today en route to an inevitable RDBMS downgrade within the value pyramid.</p> <h3>Technology</h3> <p>There have been many attempts to address real-world modeling requirements across the broader DBMS community from Object Databases to Object-Relational Databases, and more recently the emergence of simple <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id1128dad0">Entity</a>-Attribute-Value model DBMS engines. In all cases failure has come down to the existence of one or more of the following deficiencies, across each potential alternative:</p> <ol> <li>Query language standardization - nothing close to <a href="http://dbpedia.org/resource/SQL" id="link-id16002d60">SQL</a> standardization</li> <li>Data Access API standardization - nothing close to ODBC, JDBC, OLE-DB, or ADO.NET</li> <li>Wire protocol standardization - nothing close to HTTP</li> <li>Distributed Identity infrastructure - nothing close to the non-repudiatable digital Identity that <a href="http://dbpedia.org/resource/Friend_of_a_friend" id="link-id14926b18">foaf</a>+ssl accords</li> <li>Use of Identifiers as network based pointers to data sources - nothing close to RDF based <a href="http://dbpedia.org/resource/Linked_Data" id="link-id16180a28">Linked Data</a> </li> <li>Negotiable data representation - nothing close to Mime and HTTP based Content Negotiation</li> <li>Scalability especially in the era of Internet & Web scale.</li> </ol> <h4>Entity-Attribute-Value with Classes & Relationships (<a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id13e741b8">EAV</a>/CR) data models</h4> <p>A common characteristic shared by all post-relational DBMS management systems (from Object Relational to pure Object) is an orientation towards variations of EAV/CR based data models. Unfortunately, all efforts in the EAV/CR realm have typically suffered from at least one of the deficiencies listed above. In addition, the same "one DBMS model fits all" approach that lies at the heart of the RDBMS downgrade also exists in the EAV/CR realm.</p> <h3>What Comes Next?</h3> <p>The RDBMS is not going away (ever), but its era of primacy -- by virtue of its placement at the apex of the data access and data management value pyramid -- is over! I make this bold claim for the following reasons: </p> <ol> <li> The Internet aided "Global Village" has brought "<a href="http://en.wikipedia.org/wiki/Open_World_Assumption" id="link-id1148e560">Open World</a>" vs "<a href="http://en.wikipedia.org/wiki/Closed_World_Assumption" id="link-id11967cd0">Closed World</a>" assumption issues to the fore e.g., the current global economic crisis remains centered on the inability to connect dots across "Open World" and "Closed World" data frontiers </li> <li> Entity-Attribute-Value with Classes & Relationships (EAV/CR) based DBMS models are more effective when dealing with disparate data associated with disparate schemas, across disparate DBMS engines, host operating systems, and networks. </li> </ol> <p>Based on the above, it is crystal clear that a different kind of DBMS -- one with higher AVF relative to the RDBMS -- needs to sit atop today's data access and data management value pyramid. The characteristics of this DBMS must include the following:</p> <ol> <li> Every item of data (Datum/Entity/Object/Resource) has Identity</li> <li> Identity is achieved via Identifiers that aren't locked at the DBMS, OS, Network, or Application levels</li> <li> Object Identifiers and Object values are independent (extricably linked by association)</li> <li> Object values should be de-referencable via Object Identifier</li> <li> Representation of de-referenced value graph (entity, attributes, and values mesh) must be negotiable (i.e. content negotiation)</li> <li>Structured query language must provide mechanism for Creation, Deletion, Updates, and Querying of data objects</li> <li> Performance & Scalability across "Closed World" (enterprise) and "Open World" (Internet & Web) realms.</li> </ol> <p>Quick recap, I am not saying that RDBMS engine technology is dead or obsolete. I am simply stating that the era of RDBMS primacy within the data access and data management value pyramid is over. </p> <p>The problem domain (conceptual model views over heterogeneous data sources) at the apex of the aforementioned pyramid has simply evolved beyond the natural capabilities of the RDBMS which is rooted in "Closed World" assumptions re., data definition, access, and management. The need to maintain domain based conceptual interaction with data is now palpable at every echelon within our "Global Village" - Internet, Web, Enterprise, Government etc.</p> <p>It is my personal view that an EAV/CR model based DBMS, with support for the seven items enumerated above, can trigger the long anticipated RDBMS downgrade. Such a DBMS would be inherently multi-model because you would need to the best of RDBMS and EAV/CR model engines in a single product, with in-built support for HTTP and other Internet protocols in order to effectively address data representation and serialization issues.</p> <h4>EAV/CR Oriented Data Access & Management Technology</h4> <p>Examples of contemporary EAV/CR frameworks that provide concrete conceptual layers for data access and data management currently include:</p> <ul> <li> <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id115d1cb0"> Resource Description Framework</a> (RDF) - an EAV/CR based framework</li> <li> <a href="http://dbpedia.org/resource/Linked_Data" id="link-id116cf810">RDF Linked Data </a>- EAV/CR based framework that mandates de-referencable HTTP based Identifiers</li> <li> <a href="http://dbpedia.org/resource/ADO.NET_Entity_Framework" id="link-id13daa160">ADO.NET Entity Frameworks</a> - Microsoft .NET based EAV/CR framework</li> <li> <a href="http://dbpedia.org/page/Core_Data" id="link-id11111838">Core Data Services </a>- Mac OS X based EAV/CR framework that evolved from NeXT's <a href="http://dbpedia.org/resource/Enterprise_Objects_Framework" id="link-id15c27df0">Enterprise Object Frameworks</a> (EOF).</li> </ul> <p>The frameworks above provide the basis for a revised AVF pyramid, as depicted below, that reflects today's data access and management realities i.e., an Internet & Web driven global village comprised of interlinked distributed data objects, compatible with "Open World" assumptions.</p> <div> <image src="http://virtuoso.openlinksw.com/images/New_EAV_RDBMS_Pyramid.png"></image> </div> <h3>Related</h3> <ul> <li> <a href="http://allanslibrary.blogspot.com/2009/06/semantic-way.html" id="link-id0xb8c5e498">The Semantic Way</a> - Alan Cho's Summary of <a href="http://www.pwc.com/extweb/home.nsf/docid/1308AF8EA7929CCA852575BA00720F26" id="link-id0xb80f5e10">PwC 2009 tech forecast report on the Semantic Web</a> </li> <li> <a href="http://www.readwriteweb.com/archives/is_the_relational_database_doomed.php" id="link-id0xb8c20658">Is the RDBMS Doomed</a> - <a href="http://www.readwriteweb.com">ReadWriteWeb</a> Article</li> <li> <a href="http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores/" id="link-id0x1ab4778">Anti-RDBMS: a list of Distributed Key-Value Stores</a> - by <a href="http://www.last.fm/user/RJ" id="link-id0x5a968060">Richard Jones</a> (CTO Last.FM)</li> <li> <a href="http://dynamicorange.com/2009/01/22/blueblog-how-and-why-glue-is-using-amazon-simpledb-instead-of-a-relational-database/" id="link-id15e07c10">How & Why Glue is Using Amazon SimpleDB</a> </li> <li> <a href="http://www.cs.cmu.edu/afs/cs.cmu.edu/user/clamen/OODBMS/Manifesto/htManifesto/node4.html#SECTION00022000000000000000" id="link-id116cf450">Object Database Manifesto (Identity excerpt)</a> </li> <li> <a href="http://www.unixspace.com/context/databases.html" id="link-id150b2c20">Database Models Overview</a> </li> <li> <a href="http://www.youtube.com/watch?v=WEj9vqVvHPc&feature=related" id="link-id0x66b0850">Ted Nelson Explaining Irregularity and Idiosyncrasy of Data Structures</a> - ZigZag Demo </li> </ul>
New ADO.NET 3.x Provider for Virtuoso Released (Update 2)
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2009-01-08#1514
2009-01-08T04:36:47Z
2009-01-08T09:12:50.000006-05:00
<p>I am pleased to announce the immediate availability of the <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtAdoNet35Provider" id="link-id142e7390">Virtuoso ADO.NET 3.5 data provider</a> for Microsoft's .NET platform.</p> <h3>What is it?</h3> <p>A data access driver/provider that provides conceptual <a href="http://dbpedia.org/resource/Entity" id="link-id11c36c00">entity</a> oriented access to <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id12fb8618">RDBMS</a> data managed by Virtuoso. Naturally, it also uses Virtuoso's in-built virtual / <a href="http://dbpedia.org/resource/federated_database_system" id="link-id115bedc8">federated database</a> layer to provide access to <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id15153c08">ODBC</a> and <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id13418908">JDBC</a> accessible RDBMS engines such as: <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id134d72f0">Oracle</a> (7.x to latest), <a href="http://dbpedia.org/resource/SQL" id="link-id15757b88">SQL</a> Server (4.2 to latest), <a href="http://dbpedia.org/resource/Sybase" id="link-id15ef8d48">Sybase</a>, IBM <a href="http://dbpedia.org/resource/IBM_Informix" id="link-id12f56aa0">Informix</a> (5.x to latest), IBM <a href="http://dbpedia.org/resource/IBM_DB2" id="link-id119feb38">DB2</a>, <a href="http://dbpedia.org/resource/Ingres" id="link-id14e3d6c8">Ingres</a> (6.x to latest), Progress (7.x to OpenEdge), <a href="http://dbpedia.org/resource/MySQL" id="link-id11295630">MySQL</a>, PostgreSQL, <a href="http://dbpedia.org/resource/Firebird_database_server" id="link-id12f40448">Firebird</a>, and others using our ODBC or JDBC bridge drivers.</p> <h3>Benefits?</h3> <h4>Technical:</h4> <p>It delivers an <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id14012040">Entity-Attribute-Value + Classes & Relationships model</a> over disparate data sources that are materialized as .NET Entity Framework Objects, which are then consumable via ADO.NET Data Object Services, LINQ for Entities, and other ADO.NET data consumers.</p> <p>The provider is fully integrated into Visual Studio 2008 and delivers the same "ease of use" offered by Microsoft's own SQL Server provider, but across Virtuoso, Oracle, Sybase, DB2, Informix, Ingres, <a href="http://dbpedia.org/resource/Progress_4GL" id="link-id158d1fe8">Progress (OpenEdge</a>), MySQL, PostgreSQL, Firebird, and others. The same benefits also apply uniformly to Entity Frameworks compatibility.</p> <p> Bearing in mind that Virtuoso is a multi-model (hybrid) data manager, this also implies that you can use .NET Entity Frameworks against all data managed by Virtuoso. Remember, Virtuoso's SQL channel is a conduit to Virtuoso's core; thus, RDF (courtesy of <a href="http://esw.w3.org/topic/SPASQL" id="link-id133c9b70">SPASQL</a> as already implemented re. <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtJenaProvider" id="link-id11380b80">Jena</a>/<a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtSesame2Provider" id="link-id10fc0c88">Sesame</a>/<a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtRDFDriverRedland" id="link-id1390f730">Redland</a> providers), XML, and other data forms stored in Virtuoso also become accessible via .NET's Entity Frameworks.</p> <br /> <h4>Strategic:</h4> <p>You can choose which entity oriented data access model works best for you: RDF <a href="http://dbpedia.org/resource/Linked_Data" id="link-id151354f0">Linked Data</a> & <a href="http://dbpedia.org/resource/SPARQL" id="link-id15dc5eb0">SPARQL</a> or .NET Entity Frameworks & <a href="http://en.wikipedia.org/wiki/ADO.NET_Entity_Framework#Entity_SQL" id="link-id14404e80">Entity SQL</a>. Either way, Virtuoso delivers a commercial grade, high-performance, secure, and scalable solution.</p> <br /> <h3>How do I use it?</h3> Simply follow one of guides below: <ul> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtEntityFrameworkSchoolDbWinFormApp" id="link-id15e5c580">Using Visual Studio 2008 & Virtuoso to build an Entity Frameworks based Windows forms application</a> </li> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtUsingMsAdoNetDataServicesWithVirtuoso" id="link-id157912b0">Using Visual Studio 2008 & Virtuoso to build an ADO.NET Data Services based application</a> </li> </ul> <p> <b>Note:</b> When working with external or 3rd party databases, simply use the Virtuoso Conductor to link the external data source into Virtuoso. Once linked, the remote tables will simply be treated as though they are native Virtuoso tables leaving the <a href="http://dbpedia.org/resource/Virtual_Database" id="link-id15b04b18">virtual database</a> engine to handle the rest. This is similar to the role the Microsoft JET engine played in the early days of ODBC, so if you've ever linked an ODBC data source into Microsoft Access, you are ready to do the same using Virtuoso.</p> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1420" id="link-id160afdd0">Entity Oriented Data Access</a> </li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1474" id="link-id113eeb50">Yoda & the Data FORCE.</a> </li> </ul>
Virtuoso RDF: A Getting Started Guide for the Developer
http://www.openlinksw.com/blog/vdb/blog/?date=2008-12-17#1505
2008-12-17T12:31:34Z
2008-12-17T12:41:27.000006-05:00
<p>It is a long standing promise of mine to dispel the false impression that using <a href="http://virtuoso.openlinksw.com/" id="link-id113506d0">Virtuoso</a> to work with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id115d9528">RDF</a> is complicated.</p> <p>The purpose of this presentation is to show a programmer how to put RDF into Virtuoso and how to query it. This is done programmatically, with no confusing user interfaces.</p> <p>You should have a Virtuoso Open Source tree built and installed. We will look at the LUBM benchmark demo that comes with the package. All you need is a Unix shell. Running the shell under emacs (<code>m-x shell</code>) is the best. But the open source <code>isql</code> utility should have command line editing also. The emacs shell is however convenient for cutting and pasting things between shell and files.</p> <p>To get started, cd into <code>binsrc/tests/lubm</code>.</p> <p>To verify that this works, you can do </p> <blockquote> <pre>./test_server.sh virtuoso-t</pre></blockquote> <p>This will test the server with the LUBM queries. This should report 45 tests passed. After this we will do the tests step-by-step.</p> <h2>Loading the <a href="http://dbpedia.org/resource/Data" id="link-id10f7bd90">Data</a> </h2> <p>The file <code>lubm-load.sql</code> contains the commands for loading the LUBM single university qualification database.</p> <p>The data files themselves are in <code>lubm_8000</code>, 15 files in RDFXML.</p> <p>There is also a little ontology called <code>inf.nt</code>. This declares the subclass and subproperty relations used in the benchmark.</p> <p>So now let's go through this procedure.</p> <p>Start the server:</p> <blockquote> <pre>$ virtuoso-t -f & </pre></blockquote> <p>This starts the server in foreground mode, and puts it in the background of the shell.</p> <p>Now we connect to it with the isql utility.</p> <blockquote> <pre>$ isql 1111 dba dba </pre></blockquote> <p>This gives a <code>SQL></code> prompt. The default username and password are both <code>dba</code>.</p> <p>When a command is <a href="http://dbpedia.org/resource/SQL" id="link-id1176ce70">SQL</a>, it is entered directly. If it is <a href="http://dbpedia.org/resource/SPARQL" id="link-id156df468">SPARQL</a>, it is prefixed with the keyword <code>sparql</code>. This is how all the SQL clients work. Any SQL client, such as any <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id152d0a00">ODBC</a> or <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id157ad6a0">JDBC</a> application, can use SPARQL if the SQL string starts with this keyword.</p> <p>The <code>lubm-load.sql</code> file is quite self-explanatory. It begins with defining an SQL procedure that calls the RDF/XML load function, <code>DB..RDF_LOAD_RDFXML</code>, for each file in a directory.</p> <p>Next it calls this function for the <code>lubm_8000</code> directory under the server's working directory.</p> <blockquote> <pre>sparql CLEAR GRAPH <lubm>; sparql CLEAR GRAPH <inf>; load_lubm ( server_root() || '/lubm_8000/' ); </pre></blockquote> <p>Then it verifies that the right number of triples is found in the <lubm> graph.</p> <blockquote> <pre>sparql SELECT COUNT(*) FROM <lubm> WHERE { ?x ?y ?z } ; </pre></blockquote> <p>The echo commands below this are interpreted by the isql utility, and produce output to show whether the test was passed. They can be ignored for now.</p> <p>Then it adds some implied <code>subOrganizationOf</code> triples. This is part of setting up the LUBM test database.</p> <blockquote> <pre>sparql PREFIX ub: <http://www.lehigh.edu/~zhp2/2004/0401/univ-bench.owl#> INSERT INTO GRAPH <lubm> { ?x ub:subOrganizationOf ?z } FROM <lubm> WHERE { ?x ub:subOrganizationOf ?y . ?y ub:subOrganizationOf ?z . }; </pre></blockquote> <p>Then it loads the ontology file, <code>inf.nt</code>, using the Turtle load function, <code>DB.DBA.TTLP</code>. The arguments of the function are the text to load, the default namespace prefix, and the <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id15835550">URI</a> of the target graph.</p> <blockquote> <pre>DB.DBA.TTLP ( file_to_string ( 'inf.nt' ), 'http://www.lehigh.edu/~zhp2/2004/0401/univ-bench.owl', 'inf' ) ; sparql SELECT COUNT(*) FROM <inf> WHERE { ?x ?y ?z } ; </pre></blockquote> <p>Then we declare that the triples in the <code><inf></code> graph can be used for inference at run time. To enable this, a SPARQL query will declare that it uses the <code>'inft'</code> rule set. Otherwise this has no effect.</p> <blockquote> <pre>rdfs_rule_set ('inft', 'inf'); </pre></blockquote> <p>This is just a log checkpoint to finalize the work and truncate the transaction log. The server would also eventually do this in its own time.</p> <blockquote> <pre>checkpoint; </pre></blockquote> <p>Now we are ready for querying.</p> <h2>Querying the Data</h2> <p>The queries are given in 3 different versions: The first file, <code>lubm.sql</code>, has the queries with most inference open coded as <code>UNIONs</code>. The second file, <code>lubm-inf.sql</code>, has the inference performed at run time using the ontology <a href="http://dbpedia.org/resource/Information" id="link-id1109faf0">information</a> in the <code><inf></code> graph we just loaded. The last, <code>lubm-phys.sql</code>, relies on having the entailed triples physically present in the <code><lubm></code> graph. These entailed triples are inserted by the SPARUL commands in the <code>lubm-cp.sql</code> file.</p> <p>If you wish to run all the commands in a SQL file, you can type <code>load <filename>;</code> (e.g., <code>load lubm-cp.sql;</code>) at the <code>SQL></code> prompt. If you wish to try individual statements, you can paste them to the command line.</p> <p>For example: </p> <blockquote> <pre>SQL> sparql PREFIX ub: <http://www.lehigh.edu/~zhp2/2004/0401/univ-bench.owl#> SELECT * FROM <lubm> WHERE { ?x a ub:Publication . ?x ub:publicationAuthor <http://www.Department0.University0.edu/AssistantProfessor0> }; VARCHAR _______________________________________________________________________ http://www.Department0.University0.edu/AssistantProfessor0/Publication0 http://www.Department0.University0.edu/AssistantProfessor0/Publication1 http://www.Department0.University0.edu/AssistantProfessor0/Publication2 http://www.Department0.University0.edu/AssistantProfessor0/Publication3 http://www.Department0.University0.edu/AssistantProfessor0/Publication4 http://www.Department0.University0.edu/AssistantProfessor0/Publication5 6 Rows. -- 4 msec. </pre></blockquote> <p>To stop the server, simply type <code>shutdown;</code> at the <code>SQL></code> prompt.</p> <p>If you wish to use a <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id11384668">SPARQL protocol</a> end point, just enable the HTTP listener. This is done by adding a stanza like â</p> <blockquote> <pre>[HTTPServer] ServerPort = 8421 ServerRoot = . ServerThreads = 2 </pre></blockquote> <p>â to the end of the <code>virtuoso.ini</code> file in the <code>lubm</code> directory. Then shutdown and restart (type <code>shutdown;</code> at the <code>SQL></code> prompt and then <code>virtuoso-t -f &</code> at the shell prompt).</p> <p>Now you can connect to the end point with a web browser. The <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id113d02d8">URL</a> is <code>http://localhost:8421/sparql</code>. Without parameters, this will show a human readable form. With parameters, this will execute SPARQL.</p> <p>We have shown how to load and query RDF with Virtuoso using the most basic SQL tools. Next you can access RDF from, for example, <a href="http://dbpedia.org/resource/PHP" id="link-id142d0ba0">PHP</a>, using the PHP ODBC interface.</p> <p>To see how to use <a href="http://jena.sourceforge.net/" id="link-id117074f0">Jena</a> or <a href="http://sourceforge.net/projects/sesame/" id="link-id1103c9b0">Sesame</a> with Virtuoso, look at <a href="http://docs.openlinksw.com/virtuoso/rdfnativestorageproviders.html" id="link-id15488ce8">Native RDF Storage Providers</a>. To see how RDF data types are supported, see <a href="http://docs.openlinksw.com/virtuoso/VirtuosoDriverJDBC.html#jdbcrdf" id="link-id15784a40">Extension datatype for RDF</a> </p> <p>To work with large volumes of data, you must add memory to the configuration file and use the row-autocommit mode, i.e., do <code>log_enable (2);</code> before the load command. Otherwise Virtuoso will do the entire load as a single transaction, and will run out of rollback space. See <a href="http://docs.openlinksw.com/virtuoso/" id="link-id111410f0">documentation</a> for more.</p>
Virtuoso RDF: A Getting Started Guide for the Developer
http://www.openlinksw.com/weblog/oerling/?date=2008-12-17#1504
2008-12-17T12:31:34Z
2008-12-17T12:41:21.000001-05:00
<p>It is a long standing promise of mine to dispel the false impression that using <a href="http://virtuoso.openlinksw.com/" id="link-id113506d0">Virtuoso</a> to work with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id115d9528">RDF</a> is complicated.</p> <p>The purpose of this presentation is to show a programmer how to put RDF into Virtuoso and how to query it. This is done programmatically, with no confusing user interfaces.</p> <p>You should have a Virtuoso Open Source tree built and installed. We will look at the LUBM benchmark demo that comes with the package. All you need is a Unix shell. Running the shell under emacs (<code>m-x shell</code>) is the best. But the open source <code>isql</code> utility should have command line editing also. The emacs shell is however convenient for cutting and pasting things between shell and files.</p> <p>To get started, cd into <code>binsrc/tests/lubm</code>.</p> <p>To verify that this works, you can do </p> <blockquote> <pre>./test_server.sh virtuoso-t</pre></blockquote> <p>This will test the server with the LUBM queries. This should report 45 tests passed. After this we will do the tests step-by-step.</p> <h2>Loading the <a href="http://dbpedia.org/resource/Data" id="link-id10f7bd90">Data</a> </h2> <p>The file <code>lubm-load.sql</code> contains the commands for loading the LUBM single university qualification database.</p> <p>The data files themselves are in <code>lubm_8000</code>, 15 files in RDFXML.</p> <p>There is also a little ontology called <code>inf.nt</code>. This declares the subclass and subproperty relations used in the benchmark.</p> <p>So now let's go through this procedure.</p> <p>Start the server:</p> <blockquote> <pre>$ virtuoso-t -f & </pre></blockquote> <p>This starts the server in foreground mode, and puts it in the background of the shell.</p> <p>Now we connect to it with the isql utility.</p> <blockquote> <pre>$ isql 1111 dba dba </pre></blockquote> <p>This gives a <code>SQL></code> prompt. The default username and password are both <code>dba</code>.</p> <p>When a command is <a href="http://dbpedia.org/resource/SQL" id="link-id1176ce70">SQL</a>, it is entered directly. If it is <a href="http://dbpedia.org/resource/SPARQL" id="link-id156df468">SPARQL</a>, it is prefixed with the keyword <code>sparql</code>. This is how all the SQL clients work. Any SQL client, such as any <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id152d0a00">ODBC</a> or <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id157ad6a0">JDBC</a> application, can use SPARQL if the SQL string starts with this keyword.</p> <p>The <code>lubm-load.sql</code> file is quite self-explanatory. It begins with defining an SQL procedure that calls the RDF/XML load function, <code>DB..RDF_LOAD_RDFXML</code>, for each file in a directory.</p> <p>Next it calls this function for the <code>lubm_8000</code> directory under the server's working directory.</p> <blockquote> <pre>sparql CLEAR GRAPH <lubm>; sparql CLEAR GRAPH <inf>; load_lubm ( server_root() || '/lubm_8000/' ); </pre></blockquote> <p>Then it verifies that the right number of triples is found in the <lubm> graph.</p> <blockquote> <pre>sparql SELECT COUNT(*) FROM <lubm> WHERE { ?x ?y ?z } ; </pre></blockquote> <p>The echo commands below this are interpreted by the isql utility, and produce output to show whether the test was passed. They can be ignored for now.</p> <p>Then it adds some implied <code>subOrganizationOf</code> triples. This is part of setting up the LUBM test database.</p> <blockquote> <pre>sparql PREFIX ub: <http://www.lehigh.edu/~zhp2/2004/0401/univ-bench.owl#> INSERT INTO GRAPH <lubm> { ?x ub:subOrganizationOf ?z } FROM <lubm> WHERE { ?x ub:subOrganizationOf ?y . ?y ub:subOrganizationOf ?z . }; </pre></blockquote> <p>Then it loads the ontology file, <code>inf.nt</code>, using the Turtle load function, <code>DB.DBA.TTLP</code>. The arguments of the function are the text to load, the default namespace prefix, and the <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id15835550">URI</a> of the target graph.</p> <blockquote> <pre>DB.DBA.TTLP ( file_to_string ( 'inf.nt' ), 'http://www.lehigh.edu/~zhp2/2004/0401/univ-bench.owl', 'inf' ) ; sparql SELECT COUNT(*) FROM <inf> WHERE { ?x ?y ?z } ; </pre></blockquote> <p>Then we declare that the triples in the <code><inf></code> graph can be used for inference at run time. To enable this, a SPARQL query will declare that it uses the <code>'inft'</code> rule set. Otherwise this has no effect.</p> <blockquote> <pre>rdfs_rule_set ('inft', 'inf'); </pre></blockquote> <p>This is just a log checkpoint to finalize the work and truncate the transaction log. The server would also eventually do this in its own time.</p> <blockquote> <pre>checkpoint; </pre></blockquote> <p>Now we are ready for querying.</p> <h2>Querying the Data</h2> <p>The queries are given in 3 different versions: The first file, <code>lubm.sql</code>, has the queries with most inference open coded as <code>UNIONs</code>. The second file, <code>lubm-inf.sql</code>, has the inference performed at run time using the ontology <a href="http://dbpedia.org/resource/Information" id="link-id1109faf0">information</a> in the <code><inf></code> graph we just loaded. The last, <code>lubm-phys.sql</code>, relies on having the entailed triples physically present in the <code><lubm></code> graph. These entailed triples are inserted by the SPARUL commands in the <code>lubm-cp.sql</code> file.</p> <p>If you wish to run all the commands in a SQL file, you can type <code>load <filename>;</code> (e.g., <code>load lubm-cp.sql;</code>) at the <code>SQL></code> prompt. If you wish to try individual statements, you can paste them to the command line.</p> <p>For example: </p> <blockquote> <pre>SQL> sparql PREFIX ub: <http://www.lehigh.edu/~zhp2/2004/0401/univ-bench.owl#> SELECT * FROM <lubm> WHERE { ?x a ub:Publication . ?x ub:publicationAuthor <http://www.Department0.University0.edu/AssistantProfessor0> }; VARCHAR _______________________________________________________________________ http://www.Department0.University0.edu/AssistantProfessor0/Publication0 http://www.Department0.University0.edu/AssistantProfessor0/Publication1 http://www.Department0.University0.edu/AssistantProfessor0/Publication2 http://www.Department0.University0.edu/AssistantProfessor0/Publication3 http://www.Department0.University0.edu/AssistantProfessor0/Publication4 http://www.Department0.University0.edu/AssistantProfessor0/Publication5 6 Rows. -- 4 msec. </pre></blockquote> <p>To stop the server, simply type <code>shutdown;</code> at the <code>SQL></code> prompt.</p> <p>If you wish to use a <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id11384668">SPARQL protocol</a> end point, just enable the HTTP listener. This is done by adding a stanza like â</p> <blockquote> <pre>[HTTPServer] ServerPort = 8421 ServerRoot = . ServerThreads = 2 </pre></blockquote> <p>â to the end of the <code>virtuoso.ini</code> file in the <code>lubm</code> directory. Then shutdown and restart (type <code>shutdown;</code> at the <code>SQL></code> prompt and then <code>virtuoso-t -f &</code> at the shell prompt).</p> <p>Now you can connect to the end point with a web browser. The <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id113d02d8">URL</a> is <code>http://localhost:8421/sparql</code>. Without parameters, this will show a human readable form. With parameters, this will execute SPARQL.</p> <p>We have shown how to load and query RDF with Virtuoso using the most basic SQL tools. Next you can access RDF from, for example, <a href="http://dbpedia.org/resource/PHP" id="link-id142d0ba0">PHP</a>, using the PHP ODBC interface.</p> <p>To see how to use <a href="http://jena.sourceforge.net/" id="link-id117074f0">Jena</a> or <a href="http://sourceforge.net/projects/sesame/" id="link-id1103c9b0">Sesame</a> with Virtuoso, look at <a href="http://docs.openlinksw.com/virtuoso/rdfnativestorageproviders.html" id="link-id15488ce8">Native RDF Storage Providers</a>. To see how RDF data types are supported, see <a href="http://docs.openlinksw.com/virtuoso/VirtuosoDriverJDBC.html#jdbcrdf" id="link-id15784a40">Extension datatype for RDF</a> </p> <p>To work with large volumes of data, you must add memory to the configuration file and use the row-autocommit mode, i.e., do <code>log_enable (2);</code> before the load command. Otherwise Virtuoso will do the entire load as a single transaction, and will run out of rollback space. See <a href="http://docs.openlinksw.com/virtuoso/" id="link-id111410f0">documentation</a> for more.</p>
See the Lite: Embeddable/Background Virtuoso starts at 25MB
http://www.openlinksw.com/blog/vdb/blog/?date=2008-12-17#1503
2008-12-17T09:34:12Z
2008-12-17T12:03:49-05:00
<p>We have received many requests for an embeddable-scale <a href="http://virtuoso.openlinksw.com" id="link-id0x1cd69650">Virtuoso</a>. In response to this, we have added a Lite mode, where the initial size of a server process is a tiny fraction of what the initial size would be with default settings. With 2MB of disk cache buffers (ini file setting, <code>NumberOfBuffers = 256</code>), the process size stays under 30MB on 32-bit Linux.</p> <p>The value of this is that one can now have <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1ce89340">RDF</a> and full text indexing on the desktop without running a Java VM or any other memory-intensive software. And of course, all of <a href="http://dbpedia.org/resource/SQL" id="link-id0x1cfc9288">SQL</a> (transactions, stored procedures, etc.) is in the same embeddably-sized container.</p> <p>The Lite executable is a full Virtuoso executable; the Lite mode is controlled by a switch in the configuration file. The executable size is about 10MB for 32-bit Linux. A database created in the Lite mode will be converted into a fully-featured database (tables and indexes are added, among other things) if the server is started with the Lite setting "off"; functionality can be reverted to Lite mode, though it will now consume somewhat more memory, etc.</p> <p>Lite mode offers full SQL and <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1c511da8">SPARQL</a>/SPARUL (via SPASQL), but disables all <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1dac1950">HTTP</a>-based services (WebDAV, application hosting, etc.). Clients can still use all typical database access mechanisms (i.e., <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0xb19a488">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x1d93ee40">JDBC</a>, OLE-DB, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id0x1ce391c0">ADO</a>.<a href="http://dbpedia.org/resource/.NET_Framework" id="link-id0xacf1168">NET</a>, and XMLA) to connect, including the <a href="http://jena.sourceforge.net/" id="link-id0xaaf5b58">Jena</a> and <a href="http://sourceforge.net/projects/sesame/" id="link-id0x1b1e4328">Sesame</a> frameworks for RDF. ODBC now offers full support of RDF <a href="http://dbpedia.org/resource/Data" id="link-id0x1cfc9f78">data</a> types for <a href="http://dbpedia.org/resource/C%2B%2B" id="link-id0xa6059d8">C</a>-based clients. A Redland-compatible API also exists, for use with Redland v1.0.8 and later. </p> <p>Especially for embedded use, we now allow restricting the listener to be a Unix socket, which allows client connections only from the localhost.</p> <p>Shipping an embedded Virtuoso is easy. It just takes one executable and one configuration file. Performance is generally comparable to "normal" mode, except that Lite will be somewhat less scalable on multicore systems.</p> <p>The Lite mode will be included in the next Virtuoso 5 Open Source release.</p>
See the Lite: Embeddable/Background Virtuoso starts at 25MB
http://www.openlinksw.com/weblog/oerling/?date=2008-12-17#1502
2008-12-17T09:34:12Z
2008-12-17T12:03:43-05:00
<p>We have received many requests for an embeddable-scale <a href="http://virtuoso.openlinksw.com" id="link-id0xa5aa1b38">Virtuoso</a>. In response to this, we have added a Lite mode, where the initial size of a server process is a tiny fraction of what the initial size would be with default settings. With 2MB of disk cache buffers (ini file setting, <code>NumberOfBuffers = 256</code>), the process size stays under 30MB on 32-bit Linux.</p> <p>The value of this is that one can now have <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1db79ac8">RDF</a> and full text indexing on the desktop without running a Java VM or any other memory-intensive software. And of course, all of <a href="http://dbpedia.org/resource/SQL" id="link-id0xa923298">SQL</a> (transactions, stored procedures, etc.) is in the same embeddably-sized container.</p> <p>The Lite executable is a full Virtuoso executable; the Lite mode is controlled by a switch in the configuration file. The executable size is about 10MB for 32-bit Linux. A database created in the Lite mode will be converted into a fully-featured database (tables and indexes are added, among other things) if the server is started with the Lite setting "off"; functionality can be reverted to Lite mode, though it will now consume somewhat more memory, etc.</p> <p>Lite mode offers full SQL and <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1b388830">SPARQL</a>/SPARUL (via SPASQL), but disables all <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1d56b618">HTTP</a>-based services (WebDAV, application hosting, etc.). Clients can still use all typical database access mechanisms (i.e., <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x1c5abc38">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x1dade1f8">JDBC</a>, OLE-DB, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id0x25d8e0f0">ADO</a>.<a href="http://dbpedia.org/resource/.NET_Framework" id="link-id0x1d7a1a28">NET</a>, and XMLA) to connect, including the <a href="http://jena.sourceforge.net/" id="link-id0x1d929b98">Jena</a> and <a href="http://sourceforge.net/projects/sesame/" id="link-id0x1b7a9088">Sesame</a> frameworks for RDF. ODBC now offers full support of RDF <a href="http://dbpedia.org/resource/Data" id="link-id0xaf62aa0">data</a> types for <a href="http://dbpedia.org/resource/C%2B%2B" id="link-id0xa8784b0">C</a>-based clients. A Redland-compatible API also exists, for use with Redland v1.0.8 and later. </p> <p>Especially for embedded use, we now allow restricting the listener to be a Unix socket, which allows client connections only from the localhost.</p> <p>Shipping an embedded Virtuoso is easy. It just takes one executable and one configuration file. Performance is generally comparable to "normal" mode, except that Lite will be somewhat less scalable on multicore systems.</p> <p>The Lite mode will be included in the next Virtuoso 5 Open Source release.</p>
Introducing Virtuoso Universal Server (Cloud Edition) for Amazon EC2
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2008-11-28#1489
2008-11-28T19:27:12Z
2008-11-28T16:06:02.000006-05:00
<h3>What is it?</h3> <p>A pre-installed edition of <a href="http://virtuoso.openlinksw.com" id="link-id14bea838">Virtuoso</a> for Amazon's EC2 Cloud platform.</p> <h3>What does it offer?</h3> From a <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> Entrepreneur perspective it offers: <ol> <li> Low cost entry point to a game-changing Web 3.0+ (and beyond) platform that combines <a href="http://dbpedia.org/resource/SQL" id="link-id11309b38">SQL</a>, <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id135f7988">RDF</a>, XML, and Web Services functionality</li> <li> Flexible variable cost model (courtesy of <a href="http://aws.amazon.com/devpay/" id="link-id17941018">EC2 DevPay</a>) tightly bound to revenue generated by your services</li> <li> Delivers federated and/or centralized model flexibility for you SaaS based solutions</li> <li> Simple entry point for developing and deploying sophisticated database driven applications (SQL or RDF <a href="http://dbpedia.org/resource/Linked_Data" id="link-id14ea6b10">Linked Data Web</a> oriented)</li> <li> Complete framework for exploiting OpenID, OAuth (including Role enhancements) that simplifies exploitation of these vital Identity and <a href="http://dbpedia.org/resource/Data">Data</a> Access technologies</li> <li>Easily implement RDF Linked Data based Mail, Blogging, Wikis, Bookmarks, Calendaring, Discussion Forums, Tagging, Social-Networking as <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id11519928">Data Space</a> (data containers) features of your application or service offering</li> <li>Instant alleviation of challenges (e.g. service costs and agility) associated with <a href="http://dbpedia.org/resource/DataPortability" id="link-id111cb610">Data Portability</a> and Open Data Access across Web 2.0 data silos</li> <li> LDAP integration for <a href="http://dbpedia.org/resource/Intranet" id="link-id114a8270">Intranet</a> / <a href="http://dbpedia.org/resource/Extranet" id="link-id10fe4f08">Extranet</a> style applications.</li> </ol> <p>From the DBMS engine perspective it provides you with one or more pre-configured instances of Virtuoso that enable immediate exploitation of the following services:</p> <ol> <li> RDF Database (a Quad Store with <a href="http://dbpedia.org/resource/SPARQL" id="link-id11911bf8">SPARQL</a> & SPARUL Language & Protocol support)</li> <li> <a href="http://dbpedia.org/resource/SQL" id="link-id110544c8">SQL</a> Database (with <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id1524c7d0">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id14cfb658">JDBC</a>, OLE-DB, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id110ec6c8">ADO</a>.NET, and XMLA driver access)</li> <li>XML Database (XML Schema, <a href="http://dbpedia.org/resource/XQuery" id="link-id10ebf218">XQuery</a>/<a href="http://dbpedia.org/resource/XPath" id="link-id142a7898">Xpath</a>, XSLT, Full Text Indexing)</li> <li>Full Text Indexing.</li> </ol> <p>From a Middleware perspective it provides:</p> <ol> <li> RDF Views (Wrappers / Semantic Covers) over SQL, XML, and other data sources accessible via SOAP or REST style Web Services</li> <li> Sponger Service for converting non RDF <a href="http://dbpedia.org/resource/Information" id="link-id11931c60">information</a> resources into RDF <a href="http://dbpedia.org/resource/Linked_Data" id="link-id118f7168">Linked Data</a> "on the fly" via a large collection of pre-installed RDFizer Cartridges.</li> </ol> <p>From the Web Server Platform perspective it provides an alternative to LAMP stack components such as <a href="http://dbpedia.org/resource/MySQL" id="link-id10f7b780">MySQL</a> and Apace by offering</p> <ol> <li> HTTP Web Server</li> <li> WebDAV Server</li> <li> Web <a href="http://dbpedia.org/resource/Application_server" id="link-id1268daa8">Application Server</a> (includes <a href="http://dbpedia.org/resource/PHP" id="link-id1585d238">PHP</a> runtime hosting)</li> <li> SOAP or REST style Web Services Deployment</li> <li> RDF Linked Data Deployment</li> <li> SPARQL (SPARQL Query Language) and SPARUL (SPARQL Update Language) endpoints</li> <li>Virtuoso Hosted PHP packages for <a href="http://dbpedia.org/resource/MediaWiki" id="link-id15568818">MediaWiki</a>, <a href="http://dbpedia.org/resource/Drupal" id="link-id110bd7a8">Drupal</a>, <a href="http://dbpedia.org/resource/WordPress" id="link-id10f66918">Wordpress</a>, and <a href="http://dbpedia.org/resource/PhpBB" id="link-id13fda4d0">phpBB3</a> (just install the relevant Virtuoso Distro. Package). </li> </ol> <p>From the general System Administrator's perspective it provides:</p> <ol> <li> Online Backups (Backup Set dispatched to S3 buckets, FTP, or HTTP/WebDAV server locations)</li> <li>Synchronized Incremental Backups to Backup Set locations</li> <li>Backup Restore from Backup Set location (without exiting to EC2 shell).</li> </ol> <p>Higher level user oriented offerings include:</p> <ol> <li>OpenLink Data Explorer front-end for exploring the burgeoning Linked Data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id11646dc8">Web</a> </li> <li> Ajax based SPARQL Query Builder (iSPARQL) that enables SPARQL Query construction by Example</li> <li>Ajax based SQL Query Builder (QBE) that enables SQL Query construction by Example.</li> </ol> <p>For Web 2.0 / 3.0 users, developers, and entrepreneurs it offers it includes Distributed Collaboration Tools & Social Media realm functionality courtesy of <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id11009930">ODS</a> that includes:</p> <ol> <li> Point of presence on the Linked Data Web that meshes your Identity and your Data via URIs</li> <li> System generated Social Network Profile & Contact Data via <a href="http://dbpedia.org/resource/Friend_of_a_friend" id="link-id1185a1c0">FOAF</a>?</li> <li> System generated <a href="http://dbpedia.org/resource/SIOC" id="link-id14791890">SIOC</a> (Semantically Interconnected Online Community) <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id1577cad8">Data Space</a> (that includes a Social Graph) exposing all your Web data in RDF Linked Data form</li> <li> System generated OpenID and automatic integration with FOAF</li> <li> Transparent Data Integration across Facebook, Digg, LinkedIn, FriendFeed, Twitter, and any other Web 2.0 data space equipped with RSS / Atom support and/or REST style Web Services</li> <li> In-built support for SyncML which enables data synchronization with Mobile Phones.</li> </ol> <h3>How Do I Get Going with It?</h3> <ul> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/ODSInstallationEC2" id="link-id114e1600">Standard Installation Guide</a> </li> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtEC2AMIDBpediaInstall" id="link-id110a98e8">Personal or Service Specific DBpedia Installation Guide</a> </li> </ul>
Virtuoso - Are We Too Clever for Our Own Good? (updated)
http://www.openlinksw.com/blog/vdb/blog/?date=2008-10-26#1467
2008-10-26T12:15:35Z
2008-10-27T12:07:58-04:00
<p>"Physician, heal thyself," it is said. We profess to say what the messaging of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1b4a25f0">semantic web</a> ought to be, but is our own perfect?</p> <p>I will here engage in some critical introspection as well as amplify on some answers given to <a href="http://virtuoso.openlinksw.com" id="link-id0x1e4f9928">Virtuoso</a>-related questions in recent times.</p> <p>I use some conversations from the <a href="http://dbpedia.org/resource/Vienna" id="link-id0x1e6c0ca8">Vienna</a> <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1e56df88">Linked Data</a> Practitioners meeting as a starting point. These views are mine and are limited to the Virtuoso server. These do not apply to the <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id0x1e680440">ODS</a> (<a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id0x1e140068">OpenLink Data Spaces</a>) applications line, <a href="http://oat.openlinksw.com/" id="link-id0x1f4ba630">OAT</a> (<a href="http://oat.openlinksw.com/" id="link-id0x1ba4bac8">OpenLink Ajax Toolkit</a>), or <a href="http://ode.openlinksw.com/" id="link-id0x1d4159b0">ODE</a> (<a href="http://ode.openlinksw.com/" id="link-id0x1e973c80">OpenLink Data Explorer</a>).</p> <h3>"It is not always clear what the main thrust is, we get the impression that you are spread too thin," said <a href="http://www.informatik.uni-leipzig.de/~auer/foaf.rdf#me" id="link-id0x1f8bafe0">Sören Auer</a>.</h3> <p>Well, personally, I am all for core competence. This is why I do not participate in all the online conversations and groups as much as I could, for example. Time and energy are critical resources and must be invested where they make a difference. In this case, the real core competence is running in the database race. This in itself, come to think of it, is a pretty broad concept.</p> <p>This is why we put a lot of emphasis on Linked Data and the <a href="http://dbpedia.org/resource/Data" id="link-id0x200bd1f0">Data</a> Web for now, as this is the emerging game. This is a deliberate choice, not an outside imperative or built-in limitation. More specifically, this means exposing any pre-existing relational data as linked data plus being the definitive <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1fb03528">RDF</a> store.</p> <p>We can do this because we own our database and <a href="http://dbpedia.org/resource/SQL" id="link-id0x1e7dcc70">SQL</a> and data access middleware and have a history of connecting to any <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1e9baf18">RDBMS</a> out there.</p> <p>The principal message we have been hearing from the RDF field is the call for scale of triple storage. This is even louder than the call for relational mapping. We believe that in time mapping will exceed triple storage as such, once we get some real production strength mappings deployed, enough to outperform RDF warehousing.</p> <p>There are also RDF middleware things like RDF-ization and demand-driven web harvesting (i.e, the so-called Sponger). These are <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1f5f6b78">SPARQL</a> options, thus accessed via standard interfaces. We have little desire to create our own languages or APIs, or to tell people how to program. This is why we recently introduced <a href="http://sourceforge.net/projects/sesame/" id="link-id0x206818c8">Sesame</a>- and <a href="http://jena.sourceforge.net/" id="link-id0x202b3348">Jena</a>-compatible APIs to our RDF store. From what we hear, these work. On the other hand, we do not hesitate to move beyond the standards when there is obvious value or necessity. This is why we brought SPARQL up to and beyond SQL expressivity. It is not a case of E3 (Embrace, Extend, Extinguish).</p> <p>Now, this message could be better reflected in our material on the web. This <a href="http://dbpedia.org/resource/Blog" id="link-id0x1c82e508">blog</a> is a rather informal step in this direction; more is to come. For now we concentrate on delivering.</p> <p>The conventional communications wisdom is to split the message by target audience. For this, we should split the RDF, relational, and web services messages from each other. We believe that a challenger, like the semantic web technology stack, must have a compelling message to tell for it to be interesting. This is not a question of research prototypes. The new technology cannot lack something the installed technology takes for granted.</p> <p>This is why we do not tend to show things like how to insert and query a few triples: No business out there will insert and query triples for the sake of triples. There must be a more compelling story â for example, turning the whole world into a database. This is why our examples start with things like turning the <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x20832510">TPC-H</a> database into RDF, queries and all. Anything less is not interesting. Why would an enterprise that has business intelligence and integration issues way more complex than the rather stereotypical TPC-H even look at a technology that pretends to be all for integration and all for expressivity of queries, yet cannot answer the first question of the entry exam?</p> <p>The world out there is complex. But maybe we ought to make some simple tutorials? So, as a call to the people out there, tell us what a good tutorial would be. The question is more about figuring out what is out there and adapting these and making a sort of compatibility list. Jena and Sesame stuff ought to run as is. We could offer a webinar to all the data web luminaries showing how to promote the data web message with Virtuoso. After all, why not show it on the best platform?</p> <h3>"You are arrogant. When I read your papers or documentation, the impression I get is that you say you are smart and the reader is stupid."</h3> <p>We should answer in multiple parts.</p> <p>For general collateral, like web sites and documentation:</p> <p>The web site gives a confused product image. For the Virtuoso product, we should divide at the top into</p> <ul> <li> Data web and RDF - Host linked data, expose relational assets as linked data;</li> <li> Relational Database - Full function, high performance, open source, Federated/Virtual Relational DBMS, expose heterogeneous RDB assets through one point of contact for integration;</li> <li> Web Services - access all the above over standard protocols, dynamic web pages, web hosting.</li> </ul> <p>For each point, one simple statement. We all know what the above things mean?</p> <p>Then we add a new point about scalability that impacts all the above, namely the Virtuoso version 6 Cluster, meaning that you can do all these things at 10 to 1000 times the scale. This means this much more data or in some cases this much more requests per second. This too is clear.</p> <p>Far as I am concerned, hosting Java or .<a href="http://dbpedia.org/resource/.NET_Framework" id="link-id0x20283a88">NET</a> does not have to be on the front page. Also, we have no great interest in going against <a href="http://dbpedia.org/resource/Apache" id="link-id0x2024a068">Apache</a> when it comes to a web server only situation. The fact that we have a web listener is important for some things but our claim to fame does not rest on this.</p> <p>Then for documentation and training materials: The documentation should be better. Specifically it should have more of a how-to dimension since nobody reads the whole thing anyhow. About online tutorials, the order of presentation should be different. They do not really reflect what is important at the present moment either.</p> <p>Now for conference papers: Since taking the data web as a focus area, we have submitted some papers and had some rejected because these do not have enough references and do not explain what is obvious to ourselves.</p> <p>I think that the communications failure in this case is that we want to talk about end to end solutions and the reviewers expect research. For us, the solution is interesting and exists only if there is an adequate functionality mix for addressing a specific use case. This is why we do not make a paper about query cost model alone because the cost model, while indispensable, is a thing that is taken for granted where we come from. So we mention RDF adaptations to cost model, as these are important to the whole but do not find these to be the justification for a whole paper. If we made papers on this basis, we would have to make five times as many. Maybe we ought to.</p> <h3>"Virtuoso is very big and very difficult"</h3> <p>One thing that is not obvious from the Virtuoso packaging is that the minimum installation is an executable under 10MB and a config file. Two files.</p> <p>This gives you SQL and SPARQL out of the box. Adding <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x1ee61058">ODBC</a> and <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x1b8c31c0">JDBC</a> clients is as simple as it gets. After this, there is basic database functionality. Tuning is a matter of a few parameters that are explained on this blog and elsewhere. Also, the full scale installation is available as an Amazon EC2 image, so no installation required.</p> <p>Now for the difficult side:</p> <p>Use SQL and SPARQL; use stored procedures whenever there is server side business logic. For some time critical web pages, use VSP. Do not use VSPX. Otherwise, use whatever you are used to â <a href="http://dbpedia.org/resource/PHP" id="link-id0x20a13c00">PHP</a> or Java or anything else. For web services, simple is best. Stick to basics. "The engineer is one who can invent a simple thing." Use SQL statements rather than admin UI.</p> <p>Know that you can start a server with no database file and you get an initial database with nothing extra. The demo database, the way it is produced by installers is cluttered.</p> <p>We should put this into a couple of use case oriented how-tos.</p> <p>Also, we should create a network of "friendly local virtuoso geeks" for providing basic training and services so we do not have to explain these things all the time. To all you data-web-ers out there â please sign up and we will provide instructions, etc. Contact Yrjänä Rankka (ghard[at-sign]openlinksw.com), or go through the mailing lists; do not contact me directly.</p> <h3>"OK, we understand that you may be good at the large end of the spectrum but how do you reconcile this with the lightweight or embedded end, like the semantic desktop?"</h3> <p>Now, what is good for one end is usually good for the other. Namely, a database, no matter the scale, needs to have space efficient storage, fast index lookup, and correct query plans. Then there are things that occur only at the high-end, like clustering, but these are separate things. For embedding, the initial memory footprint needs to be small. With Virtuoso, this is accomplished by leaving out some 200 built-in tables and 100,000 lines of SQL procedures that are normally in by default, supporting things such as DAV and diverse other protocols. After all, if SPARQL is all one wants these are not needed.</p> <p>If one really wants to do one's server logic (like web listener and thread dispatching) oneself, this is not impossible but requires some advice from us. On the other hand, if one wants to have logic for security close to the data, then using stored procedures is recommended; these execute right next to the data, and support inline SPARQL and SQL. Depending on the license status of the other code, some special licensing arrangements may apply.</p> <p>We are talking about such things with different parties at present.</p> <h3>"How webby are you? What is webby?"</h3> <p>"Webby means distributed, heterogeneous, open; not monolithic consolidation of everything."</p> <p>We are philosophically webby. We come from open standards; we are after all called OpenLink; our history consists of connecting things. We believe in choice â the user should be able to pick the best of breed for components and have them work together. We cannot and do not wish to force replacement of existing assets. Transforming data on the fly and connecting systems, leaving data where it originally resides, is the first preference. For the data web, the first preference is a federation of independent SPARQL end points. When there is harvesting, we prefer to do it on demand, as with our Sponger. With the immense amount of data out there we believe in finding what is relevant <i>when</i> it is relevant, preferably close at hand, leveraging things like social networks. With a data web, many things which are now siloized, such as marketplaces and social networks, will return to the open.</p> <p>Google-style crawling of everything becomes less practical if one needs to run complex <i>ad hoc</i> queries against the mass of data. For these types of scenarios, if one needs to warehouse, the data cloud will offer solutions where one pays for database on demand. While we believe in loosely coupled federation where possible, we have serious work on the scalability side for the data center and the compute-on-demand cloud.</p> <h3>"How does OpenLink see the next five years unfolding?"</h3> <p>Personally, I think we have the basics for the birth of a new inflection in the <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x1fb9ae58">knowledge</a> economy. The <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x1f07c648">URI</a> is the unit of exchange; its value and competitive edge lie in the data it links you with. A name without context is worth little, but as a name gets more use, more <a href="http://dbpedia.org/resource/Information" id="link-id0x1f007d60">information</a> can be found through that name. This is anything from financial statistics, to legal precedents, to news reporting or government data. Right now, if the SEC just added one line of markup to the XBRL template, this would instantaneously make all SEC-mandated reporting into linked data via GRDDL.</p> <p>The URI is a carrier of brand. An information brand gets traffic and references, and this can be monetized in diverse ways. The key word is <i>context</i>. Information overload is here to stay, and only better context offers the needed increase in productivity to stay ahead of the flood.</p> <p>Semantic technologies on the whole can help with this. Why these should be semantic web or data web technologies as opposed to just semantic is the linked data value proposition. Even smart islands are still islands. Agility, scale, and scope, depend on the possibility of combining things. Therefore common terminologies and dereferenceability and discoverability are important. Without these, we are at best dealing with closed systems even if they were smart. The expert systems of the 1980s are a case in point.</p> <p>Ever since the .com era, the <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id0x2048e670">URL</a> has been a brand. Now it becomes a URI. Thus, entirely hiding the URI from the user experience is not always desirable. The URI is a sort of handle on the provenance and where more can be found; besides, people are already used to these.</p> <p>With linked data, information value-add products become easy to build and deploy. They can be basically just canned SPARQL queries combining data in a useful and insightful manner. And where there is traffic there can be monetization, whether by advertizing, subscription, or other means. Such possibilities are a natural adjunct to the blogosphere. To publish analysis, one no longer needs to be a think tank or media company. We could call this scenario the birth of a meshup economy.</p> <p>For OpenLink itself, this is our roadmap. The immediate future is about getting our high end offerings like clustered RDF storage generally available, both on the cloud and for private data centers. Ourselves, we will offer the whole <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1c696170">Linked Open Data</a> cloud as a database. The single feature to come in version 2 of this is fully automatic partitioning and repartitioning for on-demand scale; now, you have to choose how many partitions you have.</p> <p>This makes some things possible that were hard thus far.</p> <p>On the mapping front, we go for real-scale data integration scenarios where we can show that SPARQL can unify terms and concepts across databases, yet bring no added cost for complex queries. Enterprises can use their existing warehouses and have an added level of abstraction, the possibility of cross systems interlinking, the advantages of using the same taxonomies and ontologies across systems, and so forth.</p> <p>Then there will be developments in the direction of smarter web harvesting on demand with the Virtuoso <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id0x206ab780">Sponger</a>, and federation of heterogeneous SPARQL end points. The federation is not so unlike clustering, except the time scales are 2 orders of magnitude longer. The work on SPARQL end point statistics and data set description and discovery is a good development in the community.</p> <p>Then there will be NLP integration, as exemplified by the Open Calais linked data wrapper and more.</p> <p>Can we pull this off or is this being spread too thin? We know from experience that all this can be accomplished. Scale is already here; we show it with the billion triples set. Mapping is here; we showed it last in the Berlin Benchmark. We will also show some TPC-H results after we get a little quiet after the ISWC event. Then there is ongoing maintenance but with this we have shown a steady turnaround and quick time to fix for pretty much anything.</p>
Virtuoso - Are We Too Clever for Our Own Good? (updated)
http://www.openlinksw.com/weblog/oerling/?date=2008-10-26#1465
2008-10-26T12:15:35Z
2008-10-27T12:07:52-04:00
<p>"Physician, heal thyself," it is said. We profess to say what the messaging of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1fa3da18">semantic web</a> ought to be, but is our own perfect?</p> <p>I will here engage in some critical introspection as well as amplify on some answers given to <a href="http://virtuoso.openlinksw.com" id="link-id0x1e1eecf0">Virtuoso</a>-related questions in recent times.</p> <p>I use some conversations from the <a href="http://dbpedia.org/resource/Vienna" id="link-id0x1ec0b2e0">Vienna</a> <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x2045ac10">Linked Data</a> Practitioners meeting as a starting point. These views are mine and are limited to the Virtuoso server. These do not apply to the <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id0x2045ac38">ODS</a> (<a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id0x14f63c58">OpenLink Data Spaces</a>) applications line, <a href="http://oat.openlinksw.com/" id="link-id0x14f63c80">OAT</a> (<a href="http://oat.openlinksw.com/" id="link-id0x1e536928">OpenLink Ajax Toolkit</a>), or <a href="http://ode.openlinksw.com/" id="link-id0x1eaed7f8">ODE</a> (<a href="http://ode.openlinksw.com/" id="link-id0x1edfff88">OpenLink Data Explorer</a>).</p> <h3>"It is not always clear what the main thrust is, we get the impression that you are spread too thin," said <a href="http://www.informatik.uni-leipzig.de/~auer/foaf.rdf#me" id="link-id0x1b8a9580">Sören Auer</a>.</h3> <p>Well, personally, I am all for core competence. This is why I do not participate in all the online conversations and groups as much as I could, for example. Time and energy are critical resources and must be invested where they make a difference. In this case, the real core competence is running in the database race. This in itself, come to think of it, is a pretty broad concept.</p> <p>This is why we put a lot of emphasis on Linked Data and the <a href="http://dbpedia.org/resource/Data" id="link-id0x1b85fa38">Data</a> Web for now, as this is the emerging game. This is a deliberate choice, not an outside imperative or built-in limitation. More specifically, this means exposing any pre-existing relational data as linked data plus being the definitive <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1f5b4468">RDF</a> store.</p> <p>We can do this because we own our database and <a href="http://dbpedia.org/resource/SQL" id="link-id0x20076468">SQL</a> and data access middleware and have a history of connecting to any <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1ffd6f98">RDBMS</a> out there.</p> <p>The principal message we have been hearing from the RDF field is the call for scale of triple storage. This is even louder than the call for relational mapping. We believe that in time mapping will exceed triple storage as such, once we get some real production strength mappings deployed, enough to outperform RDF warehousing.</p> <p>There are also RDF middleware things like RDF-ization and demand-driven web harvesting (i.e, the so-called Sponger). These are <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1316f720">SPARQL</a> options, thus accessed via standard interfaces. We have little desire to create our own languages or APIs, or to tell people how to program. This is why we recently introduced <a href="http://sourceforge.net/projects/sesame/" id="link-id0x20756a68">Sesame</a>- and <a href="http://jena.sourceforge.net/" id="link-id0x1ec01ac0">Jena</a>-compatible APIs to our RDF store. From what we hear, these work. On the other hand, we do not hesitate to move beyond the standards when there is obvious value or necessity. This is why we brought SPARQL up to and beyond SQL expressivity. It is not a case of E3 (Embrace, Extend, Extinguish).</p> <p>Now, this message could be better reflected in our material on the web. This <a href="http://dbpedia.org/resource/Blog" id="link-id0x2027b410">blog</a> is a rather informal step in this direction; more is to come. For now we concentrate on delivering.</p> <p>The conventional communications wisdom is to split the message by target audience. For this, we should split the RDF, relational, and web services messages from each other. We believe that a challenger, like the semantic web technology stack, must have a compelling message to tell for it to be interesting. This is not a question of research prototypes. The new technology cannot lack something the installed technology takes for granted.</p> <p>This is why we do not tend to show things like how to insert and query a few triples: No business out there will insert and query triples for the sake of triples. There must be a more compelling story â for example, turning the whole world into a database. This is why our examples start with things like turning the <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x2051ff98">TPC-H</a> database into RDF, queries and all. Anything less is not interesting. Why would an enterprise that has business intelligence and integration issues way more complex than the rather stereotypical TPC-H even look at a technology that pretends to be all for integration and all for expressivity of queries, yet cannot answer the first question of the entry exam?</p> <p>The world out there is complex. But maybe we ought to make some simple tutorials? So, as a call to the people out there, tell us what a good tutorial would be. The question is more about figuring out what is out there and adapting these and making a sort of compatibility list. Jena and Sesame stuff ought to run as is. We could offer a webinar to all the data web luminaries showing how to promote the data web message with Virtuoso. After all, why not show it on the best platform?</p> <h3>"You are arrogant. When I read your papers or documentation, the impression I get is that you say you are smart and the reader is stupid."</h3> <p>We should answer in multiple parts.</p> <p>For general collateral, like web sites and documentation:</p> <p>The web site gives a confused product image. For the Virtuoso product, we should divide at the top into</p> <ul> <li> Data web and RDF - Host linked data, expose relational assets as linked data;</li> <li> Relational Database - Full function, high performance, open source, Federated/Virtual Relational DBMS, expose heterogeneous RDB assets through one point of contact for integration;</li> <li> Web Services - access all the above over standard protocols, dynamic web pages, web hosting.</li> </ul> <p>For each point, one simple statement. We all know what the above things mean?</p> <p>Then we add a new point about scalability that impacts all the above, namely the Virtuoso version 6 Cluster, meaning that you can do all these things at 10 to 1000 times the scale. This means this much more data or in some cases this much more requests per second. This too is clear.</p> <p>Far as I am concerned, hosting Java or .<a href="http://dbpedia.org/resource/.NET_Framework" id="link-id0x1f297540">NET</a> does not have to be on the front page. Also, we have no great interest in going against <a href="http://dbpedia.org/resource/Apache" id="link-id0x1ea29578">Apache</a> when it comes to a web server only situation. The fact that we have a web listener is important for some things but our claim to fame does not rest on this.</p> <p>Then for documentation and training materials: The documentation should be better. Specifically it should have more of a how-to dimension since nobody reads the whole thing anyhow. About online tutorials, the order of presentation should be different. They do not really reflect what is important at the present moment either.</p> <p>Now for conference papers: Since taking the data web as a focus area, we have submitted some papers and had some rejected because these do not have enough references and do not explain what is obvious to ourselves.</p> <p>I think that the communications failure in this case is that we want to talk about end to end solutions and the reviewers expect research. For us, the solution is interesting and exists only if there is an adequate functionality mix for addressing a specific use case. This is why we do not make a paper about query cost model alone because the cost model, while indispensable, is a thing that is taken for granted where we come from. So we mention RDF adaptations to cost model, as these are important to the whole but do not find these to be the justification for a whole paper. If we made papers on this basis, we would have to make five times as many. Maybe we ought to.</p> <h3>"Virtuoso is very big and very difficult"</h3> <p>One thing that is not obvious from the Virtuoso packaging is that the minimum installation is an executable under 10MB and a config file. Two files.</p> <p>This gives you SQL and SPARQL out of the box. Adding <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x20a2e7d0">ODBC</a> and <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x1e4cceb8">JDBC</a> clients is as simple as it gets. After this, there is basic database functionality. Tuning is a matter of a few parameters that are explained on this blog and elsewhere. Also, the full scale installation is available as an Amazon EC2 image, so no installation required.</p> <p>Now for the difficult side:</p> <p>Use SQL and SPARQL; use stored procedures whenever there is server side business logic. For some time critical web pages, use VSP. Do not use VSPX. Otherwise, use whatever you are used to â <a href="http://dbpedia.org/resource/PHP" id="link-id0x20b03f08">PHP</a> or Java or anything else. For web services, simple is best. Stick to basics. "The engineer is one who can invent a simple thing." Use SQL statements rather than admin UI.</p> <p>Know that you can start a server with no database file and you get an initial database with nothing extra. The demo database, the way it is produced by installers is cluttered.</p> <p>We should put this into a couple of use case oriented how-tos.</p> <p>Also, we should create a network of "friendly local virtuoso geeks" for providing basic training and services so we do not have to explain these things all the time. To all you data-web-ers out there â please sign up and we will provide instructions, etc. Contact Yrjänä Rankka (ghard[at-sign]openlinksw.com), or go through the mailing lists; do not contact me directly.</p> <h3>"OK, we understand that you may be good at the large end of the spectrum but how do you reconcile this with the lightweight or embedded end, like the semantic desktop?"</h3> <p>Now, what is good for one end is usually good for the other. Namely, a database, no matter the scale, needs to have space efficient storage, fast index lookup, and correct query plans. Then there are things that occur only at the high-end, like clustering, but these are separate things. For embedding, the initial memory footprint needs to be small. With Virtuoso, this is accomplished by leaving out some 200 built-in tables and 100,000 lines of SQL procedures that are normally in by default, supporting things such as DAV and diverse other protocols. After all, if SPARQL is all one wants these are not needed.</p> <p>If one really wants to do one's server logic (like web listener and thread dispatching) oneself, this is not impossible but requires some advice from us. On the other hand, if one wants to have logic for security close to the data, then using stored procedures is recommended; these execute right next to the data, and support inline SPARQL and SQL. Depending on the license status of the other code, some special licensing arrangements may apply.</p> <p>We are talking about such things with different parties at present.</p> <h3>"How webby are you? What is webby?"</h3> <p>"Webby means distributed, heterogeneous, open; not monolithic consolidation of everything."</p> <p>We are philosophically webby. We come from open standards; we are after all called OpenLink; our history consists of connecting things. We believe in choice â the user should be able to pick the best of breed for components and have them work together. We cannot and do not wish to force replacement of existing assets. Transforming data on the fly and connecting systems, leaving data where it originally resides, is the first preference. For the data web, the first preference is a federation of independent SPARQL end points. When there is harvesting, we prefer to do it on demand, as with our Sponger. With the immense amount of data out there we believe in finding what is relevant <i>when</i> it is relevant, preferably close at hand, leveraging things like social networks. With a data web, many things which are now siloized, such as marketplaces and social networks, will return to the open.</p> <p>Google-style crawling of everything becomes less practical if one needs to run complex <i>ad hoc</i> queries against the mass of data. For these types of scenarios, if one needs to warehouse, the data cloud will offer solutions where one pays for database on demand. While we believe in loosely coupled federation where possible, we have serious work on the scalability side for the data center and the compute-on-demand cloud.</p> <h3>"How does OpenLink see the next five years unfolding?"</h3> <p>Personally, I think we have the basics for the birth of a new inflection in the <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x2018bd98">knowledge</a> economy. The <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x1ec110d8">URI</a> is the unit of exchange; its value and competitive edge lie in the data it links you with. A name without context is worth little, but as a name gets more use, more <a href="http://dbpedia.org/resource/Information" id="link-id0x1ecfba08">information</a> can be found through that name. This is anything from financial statistics, to legal precedents, to news reporting or government data. Right now, if the SEC just added one line of markup to the XBRL template, this would instantaneously make all SEC-mandated reporting into linked data via GRDDL.</p> <p>The URI is a carrier of brand. An information brand gets traffic and references, and this can be monetized in diverse ways. The key word is <i>context</i>. Information overload is here to stay, and only better context offers the needed increase in productivity to stay ahead of the flood.</p> <p>Semantic technologies on the whole can help with this. Why these should be semantic web or data web technologies as opposed to just semantic is the linked data value proposition. Even smart islands are still islands. Agility, scale, and scope, depend on the possibility of combining things. Therefore common terminologies and dereferenceability and discoverability are important. Without these, we are at best dealing with closed systems even if they were smart. The expert systems of the 1980s are a case in point.</p> <p>Ever since the .com era, the <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id0x1c4c9248">URL</a> has been a brand. Now it becomes a URI. Thus, entirely hiding the URI from the user experience is not always desirable. The URI is a sort of handle on the provenance and where more can be found; besides, people are already used to these.</p> <p>With linked data, information value-add products become easy to build and deploy. They can be basically just canned SPARQL queries combining data in a useful and insightful manner. And where there is traffic there can be monetization, whether by advertizing, subscription, or other means. Such possibilities are a natural adjunct to the blogosphere. To publish analysis, one no longer needs to be a think tank or media company. We could call this scenario the birth of a meshup economy.</p> <p>For OpenLink itself, this is our roadmap. The immediate future is about getting our high end offerings like clustered RDF storage generally available, both on the cloud and for private data centers. Ourselves, we will offer the whole <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x20791bf0">Linked Open Data</a> cloud as a database. The single feature to come in version 2 of this is fully automatic partitioning and repartitioning for on-demand scale; now, you have to choose how many partitions you have.</p> <p>This makes some things possible that were hard thus far.</p> <p>On the mapping front, we go for real-scale data integration scenarios where we can show that SPARQL can unify terms and concepts across databases, yet bring no added cost for complex queries. Enterprises can use their existing warehouses and have an added level of abstraction, the possibility of cross systems interlinking, the advantages of using the same taxonomies and ontologies across systems, and so forth.</p> <p>Then there will be developments in the direction of smarter web harvesting on demand with the Virtuoso <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id0x1f27e6d8">Sponger</a>, and federation of heterogeneous SPARQL end points. The federation is not so unlike clustering, except the time scales are 2 orders of magnitude longer. The work on SPARQL end point statistics and data set description and discovery is a good development in the community.</p> <p>Then there will be NLP integration, as exemplified by the Open Calais linked data wrapper and more.</p> <p>Can we pull this off or is this being spread too thin? We know from experience that all this can be accomplished. Scale is already here; we show it with the billion triples set. Mapping is here; we showed it last in the Berlin Benchmark. We will also show some TPC-H results after we get a little quiet after the ISWC event. Then there is ongoing maintenance but with this we have shown a steady turnaround and quick time to fix for pretty much anything.</p>
Where Are All the RDF-based Semantic Web Applications?
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2008-10-01#1447
2008-10-01T23:09:00Z
2008-10-02T15:27:41-04:00
<p> In response to the "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id15971040">Semantic Web</a> Technology" application classification scheme espoused by <a href="http://www.readwriteweb.com" id="link-id16391540">ReadWriteWeb</a> (RWW), emphasized in the post titled: <a href="http://www.readwriteweb.com/archives/rdf_semantic_web_apps.php" id="link-id1157eaa0">Where are all the RDF-based Semantic Web Apps?</a>, here is my attempt to clarify and reintroduce what <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id15a43758">OpenLink Software</a> offers (today) in relation to Semantic Web technology. </p> <p> From the RWW Top-Down category, which I interpret as: technologies that produce RDF from non RDF <a href="http://dbpedia.org/resource/Data">data</a> sources. Our product portfolio is comprised of the following; <a href="http://virtuoso.openlinksw.com" id="link-id14f05818">Virtuoso Universal Server</a>, <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id162c8630">OpenLink Data Spaces</a>, <a href="http://oat.openlinksw.com" id="link-id134e1a00">OpenLink Ajax Toolkit</a>, and <a href="http://ode.openlinksw.com" id="link-id160b3bf8">OpenLink Data Explorer</a> (which includes ubiquity commands).</p> <h3>Virtuoso Universal Server functionality summary:</h3> <ol> <li>Generation of RDF <a href="http://dbpedia.org/resource/Linked_Data" id="link-id161d5f50">Linked Data</a> Views of <a href="http://dbpedia.org/resource/SQL" id="link-id161d5978">SQL</a>, XML, and <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> Services in general </li> <li>Deployment of RDF Linked Data </li> <li>"On the Fly" generation of RDF Linked Data from Document Web <a href="http://www.w3.org/TR/webarch/" id="link-id178bbc08">information resources</a> (i.e. distillation of entities from their containers e.g. Web pages) via Cartridges / Drivers</li> <li> <a href="http://dbpedia.org/resource/SPARQL" id="link-id162c2118">SPARQL</a> query language support </li> <li>SPARQL extensions that bring SPARQL closer to SQL e.g Aggregates, Update, Insert, Delete Named Graph support (i.e. use of logical names to partition RDF data within Virtuoso's multi-model dbms engine) </li> <li>Inference Engine (currently in use re. <a href="http://dbpedia.org/resource/DBpedia" id="link-id14f563c0">DBpedia</a> via Yago and <a href="http://umbel.org/about/" id="link-id113273b8">UMBEL</a>)</li> <li>Host and exposes data from <a href="http://dbpedia.org/resource/Drupal" id="link-id123d3bd8">Drupal</a>, <a href="http://dbpedia.org/resource/WordPress" id="link-id141adf40">Wordpress</a>, <a href="http://dbpedia.org/resource/MediaWiki" id="link-id1604b450">MediaWiki</a>, <a href="http://dbpedia.org/resource/PhpBB" id="link-id141013a8">phpBB3</a> as RDF Linked Data via in-built support for <a href="http://dbpedia.org/resource/PHP" id="link-id14661e58">PHP</a> runtime</li> <li> <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/ODSInstallationEC2" id="link-id146c84d0">Available as an EC2 AMI</a> </li> <li>etc..</li> </ol> <h3>OpenLink Data Spaces functionality summary:</h3> <ol> <li>Simple mechanism for Linked Data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id15473770">Web</a> enabling yourself by giving you an <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/GetAPersonalURIIn5MinutesOrLess" id="link-id15f6d278">HTTP based User ID</a> (a de-referencable <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id15aaeb68">URI</a>) that is linked to a <a href="http://myopenlink.net/dataspace/person/kidehen" id="link-id15a7a840">FOAF based Profile page</a> and OpenID</li> <li>Binds all your data sources (blogs, wikis, bookmarks, photos, calendar items etc. ) to your URI so can "Find" things by only remembering your URI</li> <li>Makes your profile page and personal URI the focal point of Linked Data Web presence</li> <li>Delivers Data Portability (using data access by value or <a href="http://dbpedia.org/resource/Reference_(computer_science)" id="link-id16212838">data access by reference</a>) across data silos (e.g. Web 2.0 style social networks)</li> <li>Allows you make annotations about anything in your own <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id14668010">Data Space</a>(s) on the Web without exposure to RDF markup</li> <li>A Briefcase feature that provides a WebDAV driven RDF Linked Data variant of functionality seen in Mac OS X Spotlight and WinFS with the addition of SPARQL compliance</li> <li>Automatically generates <a href="http://dbpedia.org/resource/RDFa" id="link-id14691440">RDFa</a> in its (X)HTML pages</li> <li> <a href="http://dbpedia.org/resource/Blog" id="link-id14fae7b8">Blog</a>, Wiki, WebDAV File Server, Shared Bookmarks, Calendar, and other applications that look and feel like Web 2.0 counterparts but emitt RDF Linked Data amongst a plethora of data exchange formats</li> <li>Available as an EC2 AMI</li> <li>etc..</li> </ol> <h3>OpenLink Ajax Toolkit functionality summary:</h3> <ol> <li>Provides binding to SQL, RDF, XML, and Web Services via Ajax Database Connectivity Layer (you only need an <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id11550548">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id13ae5f68">JDBC</a>, OLE-DB, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id162803e8">ADO</a>.NET, XMLA Driver, or Web Service on the backend for dynamic data access from Javascript)</li> <li>All controls are Ajax Database Connectivity bound (widgets get their data from Ajax Database Connectivity data sources)</li> <li>Bundled with Virtuoso and <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id161dfe90">ODS</a> installations.</li> <li>etc.</li> </ol> <h3>OpenLink Data Explorer functionality summary</h3> <ol> <li>Distills entities associated with information resource style containers (e.g. Web Pages or files) as RDF Linked Data</li> <li>Exposes the RDF based <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id12a42ed8">Linked Data graph</a> associated with information resources (see the Linked Data behind Web pages)</li> <li>Ubiquity commands for invoking the above</li> <li>Available as a <a href="http://linkeddata.uriburner.com/ode" id="link-id15a0d2b0">Hosted Service</a> or <a href="http://ode.openlinksw.com" id="link-id138b9fa8">Firefox Extension</a> </li> <li>Bundled with Virtuoso and ODS installations</li> <li>etc.</li> </ol> <h3>Note:</h3> <p>Of course you could have simply looked up <a href="http://www.openlinksw.com/dataspace/organization/openlink" id="link-id14ef2c10">OpenLink Software's FOAF based Profile page</a> (*note the Linked Data Explorer tab*), or simply passed the <a href="http://dbpedia.org/resource/Friend_of_a_friend" id="link-id14cbf5c8">FOAF</a> profile page <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id16453e28">URL</a> to a Linked Data aware client application such as: <a href="http://linkeddata.uriburner.com/ode" id="link-id15a80500">OpenLink Data Explorer</a>, <a href="http://zitgist.com/about/" id="link-id1586a360">Zitgist</a> <a href="http://dataviewer.zitgist.com" id="link-id16249f60">Data Viewer</a>, <a href="http://beckr.org/marbles" id="link-id15993fb0">Marbles</a>, and <a href="http://dig.csail.mit.edu/2005/ajar/release/tabulator/0.8/tab.html" id="link-id14d63048">Tabulator</a>, and obtained information. Remember, <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id138ba838">OpenLink Software</a> is an <a href="http://dbpedia.org/resource/Entity" id="link-id1173e120">Entity</a> of Type: <a href="http://xmlns.com/foaf/0.1/Organization" id="link-id138b87b8">foaf:Organization</a>, on the burgeoning Linked Data Web :-)</p> <h3>Related</h3> <ul> <li> <a href="http://virtuoso.openlinksw.com/presentations/Creating_Deploying_Exploiting_Linked_Data2/Creating_Deploying_Exploiting_Linked_Data2_TimBL_v3.html" id="link-id163a0c88">Linked Data Planet Keynote</a> (RDFa based remix edition)</li> <li> <a href="http://semanticbusiness.blogspot.com/2008/09/report-on-cusp-global-review-of.html" id="link-id11471a40">On The Cusp: A Global Review of the Semantic Web Industry.</a> </li> </ul>
Transitivity and Graphs for SQL
http://www.openlinksw.com/blog/vdb/blog/?date=2008-09-08#1435
2008-09-08T09:41:24Z
2008-09-08T15:43:07-04:00
<div> <div style="display:none;">Transitivity and Graphs for SQL</div> <h2>Background</h2> <p>I have mentioned on a couple of prior occasions that basic graph operations ought to be integrated into the <a href="http://dbpedia.org/resource/SQL" id="link-id0xa1a18c58">SQL</a> query language.</p> <p>The history of databases is by and large about moving from specialized applications toward a generic platform. The introduction of the DBMS itself is the archetypal example. It is all about extracting the common features of applications and making these the features of a platform instead.</p> <p>It is now time to apply this principle to graph traversal.</p> <p>The rationale is that graph operations are somewhat tedious to write in a parallelize-able, latency-tolerant manner. Writing them as one would for memory-based <a href="http://dbpedia.org/resource/Data" id="link-id0xaf8c730">data</a> structures is easier but totally unscalable as soon as there is any latency involved, i.e., disk reads or messages between cluster peers.</p> <p>The ad-hoc nature and very large volume of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xae41ef0">RDF</a> data makes this a timely question. Up until now, the answer to this question has been to materialize any implied facts in RDF stores. If <i>a</i> was part of <i>b</i>, and <i>b</i> part of <i><a href="http://dbpedia.org/resource/C_(programming_language)" id="link-id0xac9d8790">c</a></i>, the implied fact that <i>a</i> is part of <i>c</i> would be inserted explicitly into the database as a pre-query step.</p> <p>This is simple and often efficient, but tends to have the downside that one makes a specialized warehouse for each new type of query. The activity becomes less ad-hoc.</p> <p>Also, this becomes next to impossible when the scale approaches web scale, or if some of the data is liable to be on-and-off included-into or excluded-from the set being analyzed. This is why with <a href="http://virtuoso.openlinksw.com" id="link-id0xb68f9d0">Virtuoso</a> we have tended to favor inference on demand ("backward chaining") and mapping of relational data into RDF without copying.</p> <p>The SQL world has taken steps towards dealing with recursion with the <code>WITH - UNION</code> construct which allows definition of recursive views. The idea there is to define, for example, a tree walk as a <code>UNION</code> of the data of the starting node plus the recursive walk of the starting node's immediate children.</p> <p>The main problem with this is that I do not very well see how a SQL optimizer could effectively rearrange queries involving <code>JOIN</code>s between such recursive views. This model of recursion seems to lose SQL's non-procedural nature. One can no longer easily rearrange <code>JOIN</code>s based on what data is given and what is to be retrieved. If the recursion is written from root to leaf, it is not obvious how to do this from leaf to root. At any rate, queries written in this way are so complex to write, let alone optimize, that I decided to take another approach.</p> <p>Take a question like "list the parts of products of category <i>C</i> which have materials that are classified as toxic." Suppose that the product categories are a tree, the product parts are a tree, and the materials classification is a tree taxonomy where "toxic" has a multilevel substructure.</p> <p>Depending on the count of products and materials, the query can be evaluated as either going from products to parts to materials and then climbing up the materials tree to see if the material is toxic. Or one could do it in reverse, starting with the different toxic materials, looking up the parts containing these, going to the part tree to the product, and up the product hierarchy to see if the product is in the right category. One should be able to evaluate the identical query either way depending on what indices exist, what the cardinalities of the relations are, and so forth â regular cost based optimization.</p> <p>Especially with RDF, there are many problems of this type. In regular SQL, it is a long-standing cultural practice to flatten hierarchies, but this is not the case with RDF.</p> <p>In Virtuoso, we see <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xb3bdcc0">SPARQL</a> as reducing to SQL. Any RDF-oriented database-engine or query-optimization feature is accessed via SQL. Thus, if we address run-time-recursion in the Virtuoso query engine, this becomes, <i>ipso facto</i>, an SQL feature. Besides, we remember that SQL is a much more mature and expressive language than the current SPARQL recommendation.</p> <h2> SQL and Transitivity </h2> <p>We will here look at some simple social network queries. A later article will show how to do more general graph operations. We extend the SQL derived table construct, i.e., <code>SELECT</code> in another <code>SELECT</code>'s <code>FROM</code> clause, with a <code>TRANSITIVE</code> clause.</p> <p>Consider the data:</p> <blockquote> <pre><code>CREATE TABLE "knows" ("p1" INT, "p2" INT, PRIMARY KEY ("p1", "p2") ); ALTER INDEX "knows" ON "knows" PARTITION ("p1" INT); CREATE INDEX "knows2" ON "knows" ("p2", "p1") PARTITION ("p2" INT); </code> </pre></blockquote> <p>We represent a social network with the many-to-many relation "knows". The persons are identified by integers.</p> <blockquote> <pre><code>INSERT INTO "knows" VALUES (1, 2); INSERT INTO "knows" VALUES (1, 3); INSERT INTO "knows" VALUES (2, 4);</code> </pre> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2" FROM "knows" ) "k" WHERE "k"."p1" = 1;</code></pre></blockquote> <p>We obtain the result:</p> <blockquote> <table width="100"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> </tr> <tr> <td align="center">1</td> <td align="center">3</td> </tr> <tr> <td align="center">1</td> <td align="center">2</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> </tr> </table> </blockquote> <p>The operation is reversible:</p> <blockquote> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2" FROM "knows" ) "k" WHERE "k"."p2" = 4; </code> </pre> <table width="100"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> </tr> <tr> <td align="center">2</td> <td align="center">4</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> </tr> </table> </blockquote> <p>Since now we give <i>p2</i>, we traverse from <i>p2</i> towards <i>p1</i>. The result set states that 4 is known by 2 and 2 is known by 1.</p> <p>To see what would happen if <i>x</i> knowing <i>y</i> also meant <i>y</i> knowing <i>x</i>, one could write:</p> <blockquote> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2" FROM (SELECT "p1", "p2" FROM "knows" UNION ALL SELECT "p2", "p1" FROM "knows" ) "k2" ) "k" WHERE "k"."p2" = 4;</code> </pre> <table width="100"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> </tr> <tr> <td align="center">2</td> <td align="center">4</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> </tr> <tr> <td align="center">3</td> <td align="center">4</td> </tr> </table> </blockquote> <p>Now, since we know that 1 and 4 are related, we can ask how they are related.</p> <blockquote> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2", T_STEP (1) AS "via", T_STEP ('step_no') AS "step", T_STEP ('path_id') AS "path" FROM "knows" ) "k" WHERE "p1" = 1 AND "p2" = 4;</code> </pre> <table width="250"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> <th align="center" width="50">via</th> <th align="center" width="50">step</th> <th align="center" width="50">path</th> </tr> <tr> <td align="center">1</td> <td align="center">4</td> <td align="center">1</td> <td align="center">0</td> <td align="center">0</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> <td align="center">2</td> <td align="center">1</td> <td align="center">0</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> <td align="center">4</td> <td align="center">2</td> <td align="center">0</td> </tr> </table> </blockquote> <p>The two first columns are the ends of the path. The next column is the person that is a step on the path. The next one is the number of the step, counting from 0, so that the end of the path that corresponds to the end condition on the column designated as input, i.e., <i>p1</i>, has number 0. Since there can be multiple solutions, the last column is a sequence number allowing distinguishing multiple alternative paths from each other.</p> <p>For LinkedIn users, the friends ordered by distance and descending friend count query, which is at the basis of most LinkedIn search result views can be written as: </p> <blockquote> <pre><code>SELECT p2, dist, (SELECT COUNT (*) FROM "knows" "c" WHERE "c"."p1" = "k"."p2" ) FROM (SELECT TRANSITIVE t_in (1) t_out (2) t_distinct "p1", "p2", t_step ('step_no') AS "dist" FROM "knows" ) "k" WHERE "p1" = 1 ORDER BY "dist", 3 DESC;</code> </pre> <table width="150"> <tr> <th align="center" width="50">p2</th> <th align="center" width="50">dist</th> <th align="center" width="50">aggregate</th> </tr> <tr> <td align="center">2</td> <td align="center">1</td> <td align="center">1</td> </tr> <tr> <td align="center">3</td> <td align="center">1</td> <td align="center">0</td> </tr> <tr> <td align="center">4</td> <td align="center">2</td> <td align="center">0</td> </tr> </table> </blockquote> <h2>How?</h2> <p>The queries shown above work on Virtuoso v6. When running in cluster mode, several thousand graph traversal steps may be proceeding at the same time, meaning that all database access is parallelized and that the algorithm is internally latency-tolerant. By default, all results are produced in a deterministic order, permitting predictable slicing of result sets.</p> <p>Furthermore, for queries where both ends of a path are given, the optimizer may decide to attack the path from both ends simultaneously. So, supposing that every member of a social network has an average of 30 contacts, and we need to find a path between two users that are no more than 6 steps apart, we begin at both ends, expanding each up to 3 levels, and we stop when we find the first intersection. Thus, we reach 2 * 30^3 = 54,000 nodes, and not 30^6 = 729,000,000 nodes.</p> <p>Writing a generic database driven graph traversal framework on the application side, say in Java over <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0xa8a9ef8">JDBC</a>, would easily be over a thousand lines. This is much more work than can be justified just for a one-off, ad-hoc query. Besides, the traversal order in such a case could not be optimized by the DBMS.</p> <h2>Next</h2> <p>In a future <a href="http://dbpedia.org/resource/Blog" id="link-id0xb526a40">blog</a> post I will show how this feature can be used for common graph tasks like critical path, itinerary planning, traveling salesman, the 8 queens chess problem, etc. There are lots of switches for controlling different parameters of the traversal. This is just the beginning. I will also give examples of the use of this in SPARQL.</p> </div>
Transitivity and Graphs for SQL
http://www.openlinksw.com/weblog/oerling/?date=2008-09-08#1433
2008-09-08T09:20:11Z
2008-09-08T15:43:04.000006-04:00
<h2>Background</h2> <p>I have mentioned on a couple of prior occasions that basic graph operations ought to be integrated into the <a href="http://dbpedia.org/resource/SQL" id="link-id0xb1fe830">SQL</a> query language.</p> <p>The history of databases is by and large about moving from specialized applications toward a generic platform. The introduction of the DBMS itself is the archetypal example. It is all about extracting the common features of applications and making these the features of a platform instead.</p> <p>It is now time to apply this principle to graph traversal.</p> <p>The rationale is that graph operations are somewhat tedious to write in a parallelize-able, latency-tolerant manner. Writing them as one would for memory-based <a href="http://dbpedia.org/resource/Data" id="link-id0x1cb37218">data</a> structures is easier but totally unscalable as soon as there is any latency involved, i.e., disk reads or messages between cluster peers.</p> <p>The ad-hoc nature and very large volume of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1e1850a0">RDF</a> data makes this a timely question. Up until now, the answer to this question has been to materialize any implied facts in RDF stores. If <i>a</i> was part of <i>b</i>, and <i>b</i> part of <i><a href="http://dbpedia.org/resource/C_(programming_language)" id="link-id0xa1a08d38">c</a></i>, the implied fact that <i>a</i> is part of <i>c</i> would be inserted explicitly into the database as a pre-query step.</p> <p>This is simple and often efficient, but tends to have the downside that one makes a specialized warehouse for each new type of query. The activity becomes less ad-hoc.</p> <p>Also, this becomes next to impossible when the scale approaches web scale, or if some of the data is liable to be on-and-off included-into or excluded-from the set being analyzed. This is why with <a href="http://virtuoso.openlinksw.com" id="link-id0xa51bd10">Virtuoso</a> we have tended to favor inference on demand ("backward chaining") and mapping of relational data into RDF without copying.</p> <p>The SQL world has taken steps towards dealing with recursion with the <code>WITH - UNION</code> construct which allows definition of recursive views. The idea there is to define, for example, a tree walk as a <code>UNION</code> of the data of the starting node plus the recursive walk of the starting node's immediate children.</p> <p>The main problem with this is that I do not very well see how a SQL optimizer could effectively rearrange queries involving <code>JOIN</code>s between such recursive views. This model of recursion seems to lose SQL's non-procedural nature. One can no longer easily rearrange <code>JOIN</code>s based on what data is given and what is to be retrieved. If the recursion is written from root to leaf, it is not obvious how to do this from leaf to root. At any rate, queries written in this way are so complex to write, let alone optimize, that I decided to take another approach.</p> <p>Take a question like "list the parts of products of category <i>C</i> which have materials that are classified as toxic." Suppose that the product categories are a tree, the product parts are a tree, and the materials classification is a tree taxonomy where "toxic" has a multilevel substructure.</p> <p>Depending on the count of products and materials, the query can be evaluated as either going from products to parts to materials and then climbing up the materials tree to see if the material is toxic. Or one could do it in reverse, starting with the different toxic materials, looking up the parts containing these, going to the part tree to the product, and up the product hierarchy to see if the product is in the right category. One should be able to evaluate the identical query either way depending on what indices exist, what the cardinalities of the relations are, and so forth â regular cost based optimization.</p> <p>Especially with RDF, there are many problems of this type. In regular SQL, it is a long-standing cultural practice to flatten hierarchies, but this is not the case with RDF.</p> <p>In Virtuoso, we see <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xb4b3ce8">SPARQL</a> as reducing to SQL. Any RDF-oriented database-engine or query-optimization feature is accessed via SQL. Thus, if we address run-time-recursion in the Virtuoso query engine, this becomes, <i>ipso facto</i>, an SQL feature. Besides, we remember that SQL is a much more mature and expressive language than the current SPARQL recommendation.</p> <h2> SQL and Transitivity </h2> <p>We will here look at some simple social network queries. A later article will show how to do more general graph operations. We extend the SQL derived table construct, i.e., <code>SELECT</code> in another <code>SELECT</code>'s <code>FROM</code> clause, with a <code>TRANSITIVE</code> clause.</p> <p>Consider the data:</p> <blockquote> <pre><code>CREATE TABLE "knows" ("p1" INT, "p2" INT, PRIMARY KEY ("p1", "p2") ); ALTER INDEX "knows" ON "knows" PARTITION ("p1" INT); CREATE INDEX "knows2" ON "knows" ("p2", "p1") PARTITION ("p2" INT); </code> </pre></blockquote> <p>We represent a social network with the many-to-many relation "knows". The persons are identified by integers.</p> <blockquote> <pre><code>INSERT INTO "knows" VALUES (1, 2); INSERT INTO "knows" VALUES (1, 3); INSERT INTO "knows" VALUES (2, 4);</code> </pre> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2" FROM "knows" ) "k" WHERE "k"."p1" = 1;</code></pre></blockquote> <p>We obtain the result:</p> <blockquote> <table width="100"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> </tr> <tr> <td align="center">1</td> <td align="center">3</td> </tr> <tr> <td align="center">1</td> <td align="center">2</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> </tr> </table> </blockquote> <p>The operation is reversible:</p> <blockquote> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2" FROM "knows" ) "k" WHERE "k"."p2" = 4; </code> </pre> <table width="100"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> </tr> <tr> <td align="center">2</td> <td align="center">4</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> </tr> </table> </blockquote> <p>Since now we give <i>p2</i>, we traverse from <i>p2</i> towards <i>p1</i>. The result set states that 4 is known by 2 and 2 is known by 1.</p> <p>To see what would happen if <i>x</i> knowing <i>y</i> also meant <i>y</i> knowing <i>x</i>, one could write:</p> <blockquote> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2" FROM (SELECT "p1", "p2" FROM "knows" UNION ALL SELECT "p2", "p1" FROM "knows" ) "k2" ) "k" WHERE "k"."p2" = 4;</code> </pre> <table width="100"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> </tr> <tr> <td align="center">2</td> <td align="center">4</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> </tr> <tr> <td align="center">3</td> <td align="center">4</td> </tr> </table> </blockquote> <p>Now, since we know that 1 and 4 are related, we can ask how they are related.</p> <blockquote> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2", T_STEP (1) AS "via", T_STEP ('step_no') AS "step", T_STEP ('path_id') AS "path" FROM "knows" ) "k" WHERE "p1" = 1 AND "p2" = 4;</code> </pre> <table width="250"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> <th align="center" width="50">via</th> <th align="center" width="50">step</th> <th align="center" width="50">path</th> </tr> <tr> <td align="center">1</td> <td align="center">4</td> <td align="center">1</td> <td align="center">0</td> <td align="center">0</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> <td align="center">2</td> <td align="center">1</td> <td align="center">0</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> <td align="center">4</td> <td align="center">2</td> <td align="center">0</td> </tr> </table> </blockquote> <p>The two first columns are the ends of the path. The next column is the person that is a step on the path. The next one is the number of the step, counting from 0, so that the end of the path that corresponds to the end condition on the column designated as input, i.e., <i>p1</i>, has number 0. Since there can be multiple solutions, the last column is a sequence number allowing distinguishing multiple alternative paths from each other.</p> <p>For LinkedIn users, the friends ordered by distance and descending friend count query, which is at the basis of most LinkedIn search result views can be written as: </p> <blockquote> <pre><code>SELECT p2, dist, (SELECT COUNT (*) FROM "knows" "c" WHERE "c"."p1" = "k"."p2" ) FROM (SELECT TRANSITIVE t_in (1) t_out (2) t_distinct "p1", "p2", t_step ('step_no') AS "dist" FROM "knows" ) "k" WHERE "p1" = 1 ORDER BY "dist", 3 DESC;</code> </pre> <table width="150"> <tr> <th align="center" width="50">p2</th> <th align="center" width="50">dist</th> <th align="center" width="50">aggregate</th> </tr> <tr> <td align="center">2</td> <td align="center">1</td> <td align="center">1</td> </tr> <tr> <td align="center">3</td> <td align="center">1</td> <td align="center">0</td> </tr> <tr> <td align="center">4</td> <td align="center">2</td> <td align="center">0</td> </tr> </table> </blockquote> <h2>How?</h2> <p>The queries shown above work on Virtuoso v6. When running in cluster mode, several thousand graph traversal steps may be proceeding at the same time, meaning that all database access is parallelized and that the algorithm is internally latency-tolerant. By default, all results are produced in a deterministic order, permitting predictable slicing of result sets.</p> <p>Furthermore, for queries where both ends of a path are given, the optimizer may decide to attack the path from both ends simultaneously. So, supposing that every member of a social network has an average of 30 contacts, and we need to find a path between two users that are no more than 6 steps apart, we begin at both ends, expanding each up to 3 levels, and we stop when we find the first intersection. Thus, we reach 2 * 30^3 = 54,000 nodes, and not 30^6 = 729,000,000 nodes.</p> <p>Writing a generic database driven graph traversal framework on the application side, say in Java over <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0xb595050">JDBC</a>, would easily be over a thousand lines. This is much more work than can be justified just for a one-off, ad-hoc query. Besides, the traversal order in such a case could not be optimized by the DBMS.</p> <h2>Next</h2> <p>In a future <a href="http://dbpedia.org/resource/Blog" id="link-id0x1e4d4f18">blog</a> post I will show how this feature can be used for common graph tasks like critical path, itinerary planning, traveling salesman, the 8 queens chess problem, etc. There are lots of switches for controlling different parameters of the traversal. This is just the beginning. I will also give examples of the use of this in SPARQL.</p>
Crunchbase & Semantic Web Interview (Remix - Update 1)
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2008-08-27#1424
2008-08-27T18:16:37Z
2008-08-27T20:35:15-04:00
<p>After reading <a href="http://blog.crunchbase.com/2008/08/26/building-a-semantic-web-interview-with-benjamin-nowack/" id="link-id16b8e0e0">Bengee's interview with CrunchBase</a>, I decided to knock up a quick interview remix as part of my usual attempt to add to the developing discourse.</p> <blockquote> <cite><a href="http://www.crunchbase.com/" id="link-id17c8e7b8">CrunchBase</a>: When we released the <a href="http://www.crunchbase.com/help/api" id="link-id16681f68">CrunchBase API</a>, you were one of the first developers to step up and quickly released a <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com's%20BLOG%20%5B127%5D/1395" id="link-id1016d5f0">CrunchBase Sponger Cartridge</a>. Can you explain what a CrunchBase Sponger Cartridge is?</cite> </blockquote> <blockquote> <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id13243300">Me</a>: A Sponger Cartridge is a <a href="http://dbpedia.org/resource/Data">data</a> access driver for <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> Resources that plugs into our <a href="http://virtuoso.openlinksw.com" id="link-id17042f08">Virtuoso</a> <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id1399b588">Universal Server</a> (DBMS and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id137fd188">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id100b23d8">Web</a> Server combo amongst other things). It uses the internal structure of a resource and/or a web service associated with a resource, to materialize an RDF based <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id10418750">Linked Data graph</a> that essentially describes the resource via its properties (Attributes & Relationships). </blockquote> <br /> <img src="http://virtuoso.openlinksw.com/presentations/Creating_Deploying_Exploiting_Linked_Data2/images/ldp4.png" /> <br /> <br /> <br /> <blockquote> <cite>CrunchBase: And what inspired you to create it?</cite> </blockquote> <blockquote> <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id12fa60c0">Me</a>: Bengee built a new space with your data, and we've built a space on the fly from your data which still resides in your domain. Either solution extols the virtues of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id101a8d28">Linked Data</a> i.e. the ability to explore relationships across data items with high degrees of serendipity (also colloquially known as: following-your-nose pattern in <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id14a3ff30">Semantic Web</a> circles).</blockquote> <blockquote> <a href="http://cb.semsol.org/" id="link-id182a0170">Bengee</a> posted a notice to the <a href="http://esw.w3.org/topic/SweoIG/TaskForces/CommunityProjects/LinkingOpenData" id="link-id131e8d10">Linking Open Data Community</a>'s public <a href="http://lists.w3.org/Archives/Public/public-lod/2008Jul/0110.html" id="link-id11dd0720">mailing list announcing his effort</a>. Bearing in mind the fact that we've been using <a href="http://www.openlinksw.com/blog/~kidehen/?id=1144" id="link-id117cf6e8">middleware to mesh the realms of Web 2.0 and the Linked Data Web</a> for a while, it was a no-brainer to knock something up based on the conceptual similarities between <a href="http://wikicompany.org/wiki/Main_Page" id="link-id13b87a68">Wikicompany</a> and CrunchBase. In a sense, a quadrant of orthogonality is what immediately came to mind re. Wikicompany, CrunchBase, Bengee's RDFization efforts, and ours.</blockquote> <blockquote>Bengee created an RDF based <a href="http://dbpedia.org/resource/Linked_Data" id="link-id133c8fc8">Linked Data</a> warehouse based on the data exposed by your API, which is exposed via the <a href="http://cb.semsol.org/" id="link-id1826f928">Semantic CrunchBase</a> <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id102d8890">data space</a>. In our case we've taken the "RDFization on the fly" approach which produces a transient <a href="http://dbpedia.org/resource/Linked_Data" id="link-id16a0b8d0">Linked Data</a> View of the CrunchBase data exposed by your APIs. Our approach is in line with our world view: all resources on the Web are data sources, and the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1668e6c8">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id188e7da0">Web</a> is about incorporating HTTP into the naming scheme of these data sources so that the conventional <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id13490710">URL</a> based hyperlinking mechanism can be used to access a structured description of a resource, which is then transmitted using a range negotiable representation formats. In addition, based on the fact that we house and publish a lot of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id169aa568">Linked Data</a> on the Web (e.g. <a href="http://dbpedia.org/resource/DBpedia" id="link-id10af10e8">DBpedia</a>, <a href="http://www.pingthesemanticweb.com/about/" id="link-id10a2b710">PingTheSemanticWeb</a>, and others), we've also automatically meshed Crunchbase data with related data in <a href="http://dbpedia.org/resource/DBpedia" id="link-id1403cd40">DBpedia</a> and Wikicompany data.</blockquote> <br /> <blockquote> <cite>CrunchBase: Do you know of any apps that are using CrunchBase Cartridge to enhance their functionality?</cite> </blockquote> <blockquote> <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id177d24c8">Me</a>: Yes, the <a href="http://ode.openlinksw.com" id="link-id10725ca0">OpenLink Data Explorer</a> which provides CrunchBase site visitors with the option to explore the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id17dedea8">Linked Data</a> in the CrunchBase <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id13f02a00">data space</a>. It also allows them to "Mesh" (rather than "Mash") CrunchBase data with other <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11fb3ba0">Linked Data</a> sources on the Web without writing a single line of code. </blockquote> <br /> <blockquote> <cite>CrunchBase: You have been immersed in the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id12e18a00">Semantic Web</a> movement for a while now. How did you first get interested in the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id15132110">Semantic Web</a>?</cite> </blockquote> <blockquote> <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id0xddaa9c8">Me</a>: We saw the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id188b3330">Semantic Web</a> as a vehicle for standardizing conceptual views of heterogeneous data sources via <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id10350978">context</a> lenses (URIs). In 1998 as part of our strategy to expand our business beyond the development and deployment of <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id171d6798">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id138120a0">JDBC</a>, and OLE-DB data providers, we decided to build a <a href="http://dbpedia.org/resource/Virtual_Database" id="link-id13ea6618">Virtual Database</a> Engine (see: <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSHistory" id="link-id11a4fa30">Virtuoso History</a>), and in doing so we sought a standards based mechanism for the conceptual output of the <a href="http://dbpedia.org/resource/Federated_database_system" id="link-id101a1248">data virtualization</a> effort. As of the time of the <a href="http://www.w3.org/DesignIssues/Semantic.html" id="link-id18882cf8">seminal unveiling of the Semantic Web in 1998</a> we were clear about two things, in relation to the effects of the Web and <a href="http://dbpedia.org/resource/Internet" id="link-id12fa2c58">Internet</a> data management infrastructure inflections: 1) Existing DBMS technology had reached it limits 2) Web Servers would ultimately hit their functional limits. These fundamental realities compelled us to develop <a href="http://virtuoso.openlinksw.com" id="link-id102b09a0">Virtuoso</a> with an eye to leveraging the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id11984d98">Semantic Web</a> as a vehicle from completing its technical roadmap.</blockquote> <br /> <blockquote> <cite>CrunchBase: Can you put into laymanâs terms exactly what RDF and <a href="http://dbpedia.org/resource/SPARQL" id="link-id1066dcf0">SPARQL</a> are and why they are important? Do they only matter for developers or will they extend past developers at some point and be used by website visitors as well?</cite> </blockquote> <blockquote>Me: RDF (Resource Description Framework) is a Graph based Data Model that facilitates resource description using the <a href="http://www.eslincanada.com/englishlesson2.html" id="link-id178b94a8">Subject, Predicate, and Object principle</a>. Associated with the core data model, as part of the overall framework, are a number of markup languages for expressing your descriptions (just as you express presentation markup semantics in HTML or document structure semantics in XML) that include: <a href="http://dbpedia.org/resource/RDFa" id="link-id188db0a8">RDFa</a> (simple extension of HTML markup for embedding descriptions of things in a page), N3 (a human friendly markup for describing resources), RDF/XML (a machine friendly markup for describing resources).</blockquote> <blockquote> <a href="http://dbpedia.org/resource/SPARQL" id="link-id188c2030">SPARQL</a> is the query language associated with the RDF Data Model, just as <a href="http://dbpedia.org/resource/SQL" id="link-id13f0ffe0">SQL</a> is a query language associated with the Relational Database Model. Thus, when you have RDF based structured and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id166874d0">linked data</a> on the Web, you can query against Web using <a href="http://dbpedia.org/resource/SPARQL" id="link-id1016cc98">SPARQL</a> just as you would against an <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id101c9708">Oracle</a>/<a href="http://dbpedia.org/resource/SQL" id="link-id11cb0b18">SQL</a> Server/<a href="http://dbpedia.org/resource/IBM_DB2" id="link-id10760ec0">DB2</a>/<a href="http://dbpedia.org/resource/IBM_Informix" id="link-id1066c8c0">Informix</a>/<a href="http://dbpedia.org/resource/Ingres" id="link-id18894f40">Ingres</a>/<a href="http://dbpedia.org/resource/MySQL" id="link-iddc9ebb0">MySQL</a>/etc.. DBMS using <a href="http://dbpedia.org/resource/SQL" id="link-id1030d120">SQL</a>. That's it in a nutshell.</blockquote> <br /> <blockquote> <cite>CrunchBase: On your website you wrote that âRDF and <a href="http://dbpedia.org/resource/SPARQL" id="link-id168e9ad0">SPARQL</a> as productivity boosters in everyday web developmentâ. Can you elaborate on why you believe that to be true?</cite> </blockquote> <blockquote>Me: I think the ability to discern a formal description of anything via its discrete properties is of immense value re. productivity, especially when the capability in question results in a graph of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x179f6328">Linked Data</a> that isn't confined to a specific host operating system, database engine, application or service, programming language, or development framework. RDF <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> is about infrastructure for the true materialization of the "<a href="http://dbpedia.org/resource/Information" id="link-id13e475b8">Information</a> at Your Fingertips" vision of yore. Even though it's taken the emergence of RDF Linked Data to make the aforementioned vision tractable, the comprehension of the vision's intrinsic value have been clear for a very long time. Most organizations and/or individuals are quite familiar with the adage: <a href="http://dbpedia.org/resource/Knowledge" id="link-id13e38a30">Knowledge</a> is Power, well there isn't any <a href="http://dbpedia.org/resource/Knowledge" id="link-id188b7348">knowledge</a> without accessible <a href="http://dbpedia.org/resource/Information" id="link-id140415d0">Information</a>, and there isn't any accessible <a href="http://dbpedia.org/resource/Information" id="link-id11a976e8">Information</a> without accessible Data. The Web has always be grounded in accessibility to data (albeit via compound container documents called Web Pages).</blockquote> <blockquote>Bottom line, RDF based Linked Data is about Open <a href="http://dbpedia.org/resource/Reference_(computer_science)" id="link-id1206bfb8">Data access by reference</a> using URIs (HTTP based <a href="http://dbpedia.org/resource/Entity" id="link-idfaa6ce0">Entity</a> IDs / Data Object IDs / Data Source Names), and as I said earlier, the intrinsic value is pretty obvious bearing in mind the costs associated with integrating disparate and heterogeneous data sources -- across intranets, extranets, and the <a href="http://dbpedia.org/resource/Internet" id="link-id188ecc68">Internet</a>.</blockquote> <br /> <blockquote> <cite>CrunchBase: In his definition of Web 3.0, Nova Spivack proposes that the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id12e2d968">Semantic Web</a>, or Semanti<a href="http://dbpedia.org/resource/C_(programming_language)" id="link-id105744c0">c</a> Web technologies, will be force behind much of the innovation that will occur during Web 3.0. Do you agree with Nova Spivack? What role, if any, do you feel the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id13fa4218">Semantic Web</a> will play in Web 3.0?</cite> </blockquote> <blockquote>Me: I agree with Nova. But I see Web 3.0 as a phase within the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id188c9000">Semantic Web</a> innovation continuum. Web 3.0 exists because Web 2.0 exists. Both of these Web versions express usage and technology focus patterns. Web 2.0 is about the use of Open Source technologies to fashion Web Services that are ultimately used to drive proprietary Software as Service (SaaS) style solutions. Web 3.0 is about the use of "Smart Data Access" to fashion a new generation of Linked Data aware Web Services and solutions that exploit the federated nature of the Web to maximum effect; proprietary branding will simply be conveyed via quality of data (cleanliness, <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id188d2ef8">context</a> fidelity, and comprehension of privacy) exposed by URIs.</blockquote> <p>Here are some examples of the CrunchBase Linked Data <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id122756f8">Space</a>, as projected via our CruncBase Sponger Cartridge:</p> <ol> <li> <a href="http://demo.openlinksw.com/rdfbrowser2/?uri=http%3A%2F%2Fwww.crunchbase.com%2Fcompany%2Famazon" id="link-id13e0fd18">Amazon.com</a> </li> <li> <a href="http://demo.openlinksw.com/rdfbrowser2/?uri=http%3A%2F%2Fwww.crunchbase.com%2Fcompany%2Fmicrosoft" id="link-id13eef9e0">Microsoft</a> </li> <li> <a href="http://demo.openlinksw.com/rdfbrowser2/?uri=http%3A%2F%2Fwww.crunchbase.com%2Fcompany%2Fgoogle" id="link-id13fe47a0">Google</a> </li> <li> <a href="http://demo.openlinksw.com/rdfbrowser2/?uri=http%3A%2F%2Fwww.crunchbase.com%2Fcompany%2Fapple" id="link-id170c73b8">Apple</a> </li> </ol>
Configuring Virtuoso for Benchmarking
http://www.openlinksw.com/blog/vdb/blog/?date=2008-08-25#1419
2008-08-25T14:06:11Z
2008-08-25T15:29:06.000036-04:00
<div> <div style="display:none;">Configuring Virtuoso for Benchmarking</div> <p>I will here summarize what should be known about running benchmarks with <a href="http://virtuoso.openlinksw.com" id="link-id0xc152cf0">Virtuoso</a>.</p> <h2>Physical Memory</h2> <p>For 8G RAM, in the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set â</p> <blockquote> <code> [Parameters]<br /> ...<br /> NumberOfBuffers = 550000 </code> </blockquote> <p>For 16G RAM, double thisâ</p> <blockquote> <code> [Parameters]<br /> ...<br /> NumberOfBuffers = 1100000 </code> </blockquote> <h2>Transaction Isolation</h2> <p>For most cases, certainly all <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xb7ba270">RDF</a> cases, <i>Read Committed</i> should be the default transaction isolation. In the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set â</p> <blockquote> <code> [Parameters]<br /> ...<br /> DefaultIsolation = 2 </code> </blockquote> <h2>Multiuser Workload</h2> <p>If <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x1a40f308">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x1e003cf8">JDBC</a>, or similarly connected client applications are used, there must be more <code>ServerThreads</code> available than there will be client connections. In the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set â</p> <blockquote> <code> [Parameters]<br /> ...<br /> ServerThreads = 100 </code> </blockquote> <p>With web clients (unlike ODBC, JDBC, or similar clients), it may be justified to have fewer <code>ServerThreads</code> than there are concurrent clients. The <code>MaxKeepAlives</code> should be the maximum number of expected web clients. This can be more than the <code>ServerThreads</code> count. In the <code>[HTTPServer]</code> stanza of <code>virtuoso.ini</code>, set â</p> <blockquote> <code> [HTTPServer]<br /> ...<br /> ServerThreads = 100 <br /> MaxKeepAlives = 1000 <br /> KeepAliveTimeout = 10 </code> </blockquote> <p> <i><b>Note</b> â The <code>[HTTPServer] ServerThreads</code> are taken from the total pool made available by the <code>[Parameters] ServerThreads</code>. Thus, the <code>[Parameters] ServerThreads</code> should always be at least as large as (and is best set greater than) the <code>[HTTPServer] ServerThreads</code>, and if using the closed-source Commercial Version, should not exceed the licensed thread count.</i> </p> <h2>Disk Use</h2> <p>The basic rule is to use one stripe (file) per distinct physical device (not per file system), using no RAID. For example, one might stripe a database over 6 files (6 physical disks), with an initial size of 60000 pages (the files will grow as needed). </p> <p>For the above described example, in the <code>[Database]</code> stanza of <code>virtuoso.ini</code>, set â</p> <blockquote> <code> [Database]<br /> ...<br /> Striping = 1<br /> MaxCheckpointRemap = 2000000 </code> </blockquote> <p>â and in the <code>[Striping]</code> stanza, on one line per <code>SegmentName</code>, set â</p> <blockquote> <code> [Striping]<br /> ...<br /> Segment1 = 60000 , /virtdev/db/virt-seg1.db = q1 , /data1/db/virt-seg1-str2.db = q2 , /data2/db/virt-seg1-str3.db = q3 , /data3/db/virt-seg1-str4.db = q4 , /data4/db/virt-seg1-str5.db = q5 , /data5/db/virt-seg1-str6.db = q6</code> </blockquote> <p>As can be seen here, each file gets a background IO thread (the <code>= q<i>xxx</i></code> clause). It should be noted that all files on the same physical device should have the same <code>q<i>xxx</i></code> value. This is not directly relevant to the benchmarking scenario above, because we have only one file per device, and thus only one file per IO queue.</p> <h2> <a href="http://dbpedia.org/resource/SQL" id="link-id0xc8b97c0">SQL</a> Optimization</h2> <p>If queries have lots of joins but access little <a href="http://dbpedia.org/resource/Data" id="link-id0x193b2fa8">data</a>, as with the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1b283ca0">Berlin SPARQL Benchmark</a>, the SQL compiler must be told not to look for better plans if the best plan so far is quicker than the compilation time expended so far. Thus, in the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set â</p> <blockquote> <code> [Parameters]<br /> ...<br /> StopCompilerWhenXOverRunTime = 1 </code> </blockquote> </div>
Configuring Virtuoso for Benchmarking
http://www.openlinksw.com/weblog/oerling/?date=2008-08-25#1418
2008-08-25T14:05:46Z
2008-08-25T15:29:04-04:00
<p>I will here summarize what should be known about running benchmarks with <a href="http://virtuoso.openlinksw.com" id="link-id0xc53af18">Virtuoso</a>.</p> <h2>Physical Memory</h2> <p>For 8G RAM, in the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set â</p> <blockquote> <code> [Parameters]<br /> ...<br /> NumberOfBuffers = 550000 </code> </blockquote> <p>For 16G RAM, double thisâ</p> <blockquote> <code> [Parameters]<br /> ...<br /> NumberOfBuffers = 1100000 </code> </blockquote> <h2>Transaction Isolation</h2> <p>For most cases, certainly all <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xc2f07a0">RDF</a> cases, <i>Read Committed</i> should be the default transaction isolation. In the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set â</p> <blockquote> <code> [Parameters]<br /> ...<br /> DefaultIsolation = 2 </code> </blockquote> <h2>Multiuser Workload</h2> <p>If <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0xc1c7178">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0xd16fb40">JDBC</a>, or similarly connected client applications are used, there must be more <code>ServerThreads</code> available than there will be client connections. In the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set â</p> <blockquote> <code> [Parameters]<br /> ...<br /> ServerThreads = 100 </code> </blockquote> <p>With web clients (unlike ODBC, JDBC, or similar clients), it may be justified to have fewer <code>ServerThreads</code> than there are concurrent clients. The <code>MaxKeepAlives</code> should be the maximum number of expected web clients. This can be more than the <code>ServerThreads</code> count. In the <code>[HTTPServer]</code> stanza of <code>virtuoso.ini</code>, set â</p> <blockquote> <code> [HTTPServer]<br /> ...<br /> ServerThreads = 100 <br /> MaxKeepAlives = 1000 <br /> KeepAliveTimeout = 10 </code> </blockquote> <p> <i><b>Note</b> â The <code>[HTTPServer] ServerThreads</code> are taken from the total pool made available by the <code>[Parameters] ServerThreads</code>. Thus, the <code>[Parameters] ServerThreads</code> should always be at least as large as (and is best set greater than) the <code>[HTTPServer] ServerThreads</code>, and if using the closed-source Commercial Version, should not exceed the licensed thread count.</i> </p> <h2>Disk Use</h2> <p>The basic rule is to use one stripe (file) per distinct physical device (not per file system), using no RAID. For example, one might stripe a database over 6 files (6 physical disks), with an initial size of 60000 pages (the files will grow as needed). </p> <p>For the above described example, in the <code>[Database]</code> stanza of <code>virtuoso.ini</code>, set â</p> <blockquote> <code> [Database]<br /> ...<br /> Striping = 1<br /> MaxCheckpointRemap = 2000000 </code> </blockquote> <p>â and in the <code>[Striping]</code> stanza, on one line per <code>SegmentName</code>, set â</p> <blockquote> <code> [Striping]<br /> ...<br /> Segment1 = 60000 , /virtdev/db/virt-seg1.db = q1 , /data1/db/virt-seg1-str2.db = q2 , /data2/db/virt-seg1-str3.db = q3 , /data3/db/virt-seg1-str4.db = q4 , /data4/db/virt-seg1-str5.db = q5 , /data5/db/virt-seg1-str6.db = q6</code> </blockquote> <p>As can be seen here, each file gets a background IO thread (the <code>= q<i>xxx</i></code> clause). It should be noted that all files on the same physical device should have the same <code>q<i>xxx</i></code> value. This is not directly relevant to the benchmarking scenario above, because we have only one file per device, and thus only one file per IO queue.</p> <h2> <a href="http://dbpedia.org/resource/SQL" id="link-id0xc9fa298">SQL</a> Optimization</h2> <p>If queries have lots of joins but access little <a href="http://dbpedia.org/resource/Data" id="link-id0xb4e0aa0">data</a>, as with the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0xb2de990">Berlin SPARQL Benchmark</a>, the SQL compiler must be told not to look for better plans if the best plan so far is quicker than the compilation time expended so far. Thus, in the <code>[Parameters]</code> stanza of <code>virtuoso.ini</code>, set â</p> <blockquote> <code> [Parameters]<br /> ...<br /> StopCompilerWhenXOverRunTime = 1 </code> </blockquote>
Response to: Whole Data Post (Update 3)
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2008-08-15#1413
2008-08-15T13:06:12Z
2008-08-15T18:31:48-04:00
<p>This post is in response to <a href="http://www.furia.com" id="link-id107907b8">Glenn McDonald</a>'s post titled: <a href="http://www.furia.com/page.cgi?type=log&id=308" id="link-id13dcf2d0">Whole Data</a>, where he highlights a number of issues relating to "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id1016c1f0">Semantic Web</a>" marketing communications and overall messaging, from his perspective.</p> <p> By coincidence, Glenn and I presented at this month's Cambridge <a href="http://dbpedia.org/resource/Semantic_Web" id="link-idd526f48">Semantic Web</a> Gathering.</p> <p>I've provided a dump of Glenn's issues and my responses below:</p> <h3>Issue - RDF</h3> <ul> <li>Ingenious <a href="http://dbpedia.org/resource/Data">data</a> decomposition idea, but: </li> <li>too low-level; the assembly language of data, where we need Java or <a href="http://dbpedia.org/resource/Ruby_programming_language" id="link-id103f3dd0">Ruby</a> </li> <li>"resource" is not the issue; there's no such thing as "metadata", it's all data; "meta" is a perspective </li> <li>lists need to be effortless, not painful and obscure </li> <li>nodes need to be represented, not just implied; they need types and literals in a more pervasive, integrated way. </li> </ul> <h4>Response:</h4> <p>RDF is a Graph based Data Model it stands for Resource Description Framework. The Metadata data angle comes from it's <a href="http://dbpedia.org/resource/Meta_Content_Framework" id="link-id1690df60">Meta Content Framework (MCF)</a> origins. You can express and serialize data based on the RDF Data Model using: Turtle, N3, TriX, N-Triples, and RDF/XML.</p> <h3>Issue - <a href="http://dbpedia.org/resource/SPARQL" id="link-id10234b38">SPARQL</a> (and Freebase's MQL)</h3> <p>These are just appeasement: <br />- old query paradigm: fishing in dark water with superstitiously tied lures; only works well in carefully stocked lakes <br />- we don't ask questions by defining answer shapes and then hoping they're dredged up whole.</p> <h4>Response:</h4> <p> <a href="http://dbpedia.org/resource/SPARQL" id="link-id16e45e50">SPARQL</a>, <a href="http://www.freebase.com/view/freebase/api" id="link-id13e7d468">MQL</a>, and <a href="http://msdn.microsoft.com/en-us/library/bb387145.aspx" id="link-id1516fbd8">Entity-SQL</a> are Graph Model oriented Query Languages. Query Languages always accompany Database Engines. <a href="http://dbpedia.org/resource/SQL" id="link-id13f8c100">SQL</a> is the Relational Model equivalent. </p> <h3>Issue - <a href="http://dbpedia.org/resource/Linked_Data" id="link-id171dee68">Linked Data</a> </h3> <p>Noble attempt to ground the abstract, but: <br />- <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id1576d5f8">URI</a> dereferencing/namespace/<a href="http://dbpedia.org/resource/Open_world_assumption" id="link-id15f50180">open-world</a> issues focus too much technical attention on cross-source cases where the human issues dwarf the technical ones anyway <br />- <a href="http://dbpedia.org/resource/Friend_of_a_friend" id="link-id105df458">FOAF</a> query over the people in this room? forget it. <br />- link asymmetry doesn't scale <br />- identity doesn't scale <br />- generating RDF from non-graph sources: more appeasement, right where the win from actually converting could be biggest! </p> <h4>Response:</h4> <p>Innovative use of HTTP to deliver "<a href="http://dbpedia.org/resource/Reference_%28computer_science%29" id="link-id13eeab20">Data Access by Reference</a>" to the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id13492610">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id105dfc10">Web</a>.</p> <p>When you have a Data Model, Database Engine, and Query Language, the next thing you need is a Data Access mechanism that provides "<a href="http://dbpedia.org/resource/Reference_(computer_science)" id="link-id100ef2c0">Data Access by Reference</a>". <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id16692e88">ODBC</a> and <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id1699b970">JDBC</a> (amongst others) provide "<a href="http://dbpedia.org/resource/Reference_(computer_science)" id="link-id16034b48">Data Access by Reference</a>" via Data Source Names. <a href="http://dbpedia.org/resource/Linked_Data" id="link-id16690118">Linked Data</a> is about the same thing (URIs are Data Source Names) with the following differences:</p> <ul> <li>Naming is scoped to the <a href="http://dbpedia.org/resource/Entity" id="link-id1195dc48">entity</a> level rather than container level</li> <li>HTTP's use within the data source naming scheme expands the referencability of the Named <a href="http://dbpedia.org/resource/Entity" id="link-id10485760">Entity</a> Descriptions beyond traditional confines such as applications, operating systems, and database engines. </li> </ul> <h3> Issue - <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id104684d0">Giant Global Graph</a> </h3> <p>Hugely motivating and powerful idea, worthy of a superhero (Graphius!), but: <br />- giant and global parts are too hard, and starting global makes every problem harder <br />- local projects become unmanageable in global <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id12497088">context</a> (Cyc, Freebase data-modeling lists...). And my thus my plea, again. Forget "semantic" and "<a href="http://dbpedia.org/resource/World_Wide_Web">web</a>", let's fix the database tech first: <br />- node/arc data-model, path-based exploratory query-model <br />- data-graph applications built easily on top of this common model; building them has to be easy, because if it's hard, they'll be bad <br />- given good database tech, good web data-publishing tech will be trivial! <br />- given good tools for graphs, the problems of uniting them will be only as hard as they have to be.</p> <h4>Response:</h4> <p> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id144466d8">Giant Global Graph</a> is just another moniker for a "Web of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id15c2c738">Linked Data</a>" or "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id14e73520">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10aef200">Web</a>".</p> <p>Multi-Model Database technology that meshes the best of the Graph & Relational Models exist. In a nutshell, this is what <a href="http://virtuoso.openlinksw.com" id="link-id13492e10">Virtuoso</a> is all about and it's existed for a very long time :-)</p> <p> <a href="http://virtuoso.openlinksw.com" id="link-id105a4f58">Virtuoso</a> is also a Virtual DBMS engine (so you can see Heterogeneous Relational Data via Graph Model <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id15845110">Context</a> Lenses). Naturally, it is also a <a href="http://dbpedia.org/resource/Linked_Data" id="link-id109e2c78">Linked Data</a> Deployment platform (or <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1086d650">Linked Data</a> Sever). </p> <p>The issue isn't the "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id107f1ba8">Semantic Web</a>" moniker per se., it's about how <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0xba72818">Linked Data</a> (foundation layer of <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id101dbf50">Semantic Web</a>) gets introduced to users. As I said during the MIT Gathering: "The Web is experienced via Web Browsers primarily, so any enhancement to the Web must be exposed via traditional Web Browsers", which is why we've opted to simply add "View <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> Sources" to the existing set of common Browser options that includes:</p> <ol> <li>View page in rendered form (default)</li> <li>View page source (i.e., how you see the markup behind the page)</li> </ol> <p>By exposing the Linked Data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id15a04b70">Web</a> option as described above, you enable the Web user to knowingly transition from the traditional Rendered (X)HTML page view to the Linked Data View (i.e., structured data behind the page). This simple "User Interaction" tweak makes the notion of exploiting a Structured Web becomes somewhat clearer.</p> <p>The Linked Data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10a187d0">Web</a> isn't a panacea. It's just an addition to the existing Web that enrichens the things you can do with the Web. It's predominance, like any application feature, will be subject to the degrees to which it delivers tangible value or matrializes internal and external opportunity costs.</p> <p>Note: The Web isn't ubiquitous today becuase all it's users groked HTML Markup. It's ubquitity is a function of opportunity costs: there simply came a point in the Web boostrap when nobody could afford the opportunity costs associated with being off the Web. The same thing will play out with Linked Data and the broader <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id10a97330">Semantic Web</a> vision.</p> <b>Links:</b> <ol> <li> <a href="http://virtuoso.openlinksw.com/presentations/Creating_Deploying_Exploiting_Linked_Data2/Creating_Deploying_Exploiting_Linked_Data2_TimBL_v3.html(15)" id="link-id137fc560">Linked Data Journey part of my Linked Data Planet Presentation Remix</a>(from slides 15 to 22 - which include bits from <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id1048a968">TimBL</a>'s presentation)</li> <li> <a href="http://ode.openlinksw.com" id="link-id1667df98">OpenLink Data Explorer</a> </li> <li> <a href="http://ode.openlinksw.com/example.html" id="link-id137ee860">OpenLink Data Explorer Screenshots and examples</a>.</li> </ol>
BSBM With Triples and Mapped Relational Data
http://www.openlinksw.com/blog/vdb/blog/?date=2008-08-06#1410
2008-08-06T19:41:50Z
2008-08-06T16:29:44.000003-04:00
<div> <div style="display:none;">BSBM With Triples and Mapped Relational Data</div> <p>The special contribution of the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id10039db0">Berlin SPARQL Benchmark</a> (<a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id106b2538">BSBM</a>) to the <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id101a75f8">RDF</a> world is to raise the question of doing OLTP with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xae54170">RDF</a>.</p> <p>Of course, here we immediately hit the question of comparisons with relational databases. To this effect, <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0x1e847b08">BSBM</a> also specifies a relational schema and can generate the <a href="http://dbpedia.org/resource/Data" id="link-id1206c378">data</a> as either triples or <a href="http://dbpedia.org/resource/SQL" id="link-id1667f040">SQL</a> inserts.</p> <p>The benchmark effectively simulates the case of exposing an existing <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id10a93518">RDBMS</a> as RDF. <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id13e46d80">OpenLink Software</a> calls this <i>RDF Views</i>. <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id12027578">Oracle</a> is beginning to call this <i>semantic covers</i>. The <a href="http://www.w3.org/2005/Incubator/rdb2rdf/" id="link-id161dc678">RDB2RDF XG</a>, a W3C incubator group, has been active in this area since Spring, 2008.</p> <h3>But why an OLTP workload with RDF to begin with?</h3> <p>We believe this is relevant because RDF promises to be the interoperability factor between potentially all of traditional IS. If <a href="http://dbpedia.org/resource/Data" id="link-id0x1e7119d8">data</a> is online for human consumption, it may be online via a <a href="http://dbpedia.org/resource/SPARQL" id="link-id106a8908">SPARQL</a> end-point as well. The economic justification will come from discoverability and from applications integrating multi-source structured data. Online shopping is a fine use case.</p> <p>Warehousing all the world's publishable data as RDF is not our first preference, nor would it be the publisher's. Considerations of duplicate infrastructure and maintenance are reason enough. Consequently, we need to show that mapping can outperform an RDF warehouse, which is what we'll do here.</p> <h3>What We Got </h3> <p>First, we found that <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1400" id="link-id150ea748">making the query plan took much too long</a> in proportion to the run time. With BSBM this is an issue because the queries have lots of joins but access relatively little data. So we made a faster compiler and along the way retouched the cost model a bit.</p> <p>But the really interesting part with BSBM is mapping relational data to RDF. For us, BSBM is a great way of showing that mapping can outperform even the best triple store. A relational row store is as good as unbeatable with the query mix. And when there is a clear mapping, there is no reason the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xae5aff0">SPARQL</a> could not be directly translated.</p> <p>If Chris Bizer et al launched the mapping ship, we will be the ones to pilot it to harbor!</p> <p>We filled two <a href="http://virtuoso.openlinksw.com" id="link-id12dbdc70">Virtuoso</a> instances with a BSBM200000 data set, for 100M triples. One was filled with physical triples; the other was filled with the equivalent relational data plus mapping to triples. Performance figures are given in "query mixes per hour". (An update or follow-on to this post will provide elapsed times for each test run.)</p> <p>With the unmodified benchmark we got:</p> <blockquote> <table> <tr> <td><i>Physical Triples:</i> </td> <td>Â Â </td> <td>1297 qmph</td> </tr> <tr> <td><i>Mapped Triples:</i> </td> <td>Â Â </td> <td><b>3144 qmph</b> </td> </tr> </table> </blockquote> <p>In both cases, most of the time was spent on Q6, which looks for products with one of three words in the label. We altered Q6 to use text index for the mapping, and altered the databases accordingly. (There is no such thing as an e-commerce site without a text index, so we are amply justified in making this change.)</p> <p>The following were measured on the second run of a 100 query mix series, single test driver, warm cache.</p> <blockquote> <table> <tr> <td><i>Physical Triples:</i> </td> <td>Â Â </td> <td> 5746 qmph</td> </tr> <tr> <td><i>Mapped Triples:</i> </td> <td>Â Â </td> <td> <b>7525 qmph</b> </td> </tr> </table> </blockquote> <p>We then ran the same with 4 concurrent instances of the test driver. The qmph here is 400 / the longest run time.</p> <blockquote> <table> <tr> <td><i>Physical Triples:</i> </td> <td>Â Â </td> <td> 19459 qmph</td> </tr> <tr> <td><i>Mapped Triples:</i> </td> <td>Â Â </td> <td> <b>24531 qmph</b> </td> </tr> </table> </blockquote> <p>The system used was 64-bit Linux, 2GHz dual-Xeon 5130 (8 cores) with 8G RAM. The concurrent throughputs are a little under 4 times the single thread throughput, which is normal for SMP due to memory contention. The numbers do not evidence significant overhead from thread synchronization.</p> <p>The query compilation represents about 1/3 of total server side CPU. In an actual online application of this type, queries would be parameterized, so the throughputs would be accordingly higher. We used the <code>StopCompilerWhenXOverRunTime = 1</code> option here to cut needless compiler overhead, the queries being straightforward enough.</p> <p>We also see that the advantage of mapping can be further increased by more compiler optimizations, so we expect in the end mapping will lead RDF warehousing by a factor of 4 or so.</p> <h3>Suggestions for BSBM</h3> <ul> <li> <p> <b>Reporting Rules.</b> The benchmark spec should specify a form for disclosure of test run data, TPC style. This includes things like configuration parameters and exact text of queries. There should be accepted variants of query text, as with the TPC.</p> </li> <li> <p> <b>Multiuser operation.</b> The test driver should get a stream number as parameter, so that each client makes a different query sequence. Also, disk performance in this type of benchmark can only be reasonably assessed with a naturally parallel multiuser workload.</p> </li> <li> <p> <b>Add business intelligence.</b> SPARQL has aggregates now, at least with <a href="http://jena.sourceforge.net/" id="link-id11a25ac0">Jena</a> and <a href="http://virtuoso.openlinksw.com" id="link-id0xb003180">Virtuoso</a>, so let's use these. The BSBM business intelligence metric should be a separate metric off the same data. Adding synthetic sales figures would make more interesting queries possible. For example, producing recommendations like "customers who bought this also bought xxx."</p> </li> <li> <p> <b>For the SPARQL community</b>, BSBM sends the message that one ought to support parameterized queries and stored procedures. This would be a <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id109e2448">SPARQL protocol</a> extension; the SPARUL syntax should also have a way of calling a procedure. Something like <code>select proc (??, ??)</code> would be enough, where <code>??</code> is a parameter marker, like <code>?</code> in <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id13febf48">ODBC</a>/<a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id120416a8">JDBC</a>.</p> </li> <li> <p> <b>Add transactions.</b>Especially if we are contrasting mapping vs. storing triples, having an update flow is relevant. In practice, this could be done by having the test driver send web service requests for order entry and the SUT could implement these as updates to the triples or a mapped relational store. This could use stored procedures or logic in an app server.</p> </li> </ul> <h3>Comments on Query Mix</h3> <p>The time of most queries is less than linear to the scale factor. Q6 is an exception if it is not implemented using a text index. Without the text index, Q6 will inevitably come to dominate query time as the scale is increased, and thus will make the benchmark less relevant at larger scales.</p> <h2>Next</h2> <p>We include the sources of our RDF view definitions and other material for running BSBM with our forthcoming Virtuoso Open Source 5.0.8 release. This also includes all the query optimization work done for BSBM. This will be available in the coming days.</p> </div>
BSBM With Triples and Mapped Relational Data
http://www.openlinksw.com/weblog/oerling/?date=2008-08-06#1409
2008-08-06T19:35:27Z
2008-08-06T16:29:40-04:00
<p>The special contribution of the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id10039db0">Berlin SPARQL Benchmark</a> (<a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id106b2538">BSBM</a>) to the <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id101a75f8">RDF</a> world is to raise the question of doing OLTP with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xb230eb0">RDF</a>.</p> <p>Of course, here we immediately hit the question of comparisons with relational databases. To this effect, <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/index.html" id="link-id0xa832da8">BSBM</a> also specifies a relational schema and can generate the <a href="http://dbpedia.org/resource/Data" id="link-id1206c378">data</a> as either triples or <a href="http://dbpedia.org/resource/SQL" id="link-id1667f040">SQL</a> inserts.</p> <p>The benchmark effectively simulates the case of exposing an existing <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id10a93518">RDBMS</a> as RDF. <a href="http://www.openlinksw.com/dataspace/organization/openlink#this" id="link-id13e46d80">OpenLink Software</a> calls this <i>RDF Views</i>. <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id12027578">Oracle</a> is beginning to call this <i>semantic covers</i>. The <a href="http://www.w3.org/2005/Incubator/rdb2rdf/" id="link-id161dc678">RDB2RDF XG</a>, a W3C incubator group, has been active in this area since Spring, 2008.</p> <h3>But why an OLTP workload with RDF to begin with?</h3> <p>We believe this is relevant because RDF promises to be the interoperability factor between potentially all of traditional IS. If <a href="http://dbpedia.org/resource/Data" id="link-id0xabe48a0">data</a> is online for human consumption, it may be online via a <a href="http://dbpedia.org/resource/SPARQL" id="link-id106a8908">SPARQL</a> end-point as well. The economic justification will come from discoverability and from applications integrating multi-source structured data. Online shopping is a fine use case.</p> <p>Warehousing all the world's publishable data as RDF is not our first preference, nor would it be the publisher's. Considerations of duplicate infrastructure and maintenance are reason enough. Consequently, we need to show that mapping can outperform an RDF warehouse, which is what we'll do here.</p> <h3>What We Got </h3> <p>First, we found that <a href="http://www.openlinksw.com/dataspace/oerling/weblog/Orri%20Erling%27s%20Blog/1400" id="link-id150ea748">making the query plan took much too long</a> in proportion to the run time. With BSBM this is an issue because the queries have lots of joins but access relatively little data. So we made a faster compiler and along the way retouched the cost model a bit.</p> <p>But the really interesting part with BSBM is mapping relational data to RDF. For us, BSBM is a great way of showing that mapping can outperform even the best triple store. A relational row store is as good as unbeatable with the query mix. And when there is a clear mapping, there is no reason the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x96bb5e0">SPARQL</a> could not be directly translated.</p> <p>If Chris Bizer et al launched the mapping ship, we will be the ones to pilot it to harbor!</p> <p>We filled two <a href="http://virtuoso.openlinksw.com" id="link-id12dbdc70">Virtuoso</a> instances with a BSBM200000 data set, for 100M triples. One was filled with physical triples; the other was filled with the equivalent relational data plus mapping to triples. Performance figures are given in "query mixes per hour". (An update or follow-on to this post will provide elapsed times for each test run.)</p> <p>With the unmodified benchmark we got:</p> <blockquote> <table> <tr> <td><i>Physical Triples:</i> </td> <td>Â Â </td> <td>1297 qmph</td> </tr> <tr> <td><i>Mapped Triples:</i> </td> <td>Â Â </td> <td><b>3144 qmph</b> </td> </tr> </table> </blockquote> <p>In both cases, most of the time was spent on Q6, which looks for products with one of three words in the label. We altered Q6 to use text index for the mapping, and altered the databases accordingly. (There is no such thing as an e-commerce site without a text index, so we are amply justified in making this change.)</p> <p>The following were measured on the second run of a 100 query mix series, single test driver, warm cache.</p> <blockquote> <table> <tr> <td><i>Physical Triples:</i> </td> <td>Â Â </td> <td> 5746 qmph</td> </tr> <tr> <td><i>Mapped Triples:</i> </td> <td>Â Â </td> <td> <b>7525 qmph</b> </td> </tr> </table> </blockquote> <p>We then ran the same with 4 concurrent instances of the test driver. The qmph here is 400 / the longest run time.</p> <blockquote> <table> <tr> <td><i>Physical Triples:</i> </td> <td>Â Â </td> <td> 19459 qmph</td> </tr> <tr> <td><i>Mapped Triples:</i> </td> <td>Â Â </td> <td> <b>24531 qmph</b> </td> </tr> </table> </blockquote> <p>The system used was 64-bit Linux, 2GHz dual-Xeon 5130 (8 cores) with 8G RAM. The concurrent throughputs are a little under 4 times the single thread throughput, which is normal for SMP due to memory contention. The numbers do not evidence significant overhead from thread synchronization.</p> <p>The query compilation represents about 1/3 of total server side CPU. In an actual online application of this type, queries would be parameterized, so the throughputs would be accordingly higher. We used the <code>StopCompilerWhenXOverRunTime = 1</code> option here to cut needless compiler overhead, the queries being straightforward enough.</p> <p>We also see that the advantage of mapping can be further increased by more compiler optimizations, so we expect in the end mapping will lead RDF warehousing by a factor of 4 or so.</p> <h3>Suggestions for BSBM</h3> <ul> <li> <p> <b>Reporting Rules.</b> The benchmark spec should specify a form for disclosure of test run data, TPC style. This includes things like configuration parameters and exact text of queries. There should be accepted variants of query text, as with the TPC.</p> </li> <li> <p> <b>Multiuser operation.</b> The test driver should get a stream number as parameter, so that each client makes a different query sequence. Also, disk performance in this type of benchmark can only be reasonably assessed with a naturally parallel multiuser workload.</p> </li> <li> <p> <b>Add business intelligence.</b> SPARQL has aggregates now, at least with <a href="http://jena.sourceforge.net/" id="link-id11a25ac0">Jena</a> and <a href="http://virtuoso.openlinksw.com" id="link-id0xa83f490">Virtuoso</a>, so let's use these. The BSBM business intelligence metric should be a separate metric off the same data. Adding synthetic sales figures would make more interesting queries possible. For example, producing recommendations like "customers who bought this also bought xxx."</p> </li> <li> <p> <b>For the SPARQL community</b>, BSBM sends the message that one ought to support parameterized queries and stored procedures. This would be a <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id109e2448">SPARQL protocol</a> extension; the SPARUL syntax should also have a way of calling a procedure. Something like <code>select proc (??, ??)</code> would be enough, where <code>??</code> is a parameter marker, like <code>?</code> in <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id13febf48">ODBC</a>/<a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id120416a8">JDBC</a>.</p> </li> <li> <p> <b>Add transactions.</b>Especially if we are contrasting mapping vs. storing triples, having an update flow is relevant. In practice, this could be done by having the test driver send web service requests for order entry and the SUT could implement these as updates to the triples or a mapped relational store. This could use stored procedures or logic in an app server.</p> </li> </ul> <h3>Comments on Query Mix</h3> <p>The time of most queries is less than linear to the scale factor. Q6 is an exception if it is not implemented using a text index. Without the text index, Q6 will inevitably come to dominate query time as the scale is increased, and thus will make the benchmark less relevant at larger scales.</p> <h2>Next</h2> <p>We include the sources of our RDF view definitions and other material for running BSBM with our forthcoming Virtuoso Open Source 5.0.8 release. This also includes all the query optimization work done for BSBM. This will be available in the coming days.</p>
CrunchBase gets hooked up with the Linked Data Web!
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2008-07-25#1395
2008-07-25T14:01:01Z
2008-07-29T21:43:27-04:00
<p>It's getting really hot in <a href="http://dbpedia.org/resource/Linked_Data" id="link-id15eea8f8">Linked Data</a> land! Two days ago <a href="http://bnode.org/about" id="link-id107e2f70">Benjamin Nowack</a> pinged the <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id11b93670">LOD</a> community about his <a href="http://lists.w3.org/Archives/Public/public-lod/2008Jul/0110.html" id="link-id1022a270">RDFization of Crunchbase</a> (sample (X)HTML view: http://cb.semsol.org/company/opera-software) courtesy of Crounchbase releasing an API. As you know, I've always equated <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> Service API to Database CLIs (<a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id16327528">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id1027f410">JDBC</a>, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id10683850">ADO</a>.<a href="http://dbpedia.org/resource/ADO.NET" id="link-id13beb9b8">NET</a> etc.) as both offer code level hooks into <a href="http://dbpedia.org/resource/Data">Data</a> Spaces.</p> <p></p> <p></p> <p>Naturally, we've decided to join the Crunchbase RDFization party, and have just completed a <a href="http://virtuoso.openlinksw.com" id="link-id10282208">Virtuoso</a> <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id10acf0f8">Sponger</a> Cartridge (an RDFizer) for Crouncbase. What we add in our particular cartridge is additional meshing with <a href="http://dbpedia.org/resource/DBpedia" id="link-id115e2a98">DBpedia</a> and Wikicompany <a href="http://dbpedia.org/resource/Linked_Data" id="link-id132f0568">Linked Data</a> Spaces, plus RDFizaton of the Crunchbase (X)HTML pages :-)</p> <p></p> <p></p> <p>As I've postulated for a while, <a href="http://dbpedia.org/resource/Linked_Data" id="link-id13304010">Linked Data</a> is about data "Meshing" and "Meshups". This isn't a buzzword play. I am pointing out an important distinction between "Mashups" and "Meshpus". Which goes as follows: "Mashups" are about code level joining devoid of structured modelling, hence the revelation of code as opposed to data when you look behind a "Mashup". "Meshups" on the other hand, are about joining disparate structured data sources across the Web. And when you look behind a "Meshup" you see structured data (preferably <a href="http://dbpedia.org/resource/Linked_Data" id="link-id143bdb68">Linked Data</a>) that enables further "Meshing".</p> <p></p> <p></p> <p>I truly believe that we are now inches away from critical mass re. <a href="http://dbpedia.org/resource/Linked_Data" id="link-id14829640">Linked Data</a>, and because we are dealing with data, the network-effect will be sky-high! I shudder to think about the state of the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1c41d150">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id133364e8">Web</a> in 12 months time. Yes, I am giving the explosion 12 months (or less). These are very exciting times.</p> <p></p> <p></p> <p>Demo Links:</p> <p></p> <p></p> <ul> <li> <a href="http://demo.openlinksw.com/ode/?uri%5B%5D=http%3A%2F%2Fcb.semsol.org%2Fcompany%2Fopera-software&" id="link-id12fe1dc8">Opera Software via Benjee's Linked Data Space for Cunchbase</a> </li> <li> <a href="http://demo.openlinksw.com/ode/?uri=http%3A%2F%2Fwww.crunchbase.com%2Fcompany%2Fopera-software" id="link-id10739a18">Opera Software via our Linked Data Space for Crunchbas</a> </li> </ul> <p></p> <p></p> <p>For best experience I encourage you to look at the <a href="https://addons.mozilla.org/en-US/firefox/addon/8062" id="link-id1499a0f8">OpenLink Data Explorer extension</a> for Firefox (2.x - 3.x). This enables you to go to Crunchbase (X)HTML pages (and other sites on the Web of course), and then simply use the "View | <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> Sources" main or <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id10051b50">context</a> menu sequence to unveil the Linked Data Sources associated with any Web Page.</p> <p></p> <p>Of course there is much more to come!</p>
Response to: Where's the Killer Semantic Web Application (Update #2)
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2008-06-26#1391
2008-06-26T18:28:14Z
2008-07-19T15:50:46-04:00
<p>As is often the case these days, it's much easier to drop a <a href="http://dbpedia.org/resource/Blog" id="link-id13519b98">blog</a> post than it is to make a simple comment in an "old media" style<a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id134e92c8"> data space </a>:-(</p> <blockquote> <p> <cite>My use of "old media" implies: a place that still seeks subscriber data (no OpenID etc..), for the umpteenth time, as the toll fee for discourse development and participation on the Web.</cite> </p> </blockquote> <p>Anyway, here is what I attempted to post as a comment to Dan Grigorovici's post titled: <a href="http://www.semanticweb.com/article.php/12160_3753806_2" id="link-id134dfb80">Where is the Semantic Web Killer App?</a> </p> <p>Dan,</p> <p>An intriguing post to say the least :-) </p> <p>"<a href="http://dbpedia.org/resource/Linked_Data" id="link-id134265c0">Linked Data</a>" and "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id133d7048">Semantic Web</a>" aren't synonymous, they are simply connected, infrastructure DNA-wise. You can have "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id1096cb70">Semantic Web</a>" style graphs (i.e RDF Data) and not have "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id133f0f48">Linked Data</a>" as per <a href="http://dbpedia.org/resource/Linked_Data" id="link-id134fc7c0">Linked Data</a> deployment tenets and best practices, a very important point.</p> <p>I've stated repeatedly, the "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id134f03e8">Linked Data</a>" emphasis has more to do with focusing on a point of crystallization within the larger "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id134104f0">Semantic Web</a>" vision, so here is a quick recap:</p> <h3>What is <a href="http://dbpedia.org/resource/Linked_Data" id="link-id133decd0">Linked Data</a>?</h3> <p>A term coined by <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id1340dd28">TimBL</a> that describes an application of HTTP to the time-tested process of "<a href="http://dbpedia.org/resource/Reference_(computer_science)">Data Access by Reference</a>". "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id10695c20">Linked Data</a>" adds vital items to the "Data Access by Reference" pattern that have been erstwhile unattainable:</p> <ul> <li> The use of a Data Source Naming scoped to Database / Data Container Records as opposed to Tables, Views, Stored Procedures, Databases, and other Record Container tuple collections. Example: in <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-idd9c8af8">ODBC</a> / <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id130b0df0">JDBC</a>, a Data Source Name's scope stops at the Table / View level. In the <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> realm you get an added layer of granularity due to record level name scope</li> <li>Incorporation of HTTP into the Data Source Naming scheme, which injects the expanse of the Web into the Data Access Range of the Data Source Name (i.e. a Named Record); so you can reference a record's description directly via HTTP which is simply a major deal (to put things mildly).</li> </ul> <p>So we have HTTP based URIs as the Data Sources Names for a "Linked Data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id1044a2d8">Web</a>" i.e a Web of inter-connected Data Source Names that de-emphasize the importance of their host containers (Compound Documents / <a href="http://dbpedia.org/resource/Information" id="link-id134e0d38">Information</a> Resources).</p> <p>The business case or value proposition of "Linked Data" is synonymous with the value proposition of data access technologies such as <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id13400500">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id134f0250">JDBC</a>. <a href="http://dbpedia.org/resource/ADO.NET" id="link-id10923840">ADO</a>.<a href="http://dbpedia.org/resource/ADO.NET" id="link-id134ef878">NET</a>, OLE-DB, XMLA, and others (enterprise or consumer) in relation to the Individual and Enterprise pursuit of agility; in a realm where data is growing exponentially, and the maximum processing time in a single day remains 24 hrs. Data Access & Data Integration are timeless challenges due to the following constants:</p> <ul> <li> Structured Data Schema Heterogeneity - we will always model the same things differently</li> <li>Dirtiness of Data within Structured Data Containers - we are error prone due to laziness / sloppiness, time constraints, and the inherent limitation of our DNA based CPUs when dealing with large volumes of data.</li> </ul> <p> Note: The line between the Enterprise & Individuals continue to blur by the second, this is something I covered during my <a href="http://virtuoso.openlinksw.com/presentations/Creating_Deploying_Exploiting_Linked_Data2/Creating_Deploying_Exploiting_Linked_Data2.html" id="link-id13479488">Linked Data Planet keynote</a>, which is like most things I put on the Web (via this <a href="http://dbpedia.org/resource/Blog" id="link-id130ac870">blog</a> <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id10923ba8">data space</a>), is a live and practical demonstration of the virtues of Linked Data courtesy of <a href="http://dbpedia.org/resource/RDFa" id="link-id133fd270">RDFa</a>, the <a href="http://bibliontology.com/" id="link-id134248d8">Bibliographic Ontology</a>, and dereferencable URIs (i.e. HTTP based Data Source Names for Documents and the Entities they host).</p> <h3>Related</h3> <ul> <li> <a href="http://zitgist.com/labs/linked_data.html" id="link-id104778a8">Linked Data FAQ</a> - by <a href="http://community.linkeddata.org/dataspace/person/mkbergman#this" id="link-idd8e71b0">Mike Bergman</a> </li> <li> <a href="http://www.wired.com/special_multimedia/2008/st_infoporn_1607" id="link-id134c1e80">The Planetary Computer</a> from <a href="http://www.wired.com" id="link-id13416518">Wired Magazine</a> - which is basically the effect of Linked Data under a different label (note to Wired: "Tired of old Media repetitive Registrations" when seeking to make comments in the OpenID era!).</li> </ul>
The DARQ Matter of Federation
http://www.openlinksw.com/blog/vdb/blog/?date=2008-06-09#1381
2008-06-09T14:02:19Z
2008-06-11T15:15:14-04:00
<div> <div style="display:none;">The DARQ Matter of Federation</div> <p>Astronomers propose that the universe is held together, so to speak, by the gravity of invisible "dark matter" spread in interstellar and intergalactic space.</p> <p>For the <a href="http://dbpedia.org/resource/Data" id="link-id0x19dbf410">data</a> web, it will be held together by federation, also an invisible factor. As in Minkowski space, so in <a href="http://dbpedia.org/resource/Cyberspace" id="link-id0x9fc13ff8">cyberspace</a>.</p> <p>To take the astronomical analogy further, putting too much visible stuff in one place makes a black hole, whose chief properties are that it is very heavy, can only get heavier and that nothing comes out.</p> <p> <a href="http://darq.sourceforge.net/" id="link-id0x1d06bd88">DARQ</a> is Bastian Quilitz's federated extension of the <a href="http://jena.sourceforge.net/" id="link-id0x1cf28f70">Jena</a> <a href="http://jena.sourceforge.net/ARQ/" id="link-id0x1cba22c8">ARQ</a> <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x171c7dc8">SPARQL</a> processor. It has existed for a while and was also presented at <a href="http://www.eswc2008.org/" id="link-id0x1ed53cd0">ESWC2008</a>. There is also SPARQL FED from Andy Seaborne, an explicit means of specifying which end point will process which fragment of a distributed SPARQL query. Still, for federation to deliver in an open, decentralized world, it must be transparent. For a specific application, with a predictable workload, it is of course OK to partition queries explicitly.</p> <p>Bastian had split <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x1ce846c0">DBpedia</a> among five <a href="http://virtuoso.openlinksw.com" id="link-id0x1cad0640">Virtuoso</a> servers and was querying this set with DARQ. The end result was that there was a rather frightful cost of federation as opposed to all the data residing in a single Virtuoso. The other result was that if selectivity of predicates was not correctly guessed by the federation engine, the proposition was a non-starter. With correct join order it worked, though.</p> <p>Yet, we really want federation. Looking further down the road, we simply must make federation work. This is just as necessary as running on a server cluster for mid-size workloads.</p> <p>Since we are convinced of the cause, let's talk about the means.</p> <p>For DARQ as it now stands, there's probably an order of magnitude or even more to gain from a couple of simple tricks. If going to a SPARQL end point that is not the outermost in the loop join sequence, batch the requests together in one <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x19a48280">HTTP</a>/1.1 message. So, if the query is "get me my friends living in cities of over a million people," there will be the fragment "get city where x lives" and later "ask if population of x greater than 1000000". If I have 100 friends, I send the 100 requests in a batch to each eligible server.</p> <p>Further, if running against a server of known brand, use a client-server connection and prepared statements with array parameters. This can well improve the processing speed at the remote end point by another order of magnitude. This gain may however not be as great as the latency savings from message batching. We will provide a sample of how to do this with Virtuoso over <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x1cf18278">JDBC</a> so Bastian can try this if interested.</p> <p>These simple things will give a lot of mileage and may even decide whether federation is an option in specific applications. For the open web however, these measures will not yet win the day.</p> <p>When federating <a href="http://dbpedia.org/resource/SQL" id="link-id0x1cf7d0e8">SQL</a>, colocation of data is sort of explicit. If two tables are joined and they are in the same source, then the join can go to the source. For SPARQL this is also so but with a twist:</p> <p>If a foaf:Person is found on a given server, this does not mean that the Person's geek code or email hash will be on the same server. Thus <code>{?p name "Johnny" . ?p geekCode ?g . ?p emailHash ?h }</code> does not necessarily denote a colocated join if many servers serve items of the vocabulary.</p> <p>However, in most practical cases, for obtaining a rapid answer, treating this as a colocated fragment will be appropriate. Thus, it may be necessary to be able to declare that geek codes will be assumed colocated with names. This will save a lot of message passing and offer decent, if not theoretically total recall. For search style applications, starting with such assumptions will make sense. If nothing is found, then we can partition each join step separately for the unlikely case that there were a server that gave geek codes but not names.</p> <p>For Virtuoso, we find that a federated query's asynchronous, parallel evaluation model is not so different from that on a local cluster. So the cluster version could have the option of federated query. The difference is that a cluster is local and tightly coupled and predictably partitioned but a federated setting is none of these.</p> <p>For description, we would take DARQ's description model and maybe extend it a little where needed. Also we would enhance the protocol to allow just asking for the query cost estimate given a query with literals specified. We will do this eventually.</p> <p>We would like to talk to Bastian about large improvements to DARQ, specially when working with Virtuoso. We'll see.</p> <p>Of course, one mode of federating is the crawl-as-you-go approach of the Virtuoso <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id0x1e163140">Sponger</a>. This will bring in fragments following seeAlso or sameAs declarations or other references. This will however not have the recall of a warehouse or federation over well described SPARQL end-points. But up to a certain volume it has the speed of local storage.</p> <p>The emergence of voiD (Vocabulary of Interlinked Data) is a step in the direction of making federation a reality. There is <a href="http://www.openlinksw.com/weblog/oerling/?id=1377" id="link-id1109a4c8">a separate post</a> about this.</p> </div>
The DARQ Matter of Federation
http://www.openlinksw.com/weblog/oerling/?date=2008-06-09#1376
2008-06-09T13:57:30Z
2008-06-11T15:15:00-04:00
<p>Astronomers propose that the universe is held together, so to speak, by the gravity of invisible "dark matter" spread in interstellar and intergalactic space.</p> <p>For the <a href="http://dbpedia.org/resource/Data" id="link-id0x19bbd830">data</a> web, it will be held together by federation, also an invisible factor. As in Minkowski space, so in <a href="http://dbpedia.org/resource/Cyberspace" id="link-id0x19af2488">cyberspace</a>.</p> <p>To take the astronomical analogy further, putting too much visible stuff in one place makes a black hole, whose chief properties are that it is very heavy, can only get heavier and that nothing comes out.</p> <p> <a href="http://darq.sourceforge.net/" id="link-id0x19b7a9c8">DARQ</a> is Bastian Quilitz's federated extension of the <a href="http://jena.sourceforge.net/" id="link-id0x19ce3da0">Jena</a> <a href="http://jena.sourceforge.net/ARQ/" id="link-id0xa569a258">ARQ</a> <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1a8d2270">SPARQL</a> processor. It has existed for a while and was also presented at <a href="http://www.eswc2008.org/" id="link-id0x1aad1d00">ESWC2008</a>. There is also SPARQL FED from Andy Seaborne, an explicit means of specifying which end point will process which fragment of a distributed SPARQL query. Still, for federation to deliver in an open, decentralized world, it must be transparent. For a specific application, with a predictable workload, it is of course OK to partition queries explicitly.</p> <p>Bastian had split <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x1a8ac770">DBpedia</a> among five <a href="http://virtuoso.openlinksw.com" id="link-id0x19601d30">Virtuoso</a> servers and was querying this set with DARQ. The end result was that there was a rather frightful cost of federation as opposed to all the data residing in a single Virtuoso. The other result was that if selectivity of predicates was not correctly guessed by the federation engine, the proposition was a non-starter. With correct join order it worked, though.</p> <p>Yet, we really want federation. Looking further down the road, we simply must make federation work. This is just as necessary as running on a server cluster for mid-size workloads.</p> <p>Since we are convinced of the cause, let's talk about the means.</p> <p>For DARQ as it now stands, there's probably an order of magnitude or even more to gain from a couple of simple tricks. If going to a SPARQL end point that is not the outermost in the loop join sequence, batch the requests together in one <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x19b94818">HTTP</a>/1.1 message. So, if the query is "get me my friends living in cities of over a million people," there will be the fragment "get city where x lives" and later "ask if population of x greater than 1000000". If I have 100 friends, I send the 100 requests in a batch to each eligible server.</p> <p>Further, if running against a server of known brand, use a client-server connection and prepared statements with array parameters. This can well improve the processing speed at the remote end point by another order of magnitude. This gain may however not be as great as the latency savings from message batching. We will provide a sample of how to do this with Virtuoso over <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x17822258">JDBC</a> so Bastian can try this if interested.</p> <p>These simple things will give a lot of mileage and may even decide whether federation is an option in specific applications. For the open web however, these measures will not yet win the day.</p> <p>When federating <a href="http://dbpedia.org/resource/SQL" id="link-id0x1a651628">SQL</a>, colocation of data is sort of explicit. If two tables are joined and they are in the same source, then the join can go to the source. For SPARQL this is also so but with a twist:</p> <p>If a foaf:Person is found on a given server, this does not mean that the Person's geek code or email hash will be on the same server. Thus <code>{?p name "Johnny" . ?p geekCode ?g . ?p emailHash ?h }</code> does not necessarily denote a colocated join if many servers serve items of the vocabulary.</p> <p>However, in most practical cases, for obtaining a rapid answer, treating this as a colocated fragment will be appropriate. Thus, it may be necessary to be able to declare that geek codes will be assumed colocated with names. This will save a lot of message passing and offer decent, if not theoretically total recall. For search style applications, starting with such assumptions will make sense. If nothing is found, then we can partition each join step separately for the unlikely case that there were a server that gave geek codes but not names.</p> <p>For Virtuoso, we find that a federated query's asynchronous, parallel evaluation model is not so different from that on a local cluster. So the cluster version could have the option of federated query. The difference is that a cluster is local and tightly coupled and predictably partitioned but a federated setting is none of these.</p> <p>For description, we would take DARQ's description model and maybe extend it a little where needed. Also we would enhance the protocol to allow just asking for the query cost estimate given a query with literals specified. We will do this eventually.</p> <p>We would like to talk to Bastian about large improvements to DARQ, specially when working with Virtuoso. We'll see.</p> <p>Of course, one mode of federating is the crawl-as-you-go approach of the Virtuoso <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id0x1dddce48">Sponger</a>. This will bring in fragments following seeAlso or sameAs declarations or other references. This will however not have the recall of a warehouse or federation over well described SPARQL end-points. But up to a certain volume it has the speed of local storage.</p> <p>The emergence of voiD (Vocabulary of Interlinked Data) is a step in the direction of making federation a reality. There is <a href="http://www.openlinksw.com/weblog/oerling/?id=1377" id="link-id1109a4c8">a separate post</a> about this.</p>
ODBC & WODBC Comparison
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2008-05-20#1364
2008-05-20T19:37:53Z
2008-05-20T15:46:11-04:00
<p> <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id100eb550">ODBC</a> delivers open <a href="http://dbpedia.org/resource/Data" id="link-idffd2338">data</a> access (by reference) to a broad range of enterprise databases via a '<a href="http://dbpedia.org/resource/C_(programming_language)" id="link-id104fd1d8">C</a>' based API. Thanks to the <a href="http://www.iodbc.org" id="link-id104721b0">iODBC</a> and <a href="http://www.unixodbc.org" id="link-id10954990">unixODBC</a> projects, <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id10494670">ODBC</a> is available across broad range of platforms beyond Windows.</p> <p> <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0xc900928">ODBC</a> identifies <a href="http://dbpedia.org/resource/Data" id="link-id10f82200">data</a> sources using <a href="http://dbpedia.org/resource/Data" id="link-id0xcaad080">Data</a> Source Names (DSNs). </p> <p> WODBC (<a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> Open Database Connectivity) delivers open <a href="http://dbpedia.org/resource/Data">data</a> access to Web Databases / Data Spaces. The Data Source Naming scheme: <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id1009ce40">URI</a> or IRI, is <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id101fc1b0">HTTP</a> based thereby enabling data access by reference via the Web. </p> <p><a href="http://dbpedia.org/resource/Open_Database_Connectivity">ODBC</a> DSNs bind ODBC client applications to Tables, Views, Stored Procedures. </p> <p>WODBC DSNs bind you to a Data <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id10182a88">Space</a> (e.g. my <a href="http://myopenlink.net/dataspace/person/kidehen" id="link-id105a7858">FOAF based Profile Page</a> where you can use the "Explore Data Tab" to look around if you are a human visitor) or a specific <a href="http://dbpedia.org/resource/Entity" id="link-id10bd8578">Entity</a> within a Data <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id10780dc0">Space</a> (i.e <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id10848e08">Person Entity Me</a>).</p> <p>ODBC Drivers are built using APIs (DBMS Call Level Interfaces) provided by DBMS vendors. Thus, a DBMS vendor can chose not to release an API, or do so selectivity, for competitive advantage or market disruption purposes (it's happened!).</p> <p>WODBC Drivers are also built using APIs (Web Services associated with a Web Data <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id0xcbe6348">Space</a>). These drivers are also referred to as <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=rdf%20middleware&type=text&output=html" id="link-id16564058">RDF Middleware</a> or RDFizers. The "Web" component of WODBC ensures openness, you publish Data with URIs from your <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1064a768">Linked Data</a> Server and that's it; your data <a href="http://en.wikipedia.org/wiki/Data_Spaces">space</a> or specific data entities are live and accessible (by reference) over the Web!</p> <p>So we have come full circle (or cycle), the Web is becoming more of a structured database everyday! What's new is old, and what's old is new! </p> <p>Data Access is everything, without "Data" there is no <a href="http://dbpedia.org/resource/Information" id="link-id100a9de8">information</a> or <a href="http://dbpedia.org/resource/Knowledge" id="link-id10bb67e8">knowledge</a>. Without "Data" there's not notion of vitality, purpose, or value.</p> <p>URIs make or break everything in the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10a71638">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10494400">Web</a> just as ODBC DSNs do within the enterprise. </p> <p>I've deliberately left <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id10a05280">JDBC</a>, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id104e4a70">ADO</a>.<a href="http://dbpedia.org/resource/ADO.NET" id="link-id10215668">NET</a>, and OLE-DB out of this piece due to their respective programming languages and frameworks specificity. None of these mechanisms match the platform availability breadth of ODBC.</p> <p>The Web as a true <a href="http://dbpedia.org/resource/Model-view-controller" id="link-id108ee598">M</a>-<a href="http://dbpedia.org/resource/Model-view-controller" id="link-id0xcda5e90">V</a>-C pattern is now crystalizing. The "M" (Model) component of M-V-C is finally rising to the realm of broad attention courtesy of the "<a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id1024ff08">Linked Data" meme</a> and "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id1831b418">Semantic Web</a>" vision.</p> <p>By the way, M-V-C lines up nicely with Web 1.0 (Web Forms / Pages), Web 2.0 (Web Services based APIs), and Web 3.0 (Data Web, Web of Data, or <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0xb6d0e90">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0xb22a158">Web</a>) :-)</p>
On "Semantic", "Semantic Web", and "Linked Data Web"
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2008-05-15#1360
2008-05-15T14:11:13Z
2008-05-15T14:31:38.000004-04:00
<p> <a href="http://novaspivack.typepad.com/" id="link-id102f4e00">Nova Spivack</a> has just penned a post titled: <a href="http://novaspivack.typepad.com/nova_spivacks_weblog/2008/05/on-the-differen.html" id="link-id101a2300">On the Difference Between "Semantic" and "Semantic Web</a>", where he covers the fundamental difference between "Semantic" (what I call "Semantics Inside") and "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id11dd0578">Semantic Web</a>" applications. I would like to extend the distinctions further by adding the "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id10b54ca0">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id106f73d0">Web</a>" distinctions to the developing discourse. </p> <p>The <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1089ff48">Linked Data Web</a> (aka. <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10653828">Linked Data</a>) describes <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id134abfb0">RDF</a> <a href="http://dbpedia.org/resource/Data" id="link-id140283a8">data</a> injected into the Web, where the <a href="http://dbpedia.org/resource/Identity_(object-oriented_programming)" id="link-id1029ebf0">Data Object Identifiers</a> (URIs) in an <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id1011b180">RDF</a> graph (collection of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id103a4960">RDF</a> triples) are endowed with <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id104362d8">HTTP</a> based URIs. The net effect of this approach to <a href="http://dbpedia.org/resource/Data" id="link-id107963a0">Data</a> Object Identity is that it facilitates "Open <a href="http://dbpedia.org/resource/Data" id="link-id1331f640">Data</a> Access by Reference" on the Web (aka <a href="http://dbpedia.org/resource/Data" id="link-id10a3c608">data</a> dereferencing).</p> <p>If you recall pre Web ubiquity, in the enterprise realm for instance, Open Database Connectivity (<a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id12c6dd40">ODBC</a>) emerged as a mechanism for separating <a href="http://dbpedia.org/resource/Data" id="link-id13d6a5b0">Data</a> Access and <a href="http://dbpedia.org/resource/Data" id="link-id10b29488">Data</a> Management in the database oriented Client-Sever model. Although <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id106a8bd8">ODBC</a> gave you access to <a href="http://dbpedia.org/resource/Data">data</a>, the data access entry point took the form of a data access specific naming mechanism called a "Data Source Name" (DSN). <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id106eef18">ODBC</a> DSNs typically exposed Tables or Views. The same thing applies to <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id12c6dfe8">JDBC</a> where a non <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id104cb620">HTTP</a> based URN scheme applies.</p> <p>Zip forward to where we are today on the Web; the Web is evolving from a Document centric Database to a Distributed <a href="http://dbpedia.org/resource/Object_database" id="link-id12d15268">Object Database</a>, and you should see that in <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10716bb8">Linked Data</a> we are now truly looking at the best of all worlds: Web Open Database Connectivity (WODBC) with the following advantages:</p> <ul>- direct Access to a single Record (an <a href="http://dbpedia.org/resource/Entity" id="link-id1037d530">Entity</a>) or Record Sets (<a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id10d48e98">RDF</a> based <a href="http://dbpedia.org/resource/Entity" id="link-id1402c8f0">Entity</a> Sets) by reference over <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id10bae7a8">HTTP</a> across disparate Data Spaces on the Web</ul> <ul>- the ability to mesh disparate data sources without being impeded by back-end DBMS engine model, vendor, host operating development frameworks, or host operating system specificity</ul> <ul>- an opportunity to learn from the enterprise DBMS market and Client-Server markets of yore with regards to the shape and form of next generation <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10fe4558">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10153c98">Web</a> oriented solutions.</ul> <p>To conclude, we now have "Semantics Inside" (<a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id109d1280">RDF</a> or non <a href="http://dbpedia.org/resource/Resource_Description_Framework">RDF</a>), "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id106741a8">Semantic Web</a>" (RDF graphs with Object Identifiers that may or may not be <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id1011cc28">HTTP</a> based), and "<a href="http://dbpedia.org/resource/Linked_Data" id="link-id10793f70">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id149ecc10">Web</a>" (RDF graphs with Object Identifiers that must be <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id10a3b860">HTTP</a> based and dereferencable) oriented applications, in the emerging landscape associated with the "Semantics" moniker.</p> <p>As per usual, this post is a record in my <a href="http://dbpedia.org/resource/Blog" id="link-id1020e240">Blog</a> oriented <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id105cbf90">Data Space</a> on the Web. The permalink of this post is a <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id10ce53a8">URI</a> constructed with <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id1082f0f8">Giant Global Graph</a> enrichment in mind :-)</p>
WWW 2008
http://www.openlinksw.com/blog/vdb/blog/?date=2008-04-29#1348
2008-04-29T14:37:20Z
2008-04-29T13:35:23-04:00
<div> <div style="display:none;">WWW 2008</div> <p>Following my return from WWW 2008 in <a href="http://www2008.org/" id="link-id0x9ff7d5d0">Beijing</a>, I will write a series of <a href="http://dbpedia.org/resource/Blog" id="link-id0x9e4a7650">blog</a> posts discussing diverse topics that were brought up in presentations and conversations during the week.</p> <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x9e7ae398">Linked data</a> was our main interest in the conference and there was a one day workshop on this, unfortunately overlapping with a day of W3C Advisory Committee meetings. Hence Tim Berners-Lee, one of the chairs of the workshop, could not attend for most of the day. Still, he was present to say that "<a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0xa287d38">Linked open data</a> is the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x15372940">semantic web</a> and the web done as it ought to be done." <p>For my part, I will draw some architecture conclusions from the different talks and extrapolate about the requirements on database platforms for linked data.</p> <p>Chris Bizer predicted that 2008 would be the year of <a href="http://dbpedia.org/resource/Data" id="link-id0xa1454c58">data</a> web search, if 2007 was the year of <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xa0f73c50">SPARQL</a>. This may be the case, as linked data is now pretty much a reality and the questions of discovery become prevalent. There was a birds-of-a-feather session on this and I will make some comments on what we intend to explore in bridging between the text index based semantic web search engines and SPARQL.</p> <p>Andy Seaborne convened a birds-of-a-feather session on the future of SPARQL. Many of the already anticipated and implemented requirements were confirmed and a few were introduced. A separate blog post will discuss these further.</p> <p>From the various discussions held throughout the conference, we conclude that plug-and-play operation with the major semantic web frameworks of Jena, Sesame, and Redland, is our major immediate-term deliverable. Our efforts in this direction thus far are insufficient and we will next have these done with the right supervision and proper interop testing. The issues are fortunately simple but doing things totally right require some small server side support and some <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0xa5d4d5b8">JDBC</a>/<a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x9dc28d10">ODBC</a> tweaks, so to the interested, we advise to wait for an update to be published on this blog.</p> <p>I further had a conversation with Andy Seaborne about using Jena reasoning capabilities with <a href="http://virtuoso.openlinksw.com" id="link-id0xa2754050">Virtuoso</a> and generally the issues of "impedance mismatch" between reasoning and typical database workloads. More on this later. </p> </div>
WWW 2008
http://www.openlinksw.com/weblog/oerling/?date=2008-04-29#1345
2008-04-29T11:59:16Z
2008-04-29T13:35:17-04:00
<p>Following my return from WWW 2008 in <a href="http://www2008.org/" id="link-id0xa1901330">Beijing</a>, I will write a series of <a href="http://dbpedia.org/resource/Blog" id="link-id0xd356660">blog</a> posts discussing diverse topics that were brought up in presentations and conversations during the week.</p> <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1290cb78">Linked data</a> was our main interest in the conference and there was a one day workshop on this, unfortunately overlapping with a day of W3C Advisory Committee meetings. Hence Tim Berners-Lee, one of the chairs of the workshop, could not attend for most of the day. Still, he was present to say that "<a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0xa2754050">Linked open data</a> is the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0xa099dea8">semantic web</a> and the web done as it ought to be done." <p>For my part, I will draw some architecture conclusions from the different talks and extrapolate about the requirements on database platforms for linked data.</p> <p>Chris Bizer predicted that 2008 would be the year of <a href="http://dbpedia.org/resource/Data" id="link-id0x9e0f0a98">data</a> web search, if 2007 was the year of <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xd9e64a0">SPARQL</a>. This may be the case, as linked data is now pretty much a reality and the questions of discovery become prevalent. There was a birds-of-a-feather session on this and I will make some comments on what we intend to explore in bridging between the text index based semantic web search engines and SPARQL.</p> <p>Andy Seaborne convened a birds-of-a-feather session on the future of SPARQL. Many of the already anticipated and implemented requirements were confirmed and a few were introduced. A separate blog post will discuss these further.</p> <p>From the various discussions held throughout the conference, we conclude that plug-and-play operation with the major semantic web frameworks of Jena, Sesame, and Redland, is our major immediate-term deliverable. Our efforts in this direction thus far are insufficient and we will next have these done with the right supervision and proper interop testing. The issues are fortunately simple but doing things totally right require some small server side support and some <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0xacc8c7e8">JDBC</a>/<a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x9e48d258">ODBC</a> tweaks, so to the interested, we advise to wait for an update to be published on this blog.</p> <p>I further had a conversation with Andy Seaborne about using Jena reasoning capabilities with <a href="http://virtuoso.openlinksw.com" id="link-id0x134b8e98">Virtuoso</a> and generally the issues of "impedance mismatch" between reasoning and typical database workloads. More on this later. </p>
Linked Data enabling PHP Applications
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2008-04-10#1334
2008-04-10T18:09:49Z
2008-04-10T14:12:47-04:00
<p> <a href="http://myopenlink.net/dataspace/person/danieljohnlewis#this" id="link-id10820610">Daniel lewis</a> has penned a variation of post about <a href="http://vanirsystems.com/danielsblog/2008/04/10/simplified-adding-wordpress-blogs-into-the-linked-data-web-using-virtuoso/" id="link-id10827948">Linked Data enabling PHP applications</a> such as: <a href="http://dbpedia.org/resource/WordPress" id="link-id10426278">Wordpress</a>, <a href="http://dbpedia.org/resource/PhpBB" id="link-id13f431c0">phpBB3</a>, <a href="http://dbpedia.org/resource/MediaWiki" id="link-id10dd8760">MediaWiki</a> etc.</p> <p>Daniel simplifies my post by using diagrams to depict the different paths for <a href="http://dbpedia.org/resource/PHP" id="link-id10adcc08">PHP</a> based applications exposing <a href="http://dbpedia.org/resource/Linked_Data" id="link-id107b4e60">Linked Data</a> - especially those that already provide a significant amount of the content that drives <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id13b0ab48">Web</a> 2.0.</p> <p>If all the content in <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id0x1d499470">Web</a> 2.0 <a href="http://dbpedia.org/resource/Information" id="link-id12bd3b10">information</a> resources are distillable into discrete <a href="http://dbpedia.org/resource/Data" id="link-id10962060">data</a> objects endowed with <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id176a30e8">HTTP</a> based IDs (URIs), with zero "<a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=rdf%20tax&type=text&output=html" id="link-id1098bcd8">RDF handcrafting Tax</a>", what do we end up with? A <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id1372ce88">Giant Global Graph</a> of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0xa29f0658">Linked Data</a>; the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> as a Database.</p> <p>So, what used to apply exclusively, within enterprise settings re. <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id12d91448">Oracle</a>, <a href="http://dbpedia.org/resource/IBM_DB2" id="link-id13dd27d8">DB2</a>, <a href="http://dbpedia.org/resource/IBM_Informix" id="link-id108e6b98">Informix</a>, <a href="http://dbpedia.org/resource/Ingres" id="link-id13383708">Ingres</a>, <a href="http://dbpedia.org/resource/Sybase" id="link-idfed8aa8">Sybase</a>, <a href="http://dbpedia.org/resource/Microsoft_SQL_Server" id="link-id10b8b190">Microsoft SQL Server</a>, <a href="http://dbpedia.org/resource/MySQL" id="link-id13066ea8">MySQL</a>, PostrgeSQL, Progress Open Edge, <a href="http://dbpedia.org/resource/Firebird_database_server" id="link-id104f0a78">Firebird</a>, and others, now applies to the Web. The Web becomes the "<a href="http://dbpedia.org/resource/federated_database_system" id="link-id105a5340">Distributed Database</a> Bus" that connects database records across disparate databases (or <a href="http://dbpedia.org/resource/Data" id="link-id0xc706c68">Data</a> Spaces). These databases manage and expose records that are remotely accessible "by reference" via <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1c8f7fe0">HTTP</a>.</p> <p>As I've stated at every opportunity in the past, Web 2.0 is the greatest thing that every happened to the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id13d65278">Semantic Web</a> vision :-) Without the "<a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=Web%202.0%20%20conundrum&type=text&output=html" id="link-id100d16d0">Web 2.0 Data Silo Conundrum</a>" we wouldn't have the cry for "<a href="http://dbpedia.org/resource/Data">Data</a> Portability" that brings a lot of clarity to some fundamental Web 2.0 limitations that end-users ultimately find unacceptable.</p> <p> In the late '80s, the <a href="http://dbpedia.org/resource/SQL" id="link-idff4f0d0">SQL</a> <a href="http://dbpedia.org/resource/SQL_Access_Group" id="link-id138fbd40">Access Group</a> (now part of <a href="http://dbpedia.org/resource/X/Open" id="link-id104ee010">X</a>/<a href="http://dbpedia.org/resource/X/Open" id="link-id0xac9eab8">Open</a>) addressed a similar problem with <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id106d2008">RDBMS</a> silos within the enterprise that lead to the SAG <a href="http://dbpedia.org/resource/Call_Level_Interface" id="link-id105d45d0">CLI</a> which is exists today as Open Database Connectivity.</p> <p>In a sense we now have WODBC (Web Open Database Connectivity), comprised of Web Services based CLIs and/or traditional back-end DBMS CLIs (<a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id13f58708">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id10aa81e0">JDBC</a>, <a href="http://dbpedia.org/resource/ADO.NET" id="link-id5fddb68">ADO</a>.<a href="http://dbpedia.org/resource/ADO.NET" id="link-id0x9f085a10">NET</a>, OLE-DB, or Native), Query Language (<a href="http://dbpedia.org/resource/SPARQL" id="link-id10adb5c8">SPARQL</a> Query Language), and a Wire Protocol (<a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol">HTTP</a> based <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id126fa068">SPARQL Protocol</a>) delivering Web infrastructure equivalents of <a href="http://dbpedia.org/resource/SQL" id="link-id0x1d0a5fc8">SQL</a> and RDA, but much better, and with much broader scope for delivering profound value due to the Web's inherent openness. Today's <a href="http://dbpedia.org/resource/PHP" id="link-id0xc88ed68">PHP</a>, <a href="http://dbpedia.org/resource/Python_programming_language" id="link-id10a70530">Python</a>, <a href="http://dbpedia.org/resource/Ruby_programming_language" id="link-id13d9da18">Ruby</a>, <a href="http://dbpedia.org/resource/Tcl" id="link-id10a3c2a8">Tcl</a>, <a href="http://dbpedia.org/resource/Perl" id="link-id13e1b6f0">Perl</a>, <a href="http://dbpedia.org/resource/ASP.NET" id="link-id10810388">ASP</a>.<a href="http://dbpedia.org/resource/ASP.NET" id="link-id0xa22ce378">NET</a> developer is the enterprise <a href="http://dbpedia.org/resource/4GL" id="link-id1396a500">4GL</a> developer of yore, without enterprise confinement. We could even be talking about <a href="http://dbpedia.org/resource/5GL" id="link-id1077f250">5GL</a> development once the <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> interaction is meshed with dynamic languages (delivering higher levels of abstraction at the language and data interaction levels). Even the underlying schemas and basic design will evolve from <a href="http://dbpedia.org/resource/Closed_world_assumption" id="link-id10b280c8">Closed World</a> (solely) to a mesh of Closed & <a href="http://dbpedia.org/resource/Open_world_assumption" id="link-id104b9978">Open World</a> view schemas.</p>
Adding Wordpress Blogs into the Linked Data Web using Virtuoso
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2008-04-09#1333
2008-04-09T21:27:34Z
2008-04-10T12:33:05.000003-04:00
<p> <a href="http://dbpedia.org/resource/WordPress" id="link-id101103b0">Wordpress</a> is a Weblog platform comprised of the following: </p> <ol> <li>User Interface - <a href="http://dbpedia.org/resource/PHP" id="link-id107ba368">PHP</a> </li> <li>Application Logic - <a href="http://dbpedia.org/resource/PHP" id="link-id107066b8">PHP</a> </li> <li> <a href="http://dbpedia.org/resource/Data" id="link-id13968340">Data</a> Storage (<a href="http://dbpedia.org/resource/SQL" id="link-id104c5350">SQL</a> <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id1076d790">RDBMS</a>) - <a href="http://dbpedia.org/resource/MySQL" id="link-id109c4ea0">MySQL</a> via <a href="http://dbpedia.org/resource/PHP" id="link-id133af570">PHP</a>-<a href="http://dbpedia.org/resource/MySQL" id="link-idf0b03b0">MySQL</a> </li> <li> <a href="http://dbpedia.org/resource/Application_server" id="link-id13217630">Application Server</a> - <a href="http://dbpedia.org/resource/Apache" id="link-id108219d8">Apache</a> </li> </ol> <p>In the form above (the norm), <a href="http://dbpedia.org/resource/WordPress" id="link-id105c6d88">Wordpress</a> <a href="http://dbpedia.org/resource/Data" id="link-id104938f8">data</a> can be injected into the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id107a5f18">Linked Data</a> <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id177329c0">Web</a> via RDFization middleware such as the<a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id10531b50">Virtuoso Sponger</a> (built into all <a href="http://virtuoso.openlinksw.com" id="link-id10d7e710">Virtuoso</a> instances) and <a href="http://triplr.org/" id="link-id107dcab8">Triplr</a>. The downside of this approach is that the <a href="http://dbpedia.org/resource/Blog" id="link-id1055ab68">blog</a> owner doesn't necessary possess full control over their contributions to the emerging <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-idfed0358">Giant Global Graph</a> or <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10d70668">Linked Data</a>.</p> <p>Another route to <a href="http://dbpedia.org/resource/Linked_Data" id="link-id104c7f68">Linked Data</a> exposure is via <a href="http://virtuoso.openlinksw.com" id="link-id0xa255fb50">Virtuoso</a>'s Metaschema Language for producing <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id10968388">RDF</a> Views over <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id13f594c8">ODBC</a>/<a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id138f69a8">JDBC</a> accessible <a href="http://dbpedia.org/resource/Data" id="link-id1393c068">Data</a> Sources, that enables the following setup:</p> <ol> <li>User Interface - <a href="http://dbpedia.org/resource/PHP" id="link-id0x9fb9c478">PHP</a> </li> <li>Application Logic - PHP </li> <li> <a href="http://dbpedia.org/resource/Data" id="link-id0xc605960">Data</a> Storage (<a href="http://dbpedia.org/resource/SQL" id="link-id0xc2be608">SQL</a> <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0xc7a28a8">RDBMS</a>) - <a href="http://dbpedia.org/resource/MySQL" id="link-id0xc7228f0">MySQL</a> via the PHP-MySQL <a href="http://dbpedia.org/resource/Data">data</a> access interface </li> <li> <a href="http://dbpedia.org/resource/Virtual_Database" id="link-id134b1ee8">Virtual Database</a> linkage of MySQL Tables into Virtuoso </li> <li> <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-idfe31548">RDF</a> View generated over the Virtual SQL Tables </li> <li> <a href="http://dbpedia.org/resource/Application_server" id="link-id0xb8dfa68">Application Server</a> - Virtuoso which provides <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0xc149518">Linked Data</a> Deployment such that <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id10ad9ca0">RDF</a> <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> is exposed when requested by <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-idfd352e0">Web</a> User Agents.</li> </ol> <p>Alternatively, you can also exploit Virtuoso as the SQL DBMS, <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x9ec4f440">RDF</a> DBMS, Application Server, and Linked Data Deployment platform:</p> <ol> <li>User Interface - PHP </li> <li> Application Logic - PHP </li> <li>Data Storage (SQL RDBMS) - Virtuoso via PHP-<a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id14197218">ODBC</a> data access interface (* <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id103d1a80">ODBC</a> is Virtuoso's native SQL CLI/API *) </li> <li><a href="http://dbpedia.org/resource/Resource_Description_Framework">RDF</a> View generated over the Native SQL Tables </li> <li>Application Server - Virtuoso which provides Linked Data Deployment such that RDF Linked Data is exposed when requested by <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id13918d68">Web</a> User Agents (e.g. <a href="http://demo.openlinksw.com/rdfbrowser" id="link-idff835f0">OpenLink RDF Browser</a>, <a href="http://zitgist.com/about/" id="link-id1372e510">Zitgist</a> <a href="http://dataviewer.zitgist.com" id="link-id109c3048">Data Viewer</a>, <a href="http://www4.wiwiss.fu-berlin.de/rdf_browser" id="link-id105d97f0">DISCO Hyperdata Browser</a>, and <a href="http://dig.csail.mit.edu/2005/ajar/release/tabulator/0.8/tab.html" id="link-id10cc20d8">Tabulator</a>). </li> </ol> <h2 align="left">Benefits?</h2> <ul> <li>Each user account gets a proper Linked Data <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id108c92b0">URI</a> (ID) that can me meshed/smushed with other IDs (so you add data from this new <a href="http://dbpedia.org/resource/Blog" id="link-idfd39648">blog</a> space to other linked data sources associated with you other URIs/IDs) </li> <li>Each post gets a proper <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id10add540">URI</a> All data is now query-able via <a href="http://dbpedia.org/resource/SPARQL" id="link-id101b98f0">SPARQL</a> Discoverability increases exponentially (without drop in relevance in either direction i.e. discovering or being discovered)</li> </ul> <p>How Do I map the <a href="http://dbpedia.org/resource/WordPress" id="link-id12e448c0">WordPress</a> SQL Schema to RDF using Virtuoso? </p> <ul> <li>Determine the RDF Schema or Ontologies that define the Classes for which you will be producing instance data (e.g. SIOC and FOAF) </li> <li>Declare <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-idfaf5c80">URI</a>/IRI generator functions (*special Virtuoso functions*) </li> <li>Use <a href="http://dbpedia.org/resource/SPARQL" id="link-id100436b8">SPARQL</a> Graph patterns to apply <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x9de74950">URI</a>/IRI generator functions to Tables, Views, Table Values mode Stored Procedures, Query Resultsets as part of RDBMS to RDF mapping </li> </ul> <p> Read the <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSSQL2RDF" id="link-idfaf5d58">Meta Schema Language guide</a> or simply apply our "<a href="http://dbpedia.org/resource/WordPress" id="link-id0x9ef73c78">WordPress</a> SQL Schema to RDF" script to your Virtuoso hosted instance. Of course, there are other mappings that cover other PHP applications deployed via Virtuoso:</p> <ul> <li> <a href="http://dbpedia.org/resource/PhpBB" id="link-id179f4870">phpBB3</a> SQL Schema to RDF </li> <li> <a href="http://dbpedia.org/resource/Drupal" id="link-id10b263d8">Drupal</a> SQL Schema to RDF </li> <li> <a href="http://dbpedia.org/resource/MediaWiki" id="link-id10263a40">MediaWiki</a> SQL Schema to RDF </li> </ul> <h2>Live Demos?</h2> <ul> <li> <a href="http://demo.openlinksw.com/phpBB3" id="link-id17761e88">Virtuoso Hosting phpBB3</a> (<a href="http://demo.openlinksw.com/phpBB3/user/demo#this" id="link-id10087e68">example User URI</a>)</li> <li> <a href="http://demo.openlinksw.com/drupal" id="link-id1091f1d8">Virtuoso Hosting Drupal</a> (<a href="http://demo.openlinksw.com/drupal/user/demo#this" id="link-id13e3d468">example User URI</a>)</li> <li> <a href="http://demo.openlinksw.com/mediawiki" id="link-id10531be0">Virtuoso Hosting MediaWiki</a> (<a href="http://demo.openlinksw.com/mediawiki/user/KingsleyIdehen#this" id="link-id109c5d40">example User URI</a>)</li> </ul>
Linked Data Solution for Exposing OpenLink Product Portfolio
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2008-02-16#1317
2008-02-16T21:08:17Z
2008-02-25T15:08:04-05:00
<p>At <a href="http://www.openlinksw.com" id="link-id14a37688">OpenLink Software</a>, we've had an immense problem explaining the depth and breadth of our product porfolio via traditional Document Web pages. Thanks to <a href="http://dbpedia.org/resource/SPARQL" id="link-id101bc330">SPARQL</a> and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id105d2d28">Linked Data</a>, we are now able to use <a href="http://dbpedia.org/resource/Identity_%2528object-oriented_programming%2529" id="link-id1341fbd0">Web Data Object IDs</a> (<a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-idf101908">HTTP</a> based <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id135cd8a0">URI</a>s) to produce super SKUs for every item in our product portfolio. Even better, we are able to handle the additional challenge of exposing features and benefits which by their very nature are mercurial across an array of fronts (products releases, product formats, and supported platforms etc). </p> <p> Now I can simply state the following using Linked Data (hyperdata) links:</p> <a href="http://dbpedia.org/resource/OpenLink_Software" id="link-idfafc070">OpenLink Software</a>'s product porfolio is comprised of the following product families: <ol> <li> <a href="http://data.openlinksw.com/oplweb/product_family/uda#this" id="link-idf856aa8">Universal Data Access Drivers Suite (UDA)</a> for <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-idfc5f498">ODBC</a>, JDBC, <a href="http://dbpedia.org/resource/ADO.NET" id="link-idfc20120">ADO.NET</a>, <a href="http://dbpedia.org/resource/OLE_DB" id="link-id11579a08">OLE-DB</a>, and <a href="http://dbpedia.org/resource/XML_for_Analysis" id="link-id1067d098">XMLA</a> </li> <li> <a href="http://data.openlinksw.com/oplweb/product_family/dca#this" id="link-id1318b3f8">OpenLink Data Spaces</a> </li> <li> <a href="http://data.openlinksw.com/oplweb/product_family/virtuoso#this" id="link-idf8d7ea0">Virtuoso</a> </li> </ol> <p> We no longer have to explain (repeatedly) why our drivers exist in <a href="http://data.openlinksw.com/oplweb/product_format/express#this" id="link-idf76e1f0">Express</a>, <a href="http://data.openlinksw.com/oplweb/product_format/st#this" id="link-idf861be8">Lite</a>, and <a href="http://data.openlinksw.com/oplweb/product_format/mt#this" id="link-id128da1e0">Multi-Tier</a> Edition formats, or why you ultimately need Multi-Tier Drivers over Single Tier Drivers (Express or Lite Editions) since you ultimately heed <a href="http://data.openlinksw.com/oplweb/product_format_feature/RulesEngineBasedSecurity#this" id="link-idf8d70b0">high-performance, data encryption, and policy based security</a> across each of the data access driver formats.</p>
SPARQL based RDF Store Benchmarks via DBpedia
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2008-01-17#1298
2008-01-18T01:10:00Z
2008-05-05T08:06:45-04:00
<p> <a href="http://christianhbecker.com/contact/" id="link-id1436ad98">Christian Becker</a> has delivered the final cut of an initial iteration of his <a href="http://dbpedia.org" id="link-id143223c8">DBpedia</a> based <a href="http://www4.wiwiss.fu-berlin.de/benchmarks-200801/" id="link-id1729ebc8">RDF Data Stores benchmark</a>. This particular exercise brought some very interesting things to our attention re. <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id17178108">Virtuoso</a>'s default mode of operation:</p> <ol> <li> <a href="http://virtuoso.openlinksw.com" id="link-id0x1c519718">Virtuoso</a> is a Quad Store in a Triple Store world -- it supports <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1cce9730">RDF</a> <a href="http://dbpedia.org/resource/Data" id="link-id0x1934f0b8">data</a> set storage partitioning via <a href="http://www.w3.org/TR/rdf-sparql-query/#namedGraphs" id="link-id13f6c948">Named Graphs</a> and it requires the use of the <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1eeb1b28">SPARQL</a> FROM clause to scope query patterns to appropriate <a href="http://dbpedia.org/resource/Data">data</a> sets. Otherwise, it looks across all hosted data sets for matching patterns</li> <li>We should be able to use our server side configuration settings to make the Quad Store behave like a Triple Store (meaning we set the list of applicable named graphs as part of the session configuration)</li> <li>Provide hints to users about missing POGS, PSOG, and SOPG bitmap indexes when <a href="http://dbpedia.org/resource/SPARQL">SPARQL</a> query patterns received by the server are deemed suboptimal (we do know the execution costs of each query)</li> </ol> <p>How Do I create the missing Bitmap Indexes?</p> <p>Go to the <a href="http://dbpedia.org/resource/HTML" id="link-id1733e248">HTML</a> based Virtuoso Conductor, iSQL command line interface, or an <a href="http://dbpedia.org/resource/ODBC" id="link-id1427f338">ODBC</a> / <a href="http://dbpedia.org/resource/JDBC" id="link-id16edfa08">JDBC</a> / <a href="http://dbpedia.org/resource/ADO.NET" id="link-id1716af90">ADO.NET</a> / <a href="http://dbpedia.org/resource/OLE_DB" id="link-id1725e5e0">OLE DB</a> client and execute:</p> <b><pre>CREATE BITMAP index <a href="http://dbpedia.org/resource/Resource_Description_Framework">RDF</a>_QUAD_POGS on DB.DBA.RDF_QUAD (P,O,G,S); <br />CREATE BITMAP index RDF_QUAD_PSOG on DB.DBA.RDF_QUAD (P,S,O,G); <br />CREATE BITMAP index RDF_QUAD_SOPG on DB.DBA.RDF_QUAD (S,O,P,G); </pre></b> <h3>Related</h3> <ul> <a href="http://esw.w3.org/topic/RdfStoreBenchmarking" id="link-id14de2630">RDF Store Benchamrking Wiki</a> </ul>
OpenOffice.org, SPARQL, and the Linked Data Web
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2008-01-04#1288
2008-01-05T02:50:00Z
2008-02-04T20:42:50.000004-05:00
<p>Question posed by Dan Brickley via a blog post: SQL, OpenOffice: <a href="http://danbri.org/words/2008/01/04/245" id="link-id1689abd8">would a JDBC driver for SPARQL protocol make sense?</a> </p> <p>Writing a <a href="http://dbpedia.org/resource/JDBC_driver" id="link-id16a96580">JDBC Driver</a> for <a href="http://dbpedia.org/resource/SPARQL" id="link-id1a908a70">SPARQL</a> is a little overkill. <a href="http://dbpedia.org/resource/OpenOffice.org" id="link-id16ae69a8">OpenOffice.org</a> simply needs to make <a href="http://dbpedia.org/resource/XML" id="link-id168d3880">XML</a> or Web Data (<a href="http://dbpedia.org/resource/HTML" id="link-id1a7f1f50">HTML</a>, <a href="http://dbpedia.org/resource/XHTML" id="link-id16c1ae60">XHTML</a>, and XML) bonafide data sources within its "<a href="http://dbpedia.org/resource/Pivot_table" id="link-id16665398">Pivot Table</a>" functionality realm. Then all that would then be required is a <a href="http://www.w3.org/TR/rdf-sparql-query/#select" id="link-id168bcbe8">SPARQL SELECT Query</a> transported via the <a href="http://www.w3.org/TR/rdf-sparql-protocol/" id="link-id16c1bbc0">SPARQL Protocol</a> with results sent back using the <a href="http://www.w3.org/TR/rdf-sparql-XMLres/" id="link-id1aa61118">SPARQL XML results serialization</a> format (all part of a single SPARQL Protocol URL).</p> <p>Excel successfully consumes the following information resource URI: http://tinyurl.com/yvoccj (a tiny url for a SPARQL SELECT against my<a href="http://myopenlink.net/dataspace/person/kidehen" id="link-id16702ba8"> FOAF file</a>).</p> <p>Alternatively, and currently achievable, you could simply use <a href="http://esw.w3.org/topic/SPASQL" id="link-id1a1b6b78">SPASQL</a> (SPARQL within SQL) using a <a href="http://dbpedia.org/resource/DBMS" id="link-id1661f240">DBMS</a> engine that supports SQL, SPARQL, and SPARQL e.g. <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id168bba60">Virtuoso</a>. </p> <p> <a href="http://docs.openlinksw.com/virtuoso/rdfapiandsql.html" id="link-id167d9508">Virtuoso SPASQL support</a> is exposed via it's <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id16c62160">ODBC</a> and/or JDBC Drivers. Thus you can do things such as: </p> <ol> <li>Use a SPARQL Query in the FROM CLAUSE of a <a href="http://dbpedia.org/resource/SQL" id="link-id1657a3a8">SQL</a> statement</li> <li>Execute SPARQL via SQL processor by prepending SPARQL query text with the literals "sparql" </li> </ol> <p>BTW - My News Years Resolution: get my act together and shrink the ever increasing list of "simple & practical Virtuoso use case demos" on my todo which now spans all the way back to 2006 :-(</p>
Linked Data & The Web Information BUS
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2007-07-20#1231
2007-07-20T07:50:00Z
2007-08-08T18:26:55-04:00
<p> <a href="http://www.wiwiss.fu-berlin.de/suhl/ueber_uns/team/chris_bizer.htm">Chris Bizer</a>, <a href="http://richard.cyganiak.de/">Richard Cyganiak</a>, and <a href="http://kmi.open.ac.uk/people/tom/html">Tom Heath</a> have just published a <a href="http://sites.wiwiss.fu-berlin.de/suhl/bizer/pub/LinkedDataTutorial/">Linked Data Publishing Tutorial</a> that provides a guide to the mechanics of Linked Data injection into the Semantic Data Web.</p> <p> On different, but related, thread, <a href="http://www.mkbergman.com">Mike Bergman</a> recently penned a post titled: <a href="http://feeds.feedburner.com/~r/AI3_AdaptiveInformation/~3/134989485/">What is the Structured Web?</a>. Both of these public contributions shed light on the "Information BUS" essence of the World Wide Web by describing the evolving nature of the payload shuttled by the BUS. </p> <h3>What is an Information BUS? </h3> <p>Middleware infrastructure for shuttling "Information" between endpoints using a messaging protocol.</p> <p>The Web is the dominant Information BUS within the Network Computer we know as the "Internet". It uses HTTP to shuttle information payloads between "Data Sources" and "Information Consumers" - what happens when we interact with Web via User Agents / Clients (e.g Browsers). </p> <h3>What are Web Information Payloads?</h3> <p>HTTP transported streams of contextualized data. Hence the terms: "Information Resource" and "Non Information" when reading material related to <a href="http://www.w3.org/2001/tag/doc/httpRange-14/2007-05-31/HttpRange-14#iddiv2104118728">http-range-14 and Web Architecture</a>. For example, an (X)HTML document is a specific data context (representation) that enables us to perceive, or comprehend, a data stream originating from a Web Server as a Web Page. On the other hand, if the payload lacks contextualized data, a fundamental Web requirement, then the resource is referred to as a "Non Information" resource. Of course, there is really no such thing as a "Non Information" resource, but with regards to Web Architecture, it's the short way of saying: "the Web Transmits Information only". That said, I prefer to refer to these "Non Information" resources as "Data Sources", are term well understood in the world of Data Access Middleware (ODBC, JDBC, OLEDB, ADO.NET etc.) and Database Management Systems (Relational, Objec-Relational, Object etc).</p> <p>Examples of Information Resource and Data Source URIs:</p> <ul> <a href="http://demo.openlinksw.com/Northwind/Customer/ALFKI">http://demo.openlinksw.com/Northwind/Customer/ALFKI</a> (Information Resource)</ul> <ul> <a href="http://demo.openlinksw.com/sparql?query=CONSTRUCT+{+%3Chttp%3A//demo.openlinksw.com/Northwind/Customer/ALFKI%23this%3E+%3Fp+%3Fo+}+FROM+%3Chttp%3A//demo.openlinksw.com/Northwind%3E+WHERE+{+%3Chttp%3A//demo.openlinksw.com/Northwind/Customer/ALFKI%23this%3E+%3Fp+%3Fo+}&format=application/rdf%2Bxml">http://demo.openlinksw.com/Northwind/Customer/ALFKI#this</a> (Data Source)</ul> <p>Explanation: The Information Resource is a conduit to the Entity identified by Data Source (an entity in my RDF Data Space that is the Subject or Object of one of more Triple based Statements. The triples in question can that can be represented as an RDF resource when transmitted over the Web via an Information Resource that takes the form of a SPARQL REST Service URL or a Physical RDF based Information Resource URL). </p> <h3>What about Structured Data?</h3> <p>Prior to the emergence of the Semantic Data Web, the payloads shuttled across the Web Information BUS comprised primarily of the following:</p> <ol> <li>HTML - Web Resource with presentation focused structure (Web 1.0 dominant payload form)</li> <li>XML - Web Resource with structure that separates presentation and data (Web 2.0's dominant payload form).</li> </ol> <p>The Semantic Data Web simply adds <a href="http://blogs.usnet.private:8893/Resource_Description_Framework">RDF</a> to the payload formats that shuttle the Web Information BUS. RDF addresses formal data structure which XML doesn't cover since it is semi-structured (distinct data entities aren't formally discernible). In a nutshell, an RDF payload is basically a conceptual model database packaged as an Information Resource. It's comprised of granular data items called "Entities", that expose fine grained properties values, individual and/or group characteristics (attributes), and relationships (associations) with other Entities.</p> <h3>Where is this all headed? </h3> <p>The Web is in the final stages of the 3rd phase of it's evolution. A phase characterized by the shuttling of structured data payloads (RDF) alongside less data oriented payloads (HTML, XHTML, XML etc.). As you can see, <a href="http://linkeddata.org">Linked Data</a> and Structured Data are both terms used to describe the addition of more data centric payloads to the Web. Thus, you could view the process of creating a Structured Web of Linked Data as follows:</p> <ol> <li>Identify or Create Structured Data Sources</li> <li>Name these Data Sources using Data Source URIs</li> <li>Expose Structured Data Sources to the Web as Linked Data using Information Resource (conduit) URIs</li> </ol> <h3>Conclusions</h3> <p>The Semantic Data Web is an evolution of the current Web (an Information Space) that adds structured data payloads (RDF) to current, less data oriented, structured payloads (HTML, XHTML, XML, and others).</p> <p>The Semantic Data Web is increasingly seen as an inevitability because it's rapidly reaching the point of critical mass (i.e. network effect kick-in). As a result, Data Web emphasis is moving away from: "What is the Semantic Data Web?" To: "How will Semantic Data Web make our globally interconnected village an even better place?", relative to the contributions accrued from the Web thus far. Remember, the initial "Document Web" (Web 1.0) bootstrapped because of the benefits it delivered to blurb-style content publishing (remember the term electronic brochure-ware?). Likewise, in the case of the "Services Web" (Web 2.0), the bootstrap occurred because it delivered platform independence to Web Application Developers - enabling them to expose application logic behind Web Services. It is my expectation that the Data Integration prowess of the Data Web will create a value exchange realm for data architects and other practitioners from the database and data access realms.</p> <h3>Related Items</h3> <ol> <li> <a href="http://www.mkbergman.com/?p=153">Mike Bergman's post about Semi-Structured Data</a> </li> <li> <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=structured%20data&type=text&output=html">My Posts covering Structured and Un-Structured Containers</a> </li> </ol>
Enterprise 0.0, Linked Data, and Semantic Data Web
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2007-06-14#1224
2007-06-14T15:28:26Z
2008-02-04T23:19:26.000001-05:00
<p>Last week we <a href="http://www.openlinksw.com/press/virt_501.htm">officially released Virtuoso 5.0.1</a> (in Commercial and Open Source Editions). The press release provided us with an official mechanism and timestamp for the current Virtuoso feature set.</p> <p>A vital component of the new Virtuoso release is the finalization of our SQL to RDF mapping functionality -- enabling the declarative mapping of SQL Data to RDF. Additional technical insight covering other new features (delivered and pending) is provided by <a href="http://www.openlinksw.com/weblogs/oerling/">Orri Erling</a>, as part of a series of post-Banff posts.</p> <h2>Why is SQL to RDF Mapping a Big Deal?</h2> <p>A majority of the world's data (especially in the enterprise realm) resides in SQL Databases. In addition, Open Access to the data residing in said databases remains the biggest challenge to enterprises for the following reasons:</p> <ol> <li> SQL Data Sources are inherently heterogeneous because they are acquired with business applications that are in many cases inextricably bound to a particular DBMS engine </li> <li> Data is predictably dirty </li> <li> DBMS vendors ultimately hold the data captive and have traditionally resisted data access standards such as ODBC (*trust me they have, just look at the unprecedented bad press associated with ODBC the only truly platform independent data access API. Then look at how this bad press arose..*) </li> </ol> <p> Enterprises have known from the beginning of modern corporate times that data access, discovery, and manipulation capabilities are inextricably linked to the "Real-time Enterprise" nirvana (hence my use of 0.0 before this becomes 3.0).</p> <p>In my experience, as someone whose operated in the data access and data integration realms since the late '80s, I've painfully observed enterprises pursue, but unsuccessfully attain, full control over enterprise data (the prized asset of any organization) such that data-, information-, knowledge-workers are just a click away from commencing coherent platform and database independent data drill-downs and/or discovery that transcend intranet, internet, and extranet boundaries -- serendipitous interaction with relevant data, without compromise!</p> <p>Okay, situation analysis done, we move on.. </p> <p>At our most recent (<a href="http://esw.w3.org/topic/CambridgeSemanticWebGatherings/Meeting/2007-06-12_Gathering">12th June</a>) monthly <a href="http://esw.w3.org/topic/CambridgeSemanticWebGatherings">Semantic Web Gathering</a>, I unveiled to <a href="http://www.w3.org/People/Berners-Lee/card#i">TimBL</a> and a host of other attendees a simple, but powerful, demonstration of how <a href="http://en.wikipedia.org/wiki/Linked_Data">Linked Data</a>, as an aspect of the <a href="http://www.businessweek.com/technology/content/apr2007/tc20070409_961951.htm">Semantic Data Web</a>, can be applied to enterprise data integration challenges.</p> <h2>Actual SQL to RDF Mapping Demo / Experiment</h2> <h4>Hypothesis</h4> A SQL Schema can be effectively mapped declaratively to RDF such that SQL Rows morph into RDF Instance Data (Entity Sets) based on the Concepts & Properties defined in a Concrete Conceptual Data Model oriented Data Dictionary (<a href="http://www.w3schools.com/rdf/rdf_schema.asp">RDF Schema</a> and/or <a href="http://www.w3schools.com/rdf/rdf_owl.asp">OWL Ontology</a>). In addition, the solution must demonstrate how "Linked Data in the Web" is completely different from "Data on the Web" or "Linked Data on the Web" (btw - <a href="http://kasei.us/people/Tom_Heath/">Tom Heath</a> eloquently unleashed this point in his recent <a href="http://blogs.talis.com/nodalities/2007/06/tom_heath_talks_with_talis_abo.php">podcast interview with Talis</a>). <h4>Apparatus</h4> An Ontology - in this case we simply derived the <a href="http://demo.openlinksw.com/DAV/home/demo/Public/Queries/SQLRDFIntegraton/Explore_Northwind_Ontology.isparql">Northwind Ontology</a> from the XML Schema based CSDL (<a href="http://blogs.msdn.com/adonet/archive/2007/01/30/entity-data-model-part-1.aspx">Conceptual Schema Definition Language</a>) used by Microsoft's public <a href="http://astoria.mslivelabs.com/Default.aspx">Astoria demo</a> (specifically the <a href="http://astoria.mslivelabs.com/termsOfUseNorthwind.aspx?returnURL=Northwind">Northwind Data Services demo</a>). SQL Database Schema - <a href="http://www.microsoft.com/library/media/1033/technet/images/prodtechnol/sql/2000/maintain/sscpop07_big.gif">Northwind</a> (comes bundled with ACCESS, SQL Server, and Virtuoso) comprised of tables such as: <a href="http://www.openlinksw.com/schemas/northwind#Customer">Customer</a>, <a href="http://www.openlinksw.com/schemas/northwind#Employee">Employee</a>, <a href="http://www.openlinksw.com/schemas/northwind#Product">Product</a>, <a href="http://www.openlinksw.com/schemas/northwind#Category">Category</a>, <a href="http://www.openlinksw.com/schemas/northwind#Supplier">Supplier</a>, <a href="http://www.openlinksw.com/schemas/northwind#Shipper">Shipper</a> etc. <a href="http://www.openlinksw.com/virtuoso/">OpenLink Virtuoso</a> - SQL DBMS Engine (although this could have been any <a href="http://en.wikipedia.org/wiki/Open_Database_Connectivity">ODBC</a> or <a href="http://en.wikipedia.org/wiki/Java_Database_Connectivity">JDBC</a> accessible Database), <a href="http://www.openlinksw.com/virtuoso/Whitepapers/pdf/Virtuoso_SQL_to_RDF_Mapping.pdf">SQL-RDF Metaschema Language</a>, HTTP URL-rewriter, WebDAV Engine, and DBMS hosted XSLT processor Client Tools -<a href="http://demo.openlinksw.com/isparql/"> iSPARQL Query Builder</a>, <a href="http://demo.openlinksw.com/DAV/JS/rdfbrowser/index.html">RDF Browser</a> (which could also have been <a href="http://www.w3.org/2005/ajar/tab">Tabulator</a> or<a href="http://sites.wiwiss.fu-berlin.de/suhl/bizer/ng4j/disco/"> DISCO</a> or a standard Web Browser) <h4>Experiment / Demo</h4> <ol> <li> Declaratively map the Northwind SQL Schema to RDF using the Virtuoso Meta Schema Language (see: <a href="http://demo.openlinksw.com/DAV/home/demo/Public/Queries/SQLRDFIntegraton/northwind_sql_rdf.sql">Virtuoso PL based Northwind_SQL_RDF script</a>) </li> <li> Start browsing the data by clicking on the URIs that represent the RDF Data Model Entities resulting from the SQL to RDF Mapping </li> </ol> <h4>Observations</h4> <ol> <li> Via a single Data Link click I was able to obtain specific information about the Customer represented by the URI <a href="http://demo.openlinksw.com/Northwind/Customer/ALFKI">"ALFKI"</a> (act of URI Dereferencing as you would an Object ID in an Object or Object-Relational Database) </li> <li> Via a <a href="http://demo.openlinksw.com/DAV/home/demo/Public/Queries/SQLRDFIntegraton/Explore_Northwind.isparql">Dynamic Data Page </a> I was able to explore all the entity relationships or specific entity data (i.e Exploratory or Entity specific dereferencing) in the Northwind Data Space </li> <li> I was able to perform similar exploration (as per item 2) using our <a href="http://demo.openlinksw.com/DAV/home/demo/Public/Queries/SQLRDFIntegraton/Explore_Northwind_Customer_ALFKI.wqx">OpenLink Browser. </a> </li> </ol> <h4>Conclusions</h4> <p>The vision of data, information, or knowledge at your fingertips is nigh! Thanks to the infrastructure provided by the Semantic Data Web (URIs, <a href="http://en.wikipedia.org/wiki/Resource_Description_Framework">RDF Data Model</a>, variety of RDF Serialization Formats[<a href="http://www.dajobe.org/2004/01/turtle/">1</a>][<a href="http://www.w3.org/DesignIssues/Notation3">2</a>][<a href="http://www.w3.org/TR/2002/WD-rdf-syntax-grammar-20020325/">3</a>], and Shared Data Dictionaries / Schemas / Ontologies [<a href="http://xmlns.com/foaf/spec/">1</a>][<a href="http://rdfs.org/sioc/spec/">2</a>][<a href="http://www.w3.org/TR/swbp-skos-core-guide/">3</a>][<a href="http://musicontology.com/">4</a>][<a href="http://bblfish.net/work/atom-owl/2006-06-06/AtomOwl.html">5</a>]) it's now possible to Virtualize enterprise data from the Physical Storage Level, through the Logical Data Management Levels (Relational), up to a Concrete Conceptual Model (Graph) without operating system, development environment or framework, or database engine lock-in.</p> <h2>Next Steps</h2> <p>We produce a shared ontology for the CRM and Business Reporting Domains. I hope this experiment clarifies how this is quite achievable by converting XML Schemas to RDF Data Dictionaries (RDF Schemas or Ontologies). Stay tuned :-) </p> <p>Also watch <a href="http://news.com.com/1606-2-6189377.html">TimBL amplify and articulate Linked Data value</a> in a recent interview.</p> <h2>Other Related Matters</h2> <p>To deliver a mechanism that facilitates the crystallization of this reality is a contribution of boundless magnitude (as we shall all see in due course). Thus, it is easy to understand why even "her majesty", the queen of England, simply had to get in on the act and <a href="http://www.royal.gov.uk/output/Page1880.asp">appoint TimBL to the "British Order of Merit</a>" :-)</p> <p>Note: All of the demos above now work with IE & Safari (a "remember what Virtuoso is epiphany") by simply putting Virtuoso's DBMS hosted XSLT engine to use :-) This also applies to my earlier collection of demos from the <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=hello%20data%20web&type=text&output=html">Hello Data Web</a> and other <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=.isparql&type=text&output=html">Data Web & Linked Data related demo style posts</a>.</p>
OAT: OpenAjax Alliance Compliant Toolkit (Live Links Version)
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2007-01-29#1129
2007-01-29T16:16:14Z
2007-02-02T10:29:55-05:00
<blockquote> <p> <a href="http://ajaxian.com/archives/oat-openajax-alliance-compliant-toolkit">OAT: OpenAjax Alliance Compliant Toolkit</a>: "</p> <p>Ondrej Zara and his team at Openlink Software have created a Openlink Software JS Toolkit, known as OAT. It is a full-blown JS framework, suitable for developing<br /> rich applications with special focus to data access.</p> <p>OAT works standalone, offers vast number of widgets and has some rarely seen features, such as on-demand library loading (which reduces the total amount of downloaded JS code).</p> <p>OAT is one of the first JS toolkits which show full OpenAjax Alliance conformance: see the appropriate <a href="http://www.openajax.org/member/wiki/InteropFest_2007_March)">wiki page</a> and <a href="http://demo.openlinksw.com/DAV/JS/openajax/HubTest-OATConformance.html">conformance test page</a>.</p> <p>There is a lot to see with this toolkit:</p> <p>You can see some of the widgets in a <a href="http://demo.openlinksw.com/DAV/JS/demo/index.html">Kitchen sink application</a> </p> <p>Sample data access applications:</p> <ul> <li> <a href="http://demo.openlinksw.com/DAV/JS/qbe/index.html">SQL Query By Example</a> </li> <li> <a href="http://demo.openlinksw.com/DAV/JS/formdesigner/index.html">Forms designer</a> </li> <li> <a href="http://demo.openlinksw.com/DAV/JS/designer/index.html">DB Designer</a> </li> </ul> <p>OAT is Open Source and GPLâed over at <a href="http://sourceforge.net/project/showfiles.php?group_id=168143">sourceforge</a> and the team has recently managed to incorporate our OAT data access layer as a<br /> module to <a href="http://demo.openlinksw.com/DAV/JS/dojo-oatstore-demo/test_OATStore_in_FilteringTable.html">dojo datastore</a>.</p> <p>(Via <a href="http://ajaxian.com">Ajaxian Blog</a>.)</p> </blockquote> <p>This is a corrected version of the initial post. Unfortunately, the initial post was inadvertently littered with invalid links :-( Also, since the original post we have released <a href="http://sourceforge.net/mailarchive/forum.php?thread_id=31568932&forum_id=49207">OAT 1.2</a> that includes integration of our iSPARQL QBE into the OAT Form Designer application.</p> <p>Re. Data Access, It is important to note that OAT's Ajax Database Connectivity layers supports data binding to the following data source types:</p> <ol> <li> <a href="http://www.w3.org/TR/rdf-primer/">RDF</a> - via <a href="http://demo.openlinksw.com/DAV/JS/demo/index.html?dav">SPARQL</a> (Query Language, Protocol, and Resultset Serialization formats: <a href="http://en.wikipedia.org/wiki/JDBC">RDF/XML</a>, <a href="http://www.w3.org/DesignIssues/Notation3">RDF/N3</a>, <a href="http://www.dajobe.org/2004/01/turtle/">RDF/Turtle</a>, <a href="http://en.wikipedia.org/wiki/XML">XML</a>, and <a href="http://www.json.org/">JSON</a>)</li> <li> <a href="http://en.wikipedia.org/wiki/SQL">SQL</a> - via <a href="http://www.xmla.org/faq.asp">XMLA</a> (somewhat forgotten <a href="http://en.wikipedia.org/wiki/SOAP">SOAP</a> protocol for SQL Data Access that can sit atop <a href="http://en.wikipedia.org/wiki/Open_Database_Connectivity">ODBC</a>, <a href="http://en.wikipedia.org/wiki/ADO.NET">ADO.NET</a>, <a href="http://en.wikipedia.org/wiki/OLE_DB">OLE-DB</a>, and even <a href="http://en.wikipedia.org/wiki/JDBC">JDBC</a>)</li> <li>XML - via SOAP or <a href="http://en.wikipedia.org/wiki/Representational_State_Transfer">REST</a> style Web Services</li> </ol> In all cases, OAT also provides Data Aware controls for the above that include: <ol> <li> <a href="http://demo.openlnksw.com/DAV/JS/demo/index.html?grid">Tabular Grids</a> </li> <li> <a href="http://demo.openlinksw.com/DAV/JS/demo/index.html?pivot">Pivot Tables</a> </li> <li> <a href="http://demo.openlnksw.com/DAV/JS/demo/index.html?timeline">TimeLines</a> </li> <li> <a href="http://demo.openlinksw.com/DAV/JS/demo/index.html?anchor">Extended Anchor Tags</a> </li> <li> <a href="http://demo.openlnksw.com/DAV/JS/demo/index.html?mashups">Map Service Controls</a> (Google, Yahoo!, OpenLayers, Microsoft Visual Earth)</li> <li> <a href="http://demo.openlinksw.com/DAV/JS/demo/index.html?rdf">SVG based RDF Graph Control</a> (Opera 9.x provides best viewing experience at the current time)</li> </ol> <p>OAT also includes a number of prototype applications that are completely developed using OAT Controls and Libraries:</p> <ol> <li> <a href="http://demo.openlnksw.com/isparql/">Visual SPARQL Query Builder</a> </li> <li> <a href="http://demo.openlinksw.com/DAV/JS/qbe/index.html">Visual SQL Query Builder</a> </li> <li> <a href="http://demo.openlinksw.com/DAV/JS/formdesigner/index.html">Web Forms Designer</a> (includes Drag-Drop usage of Data Aware Controls etc.)</li> <li> <a href="http://demo.openlinksw.com/DAV/JS/designer/index.html">Visual DB Designer</a> </li> </ol> <p>Note: Pick "Local DSN" from page initialization dialog's drop-down list control when prompted</p>
SPARQL End Point Self Description
http://www.openlinksw.com/weblog/oerling/?date=2006-11-21#1085
2006-11-21T14:30:45Z
2008-04-16T16:53:26.000002-04:00
<p> <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xd750548">SPARQL</a> End Point Self Description</p> <p>I was at the ISWC 2006 conference a week back. One of the items discussed there, at least informally, was the topic of SPARQL end point discovery. I have below put together a summary of points that were discussed and of my own views on their possible resolution.</p> <p>This is intended as a start for conversation and as a summary of ideas.</p> <h4>Use Cases</h4> <p>Self-description of end points may serve at least the following purposes:</p> <ul> <li>Query composition - A client must know the capabilities of a server in order to compose suitable queries. <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x17692468">ODBC</a> and <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0xd65d538">JDBC</a> have fairly extensive metadata about each DBMS's <a href="http://dbpedia.org/resource/SQL" id="link-id0xd3993d0">SQL</a> dialect and other properties. These may in part serve as a model.</li> <li>Content Discovery - What is the <a href="http://dbpedia.org/resource/Data" id="link-id0xd6ef6a8">data</a> about? What graphs does the end point contain?</li> <li>Query planning - When making an execution plan for federated queries, it is almost necessary to know the cardinalities of predicates and other things for evaluating join orders and the like.</li> <li>Query targeting - Does it make sense to send a particular query to this end point? The answer may contain things like whether the query could be parsed in the first place, whether it is known to be identically empty, estimated computation time, estimated count of results, optionally a platform dependent query plan.</li> </ul> <p>We will look at each one in turn.</p> <h4>End Point Data and Capabilities</h4> <ul> <li>Server software name and version</li> <li>Must the predicate be constant? Must a rdfs:type be given for a subject? Must a graph be given? Can the graph be a variable known at execution time only?</li> <li>List of supported built-in SPARQL functions.</li> <li>Language extensions - For example, whether there is a full text match predicate.</li> </ul> <h4>Content Discovery</h4> <ul> <li>Name and general description of the purpose of the end point.</li> <li>What organization/individual is maintaining the end point?</li> <li>Contact for technical support, legal or administrative matters. Support and webmaster.</li> <li>Ontologies used. This could be a list of graphs, each with a list of ontologies describing the data therein. Each graph would be listed with a rough estimate of size expressed in triples.</li> <li>Topic - Each graph/ontology pair could have a number of identifiers drawn from standard taxonomies. Examples would be the Latin names of geni and species for biology, the HS code for customs, ISO code for countries, various industry specific classifications of goods and services.</li> </ul> <h4>Query Planning</h4> <p>The end point should give a ballpark cardinality for the following combinations of G, S, P, O.</p> <ul> <li>G</li> <li>G, P</li> <li>G, P, O</li> <li>G, S</li> <li>G, S, P</li> </ul> <p>Based on our experience, these are the most interesting questions but for completeness, the entry point might offer an API allowing specifying a constant or wildcard for each of the four parts of a quad. If the <a href="http://dbpedia.org/resource/Information" id="link-id0x1a59ab78">information</a> is not readily available, "unknown" could be returned, together with the count of triples in the whole end point or the graph, if the graph is specified. Even if the end point does not support real time sampling of data for cardinality estimates, it would at least have an idea of the count of triples per graph, which is still far better than nothing.</p> <h4>Query Feasibility</h4> <p>Given the full SPARQL request, the end point could return the following data, without executing the query itself.</p> <ul> <li>Syntax errors vs. parsed successfully?</li> <li>Are there graph, predicate or subject literals which do not exist in this end point? Does this cause the query result to always be empty? What are these?</li> <li>How many results are expected, according to the SPARQL compiler cost model? This is a row count, if the query is a construct or describe query, this is the count of rows that will go as input to the construct/describe.</li> <li>What is the execution time, as guessed by the SPARQL compiler cost model?</li> <li>Execution plan, in whatever implementation specific, in principle human readable format.</li> </ul> <p>All these elements would be optional.</p> <p>This somewhat overlaps with the optimization questions but it may still be the case that it is more efficient to support a special interface for the optimization related questions.</p>
SPARQL End Point Self Description
http://www.openlinksw.com/blog/vdb/blog/?date=2006-11-21#1087
2006-11-21T14:22:53Z
2008-04-16T16:53:46.000002-04:00
<div> <div style="display:none;">SPARQL End Point Self Description</div> <p> <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x18b5cde0">SPARQL</a> End Point Self Description</p> <p>I was at the ISWC 2006 conference a week back. One of the items discussed there, at least informally, was the topic of SPARQL end point discovery. I have below put together a summary of points that were discussed and of my own views on their possible resolution.</p> <p>This is intended as a start for conversation and as a summary of ideas.</p> <h4>Use Cases</h4> <p>Self-description of end points may serve at least the following purposes:</p> <ul> <li>Query composition - A client must know the capabilities of a server in order to compose suitable queries. <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x171af558">ODBC</a> and <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0xdf3ec80">JDBC</a> have fairly extensive metadata about each DBMS's <a href="http://dbpedia.org/resource/SQL" id="link-id0x1b5a6ec8">SQL</a> dialect and other properties. These may in part serve as a model.</li> <li>Content Discovery - What is the <a href="http://dbpedia.org/resource/Data" id="link-id0xc518ac8">data</a> about? What graphs does the end point contain?</li> <li>Query planning - When making an execution plan for federated queries, it is almost necessary to know the cardinalities of predicates and other things for evaluating join orders and the like.</li> <li>Query targeting - Does it make sense to send a particular query to this end point? The answer may contain things like whether the query could be parsed in the first place, whether it is known to be identically empty, estimated computation time, estimated count of results, optionally a platform dependent query plan.</li> </ul> <p>We will look at each one in turn.</p> <h4>End Point Data and Capabilities</h4> <ul> <li>Server software name and version</li> <li>Must the predicate be constant? Must a rdfs:type be given for a subject? Must a graph be given? Can the graph be a variable known at execution time only?</li> <li>List of supported built-in SPARQL functions.</li> <li>Language extensions - For example, whether there is a full text match predicate.</li> </ul> <h4>Content Discovery</h4> <ul> <li>Name and general description of the purpose of the end point.</li> <li>What organization/individual is maintaining the end point?</li> <li>Contact for technical support, legal or administrative matters. Support and webmaster.</li> <li>Ontologies used. This could be a list of graphs, each with a list of ontologies describing the data therein. Each graph would be listed with a rough estimate of size expressed in triples.</li> <li>Topic - Each graph/ontology pair could have a number of identifiers drawn from standard taxonomies. Examples would be the Latin names of geni and species for biology, the HS code for customs, ISO code for countries, various industry specific classifications of goods and services.</li> </ul> <h4>Query Planning</h4> <p>The end point should give a ballpark cardinality for the following combinations of G, S, P, O.</p> <ul> <li>G</li> <li>G, P</li> <li>G, P, O</li> <li>G, S</li> <li>G, S, P</li> </ul> <p>Based on our experience, these are the most interesting questions but for completeness, the entry point might offer an API allowing specifying a constant or wildcard for each of the four parts of a quad. If the <a href="http://dbpedia.org/resource/Information" id="link-id0x1891a2e0">information</a> is not readily available, "unknown" could be returned, together with the count of triples in the whole end point or the graph, if the graph is specified. Even if the end point does not support real time sampling of data for cardinality estimates, it would at least have an idea of the count of triples per graph, which is still far better than nothing.</p> <h4>Query Feasibility</h4> <p>Given the full SPARQL request, the end point could return the following data, without executing the query itself.</p> <ul> <li>Syntax errors vs. parsed successfully?</li> <li>Are there graph, predicate or subject literals which do not exist in this end point? Does this cause the query result to always be empty? What are these?</li> <li>How many results are expected, according to the SPARQL compiler cost model? This is a row count, if the query is a construct or describe query, this is the count of rows that will go as input to the construct/describe.</li> <li>What is the execution time, as guessed by the SPARQL compiler cost model?</li> <li>Execution plan, in whatever implementation specific, in principle human readable format.</li> </ul> <p>All these elements would be optional.</p> <p>This somewhat overlaps with the optimization questions but it may still be the case that it is more efficient to support a special interface for the optimization related questions.</p> </div>
Virtuoso's SQL Schema to RDF Ontology Mapping Language (1.0)
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2006-10-18#1064
2006-10-18T22:18:00Z
2006-11-17T18:24:25-05:00
<p>A new technical white paper about our declarative language for SQL Schema to RDF Ontology Mapping has just been published.</p> <h2>What is this?</h2> <p>A declarative language adapted from SPARQL's graph pattern language (N3/Turtle) for mapping SQL Data to RDF Ontologies. We currently refer to this as a Graph Pattern based RDF VIEW Definition Language.</p> <h2>Why is it important?</h2> <p>It provides an effective mechanism for exposing existing SQL Data as virtual RDF Data Sets (Graphs) negating the data duplication associated with generating physical RDF Graphs from SQL Data en route to persistence in a dedicated Triple Store. </p> <p>Enterprise applications (traditional and web based) and most Web Applications (Web 1.0 and Web 2.0) sit atop relational databases, implying that SQL/RDF model and data integration is an essential element of the burgeoning "Data Web" (Semantic Web - Layer 1) comprehension and adoption process.</p> <p>In a nutshell, this is a quick route for non disruptive exposure of existing SQL Data to SPARQL supporting RDF Tools and Development Environments.</p> <h2>How does it work?</h2> <h3>RDF Side</h3> <ol> <li>locate one or more Ontologies (e.g FOAF, SIOC, AtomOWL, SKOS etc.) that effectively defines the Concepts (Classes) and Terms (Predicates) to be exposed via your RDF Graph</li> <li>Using the Virtuoso's RDF View Definition Language declare a International Resource Identifier (or URI) for your Graph. Example:<pre>CREATE GRAPH IRI("http://myopenlink.net/dataspace")</pre> </li> <li>Then create Classes (Concepts), Class Properties/Predicates (Memb), and Class Instances (Inst) for the new Graph. Example: <pre>CREATE IRI CLASS odsWeblog:feed_iri "http://myopenlink.net/dataspace/kidehen/weblog/MyFeeds" ( in memb varchar not null, in inst varchar not null)</pre> </li> </ol> <h3>SQL Side</h3> <ol> <li>If Virtuoso isn't your SQL Data Store, Identify the ODBC or JDBC SQL data source(s) containing the SQL data to be mapped to RDF and then link the relevant tables into Virtuoso's Virtual DBMS Layer</li> <li>Then use the RDF View Definition Language's graph pattern feature to generate SQL to RDF Mapping Template for your Graph. As shown in this <a href="http://www.usnet.private:8889/wiki/main/Main/VOSSQLRDF#MappingOdsBlogToAtomOwlExample">ODS Weblog -> AtomOWL Mapping example</a>.</li> </ol>
Object Relational Rediscovered?
http://www.openlinksw.com/blog/vdb/blog/?date=2006-07-13#1003
2006-07-13T12:33:32Z
2008-04-16T16:13:26-04:00
<div> <div style="display:none;">Object Relational Rediscovered?</div> <p>I have recently read some of Microsoft's <a href="http://dbpedia.org/resource/ADO.NET" id="link-id0x173cea20">ADO</a> .NET 3 papers. I am reminded of the distant past when I designed Kubl, which later became OpenLink <a href="http://virtuoso.openlinksw.com" id="link-id0x18bdfe68">Virtuoso</a>. So I will reminisce and speculate a little.</p> <p>So now is the time when polymorphic queries and mixing relational style joins and object style navigation become politically acceptable and even recommended and there finally is a workable solution to having a foreign key in the database and a pointer or set of pointers in the client application. Not to mention change tracking so as to be able to update in-memory <a href="http://dbpedia.org/resource/Data" id="link-id0xd6f0ae0">data</a> structures and commit a delta against the database without explicit update statements.</p> <p>All these questions existed already in the mid 90s and earlier. Since I was coming from OO and LISP into the database world, I even felt these questions to be important. The solution in the earliest Kubl was to have inheritance between tables, what became the <a href="http://dbpedia.org/resource/SQL" id="link-id0xddcdac0">SQL</a> 2K <code>UNDER</code> clause, and a virtual column called <code>_ROW</code> that would select a serialization of the primary key entry. Then there was the function <code>row_key()</code>, which when applied to a <code>_ROW</code> virtual column would return a database-wide unique identifier of the row, containing the key info and the key part values plus which subtable of the table was at hand. Then there was a function for dereferencing a <code>row_key</code> for getting the <code>_ROW</code>. And one could store <code>row_keys</code> into columns and dereference these in queries. Within SQL, one could use the <code>row_column</code> function to extract individual column values from a <code>row_key</code> or <code>_ROW</code>.</p> <p>This was all fine server side. But we also had a client for Franz Inc.'s Allegro Common Lisp that talked to Kubl's <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0xde2c348">ODBC</a> listener. This client had the basic statements and prepared statements and result sets, parameters and array parameters, a little like <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x156409f8">JDBC</a> does now. But the extra was that we could do a mapping between a Lisp struct or object and a database key, so the <code>_ROW</code> would automatically materialize into the Lisp struct or class instance. And the mapping between these materializations and the <code>row_keys</code> identifying them in the database were kept in a thread environment called object space. Updates could be relational-style <code>UPDATEs</code> or consist of putting a <code>_ROW</code> serialization in database format back into the Kubl store with a single SQL function.</p> <p>This was different from just storing object serializations into LOB columns, as is often done, insofar as the object classes and data members were really database tables and columns, thus native to the DBMS, not just opaque data to be processed client-side only.</p> <p>So it was then possible to program a little like is shown in the ADO .NET 3 demos today, some ten years later.</p> <p>Some of these functions still exist in Virtuoso, albeit in a deprecated state, and there is no client that can use these to any advantage. Indeed, we dropped this line of work when Kubl became Virtuoso, mostly because there was no standard and no client applications that would use such features. Instead, we concentrated on virtual <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x175a7b10">RDBMS</a>, transparently accessing any third party data via ODBC.</p> <p>Now however, as objects, both native SQL and Java and .NET, have become mainstream citizens of relational databases in general, Virtuoso and otherwise, and as Microsoft has legitimized accessing whole objects and not only scalar columns in result sets as part of ADO .NET 3, these things might be worth a second look.</p> </div>
Object Relational Rediscovered?
http://www.openlinksw.com/weblog/oerling/?date=2006-07-13#1002
2006-07-13T11:38:41Z
2008-04-16T16:13:08.000001-04:00
<p>I have recently read some of Microsoft's <a href="http://dbpedia.org/resource/ADO.NET" id="link-id0x151feed0">ADO</a> .NET 3 papers. I am reminded of the distant past when I designed Kubl, which later became OpenLink <a href="http://virtuoso.openlinksw.com" id="link-id0xd1e5a50">Virtuoso</a>. So I will reminisce and speculate a little.</p> <p>So now is the time when polymorphic queries and mixing relational style joins and object style navigation become politically acceptable and even recommended and there finally is a workable solution to having a foreign key in the database and a pointer or set of pointers in the client application. Not to mention change tracking so as to be able to update in-memory <a href="http://dbpedia.org/resource/Data" id="link-id0xd6877c0">data</a> structures and commit a delta against the database without explicit update statements.</p> <p>All these questions existed already in the mid 90s and earlier. Since I was coming from OO and LISP into the database world, I even felt these questions to be important. The solution in the earliest Kubl was to have inheritance between tables, what became the <a href="http://dbpedia.org/resource/SQL" id="link-id0xc703880">SQL</a> 2K <code>UNDER</code> clause, and a virtual column called <code>_ROW</code> that would select a serialization of the primary key entry. Then there was the function <code>row_key()</code>, which when applied to a <code>_ROW</code> virtual column would return a database-wide unique identifier of the row, containing the key info and the key part values plus which subtable of the table was at hand. Then there was a function for dereferencing a <code>row_key</code> for getting the <code>_ROW</code>. And one could store <code>row_keys</code> into columns and dereference these in queries. Within SQL, one could use the <code>row_column</code> function to extract individual column values from a <code>row_key</code> or <code>_ROW</code>.</p> <p>This was all fine server side. But we also had a client for Franz Inc.'s Allegro Common Lisp that talked to Kubl's <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0xdbceda0">ODBC</a> listener. This client had the basic statements and prepared statements and result sets, parameters and array parameters, a little like <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x181a27b0">JDBC</a> does now. But the extra was that we could do a mapping between a Lisp struct or object and a database key, so the <code>_ROW</code> would automatically materialize into the Lisp struct or class instance. And the mapping between these materializations and the <code>row_keys</code> identifying them in the database were kept in a thread environment called object space. Updates could be relational-style <code>UPDATEs</code> or consist of putting a <code>_ROW</code> serialization in database format back into the Kubl store with a single SQL function.</p> <p>This was different from just storing object serializations into LOB columns, as is often done, insofar as the object classes and data members were really database tables and columns, thus native to the DBMS, not just opaque data to be processed client-side only.</p> <p>So it was then possible to program a little like is shown in the ADO .NET 3 demos today, some ten years later.</p> <p>Some of these functions still exist in Virtuoso, albeit in a deprecated state, and there is no client that can use these to any advantage. Indeed, we dropped this line of work when Kubl became Virtuoso, mostly because there was no standard and no client applications that would use such features. Instead, we concentrated on virtual <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0xd06fd40">RDBMS</a>, transparently accessing any third party data via ODBC.</p> <p>Now however, as objects, both native SQL and Java and .NET, have become mainstream citizens of relational databases in general, Virtuoso and otherwise, and as Microsoft has legitimized accessing whole objects and not only scalar columns in result sets as part of ADO .NET 3, these things might be worth a second look.</p>
Contd: Ajax Database Connectivity Demos
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2006-06-01#988
2006-06-02T02:48:00Z
2006-06-22T08:56:58-04:00
<p> Last week I put out a series of screencast style demos that sought to demonstrate the core elements of our soon to be released Javascript Toolkit called OAT (<a href="http://www.openlinksw.com/oat/">OpenLink Ajax Toolkit</a>) and its Ajax Database Connectivity layer. </p> <p> The screencasts covered the following functionality realms: </p> <ol> <li> <a href="http://www.openlinksw.com/blog/%7Ekidehen/index.vspx?page=&id=982">SQL Query By Example (basic)</a> </li> <li> <a href="http://www.openlinksw.com/blog/%7Ekidehen/index.vspx?page=&id=983">SQL Query By Example (advanced - pivot table construction)</a> </li> <li> <a href="http://www.openlinksw.com/blog/%7Ekidehen/index.vspx?page=&id=981">Web Form Design (basic database driven map based mashup)</a> </li> <li> <a href="http://www.openlinksw.com/blog/%7Ekidehen/index.vspx?page=&id=985">Web Form Design (advanced database driven map based mashup)</a> </li> </ol> <p> To bring additional clarity to the screencasts demos and OAT in general, I have saved a number of documents that are the by products of activities in the screenvcasts: </p> <ol> <li> <a href="http://demo.openlinksw.com/public_demos/queries/customer_qry1.xml">Live XML Document produced using SQL Query By Example (basic)</a> (you can use drag and drop columns across the grid to reorder and sort presentation)</li> <li> <a href="http://demo.openlinksw.com/public_demos/reports/Pivots/employee_sales_by_ship_country_pivot.xml">Live XML Document produced using QBE and Pivot Functionality</a> (you can drag and drop the aggregate columns and rows to create your own views etc..)</li> <li> <a href="http://demo.openlinksw.com/public_demos/reports/MapMashups/country_flags_google_frm2.xml">Basic database driven map based mashup</a> (works with FireFox, Webkit, Camino; click on pins to see national flag)</li> <li> <a href="http://demo.openlinksw.com/public_demos/reports/MapMashups/employee_sales_by_ship_country_pivot_google.xml">Advanced database driven map based mashup</a> (works with FireFox, Webkit, Camino; records, 36, 87, and 257 will unveil pivots via lookup pin)</li> </ol> <p> Notes: </p> <ul> <li>“Advanced”, as used above, simply means that I am embedding images (employee photos and national flags) and a database driven pivot into the map pins that serve as details lookups in classic SQL master/details type scenarios.</li> <li>The “Ajax Call In Progress..” dialog is there to show live interaction with a remote database (in this case <a href="http://virtuoso.openlinksw.com">Virtuoso</a> but this could be any ODBC, JDBC, OLEDB, ADO.NET, or XMLA accessible data source)</li> <li>The data access magic source (if you want to call it that) is XMLA - a standard that has been in place for years but completely misunderstood and as a result under utilized</li> </ul> <p> You can see a full collection of saved documents at the following locations: </p> <ul> <li> <a href="http://demo.openlinksw.com/public_demos/reports/MapMashups/">My Mashups demo directory</a> (Google and Yahoo! demo variants but note these do not work with Safari or IE at the current time. IE7 issues will be resolved in the next day or so) </li> <li> <a href="http://demo.openlinksw.com/public_demos/reports/Pivots/">My Pivots demo directory</a> (other Pivots will be added as I build and save them) </li> <li> <a href="http://demo.openlinksw.com/public_demos/queries/">My Saved Queries</a> (a collection of saved QBE generated queries)</li> </ul>
Screencast: Yahoo! Maps variation of Ajax Database Connectivity Maps Mash-up
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2006-05-26#986
2006-05-26T22:49:00Z
2006-06-22T08:56:58-04:00
This is a Yahoo! maps variation of the <a href="http://www.openlinksw.com/blog/%7Ekidehen/index.vspx?page=&id=985">Google Maps based Forms Designer mash-up screencast</a>.<br /> <br /> <br />
Screencast: Building Database Centric Web 2.0 Mash-ups using Ajax Database Connectivity
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2006-05-26#985
2006-05-26T22:38:00Z
2006-06-22T08:56:58-04:00
This screencast covers the actual codeless process of building a database centric Web 2.0 mash-up using OAT's database-aware Forms Designer. This is basically the simplicity of Paradox or Microsoft ACCESS form building delivered via Ajax without any database or operating system lock-in. This demo uses the Google Mapping Service (note: there is a <a href="http://www.openlinksw.com/dataspace/%7Ekidehen/blog/public/Screencasts/oat-formdesigner-mashup-yahoo-maps-demo1.mov">Yahoo! Mapping Service screencast demo</a> that follows this post). Also note that fact that in this demonstration I actually incorporate the Pivot building functionality from an earlier <a href="http://www.openlinksw.com/blog/%7Ekidehen/index.vspx?page=&id=983">Ajax based Pivot Building screencast</a>.<br /> <br />
Screencast: Using a Live Report (mash-up) that exploits AJAX Database Connectivity
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2006-05-26#984
2006-05-26T22:27:00Z
2006-06-22T08:56:58-04:00
Another demo. This time around you are looking at a quick and dirty mashup assembled using the <a href="http://www.openlinksw.com/blog/%7Ekidehen/index.vspx?page=&id=981">OAT FormDesigner</a>. There is a follow-on demo that shows how this was assembled (no coding whatsoever!).<br />
Building Pivot Tables using Ajax Database Connectivity
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2006-05-26#983
2006-05-26T22:08:00Z
2008-02-04T20:43:35.000004-05:00
This screencast demo (enclosure attached) is a continuation from my earlier <a href="http://www.openlinksw.com/blog/%7Ekidehen/index.vspx?page=&id=982">Ajax and QBE screencast</a> demo. This time the focus is on building Excel like Pivot tables using data exposed via Ajax Database Connectivity.<br />
Screencast: Ajax Database Connectivity and SQL Query By Example
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2006-05-26#982
2006-05-26T21:59:00Z
2006-06-22T08:56:58-04:00
AJAX Database Connectivity is the Data Access Component of OAT (<a href="http://www.openlinksw.com/oat/">OpenLink AJAX Toolkit</a>). It's basically an <a href="http://www.xmla.org/">XML for Analysis</a> (XMLA) client that enables the development and deployment of database independent Rich Internet Applications (RIAs). Thus, you can now develop database centric AJAX applications without lock-in at the Operating System, Database Connectivity mechanism (ODBC, JDBC, OLEDB, ADO.NET), or back-end Database levels. <br /> <br />XMLA has been around for a long time. Its fundamental goal was to provide Web Applications with Tabular and Multi-dimensional data access before it fell off the radar (a story too long to tell in this post).<br /> <br />AJAX Database connectivity only requires your target DBMS to be XMLA (direct), ODBC, JDBC, OLEDB, or ADO.NET accessible. <br /> <br />I have attached a Query By Example (QBE) screencast movie enclosure to this post (should you be reading this post Web 1.0 style). The demo shows how Paradox-, Quattro Pro-, Access-, and MS Query-like user friendly querying is achieved using AJAX Database Connect Connectivity<br /> <br />
SPARQL Parameterized Queries (Virtuoso using SPARQL in SQL)
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2006-05-11#973
2006-05-11T18:54:47Z
2006-06-22T08:56:58-04:00
<h2>SPARQL with SQL (Inline) </h2> <p>Virtuoso extends its SQL3 implementation with syntax for integrating SPARQL into queries and subqueries.Thus, as part of a SQL SELECT query or subquery, one can write the SPARQL keyword and a SPARQL query as part of query text processed by Virtuoso's SQL Query Processor.</p> <h4>Example 1 (basic) : </h4> <p>Using Virtuoso's Command line or the Web Based ISQL utility type in the following (note: "SQL>" is the command line prompt for the native ISQL utility): </p> <pre>SQL> sparql select distinct ?p where { graph ?g { ?s ?p ?o } };</pre> <p>Which will return the following: </p> <blockquote> <pre> p varchar ---------- http://example.org/ns#b http://example.org/ns#d http://xmlns.com/foaf/0.1/name http://xmlns.com/foaf/0.1/mbox ... </pre> </blockquote> <h4>Example 2 (a subquery variation):</h4> <pre>SQL> select distinct subseq (p, strchr (p, '#')) as fragment from (sparql select distinct ?p where { graph ?g { ?s ?p ?o } } ) as all_predicates where p like '%#%' ;</pre> <blockquote> <pre> fragment varchar ---------- #query #data #name #comment ...</pre> </blockquote> <h3>Parameterized Queries:</h3> <p>You can pass parameters to a SPARQL query using a Virtuoso-specific syntax extension. '??' or '$?' indicates a positional parameter similar to '?' in standard SQL. '??' can be used in graph patterns or anywhere else where a SPARQL variable is accepted. The value of a parameter should be passed in SQL form, i.e. this should be a number or an untyped string. An IRI ID can not be passed, but an absolute IRI can. Using this notation, a dynamic SQL capable client (ODBC, JDBC, ADO.NET, OLEDB, XMLA, or others) can execute parametrized SPARQL queries using parameter binding concepts that are common place in dynamic SQL. Which implies that existing SQL applications and development environments (PHP, Ruby, Python, Perl, VB, C#, Java, etc.) are capable of issuing SPARQL queries via their existing SQL bound data access channels against RDF Data stored in Virtuoso. </p> <p>Note: This is the Virtuoso equivalent of a <a href="http://seaborne.blogspot.com/2006/05/parameterized-queries_07.html">recently published example using Jena </a>(a Java based RDF Triple Store).</p> <h3>Example:</h3> <p>Create a Virtuoso Function by execting the following: </p> <pre>SQL> create function param_passing_demo (); { declare stat, msg varchar; declare mdata, rset any; exec ('sparql select ?s where { graph ?g { ?s ?? ?? }}', stat, msg, vector ('http://www.w3.org/2001/sw/DataAccess/tests/data/Sorting/sort-0#int1', 4 ), -- Vector of two parameters 10, -- Max. result-set rows mdata, -- Variable for handling result-set metadata rset -- Variable for handling query result-set ); return rset[0][0]; } </pre> Test new "param_passing_demo" function by executing the following: <br /> <pre>SQL> select param_passing_demo (); </pre> <p>Which returns: </p> <blockquote> <pre> callret VARCHAR _______________________________________________________________________________</pre> <pre>http://www.w3.org/2001/sw/DataAccess/tests/data/Sorting/sort-0#four</pre> <pre>1 Rows. -- 00000 msec.</pre> </blockquote> <h3>Â </h3> <h3>Using SPARQL in SQL Predicates:</h3> <p>A SPARQL ASK query can be used as an argument of the SQL EXISTS predicate.</p> <pre>create function sparql_ask_demo () returns varchar { if (exists (sparql ask where { graph ?g { ?s ?p 4}})) return 'YES'; else return 'NO'; }; </pre> <p> <br /> Test by executing: </p> <pre>SQL> select sparql_ask_demo (); </pre> <p>Which returns:</p> <pre>_________________________ YES</pre>
"Free" Databases: Express vs. Open-Source RDBMSs
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2006-05-05#968
2006-05-05T16:02:17Z
2006-07-21T07:21:57.000006-04:00
<p>Very detailed and insightful peek into the state of affairs re. database engines (Open & Closed Source).</p> <p>I added the missing piece regarding the "Virtuoso Conductor" (the Web based Admin UI for Virtuoso) to the original post below. I also added a link to our live SPARQL Demo so that anyone interested can start playing around with SPARQL and SPARQL integrated into SQL right away.</p> <p>Another good thing about this post is the vast amount of valuable links that it contains. To really appreciate this point simply visit my Linkblog (excuse the current layout :-) - a Tab if you come in via the front door of this <a href="http://www.infoworld.com/opinions/index.html">Data Space</a> (what I used to call <a href="http://www.openlinksw.com/blog/%7Ekidehen/">My Weblog Home Page</a>).</p> <blockquote> <p> <a href="http://oakleafblog.blogspot.com/2006/05/free-databases-express-vs-open-source.html">"Free" Databases: Express vs. Open-Source RDBMSs</a>: "<span style="font-family: verdana;">Open-source relational database management systems (RDBMSs) are gaining IT mindshare at a rapid pace. As an example, <em>BusinessWeek</em>'s February 6, 2006 '</span> <a href="http://www.businessweek.com/technology/content/feb2006/tc20060206_918648.htm"><span style="font-family: verdana;">Taking On the Database Giants</span> </a><span style="font-family: verdana;">' article asks 'Can open-source upstarts compete with Oracle, IBM, and Microsoft?' and then provides the answer: 'It's an uphill battle, but customers are starting to look at the alternatives.'</span> <br /> <span style="font-family: Verdana;"></span> <br /> <span style="font-family: Verdana;">There's no shortage of open-source alternatives to look at. The <em>BusinessWeek</em> article concentrates on <a href="http://www.mysql.com/">MySQL</a>, which <em>BW</em> says 'is trying to be the Ikea of the database world: cheap, needs some assembly, but has a sleek, modern design and does the job.' The article also discusses <a href="http://www.postgresql.org/">Postgre[SQL]</a> and <a href="http://www.ingres.com/products/Prod_Ingres_2006.html">Ingres</a>, as well as <a href="http://www.enterprisedb.com/">EnterpriseDB</a>, an Oracle clone created from PostgreSQL code*. Sun includes <a href="http://www.sun.com/software/solaris/postgres.jsp">PostgreSQL with Solaris 10</a> and, as of April 6, 2006, with <a href="http://docs.sun.com/app/docs/doc/819-2183/6n4g726uc?a=view">Solaris Express</a>.**</span> <br /> <span style="font-family: Verdana;"></span> <br /> <span style="font-family: Verdana;"><span style="font-size: 85%;">*Frank Batten, Jr., the investor who originally funded Red Hat, invested a reported </span> <a href="http://www.theinquirer.net/?article=28201"><span style="font-size: 85%;">$16 million into Great Bridge</span> </a><span style="font-size: 85%;"> with the hope of making a business out of providing paid support to PostgreSQL users. </span> <a href="http://news.com.com/2100-1001-272715.html"><span style="font-size: 85%;">Great Bridge stayed in business only 18 months</span> </a><span style="font-size: 85%;">, having </span> <a href="http://news.com.com/2100-1001-268915.html"><span style="font-size: 85%;">missed an opportunity to sell the business to Red Hat</span> </a><span style="font-size: 85%;"> and finding that selling </span> <a href="http://news.com.com/2100-1001-269729.html"><span style="font-size: 85%;">$50,000-per-year support packages</span> </a><span style="font-size: 85%;"> for an open-source database wasn't easy. As Batten concluded, 'We could not get customers to pay us big dollars for support contracts.' Perhaps EnterpriseDB will be more successful with a choice of </span> <a href="http://www.enterprisedb.com/shop.do?cID=10000&pID=10001"><span style="font-size: 85%;">$5,000, $3,000, or $1,000 annual support subscriptions</span> </a><span style="font-size: 85%;">.</span> </span> <br /> <span style="font-family: Verdana;"></span><span style="font-family: Verdana;"></span><span style="font-family: Verdana;"></span> <br /> <span style="font-family: Verdana; font-size: 85%;">**Interestingly, <a href="http://www.sun.com/smi/Press/sunflash/2005-11/sunflash.20051115.4.xml">Oracle announced in November 2005</a> that Solaris 10 is 'its preferred development and deployment platform for most x64 architectures, including x64 (x86, 64-bit) AMD Opteron and Intel Xeon processor-based systems and Sun's UltraSPARC(R)-based systems.'</span> <br /> <br /> <span style="font-family: Verdana;">There is a surfeit of reviews of current MySQL, PostgreSQL andâto a lesser extentâIngres implementations. These three open-source RDBMSs come with their own or third-party management tools. These systems compete against free versions of commercial (proprietary) databases: <a href="http://msdn.microsoft.com/vstudio/express/sql/">SQL Server 2005 Express Edition</a> (and its MSDE 2000 and 1.0 predecessors), <a href="http://www.oracle.com/technology/products/database/xe/index.html" target="_blank">Oracle Database 10g Express Edition</a>, <a href="http://www-306.ibm.com/software/data/db2/udb/db2express/download.html" target="_blank">IBM DB2 Express-C</a>, and <a href="http://www.sybase.com/linux_promo" target="_blank">Sybase ASE Express Edition for Linux</a> where database size and processor count limitations aren't important. Click <a href="http://www.ftponline.com/special/sqlserver/rjennings-overview/table4.aspx">here</a> for a summary of recent <em>InfoWorld</em> reviews of the full versions of these four databases plus MySQL, which should be valid for Express editions also. The <a href="http://www.ftponline.com/special/sqlserver/">FTPOnline Special Report</a> article, 'Microsoft SQL Server Turns 17,' that contains the preceding table is <a href="http://www.ftponline.com/special/sqlserver/rjennings-overview/">here</a> (requires registration.)</span> <br /> <br /> </p> <p> <strong><span style="font-family: verdana;">SQL Server 2005 Express Edition SP-1 Advanced Features</span> </strong> </p> <p> <span style="font-family: Verdana;"><a href="http://www.microsoft.com/downloads/details.aspx?familyid=4C6BA9FD-319A-4887-BC75-3B02B5E48A40&displaylang=en">SQL Server 2005 Express Edition with Advanced Features</a> enhances SQL Server 2005 Express Edition (SQL Express or SSX) dramatically, so it deserves special treatment here. SQL Express gains full text indexing and now supports SQL Server Reporting Services (SSRS) on the local SSX instance. The SP-1 with Advanced Features setup package, which Microsoft released on April 18, 2006, installs the release version of SQL Server Management Studio Express (SSMSE) and the full version of Business Intelligence Development Studio (BIDS) for designing and editing SSRS reports. My '<a href="http://www.ftponline.com/special/sqlserver/rjennings-sp1/">Install SP-1 for SQL Server 2005 and Express</a>' article for FTPOnline's <a href="http://www.ftponline.com/special/sqlserver/">SQL Server Special Report</a> provides detailed, illustrated installation instructions for and related information about the release version of SP-1. SP-1 makes SSX the most capable of all currently available Express editions of commercial RDBMSs for Windows.</span> </p> <p> <strong><span style="font-family: verdana;">OpenLink Software's Virtuoso Open-Source Edition</span> </strong> <br /> <span style="font-family: verdana;"></span> <br /> <span style="font-family: verdana;"><a href="http://openlinksw.com/">OpenLink Software</a> announced an <a href="http://virtuoso.openlinksw.com/wiki/main/Main/">open-source version</a> of it's <a href="http://virtuoso.openlinksw.com/">Virtuoso Universal Server</a> commercial DBMS on April 11, 2006. On the initial date of this post, May 2, 2006, Virtuoso Open-Source Edition (VOS) was virtually under the radar as an open-source product. According to <a href="http://www.openlinksw.com/press/VOSPressRelease.htm">this press release</a>, the new edition includes:</span> <span style="font-family: Verdana;"></span> </p> <blockquote> <span style="font-family: Verdana;"></span> </blockquote> <blockquote></blockquote> <blockquote></blockquote> <ul> <li> <a href="http://demo.openlinksw.com/sparql_demo/">SPARQL compliant RDF Triple Store</a> </li> <li>SQL-200n Object-Relational Database Engine (SQL, XML, and Free Text) </li> <li>Integrated BPEL Server and Enterprise Service Bus</li> <li>WebDAV and Native File Server </li> <li>Web Application Server that supports PHP, Perl, Python, ASP.NET, JSP, etc. </li> <li>Runtime Hosting for Microsoft .NET, Mono, and Java </li> </ul>VOS only lacks the virtual server and replication features that are offered by the commercial edition. VOS includes a Web-based administration tool called the "Virtuoso Conductor" According to <a href="http://www.openlinksw.com/blog/%7Ekidehen/index.vspx?page=&id=951&sid=&realm=">Kingsley Idehen's Weblog</a>, 'The Virtuoso build scripts have been successfully tested on Mac OS X (Universal Binary Target), Linux, FreeBSD, and Solaris (AIX, HP-UX, and True64 UNIX will follow soon). A Windows Visual Studio project file is also in the works (ETA some time this week).'<br /> <br /> <em>InfoWorld</em>'s Jon Udell has tracked Virtuoso's progress since <a href="http://www.infoworld.com/article/02/04/12/020415plvirtuoso_1.html">2002</a>, with an <a href="http://www.infoworld.com/article/03/03/21/12virtuoso_1.html">additional article in 2003</a> and a <a href="http://weblog.infoworld.com/udell/2006/04/28.html#a1437">one-hour podcast with Kingsley Idehen</a> on April 26, 2006. A major talking point for Virtuoso is its support for Atom 0.3 syndication and publication, Atom 1.0 syndication and (forthcoming) publication, and future support for Google's <a href="http://code.google.com/apis/gdata/overview.html">GData protocol</a>, as mentioned in <a href="http://www.openlinksw.com/blog/%7Ekidehen/index.vspx?page=&id=965">this Idehen post</a>. Yahoo!'s <a href="http://jeremy.zawodny.com/blog/archives/006687.html">Jeremy Zawodny</a> points out that the 'fingerprints' of <a href="http://oakleafblog.blogspot.com/2005/11/adam-bosworth-learning-from-web-and.html">Adam Bosworth</a>, Google's VP of Engineering and the primary force behind the development of Microsoft Access, 'are all over GData.' Click <a href="http://search.blogger.com/?as_q=bosworth&ie=UTF-8&ui=blg&bl_url=oakleafblog.blogspot.com&x=50&y=10">here</a> to display a list of all OakLeaf posts that mention Adam Bosworth.<br /> <br />One application for the GData protocol is querying and updating the Google Base database independently of the Google Web client, as mentioned by Jeremy: 'It's not about building an easier onramp to Google Base. ... Well, it is. But, again, that's the small stuff.' Click <a href="http://search.blogger.com/?as_q=%22google+base%22&ie=UTF-8&x=50&y=9&q=%22google+base%22+blogurl:oakleafblog.blogspot.com&filter=0&ui=blg&sa=N&start=0">here</a> for a list of posts about my experiences with Google Base. Watch for a future OakLeaf post on the subject as the GData APIs gain ground.<br /> <span style="font-family: Verdana;"></span> <br /> <span style="font-family: Verdana;"><strong>Open-Source and Free Embedded Database Contenders</strong> </span> <br /> <span style="font-family: Verdana;"></span> <br /> <span style="font-family: Verdana;">Open-source and free embedded SQL databases are gaining importance as the number and types of mobile devices and OSs proliferate. Embedded databases usually consist of Java classes or Windows DLLs that are designed to minimize file size and memory consumption. Embedded databases avoid the installation hassles, heavy resource usage and maintenance cost associated with client/server RDBMSs that run as an operating system service.</span> <br /> <br /> <span style="font-family: Verdana;">Andrew Hudson's December 2005 '<a href="http://www.theinquirer.net/?article=28201">Open Source databases rounded up and rodeoed</a>' review for The Enquirer provides brief descriptions of one commercial and eight open source database purveyors/products: Sleepycat, MySQL, PostgreSQL, Ingres, InnoBase, Firebird, IBM Cloudscape (a.k.a, Derby), Genezzo, and Oracle. Oracle <a href="http://www.sleepycat.com/">Sleepycat</a>* isn't an SQL Database, Oracle <a href="http://www.innodb.com/index.php">InnoDB</a>* is an OEM database engine that's used by MySQL, and <a href="http://www.genezzo.com/">Genezzo</a> is a multi-user, multi-server distributed database engine written in Perl. These special-purpose databases are beyond the scope of this post.</span> <br /> <br /> <span style="font-family: Verdana;"><span style="font-size: 85%;">* Oracle <a href="http://www.oracle.com/sleepycat/index.html">purchased Sleepycat Software, Inc. in February 2006</a> and </span> <a href="http://www.oracle.com/innodb/index.html"><span style="font-size: 85%;">purchased Innobase OY in October 2005</span> </a><span style="font-size: 85%;">. The press release states: 'Oracle intends to continue developing the InnoDB technology and expand our commitment to open source software.' </span> </span> <br /> <span style="font-family: Verdana; font-size: 85%;"></span> <br /> <span style="font-family: Verdana;"> <a href="http://db.apache.org/derby/"><strong>Derby</strong> </a> is an open-source release by the <a href="http://www.apache.org/">Apache Software Foundation</a> of the <a href="http://www.infoworld.com/article/04/08/03/HNcloudscape_1.html">Cloudscape Java-based database that IBM acquired</a> when it bought Informix in 2001. IBM offers a commercial release of Derby as <a href="http://www-128.ibm.com/developerworks/db2/library/techarticle/dm-0408cline/">IBM Cloudscape 10.1</a>. Derby is a Java class library that has a relatively light footprint (2 MB), which make it suitable for <a href="http://www-128.ibm.com/developerworks/db2/library/techarticle/dm-0503stumpf/">client/server synchronization</a> with the IBM DB2 Everyplace Sync Server in <a href="http://www-128.ibm.com/developerworks/library/wi-cloud/">mobile applications</a>. The IBM DB2 Everyplace Express Edition isn't open source or free*, so it doesn't qualify for this post. The same is true for the corresponding Sybase SQL Anywhere components.**</span> <br /> <br /> <br /> <p> <span style="font-family: verdana; font-size: 85%;">* IBM DB2 Everyplace Express Edition with synchronization costs $379 per server (up to two processors) and $79 per user. DB2 Everyplace Database Edition (without DB2 synchronization) is $49 per user. (Prices are based on those when </span> <a href="http://news.earthweb.com/wireless/article.php/3107101"><span style="font-family: verdana; font-size: 85%;">IBM announced version 8</span> </a><span style="font-family: verdana; font-size: 85%;"> in November 2003.)</span> </p> <p> <span style="font-family: verdana; font-size: 85%;">** Sybase's iAnywhere subsidiary calls SQL Anywhere 'the industry's leading mobile database.' A Sybase SQL Anywhere Personal DB seat license with synchronization to SQL Anywhere Server is $119; the cost without synchronization wasn't available from the Sybase Web site. Sybase SQL Anywhere and IBM DB2 Everyplace perform similar replication functions.</span> </p> <p> <span style="font-family: Verdana;">Sun's <a href="http://developers.sun.com/prodtech/javadb/"><strong>Java DB</strong></a>, another commercial version of Derby, comes with the <a href="http://www.sun.com/software/solaris/">Solaris Enterprise Edition</a>, which bundles Solaris 10, the Java Enterprise System, developer tools, desktop infrastructure and N1 management software. A recent Between the Lines blog entry by ZDNet's David Berlind waxes enthusiastic over the use of <a href="http://blogs.zdnet.com/BTL/?p=2298">Java DB embedded in a browser</a> to provide offline persistence. RedMonk analyst <a href="http://www.redmonk.com/jgovernor/archives/001151.html">James Governor</a> and <em>eWeek</em>'s <a href="http://www.eweek.com/article2/0,1895,1902407,00.asp">Lisa Vaas</a> wrote about the use of Java DB as a local data store when <a href="http://www.sauria.com/blog/2005/12/13#1440">Tim Bray announced Sun's Derby derivative</a> and <a href="http://blogs.sun.com/roller/page/FrancoisOrsini?entry=derby_apachecon_demo">Francois Orsini</a> demonstrated Java DB embedded in the Firefox browser at the ApacheCon 2005 conference.</span> <br /> <span style="font-family: Verdana;"></span> <br /> <span style="font-family: Verdana;"> <a href="http://www.firebirdsql.org/"><strong>Firebird</strong> </a> is derived from Borland's InterBase 6.0 code, the first commercial relational database management system (RDBMS) to be released as open source. Firebird has excellent support for SQL-92 and comes in three versions: Classic, SuperServer and Embedded for Windows, Linux, Solaris, HP-UX, FreeBSD and MacOS X. The embedded version has a 1.4-MB footprint. Release Candidate 1 for Firebird 2.0 became available on March 30, 2006 and is a major improvement over earlier versions. <a href="http://www.borland.com/us/products/interbase/index.html">Borland continues to promote InterBase</a>, now at version 7.5, as a small-footprint, embedded database with commercial Server and Client licenses.</span> <br /> <span style="font-family: Verdana;"></span> <br /> <span style="font-family: Verdana;"> <a href="http://www.sqlite.org/index.html"><strong>SQLite</strong> </a> is a featherweight C library for an embedded database that implements most SQL-92 entry- and transitional-level requirements (some through the JDBC driver) and supports transactions within a tiny 250-KB code footprint. <a href="http://www.sqlite.org/cvstrac/wiki?p=SqliteWrappers">Wrappers</a> support a multitude of languages and operating systems, including Windows CE, SmartPhone, Windows Mobile, and Win32. SQLite's primary <a href="http://www.sqlite.org/omitted.html">SQL-92 limitations</a> are lack of nested transactions, inability to alter a table design once committed (other than with RENAME TABLE and ADD COLUMN operations), and foreign-key constraints. SQLite provides read-only views, triggers, and 256-bit encryption of database files. A downside is the the entire database file is <a href="http://weblogs.asp.net/jgalloway/archive/2006/04/12/442615.aspx">locked when while a transaction is in progress</a>. SQLite uses file access permissions in lieu of GRANT and REVOKE commands. Using SQLite involves no license; its code is entirely in the public domain.</span> </p> <p> <span style="font-family: Verdana; font-size: 85%;">The Mozilla Foundation's <a href="http://wiki.mozilla.org/Mozilla2:Unified_Storage">Unified Storage wiki</a> says this about SQLite: 'SQLite will be the back end for the unified store [for Firefox]. Because it implements a SQL engine, we get querying 'for free', without having to invent our own query language or query execution system. Its code-size footprint is moderate (250k), but it will hopefully simplify much existing code so that the net code-size change should be smaller. It has exceptional performance, and supports concurrent access to the database. Finally, it is released into the public domain, meaning that we will have no licensing issues.'</span> </p> <p> <span style="font-family: verdana;">Vieka Technology, Inc.'s <a href="http://vieka.com/esql.htm"><strong>eSQL 2.11</strong></a> is a port of SQLite to Windows Mobile (Pocket PC and Smartphone) and Win32, and includes development tools for Windows devices and PCs, as well as a .NET native data provider. A conventional ODBC driver also is available. eSQL for Windows (Win32) is free for personal and commercial use; eSQL for Windows Mobile requires a license for commercial (for-profit or business) use.</span> </p> <p> <span style="font-family: verdana;"> <a href="http://hsqldb.org/"><strong>HSQLDB</strong> </a> isn't on most reviewers' radar, which is surprising because it's the default database for <a href="http://www.openoffice.org/">OpenOffice.org</a> (OOo) 2.0's <a href="http://www.openoffice.org/product/base.html">Base</a> suite member. HSQLDB 1.8.0.1 is an open-source (BSD license) Java dembedded database engine based on Thomas Mueller's original Hypersonic SQL Project. Using OOo's Base feature requires installing the Java 2.0 Runtime Engine (which is not open-source) or the presence of an alternative open-source engine, such as Kaffe. My prior posts about OOo Base and HSQLDB are <a href="http://oakleafblog.blogspot.com/2005/10/openoffice-base-20-vs-microsoft-access.html">here</a>, <a href="http://oakleafblog.blogspot.com/2005/10/openoffice-base-20-vs-microsoft-access_22.html">here</a> and <a href="http://oakleafblog.blogspot.com/2005/10/openoffice-20-base-matches-microsoft.html">here</a>.</span> </p> <p> <span style="font-family: verdana;">The <a href="http://hsqldb.sourceforge.net/web/hsqlDocsFrame.html">HSQLDB 1.8.0 documentation</a> on SourceForge states the following regarding SQL-92 and later conformance:</span> </p> <span style="font-family: verdana;"> <blockquote> <p> <span style="font-family: verdana;">HSQLDB 1.8.0 supports the dialect of SQL defined by SQL standards 92, 99 and 2003. This means where a feature of the standard is supported, e.g. left outer join, the syntax is that specified by the standard text. Many features of SQL92 and 99 up to Advanced Level are supported and here is support for most of SQL 2003 Foundation and several optional features of this standard. However, certain features of the Standards are not supported so no claim is made for full support of any level of the standards. </span> </p> </blockquote> <span style="font-family: verdana;"><span style="font-size: 85%;">Other less well-known embedded databases designed for or suited to mobile deployment are </span> <a href="http://www.mimer.com/leftright.asp?secId=172"><span style="font-size: 85%;">Mimer SQL Mobile</span> </a><span style="font-size: 85%;"> and </span> <a href="http://www.vistadb.net/"><span style="font-size: 85%;">VistaDB 2.1</span> </a><span style="font-size: 85%;">. Neither product is open-source and require paid licensing; VistaDB requires a small up-front payment by developers but offers royalty-free distribution.</span> </span> <br /> <br /> <span style="font-family: Verdana;">Java DB, Firebird embedded, SQLite and eSQL 2.11 are contenders for lightweight PC and mobile device database projects that aren't Windows-only.</span> <br /> <br /> <strong> <span style="font-family: verdana;">SQL Server 2005 Everywhere<br /> </span><span style="font-family: Verdana;"></span> </strong> <br /> <span style="font-family: verdana;">If you're a Windows developer, SQL Server Mobile is the logical embedded database choice for mobile applications for Pocket PCs and Smartphones. Microsoft's April 19, 2006 press release delivered the news that SQL Server 2005 Mobile Editon (SQL Mobile or SSM) would gain a big brotherâSQL Server 2005 Everywhere Edition. </span> <br /> <span style="font-family: verdana;"></span> <br /> <span style="font-family: verdana;">Currently, the SSM client is licensed (at no charge) to run in production on devices with Windows CE 5.0, Windows Mobile 2003 for Pocket PC or Windows Mobile 5.0, or on PCs with Windows XP Tablet Edition only. SSM also is licensed for development purposes on PCs running Visual Studio 2005.</span> <span style="font-family: verdana;"> Smart Device replication with SQL Server 2000 SP3 and later databases has been the most common application so far for SSM.<br /> <br /> </span><span style="font-family: verdana;">By the end of 2006, Microsoft will license SSE for use on <em>all</em> PCs running any Win32 version or the preceding device OSs. A version of SQL Server Management Studio Express (SSMSE)âupdated to support SSEâis expected to release by the end of the year. These features will qualify SSE as <em>the universal embedded database</em> for Windows client and smart-device applications. </span> <br /> <br /> <span style="font-family: verdana;">For more details on SSE, read <a href="http://weblogs.asp.net/jgalloway/archive/2006/04/11/442451.aspx">John Galloway's April 11, 2006 blog post</a> and my '<a href="http://www.ftponline.com/special/sqlserver/rjennings-mobile/">SQL Server 2005 Mobile Goes Everywhere</a>' article for the <a href="http://www.ftponline.com/special/sqlserver/">FTPOnline Special Report on SQL Server</a>.</span><span style="font-family: verdana;"></span>" <p>(Via <a href="http://oakleafblog.blogspot.com">OakLeaf Systems</a>.)</p> </span> </blockquote>
Have RSS feeds killed the email star?
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2005-02-28#704
2005-02-28T20:36:19Z
2006-06-22T08:56:58-04:00
<blockquote dir="ltr" style="MARGIN-RIGHT: 0px"> <p><a href="http://networks.silicon.com/webwatch/0,39024667,39128215,00.htm">Have RSS feeds killed the email star? </a>silicon.com Feb 28 2005 12:58PM GMT </p></blockquote> <div align="right">[via <a href="http://www.moreover.com/rss">Moreover - XML and metadata news</a>]</div> <div align="left"> </div> <div align="left"><a href="http://www.answers.com/topic/rss-protocol">RSS</a> and other <a href="http://www.answers.com/main/ntquery?s=xml&method=2&gwp=13">XML</a> based syndication formats (<a href="http://www.answers.com/main/ntquery?method=4&dsid=2222&dekey=Resource+Description+Framework&gwp=8&curtab=2222_1">RDF</a>, <a href="http://www.answers.com/topic/atom-standard?hl=atom&hl=syndication">Atom</a>, etc.) allow organizations to syndicate their value propositions via feeds. Thus, instead of, depending solely on sending out HTML based advertorial emails (which end up in Spam Folders 75% of the time anyhow) to targets such as; suspects, leads, and customers. You can rely on the <a href="http://en.wikipedia.org/wiki/Web_2.0">Web 2.0 </a>fabric for <a href="http://diveintomark.org/archives/2002/06/02/important_change_to_the_link_tag">auto-discovery</a> of syndicated feeds covering marketing collateral such as; <a href="http://rss.openlinksw.com/uda.xml">features & benefits data</a>, product documentation (ODBC/JDBC <a href="http://docs.openlinksw.com/mt/mtdocs.opml">Multi-Tier</a>, ODBC/JDBC <a href="http://docs.openlinksw.com/st/litedocs.opml">Single-Tier</a>, and <a href="http://docs.openlinksw.com/virtuoso/virtdocs.opml">Virtuoso</a> ), <a href="http://demo.openlinksw.com/tutorial/rss.vsp">product functionality tutorials</a>, and screencasts (<a href="http://support.openlinksw.com/viewlets/uda_viewlets_rss.vsp">UDA </a>, <a href="http://support.openlinksw.com/viewlets/virtuoso_viewlets_rss.vsp">Virtuoso</a>, and <a href="http://support.openlinksw.com/viewlets/utilities_viewlets_rss.vsp">ODBC Benchmark & Troubleshooting Utilities</a>) etc. </div>
Cognitive Dissonance
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2005-02-25#695
2005-02-25T00:58:21Z
2006-06-22T08:56:58-04:00
<p><a href="http://www.answers.com/main/ntquery?s=definition+cognitive+dissonance&method=2&gwp=13">Cognitive dissonance</a> is how <a href="http://blogs.msdn.com/dareobasanjo/archive/2005/02/17/375367.aspx">Dare Obasanjo</a> aptly describes the emergence of some of the <a href="http://www.answers.com/main/ntquery?s=smart+tags&method=2&gwp=13">Smart Tags </a>concepts previously introduced by Microsoft and now emulated by the new google toolbar's autolink feature (<a href="http://glinden.blogspot.com/">Greg Linden</a> explains the problem with <a href="http://glinden.blogspot.com/2005/02/autolink-in-google-toolbar.html">clarity</a>).</p> <p>Anyway, back to cognitive dissonance. Could this be the reason for the following?</p> <ol> <li>Open Source products are increasingly database specific even though they could be database independent via Open Source ODBC SDK efforts such as <a href="http://www.iodbc.org">iODBC</a> and <a href="http://www.unixodbc.org">unixODBC</a>. We increasingly narrowing our choices down to database specific "Closed Source" or database specific "Open Source" solutions and somehow deem this to be progress<br></li> <li>The prevalent use of free standards compliant data access drivers (<a href="http://en.wikipedia.org/wiki/ODBC">ODBC</a>, <a href="http://en.wikipedia.org/wiki/JDBC">JDBC</a>, and <a href="http://en.wikipedia.org/wiki/ADO.NET">ADO.NET</a>) or their native counterparts that remain vulnerable to simple password hacks (there are databases behind those dynamic web sites!!) as none of these have any notion of "rules based" authentication and data access policy<br></li> <li>The time-tested fallacy that: "select * from table" defines a viable <a href="http://en.wikipedia.org/wiki/RDBMS">RDBMS</a> engine since Transaction Atomicity, Concurrency, Isolation, and Durability (ACID) mean zip! Ditto scrollable cursors, stored procedures, and other presumably useless aspects of any marginably decent RDBMS engine<br></li> <li>Failing to comprehend that a <a href="http://en.wikipedia.org/wiki/Weblog">Weblog</a> is your property (if you have a personal blog) not the property of the vendor hosting your service (that important issue of separating data ownership and data storage again). You may have heard about, or experienced, total loss of weblog and/or weblog archives arising from weblog engine or blog service provider changeovers<br></li> <li>Failing to see the synergy between personal/group/corporate information stores (aka <a href="http://www.openlinksw.com/blog/search.vspx?blogid=127&q=infoBase&type=text&output=html">infobase</a>) such as Wikis, Weblogs, and the burgeoning semantic web. <a href="http://weblog.infoworld.com/udell/2005/02/22.html#a1183">Jon Udell</a> for instance, is trying to get the point across via his tireless collection of <a href="http://en.wikipedia.org/wiki/XML_query_language">XQuery</a>/<a href="http://en.wikipedia.org/wiki/XPath">XPath</a> based queries aimed at the blogosphere section of the burgeoning semantic web. Here are some of mine (scoped to this weblog):<br></li> <ul> <li>Security related posts to date (<a href="http://www.openlinksw.com/blog/search.vspx?blogid=127&q=//p[contains%28.%2C%27security%27%29]&type=xpath&output=html">XPath</a> query)<br></li> <li>Infobase related posts to date (<a href="http://www.openlinksw.com/blog/search.vspx?blogid=127&q=infoBase&type=text&output=html">Free Text</a> search)<br></li></ul></ol> <p>And more...</p>
Email As A Platform
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2005-02-10#680
2005-02-10T17:01:57Z
2006-06-22T08:56:58-04:00
<blockquote dir="ltr" style="MARGIN-RIGHT: 0px"> <p><a href="http://techdirt.com/articles/20050209/1329235_F.shtml">Email As A Platform</a> It looks like more people are starting to realize that email is more than it seems. Especially given the drastic increase in storage size of web-based email applications, more people are realizing that <a href="http://news.bbc.co.uk/2/hi/business/4167633.stm">email is basically a personal database</a>. People simply store information in their email, from contact information that was emailed to them to schedule information to purchase tracking from emailed receipts. Lots of people email messages to themselves, realizing that email is basically the best "permanent" filing system they have. That's part of the reason why good email search is so important. Of course, what the article doesn't discuss is the next stage of this evolution. If you have a database of important information, the next step is to build useful applications on top of it. In other words, people are starting to realize that email, itself, is a <i>platform</i> for personal information management. </p></blockquote> <div align="right">[via <a href="http://www.techdirt.com/">Techdirt</a>]</div> <div align="left"> </div> <div align="left">Yep! And this is where the Unified Storage vision comes into play. Many years ago the same issues emerged in the business application realm, and at the time the issue at hand was: separating the DBMS engine from the Application logic. This is what the SQL Access Group (SAG) addressed via the CLI that laid the foundation for ODBC, JDBC, and recent derivatives; OLE DB and ADO.NET. </div> <div align="left"> </div> <div align="left">Most of us live inside our email applications and the need to integrate the content of emails, address books, notes, calendars with other data sources (Web Portal, Blogs, Wikis, CRM, ERP, and more) as part of our application interaction cycles and domain specific workflow is finally becoming obvious. There is a need for separation of the application/service layer from the storage engine across each one of these functionality realms. XML, RDF, and Triple Stores (RDF / Semantic Data Stores) collectively provide a standards based framework for achieving this goal. On the other hand so does WinFS albeit total proprietary (by this I mean none standards compliant) at the current time.</div> <div align="left"> </div> <div align="left">As you can already see there are numerous applications (conventional or hosted) that address email, address books, bookmarking, notes, calendars, blogs, wikis, crm etc. specifically, but next to none that address the obvious need for transparent integration across each functionality realm - the ultimate goal.</div> <div align="left"> </div> <div align="left">Yes, you know what I am about to say! <a href="http://virtuoso.openlinksw.com">OpenLink Virtuoso</a> is the platform for developing and/or implementing these next generation solutions. We have also decided to go one step further by developing a number of applications that demonstrate the vision (and ultimate reality); and each of these applications (and the inherent integration tapestry) will be the subject of a future Virtuoso Application specific post.</div>
Preventable SQL DBMS Vulnerabilities
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2004-05-17#546
2004-05-18T00:42:08Z
2006-06-22T08:56:58-04:00
<p>Here are some excerpts (inlined) with my comments (outlined) from an <a href="http://www.db2mag.com/showArticle.jhtml?articleID=18901175">interesting article</a> on SQL DBMS exploits and vulnerabilities by <a href="http://www.appsecinc.com/">Aaron C. Newman</a>, for <a href="http://www.db2mag.com/show">DB2 Magazine</a> titled "6 Security Secrets Attackers don't want You To Know".</p> <blockquote dir="ltr" style="MARGIN-RIGHT: 0px"> <p>How secure is your data? Looking at your information management resources through a would-be intruder's eyes can help you find (and fix) vulnerabilities.</p></blockquote> <p dir="ltr">Naturally :-)</p> <p></p> <blockquote dir="ltr" style="MARGIN-RIGHT: 0px"> <p>When E. F. Codd developed his relational data model in 1970, the business world was a different place. Almost 35 years after his seminal work appeared, RDBMSs that sprung from Codd's ideas are the standard for storing corporate information. And, with government and industry regulations dictating what kinds of information companies have to store, manage, and audit (and for how long), protecting this information is more important than ever. Unfortunately, it's also more challenging</p> <p>Even in 1985, when <a href="http://www.databaseanswers.com/codds_rules.htm">Dr. Codd published 12 guidelines for RDBMSs</a>, there was little concern for data security. In those days, gaining access to a database was so difficult that advanced security features on the database were irrelevant. </p> <p>Today, RDBMSs carry the lifeblood of every organization. Note the use of the plural: Organizations now have many databases that are decentralized in terms of use and security controls. E-business demands that data access be extended to customers, partners, suppliers, and other parties who were rarely considered in the early data management days. With all this availability ? not to mention pressure from an array of government and industry regulations (see the sidebar, <a href="http://www.db2mag.com/showArticle.jhtml?articleID=18901175#sidebar">"Security and Compliance"</a>) ? the need to control exactly who can access or modify data is becoming paramount. </p></blockquote> <p dir="ltr">Absolute facts, that are still partially understood at best. For instance we are still in a so called "Information Age" in which standards based data access remains an issue of contempt instead of absolute necessity. </p> <p dir="ltr">There are a number of prevailing myths about standards based data access that continue to cloak reality:</p> <ol dir="ltr"> <li> <div>ODBC, JDBC, ADO.NET, OLEDB all deliver poor performance (compared to their native, proprietary, and database specific counterparts; native interfaces)<br></div></li> <li> <div>You can't really right generic database applications with these standards due to inconsistencies in the DBMS implementations of SQL (not true! there are many aspects of the specs that address these concerns if only a majority of driver vendors would implement these features, and the application developers actually used them by seeking drivers with full implementations).</div></li></ol> <p>Even if the above were true (which I refute strongly), how about the general security vulnerabilities that affect both Native, and Standards compliant, data access interfaces?</p> <p>Aaron's article does a good job of highlighting 6 areas of vulnerability:</p> <ol> <li> <div>DBMS Defaults (usernames and passwords)</div></li> <li> <div>Authentication (at connect time)</div></li> <li> <div>Database Privileges</div></li> <li> <div>Fixpaks </div></li> <li> <div>Buffer Overflows</div></li> <li> <div>SQL Injection</div></li></ol> <p>What I have been able to do very quickly (thanks to blogging, and the power of a blog engine that supports <a href="http://www.openlinksw.com/blog/~kidehen/index.vspx?id=543">WebDAV</a>), is write a <a href="http://www.openlinksw.com/blog/~kidehen/articles/uda_rule_book_sql_attacks.htm">tabulated response to each of the items </a>(bar Fixpaks) indicating how the <a href="http://www.openlinksw.com/info/mtproduct.htm">OpenLink Multi-Tier Data Access Drivers </a>(for ODBC, JDBC, ADO.NET, and OLEDB) protect corporate databases from each of these vulnerabilities.</p> <p>To cut a long story short, we are increasingly living a contradiction where the terms "simple" and "free" are supposed to lead us to products that can adequately handle the challenges of an increasingly sophisticated grid of inter-connecting point. </p> <p>I have been asked on numerous occassions, "How can you build a company and business based on data access technology?". My reply is the same as usual, "because everything comes down to data". If the data is compromised in anyway, then kiss Information, Knowledge, and everything else goodbye!</p> <table align="right" border="0" cellpadding="5" cellspacing="0" width="336"> <tbody> <tr> <td></td></tr></tbody></table> <p> </p>
Enterprise Databases get a grip on XML
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2004-01-06#442
2004-01-06T23:17:07Z
2006-06-22T08:56:58-04:00
<blockquote style="margin-right: 0px;" dir="ltr"> <p><a class="listLinkLrg" title="http://newsletter.infoworld.com/t?ctl=4FEDB6:1F3948D" href="http://newsletter.infoworld.com/t?ctl=4FEDB6:1F3948D" target="_new"><strong><font face="Verdana">Databases get a grip on XML</font></strong></a><br /><font size="2"></font><font face="Verdana">From <a href="http://newsletter.infoworld.com/t?ctl=4FEDB6:1F3948D">Inforworld</a>.</font><br /><br /><font face="Verdana,Geneva,Arial,sans-serif" size="2">The next iteration of the SQL standard was supposed to arrive in 2003. But SQL standardization has always been a glacially slow process, so nobody should be surprised that SQL:2003 ? now known as SQL:200n ? isn?t ready yet. Even so, 2003 was a year in which XML-oriented data management, one of the areas addressed by the forthcoming standard, showed up on more and more developers? radar screens.ÃÂ <a title="http://newsletter.infoworld.com/t?ctl=4FEDB6:1F3948D" href="http://newsletter.infoworld.com/t?ctl=4FEDB6:1F3948D" target="_blank">>> READ MORE</a></font></p></blockquote> <p dir="ltr"><font face="Verdana" size="2">This article rounds up product for 2003 in the critical area of Enterprise Database Technology. It's certainly provides an apt reflection of how Virtuoso compares with offerings from some the larger (but certainly slower to implement) database vendors in this space. As usual Jon Udell's quote pretty much sums this up:</font></p> <blockquote style="margin-right: 0px;" dir="ltr"> <p dir="ltr"><!--StartFragment --><span class="artText"><em>"While the spotlight shone on the heavyweight contenders, a couple of agile innovators made noteworthy advances in 2003. </em><a class="regularArticleU" href="http://www.infoworld.com/699"><em>OpenLink Software?s Virtuoso 3.0</em></a><em>, which we reviewed in March, stole thunder from all three major players. Like Oracle, it offers a WebDAV-accessible XML repository. Like DB2 Information Integrator, it functions as database middleware that can perform federated ?joins? across SQL and XML sources. And like the forthcoming Yukon, it embeds the .Net CLR (Common Language Runtime), or in the case of Linux, Novell/Ximian?s Mono."</em></span> </p></blockquote> <p dir="ltr"><font face="Verdana" size="2">Albeit still somewhat unknown to the broader industry we have remained true our "innovator" discipline, which still remains our chosen path to market leadership. Thus, its worth a quick Virtuoso release history, and featuresÃÂ recap as we get set to up the ante even further in 2004:</font></p> <p dir="ltr"><font face="Verdana" size="2"><a href="http://www.openlinksw.com/press/virtuoso.htm">1998 - Virtuoso's initial public beta</a> release with functional emphasis on Virtual Database Engine for ODBC and JDBC Data Sources.</font></p> <p dir="ltr"><font face="Verdana" size="2"><a href="http://www.openlinksw.com/press/virtuoso1.htm">1999 - Virtuoso's official commercial</a> release, with emphasis stillÃÂ on Virtual Database functionality for ODBC, JDBC accessible SQL Databases.</font></p> <p dir="ltr"><font face="Verdana" size="2"><a href="http://www.openlinksw.com/press/v2releas.htm">2000 - Virtuoso 2.0</a> adds XML Storage, XPath, XML Schema, XQuery, XSL-T, WebDAV, SOAP, UDDI, HTTP, Replication, Free Text Indexing (*feature update*), POP3, and NNTP support.</font></p> <p dir="ltr"><font face="Verdana" size="2"><a href="http://www.openlinksw.com/press/v27releas.htm">2002 - Virtuoso 2.7</a> extends Virtualization prowess beyond data access via enhancements to its Web Services protocol stack implementation by enabling SQL Stored Procedures to be published as Web Services. It also debutsÃÂ its Object-Relational engine enhancements that include theÃÂ incorporation of Java and Microsoft .NET Objects into its User Defined Type, User Defined Functions, and Stored ProcedureÃÂ offerings.</font></p> <p dir="ltr"><font face="Verdana" size="2"><a href="http://www.openlinksw.com/press/virt3beta.htm">2003 - Virtuoso 3.0</a> extends data and application logic virtualization into the Application Server realm (basically a Virtual Application server too!), by adding support for ASP.NET, PHP, Java Server Pages runtime hosting (making applications built using any of these languages deployable using Virtuoso across all supported platforms).</font></p> <p dir="ltr"><font face="Verdana" size="2">Collectively each of these releases have contributed to a very premeditated architecture and vision that will ultimately unveil the inherent power of critical I.S infrastructure virtualizationÃÂ along the following lines; data storage, data access , and application logic via coherent integration of SQL, XML, Web Services, and Persistent Stored Modules (.NET, Java, and other object based component building blocks).</font></p> <p dir="ltr"><font face="Verdana"></font>ÃÂ </p>
A Virtuoso of a Server
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2003-10-23#395
2003-10-23T21:57:48Z
2006-06-22T08:56:58-04:00
<font size="2"> <p><a href="http://www.nwfusion.com/index.html">NETWORK WORLD</a> NEWSLETTER: MARK GIBBS ON WEB APPLICATIONS </p> <p><font size="2">Today's focus: A Virtuoso of a server</font></p> <p>By <a href="http://www.nwfusion.com/columnists/gibbs.html">Mark Gibbs</a></p> <p>One of the bigger drags of Web applications development is that building a system of even modest complexity is a lot like herding cats - you need a database, an applications server, an XML engine, etc., etc. And as they all come from different vendors you are faced with solving the constellation of integration issues that inevitably arise.</p> <p>If you are lucky, your integration results in a smoothly functioning system. If not, you have a lot of spare parts flying in loose formation with the risk of a crash and burn at any moment.</p> <p>An alternative is to look for all of these features and services in a single package but you'll find few choices in this arena.</p> <p>One that is available and looks very promising is OpenLink's Virtuoso (see links below).</p> <p>Virtuoso is described as a cross platform (runs on Windows, all Unix flavors, Linux, and Mac OS X) universal server that provides databases, XML services, a Web application server and supporting services all in a single package.</p> <p>OpenLink's list of supported standards is impressive and includes .Net, Mono, J2EE, XML Web Services (Simple Object Application Protocol, Web Services Description Language, WS-Security, Universal Description, Discovery and Integration), XML, XPath, XQuery, XSL-T, WebDav, HTTP, SMTP, LDAP, POP3, SQL-92, ODBC, JDBC and OLE-DB.</p> <p>Virtuoso provides an HTTP-compliant Web Server; native XML document creation, storage and management; a Web services platform for creation, hosting and consumption of Web services; content replication and synchronization services; free text index server, mail delivery and storage and an NNTP server.</p> <p>Another interesting feature is that with Virtuoso you can create Web services from existing SQL Stored Procedures, Java classes,</p> <p>C++ classes, and 'C' functions as well as create dynamic XML</p> <p>documents from ODBC and JDBC data sources.</p> <p>This is an enormous product and implies a serious commitment on the part of adopters due to its scope and range of services.</p> <blockquote dir="ltr" style="MARGIN-RIGHT: 0px"> <p><em>Virtuoso is enormous by virtue of its architectural ambitions, but actual disk requirements are</em></p></blockquote></font>
Howl is Rendezvous for Windows and Linux
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2003-08-05#232
2003-08-05T19:18:55Z
2006-06-22T08:56:58-04:00
<p dir="ltr">Very interesting, we have basically ported Zeroconfig as released by Apple (in Open Source) too, and used it in both our <a href="http://www.openlinksw.com/virtuoso/whatis.htm">Virtuoso</a> 3.x and <a href="http://www.openlinksw.com/product.htm">UDA</a> 5.x products.</p> <blockquote dir="ltr" style="MARGIN-RIGHT: 0px"> <p><a href="http://www.swampwolf.com/products/">Howl</a> is Rendezvous for Windows and Linux. [via <a href="http://www.scripting.com/">Scripting News</a>]</p></blockquote> <p dir="ltr">In the case of UDA you can configure ODBC and JDBC consumable data source names that are hosted on the server. Users can nownbspsimply picknbspDSNs from anbspcombo box and they are ready to make connections to remote databases from any ODBC, JDBC, OLE DB, or ADO.NET application.nbspAnother benefit ofnbspZeroconfignbspis that it facilitates centralized server side configuration which further enhances our server side session rules book;nbspwhich serves all our Multi-Tier data access drivers.</p> <p dir="ltr">In the case of Virtuoso you are able to bind to pre-configured Virtuoso instances in exactly the same way.</p> <p dir="ltr">Our Zeroconfig support has beennbspimplemented across Solaris, AIX, Digital UNIX, IRIX, HP-UX amongst others, but this is a project of interest all the same, and we may end up contributing to this effort.</p>
Tim O'Reilly about network aware software
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2003-07-07#201
2003-07-07T20:51:35Z
2006-06-22T08:56:58-04:00
<p><a href="http://blog.bitflux.ch/p1077.html">Tim O'Reilly about network aware software</a> </p> <p>Tim O'Reilly wrote some thoughts about network aware software. Good sumup and nice ideas, why not only blogs should be net-aware (and where even blogs can be improved ;) ) </p> <blockquote dir="ltr" style="MARGIN-RIGHT: 0px"> <div align="left">"<i>For the desktop, my personal vision is to see existing software instrumented to become increasingly web aware. It seems that Apple are doing a good job with this. (What does web aware mean for me? Being able to grok URIs, speaking WebDAV, and using open standard data formats.)</i>" -- <strong>Edd Dumbill</strong> </div> <div align="left"></div> <div align="left">[via <a href="http://blog.bitflux.ch/">Bitflux Blog</a>]</div></blockquote> <div align="left">I agree, but you do have to add Open Data Access formats (such as ODBC and to some degree JDBC) to this mix otherwise the you will need to create data for Open Standard Data Formats from sratch (tough for any enterprise irrespective of size).</div> <div align="left"></div> <div align="left">Tim O'Reilly added the following items to Edd's list:</div> <div align="left"> <ul> <li> <p>Rendezvous-like functionality for automatic discovery of and potential synchronization with other instances of the application on other computers. Apple is showing the power of this idea with iChat and iTunes, but it really could be applied in so many other places. For example, if every PIM supported this functionality, we could have the equivalent of "phonester" where you could automatically ask peers for contact information. Of course, that leads to guideline 2. </p></li></ul></div> <p>Another application is discovery of <a href="http://www.openlinksw.com/info/docs/uda50/mt/features.html#features">ODBC data sources</a>, and database servers. Rendezvous can also simply security and administration of data sources accessible by either one of these standards data access mechanisms. It can also apply to XML databases and data sources exposed by <a href="http://www.openlinksw.com/virtuoso/whatis.htm">XML Databases</a>.</p> <p></p> <p></p> <ul> <li>If you assume ad-hoc networking, you have to automatically define levels of access. I've always thought that the old Unix ugo (user, group, other) three-level permission system was simple and elegant, and if you replace the somewhat arbitrary "group" with "on my buddy list", you get something quite powerful. Which leads me to... <p></p> <p></p></li> <ul> <li>Buddy lists ought to be supported as a standard feature of many apps, and in a consistent way. What's more, our address books really ought to make it easy to indicate who is in a "buddy list" and support numerous overlapping lists for different purposes. <br></li></ul> <li>Every application ought to expose some version of its data as an XML feed via some well-defined and standard access mechanism. It strikes me that one of the really big wins that fueled the early web was a simple naming scheme: you could go to a site called www.foo.com, and you'd find a web server there. While it wasn't required, it made web addresses eminently guessable. We missed the opportunity for xml.foo.com to mean "this is where you get the data feed" but it's probably still possible to come up with a simple, consistent naming scheme. And of course, if we can do it for web sites, we also need to think about how to do it for local applications, since... </li></ul> <p>The very point I continue to make about Internet Points of Presence beingactual data acces points, in short these end points should be served by database serverprocesses. This is the very basis of <a href="http://www.openlinksw.com/virtuoso">Virtuoso</a>, the inevitability of this realization remains the undepinings of this product. There are other products out there that have some sense of this vision too, but there is a little snag (at least so far in my research efforts), and that is the tendency to create dedicated independent server per protocol (an ultimate integration, administration, and maintenance nightmare).</p> <ul> <li>We ought to be able to have the expectation that all applications, whether local or remote (web) will be set up for two-way interactions. That is, they can be either a source or sink of online data. So, for example, the natural complement to amazon's web services data feeds is data input (for example, the ability to comment on a book on your local blog, and syndicate the review via RSS to amazon's detail page for the book.) And that leads to: <p></p> <p></p></li> <li>We really need to understand who owns what, and come up with mechanisms that protect the legitimate rights of individuals and businesses to their own data, while creating the "liquidity" and free movement of data that will fuel the next great revolution in computer functionality. (I'm doing a panel on this subject at next week's Open Source Convention, entitled "<a href="http://conferences.oreillynet.com/cs/os2003/view/e_sess/4526">We Need a Bill of Rights for Web Services</a>.") <p></p> <p></p></li> <li>We need easy gateways between different application domains. I was recently in Finland at a Nokia retreat, and we used camera-enabled cell phones to create a mobile photoblog. That was great. But even more exciting was the ease with which I could send a photo from the phone not just to another phone but also to an email address. This is the functionality that enabled the blog gateway, but it also made it trivial to send photos home to my family and friends. Similarly, I often blog things that I hear on mailing lists, and read many web sites via screen-scraping enabled email lists. It would be nice to have cross-application gateways be a routine part of software, rather than something that has to be hacked on after the fact.</li></ul> <div align="left">The wish list is pretty much a clear articulation of key items that should matter most to decision makers (CTOs and CIOs) ; in particular those that continue to wrestle with the identification and isolation of relevantcomponentsfor their enterprisearchitectures. </div>
<p>We all know that the only benchmark that matters, is the one that you run in-house using the systems that comprise your IT infrastructure. </p>
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2003-06-25#183
2003-06-25T13:27:02Z
2006-06-22T08:56:58-04:00
<P>We all know that the only benchmark that matters, is the one that you run in-house using the systems that comprise your IT infrastructure. </P> <BLOCKQUOTE dir=ltr style="MARGIN-RIGHT: 0px"> <P><A href="http://c.moreover.com/click/here.pl?r77267478">Apple's benchmarks under fire</A> ZDNet Jun 25 2003 7:13AM ET </P> <P>[via <A href="http://www.moreover.com/">Moreover - ZDNet</A>]</P></BLOCKQUOTE> <P>OpenLink Software has provided an Open Source benchmark utility that support Mac OS X, Linux, and UNIX. Thus, if mission critical database oriented performance is what is most relevant to your needs (as opposed to Photoshop) then simply download either one, or both of the following:</P> <P><A href="http://oplweb2.openlinksw.com:8080/download/util.vsp">OpenLink ODBC Bench</A> (you can test TPC-A and TPC-C like performance of the G5 and compare against other platforms) via ODBC)</P> <P><A href="http://oplweb2.openlinksw.com:8080/download/util.vsp">OpenLink JDBC Bench </A>(same thing using JDBC)</P>
Ingres - A Forgotten Database, the untold story
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2003-06-17#279
2003-06-17T11:18:57Z
2006-06-22T08:56:58-04:00
<P><A href="http://www.it-director.com/article.php?articleid=10951">Ingres - A Forgottent Database The Untold Story</A></P> <P><EM>Ingres (technically, Advantage Ingres Enterprise) is, arguably, the forgotten database. There used to be five major databases: Oracle, DB2, Sybase, Informix and Ingres. Then along came Microsoft and, if you listened to most press comment (or the lack of it), you would think that there were only two of these left, plus SQL Server</EM>. [From <A href="http://www.it-director.com/article.php?articleid=10951">IT-Director</A>]</P> <BLOCKQUOTE dir=ltr style="MARGIN-RIGHT: 0px"> <P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Oracle, Microsoft, and IBM would certainly like the illusion of a 3 horse race, as this is the only way they can induce Ingres, Informix, and Sybase users to jump ship, and this, even though database migrations are by far the most risk prone and problematic aspects of any IT infrastructure. <?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" /><o:p></o:p></SPAN></P> <P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Here is the interesting logic from the self-made big three, if you want to take advanatage of new paradigms and technologies such as XML, Web Services, and anything else in the pipeline you have to move all your data out of these databases, and then get all the mission critical applications re-associated with one of these databases, and by the way when you do so it is advisable that you use native interfaces (so that sometime in the future you have no chance whatsoever of repeating this folly at their expense).<o:p></o:p></SPAN></P> <P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">The simple fact of the matter (which the self-made big three do not want you to know) is that you can put ODBC, JDBC, even platform specific data access APIs such as OLE DB and ADO.NET atop any of these databases, and then explore and exploit the benefits of new technologies and paradigms as long as the tool pool supports one of more of these standards.<o:p></o:p></SPAN></P> <P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Unfortunately the no-brainer above appears to be the more difficult of the choices before decision makers. In other words, many would rather dig themselves into a deeper hole (unknowingly i can only presume) that ultimately leads to technology lock-in.<o:p></o:p></SPAN></P> <P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">The biggest challenge before any RDBMS based infrastructure today isn't which of the self-made big three to migrate to wholesale, rather, how to make progressive use of the pool of disparate applications, and application databases that proliferate the enterprise. <o:p></o:p></SPAN></P> <P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">This is another way of understanding the burgeoning market for Virtual Databases, which in my opiion present the new frontier in database technology.<o:p></o:p></SPAN></P> <P> </P></BLOCKQUOTE>
Web-enable Your Business Intelligence Using XML/A and ASP.NET
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2003-05-29#338
2003-05-29T20:01:48Z
2006-06-22T08:56:58-04:00
0
Web-enable Your Business Intelligence Using XML/A and ASP.NET
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2003-05-29#66
2003-05-29T20:01:48Z
2006-06-22T08:56:58-04:00
<P>Don't you sometimes wish that accessing database data were easier, and you didn't have to deal with such things as OLE DB, ODBC, and JDBCthat you could use just one uniform approach to access the data regardless of the location and type? </P> <P><EM>Yes, but I don't see how any of the data access mechanisms listed have anything to do with the problem as each serves a different application domain.</EM></P> <P>Well, guess what? You won't have to wait too much longer to see this dream become reality. All major vendors are not only working on this issue seriously but have already started releasing SDKs for exactly this purpose. The point of commonality between them is XML. Like it or not, XML is becoming a standard part of programmers' lives because it's simple and easy to understand structure make it a suitable format for transferring data between disparate applications and platforms. But the real power of XML lies in its ability to not only extend itself, but also define itself.<BR><BR><EM>This is all well and good but it is XML, yes a uniforming technology, but the opening paragraph undermines the pont! </EM></P> <P><EM>XML For Analysis is just another addition to the pool of data access mechanisms, it is an XML./SOAP based data access mechanism for Tabular and Mulitdimensional Data and without our implementation XMLA simply offers little to anyone trying to get at tabular data (SQL Data) as many of the current implementers fall into the OLAP camp, and there is no standard query language for persistenf CUBES (MDX from Microsoft is de facto at best). Thus, without Virtuoso tabular data (where the real data actually resides) is out of the picture.</EM></P> <P><EM>So we say goodbye to ODBC, JDBC, OLE DB to get what exactly? From our perspective, another showcase opportunity, but for the innocent readers of this article I haven't got the faintest idea.</EM></P> <P><EM></EM> </P>
<big>SQL Injection FAQ </big>
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2003-05-21#319
2003-05-21T22:27:45Z
2006-12-04T08:41:24.000003-05:00
<p align="center"><font color="#0080c0" size="2"><big><strong><big>SQL Injection FAQ </big></strong></big></font></p> <p align="center"> </p><p align="center"><strong><font color="red"></font></strong></p><strong><font color="red"> </font></strong><a href="http://www.sqlsecurity.com/DesktopDefault.aspx?tabindex=2&tabid=3"><strong><font color="red"><a href="http://www.sqlsecurity.com/DesktopDefault.aspx?tabindex=2&;tabid=3">http://www.sqlsecurity.com/DesktopDefault.aspx?tabindex=2&;tabid=3</a><br /></font></strong></a><strong><font color="red"> </font></strong> <br /> <div align="center"> <center> <table width="80%" border="0"> <tbody> <tr> <td width="100%"> <p><big><strong><font size="2">Are other SQL Servers (Sybase, Oracle, DB2) subject to SQL injection?</font></strong></big></p> <p><font size="2">Yes, to varying degrees. Here is a site that can get you more details on some of the issues with other SQL Servers. </font><a href="http://www.owasp.org/" target="_blank"><a href="http://www.owasp.org/"><font size="2">http://www.owasp.org</font></a></a></p><font size="2"><b>What is SQL Injection and why is all this information not included in the regular FAQ?</b> </font> <p><font size="2">SQL Injection is simply a term describing the act of passing SQL code into an application that was not intended by the developer.  Since this topic is not specifically restricted to SQL Server it is not included in the normal FAQ. In fact, much of the problems that allow SQL injection are not the fault of the database server per-se but rather are due to poor input validation and coding at other code layers. However, due to the serious nature and prevalence of this problem I feel its inclusion in a thorough discussion of SQL Server security is warranted.</font></p> <p><big><strong><font size="2">What causes SQL Injection?</font></strong></big></p> <p><font size="2">SQL injection is usually caused by developers who use "string-building" techniques in order to execute SQL code. For example, in a search page, the developer may use the following code to execute a query (VBScript/ASP sample shown):</font></p> <p><font face="Courier New" color="#ff0000" size="2">Set myRecordset = myConnection.execute("SELECT * FROM myTable WHERE someText ='" & request.form("inputdata") & "'")</font></p> <p><font size="2">The reason this statement is likely to introduce an SQL injection problem is that the developer has made a classic mistake - poor input validation. We are trusting that user has not entered something malicious - something like the innocent looking single quote ('). Let's consider what would happen if a user entered the following text into the search form:</font></p> <p><font size="2">' exec master..xp_cmdshell 'net user test testpass /ADD' --</font></p> <p><font size="2">Then, when the query string is assembled and sent to SQL Server, the server will process the following code:</font></p> <p><font face="Courier New" color="#ff0000" size="2">SELECT * FROM myTable WHERE someText ='' exec master..xp_cmdshell 'net user test testpass /ADD'--'</font></p> <p><font size="2">Notice, the first single quote entered by the user closed the string and SQL Server eagerly executes the next SQL statements in the batch including a command to add a new user to the local accounts database. If this application were running as 'sa' and the MSSQLSERVER service is running with sufficient privileges we would now have an account with which to access this machine. Also note the use of the comment operator (--) to force the SQL Server to ignore the trailing quote placed by the developer's code.</font></p> <p><a href="http://www.sqlsecurity.com/faq-inj.asp"><font size="2">More</font></a></p> <p><em><font color="#000000" size="2">Very intresting that these are all Native Interface based exploits. So the security issue isn't ODBC, JDBC, ADO.NET, or OLE DB specific (although they certainly increase the potential damage that can be unleashed via metadata analysis en route to that huge Cartesian Product ; the mother of all Exploits!). Our Session Rules Book was devised in 1993 with many of these issues in mind, and to this date there are no other ODBC/JDBC/OLE DB products out there that even come close to acknowledging this reality.</font></em></p></td></tr></tbody></table></center></div>
<big>SQL Injection FAQ </big>
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2003-05-21#48
2003-05-21T22:27:45Z
2006-06-22T08:56:58-04:00
<P align=center><FONT color=#0080c0 size=2><BIG><STRONG><BIG>SQL Injection FAQ </BIG></STRONG></BIG></FONT></P> <P align=center> <P align=center><STRONG><FONT color=red></FONT></STRONG></P><STRONG><FONT color=red> </FONT></STRONG><A href="http://www.sqlsecurity.com/DesktopDefault.aspx?tabindex=2&tabid=3"><STRONG><FONT color=red><A href="http://www.sqlsecurity.com/DesktopDefault.aspx?tabindex=2&;tabid=3">http://www.sqlsecurity.com/DesktopDefault.aspx?tabindex=2&;tabid=3</A><BR></FONT></STRONG></A><STRONG><FONT color=red> </FONT></STRONG> <BR> <DIV align=center> <CENTER> <TABLE width="80%" border=0> <TBODY> <TR> <TD width="100%"> <P><BIG><STRONG><FONT size=2>Are other SQL Servers (Sybase, Oracle, DB2) subject to SQL injection?</FONT></STRONG></BIG></P> <P><FONT size=2>Yes, to varying degrees. Here is a site that can get you more details on some of the issues with other SQL Servers. </FONT><A href="http://www.owasp.org/" target=_blank><A href="http://www.owasp.org/"><FONT size=2>http://www.owasp.org</FONT></A></A></P><FONT size=2><B>What is SQL Injection and why is all this information not included in the regular FAQ?</B> </FONT> <P><FONT size=2>SQL Injection is simply a term describing the act of passing SQL code into an application that was not intended by the developer. Since this topic is not specifically restricted to SQL Server it is not included in the normal FAQ. In fact, much of the problems that allow SQL injection are not the fault of the database server per-se but rather are due to poor input validation and coding at other code layers. However, due to the serious nature and prevalence of this problem I feel its inclusion in a thorough discussion of SQL Server security is warranted.</FONT></P> <P><BIG><STRONG><FONT size=2>What causes SQL Injection?</FONT></STRONG></BIG></P> <P><FONT size=2>SQL injection is usually caused by developers who use "string-building" techniques in order to execute SQL code. For example, in a search page, the developer may use the following code to execute a query (VBScript/ASP sample shown):</FONT></P> <P><FONT face="Courier New" color=#ff0000 size=2>Set myRecordset = myConnection.execute("SELECT * FROM myTable WHERE someText ='" & request.form("inputdata") & "'")</FONT></P> <P><FONT size=2>The reason this statement is likely to introduce an SQL injection problem is that the developer has made a classic mistake - poor input validation. We are trusting that user has not entered something malicious - something like the innocent looking single quote ('). Let's consider what would happen if a user entered the following text into the search form:</FONT></P> <P><FONT size=2>' exec master..xp_cmdshell 'net user test testpass /ADD' --</FONT></P> <P><FONT size=2>Then, when the query string is assembled and sent to SQL Server, the server will process the following code:</FONT></P> <P><FONT face="Courier New" color=#ff0000 size=2>SELECT * FROM myTable WHERE someText ='' exec master..xp_cmdshell 'net user test testpass /ADD'--'</FONT></P> <P><FONT size=2>Notice, the first single quote entered by the user closed the string and SQL Server eagerly executes the next SQL statements in the batch including a command to add a new user to the local accounts database. If this application were running as 'sa' and the MSSQLSERVER service is running with sufficient privileges we would now have an account with which to access this machine. Also note the use of the comment operator (--) to force the SQL Server to ignore the trailing quote placed by the developer's code.</FONT></P> <P><A href="http://www.sqlsecurity.com/faq-inj.asp"><FONT size=2>More</FONT></A></P> <P><EM><FONT color=#000000 size=2>Very intresting that these are all Native Interface based exploits. So the security issue isn't ODBC, JDBC, ADO.NET, or OLE DB specific (although they certainly increase the potential damage that can be unleashed via metadata analysis en route to that huge Cartesian Product ; the mother of all Exploits!). Our Session Rules Book was devised in 1993 with many of these issues in mind, and to this date there are no other ODBC/JDBC/OLE DB products out there that even come close to acknowledging this reality.</FONT></EM></P></TD></TR></TBODY></TABLE></CENTER></DIV>