Kingsley Idehen's Blog Data Space

In recent times a lot of the commentary and focus re. Virtuoso has centered on the RDF Quad Store and Linked Data. What sometimes gets overlooked is the sophisticated Virtual Database Engine that provides the foundation for all of Virtuoso's data integration capabilities.

In this post I provide a brief re-introduction to this essential aspect of Virtuoso.

What is it?

This component of Virtuoso is known as the Virtual Database Engine (VDBMS). It provides transparent high-performance and secure access to disparate data sources that are external to Virtuoso. It enables federated access and integration of data hosted by any ODBC- or JDBC-accessible RDBMS, RDF Store, XML database, or Document (Free Text)-oriented Content Management System. In addition, it facilitates integration with Web Services (SOAP-based SOA RPCs or REST-fully accessible Web Resources).

Why is it important?

In the most basic sense, you shouldn't need to upgrade your existing database engine version simply because your current DBMS and Data Access Driver combo isn't compatible with ODBC-compliant desktop tools such as Microsoft Access, Crystal Reports, BusinessObjects, Impromptu, or other of ODBC, JDBC, ADO.NET, or OLE DB-compliant applications. Simply place Virtuoso in front of your so-called "legacy database," and let it deliver the compliance levels sought by these tools

In addition, it's important to note that today's enterprise, through application evolution, company mergers, or acquisitions, is often faced with disparately-structured data residing in any number of line-of-business-oriented data silos. Compounding the problem is the exponential growth of user-generated data via new social media-oriented collaboration tools and platforms. For companies to cost-effectively harness the opportunities accorded by the increasing intersection between line-of-business applications and social media, virtualization of data silos must be achieved, and this virtualization must be delivered in a manner that doesn't prohibitively compromise performance or completely undermine security at either the enterprise or personal level. Again, this is what you get by simply installing Virtuoso.

How do I use it?

The VDBMS may be used in a variety of ways, depending on the data access and integration task at hand. Examples include:

Relational Database Federation

You can make a single ODBC, JDBC, ADO.NET, OLE DB, or XMLA connection to multiple ODBC- or JDBC-accessible RDBMS data sources, concurrently, with the ability to perform intelligent distributed joins against externally-hosted database tables. For instance, you can join internal human resources data against internal sales and external stock market data, even when the HR team uses Oracle, the Sales team uses Informix, and the Stock Market figures come from Ingres!

Conceptual Level Data Access using the RDF Model

You can construct RDF Model-based Conceptual Views atop Relational Data Sources. This is about generating HTTP-based Entity-Attribute-Value (E-A-V) graphs using data culled "on the fly" from native or external data sources (Relational Tables/Views, XML-based Web Services, or User Defined Types).

You can also derive RDF Model-based Conceptual Views from Web Resource transformations "on the fly" -- the Virtuoso Sponger (RDFizing middleware component) enables you to generate RDF Model Linked Data via a RESTful Web Service or within the process pipeline of the SPARQL query engine (i.e., you simply use the URL of a Web Resource in the FROM clause of a SPARQL query).

It's important to note that Views take the form of HTTP links that serve as both Data Source Names and Data Source Addresses. This enables you to query and explore relationships across entities (i.e., People, Places, and other Real World Things) via HTTP clients (e.g., Web Browsers) or directly via SPARQL Query Language constructs transmitted over HTTP.

Conceptual Level Data Access using ADO.NET Entity Frameworks

As an alternative to RDF, Virtuoso can expose ADO.NET Entity Frameworks-based Conceptual Views over Relational Data Sources. It achieves this by generating Entity Relationship graphs via its native ADO.NET Provider, exposing all externally attached ODBC- and JDBC-accessible data sources. In addition, the ADO.NET Provider supports direct access to Virtuoso's native RDF database engine, eliminating the need for resource intensive Entity Frameworks model transformations.

Related

• Attaching ODBC or JDBC accessible Relational Tables to Virtuoso
• Using an HTML based Wizard to Generate RDF based Linked Views over Relational Tables
• Screencast Demonstrating Wizard based generation of RDF based Linked Data Views Part 1
• Screencast Demonstrating Wizard based generation of RDF based Linked Data Views Part 1
• Generating RDF based Linked Data from non RDF based Web Resources via the Sponger
• Building ADO.NET based Entity Frameworks Views over Relational Data
• Building Silverlight Rich Internat Applicaitons using ADO.NET, Entity Frameworks, and RDF based Linked Data.

After listening to the latest Semantic Web Gang podcast, I found myself agreeing with some of the points made by Alex Iskold, specifically:

-- Linked Data does not implicitly imply making all your data public

-- Linked Data principles benefit Intranet and Extranet style data integration (trumps alternative distributed database integration approaches any day)

-- Business exploitation of Linked Data on the Web will certainly be driven by the correlation of opportunity costs (which is more than likely what Alex meant by "use cases") associated with the lack of URIs originating from the domain of a given business (Tom Heath: also effectively alluded to this via his BBC and URI land grab anecdotes; same applies Georgi's examples)

-- History is a great tutor, answers to many of today's problems always lie somewhere in plain sight of the past.

Of course, I also believe that Linked Data serves Web Data Integration across the Internet very well too, and the fact that it will be beneficial to businesses in a big way. No individual or organization is an island, I think the Internet and Web have done a good job of demonstrating that thus far :-) We're all data nodes in a Giant Global Graph.

Daniel lewis did shed light on the read-write aspects of the Linked Data Web, which is actually very close to the callout for a Wikipedia for Data. TimBL has been working on this via Tabulator (see Tabulator Editing Screencast), Bengamin Nowack also added similar functionality to ARC, and of course we support the same SPARQL UPDATE into an RDF information resource via the RDF Sink feature of our WebDAV and ODS-Briefcase implementations.

One of the biggest impediments to the adoption of technology is the cost burden typically associated with doing the right thing. For instance, requirements for making the Linked Data Web (GGG) buzz would include the following (paraphrasing TimBL's original Linked Data meme):

-- identifying the things you observe, or stumble upon, using URIs (aka Entity IDs)

-- construct URIs using HTTP so that the Web provides a channel for referencing things elsewhere (remote object referencing)

-- Expose things in your Data Space(s) that are potentially useful to other Web users via URIs

-- Link to other Web accessible things using their URIs.

The list is nice, but actual execution can be challenging. For instance, when writing a blog post, or constructing a WikiWord, would you have enough disposable time to go searching for these URIs? Or would you compromise and continue to inject "Literal" values into the Web, leaving it to the reasoning endowed human reader to connect the dots?

Anyway, OpenLink Data Spaces is now equipped with a Glossary system that allows me to manage terms, meaning of terms, and hyper-linking of phrases and words matching associated with my terms. The great thing about all of this is that everything I do is scoped to my Data Space (my universe of discourse), I don't break or impede the other meanings of these terms outside my Data Space. The Glossary system can be shared with anyone I choose to share it with, and even better, it makes my upstreaming (rules based replication) style of blogging even more productive :-)

Remember, on the Linked Data Web, who you know doesn't matter as much as what your are connected to, directly or indirectly. Jason Kolb covers this issue in his post: People as Data Connectors, and so doesFrederick Giasson via a recent post titled: Networks are everywhere. For instance, this blog post (or the entire Blog) is a bona fide RDF Linked Data Source, you can use it as the Data Source of a SPARQL Query to find things that aren't even mentioned in this post, since all you are doing is beaming a query through my Data Space (a container of Linked Data Graphs). On that note, let's re-watch Jon Udell's "On-Demand-Blogosphere" screencast from 2006 :-)

While I continue to wrestle with screencast production etc.. Here is are some screenshots that guide you through the process of providing Data Web URIs to the SPARQL Query Builder (first cut of an MS Query or MS ACCESS type tool for the Data Web).

1. Step 1 - Enter a Data Source URI
2. Step 2 - Click on the Run Control (">" video control icon)
3. Step 3 - Interact with Custom Grid hosted results (comprised of Resource Identifiers (S), Properties (P), and Property Values (O).

Once you grasp the concept of entering values into the "Default Data Source URI field", take a look at: http://programmableweb.com and other URIs (hint: scroll through the results grid to the QEDWiki demo item)

The simple demo use our Ajax based Visual Query Builder for the SPARQL Query Language (this isn't Grandma's Data Web UI, but not to worry, that is on it's way also). Here goes:

1. go to http://demo.openlinksw.com/isparql
2. Enter any of the following values into the "Default Data URI"; field:
- http://www.mkbergman.com/?p=336
- http://radar.oreilly.com/archives/2007/02/pipes_and_filte.html
- http://jeremy.zawodny.com/blog/archives/008513.html
- Other URIs

What I am demonstrating is how existing Web Content hooks transperently into the "Data Web". Zero RDF Tax :-) Everything is good!

Note: Please look to the bottom of the screen for the "Run Query" Button. Remember, it not quite Grandma's UI but should do for Infonauts etc.. A screencast will follow.

A quick dump that demonstrates how I integrate tags and links from del.icio.us with links from my local bookmark database via one of my public Data Spaces (this demo uses the kidehen Data Space).

SPARQL (query language for the Semantic Web) basically enables me to query a collection of typed links (predicates/properties/attributes) in my Data Space (ODS based of course) without breaking my existing local bookmarks database or the one I maintain at del.icio.us.

I am also demonstrating how Web 2.0 concepts such as Tagging mesh nicely with the more formal concepts of Topics in the Semantic Web realm. The key to all of this is the ability to generate RDF Data Model Instance Data based on Shared Ontologies such as SIOC (from DERI's SIOC Project) and SKOS (again showing that Ontologies and Folksonomies are complimentary).

This demo also shows that Ajax also works well in the Semantic Web realm (or web dimension of interaction 3.0) especially when you have a toolkit with Data Aware controls (for SQL, RDF, and XML) such as OAT (OpenLink Ajax Toolkit). For instance, we've successfully used this to build a Visual Query Building Tool for SPARQL (alpha) that really takes a lot of the pain out of constructing SPARQL Queries (there is much more to come on this front re. handling of DISTINCT, FILTER, ORDER BY etc..).

For now, take a look at the SPARQL Query dump generated by this SIOC & SKOS SPARQL QBE Canvas Screenshot.

You can cut and paste the queries that follow into the Query Builder or use the screenshot to build your variation of this query sample. Alternatively, you can simply click on *This* SPARQL Protocol URL to see the query results in a basic HTML Table. And one last thing, you can grab the SPARQL Query File saved into my ODS-Briefcase (the WebDAV repository aspect of my Data Space).

Note the following SPARQL Protocol Endpoints:

1. MyOpenLink Data Space
2. Experimental Data Space SPARQL Query Builder (you need to register at http://myopenlink.net:8890/ods to use this version)
3. Live Demo Sever
4. Demo Server SPARQL Query Builder (use: demo for both username and pwd when prompted)

My beautified Version of the SPARQL Generated by QBE (you can cut and paste into "Advanced Query" section of QBE) is presented below:

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX sioc: <http://rdfs.org/sioc/ns#>
PREFIX dct: <http://purl.org/dc/elements/1.1/>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>


SELECT distinct 
       ?forum_name, 
       ?owner, 
       ?post, 
       ?title, 
       ?link, 
       ?url, 
       ?tag
FROM <http://myopenlink.net/dataspace>
WHERE {
       ?forum a sioc:Forum;
                   sioc:type "bookmark";
                   sioc:id ?forum_name;
                   sioc:has_member ?owner.
       ?owner sioc:id "kidehen".
       ?forum sioc:container_of ?post .
       ?post  dct:title ?title .
       optional { ?post sioc:link ?link  }
       optional { ?post sioc:links_to ?url }
       optional { ?post sioc:topic ?topic.
                        ?topic a skos:Concept;
                                  skos:prefLabel ?tag}.
     } 

Unmodified dump from the QBE (this will be beautified automatically in due course by the QBE):

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX sioc: <http://rdfs.org/sioc/ns#>
PREFIX dct: <http://purl.org/dc/elements/1.1/>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>


SELECT ?var8 ?var9 ?var13 ?var14 ?var24 ?var27 ?var29 ?var54 ?var56
WHERE
{
graph ?graph {
 ?var8 rdf:type sioc:Forum .
 ?var8 sioc:container_of ?var9 .
 ?var8 sioc:type "bookmark" .
 ?var8 sioc:id ?var54 .
 ?var8 sioc:has_member ?var56 .
 ?var9 rdf:type sioc:Post .
 OPTIONAL {?var9 dc:title ?var13} .
 OPTIONAL {?var9 sioc:links_to ?var14} .
 OPTIONAL {?var9 sioc:link ?var29} .
 ?var9 sioc:has_creator ?var37 .
 OPTIONAL {?var9 sioc:topic ?var24} .
 ?var24 rdf:type skos:Concept .
 OPTIONAL {?var24 skos:prefLabel ?var27} .
 ?var56 rdf:type sioc:User .
 ?var56 sioc:id "kidehen" .
 }
} 

Current missing items re. Visual QBE for SPARQL are:

1. Ability to Save properly to WebDAV so that I can then expose various saved SPARQL Queries (.rq file) from my Data Space via URIs
2. Handling of DISTINCT, FILTERS (note: OPTIONAL is handled via dotted predicate-links)
3. General tidying up re. click event handling etc.

1. Export some bookmarks from your local browser to the usual HTML bookmarks dump file
2. Create an ODS-Bookmarks Instance using your new ODS account
3. Use the ODS-Bookmark Instance to import your local bookmarks from the HTML dump file
4. Repeat the same import sequence using the ODS-Bookmark Instance, but this time pick the del.icio.us option
5. Build your query (change 'kidehen' to your ODS-user-name)
6. That's it you now have Semantic Web presence in the form of a Data Space for your local and del.icio.us hosted bookmarks with tags integrated

Quick Query Builder Tip: You will need to import the following (using the Import Button in the Ontologies & Schemas side-bar);

1. http://www.w3.org/1999/02/22-rdf-syntax-ns# (RDF)
2. http://rdfs.org/sioc/ns# (SIOC)
3. http://purl.org/dc/elements/1.1/ (Dublin Core)
4. http://www.w3.org/2004/02/skos/core# (SKOS)

Browser Support: The SPARQL QBE is SVG based and currently works fine with the following browsers; Firefox 1.5/2.0, Camino (Cocoa variant of Firefox for Mac OS X), Webkit (Safari pre-release / advanced sibling), Opera 9.x. We are evaluating the use of the Adobe SVG plugin re. IE 6/7 support.

Of course this should be a screencast, but I am the middle of a plethora of things right now :-)

Last week I put out a series of screencast style demos that sought to demonstrate the core elements of our soon to be released Javascript Toolkit called OAT (OpenLink Ajax Toolkit) and its Ajax Database Connectivity layer.

The screencasts covered the following functionality realms:

1. SQL Query By Example (basic)
2. SQL Query By Example (advanced - pivot table construction)
3. Web Form Design (basic database driven map based mashup)
4. Web Form Design (advanced database driven map based mashup)

To bring additional clarity to the screencasts demos and OAT in general, I have saved a number of documents that are the by products of activities in the screenvcasts:

1. Live XML Document produced using SQL Query By Example (basic) (you can use drag and drop columns across the grid to reorder and sort presentation)
2. Live XML Document produced using QBE and Pivot Functionality (you can drag and drop the aggregate columns and rows to create your own views etc..)
3. Basic database driven map based mashup (works with FireFox, Webkit, Camino; click on pins to see national flag)
4. Advanced database driven map based mashup (works with FireFox, Webkit, Camino; records, 36, 87, and 257 will unveil pivots via lookup pin)

Notes:

• “Advanced”, as used above, simply means that I am embedding images (employee photos and national flags) and a database driven pivot into the map pins that serve as details lookups in classic SQL master/details type scenarios.
• The “Ajax Call In Progress..” dialog is there to show live interaction with a remote database (in this case Virtuoso but this could be any ODBC, JDBC, OLEDB, ADO.NET, or XMLA accessible data source)
• The data access magic source (if you want to call it that) is XMLA - a standard that has been in place for years but completely misunderstood and as a result under utilized

You can see a full collection of saved documents at the following locations:

• My Mashups demo directory (Google and Yahoo! demo variants but note these do not work with Safari or IE at the current time. IE7 issues will be resolved in the next day or so)
• My Pivots demo directory (other Pivots will be added as I build and save them)
• My Saved Queries (a collection of saved QBE generated queries)