A simple guide usable by any Perl developer seeking to exploit SPARQL without hassles.

Why?

SPARQL is a powerful query language, results serialization format, and an HTTP based data access protocol from the W3C. It provides a mechanism for accessing and integrating data across Deductive Database Systems (colloquially referred to as triple or quad stores in Semantic Web and Linked Data circles) -- database systems (or data spaces) that manage proposition oriented records in 3-tuple (triples) or 4-tuple (quads) form.

How?

SPARQL queries are actually HTTP payloads (typically). Thus, using a RESTful client-server interaction pattern, you can dispatch calls to a SPARQL compliant data server and receive a payload for local processing.

Steps:

1. Determine which SPARQL endpoint you want to access e.g. DBpedia or a local Virtuoso instance (typically: http://localhost:8890/sparql).
2. If using Virtuoso, and you want to populate its quad store using SPARQL, assign "SPARQL_SPONGE" privileges to user "SPARQL" (this is basic control, more sophisticated WebID based ACLs are available for controlling SPARQL access).

Script:

#
# Demonstrating use of a single query to populate a 
# Virtuoso Quad Store via Perl. 
#

# 
# HTTP URL is constructed accordingly with CSV query results format as the default via mime type.
#

use CGI qw/:standard/;
use LWP::UserAgent;
use Data::Dumper;
use Text::CSV_XS;

sub sparqlQuery(@args) {
  my $query=shift;
  my $baseURL=shift;
  my $format=shift;
	
	%params=(
		"default-graph" => "", "should-sponge" => "soft", "query" => $query,
		"debug" => "on", "timeout" => "", "format" => $format,
		"save" => "display", "fname" => ""
	);
	
	@fragments=();
	foreach $k (keys %params) {
		$fragment="$k=".CGI::escape($params{$k});
		push(@fragments,$fragment);
	}
	$query=join("&", @fragments);
	
	$sparqlURL="${baseURL}?$query";
	
	my $ua = LWP::UserAgent->new;
	$ua->agent("MyApp/0.1 ");
	my $req = HTTP::Request->new(GET => $sparqlURL);
	my $res = $ua->request($req);
	$str=$res->content;
	
	$csv = Text::CSV_XS->new();
	
	foreach $line ( split(/^/, $str) ) {
		$csv->parse($line);
		@bits=$csv->fields();
	  push(@rows, [ @bits ] );
	}
	return \@rows;
}


# Setting Data Source Name (DSN)

$dsn="http://dbpedia.org/resource/DBpedia";

# Virtuoso pragmas for instructing SPARQL engine to perform an HTTP GET using the IRI in
# FROM clause as Data Source URL en route to DBMS
# record Inserts.

$query="DEFINE get:soft \"replace\"\n

# Generic (non Virtuoso specific SPARQL
# Note: this will not add records to the 
# DBMS 

SELECT DISTINCT * FROM <$dsn> WHERE {?s ?p ?o}"; 

$data=sparqlQuery($query, "http://localhost:8890/sparql/", "text/csv");

print "Retrieved data:\n";
print Dumper($data);

Output

Retrieved data:
$VAR1 = [
          [
            's',
            'p',
            'o'
          ],
          [
            'http://dbpedia.org/resource/DBpedia',
            'http://www.w3.org/1999/02/22-rdf-syntax-ns#type',
            'http://www.w3.org/2002/07/owl#Thing'
          ],
          [
            'http://dbpedia.org/resource/DBpedia',
            'http://www.w3.org/1999/02/22-rdf-syntax-ns#type',
            'http://dbpedia.org/ontology/Work'
          ],
          [
            'http://dbpedia.org/resource/DBpedia',
            'http://www.w3.org/1999/02/22-rdf-syntax-ns#type',
            'http://dbpedia.org/class/yago/Software106566077'
          ],
...

Conclusion

CSV was chosen over XML (re. output format) since this is about a "no-brainer installation and utilization" guide for a Perl developer that already knows how to use Perl for HTTP based data access within HTML. SPARQL just provides an added bonus to URL dexterity (delivered via URI abstraction) with regards to constructing Data Source Names or Addresses.

Related

• RDF::Query::Client Guide
• SPARQL Guide for the Perl Developer
• SPARQL Guide for the PHP Developer
• SPARQL Guide for the Python Developer
• SPARQL Guide for the Ruby Developer
• Simple Guide for using SPARQL with Virtuoso
• General SPARQL Tutorial Collection
• Virtuoso Specific SPARQL Tutorial Collection
• The URI, URL, and Linked Data Meme's Generic HTTP URI.

A simple guide usable by any Javascript developer seeking to exploit SPARQL without hassles.

Why?

SPARQL is a powerful query language, results serialization format, and an HTTP based data access protocol from the W3C. It provides a mechanism for accessing and integrating data across Deductive Database Systems (colloquially referred to as triple or quad stores in Semantic Web and Linked Data circles) -- database systems (or data spaces) that manage proposition oriented records in 3-tuple (triples) or 4-tuple (quads) form.

How?

SPARQL queries are actually HTTP payloads (typically). Thus, using a RESTful client-server interaction pattern, you can dispatch calls to a SPARQL compliant data server and receive a payload for local processing.

Steps:

1. Determine which SPARQL endpoint you want to access e.g. DBpedia or a local Virtuoso instance (typically: http://localhost:8890/sparql).
2. If using Virtuoso, and you want to populate its quad store using SPARQL, assign "SPARQL_SPONGE" privileges to user "SPARQL" (this is basic control, more sophisticated WebID based ACLs are available for controlling SPARQL access).

Script:

/*
Demonstrating use of a single query to populate a # Virtuoso Quad Store via Javascript. 
*/

/* 
HTTP URL is constructed accordingly with JSON query results format as the default via mime type.
*/

function sparqlQuery(query, baseURL, format) {
	if(!format)
		format="application/json";
	var params={
		"default-graph": "", "should-sponge": "soft", "query": query,
		"debug": "on", "timeout": "", "format": format,
		"save": "display", "fname": ""
	};
	
	var querypart="";
	for(var k in params) {
		querypart+=k+"="+encodeURIComponent(params[k])+"&";
	}
	var queryURL=baseURL + '?' + querypart;
	if (window.XMLHttpRequest) {
  	xmlhttp=new XMLHttpRequest();
  }
  else {
  	xmlhttp=new ActiveXObject("Microsoft.XMLHTTP");
  }
  xmlhttp.open("GET",queryURL,false);
  xmlhttp.send();
  return JSON.parse(xmlhttp.responseText);
}

/*
setting Data Source Name (DSN)
*/

var dsn="http://dbpedia.org/resource/DBpedia";

/*
Virtuoso pragma "DEFINE get:soft "replace" instructs Virtuoso SPARQL engine to perform an HTTP GET using the IRI in FROM clause as Data Source URL with regards to 
DBMS record inserts
*/

var query="DEFINE get:soft \"replace\"\nSELECT DISTINCT * FROM <"+dsn+"> WHERE {?s ?p ?o}"; 
var data=sparqlQuery(query, "/sparql/");

Output

Place the snippet above into the <script/> section of an HTML document to see the query result.

Conclusion

JSON was chosen over XML (re. output format) since this is about a "no-brainer installation and utilization" guide for a Javascript developer that already knows how to use Javascript for HTTP based data access within HTML. SPARQL just provides an added bonus to URL dexterity (delivered via URI abstraction) with regards to constructing Data Source Names or Addresses.

Related

• SPARQL Guide for the PHP Developer
• SPARQL Guide for the Python Developer
• SPARQL Guide for the Ruby Developer
• Simple Guide for using SPARQL with Virtuoso
• General SPARQL Tutorial Collection
• Virtuoso Specific SPARQL Tutorial Collection
• The URI, URL, and Linked Data Meme's Generic HTTP URI.

A simple guide usable by any PHP developer seeking to exploit SPARQL without hassles.

Why?

SPARQL is a powerful query language, results serialization format, and an HTTP based data access protocol from the W3C. It provides a mechanism for accessing and integrating data across Deductive Database Systems (colloquially referred to as triple or quad stores in Semantic Web and Linked Data circles) -- database systems (or data spaces) that manage proposition oriented records in 3-tuple (triples) or 4-tuple (quads) form.

How?

SPARQL queries are actually HTTP payloads (typically). Thus, using a RESTful client-server interaction pattern, you can dispatch calls to a SPARQL compliant data server and receive a payload for local processing e.g. local object binding re. PHP.

Steps:

1. From your command line execute: aptitude search '^PHP26', to verify PHP is in place
2. Determine which SPARQL endpoint you want to access e.g. DBpedia or a local Virtuoso instance (typically: http://localhost:8890/sparql).
3. If using Virtuoso, and you want to populate its quad store using SPARQL, assign "SPARQL_SPONGE" privileges to user "SPARQL" (this is basic control, more sophisticated WebID based ACLs are available for controlling SPARQL access).

Script:

#!/usr/bin/env php
<?php
#
# Demonstrating use of a single query to populate a # Virtuoso Quad Store via PHP. 
#

# HTTP URL is constructed accordingly with JSON query results format in mind.

function sparqlQuery($query, $baseURL, $format="application/json")

  {
	$params=array(
		"default-graph" =>  "",
		"should-sponge" =>  "soft",
		"query" =>  $query,
		"debug" =>  "on",
		"timeout" =>  "",
		"format" =>  $format,
		"save" =>  "display",
		"fname" =>  ""
	);

	$querypart="?";	
	foreach($params as $name => $value) 
  {
		$querypart=$querypart . $name . '=' . urlencode($value) . "&";
	}
	
	$sparqlURL=$baseURL . $querypart;
	
	return json_decode(file_get_contents($sparqlURL));
};



# Setting Data Source Name (DSN)
$dsn="http://dbpedia.org/resource/DBpedia";

#Virtuoso pragmas for instructing SPARQL engine to perform an HTTP GET
#using the IRI in FROM clause as Data Source URL

$query="DEFINE get:soft \"replace\"
SELECT DISTINCT * FROM <$dsn> WHERE {?s ?p ?o}"; 

$data=sparqlQuery($query, "http://localhost:8890/sparql/");

print "Retrieved data:\n" . json_encode($data);

?>

Output

Retrieved data:
  {"head":
  {"link":[],"vars":["s","p","o"]},
  "results":
		{"distinct":false,"ordered":true,
		"bindings":[
			{"s":
			{"type":"uri","value":"http:\/\/dbpedia.org\/resource\/DBpedia"},"p":
			{"type":"uri","value":"http:\/\/www.w3.org\/1999\/02\/22-rdf-syntax-ns#type"},"o":
			{"type":"uri","value":"http:\/\/www.w3.org\/2002\/07\/owl#Thing"}},
			{"s":
			{"type":"uri","value":"http:\/\/dbpedia.org\/resource\/DBpedia"},"p":
			{"type":"uri","value":"http:\/\/www.w3.org\/1999\/02\/22-rdf-syntax-ns#type"},"o":
			{"type":"uri","value":"http:\/\/dbpedia.org\/ontology\/Work"}},
			{"s":
			{"type":"uri","value":"http:\/\/dbpedia.org\/resource\/DBpedia"},"p":
			{"type":"uri","value":"http:\/\/www.w3.org\/1999\/02\/22-rdf-syntax-ns#type"},"o":
			{"type":"uri","value":"http:\/\/dbpedia.org\/class\/yago\/Software106566077"}},
...

Conclusion

JSON was chosen over XML (re. output format) since this is about a "no-brainer installation and utilization" guide for a PHP developer that already knows how to use PHP for HTTP based data access. SPARQL just provides an added bonus to URL dexterity (delivered via URI abstraction) with regards to constructing Data Source Names or Addresses.

Related

• SPARQL Guide for the Python Developer
• SPARQL Guide for the Ruby Developer
• Simple Guide for using SPARQL with Virtuoso
• General SPARQL Tutorial Collection
• Virtuoso Specific SPARQL Tutorial Collection
• The URI, URL, and Linked Data Meme's Generic HTTP URI.

A simple guide usable by any Python developer seeking to exploit SPARQL without hassles.

Why?

SPARQL is a powerful query language, results serialization format, and an HTTP based data access protocol from the W3C. It provides a mechanism for accessing and integrating data across Deductive Database Systems (colloquially referred to as triple or quad stores in Semantic Web and Linked Data circles) -- database systems (or data spaces) that manage proposition oriented records in 3-tuple (triples) or 4-tuple (quads) form.

How?

SPARQL queries are actually HTTP payloads (typically). Thus, using a RESTful client-server interaction pattern, you can dispatch calls to a SPARQL compliant data server and receive a payload for local processing e.g. local object binding re. Python.

Steps:

1. From your command line execute: aptitude search '^python26', to verify Python is in place
2. Determine which SPARQL endpoint you want to access e.g. DBpedia or a local Virtuoso instance (typically: http://localhost:8890/sparql).
3. If using Virtuoso, and you want to populate its quad store using SPARQL, assign "SPARQL_SPONGE" privileges to user "SPARQL" (this is basic control, more sophisticated WebID based ACLs are available for controlling SPARQL access).

Script:

#!/usr/bin/env python
#
# Demonstrating use of a single query to populate a # Virtuoso Quad Store via Python. 
#

import urllib, json

# HTTP URL is constructed accordingly with JSON query results format in mind.

def sparqlQuery(query, baseURL, format="application/json"):
	params={
		"default-graph": "",
		"should-sponge": "soft",
		"query": query,
		"debug": "on",
		"timeout": "",
		"format": format,
		"save": "display",
		"fname": ""
	}
	querypart=urllib.urlencode(params)
	response = urllib.urlopen(baseURL,querypart).read()
	return json.loads(response)

# Setting Data Source Name (DSN)
dsn="http://dbpedia.org/resource/DBpedia"

# Virtuoso pragmas for instructing SPARQL engine to perform an HTTP GET
# using the IRI in FROM clause as Data Source URL

query="""DEFINE get:soft "replace"
SELECT DISTINCT * FROM <%s> WHERE {?s ?p ?o}""" % dsn 

data=sparqlQuery(query, "http://localhost:8890/sparql/")

print "Retrieved data:\n" + json.dumps(data, sort_keys=True, indent=4)

#
# End

Output

Retrieved data:
{
    "head": {
        "link": [], 
        "vars": [
            "s", 
            "p", 
            "o"
        ]
    }, 
    "results": {
        "bindings": [
            {
                "o": {
                    "type": "uri", 
                    "value": "http://www.w3.org/2002/07/owl#Thing"
                }, 
                "p": {
                    "type": "uri", 
                    "value": "http://www.w3.org/1999/02/22-rdf-syntax-ns#type"
                }, 
                "s": {
                    "type": "uri", 
                    "value": "http://dbpedia.org/resource/DBpedia"
                }
            }, 
...

Conclusion

JSON was chosen over XML (re. output format) since this is about a "no-brainer installation and utilization" guide for a Python developer that already knows how to use Python for HTTP based data access. SPARQL just provides an added bonus to URL dexterity (delivered via URI abstraction) with regards to constructing Data Source Names or Addresses.

Related

• SPARQL Guide for the Ruby Developer
• Simple Guide for using SPARQL with Virtuoso
• General SPARQL Tutorial Collection
• Virtuoso Specific SPARQL Tutorial Collection
• The URI, URL, and Linked Data Meme's Generic HTTP URI.

- Produce RDF Linked Data from non RDF Data Sources (Heterogeneous SQL, XML, Web Services)

- Provide highly scalable RDF Data Management via a Quad Store (DBpedia is an example of a live demonstration)

- Sophisticated Deployment of Linked Data that exploits the power of SPARQL

- Powerful WebDAV innovations that simplify read-write mode interaction with Linked Data

- Use Web Data Virtualization to address the pain and frustration associated with Web Data Silos (e.g. OpenLink Data Spaces layer stop Virtuoso that delivers Personal Data Spaces / Unified Storage in the Clouds)

- Deliver a Linked Data development and deployment platform to .NET (VB, C#) , Java, PHP, Ruby, Perl, Python, 'C', C++, and other developers

- More...

Daniel simplifies my post by using diagrams to depict the different paths for PHP based applications exposing Linked Data - especially those that already provide a significant amount of the content that drives Web 2.0.

If all the content in Web 2.0 information resources are distillable into discrete data objects endowed with HTTP based IDs (URIs), with zero "RDF handcrafting Tax", what do we end up with? A Giant Global Graph of Linked Data; the Web as a Database.

So, what used to apply exclusively, within enterprise settings re. Oracle, DB2, Informix, Ingres, Sybase, Microsoft SQL Server, MySQL, PostrgeSQL, Progress Open Edge, Firebird, and others, now applies to the Web. The Web becomes the "Distributed Database Bus" that connects database records across disparate databases (or Data Spaces). These databases manage and expose records that are remotely accessible "by reference" via HTTP.

As I've stated at every opportunity in the past, Web 2.0 is the greatest thing that every happened to the Semantic Web vision :-) Without the "Web 2.0 Data Silo Conundrum" we wouldn't have the cry for "Data Portability" that brings a lot of clarity to some fundamental Web 2.0 limitations that end-users ultimately find unacceptable.

In the late '80s, the SQL Access Group (now part of X/Open) addressed a similar problem with RDBMS silos within the enterprise that lead to the SAG CLI which is exists today as Open Database Connectivity.

In a sense we now have WODBC (Web Open Database Connectivity), comprised of Web Services based CLIs and/or traditional back-end DBMS CLIs (ODBC, JDBC, ADO.NET, OLE-DB, or Native), Query Language (SPARQL Query Language), and a Wire Protocol (HTTP based SPARQL Protocol) delivering Web infrastructure equivalents of SQL and RDA, but much better, and with much broader scope for delivering profound value due to the Web's inherent openness. Today's PHP, Python, Ruby, Tcl, Perl, ASP.NET developer is the enterprise 4GL developer of yore, without enterprise confinement. We could even be talking about 5GL development once the Linked Data interaction is meshed with dynamic languages (delivering higher levels of abstraction at the language and data interaction levels). Even the underlying schemas and basic design will evolve from Closed World (solely) to a mesh of Closed & Open World view schemas.

Virtuoso extends its SQL3 implementation with syntax for integrating SPARQL into queries and subqueries.Thus, as part of a SQL SELECT query or subquery, one can write the SPARQL keyword and a SPARQL query as part of query text processed by Virtuoso's SQL Query Processor.

Example 1 (basic) :

Using Virtuoso's Command line or the Web Based ISQL utility type in the following (note: "SQL>" is the command line prompt for the native ISQL utility):

SQL> sparql select distinct ?p where { graph ?g { ?s ?p ?o } };

Which will return the following:

	  p varchar
     ----------
     http://example.org/ns#b
     http://example.org/ns#d
     http://xmlns.com/foaf/0.1/name
     http://xmlns.com/foaf/0.1/mbox
     ...   

Example 2 (a subquery variation):

SQL> select distinct subseq (p, strchr (p, '#')) as fragment
 from (sparql select distinct ?p where { graph ?g { ?s ?p ?o } } ) as all_predicates
 where p like '%#%' ;

     fragment varchar
     ----------
     #query
     #data
     #name
     #comment
     ...

Parameterized Queries:

You can pass parameters to a SPARQL query using a Virtuoso-specific syntax extension. '??' or '$?' indicates a positional parameter similar to '?' in standard SQL. '??' can be used in graph patterns or anywhere else where a SPARQL variable is accepted. The value of a parameter should be passed in SQL form, i.e. this should be a number or an untyped string. An IRI ID can not be passed, but an absolute IRI can. Using this notation, a dynamic SQL capable client (ODBC, JDBC, ADO.NET, OLEDB, XMLA, or others) can execute parametrized SPARQL queries using parameter binding concepts that are common place in dynamic SQL. Which implies that existing SQL applications and development environments (PHP, Ruby, Python, Perl, VB, C#, Java, etc.) are capable of issuing SPARQL queries via their existing SQL bound data access channels against RDF Data stored in Virtuoso.

Note: This is the Virtuoso equivalent of a recently published example using Jena (a Java based RDF Triple Store).

Example:

Create a Virtuoso Function by execting the following:

SQL> create function param_passing_demo ();
 {
 	declare stat, msg varchar;
 	declare mdata, rset any;
 	exec ('sparql select ?s where { graph ?g { ?s ?? ?? }}',
 			stat, msg,
 			vector ('http://www.w3.org/2001/sw/DataAccess/tests/data/Sorting/sort-0#int1',
 		  		   4 ),	-- Vector of two parameters 
			10,			-- Max. result-set rows
			mdata, 		-- Variable for handling result-set metadata
 		 	rset   		-- Variable for handling query result-set
		 ); 
     return rset[0][0];
 }

SQL> select param_passing_demo ();

Which returns:

callret VARCHAR
 _______________________________________________________________________________

http://www.w3.org/2001/sw/DataAccess/tests/data/Sorting/sort-0#four

1 Rows. -- 00000 msec.

 

Using SPARQL in SQL Predicates:

A SPARQL ASK query can be used as an argument of the SQL EXISTS predicate.

create function sparql_ask_demo () returns varchar
  {
 		if (exists (sparql ask where { graph ?g { ?s ?p 4}})) return 'YES';
 		else return 'NO';
   };

Test by executing:

SQL> select sparql_ask_demo ();

Which returns:

_________________________
YES

I added the missing piece regarding the "Virtuoso Conductor" (the Web based Admin UI for Virtuoso) to the original post below. I also added a link to our live SPARQL Demo so that anyone interested can start playing around with SPARQL and SPARQL integrated into SQL right away.

Another good thing about this post is the vast amount of valuable links that it contains. To really appreciate this point simply visit my Linkblog (excuse the current layout :-) - a Tab if you come in via the front door of this Data Space (what I used to call My Weblog Home Page).

"Free" Databases: Express vs. Open-Source RDBMSs: "Open-source relational database management systems (RDBMSs) are gaining IT mindshare at a rapid pace. As an example, BusinessWeek's February 6, 2006 ' Taking On the Database Giants ' article asks 'Can open-source upstarts compete with Oracle, IBM, and Microsoft?' and then provides the answer: 'It's an uphill battle, but customers are starting to look at the alternatives.'

There's no shortage of open-source alternatives to look at. The BusinessWeek article concentrates on MySQL, which BW says 'is trying to be the Ikea of the database world: cheap, needs some assembly, but has a sleek, modern design and does the job.' The article also discusses Postgre[SQL] and Ingres, as well as EnterpriseDB, an Oracle clone created from PostgreSQL code*. Sun includes PostgreSQL with Solaris 10 and, as of April 6, 2006, with Solaris Express.**

*Frank Batten, Jr., the investor who originally funded Red Hat, invested a reported $16 million into Great Bridge with the hope of making a business out of providing paid support to PostgreSQL users. Great Bridge stayed in business only 18 months , having missed an opportunity to sell the business to Red Hat and finding that selling $50,000-per-year support packages for an open-source database wasn't easy. As Batten concluded, 'We could not get customers to pay us big dollars for support contracts.' Perhaps EnterpriseDB will be more successful with a choice of $5,000, $3,000, or $1,000 annual support subscriptions .

**Interestingly, Oracle announced in November 2005 that Solaris 10 is 'its preferred development and deployment platform for most x64 architectures, including x64 (x86, 64-bit) AMD Opteron and Intel Xeon processor-based systems and Sun's UltraSPARC(R)-based systems.'

There is a surfeit of reviews of current MySQL, PostgreSQL and—to a lesser extent—Ingres implementations. These three open-source RDBMSs come with their own or third-party management tools. These systems compete against free versions of commercial (proprietary) databases: SQL Server 2005 Express Edition (and its MSDE 2000 and 1.0 predecessors), Oracle Database 10g Express Edition, IBM DB2 Express-C, and Sybase ASE Express Edition for Linux where database size and processor count limitations aren't important. Click here for a summary of recent InfoWorld reviews of the full versions of these four databases plus MySQL, which should be valid for Express editions also. The FTPOnline Special Report article, 'Microsoft SQL Server Turns 17,' that contains the preceding table is here (requires registration.)

SQL Server 2005 Express Edition SP-1 Advanced Features

SQL Server 2005 Express Edition with Advanced Features enhances SQL Server 2005 Express Edition (SQL Express or SSX) dramatically, so it deserves special treatment here. SQL Express gains full text indexing and now supports SQL Server Reporting Services (SSRS) on the local SSX instance. The SP-1 with Advanced Features setup package, which Microsoft released on April 18, 2006, installs the release version of SQL Server Management Studio Express (SSMSE) and the full version of Business Intelligence Development Studio (BIDS) for designing and editing SSRS reports. My 'Install SP-1 for SQL Server 2005 and Express' article for FTPOnline's SQL Server Special Report provides detailed, illustrated installation instructions for and related information about the release version of SP-1. SP-1 makes SSX the most capable of all currently available Express editions of commercial RDBMSs for Windows.

OpenLink Software's Virtuoso Open-Source Edition

OpenLink Software announced an open-source version of it's Virtuoso Universal Server commercial DBMS on April 11, 2006. On the initial date of this post, May 2, 2006, Virtuoso Open-Source Edition (VOS) was virtually under the radar as an open-source product. According to this press release, the new edition includes:

• SPARQL compliant RDF Triple Store
• SQL-200n Object-Relational Database Engine (SQL, XML, and Free Text)
• Integrated BPEL Server and Enterprise Service Bus
• WebDAV and Native File Server
• Web Application Server that supports PHP, Perl, Python, ASP.NET, JSP, etc.
• Runtime Hosting for Microsoft .NET, Mono, and Java

VOS only lacks the virtual server and replication features that are offered by the commercial edition. VOS includes a Web-based administration tool called the "Virtuoso Conductor" According to Kingsley Idehen's Weblog, 'The Virtuoso build scripts have been successfully tested on Mac OS X (Universal Binary Target), Linux, FreeBSD, and Solaris (AIX, HP-UX, and True64 UNIX will follow soon). A Windows Visual Studio project file is also in the works (ETA some time this week).'

InfoWorld's Jon Udell has tracked Virtuoso's progress since 2002, with an additional article in 2003 and a one-hour podcast with Kingsley Idehen on April 26, 2006. A major talking point for Virtuoso is its support for Atom 0.3 syndication and publication, Atom 1.0 syndication and (forthcoming) publication, and future support for Google's GData protocol, as mentioned in this Idehen post. Yahoo!'s Jeremy Zawodny points out that the 'fingerprints' of Adam Bosworth, Google's VP of Engineering and the primary force behind the development of Microsoft Access, 'are all over GData.' Click here to display a list of all OakLeaf posts that mention Adam Bosworth.

One application for the GData protocol is querying and updating the Google Base database independently of the Google Web client, as mentioned by Jeremy: 'It's not about building an easier onramp to Google Base. ... Well, it is. But, again, that's the small stuff.' Click here for a list of posts about my experiences with Google Base. Watch for a future OakLeaf post on the subject as the GData APIs gain ground.

Open-Source and Free Embedded Database Contenders

Open-source and free embedded SQL databases are gaining importance as the number and types of mobile devices and OSs proliferate. Embedded databases usually consist of Java classes or Windows DLLs that are designed to minimize file size and memory consumption. Embedded databases avoid the installation hassles, heavy resource usage and maintenance cost associated with client/server RDBMSs that run as an operating system service.

Andrew Hudson's December 2005 'Open Source databases rounded up and rodeoed' review for The Enquirer provides brief descriptions of one commercial and eight open source database purveyors/products: Sleepycat, MySQL, PostgreSQL, Ingres, InnoBase, Firebird, IBM Cloudscape (a.k.a, Derby), Genezzo, and Oracle. Oracle Sleepycat* isn't an SQL Database, Oracle InnoDB* is an OEM database engine that's used by MySQL, and Genezzo is a multi-user, multi-server distributed database engine written in Perl. These special-purpose databases are beyond the scope of this post.

* Oracle purchased Sleepycat Software, Inc. in February 2006 and purchased Innobase OY in October 2005 . The press release states: 'Oracle intends to continue developing the InnoDB technology and expand our commitment to open source software.'

Derby is an open-source release by the Apache Software Foundation of the Cloudscape Java-based database that IBM acquired when it bought Informix in 2001. IBM offers a commercial release of Derby as IBM Cloudscape 10.1. Derby is a Java class library that has a relatively light footprint (2 MB), which make it suitable for client/server synchronization with the IBM DB2 Everyplace Sync Server in mobile applications. The IBM DB2 Everyplace Express Edition isn't open source or free*, so it doesn't qualify for this post. The same is true for the corresponding Sybase SQL Anywhere components.**

* IBM DB2 Everyplace Express Edition with synchronization costs $379 per server (up to two processors) and $79 per user. DB2 Everyplace Database Edition (without DB2 synchronization) is $49 per user. (Prices are based on those when IBM announced version 8 in November 2003.)

** Sybase's iAnywhere subsidiary calls SQL Anywhere 'the industry's leading mobile database.' A Sybase SQL Anywhere Personal DB seat license with synchronization to SQL Anywhere Server is $119; the cost without synchronization wasn't available from the Sybase Web site. Sybase SQL Anywhere and IBM DB2 Everyplace perform similar replication functions.

Sun's Java DB, another commercial version of Derby, comes with the Solaris Enterprise Edition, which bundles Solaris 10, the Java Enterprise System, developer tools, desktop infrastructure and N1 management software. A recent Between the Lines blog entry by ZDNet's David Berlind waxes enthusiastic over the use of Java DB embedded in a browser to provide offline persistence. RedMonk analyst James Governor and eWeek's Lisa Vaas wrote about the use of Java DB as a local data store when Tim Bray announced Sun's Derby derivative and Francois Orsini demonstrated Java DB embedded in the Firefox browser at the ApacheCon 2005 conference.

Firebird is derived from Borland's InterBase 6.0 code, the first commercial relational database management system (RDBMS) to be released as open source. Firebird has excellent support for SQL-92 and comes in three versions: Classic, SuperServer and Embedded for Windows, Linux, Solaris, HP-UX, FreeBSD and MacOS X. The embedded version has a 1.4-MB footprint. Release Candidate 1 for Firebird 2.0 became available on March 30, 2006 and is a major improvement over earlier versions. Borland continues to promote InterBase, now at version 7.5, as a small-footprint, embedded database with commercial Server and Client licenses.

SQLite is a featherweight C library for an embedded database that implements most SQL-92 entry- and transitional-level requirements (some through the JDBC driver) and supports transactions within a tiny 250-KB code footprint. Wrappers support a multitude of languages and operating systems, including Windows CE, SmartPhone, Windows Mobile, and Win32. SQLite's primary SQL-92 limitations are lack of nested transactions, inability to alter a table design once committed (other than with RENAME TABLE and ADD COLUMN operations), and foreign-key constraints. SQLite provides read-only views, triggers, and 256-bit encryption of database files. A downside is the the entire database file is locked when while a transaction is in progress. SQLite uses file access permissions in lieu of GRANT and REVOKE commands. Using SQLite involves no license; its code is entirely in the public domain.

The Mozilla Foundation's Unified Storage wiki says this about SQLite: 'SQLite will be the back end for the unified store [for Firefox]. Because it implements a SQL engine, we get querying 'for free', without having to invent our own query language or query execution system. Its code-size footprint is moderate (250k), but it will hopefully simplify much existing code so that the net code-size change should be smaller. It has exceptional performance, and supports concurrent access to the database. Finally, it is released into the public domain, meaning that we will have no licensing issues.'

Vieka Technology, Inc.'s eSQL 2.11 is a port of SQLite to Windows Mobile (Pocket PC and Smartphone) and Win32, and includes development tools for Windows devices and PCs, as well as a .NET native data provider. A conventional ODBC driver also is available. eSQL for Windows (Win32) is free for personal and commercial use; eSQL for Windows Mobile requires a license for commercial (for-profit or business) use.

HSQLDB isn't on most reviewers' radar, which is surprising because it's the default database for OpenOffice.org (OOo) 2.0's Base suite member. HSQLDB 1.8.0.1 is an open-source (BSD license) Java dembedded database engine based on Thomas Mueller's original Hypersonic SQL Project. Using OOo's Base feature requires installing the Java 2.0 Runtime Engine (which is not open-source) or the presence of an alternative open-source engine, such as Kaffe. My prior posts about OOo Base and HSQLDB are here, here and here.

The HSQLDB 1.8.0 documentation on SourceForge states the following regarding SQL-92 and later conformance:

HSQLDB 1.8.0 supports the dialect of SQL defined by SQL standards 92, 99 and 2003. This means where a feature of the standard is supported, e.g. left outer join, the syntax is that specified by the standard text. Many features of SQL92 and 99 up to Advanced Level are supported and here is support for most of SQL 2003 Foundation and several optional features of this standard. However, certain features of the Standards are not supported so no claim is made for full support of any level of the standards.

Other less well-known embedded databases designed for or suited to mobile deployment are Mimer SQL Mobile and VistaDB 2.1 . Neither product is open-source and require paid licensing; VistaDB requires a small up-front payment by developers but offers royalty-free distribution.

Java DB, Firebird embedded, SQLite and eSQL 2.11 are contenders for lightweight PC and mobile device database projects that aren't Windows-only.

SQL Server 2005 Everywhere

If you're a Windows developer, SQL Server Mobile is the logical embedded database choice for mobile applications for Pocket PCs and Smartphones. Microsoft's April 19, 2006 press release delivered the news that SQL Server 2005 Mobile Editon (SQL Mobile or SSM) would gain a big brother—SQL Server 2005 Everywhere Edition.

Currently, the SSM client is licensed (at no charge) to run in production on devices with Windows CE 5.0, Windows Mobile 2003 for Pocket PC or Windows Mobile 5.0, or on PCs with Windows XP Tablet Edition only. SSM also is licensed for development purposes on PCs running Visual Studio 2005. Smart Device replication with SQL Server 2000 SP3 and later databases has been the most common application so far for SSM.

By the end of 2006, Microsoft will license SSE for use on all PCs running any Win32 version or the preceding device OSs. A version of SQL Server Management Studio Express (SSMSE)—updated to support SSE—is expected to release by the end of the year. These features will qualify SSE as the universal embedded database for Windows client and smart-device applications.

For more details on SSE, read John Galloway's April 11, 2006 blog post and my 'SQL Server 2005 Mobile Goes Everywhere' article for the FTPOnline Special Report on SQL Server."

(Via OakLeaf Systems.)

Anyway, Marc's article is a very refreshing read because it provides a really good insight into the general landscape of a rapidly evolving Web alongside genuine appreciation of our broader timeless pursuit of "Openness".

To really help this document provide additional value have scrapped the content of the original post and dumped it below so that we can appreciate the value of the links embedded within the article (note: thanks to Virtuoso I only had to paste the content into my blog, the extraction to my Linkblog and Blog Summary Pages are simply features of my Virtuoso based Blog Engine):

Breaking the Web Wide Open! (complete story)

Even the web giants like AOL, Google, MSN, and Yahoo need to observe these open standards, or they'll risk becoming the "walled gardens" of the new web and be coolio no more.

Marc Canter [Broadband Mechanics, Inc.] | POSTED: 09.26.05 @12:00

Editorial Note: Several months ago, AlwaysOn got a personal invitation from Yahoo founder Jerry Yang "to see and give us feedback on our new social media product, y!360." We were happy to oblige and dutifully showed up, joining a conference room full of hard-core bloggers and new, new media types. The geeks gave Yahoo 360 an overwhelming thumbs down, with comments like, "So the only services I can use within this new network are Yahoo services? What if I don't use Yahoo IM?" In essence, the Yahoo team was booed for being "closed web," and we heartily agreed. With Yahoo 360, Yahoo continues building its own "walled garden" to control its 135 million customers—an accusation also hurled at AOL in the early 1990s, before AOL migrated its private network service onto the web. As the  Economist recently noted, "Yahoo, in short, has old media plans for the new-media era." The irony to our view here is, of course, that today's AO Network is also a "closed web." In the end, Mr. Yang's thoughtful invitation and our ensuing disappointment in his new service led to the assignment of this article. It also confirmed our existing plan to completely revamp the AO Network around open standards. To tie it all together, we recruited the chief architect of our new site, the notorious Marc Canter, to pen this piece. We look forward to our reader feedback. Breaking the Web Wide Open! By Marc Canter For decades, "walled gardens" of proprietary standards and content have been the strategy of dominant players in mainframe computer software, wireless telecommunications services, and the World Wide Web—it was their successful lock-in strategy of keeping their customers theirs. But like it or not, those walls are tumbling down. Open web standards are being adopted so widely, with such value and impact, that the web giants—Amazon, AOL, eBay, Google, Microsoft, and Yahoo—are facing the difficult decision of opening up to what they don't control. The online world is evolving into a new open web (sometimes called the Web 2.0), which is all about being personalized and customized for each user. Not only open source software, but open standards are becoming an essential component. Many of the web giants have been using open source software for years. Most of them use at least parts of the LAMP (Linux, Apache, MySQL, Perl/Python/PHP) stack, even if they aren't well-known for giving back to the open source community. For these incumbents that grew big on proprietary web services, the methods, practices, and applications of open source software development are difficult to fully adopt. And the next open source movements—which will be as much about open standards as about code—will be a lot harder for the incumbents to exploit. While the incumbents use cheap open source software to run their back-ends systems, their business models largely depend on proprietary software and algorithms. But our view a new slew of open software, open protocols, and open standards will confront the incumbents with the classic Innovator's Dilemma.  Should they adopt these tools and standards, painfully cannibalizing their existing revenue for a new unproven concept, or should they stick with their currently lucrative model with the risk that eventually a bunch of upstarts eat their lunch? Credit should go to several of the web giants who have been making efforts to "open up." Google, Yahoo, eBay, and Amazon all have Open APIs (Application Programming Interfaces) built into their data and systems. Any software developer can access and use them for whatever creative purposes they wish. This means that the API provider becomes an open platform for everyone to use and build on top of. This notion has expanded like wildfire throughout the blogosphere, so nowadays, Open APIs are pretty much required. Other incumbents also have open strategies. AOL has got the RSS religion, providing a feedreader and RSS search in order to escape the "walled garden of content" stigma. Apple now incorporates podcasts, the "personal radio shows" that are latest rage in audio narrowcasting, into iTunes. Even Microsoft is supporting open standards, for example by endorsing SIP (Session Initiation Protocol) for internet telephony and conferencing over Skype's proprietary format or one of its own devising. But new open standards and protocols are in use, under construction, or being proposed every day, pushing the envelope of where we are right now. Many of these standards are coming from startup companies and small groups of developers, not from the giants. Together with the Open APIs, those new standards will contribute to a new, open infrastructure. Tens of thousands of developers will use and improve this open infrastructure to create new kinds of web-based applications and services, to offer web users a highly personalized online experience. A Brief History of Openness At this point, I have to admit that I am not just a passive observer, full-time journalist or "just some blogger"—but an active evangelist and developer of these standards. It's the vision of "open infrastructure" that's driving my company and the reason why I'm writing this article. This article will give you some of the background behind on these standards, and what the evolution of the next generation of open standards will look like. Starting back in the 1980s, establishing a software standard was a key strategy for any software company. My former company, MacroMind (which became Macromedia), achieved this goal early on with Director. As Director evolved into Flash, the world saw that other companies besides Microsoft, Adobe, and Apple could establish true cross-platform, independent media standards. Then Tim Berners-Lee and Marc Andreessen came along, and changed the rules of the software business and of entrepreneurialism. No matter how entrenched and "standardized" software was, the rug could still get pulled out from under it. Netscape did it to Microsoft, and then Microsoft did it back  to Netscape. The web evolved, and lots of standards evolved with it. The leading open source standards (such as the LAMP stack) became widely used alternatives to proprietary closed-source offerings. Open standards are more than just technology. Open standards mean sharing, empowering, and community support. Someone floats a new idea (or meme) and the community runs with it – with each person making their own contributions to the standard – evolving it without a moment's hesitation about "giving away their intellectual property." One good example of this was Dave Sifry, who built the Technorati blog-tracking technology inspired by the Blogging Ecosystem, a weekend project by young hacker Phil Pearson. Dave liked what he saw and he ran with it—turning Technorati into what it is today. Dave Winer has contributed enormously to this area of open standards. He defined and personally created several open standards and protocols—such as RSS, OPML, and XML-RPC. Dave has also helped build the blogosphere through his enthusiasm and passion. By 2003, hundreds of programmers were working on creating and establishing new standards for almost everything. The best of these new standards have evolved into compelling web services platforms – such as del.icio.us, Webjay, or Flickr. Some have even spun off formal standards – like XSPF (a standard for playlists) or instant messaging standard XMPP (also known as Jabber). Today's Open APIs are complemented by standardized Schemas—the structure of the data itself and its associated meta-data. Take for example a podcasting feed. It consists of: a) the radio show itself, b) information on who is on the show, what the show is about and how long the show is (the meta-data) and also c) API calls to retrieve a show (a single feed item) and play it from a specified server. The combination of Open APIs, standardized schemas for handling meta-data, and an industry which agrees on these standards are breaking the web wide open right now. So what new open standards should the web incumbents—and you—be watching? Keep an eye on the following developments: Identity Attention Open Media Microcontent Publishing Open Social Networks Tags Pinging Routing Open Communications Device Management and Control 1. Identity Right now, you don't really control your own online identity. At the core of just about every online piece of software is a membership system. Some systems allow you to browse a site anonymously—but unless you register with the site you can't do things like search for an article, post a comment, buy something, or review it. The problem is that each and every site has its own membership system. So you constantly have to register with new systems, which cannot share data—even you'd want them to. By establishing a "single sign-on" standard, disparate sites can allow users to freely move from site to site, and let them control the movement of their personal profile data, as well as any other data they've created. With Passport, Microsoft unsuccessfully attempted to force its proprietary standard on the industry. Instead, a world is evolving where most people assume that users want to control their own data, whether that data is their profile, their blog posts and photos, or some collection of their past interactions, purchases, and recommendations. As long as users can control their digital identity, any kind of service or interaction can be layered on top of it. Identity 2.0 is all about users controlling their own profile data and becoming their own agents. This way the users themselves, rather than other intermediaries, will profit from their ID info. Once developers start offering single sign-on to their users, and users have trusted places to store their data—which respect the limits and provide access controls over that data, users will be able to access personalized services which will understand and use their personal data. Identity 2.0 may seem like some geeky, visionary future standard that isn't defined yet, but by putting each user's digital identity at the core of all their online experiences, Identity 2.0 is becoming the cornerstone of the new open web. The Initiatives: Right now, Identity 2.0 is under construction through various efforts from Microsoft (the "InfoCard" component built into the Vista operating system and its "Identity Metasystem"), Sxip Identity, Identity Commons, Liberty Alliance, LID (NetMesh's Lightweight ID), and SixApart's OpenID. More Movers and Shakers: Identity Commons and Kaliya Hamlin, Sxip Identity and Dick Hardt, the Identity Gang and Doc Searls, Microsoft's Kim Cameron, Craig Burton, Phil Windley, and Brad Fitzpatrick, to name a few. 2. Attention How many readers know what their online attention is worth? If you don't, Google and Yahoo do—they make their living off our attention. They know what we're searching for, happily turn it into a keyword, and sell that keyword to advertisers. They make money off our attention. We don't. Technorati and friends proposed an attention standard, Attention.xml, designed to "help you keep track of what you've read, what you're spending time on, and what you should be paying attention to." AttentionTrust is an effort by Steve Gillmor and Seth Goldstein to standardize on how captured end-user performance, browsing, and interest data are used. Blogger Peter Caputa gives a good summary of AttentionTrust: "As we use the web, we reveal lots of information about ourselves by what we pay attention to. Imagine if all of that information could be stored in a nice neat little xml file. And when we travel around the web, we can optionally share it with websites or other people. We can make them pay for it, lease it ... we get to decide who has access to it, how long they have access to it, and what we want in return. And they have to tell us what they are going to do with our Attention data." So when you give your attention to sites that adhere to the AttentionTrust, your attention rights (you own your attention, you can move your attention, you can pay attention and be paid for it,  and you can see how your attention is used) are guaranteed. Attention data is crucial to the future of the open web, and Steve and Seth are making sure that no one entity or oligopoly controls it. Movers and Shakers: Steve Gillmor, Seth Goldstein, Dave Sifry and the other Attention.xml folks. 3. Open Media Proprietary media standards—Flash, Windows Media, and QuickTime, to name a few —helped liven up the web. But they are proprietary standards that try to keep us locked in, and they weren't created from scratch to handle today's online content. That's why, for many of us, an Open Media standard has been a holy grail. Yahoo's new Media RSS standard brings us one step closer to achieving open media, as do Ogg Vorbis audio codecs, XSPF playlists, or MusicBrainz. And several sites offer digital creators not only a place to store their content, but also to sell it. Media RSS (being developed by Yahoo with help from the community) extends RSS and combines it with "RSS enclosures" —adds metadata to any media item—to create a comprehensive solution for media "narrowcasters." To gain acceptance for Media RSS, Yahoo knows it has to work with the community. As an active member of this community, I can tell you that we'll create Media RSS equivalents for rdf (an alternative subscription format) and Atom (yet another  subscription format), so no one will be able to complain that Yahoo is picking sides in format wars. When Yahoo announced the purchase of Flickr, Yahoo founder Jerry Yang insinuated that Yahoo is acquiring "open DNA" to turn Yahoo into an open standards player. Yahoo is showing what happens when you take a multi-billion dollar company and make openness one of its core values—so Google, beware, even if Google does have more research fellows and Ph.D.s. The open media landscape is far and wide, reaching from game machine hacks and mobile phone downloads to PC-driven bookmarklets, players, and editors, and it includes many other standardization efforts. XSPF is an open standard for playlists, and MusicBrainz is an alternative to the proprietary (and originally effectively stolen) database that Gracenote licenses. Ourmedia.org is a community front-end to Brewster Kahle's Internet Archive. Brewster has promised free bandwidth and free storage forever to any content creators who choose to share their content via the Internet Archive. Ourmedia.org is providing an easy-to-use interface and community to get content in and out of the Internet Archive, giving ourmedia.org users the ability to share their media anywhere they wish, without being locked into a particular service or tool. Ourmedia plans to offer open APIs and an open media registry that interconnects other open media repositories into a DNS-like registry (just like the www domain system), so folks can browse and discover open content across many open media services. Systems like Brightcove and Odeo support the concept of an open registry, and hope to work with digital creators to sell their work to fulfill the financial aspect of the "Long Tail." More Movers and Shakers: Creative Commons, the Open Media Network, Jay Dedman, Ryanne Hodson, Michael Verdi, Eli Chapman, Kenyatta Cheese, Doug Kaye, Brad Horowitz, Lucas Gonze, Robert Kaye, Christopher Allen, Brewster Kahle, JD Lasica, and indeed, Marc Canter, among others. 4. Microcontent Publishing Unstructured content is cheap to create, but hard to search through. Structured content is expensive to create, but easy to search. Microformats resolve the dilemma with simple structures that are cheap to use and easy to search. The first kind of widely adopted microcontent is blogging. Every post is an encapsulated idea, addressable via a URL called a permalink. You can syndicate or subscribe to this microcontent using RSS or an RSS equivalent, and news or blog aggregators can then display these feeds in a convenient readable fashion. But a blog post is just a block of unstructured text—not a bad thing, but just a first step for microcontent. When it comes tostructured data, such as personal identity profiles, product reviews, or calendar-type event data, RSS was not designed to maintain the integrity of the structures. Right now, blogging doesn't have the underlying structure necessary for full-fledged microcontent publishing. But that will change. Think of local information services (such as movie listings, event guides, or restaurant reviews) that any college kid can access and use in her weekend programming project to create new services and tools. Today's blogging tools will evolve into microcontent publishing systems, and will help spread the notion of structured data across the blogosphere. New ways to store, represent and produce microcontent will create new standards, such as Structured Blogging and Microformats. Microformats differ from RSS feeds in that you can't subscribe to them. Instead, Microformats are embedded into webpages and discovered by search engines like Google or Technorati. Microformats are creating common definitions for "What is a review or event? What are the specific fields in the data structure?" They can also specify what we can do with all this information.OPML (Outline Processor Markup Language) is a hierarchical file format for storing microcontent and structured data. It was developed by Dave Winer of RSS and podcast fame. Events are one popular type of microcontent. OpenEvents is already working to create shared databases of standardized events, which would get used by a new generation of event portals—such as Eventful/EVDB, Upcoming.org, and WhizSpark. The idea of OpenEvents is that event-oriented systems and services can work together to establish shared events databases (and associated APIs) that any developer could then use to create and offer their own new service or application. OpenReviews is still in the conceptual stage, but it would make it possible to provide open alternatives to closed systems like Epinions, and establish a shared database of local and global reviews. Its shared open servers would be filled with all sorts of reviews for anyone to access. Why is this important? Because I predict that in the future, 10 times more people will be writing reviews than maintaining their own blog. The list of possible microcontent standards goes on: OpenJobpostings, OpenRecipes, and even OpenLists. Microsoft recently revealed that it has been working on an important new kind of microcontent: Lists—so OpenLists will attempt to establish standards for the kind of lists we all use, such as lists of Links, lists of To Do Items, lists of People, Wish Lists, etc. Movers and Shakers: Tantek Çelik and Kevin Marks of Technorati, Danny Ayers, Eric Meyer, Matt Mullenweg, Rohit Khare, Adam Rifkin, Arnaud Leene, Seb Paquet, Alf Eaton, Phil Pearson, Joe Reger, Bob Wyman among others. 5. Open Social Networks I'll never forget the first time I met Jonathan Abrams, the founder of Friendster. He was arrogant and brash and he claimed he "owned"  all his users, and that he was going to monetize them and make a fortune off them. This attitude robbed Friendster of its momentum, letting MySpace, Facebook, and other social networks take Friendster's place. Jonathan's notion of social networks as a way to control users is typical of the Web 1.0 business model and its attitude towards users in general. Social networks have become one of the battlegrounds between old and new ways of thinking. Open standards for Social Networking will define those sides very clearly. Since meeting Jonathan, I have been working towards finding and establishing open standards for social networks. Instead of closed, centralized social networks with 10 million people in them, the goal is making it possible to have 10 million social networks that each have 10 people in them. FOAF (which stands for Friend Of A Friend, and describes people and relationships in a way that computers can parse) is a schema to represent not only your personal profile's meta-data, but your social network as well. Thousands of researchers use the FOAF schema in their "Semantic Web" projects to connect people in all sorts of new ways. XFN is a microformat standard for representing your social network, while vCard (long familiar to users of contact manager programs like Outlook) is a microformat that contains your profile information. Microformats are baked into any xHTML webpage, which means thatany blog, social network page, or any webpage in general can "contain" your social network in it—and be used byany compatible tool, service or application. PeopleAggregator is an earlier project now being integrated into open content management framework Drupal. The PeopleAggregator APIs will make it possible to establish relationships, send messages, create or join groups, and post between different social networks. (Sneak preview: this technology will be available in the upcoming GoingOn Network.) All of these open social networking standards mean that inter-connected social networks will form a mesh that will parallel the blogosphere. This vibrant, distributed, decentralized world will be driven by open standards: personalized online experiences are what the new open web will be all about—and what could be more personalized than people's networks? Movers and Shakers: Eric Sigler, Joel De Gan, Chris Schmidt, Julian Bond, Paul Martino, Mary Hodder, Drummond Reed, Dan Brickley, Randy Farmer, and Kaliya Hamlin, to name a few. 6. Tags Nowadays, no self-respecting tool or service can ship without tags. Tags are keywords or phrases attached to photos, blog posts, URLs, or even video clips. These user- and creator-generated tags are an open alternative to what used to be the domain of librarians and information scientists: categorizing information and content using taxonomies. Tags are instead creating "folksonomies." The recently proposed OpenTags concept would be an open, community-owned version of the popular Technorati Tags service. It would aggregate the usage of tags across a wide range of services, sites, and content tools. In addition to Technorati's current tag features, OpenTags would let groups of people share their tags in "TagClouds." Open tagging is likely to include some of the open identity features discussed above, to create a tag system that is resilient to spam, and yet trustable across sites all over the web. OpenTags owes a debt to earlier versions of shared tagging systems, which include Topic Exchange and something called the k-collector—a knowledge management tag aggregator—from Italian company eVectors. Movers & Shakers: Phil Pearson, Matt Mower , Paolo Valdemarin, and Mary Hodder and Drummond Reed again, among others. 7. Pinging Websites used to be mostly static. Search engines that crawled (or "spidered") them every so often did a good enough job to show reasonably current versions of your cousin's homepage or even Time magazine's weekly headlines. But when blogging took off, it became hard for search engines to keep up. (Google has only just managed to offer blog-search functionality, despite buying Blogger back in early 2003.) To know what was new in the blogosphere, users couldn't depend on services that spidered webpages once in a while. The solution: a way for blogs themselves to automatically notify blog-tracking sites that they'd been updated. Weblogs.com was the first blog "ping service": it displayed the name of a blog whenever that blog was updated. Pinging sites helped the blogosphere grow, and more tools, services, and portals started using pinging in new and different ways. Dozens of pinging services and sites—most of which can't talk to each other—sprang up. Matt Mullenweg (the creator of open source blogging software WordPress) decided that a one-stop service for pinging was needed. He created Ping-o-Matic—which aggregates ping services and simplifies the pinging process for bloggers and tool developers. With Ping-o-Matic, any developer can alert all of the industry's blogging tools and tracking sites at once. This new kind of open standard, with shared infrastructure, is a critical to the scalability of Web 2.0 services. As Matt said: There are a number of services designed specifically for tracking and connecting blogs. However it would be expensive for all the services to crawl all the blogs in the world all the time. By sending a small ping to each service you let them know you've updated so they can come check you out. They get the freshest data possible, you don't get a thousand robots spidering your site all the time. Everybody wins. Movers and Shakers: Matt Mullenweg, Jim Winstead, Dave Winer 8. Routing Bloggers used to have to manually enter the links and content snippets of blog posts or news items they wanted to blog. Today, some RSS aggregators can send a specified post directly into an associated blogging tool: as bloggers browse through the feeds they subscribe to, they can easily specify and send any post they wish to "reblog" from their news aggregator or feed reader into their blogging tool. (This is usually referred to as "BlogThis.") As structured blogging comes into its own (see the section on Microcontent Publishing), it will be increasingly important to maintain the structural integrity of these pieces of microcontent when reblogging them. Promising standard RedirectThis will combine a "BlogThis"-like capability while maintaining the integrity of the microcontent. RedirectThis will let bloggers and content developers attach a simple "PostThis" button to their posts. Clicking on that button will send that post to the reader/blogger's favorite blogging tool. This favorite tool is specified at the RedirectThis web service, where users register their blogging tool of choice. RedirectThis also helps maintain the integrity and structure of microcontent—then it's just up to the user to prefer a blogging tool that also attains that lofty goal of microcontent integrity. OutputThis is another nascent web services standard, to let bloggers specify what "destinations" they'd like to have as options in their blogging tool. As new destinations are added to the service, more checkboxes would get added to their blogging tool—allowing them to route their published microcontent to additional destinations. Movers and Shakers: Michael Migurski, Lucas Gonze 9. Open Communications Likely, you've experienced the joys of finding friends on AIM or Yahoo Messenger, or the convenience of Skyping with someone overseas. Not that you're about to throw away your mobile phone or BlackBerry, but for many, also having access to Instant Messaging (IM) and Voice over IP (VoIP) is crucial. IM and VoIP are mainstream technologies that already enjoy the benefits of open standards. Entire industries are born—right this second—based around these open standards. Jabber has been an open IM technology for years—in fact, as XMPP, it was officially dubbed a standard by the IETF. Although becoming an official IETF standard is usually the kiss of death, Jabber looks like it'll be around for a while, as entire generations of collaborative, work-group applications and services have been built on top of its messaging protocol. For VoIP, Skype is clearly the leading standard today—though one could argue just how "open" it is (and defenders of the IETF's SIP standard often do). But it is free and user-friendly, so there won't be much argument from users  about it being insufficiently open. Yet there may be a cloud on Skype's horizon: web behemoth Google recently released a beta of Google Talk, an IM client committed to open standards. It currently supports XMPP, and will support SIP for VoIP calls. Movers and Shakers: Jeremie Miller, Henning Schulzrinne, Jon Peterson, Jeff Pulver 10. Device Management and Control To access online content, we're using more and more devices. BlackBerrys, iPods, Treos, you name it. As the web evolves, more and more different devices will have to communicate with each other to give us the content we want when and where we want it. No-one wants to be dependent on one vendor anymore—like, say, Sony—for their laptop, phone, MP3 player, PDA, and digital camera, so that it all works together. We need fully interoperable devices, and the standards to make that work. And to fully make use of how content is moving online content and innovative web services, those standards need to be open. MIDI (musical instrument digital interface), one of the very first open standards in music, connected disparate vendors' instruments, post-production equipment, and recording devices. But MIDI is limited, and MIDI II has been very slow to arrive. Now a new standard for controlling musical devices has emerged: OSC (Open SoundControl). This protocol is optimized for modern networking technology and inter-connects music, video and controller devices with "other multimedia devices." OSC is used by a wide range of developers, and is being taken up in the mainstream MIDI marketplace. Another open-standards-based device management technology is ZigBee, for building wireless intelligence and network monitoring into all kinds of devices. ZigBee is supported by many networking, consumer electronics, and mobile device companies.       · · · · · ·     The Change to Openness The rise of open source software and its "architecture of participation" are completely shaking up the old proprietary-web-services-and-standards approach. Sun Microsystems—whose proprietary Java standard helped define the Web 1.0—is opening its Solaris OS and has even announced the apparent paradox of an open-source Digital Rights Management system. Today's incumbents will have to adapt to the new openness of the Web 2.0. If they stick to their proprietary standards, code, and content, they'll become the new walled gardens—places users visit briefly to retrieve data and content from enclosed data silos, but not where users "live." The incumbents' revenue models will have to change. Instead of "owning" their users, users will know they own themselves, and will expect a return on their valuable identity and attention. Instead of being locked into incompatible media formats, users will expect easy access to digital content across many platforms. Yesterday's web giants and tomorrow's users will need to find a mutually beneficial new balance—between open and proprietary, developer and user, hierarchical and horizontal, owned and shared, and compatible and closed. Marc Canter is an active evangelist and developer of open standards. Early in his career, Marc founded MacroMind, which became Macromedia. These days, he is CEO of Broadband Mechanics, a founding member of the Identity Gang and of ourmedia.org. Broadband Mechanics is currently developing the GoingOn Network (with the AlwaysOn Network), as well as an open platform for social networking called the PeopleAggregator. A version of the above post appears in the Fall 2005 issue of AlwaysOn's quarterly print blogozine, and ran as a four-part series on the AlwaysOn Network website.

(Via Marc's Voice.)

An immediate implication is that you can generate Google AdWords based adds using any development environment (Virtuoso's SQL Stored Procedure Language,  any .NET bound language, Java, C/C++, PHP, Ruby, Perl, Python, TCL etc.) that supports SOAP, WSDL, and I would presume WS-Security.

An even more interesting offshoot of this initiative from Google, is the fact that it could bring a degree of clarity to the issue of multi-protocol and multi-purpose servers (what I call Universal Servers e.g. OpenLink Virtuoso). For instance, you could manage AdWords campaigns across product portfolios using Triggers (the SQL database kind) or Notification Services.

Mozilla's SQL Support allows applications to directly connect to SQL databases. A web application no longer needs to pass information through a scripting language, such as Perl or Python, in order to recieve information it can use. The removal of the layer seperating applications and data simplifies the job of the programmer.

Somehow I missed this effort, and only stumbled across it today after experimenting with Virtuoso's SyncML features (and then pondering about OutLook, WinFS, and what may or may not happen with SyncML support - another story).

As usual the SQL binding to Mozilla caught my attention (I do recall trying to get Marc and Jim Clark to head down this path many years ago via an email; at least Jim acknowledged not knowing that much about SQL and past it on.., and as for Marc well... nothing happened).

A few

Email: Killer App Or Just A Killer? While many people consider email to be the "killer app" that brought the internet into homes and businesses, now some are saying that email has become annoying and costly. The most obvious issue is with spam, but there are other things as well. Maintaining an email server is a pain, and keeping email free from viruses is an additional cost. For companies that monitor email (and there are more and more), that's yet another expense. Finally, since there are questions about email security, some companies are telling employees not to use email for sensitive material. Thus, for many companies, email is only useful for informal communications, and you can only find those messages once you wade through all the spam and viruses - or so this article would have you believe. It's really not that bad - and there are reasonable technology solutions that should be able to keep most companies afloat with minimal costs. Yes, it's annoying, but the benefits of having email certainly outweigh the annoyances associated with it. [via Techdirt]

One of the reasons for e-mail enabling Virtuoso (circa. 2000) was to set the stage for addressing what I anticipated would ultimately become the Spam Monster. This is how the solution was envisioned.

Build a Driver/Sink that could be attached to the SMTP Agents such as Sendmail, Excim, Exchange etc.. such that the mail received is actually stored in a DBMS Engine (in this case Virtuoso or an ODBC accessible database). Once the mail is in the database it is then possible for Triggers to handle filtering of the Mime headers and mail body (using regular experessions). The end result being that Spam and Virulent mails are already filtered prior to POP or IMAP retrieval.

With the emergence of Bayesian Spam Filters and other Anti Spam solutions there remains a possibility for this pursuing the best of both worlds. Enhance the DB Engine via its extensions API (In the case of Virtuosowhich supports Python, Perl etc..), or enhance the Mail Driver/Sink by extending it in a similar manner (a little more work if extensibility isn't part of the original Mail Sink design). My preference is obviously to handle this at the database level so that the Bayesian spam filter becomes a Trigger on the table into which the mail is stored.

With a database in the mix I pretty much have a rules based engine for e-mail and also a pretty flexible mechanism for dealing with false positives (nothing's perfect!) since they remain in the database too, but not automatically part of the IMAP or POP retrieval process.

At the end of the day e-mail is data and we simply need to look at data

This Blog Site is actually powered by Virtuoso 3.2 (has been doing so prior to the announcement). Hmm. product utilization preceding press release? Why not?

OpenLink adds Weblog client and server functionality to
Virtual Database Engine for SQL, XML, and Web Services

Burlington, MA. June 25, 2003 - OpenLink Software, Inc., a leading provider of universal data access and enterprise information integration middleware, announces Virtuoso 3.2 — the latest edition of its cross platform Virtual Database for SQL, XML, and Web Services — for Mac® OS X.

The new release incorporates full client and server support for the Blogger, Moveable Type, and MetaWeblog APIs, providing users with choice over location, format, data storage, development environment, and host operating system, for personal, community, and corporate Weblogs. The new release also facilitates the transparent integration of Weblog data with other enterprise data sources.

Full Press Release

Putting together the community site took 5 minutes and it basically involved the following steps:

1. Standard installation from installer program (Mac OS X in this case, but Windows, Linux, and UNIX supported)

2. Creation of WebDAV user account for WebDAV repository (where all the gems reside)

3. Clicking on the "Generate Web Site" button situated in the Weblog menu tree with the Virtuoso HTML based Admin UI

4. Filled up my channel and blogrolls by asking Virtuoso to use it’s very old web content aggregation functionality

5. Setup my upstreams (so that I post once and propagate to my numerous blog sites on a conditional basis)

6. Create a Virtuoso HTTP Virtual Domain for the community/personal Blog

7. Start blogging using any Blog Client that supports; Blogger API, MetaWeblog, or Moveable Type

No more no less. Most importantly I have a choice of programming languages (VSP, VSX, PHP, ASP.NET, JSP, Perl, Python), operating systems, and databases that constitute the shape and form of my blog home.

See the Virtuoso FAQ for how this all comes together.

More .NET languages are popping up everyday.  I've put together one of the most extensive lists of languages and posted it here. 

Its pulled from many sources including: .NET Languages (ASP.NET), Cetus, Language Vendors (MS GotDotNET), .NET Language Group (MS GotDotNET), Visual Studio Partners: Language Vendors (MS), Mono-list, Google, SourceForge.net

• Ada
  • A# - port of Ada to .NET (Dr. Martin C. Carlisle)
• APL
  • Dyalog.Net - Dyalog APL (Dyadic)
• AsmL
  • Abstract State Machine Language (MS Research)
• Visual Basic
  • VB.NET (MS)
  • mbas (Mono/Ximian)
• C#
  • C# (MS)
  • mcs (Mono/Ximian)
  • cscc (DotGNU Portable.NET)
• Caml
  • F# (ML and Caml), Abstract IL, ILX (MS Research)
• C++
  • Managed Extensions for C++ (MS)
  • Managed and Unmanaged C++ (GotDotNet)
• Cobol
  • NetCOBOL - COBOL for .NET (Fujitsu)
  • Net Express (Micro Focus)
• Delphi
  • Borland Delphi and C++Builder Support for .NET (Borland)
  • Delphi.NET - interoperability tools (Marcus Schmidt)
• Eiffel
  • Eiffel for .NET (Interactive Software Engineering)
• Forth
  • Delta Forth .NET (Valer BOCAN)
• Fortran
  • Lahey/Fujitsu Fortran for .NET (Lahey Computer Systems, Inc.)
  • FTN95 - Fortran for Microsoft .NET (Salford Software Ltd.)
• Java
  • Visual J# .NET (MS)
  • IKVM.NET - Java VM for .NET
• JavaScript
• JScript .NET (GotDotNet)
• JANET - JavaScript-compatible language
• LOGO
  • MonoLOGO (Richard Hestilow)
• Lua
  • Lua.NET: Integrating Lua with Rotor (PUC-RIO
• Mercury
  • Mercury on .NET
• Mondrian
  • Mondrian and Haskell for .NET (Nigel Perry)
• Oberon
  • Active Oberon for .net (ETH Zuerich)
• Perl
  • Perl for .NET, PerlNET (ActiveState SRL.)
• Pascal
  • Component Pascal (QUT)
• PHP
  • PHP Sharp
• Python
  • KOBRA
  • Open Source Python for .NET (Mark Hammond)
• Ruby
  • NetRuby
• RPG
  • ASNA Visual RPG for .NET
• Scheme
  • Scheme (Northwestern University)
• Small Talk
  • S# (SmallScript LLC)
• SML (Standard Meta Language)
  • SML.NET (MS Research, University of Cambridge)
• Visual Basic
  • VB.NET (MS)
  • mbas (Mono/Ximian)

I'll try to keep this updated when I run across a new language.  If anyone knows of any others, let me know.[via Brian Ritchie's Blog]

More .NET languages are popping up everyday.  I've put together one of the most extensive lists of languages and posted it here. 

Its pulled from many sources including: .NET Languages (ASP.NET), Cetus, Language Vendors (MS GotDotNET), .NET Language Group (MS GotDotNET), Visual Studio Partners: Language Vendors (MS), Mono-list, Google, SourceForge.net

• Ada
  • A# - port of Ada to .NET (Dr. Martin C. Carlisle)
• APL
  • Dyalog.Net - Dyalog APL (Dyadic)
• AsmL
  • Abstract State Machine Language (MS Research)
• Visual Basic
  • VB.NET (MS)
  • mbas (Mono/Ximian)
• C#
  • C# (MS)
  • mcs (Mono/Ximian)
  • cscc (DotGNU Portable.NET)
• Caml
  • F# (ML and Caml), Abstract IL, ILX (MS Research)
• C++
  • Managed Extensions for C++ (MS)
  • Managed and Unmanaged C++ (GotDotNet)
• Cobol
  • NetCOBOL - COBOL for .NET (Fujitsu)
  • Net Express (Micro Focus)
• Delphi
  • Borland Delphi and C++Builder Support for .NET (Borland)
  • Delphi.NET - interoperability tools (Marcus Schmidt)
• Eiffel
  • Eiffel for .NET (Interactive Software Engineering)
• Forth
  • Delta Forth .NET (Valer BOCAN)
• Fortran
  • Lahey/Fujitsu Fortran for .NET (Lahey Computer Systems, Inc.)
  • FTN95 - Fortran for Microsoft .NET (Salford Software Ltd.)
• Java
  • Visual J# .NET (MS)
  • IKVM.NET - Java VM for .NET
• JavaScript
• JScript .NET (GotDotNet)
• JANET - JavaScript-compatible language
• LOGO
  • MonoLOGO (Richard Hestilow)
• Lua
  • Lua.NET: Integrating Lua with Rotor (PUC-RIO
• Mercury
  • Mercury on .NET
• Mondrian
  • Mondrian and Haskell for .NET (Nigel Perry)
• Oberon
  • Active Oberon for .net (ETH Zuerich)
• Perl
  • Perl for .NET, PerlNET (ActiveState SRL.)
• Pascal
  • Component Pascal (QUT)
• PHP
  • PHP Sharp
• Python
  • KOBRA
  • Open Source Python for .NET (Mark Hammond)
• Ruby
  • NetRuby
• RPG
  • ASNA Visual RPG for .NET
• Scheme
  • Scheme (Northwestern University)
• Small Talk
  • S# (SmallScript LLC)
• SML (Standard Meta Language)
  • SML.NET (MS Research, University of Cambridge)
• Visual Basic
  • VB.NET (MS)
  • mbas (Mono/Ximian)

I'll try to keep this updated when I run across a new language.  If anyone knows of any others, let me know.[via Brian Ritchie's Blog]