http://www.openlinksw.com/weblog/dav/dav-blog-1/
OpenLink Community Blog
A Collection of blogs by OpenLink Staff
kidehen@openlinksw.com
kidehen@openlinksw.com
2009-11-23T14:21:42Z
Virtuoso Universal Server 05.12.3041
http://www.openlinksw.com/weblog/public/images/vbloglogo.gif
European Commission and the Data Overflow
http://www.openlinksw.com/blog/vdb/blog/?date=2009-10-27#1586
2009-10-27T18:29:51Z
2009-10-27T14:57:31-04:00
<p>The European Commission recently circulated a questionnaire to selected experts on what could be done for the future of big <a href="http://dbpedia.org/resource/Data" id="link-id0x43bae00">data</a>.</p> <p>Since the <a href="http://cordis.europa.eu/fp7/ict/content-knowledge/consultation_en.html" id="link-id1191c0f8">questionnaire is public</a>, I am publishing my answers below.</p> <ol type="1" start="1"> <li> <p> <b>Data and data types</b> </p> <ol type="a" start="1"> <li> <p> <b>What volumes of data are we dealing with today? What is the growth rate? Where can we expect to be in 2015? </b> </p> <p>Private data warehouses of corporations have more than doubled yearly for the past years; hundreds of TB is not exceptional. This will continue. The real shift is in structured data being published in increasing quantities with a minimum level of integrate-ability through use of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x5c7add0">RDF</a> and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x5c7adb8">linked data</a> principles. There are rewards for use of standard vocabularies and identifiers through search engines recognizing such data. There is convergence around <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x5c7ada0">DBpedia</a> identifiers for real-world entities, e.g., most things that would be in the news.</p> <p>This also means that internal data processes and silos may be enriched with this content. There is consequent pressure for accommodating more diversity of data, with more flexible <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x7d87a88">schema</a>.</p> <p>Ultimately, all content presently stored in RDBs and presented in public accessible dynamic web pages will end up on the web of linked data. Examples are product catalogs, price lists, event schedules and the like.</p> <p>The volume of the well known linked data sets is around 10 billion statements. With the above mentioned trends, growth by two or three orders of magnitude by 2015 seems reasonable, This is so especially if explicit semantics are extracted from the document web and if there is some further progress in the precision/recall of such extraction.</p> <p>Relevant sections of this mass of data are a potential addition to any present or future analytics application.</p> <p>Since arbitrary analytics over the database which is the web cannot be economically provided by a centralized search engine, a cloud model may be used for on-demand selection of relevant data and mixing it with private data. This will drive database innovation for the next years even more than the continued classical warehouse growth.</p> <p>Science data is another driver of the data overflow. For example, faster gene sequencing, more accurate measurements in high energy physics, better imaging, and remote sensing will produce large volumes of data. This data has highly regular structure but labeling this data with source and lineage calls for a flexible, schema-last, self-describing model, such as RDF and linked data. Data and <a href="http://dbpedia.org/resource/Metadata" id="link-id0x7a3fb40">metadata</a> should travel together but may have different data models.</p> <p>By and large, the metadata of science data will be another stream to the web of linked data, at least to the degree it is publicly accessible. Restricted circles can and likely will implement similar ideas.</p> </li> <li> <p> <b>What types of data can we deal with intelligently due to their inherent structure (geospatial, temporal, social or <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x5a48058">knowledge</a> graphs, 3D, sensor streams...)?</b> </p> <p>All the above types should be supported inside one DBMS so as to allow efficient querying combining conditions on all these types of data, e.g., <i>photos of sunsets taken last summer in Ibiza, with over 20 megapixels, by people I know.</i> </p> <p>Note that the test for being a sunset is an operation on the image blob that should be taken to the data; the images cannot be economically transferred.</p> <p>Interleaving of all database functions and types becomes increasingly important.</p> </li> </ol> </li> <li> <p> <b>Industries, communities</b> </p> <ol type="a" start="1"> <li> <p> <b>Who is producing these data and why? Could they do it better? How?</b> </p> <p>Right now, projects such as <a href="http://www.bio2rdf.org/" id="link-id0x2a29de8">Bio2RDF</a>, <a href="http://neurocommons.org/page/Main_Page" id="link-id0x7ddaed0">Neurocommons</a>, and DBPedia produce this data. The processes are in place and are reasonable. Incremental improvement is to be expected. These processes, along with the <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id0xbab4dfd0">linked data meme</a> generally taking off, drive demand for better <a href="http://dbpedia.org/resource/Natural_language_processing" id="link-id0x51f4e0">NLP</a> (<a href="http://dbpedia.org/resource/Natural_language_processing" id="link-id0x51a1b48">Natural Language Processing</a>), e.g., <a href="http://dbpedia.org/resource/Entity" id="link-id0x956680">entity</a> and relationship extraction, especially extraction that can produce instance data in given ontologies (e.g., events) using common identifiers (e.g., DBPedia URIs).</p> <p>Mapping of RDBs to RDF is possible, and a W3C working group is developing standards for this. The required baseline level has been reached; the rest is a matter of automating deployment. Within the enterprise, there are advantages to be gained for <a href="http://dbpedia.org/resource/Information" id="link-id0x7da9e80">information</a> integration; e.g., all entities in the CRM space can be integrated with all email and support tickets through giving everything a <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x71673f8">URI</a>. Some of this information may even be published on an <a href="http://dbpedia.org/resource/Extranet" id="link-id0x9aa6e0">extranet</a> for self-service and web-service interfaces. This has been done at small scales and the rest is a matter of spreading adoption and lowering the entry barrier. Incremental progress will take place, eventually resulting in qualitatively better integration along the value chain when adoption is sufficiently widespread.</p> </li> <li> <p> <b>Who is consuming these data and why? Could they do it better? How?</b> </p> <p>Consumers are various. The greatest need is for tools that summarize complex data and allow getting a bird's eye view of what data is in the first instance available. Consuming the data is hindered by the user not even necessarily knowing what data there is. This is somewhat new, as traditionally the business analyst did know the schema of the warehouse and was proficient with <a href="http://dbpedia.org/resource/SQL" id="link-id0x7f7b148">SQL</a> report generators and statistics packages.</p> <p>Where Web 2.0 made the <i>citizen journalist</i>, the web of linked data will make the <i>citizen analyst</i>. For this to happen, with benefits for individuals, enterprises, and governments alike, more work in user interfaces, knowledge discovery, and query composition will be useful. We may envision a "meshup economy" where data is plentiful, but the unit of value and exchange is the smart report that crystallizes actionable value from this ocean.</p> </li> <li> <p> <b>What industrial sectors in Europe could become more competitive if they became much better at managing data?</b> </p> <p>Any sector could benefit. Early adopters are seen in the biomedical field and to an extent in media. </p> </li> <li> <p> <b>Is the regulation landscape imposing constraints (privacy, compliance ...) that don't have today good tool support?</b> </p> <p>The regulation landscape drives database demand through data retention requirements and the like.</p> <p>With data integration, especially with privacy-sensitive data (as in medicine), there are issues of whether one dares put otherwise-shareable information online. Regulation is needed to protect individuals, but integration should still be possible for science.</p> <p>For this, we see a need for progress in applying policy-based approaches (e.g., row level security) to relatively schema-last data such as RDF. This is possible but needs some more work. Also, creating on-the-fly-anonymizing views on data might help.</p> <p>More research is needed for reconciling the need for security with the advantages of broad-based <i>ad hoc</i> integration. Ideally, data should be intelligent, aware of its origins and classification and cautious of whom it interacts with, all of this supported under the covers so that the user could ask anything but the data might refuse to answer or might restrict answers according to the user's profile. This is a tall order and implementing something of the sort is an open question.</p> </li> <li> <p> <b>What are the main practical problem identified for individuals and organizations? Please give examples and tell us about the main obstacles and barriers.</b> </p> <p>We have come across the following:</p> <ul> <li>Knowing that the data exists in the first place.</li> <li>If the data is found, figuring out the provenance, units and precision of measurement, identifiers, and the like.</li> <li>Compatible subject matter but incompatible representation: For example, one has numbers on a map with different maps for different points in time; another has time series of instrument data with geo-location for the instrument. It is only to be expected that the time interval between measurements is not the same. So there is need for a lot of one-off programming to align data.</li> </ul> <p>Other problems have to do with sheer volume, i.e., transfer of data even in a local area network is too slow, let alone over a wide area network. Computation needs to go to the data, and databases need to support this.</p> </li> </ol> </li> <li> <p> <b>Services, software stacks, protocols, standards, benchmarks</b> </p> <ol type="a" start="1"> <li> <p> <b>What combinations of components are needed to deal with these problems?</b> </p> <p>Recent times have seen a proliferation of special purpose databases. Since the data needs of the future are about combining data with maximum agility and minimum performance hit, there is need to gather the currently-separate functionality into an integrated system with sufficient flexibility. We see some of this in integration of map-reduce and scale-out databases. The former antagonists have become partners. Vertica, <a href="http://dbpedia.org/resource/Greenplum" id="link-id0x7a94e70">Greenplum</a>, and OpenLink <a href="http://virtuoso.openlinksw.com" id="link-id0x2ab2868">Virtuoso</a> are example of DBMS featuring work in this direction.</p> <p>Interoperability and at least <i>de facto</i> standards in ways of doing this will emerge.</p> </li> <li> <p> <b>What data exchange and processing mechanisms will be needed to work across platforms and programming languages?</b> </p> <p> <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x78a0458">HTTP</a>, <a href="http://dbpedia.org/resource/XML" id="link-id0x7ff2360">XML</a>, and RDF are in fact very verbose, yet these are the formats and models that have uptake. Thus, these will continue to be used even though one might think binary formats to be more efficient.</p> <p>There are of course science data set standards that are more compressed and these will continue, hopefully adding a practice of rich metadata in RDF.</p> <p>For internals of systems, MPI and TCP/IP with proprietary optimized wire formats will continue. Inter-system communication will likely continue to be HTTP, XML, and RDF as appropriate.</p> </li> <li> <p> <b>What data environments are today so wastefully messy that they would benefit from the development of standards?</b> </p> <p>RDF and <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x5643d70">OWL</a> are not messy but they could use some more performance; we are working on this. <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x152ab18">SPARQL</a> is finally acquiring the capabilities of a serious query language, so things are slowly coming together.</p> <p>Community process for developing application domain specific vocabularies works quite well, even though one could argue it is <i>ad hoc</i> and not up to what a modeling purist might wish.</p> <p>Top-down imposition of standards has a mixed history, with long and expensive development and sometimes no or little uptake, consider some WS* standards for example.</p> </li> <li> <p> <b>What kind of performance is expected or required of these systems? Who will measure it reliably? How?</b> </p> <p>Relational databases have a history of substantial investment in <a href="http://dbpedia.org/resource/Program_optimization" id="link-id0xecc100">optimization</a> and some of them are very good for what they do, e.g., the newer generation of analytics databases.</p> <p>The very large schema-last, no-SQL, sometimes eventually consistent key-value stores have a somewhat shorter history but do fill a real need.</p> <p>These trends will merge: Extreme scale, schema-last, complex queries, even more complex inference, custom code for in-database machine learning and other bulk processing.</p> <p>We find RDF augmented with some binary types at this crossroads. This point of the design space will have to provide performance roughly on the level of today's best relational solution for workloads that fit the relational model. The added cost of schema-last and inference must come down. We are working on this. Research work such as carried out with <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x7ae2890">MonetDB</a> gives clues as to how these aims can be reached.</p> <p>The separation of query language and inference is artificial. After the concepts are mature, these functions will merge and execute close to the data; there are clear evolutionary pressures in this direction.</p> <p>Benchmarks are key. Some gain can be had even from repurposing standard relational benchmarks like <a href="http://www.tpc.org/" id="link-id0x71eb528">TPC</a>-<a href="http://dbpedia.org/resource/TPC-H" id="link-id0x5e16a40">H</a>. But the TPC-H rules do not allow official reporting of such.</p> <p>Development of benchmarks for RDF, complex queries, and inference is needed. A bold challenge to the community, it should be rooted in real-life integration needs and involve high heterogeneity. A key-value store benchmark might also be conceived. A transaction benchmark like TPC-<a href="http://dbpedia.org/resource/C%2B%2B" id="link-id0x78562d0">C</a> might be the basis, maybe augmented with massive user-generated content like reviews and blogs.</p> <p>If benchmarks exist and are not too easy nor inaccessibly difficult nor too expensive to run â think of the high end TPC-C results â then TPC-style rules and processes would be quite adequate. The threshold to publish should be lowered: Everybody runs the TPC workloads internally but few publish.</p> <p>Some EC initiative for benchmarking could make sense, similar to the TREC initiative of the US government. Industry should be consulted for the specific content; possibly the answers to the present questionnaire can provide an approximate direction.</p> <p>Benchmarks should be run by software vendors on their own systems, tuned by themselves. But there should be a process of disclosure and auditing; the TPC rules give an example. Compliance should not be too expensive or time consuming. Some community development for automating these things would be a worthwhile target for EC funding.</p> </li> </ol> </li> <li> <p> <b>Usability and training</b> </p> <ol type="a" start="1"> <li> <p> <b>How difficult will it be for a developer of average competence to deploy components whose core is based on rather deep computer science? Do we all need to understand Monads and Continuations? What can be done to make it ever easier?</b> </p> <p>In the database world, huge advances in technology have taken place behind a relatively simple and stable interface: SQL. For the linked data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x7761e50">web</a>, the same will take place behind SPARQL.</p> <p>Beyond these, for example, programming with MPI with good utilization of a cluster platform for an arbitrary algorithm, is quite difficult. The casual amateur is hereby warned.</p> <p>There is no single solution. For automatic parallelization, since explicit, programmatic parallelization of things with MPI for example is very unscalable in terms of required skill, we should favor declarative and/or functional approaches.</p> <p>Developing a debugger and explanation engine for rule-based and description-logics-based inference would be an idea.</p> <p>For procedural workloads, things like Erlang may be good in cases and are not overly difficult in principle, especially if there are good debugging facilities.</p> <p>For shipping functions in a cluster or cloud, the <a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id0x5494b0">BOOM</a> (<a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id0x7f1f148">Berkeley Orders Of Magnitude</a>) approach or logic programming with explicit specification of compute location seem promising, surely more flexible than map-reduce. The question is whether a <a href="http://dbpedia.org/resource/PHP" id="link-id0x5c758c8">PHP</a> developer can be made to do logic programming.</p> <p>This bridge will be crossed only with actual need and even then reluctantly. We may look at the Web 2.0 practice of sharding <a href="http://dbpedia.org/resource/MySQL" id="link-id0x432f868">MySQL</a>, inconvenient as this may be, for an example. There is inertia and thus re-architecting is a constant process that is generally in reaction to facts, <i>post hoc</i>, often a point solution. One could argue that planning ahead would be smarter but by and large the world does not work so.</p> <p>One part of the answer is an infinitely-scalable SQL database that expands and shrinks in the clouds, with the usual semantics, maybe optional eventual consistency and built-in map reduce. If such a thing is inexpensive enough and syntax-level-compatible with present installed base, many developers do not have to learn very much more.</p> <p>This is maybe good for the bread-and-butter IT, but European competitiveness should not rest on this. Therefore we wish to go for bold new application types for which the client-server database application is not the model. Data-centric languages like BOOM, if they can be made very efficient and have good debugging support, are attractive there. These do require more intellectual investment but that is not a problem since the less-inquisitive part of the developer community is served by the first part of the answer.</p> </li> <li> <p> <b>How is a developer of average skills going to learn about these new advanced tools? How can we plan for excellent documentation and training, community mentoring, exchange of good practices, etc... across all EU countries?</b> </p> <p>For the most part, developers do not learn things for the sake of learning. When they have learned something and it is adequate, they stay with it for the most part and are even reluctant to engage in cross-camps interaction. The research world is often similarly insular. A new inflection in the application landscape is needed to drive learning. This inflection is provided by the <a href="https://wiki.mozilla.org/Labs/Ubiquity" id="link-id0x7f051c8">ubiquity</a> of mobile devices, sensor data, explicit semantics, NLP concept extraction, web of linked data, and such factors.</p> <p>RDFa is a good example of a new technique piggybacking on something everybody uses, namely HTML. These new things should, within possibility, be deployed in the usual technology stack, <a href="http://en.wikipedia.org/wiki/LAMP_%28software_bundle%29" id="link-id0x77151e0">LAMP</a> or Java. Of course these do not have to be LAMP or Java or HTML or HTTP themselves but they must manifest through these.</p> <p>A lot of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x7940cd0">semantic web</a> potential can be realized within the client-server database application model, thus no fundamental re-architecting, just some new data types and queries.</p> <p>For data- or processing-intensive tasks, an on-demand hookup to cloud-based servers with Erlang and/or BOOM for programming model would be easy enough to learn and utilize.</p> <p>The question is one of providing challenges. Addressing actual challenges with these techniques will lead to maturity, documentation, examples, and training. With virtual, Europe-wide distributed teams a reality in many places, Europe-wide dissemination is no longer insurmountable.</p> <p>As the data overflow proceeds, its victims will multiply and create demand for solutions. The EC could here encourage research project use cases gaining an extended life past the end of research projects, possibly being maintained and multiplied and spun off.</p> <p>If such things could be mutated into self-sustaining service businesses with pay-per-use revenue, say through a cloud SaaS business model, still primarily leveraging an open source technology stack, we could have self-propagating and self-supporting models for exploiting advanced IT. This would create interest, and interest would drive training and dissemination.</p> <p>The problem is creating the pull.</p> </li> </ol> </li> <li> <p> <b>Challenges</b> </p> <ol type="a" start="1"> <li> <p> <b>What should be, in this domain, the equivalent of the Netflix challenge, Ansari X Prize, <a href="http://dbpedia.org/resource/Google" id="link-id0x7e72f40">Google</a> Lunar X Prize, etc. ... ?</b> </p> <p>The EC itself no doubt suffers from data overflow in one function or another. Unless security/secrecy prohibits, simply publishing a large data set and a description of what operations should be done on it would be a start. The more real the data, the better â reality is consistently more complex and surprising than imagination. Since many interesting problems touch on fraud detection and law enforcement, there may be some security obstacles for using these application domains as subject matters of open challenges.</p> <p>Once there is a good benchmark, as discussed above, there can be some prize money allocated for the winners, specially if the race is tight.</p> <p>The Semantic Web Challenge and the Billion Triples Challenge exist and are useful as such, but do not seem to have any huge impact.</p> <p>The incentives should be sufficient and part of the expenses arising from running for such challenges could be funded. Otherwise investing in existing business development will be more interesting to industry. Some industry participation seems necessary; we would wish academia and industry to work closer. Also, having industry supply the baseline guarantees that academia actually does further the state of the art. This is not always certain.</p> <p>If challenges are based on actual problems, whether of the EC, its member governments, or private entities, and winning the challenge may lead to a contract for supplying an actual solution, these will naturally become more interesting for consortia involving integrators, specialist software vendors, and academia. Such a model would build actual capacity to deploy leading edge technologies in production, which is sorely needed.</p> </li> <li> <p> <b>What should one do to set up such a challenge, administer, and monitor it?</b> </p> <p>The EC should probably circulate a call for actual problem scenarios involving big data. If the matter of the overflow is as dire as represented, cases should be easy to find. A few should be selected and then anonymized if needed.</p> <p>The party with the use case would benefit by having hopefully the best work on it. The contestants would benefit from having real world needs guide R&D. The EC would not have to do very much, except possibly use some money for funding the best proposals. The winner would possibly get a large account and related sales and service income. The contestants would have to be teams possibly involving many organizations; for example, development and first-line services and support could come from different companies along a systems integrator model such as is widely used in the US.</p> <p>There may be a good benchmark at the time, possibly resulting from FP7 itself. In such a case, the EC could offer a prize for winners. Details would have to be worked out case by case. Such a challenge could be repeated a few times, as benchmark-driven progress in databases or TREC for example have taken some years to reach a point of slowdown in progress.</p> <p>Administrating such an activity should not be prohibitive, as most of the expertise can be found with the stakeholders.</p> </li> </ol> </li> </ol>
European Commission and the Data Overflow
http://www.openlinksw.com/weblog/oerling/?date=2009-10-27#1585
2009-10-27T18:29:51Z
2009-10-27T14:57:28.000002-04:00
<p>The European Commission recently circulated a questionnaire to selected experts on what could be done for the future of big <a href="http://dbpedia.org/resource/Data" id="link-id0x79cfe58">data</a>.</p> <p>Since the <a href="http://cordis.europa.eu/fp7/ict/content-knowledge/consultation_en.html" id="link-id1191c0f8">questionnaire is public</a>, I am publishing my answers below.</p> <ol type="1" start="1"> <li> <p> <b>Data and data types</b> </p> <ol type="a" start="1"> <li> <p> <b>What volumes of data are we dealing with today? What is the growth rate? Where can we expect to be in 2015? </b> </p> <p>Private data warehouses of corporations have more than doubled yearly for the past years; hundreds of TB is not exceptional. This will continue. The real shift is in structured data being published in increasing quantities with a minimum level of integrate-ability through use of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x7d7e7a0">RDF</a> and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x7f2a788">linked data</a> principles. There are rewards for use of standard vocabularies and identifiers through search engines recognizing such data. There is convergence around <a href="http://dbpedia.org/resource/DBpedia" id="link-id0x7dfbca8">DBpedia</a> identifiers for real-world entities, e.g., most things that would be in the news.</p> <p>This also means that internal data processes and silos may be enriched with this content. There is consequent pressure for accommodating more diversity of data, with more flexible <a href="http://dbpedia.org/resource/Database_schema" id="link-id0x7babaf8">schema</a>.</p> <p>Ultimately, all content presently stored in RDBs and presented in public accessible dynamic web pages will end up on the web of linked data. Examples are product catalogs, price lists, event schedules and the like.</p> <p>The volume of the well known linked data sets is around 10 billion statements. With the above mentioned trends, growth by two or three orders of magnitude by 2015 seems reasonable, This is so especially if explicit semantics are extracted from the document web and if there is some further progress in the precision/recall of such extraction.</p> <p>Relevant sections of this mass of data are a potential addition to any present or future analytics application.</p> <p>Since arbitrary analytics over the database which is the web cannot be economically provided by a centralized search engine, a cloud model may be used for on-demand selection of relevant data and mixing it with private data. This will drive database innovation for the next years even more than the continued classical warehouse growth.</p> <p>Science data is another driver of the data overflow. For example, faster gene sequencing, more accurate measurements in high energy physics, better imaging, and remote sensing will produce large volumes of data. This data has highly regular structure but labeling this data with source and lineage calls for a flexible, schema-last, self-describing model, such as RDF and linked data. Data and <a href="http://dbpedia.org/resource/Metadata" id="link-id0x96ce60">metadata</a> should travel together but may have different data models.</p> <p>By and large, the metadata of science data will be another stream to the web of linked data, at least to the degree it is publicly accessible. Restricted circles can and likely will implement similar ideas.</p> </li> <li> <p> <b>What types of data can we deal with intelligently due to their inherent structure (geospatial, temporal, social or <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x7e8e248">knowledge</a> graphs, 3D, sensor streams...)?</b> </p> <p>All the above types should be supported inside one DBMS so as to allow efficient querying combining conditions on all these types of data, e.g., <i>photos of sunsets taken last summer in Ibiza, with over 20 megapixels, by people I know.</i> </p> <p>Note that the test for being a sunset is an operation on the image blob that should be taken to the data; the images cannot be economically transferred.</p> <p>Interleaving of all database functions and types becomes increasingly important.</p> </li> </ol> </li> <li> <p> <b>Industries, communities</b> </p> <ol type="a" start="1"> <li> <p> <b>Who is producing these data and why? Could they do it better? How?</b> </p> <p>Right now, projects such as <a href="http://www.bio2rdf.org/" id="link-id0x43bd098">Bio2RDF</a>, <a href="http://neurocommons.org/page/Main_Page" id="link-id0x5c074b0">Neurocommons</a>, and DBPedia produce this data. The processes are in place and are reasonable. Incremental improvement is to be expected. These processes, along with the <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id0x72131d0">linked data meme</a> generally taking off, drive demand for better <a href="http://dbpedia.org/resource/Natural_language_processing" id="link-id0x71e7798">NLP</a> (<a href="http://dbpedia.org/resource/Natural_language_processing" id="link-id0x7e0e2f0">Natural Language Processing</a>), e.g., <a href="http://dbpedia.org/resource/Entity" id="link-id0x71ab500">entity</a> and relationship extraction, especially extraction that can produce instance data in given ontologies (e.g., events) using common identifiers (e.g., DBPedia URIs).</p> <p>Mapping of RDBs to RDF is possible, and a W3C working group is developing standards for this. The required baseline level has been reached; the rest is a matter of automating deployment. Within the enterprise, there are advantages to be gained for <a href="http://dbpedia.org/resource/Information" id="link-id0x7a8e9a8">information</a> integration; e.g., all entities in the CRM space can be integrated with all email and support tickets through giving everything a <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x599f630">URI</a>. Some of this information may even be published on an <a href="http://dbpedia.org/resource/Extranet" id="link-id0x2a28f98">extranet</a> for self-service and web-service interfaces. This has been done at small scales and the rest is a matter of spreading adoption and lowering the entry barrier. Incremental progress will take place, eventually resulting in qualitatively better integration along the value chain when adoption is sufficiently widespread.</p> </li> <li> <p> <b>Who is consuming these data and why? Could they do it better? How?</b> </p> <p>Consumers are various. The greatest need is for tools that summarize complex data and allow getting a bird's eye view of what data is in the first instance available. Consuming the data is hindered by the user not even necessarily knowing what data there is. This is somewhat new, as traditionally the business analyst did know the schema of the warehouse and was proficient with <a href="http://dbpedia.org/resource/SQL" id="link-id0x5999558">SQL</a> report generators and statistics packages.</p> <p>Where Web 2.0 made the <i>citizen journalist</i>, the web of linked data will make the <i>citizen analyst</i>. For this to happen, with benefits for individuals, enterprises, and governments alike, more work in user interfaces, knowledge discovery, and query composition will be useful. We may envision a "meshup economy" where data is plentiful, but the unit of value and exchange is the smart report that crystallizes actionable value from this ocean.</p> </li> <li> <p> <b>What industrial sectors in Europe could become more competitive if they became much better at managing data?</b> </p> <p>Any sector could benefit. Early adopters are seen in the biomedical field and to an extent in media. </p> </li> <li> <p> <b>Is the regulation landscape imposing constraints (privacy, compliance ...) that don't have today good tool support?</b> </p> <p>The regulation landscape drives database demand through data retention requirements and the like.</p> <p>With data integration, especially with privacy-sensitive data (as in medicine), there are issues of whether one dares put otherwise-shareable information online. Regulation is needed to protect individuals, but integration should still be possible for science.</p> <p>For this, we see a need for progress in applying policy-based approaches (e.g., row level security) to relatively schema-last data such as RDF. This is possible but needs some more work. Also, creating on-the-fly-anonymizing views on data might help.</p> <p>More research is needed for reconciling the need for security with the advantages of broad-based <i>ad hoc</i> integration. Ideally, data should be intelligent, aware of its origins and classification and cautious of whom it interacts with, all of this supported under the covers so that the user could ask anything but the data might refuse to answer or might restrict answers according to the user's profile. This is a tall order and implementing something of the sort is an open question.</p> </li> <li> <p> <b>What are the main practical problem identified for individuals and organizations? Please give examples and tell us about the main obstacles and barriers.</b> </p> <p>We have come across the following:</p> <ul> <li>Knowing that the data exists in the first place.</li> <li>If the data is found, figuring out the provenance, units and precision of measurement, identifiers, and the like.</li> <li>Compatible subject matter but incompatible representation: For example, one has numbers on a map with different maps for different points in time; another has time series of instrument data with geo-location for the instrument. It is only to be expected that the time interval between measurements is not the same. So there is need for a lot of one-off programming to align data.</li> </ul> <p>Other problems have to do with sheer volume, i.e., transfer of data even in a local area network is too slow, let alone over a wide area network. Computation needs to go to the data, and databases need to support this.</p> </li> </ol> </li> <li> <p> <b>Services, software stacks, protocols, standards, benchmarks</b> </p> <ol type="a" start="1"> <li> <p> <b>What combinations of components are needed to deal with these problems?</b> </p> <p>Recent times have seen a proliferation of special purpose databases. Since the data needs of the future are about combining data with maximum agility and minimum performance hit, there is need to gather the currently-separate functionality into an integrated system with sufficient flexibility. We see some of this in integration of map-reduce and scale-out databases. The former antagonists have become partners. Vertica, <a href="http://dbpedia.org/resource/Greenplum" id="link-id0x45ecfa0">Greenplum</a>, and OpenLink <a href="http://virtuoso.openlinksw.com" id="link-id0x7f73fc8">Virtuoso</a> are example of DBMS featuring work in this direction.</p> <p>Interoperability and at least <i>de facto</i> standards in ways of doing this will emerge.</p> </li> <li> <p> <b>What data exchange and processing mechanisms will be needed to work across platforms and programming languages?</b> </p> <p> <a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x776a1a0">HTTP</a>, <a href="http://dbpedia.org/resource/XML" id="link-id0x2a4e8d0">XML</a>, and RDF are in fact very verbose, yet these are the formats and models that have uptake. Thus, these will continue to be used even though one might think binary formats to be more efficient.</p> <p>There are of course science data set standards that are more compressed and these will continue, hopefully adding a practice of rich metadata in RDF.</p> <p>For internals of systems, MPI and TCP/IP with proprietary optimized wire formats will continue. Inter-system communication will likely continue to be HTTP, XML, and RDF as appropriate.</p> </li> <li> <p> <b>What data environments are today so wastefully messy that they would benefit from the development of standards?</b> </p> <p>RDF and <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id0x2a35960">OWL</a> are not messy but they could use some more performance; we are working on this. <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x12362e8">SPARQL</a> is finally acquiring the capabilities of a serious query language, so things are slowly coming together.</p> <p>Community process for developing application domain specific vocabularies works quite well, even though one could argue it is <i>ad hoc</i> and not up to what a modeling purist might wish.</p> <p>Top-down imposition of standards has a mixed history, with long and expensive development and sometimes no or little uptake, consider some WS* standards for example.</p> </li> <li> <p> <b>What kind of performance is expected or required of these systems? Who will measure it reliably? How?</b> </p> <p>Relational databases have a history of substantial investment in <a href="http://dbpedia.org/resource/Program_optimization" id="link-id0x7b2d7c8">optimization</a> and some of them are very good for what they do, e.g., the newer generation of analytics databases.</p> <p>The very large schema-last, no-SQL, sometimes eventually consistent key-value stores have a somewhat shorter history but do fill a real need.</p> <p>These trends will merge: Extreme scale, schema-last, complex queries, even more complex inference, custom code for in-database machine learning and other bulk processing.</p> <p>We find RDF augmented with some binary types at this crossroads. This point of the design space will have to provide performance roughly on the level of today's best relational solution for workloads that fit the relational model. The added cost of schema-last and inference must come down. We are working on this. Research work such as carried out with <a href="http://dbpedia.org/resource/MonetDB" id="link-id0x794ee48">MonetDB</a> gives clues as to how these aims can be reached.</p> <p>The separation of query language and inference is artificial. After the concepts are mature, these functions will merge and execute close to the data; there are clear evolutionary pressures in this direction.</p> <p>Benchmarks are key. Some gain can be had even from repurposing standard relational benchmarks like <a href="http://www.tpc.org/" id="link-id0x7d45c58">TPC</a>-<a href="http://dbpedia.org/resource/TPC-H" id="link-id0x45b0198">H</a>. But the TPC-H rules do not allow official reporting of such.</p> <p>Development of benchmarks for RDF, complex queries, and inference is needed. A bold challenge to the community, it should be rooted in real-life integration needs and involve high heterogeneity. A key-value store benchmark might also be conceived. A transaction benchmark like TPC-<a href="http://dbpedia.org/resource/C%2B%2B" id="link-id0x7e32178">C</a> might be the basis, maybe augmented with massive user-generated content like reviews and blogs.</p> <p>If benchmarks exist and are not too easy nor inaccessibly difficult nor too expensive to run â think of the high end TPC-C results â then TPC-style rules and processes would be quite adequate. The threshold to publish should be lowered: Everybody runs the TPC workloads internally but few publish.</p> <p>Some EC initiative for benchmarking could make sense, similar to the TREC initiative of the US government. Industry should be consulted for the specific content; possibly the answers to the present questionnaire can provide an approximate direction.</p> <p>Benchmarks should be run by software vendors on their own systems, tuned by themselves. But there should be a process of disclosure and auditing; the TPC rules give an example. Compliance should not be too expensive or time consuming. Some community development for automating these things would be a worthwhile target for EC funding.</p> </li> </ol> </li> <li> <p> <b>Usability and training</b> </p> <ol type="a" start="1"> <li> <p> <b>How difficult will it be for a developer of average competence to deploy components whose core is based on rather deep computer science? Do we all need to understand Monads and Continuations? What can be done to make it ever easier?</b> </p> <p>In the database world, huge advances in technology have taken place behind a relatively simple and stable interface: SQL. For the linked data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x7e01618">web</a>, the same will take place behind SPARQL.</p> <p>Beyond these, for example, programming with MPI with good utilization of a cluster platform for an arbitrary algorithm, is quite difficult. The casual amateur is hereby warned.</p> <p>There is no single solution. For automatic parallelization, since explicit, programmatic parallelization of things with MPI for example is very unscalable in terms of required skill, we should favor declarative and/or functional approaches.</p> <p>Developing a debugger and explanation engine for rule-based and description-logics-based inference would be an idea.</p> <p>For procedural workloads, things like Erlang may be good in cases and are not overly difficult in principle, especially if there are good debugging facilities.</p> <p>For shipping functions in a cluster or cloud, the <a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id0x43665a8">BOOM</a> (<a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id0x7718f00">Berkeley Orders Of Magnitude</a>) approach or logic programming with explicit specification of compute location seem promising, surely more flexible than map-reduce. The question is whether a <a href="http://dbpedia.org/resource/PHP" id="link-id0x7d64f68">PHP</a> developer can be made to do logic programming.</p> <p>This bridge will be crossed only with actual need and even then reluctantly. We may look at the Web 2.0 practice of sharding <a href="http://dbpedia.org/resource/MySQL" id="link-id0xbab1ae98">MySQL</a>, inconvenient as this may be, for an example. There is inertia and thus re-architecting is a constant process that is generally in reaction to facts, <i>post hoc</i>, often a point solution. One could argue that planning ahead would be smarter but by and large the world does not work so.</p> <p>One part of the answer is an infinitely-scalable SQL database that expands and shrinks in the clouds, with the usual semantics, maybe optional eventual consistency and built-in map reduce. If such a thing is inexpensive enough and syntax-level-compatible with present installed base, many developers do not have to learn very much more.</p> <p>This is maybe good for the bread-and-butter IT, but European competitiveness should not rest on this. Therefore we wish to go for bold new application types for which the client-server database application is not the model. Data-centric languages like BOOM, if they can be made very efficient and have good debugging support, are attractive there. These do require more intellectual investment but that is not a problem since the less-inquisitive part of the developer community is served by the first part of the answer.</p> </li> <li> <p> <b>How is a developer of average skills going to learn about these new advanced tools? How can we plan for excellent documentation and training, community mentoring, exchange of good practices, etc... across all EU countries?</b> </p> <p>For the most part, developers do not learn things for the sake of learning. When they have learned something and it is adequate, they stay with it for the most part and are even reluctant to engage in cross-camps interaction. The research world is often similarly insular. A new inflection in the application landscape is needed to drive learning. This inflection is provided by the <a href="https://wiki.mozilla.org/Labs/Ubiquity" id="link-id0x770df38">ubiquity</a> of mobile devices, sensor data, explicit semantics, NLP concept extraction, web of linked data, and such factors.</p> <p>RDFa is a good example of a new technique piggybacking on something everybody uses, namely HTML. These new things should, within possibility, be deployed in the usual technology stack, <a href="http://en.wikipedia.org/wiki/LAMP_%28software_bundle%29" id="link-id0x55596a8">LAMP</a> or Java. Of course these do not have to be LAMP or Java or HTML or HTTP themselves but they must manifest through these.</p> <p>A lot of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x3d5378">semantic web</a> potential can be realized within the client-server database application model, thus no fundamental re-architecting, just some new data types and queries.</p> <p>For data- or processing-intensive tasks, an on-demand hookup to cloud-based servers with Erlang and/or BOOM for programming model would be easy enough to learn and utilize.</p> <p>The question is one of providing challenges. Addressing actual challenges with these techniques will lead to maturity, documentation, examples, and training. With virtual, Europe-wide distributed teams a reality in many places, Europe-wide dissemination is no longer insurmountable.</p> <p>As the data overflow proceeds, its victims will multiply and create demand for solutions. The EC could here encourage research project use cases gaining an extended life past the end of research projects, possibly being maintained and multiplied and spun off.</p> <p>If such things could be mutated into self-sustaining service businesses with pay-per-use revenue, say through a cloud SaaS business model, still primarily leveraging an open source technology stack, we could have self-propagating and self-supporting models for exploiting advanced IT. This would create interest, and interest would drive training and dissemination.</p> <p>The problem is creating the pull.</p> </li> </ol> </li> <li> <p> <b>Challenges</b> </p> <ol type="a" start="1"> <li> <p> <b>What should be, in this domain, the equivalent of the Netflix challenge, Ansari X Prize, <a href="http://dbpedia.org/resource/Google" id="link-id0x6a6c2b0">Google</a> Lunar X Prize, etc. ... ?</b> </p> <p>The EC itself no doubt suffers from data overflow in one function or another. Unless security/secrecy prohibits, simply publishing a large data set and a description of what operations should be done on it would be a start. The more real the data, the better â reality is consistently more complex and surprising than imagination. Since many interesting problems touch on fraud detection and law enforcement, there may be some security obstacles for using these application domains as subject matters of open challenges.</p> <p>Once there is a good benchmark, as discussed above, there can be some prize money allocated for the winners, specially if the race is tight.</p> <p>The Semantic Web Challenge and the Billion Triples Challenge exist and are useful as such, but do not seem to have any huge impact.</p> <p>The incentives should be sufficient and part of the expenses arising from running for such challenges could be funded. Otherwise investing in existing business development will be more interesting to industry. Some industry participation seems necessary; we would wish academia and industry to work closer. Also, having industry supply the baseline guarantees that academia actually does further the state of the art. This is not always certain.</p> <p>If challenges are based on actual problems, whether of the EC, its member governments, or private entities, and winning the challenge may lead to a contract for supplying an actual solution, these will naturally become more interesting for consortia involving integrators, specialist software vendors, and academia. Such a model would build actual capacity to deploy leading edge technologies in production, which is sorely needed.</p> </li> <li> <p> <b>What should one do to set up such a challenge, administer, and monitor it?</b> </p> <p>The EC should probably circulate a call for actual problem scenarios involving big data. If the matter of the overflow is as dire as represented, cases should be easy to find. A few should be selected and then anonymized if needed.</p> <p>The party with the use case would benefit by having hopefully the best work on it. The contestants would benefit from having real world needs guide R&D. The EC would not have to do very much, except possibly use some money for funding the best proposals. The winner would possibly get a large account and related sales and service income. The contestants would have to be teams possibly involving many organizations; for example, development and first-line services and support could come from different companies along a systems integrator model such as is widely used in the US.</p> <p>There may be a good benchmark at the time, possibly resulting from FP7 itself. In such a case, the EC could offer a prize for winners. Details would have to be worked out case by case. Such a challenge could be repeated a few times, as benchmark-driven progress in databases or TREC for example have taken some years to reach a point of slowdown in progress.</p> <p>Administrating such an activity should not be prohibitive, as most of the expertise can be found with the stakeholders.</p> </li> </ol> </li> </ol>
VLDB 2009 Web Scale Data Management Panel (5 of 5)
http://www.openlinksw.com/blog/vdb/blog/?date=2009-09-01#1583
2009-09-01T16:24:17Z
2009-09-02T12:05:26-04:00
<blockquote> <p> <i>"The universe of cycles is not exactly one of literal cycles, but rather one of spirals," mused <a href="http://db.cs.berkeley.edu/jmh/" id="link-id117455a0">Joe Hellerstein</a> of UC Berkeley.</i> </p> <p> <i>"Come on, let's all drop some <a href="http://dbpedia.org/resource/ACID" id="link-id16b3db50">ACID</a>," interjected another.</i> </p> <p> <i>"It is not that we end up repeating the exact same things, rather even if some patterns seem to repeat, they do so at a higher level, enhanced by the experience gained," continued Joe.</i> </p> </blockquote> <p>Thus did the Web Scale <a href="http://dbpedia.org/resource/Data" id="link-id11061ae0">Data</a> Management panel conclude.</p> <p>Whether successive generations are made wiser by the ones that have gone before may be argued either way.</p> <p>The cycle in question was that of developers discovering ACID in the 1960s, i.e. Atomicity, Consistency, Integrity, Durability. Thus did the DBMS come into being. Then DBMSs kept becoming more complex until, as there will be a counter-force to each force, came the <a href="http://dbpedia.org/resource/Meme" id="link-id11076cc8">meme</a> of key value stores and BASE, no multiple-row transactions, eventual consistency, no query language but scaling to thousands of computers. So now, the DBMS community asks itself what went wrong.</p> <p>In the words of one panelist, another demonstrated a "shocking familiarity with the subject matter of substance abuse" when he called for the DBMS community to get on a <a href="http://dbpedia.org/resource/Twelve-step_program" id="link-id15d954a8">12 step program</a> and to look where addiction to certain ideas, among which ACID, had brought its life. Look at yourself: The influential papers in what ought to be your space by rights are coming from the OS community: <a href="http://dbpedia.org/resource/Google" id="link-id166675f0">Google</a> Bigtable, Amazon Dynamo, want more? When you ought to drive, you give excuses and play catch up! Stop denial, drop <a href="http://dbpedia.org/resource/SQL" id="link-id1105adf0">SQL</a>, drop ACID!</p> <p>The web developers have revolted against the time-honored principles of the DBMS. This is true. Sharded <a href="http://dbpedia.org/resource/MySQL" id="link-id1221c230">MySQL</a> is not the ticket â or is it? Must they rediscover the virtues of ACID, just like the previous generation did?</p> <p>Nothing under the sun is new. As in music and fashion, trends keep cycling also in science and engineering.</p> <p>But seriously, does the full-featured DBMS scale to web scale? <a href="http://dbpedia.org/resource/Microsoft" id="link-id10ffcaf8">Microsoft</a> says the Azure version of SQL server does. <a href="http://dbpedia.org/resource/Yahoo%21" id="link-id16b3f138">Yahoo</a> says they want no SQL but <a href="http://dbpedia.org/resource/Hadoop" id="link-id11046ef0">Hadoop</a> and <a href="http://research.yahoo.com/node/2304" id="link-id110a0040">PNUTS</a>.</p> <p>Twitter, Facebook, and other web names got their own discussion. Why do they not go to serious DBMS vendors for their data but make their own, like Facebook with Hive?</p> <p>Who can divine the mind of the web developer? What makes them go to <a href="http://www.danga.com/memcached/" id="link-id1109e280">memcached</a>, manually sharded MySQL, and <a href="http://dbpedia.org/resource/MapReduce" id="link-id1107cd60">MapReduce</a>, walking away from the 40 years of technology invested in declarative query and ACID? What is this highly visible but hard to grasp <a href="http://dbpedia.org/resource/Entity" id="link-id1105b6b8">entity</a>? My guess is that they want something they can understand, at least at the beginning. A DBMS, especially on a cluster, is complicated, and it is not so easy to say how it works and how its performance is determined. The big brands, if deployed on a thousand PCs, would also be prohibitively expensive. But if all you do with the DBMS is single row selects and updates, it is no longer so scary, but you end up doing all the distributed things in a middle layer, and abandoning expressive queries, transactions, and database-supported transparency of location. But at least now you know how it works and what it is good/not good for.</p> <p>This would be the case for those who make a conscious choice. But by and large the choice is not deliberate; it is something one drifts into: The application gains popularity; the single <a href="http://en.wikipedia.org/wiki/LAMP_%28software_bundle%29" id="link-iddc68d28">LAMP</a> can no longer keep all in memory; you need a second MySQL in the LAMP and you decide that users AâM go left and NâZ right (horizontal partitioning). This siren of sharding beckons you and all is good until you hit the reef of re-architecting. Memcached and duct-tape help, like aspirin helps with hangover, but the root cause of the headache lies unaddressed.</p> <p>The conclusion was that there ought to be something incrementally scalable from the get-go. Low cost of entry and built-in scale-out. No, the web developers do not hate SQL; they just have gotten the idea that it does not scale. But they would really wish it to. So, DBMS people, show there is life in you yet.</p> <p>Joe Hellerstein was the philosopher and paradigmatician of the panel. His team had developed a protocol-compatible Hadoop in a few months using a declarative logic programming style approach. His claim was that developers made the market. Thus, for writing applications against web scale data, there would have to be data centric languages. Why not? These are discussed in <a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id110ba0e0">Berkeley Orders Of Magnitude</a> (<a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id16aab768">BOOM</a>).</p> <p>I come from <a href="http://en.wikipedia.org/wiki/Lisp_%28programming_language%29" id="link-id10f2cd68">Lisp</a> myself, way back. I have since abandoned any desire to tell anybody what they ought to program in. This is a bit like religion: Attempting to impose or legislate or ram it on somebody just results in anything from lip service to rejection to war. The appeal exerted by the diverse language/paradigm -isms on their followers seems to be based on hitting a simplification of reality that coincides with a problem in the air. MapReduce is an example of this. <a href="http://dbpedia.org/resource/PHP" id="link-ide22cdd0">PHP</a> is another. A quick fix for a present need: Scripting web servers (PHP) or processing tons of files (MapReduce). The full database is not as quick a fix, even though it has many desirable features. It is also not as easy to tell what happens inside one, so MapReduce may give a greater feeling of control.</p> <p>Totally self-managing, dynamically-scalable <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id152864b0">RDF</a> would be a fix for not having to design or administer databases: Since it would be indexed on everything, complex queries would be possible; no full database scans would stop everything. For the mid-size segment of web sites this might be a fit. For the extreme ends of the spectrum, the choice is likely something custom built and much less expressive.</p> <p>The BOOM rule language for data-centric programming would be something very easy for us to implement, in fact we will get something of the sort essentially for free when we do the rule support already planned.</p> <p>The question is, can one induce web developers to do logic? The history is one of procedures, both in LAMP and MapReduce. On the other hand, the query languages that were ever universally adopted were declarative, i.e., keyword search and SQL. There certainly is a quest for an application model for the cloud space beyond just migrating apps. We'll see. More on this another time.</p>
VLDB 2009 Web Scale Data Management Panel (5 of 5)
http://www.openlinksw.com/weblog/oerling/?date=2009-09-01#1582
2009-09-01T16:24:17Z
2009-09-02T12:05:20.000001-04:00
<blockquote> <p> <i>"The universe of cycles is not exactly one of literal cycles, but rather one of spirals," mused <a href="http://db.cs.berkeley.edu/jmh/" id="link-id117455a0">Joe Hellerstein</a> of UC Berkeley.</i> </p> <p> <i>"Come on, let's all drop some <a href="http://dbpedia.org/resource/ACID" id="link-id16b3db50">ACID</a>," interjected another.</i> </p> <p> <i>"It is not that we end up repeating the exact same things, rather even if some patterns seem to repeat, they do so at a higher level, enhanced by the experience gained," continued Joe.</i> </p> </blockquote> <p>Thus did the Web Scale <a href="http://dbpedia.org/resource/Data" id="link-id11061ae0">Data</a> Management panel conclude.</p> <p>Whether successive generations are made wiser by the ones that have gone before may be argued either way.</p> <p>The cycle in question was that of developers discovering ACID in the 1960s, i.e. Atomicity, Consistency, Integrity, Durability. Thus did the DBMS come into being. Then DBMSs kept becoming more complex until, as there will be a counter-force to each force, came the <a href="http://dbpedia.org/resource/Meme" id="link-id11076cc8">meme</a> of key value stores and BASE, no multiple-row transactions, eventual consistency, no query language but scaling to thousands of computers. So now, the DBMS community asks itself what went wrong.</p> <p>In the words of one panelist, another demonstrated a "shocking familiarity with the subject matter of substance abuse" when he called for the DBMS community to get on a <a href="http://dbpedia.org/resource/Twelve-step_program" id="link-id15d954a8">12 step program</a> and to look where addiction to certain ideas, among which ACID, had brought its life. Look at yourself: The influential papers in what ought to be your space by rights are coming from the OS community: <a href="http://dbpedia.org/resource/Google" id="link-id166675f0">Google</a> Bigtable, Amazon Dynamo, want more? When you ought to drive, you give excuses and play catch up! Stop denial, drop <a href="http://dbpedia.org/resource/SQL" id="link-id1105adf0">SQL</a>, drop ACID!</p> <p>The web developers have revolted against the time-honored principles of the DBMS. This is true. Sharded <a href="http://dbpedia.org/resource/MySQL" id="link-id1221c230">MySQL</a> is not the ticket â or is it? Must they rediscover the virtues of ACID, just like the previous generation did?</p> <p>Nothing under the sun is new. As in music and fashion, trends keep cycling also in science and engineering.</p> <p>But seriously, does the full-featured DBMS scale to web scale? <a href="http://dbpedia.org/resource/Microsoft" id="link-id10ffcaf8">Microsoft</a> says the Azure version of SQL server does. <a href="http://dbpedia.org/resource/Yahoo%21" id="link-id16b3f138">Yahoo</a> says they want no SQL but <a href="http://dbpedia.org/resource/Hadoop" id="link-id11046ef0">Hadoop</a> and <a href="http://research.yahoo.com/node/2304" id="link-id110a0040">PNUTS</a>.</p> <p>Twitter, Facebook, and other web names got their own discussion. Why do they not go to serious DBMS vendors for their data but make their own, like Facebook with Hive?</p> <p>Who can divine the mind of the web developer? What makes them go to <a href="http://www.danga.com/memcached/" id="link-id1109e280">memcached</a>, manually sharded MySQL, and <a href="http://dbpedia.org/resource/MapReduce" id="link-id1107cd60">MapReduce</a>, walking away from the 40 years of technology invested in declarative query and ACID? What is this highly visible but hard to grasp <a href="http://dbpedia.org/resource/Entity" id="link-id1105b6b8">entity</a>? My guess is that they want something they can understand, at least at the beginning. A DBMS, especially on a cluster, is complicated, and it is not so easy to say how it works and how its performance is determined. The big brands, if deployed on a thousand PCs, would also be prohibitively expensive. But if all you do with the DBMS is single row selects and updates, it is no longer so scary, but you end up doing all the distributed things in a middle layer, and abandoning expressive queries, transactions, and database-supported transparency of location. But at least now you know how it works and what it is good/not good for.</p> <p>This would be the case for those who make a conscious choice. But by and large the choice is not deliberate; it is something one drifts into: The application gains popularity; the single <a href="http://en.wikipedia.org/wiki/LAMP_%28software_bundle%29" id="link-iddc68d28">LAMP</a> can no longer keep all in memory; you need a second MySQL in the LAMP and you decide that users AâM go left and NâZ right (horizontal partitioning). This siren of sharding beckons you and all is good until you hit the reef of re-architecting. Memcached and duct-tape help, like aspirin helps with hangover, but the root cause of the headache lies unaddressed.</p> <p>The conclusion was that there ought to be something incrementally scalable from the get-go. Low cost of entry and built-in scale-out. No, the web developers do not hate SQL; they just have gotten the idea that it does not scale. But they would really wish it to. So, DBMS people, show there is life in you yet.</p> <p>Joe Hellerstein was the philosopher and paradigmatician of the panel. His team had developed a protocol-compatible Hadoop in a few months using a declarative logic programming style approach. His claim was that developers made the market. Thus, for writing applications against web scale data, there would have to be data centric languages. Why not? These are discussed in <a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id110ba0e0">Berkeley Orders Of Magnitude</a> (<a href="http://www.eecs.berkeley.edu/Research/Projects/Data/105733.html" id="link-id16aab768">BOOM</a>).</p> <p>I come from <a href="http://en.wikipedia.org/wiki/Lisp_%28programming_language%29" id="link-id10f2cd68">Lisp</a> myself, way back. I have since abandoned any desire to tell anybody what they ought to program in. This is a bit like religion: Attempting to impose or legislate or ram it on somebody just results in anything from lip service to rejection to war. The appeal exerted by the diverse language/paradigm -isms on their followers seems to be based on hitting a simplification of reality that coincides with a problem in the air. MapReduce is an example of this. <a href="http://dbpedia.org/resource/PHP" id="link-ide22cdd0">PHP</a> is another. A quick fix for a present need: Scripting web servers (PHP) or processing tons of files (MapReduce). The full database is not as quick a fix, even though it has many desirable features. It is also not as easy to tell what happens inside one, so MapReduce may give a greater feeling of control.</p> <p>Totally self-managing, dynamically-scalable <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id152864b0">RDF</a> would be a fix for not having to design or administer databases: Since it would be indexed on everything, complex queries would be possible; no full database scans would stop everything. For the mid-size segment of web sites this might be a fit. For the extreme ends of the spectrum, the choice is likely something custom built and much less expressive.</p> <p>The BOOM rule language for data-centric programming would be something very easy for us to implement, in fact we will get something of the sort essentially for free when we do the rule support already planned.</p> <p>The question is, can one induce web developers to do logic? The history is one of procedures, both in LAMP and MapReduce. On the other hand, the query languages that were ever universally adopted were declarative, i.e., keyword search and SQL. There certainly is a quest for an application model for the cloud space beyond just migrating apps. We'll see. More on this another time.</p>
The URI, URL, and Linked Data Meme's Generic HTTP URI (Updated)
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2009-08-07#1567
2009-08-07T18:34:50Z
2009-10-07T08:02:34-04:00
<h3>Situation Analysis</h3> <p>As the "<a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id12f96a00">Linked Data" meme</a> has gained momentum you've more than likely been on the receiving end of dialog with Linked Open <a href="http://dbpedia.org/resource/Data">Data</a> community members (myself included) that goes something like this:</p> <blockquote> <cite>"Do you have a <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id139252a0">URI</a>", "Get yourself a URI", "Give <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id140eab68">me</a> a de-referencable URI" etc..</cite> </blockquote> <p>And each time, you respond with a <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id112c1860">URL</a> -- which to the best of your <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> <a href="http://dbpedia.org/resource/Knowledge" id="link-id140b51c0">knowledge</a> is a bona fide URI. But to your utter confusion you are told: Nah! You gave me a Document URI instead of the URI of a real-world thing or object etc..</p> <h3>What's up with that?</h3> <p>Well our everyday use of the Web is an unfortunate conflation of two distinct things, which have Identity: Real World Objects (RWOs) & Address/Location of Documents (<a href="http://dbpedia.org/resource/Information" id="link-id144838b0">Information</a> bearing Resources).</p> <p>The "Linked Data" meme is about enhancing the Web by unobtrusively reintroducing its core essence: the generic HTTP URI, a vital piece of Web Architecture DNA. Basically, its about so realizing the full capabilities of the Web as a platform for Open Data Identification, Definition, Access, Storage, Representation, Presentation, and Integration.</p> <h3>What is a Real World Object?</h3> <p>People, Places, Music, Books, Cars, Ideas, Emotions etc..</p> <h3>What is a URI?</h3> <p>A Uniform Resource Identifier. A global identifier mechanism for network addressable data items. Its sole function is Name oriented Identification.</p> <h4>URI Generic Syntax</h4> <p>The constituent parts of a URI (from <a href="http://www.ietf.org/rfc/rfc2396.txt" id="link-id1180c700">URI Generic Syntax RFC</a>) are depicted below: <img src="http://virtuoso.openlinksw.com/images/generic_uri_syntax_image.png" /> </p> <h3>What is a URL?</h3> <p>A location oriented HTTP scheme based URI. The HTTP scheme introduces a powerful and inherent duality that delivers:</p> <ol> <li> Resource Address/Location Identifier</li> <li> Data Access mechanism for an Information bearing Resource (Document, File etc..) </li> </ol> <p>So far so good!</p> <h3>What is an HTTP based URI?</h3> <p>The kind of URI <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11100a28">Linked Data</a> aficionados mean when they use the term: URI.</p> <p>An HTTP URI is an HTTP scheme based URI. Unlike a URL, this kind of HTTP scheme URI is devoid of any Web Location orientation or specificity. Thus, Its inherent duality provides a more powerful level of abstraction. Hence, you can use this form of URI to assign Names/Identifiers to Real World Objects (RWO). Even better, courtesy of the Identity/Address duality of the HTTP scheme, a single URI can deliver the following:</p> <ol> <li> RWO Identfier/Name</li> <li> RWO Metadata document Locator (courtesy of URL aspect) </li> <li> Negotiable Representation of the Located Document (courtesy of HTTP's content negotiation feature).</li> </ol> <h3>What is Metadata?</h3> <p> Data about Data. Put differently, data that describes other data in a structured manner.</p> <h3>How Do we Model Metadata?</h3> <p>The predominant model for metadata is the <a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id11193d30">Entity</a>-Attribute-Value + Classes & Relationships model (<a href="http://dbpedia.org/resource/Entity-attribute-value_model" id="link-id11725710">EAV</a>/CR). A model that's been with us since the inception of modern computing (long before the Web). </p> <h3>What about RDF?</h3> <p>The Resource Description Framework (RDF) is a framework for describing Web addressable resources. In a nutshell, its a framework for adding Metadata bearing Information Resources to the current Web. Its comprised of:</p> <ol> <li> Entity-Attribute-Value (aka. Subject-Predictate-Object) plus Classes & Relationships (<a href="http://dbpedia.org/resource/Data_dictionary" id="link-id138df0f8">Data Dictionaries</a> e.g., <a href="http://dbpedia.org/resource/Web_Ontology_Language" id="link-id116bf590">OWL</a>) metadata model</li> <li> A plethora of instance data representation formats that include: <a href="http://dbpedia.org/resource/RDFa" id="link-id13360b90">RDFa</a> (when doing so within (X)HTML docs), Turtle, N3, TriX, RDF/XML etc. </li> </ol> <h3>What's the Problem Today?</h3> <p>The ubiquitous use of the Web is primarily focused on a Linked Mesh of Information bearing Documents. URLs rather than generic HTTP URIs are the prime mechanism for Web tapestry; basically, we use URLs to conduct Information -- which is inherently subjective -- instead of using HTTP URIs to conduct "Raw Data" -- which is inherently objective. </p> <blockquote> <strong>Note:</strong> Information is "data in <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id1395ca50">context</a>", it isn't the same thing as "Raw Data". Thus, if we can link to Information via the Web, why shouldn't we be able to do the same for "Raw Data"?</blockquote> <h3>How Does the Link Data <a href="http://dbpedia.org/resource/Meme" id="link-id1160ab70">meme</a> solve the problem?</h3> <p>The meme simply provides a set of guidelines (best practices) for producing Web architecture friendly metadata. Meaning: when producing EAV/CR model based metadata, endow Subjects, their Attributes, and Attribute Values (optionally) with HTTP URIs. By doing so, a new level of Link Abstraction on the Web is possible i.e., "Data Item to Data Item" level links (aka <a href="http://dbpedia.org/resource/Linked_Data" id="link-id137a78a0">hyperdata</a> links). Even better, when you de-reference a RWO hyperdata link you end up with a negotiated representations of its metadata.</p> <h3>Conclusion</h3> <p>Linked Data is ultimately about an HTTP URI for each item in the <a href="http://dbpedia.org/resource/Data_hierarchy" id="link-id1393c3e0">Data Organization Hierarchy</a> :-)</p> <h3>Related</h3> <ol> <li> <a href="http://www.nabble.com/Review-of-new-HTTPbis-text-for-303-See-Other-to24035004.html#a24774368" id="link-id10fcaba8">History of how "Resource" became part of URI</a> - historic account by <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id1172b128">TimBL</a> </li> <li> <a href="http://www.w3.org/DesignIssues/LinkedData.html" id="link-id1338cbd0">Linked Data Design Issues Document</a> - <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id13536ad8">TimBL</a>'s initial Linked Data Guide</li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1561" id="link-id116c1af8">Linked Data Rules Simplified</a> - My attempt at simplifying the Linked Data Meme without <a href="http://dbpedia.org/resource/SPARQL" id="link-id116c3b40">SPARQL</a> & RDF distraction</li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1547" id="link-id135dd1b8">Linked Data & Identity</a> - another related post</li> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1565" id="link-id134afc50">The Linked Data Meme's Value Proposition</a> </li> <li> <a href="http://delicious.com/kidehen/identifier_scheme" id="link-id14cc7e18">My Del.icio.us hosted Bookmark Data Space for Identity Schemes</a> </li> <li> <a href="http://www.ted.com/talks/tim_berners_lee_on_the_next_web.html" id="link-id115a3748">TimBL's Ted Talk re. "Raw Linked Data"</a>.</li> </ol>
Social Web Camp (#5 of 5)
http://www.openlinksw.com/blog/vdb/blog/?date=2009-04-30#1555
2009-04-30T16:14:02Z
2009-04-30T12:51:54-04:00
<p>(Last of five posts related to the <a href="http://www2009.org/" id="link-id0x112efd58">WWW 2009</a> conference, held the week of April 20, 2009.) </p> <p>The social networks camp was interesting, with a special meeting around Twitter. Half jokingly, we (that is, the OpenLink folks attending) concluded that societies would never be completely classless, although mobility between, as well as criteria for membership in, given classes would vary with time and circumstance. Now, there would be a new class division between people for whom micro-blogging is obligatory and those for whom it is an option.</p> <p>By my experience, a great deal is possible in a short time, but this possibility depends on focus and concentration. These are increasingly rare. I am a great believer in core competence and focus. This is not only for geeks â one can have a lot of breadth-of-scope but this too depends on not getting sidetracked by constant <a href="http://dbpedia.org/resource/Information" id="link-id0x14e380b8">information</a> overload.</p> <p>Insofar as personal success depends on constant reaction to online social media, this comes at a cost in time and focus and this cost will have to be managed somehow, for example by automation or outsourcing. But if the social media is only automated fronts twitting and re-twitting among themselves, a bit like electronic trading systems do with securities, with or without human operators, the value of the medium decreases.</p> <p>There are contradictory requirements. On one hand, what is said in electronic media is essentially permanent, so one had best only say things that are well considered. On the other hand, one must say these things without adequate time for reflection or analysis. To cope with this, one must have a well-rehearsed position that is compacted so that it fits in a short format and is easy to remember and unambiguous to express. A culture of pre-cooked fast-food advertising cuts down on depth. Real-world things are complex and multifaceted. Besides, prevalent patterns of communication train the brain for a certain mode of functioning. If we train for rapid-fire 140-character messaging, we optimize one side but probably at the expense of another. In the meantime, the world continues developing increased complexity by all kinds of emergent effects. Connectivity is good but don't get lost in it.</p> <p>There is <a href="https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis/index.html" id="link-id170cb010">a CIA memorandum about how analysts misinterpret data and see what they want to see</a>. This is a relevant resource for understanding some psychology of perception and memory. With the information overload, largely driven by user generated content, interpreting fragmented and variously-biased real-time information is not only for the analyst but for everyone who needs to intelligently function in cyber-social space.</p> <p>I participated in discussions on security and privacy and on mobile social networks and context.</p> <p>For privacy, the main thing turned out to be whether people should be protected from themselves. Should information expire? Will it get buried by itself under huge volumes of new content? Well, for purposes of visibility, it will certainly get buried and will require constant management to stay visible. But for purposes of future finding of dirt, it will stay findable for those who are looking.</p> <p>There is also the corollary of setting security for resources, like documents, versus setting security for statements, i.e., structured data like social networks. As I have blogged before, policies <a id="link-id14aaff90">à la</a> <a href="http://dbpedia.org/resource/SQL" id="link-id0x13d77830">SQL</a> do not work well when schema is fluid and end-users can't be expected to formulate or understand these. Remember <a href="http://dbpedia.org/resource/Ted_Nelson" id="link-id0x156ceae0">Ted Nelson</a>? A user interface should be such that a beginner understands it in 10 seconds in an emergency. The user interaction question is how to present things so that the user understands who will have access to what content. Also, users should themselves be able to check what potentially sensitive information can be found out about them. A service along the lines of Garlic's Data Patrol should be a part of the social web infrastructure of the future.</p> <p>People at MIT have developed AIR (Accountability In <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x14e2abc0">RDF</a>) for expressing policies about what can be done with data and for explaining why access is denied if it is denied. However, if we at all look at the history of secrets, it is rather seldom that one hears that access to information about X is restricted to compartment so-and-so; it is much more common to hear that there is no X. I would say that a policy system that just leaves out information that is not supposed to be available will please the users more. This is not only so for organizations; it is fully plausible that an individual might not wish to expose even the existence of some selected inner circle of friends, their parties together, or whatever.</p> <p>In conclusion, there is no self-evident solution for careless use of social media. A site that requires people to confirm multiple times that they know what they are doing when publishing a photo will not get much use. We will see.</p> <p>For mobility, there was some talk about the context of usage. Again, this is difficult. For different contexts, one would for example disclose one's location at the granularity of the city; for some other purposes, one would say which conference room one is in.</p> <p>Embarrassing social situations may arise if mobile devices are too clever: If information about travel is pushed into the social network, one would feel like having to explain why one does not call on such-and-such a person and so on. Too much initiative in the mobile phone seems like a recipe for problems.</p> <p>There is a thin line between convenience and having IT infrastructure rule one's life. The complexities and subtleties of social situations ought not to be reduced to the level of if-then rules. People and their interactions are more complex than they themselves often realize. A system is not its own metasystem, as Gödel put it. Similarly, human self-<a href="http://dbpedia.org/resource/Knowledge" id="link-id0x70d82ff8">knowledge</a>, let alone knowledge about another, is by this very principle only approximate. Not to forget what psychology tells us about state-dependent recall and of how circumstance can evoke patterns of behavior before one even notices. The history of expert systems did show that people do not do very well at putting their skills in the form of if-then rules. Thus automating sociality past a certain point seems a problematic proposition.</p>
Social Web Camp (#5 of 5)
http://www.openlinksw.com/weblog/oerling/?date=2009-04-30#1554
2009-04-30T16:14:02Z
2009-04-30T12:51:49-04:00
<p>(Last of five posts related to the <a href="http://www2009.org/" id="link-id0xd28c860">WWW 2009</a> conference, held the week of April 20, 2009.) </p> <p>The social networks camp was interesting, with a special meeting around Twitter. Half jokingly, we (that is, the OpenLink folks attending) concluded that societies would never be completely classless, although mobility between, as well as criteria for membership in, given classes would vary with time and circumstance. Now, there would be a new class division between people for whom micro-blogging is obligatory and those for whom it is an option.</p> <p>By my experience, a great deal is possible in a short time, but this possibility depends on focus and concentration. These are increasingly rare. I am a great believer in core competence and focus. This is not only for geeks â one can have a lot of breadth-of-scope but this too depends on not getting sidetracked by constant <a href="http://dbpedia.org/resource/Information" id="link-id0x10019a70">information</a> overload.</p> <p>Insofar as personal success depends on constant reaction to online social media, this comes at a cost in time and focus and this cost will have to be managed somehow, for example by automation or outsourcing. But if the social media is only automated fronts twitting and re-twitting among themselves, a bit like electronic trading systems do with securities, with or without human operators, the value of the medium decreases.</p> <p>There are contradictory requirements. On one hand, what is said in electronic media is essentially permanent, so one had best only say things that are well considered. On the other hand, one must say these things without adequate time for reflection or analysis. To cope with this, one must have a well-rehearsed position that is compacted so that it fits in a short format and is easy to remember and unambiguous to express. A culture of pre-cooked fast-food advertising cuts down on depth. Real-world things are complex and multifaceted. Besides, prevalent patterns of communication train the brain for a certain mode of functioning. If we train for rapid-fire 140-character messaging, we optimize one side but probably at the expense of another. In the meantime, the world continues developing increased complexity by all kinds of emergent effects. Connectivity is good but don't get lost in it.</p> <p>There is <a href="https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis/index.html" id="link-id170cb010">a CIA memorandum about how analysts misinterpret data and see what they want to see</a>. This is a relevant resource for understanding some psychology of perception and memory. With the information overload, largely driven by user generated content, interpreting fragmented and variously-biased real-time information is not only for the analyst but for everyone who needs to intelligently function in cyber-social space.</p> <p>I participated in discussions on security and privacy and on mobile social networks and context.</p> <p>For privacy, the main thing turned out to be whether people should be protected from themselves. Should information expire? Will it get buried by itself under huge volumes of new content? Well, for purposes of visibility, it will certainly get buried and will require constant management to stay visible. But for purposes of future finding of dirt, it will stay findable for those who are looking.</p> <p>There is also the corollary of setting security for resources, like documents, versus setting security for statements, i.e., structured data like social networks. As I have blogged before, policies <a id="link-id14aaff90">à la</a> <a href="http://dbpedia.org/resource/SQL" id="link-id0x10b058d0">SQL</a> do not work well when schema is fluid and end-users can't be expected to formulate or understand these. Remember <a href="http://dbpedia.org/resource/Ted_Nelson" id="link-id0x145b3070">Ted Nelson</a>? A user interface should be such that a beginner understands it in 10 seconds in an emergency. The user interaction question is how to present things so that the user understands who will have access to what content. Also, users should themselves be able to check what potentially sensitive information can be found out about them. A service along the lines of Garlic's Data Patrol should be a part of the social web infrastructure of the future.</p> <p>People at MIT have developed AIR (Accountability In <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x10dec8f8">RDF</a>) for expressing policies about what can be done with data and for explaining why access is denied if it is denied. However, if we at all look at the history of secrets, it is rather seldom that one hears that access to information about X is restricted to compartment so-and-so; it is much more common to hear that there is no X. I would say that a policy system that just leaves out information that is not supposed to be available will please the users more. This is not only so for organizations; it is fully plausible that an individual might not wish to expose even the existence of some selected inner circle of friends, their parties together, or whatever.</p> <p>In conclusion, there is no self-evident solution for careless use of social media. A site that requires people to confirm multiple times that they know what they are doing when publishing a photo will not get much use. We will see.</p> <p>For mobility, there was some talk about the context of usage. Again, this is difficult. For different contexts, one would for example disclose one's location at the granularity of the city; for some other purposes, one would say which conference room one is in.</p> <p>Embarrassing social situations may arise if mobile devices are too clever: If information about travel is pushed into the social network, one would feel like having to explain why one does not call on such-and-such a person and so on. Too much initiative in the mobile phone seems like a recipe for problems.</p> <p>There is a thin line between convenience and having IT infrastructure rule one's life. The complexities and subtleties of social situations ought not to be reduced to the level of if-then rules. People and their interactions are more complex than they themselves often realize. A system is not its own metasystem, as Gödel put it. Similarly, human self-<a href="http://dbpedia.org/resource/Knowledge" id="link-id0xd7b1808">knowledge</a>, let alone knowledge about another, is by this very principle only approximate. Not to forget what psychology tells us about state-dependent recall and of how circumstance can evoke patterns of behavior before one even notices. The history of expert systems did show that people do not do very well at putting their skills in the form of if-then rules. Thus automating sociality past a certain point seems a problematic proposition.</p>
YODA & the Data FORCE
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2008-11-03#1474
2008-11-03T17:32:49Z
2008-11-06T09:04:34.000004-05:00
<p>The original <a href="http://www.w3.org/History/1989/proposal.html" id="link-id13b25ba8">design document</a> (by <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id181e4c70">TimBL</a>) that lead to the WWW (*an important read*) was very clear about the need to create an "<a href="http://dbpedia.org/resource/Information" id="link-id10f23918">information</a> space" that connects heterogeneous <a href="http://dbpedia.org/resource/Data">data</a> sources. Unfortunately, in trying to create a moniker to distinguish one aspect of the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> (the Linked Document Web) from the part that was overlooked (the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11096818">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id1b9c6b98">Web</a>), we ended up with a project code name that's fundamentally a misnomer in the form of: "The <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id10ffe228">Semantic Web</a>".</p> <p>If we could just take "The Semantic Web" moniker for what it was -- a code name for an aspect of the Web -- and move on, things will get much clearer, fast!</p> <p>Basically, what is/was the "Semantic Web" should really have been code named: ("You" Oriented Data Access) as a play on: Yoda's appreciation of the FORCE (Fact ORiented Connected Entities) -- the power of inter galactic, interlinked, structured data, fashioned by the <a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id191b22e0">World Wide Web</a> courtesy of the HTTP protocol.</p> <div> <img src="http://www.the-planets.com/star-biography/yoda_biography_3.jpg" /> </div> <p>As stated in a earlier post, the next phase of the Web is all about the magic of <a href="http://dbpedia.org/resource/Entity" id="link-id1a7395f0">entity</a> "You". The single most important item of reference to every Web user would be the Person Entity <a href="http://dbpedia.org/resource/Identity_%28object-oriented_programming%29" id="link-id16ab9308">ID</a> (<a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id1d403c88">URI</a>). Just by remembering your Entity ID, you will have intelligent pathways across, and into, the FORCE that the Linked Data Web delivers. The quality of the pathways and increased density of the FORCE are the keys to high <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1442" id="link-id1c549b28">SDQ</a> (tomorrows SEO). Thus, the SDQ of URIs will ultimately be the unit determinant of value to Web Users along the following personal lines:</p> <ul> <li> Does your platform give <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id175afe00">me</a> Identity (a URI) with high SDQ?</li> <li> Do the Data Source Names (URIs) in your Data Spaces deliver high SDQ? </li> </ul> <p>While most industry commentators continue to ponder and pontificate about what "The Semantic Web" is (unfortunately), the real thing (the "FORCE") is already here, and self-enhancing rapidly. </p> <p>Assuming we now accept the FORCE is simply an RDF based Linked Data moniker, and that RDF Linked Data is all about the Web as a structured database, we should start to move our attention over to practical exploitation of this burgeoning global database, and in doing so we should not discard <a href="http://dbpedia.org/resource/Knowledge" id="link-id19e2c6e0">knowledge</a> from the past such as the many great examples available gratis from the Relational Database realm. For instance, we should start paying attention to the discovery, development, and deployment of high level tools such as query builders, report writers, and intelligence oriented analytic tools, none of which should -- at first point of interaction -- expose raw RDF or the <a href="http://dbpedia.org/resource/SPARQL" id="link-id117921f0">SPARQL</a> query language. Along similar lines of thinking, we also need development environments and frameworks that are counterparts to Visual Studio, ACCESS, File Maker, and the like.</p> <h3>Related</h3> <ul> <li> <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1458" id="link-id1cec1a40">Numerati & The Magic of You!</a> </li> </ul>
Virtuoso - Are We Too Clever for Our Own Good? (updated)
http://www.openlinksw.com/blog/vdb/blog/?date=2008-10-26#1467
2008-10-26T12:15:35Z
2008-10-27T12:07:58-04:00
<p>"Physician, heal thyself," it is said. We profess to say what the messaging of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1b4a25f0">semantic web</a> ought to be, but is our own perfect?</p> <p>I will here engage in some critical introspection as well as amplify on some answers given to <a href="http://virtuoso.openlinksw.com" id="link-id0x1e4f9928">Virtuoso</a>-related questions in recent times.</p> <p>I use some conversations from the <a href="http://dbpedia.org/resource/Vienna" id="link-id0x1e6c0ca8">Vienna</a> <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1e56df88">Linked Data</a> Practitioners meeting as a starting point. These views are mine and are limited to the Virtuoso server. These do not apply to the <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id0x1e680440">ODS</a> (<a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id0x1e140068">OpenLink Data Spaces</a>) applications line, <a href="http://oat.openlinksw.com/" id="link-id0x1f4ba630">OAT</a> (<a href="http://oat.openlinksw.com/" id="link-id0x1ba4bac8">OpenLink Ajax Toolkit</a>), or <a href="http://ode.openlinksw.com/" id="link-id0x1d4159b0">ODE</a> (<a href="http://ode.openlinksw.com/" id="link-id0x1e973c80">OpenLink Data Explorer</a>).</p> <h3>"It is not always clear what the main thrust is, we get the impression that you are spread too thin," said <a href="http://www.informatik.uni-leipzig.de/~auer/foaf.rdf#me" id="link-id0x1f8bafe0">Sören Auer</a>.</h3> <p>Well, personally, I am all for core competence. This is why I do not participate in all the online conversations and groups as much as I could, for example. Time and energy are critical resources and must be invested where they make a difference. In this case, the real core competence is running in the database race. This in itself, come to think of it, is a pretty broad concept.</p> <p>This is why we put a lot of emphasis on Linked Data and the <a href="http://dbpedia.org/resource/Data" id="link-id0x200bd1f0">Data</a> Web for now, as this is the emerging game. This is a deliberate choice, not an outside imperative or built-in limitation. More specifically, this means exposing any pre-existing relational data as linked data plus being the definitive <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1fb03528">RDF</a> store.</p> <p>We can do this because we own our database and <a href="http://dbpedia.org/resource/SQL" id="link-id0x1e7dcc70">SQL</a> and data access middleware and have a history of connecting to any <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1e9baf18">RDBMS</a> out there.</p> <p>The principal message we have been hearing from the RDF field is the call for scale of triple storage. This is even louder than the call for relational mapping. We believe that in time mapping will exceed triple storage as such, once we get some real production strength mappings deployed, enough to outperform RDF warehousing.</p> <p>There are also RDF middleware things like RDF-ization and demand-driven web harvesting (i.e, the so-called Sponger). These are <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1f5f6b78">SPARQL</a> options, thus accessed via standard interfaces. We have little desire to create our own languages or APIs, or to tell people how to program. This is why we recently introduced <a href="http://sourceforge.net/projects/sesame/" id="link-id0x206818c8">Sesame</a>- and <a href="http://jena.sourceforge.net/" id="link-id0x202b3348">Jena</a>-compatible APIs to our RDF store. From what we hear, these work. On the other hand, we do not hesitate to move beyond the standards when there is obvious value or necessity. This is why we brought SPARQL up to and beyond SQL expressivity. It is not a case of E3 (Embrace, Extend, Extinguish).</p> <p>Now, this message could be better reflected in our material on the web. This <a href="http://dbpedia.org/resource/Blog" id="link-id0x1c82e508">blog</a> is a rather informal step in this direction; more is to come. For now we concentrate on delivering.</p> <p>The conventional communications wisdom is to split the message by target audience. For this, we should split the RDF, relational, and web services messages from each other. We believe that a challenger, like the semantic web technology stack, must have a compelling message to tell for it to be interesting. This is not a question of research prototypes. The new technology cannot lack something the installed technology takes for granted.</p> <p>This is why we do not tend to show things like how to insert and query a few triples: No business out there will insert and query triples for the sake of triples. There must be a more compelling story â for example, turning the whole world into a database. This is why our examples start with things like turning the <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x20832510">TPC-H</a> database into RDF, queries and all. Anything less is not interesting. Why would an enterprise that has business intelligence and integration issues way more complex than the rather stereotypical TPC-H even look at a technology that pretends to be all for integration and all for expressivity of queries, yet cannot answer the first question of the entry exam?</p> <p>The world out there is complex. But maybe we ought to make some simple tutorials? So, as a call to the people out there, tell us what a good tutorial would be. The question is more about figuring out what is out there and adapting these and making a sort of compatibility list. Jena and Sesame stuff ought to run as is. We could offer a webinar to all the data web luminaries showing how to promote the data web message with Virtuoso. After all, why not show it on the best platform?</p> <h3>"You are arrogant. When I read your papers or documentation, the impression I get is that you say you are smart and the reader is stupid."</h3> <p>We should answer in multiple parts.</p> <p>For general collateral, like web sites and documentation:</p> <p>The web site gives a confused product image. For the Virtuoso product, we should divide at the top into</p> <ul> <li> Data web and RDF - Host linked data, expose relational assets as linked data;</li> <li> Relational Database - Full function, high performance, open source, Federated/Virtual Relational DBMS, expose heterogeneous RDB assets through one point of contact for integration;</li> <li> Web Services - access all the above over standard protocols, dynamic web pages, web hosting.</li> </ul> <p>For each point, one simple statement. We all know what the above things mean?</p> <p>Then we add a new point about scalability that impacts all the above, namely the Virtuoso version 6 Cluster, meaning that you can do all these things at 10 to 1000 times the scale. This means this much more data or in some cases this much more requests per second. This too is clear.</p> <p>Far as I am concerned, hosting Java or .<a href="http://dbpedia.org/resource/.NET_Framework" id="link-id0x20283a88">NET</a> does not have to be on the front page. Also, we have no great interest in going against <a href="http://dbpedia.org/resource/Apache" id="link-id0x2024a068">Apache</a> when it comes to a web server only situation. The fact that we have a web listener is important for some things but our claim to fame does not rest on this.</p> <p>Then for documentation and training materials: The documentation should be better. Specifically it should have more of a how-to dimension since nobody reads the whole thing anyhow. About online tutorials, the order of presentation should be different. They do not really reflect what is important at the present moment either.</p> <p>Now for conference papers: Since taking the data web as a focus area, we have submitted some papers and had some rejected because these do not have enough references and do not explain what is obvious to ourselves.</p> <p>I think that the communications failure in this case is that we want to talk about end to end solutions and the reviewers expect research. For us, the solution is interesting and exists only if there is an adequate functionality mix for addressing a specific use case. This is why we do not make a paper about query cost model alone because the cost model, while indispensable, is a thing that is taken for granted where we come from. So we mention RDF adaptations to cost model, as these are important to the whole but do not find these to be the justification for a whole paper. If we made papers on this basis, we would have to make five times as many. Maybe we ought to.</p> <h3>"Virtuoso is very big and very difficult"</h3> <p>One thing that is not obvious from the Virtuoso packaging is that the minimum installation is an executable under 10MB and a config file. Two files.</p> <p>This gives you SQL and SPARQL out of the box. Adding <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x1ee61058">ODBC</a> and <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x1b8c31c0">JDBC</a> clients is as simple as it gets. After this, there is basic database functionality. Tuning is a matter of a few parameters that are explained on this blog and elsewhere. Also, the full scale installation is available as an Amazon EC2 image, so no installation required.</p> <p>Now for the difficult side:</p> <p>Use SQL and SPARQL; use stored procedures whenever there is server side business logic. For some time critical web pages, use VSP. Do not use VSPX. Otherwise, use whatever you are used to â <a href="http://dbpedia.org/resource/PHP" id="link-id0x20a13c00">PHP</a> or Java or anything else. For web services, simple is best. Stick to basics. "The engineer is one who can invent a simple thing." Use SQL statements rather than admin UI.</p> <p>Know that you can start a server with no database file and you get an initial database with nothing extra. The demo database, the way it is produced by installers is cluttered.</p> <p>We should put this into a couple of use case oriented how-tos.</p> <p>Also, we should create a network of "friendly local virtuoso geeks" for providing basic training and services so we do not have to explain these things all the time. To all you data-web-ers out there â please sign up and we will provide instructions, etc. Contact Yrjänä Rankka (ghard[at-sign]openlinksw.com), or go through the mailing lists; do not contact me directly.</p> <h3>"OK, we understand that you may be good at the large end of the spectrum but how do you reconcile this with the lightweight or embedded end, like the semantic desktop?"</h3> <p>Now, what is good for one end is usually good for the other. Namely, a database, no matter the scale, needs to have space efficient storage, fast index lookup, and correct query plans. Then there are things that occur only at the high-end, like clustering, but these are separate things. For embedding, the initial memory footprint needs to be small. With Virtuoso, this is accomplished by leaving out some 200 built-in tables and 100,000 lines of SQL procedures that are normally in by default, supporting things such as DAV and diverse other protocols. After all, if SPARQL is all one wants these are not needed.</p> <p>If one really wants to do one's server logic (like web listener and thread dispatching) oneself, this is not impossible but requires some advice from us. On the other hand, if one wants to have logic for security close to the data, then using stored procedures is recommended; these execute right next to the data, and support inline SPARQL and SQL. Depending on the license status of the other code, some special licensing arrangements may apply.</p> <p>We are talking about such things with different parties at present.</p> <h3>"How webby are you? What is webby?"</h3> <p>"Webby means distributed, heterogeneous, open; not monolithic consolidation of everything."</p> <p>We are philosophically webby. We come from open standards; we are after all called OpenLink; our history consists of connecting things. We believe in choice â the user should be able to pick the best of breed for components and have them work together. We cannot and do not wish to force replacement of existing assets. Transforming data on the fly and connecting systems, leaving data where it originally resides, is the first preference. For the data web, the first preference is a federation of independent SPARQL end points. When there is harvesting, we prefer to do it on demand, as with our Sponger. With the immense amount of data out there we believe in finding what is relevant <i>when</i> it is relevant, preferably close at hand, leveraging things like social networks. With a data web, many things which are now siloized, such as marketplaces and social networks, will return to the open.</p> <p>Google-style crawling of everything becomes less practical if one needs to run complex <i>ad hoc</i> queries against the mass of data. For these types of scenarios, if one needs to warehouse, the data cloud will offer solutions where one pays for database on demand. While we believe in loosely coupled federation where possible, we have serious work on the scalability side for the data center and the compute-on-demand cloud.</p> <h3>"How does OpenLink see the next five years unfolding?"</h3> <p>Personally, I think we have the basics for the birth of a new inflection in the <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x1fb9ae58">knowledge</a> economy. The <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x1f07c648">URI</a> is the unit of exchange; its value and competitive edge lie in the data it links you with. A name without context is worth little, but as a name gets more use, more <a href="http://dbpedia.org/resource/Information" id="link-id0x1f007d60">information</a> can be found through that name. This is anything from financial statistics, to legal precedents, to news reporting or government data. Right now, if the SEC just added one line of markup to the XBRL template, this would instantaneously make all SEC-mandated reporting into linked data via GRDDL.</p> <p>The URI is a carrier of brand. An information brand gets traffic and references, and this can be monetized in diverse ways. The key word is <i>context</i>. Information overload is here to stay, and only better context offers the needed increase in productivity to stay ahead of the flood.</p> <p>Semantic technologies on the whole can help with this. Why these should be semantic web or data web technologies as opposed to just semantic is the linked data value proposition. Even smart islands are still islands. Agility, scale, and scope, depend on the possibility of combining things. Therefore common terminologies and dereferenceability and discoverability are important. Without these, we are at best dealing with closed systems even if they were smart. The expert systems of the 1980s are a case in point.</p> <p>Ever since the .com era, the <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id0x2048e670">URL</a> has been a brand. Now it becomes a URI. Thus, entirely hiding the URI from the user experience is not always desirable. The URI is a sort of handle on the provenance and where more can be found; besides, people are already used to these.</p> <p>With linked data, information value-add products become easy to build and deploy. They can be basically just canned SPARQL queries combining data in a useful and insightful manner. And where there is traffic there can be monetization, whether by advertizing, subscription, or other means. Such possibilities are a natural adjunct to the blogosphere. To publish analysis, one no longer needs to be a think tank or media company. We could call this scenario the birth of a meshup economy.</p> <p>For OpenLink itself, this is our roadmap. The immediate future is about getting our high end offerings like clustered RDF storage generally available, both on the cloud and for private data centers. Ourselves, we will offer the whole <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1c696170">Linked Open Data</a> cloud as a database. The single feature to come in version 2 of this is fully automatic partitioning and repartitioning for on-demand scale; now, you have to choose how many partitions you have.</p> <p>This makes some things possible that were hard thus far.</p> <p>On the mapping front, we go for real-scale data integration scenarios where we can show that SPARQL can unify terms and concepts across databases, yet bring no added cost for complex queries. Enterprises can use their existing warehouses and have an added level of abstraction, the possibility of cross systems interlinking, the advantages of using the same taxonomies and ontologies across systems, and so forth.</p> <p>Then there will be developments in the direction of smarter web harvesting on demand with the Virtuoso <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id0x206ab780">Sponger</a>, and federation of heterogeneous SPARQL end points. The federation is not so unlike clustering, except the time scales are 2 orders of magnitude longer. The work on SPARQL end point statistics and data set description and discovery is a good development in the community.</p> <p>Then there will be NLP integration, as exemplified by the Open Calais linked data wrapper and more.</p> <p>Can we pull this off or is this being spread too thin? We know from experience that all this can be accomplished. Scale is already here; we show it with the billion triples set. Mapping is here; we showed it last in the Berlin Benchmark. We will also show some TPC-H results after we get a little quiet after the ISWC event. Then there is ongoing maintenance but with this we have shown a steady turnaround and quick time to fix for pretty much anything.</p>
Virtuoso - Are We Too Clever for Our Own Good? (updated)
http://www.openlinksw.com/weblog/oerling/?date=2008-10-26#1465
2008-10-26T12:15:35Z
2008-10-27T12:07:52-04:00
<p>"Physician, heal thyself," it is said. We profess to say what the messaging of the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x1fa3da18">semantic web</a> ought to be, but is our own perfect?</p> <p>I will here engage in some critical introspection as well as amplify on some answers given to <a href="http://virtuoso.openlinksw.com" id="link-id0x1e1eecf0">Virtuoso</a>-related questions in recent times.</p> <p>I use some conversations from the <a href="http://dbpedia.org/resource/Vienna" id="link-id0x1ec0b2e0">Vienna</a> <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x2045ac10">Linked Data</a> Practitioners meeting as a starting point. These views are mine and are limited to the Virtuoso server. These do not apply to the <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id0x2045ac38">ODS</a> (<a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id0x14f63c58">OpenLink Data Spaces</a>) applications line, <a href="http://oat.openlinksw.com/" id="link-id0x14f63c80">OAT</a> (<a href="http://oat.openlinksw.com/" id="link-id0x1e536928">OpenLink Ajax Toolkit</a>), or <a href="http://ode.openlinksw.com/" id="link-id0x1eaed7f8">ODE</a> (<a href="http://ode.openlinksw.com/" id="link-id0x1edfff88">OpenLink Data Explorer</a>).</p> <h3>"It is not always clear what the main thrust is, we get the impression that you are spread too thin," said <a href="http://www.informatik.uni-leipzig.de/~auer/foaf.rdf#me" id="link-id0x1b8a9580">Sören Auer</a>.</h3> <p>Well, personally, I am all for core competence. This is why I do not participate in all the online conversations and groups as much as I could, for example. Time and energy are critical resources and must be invested where they make a difference. In this case, the real core competence is running in the database race. This in itself, come to think of it, is a pretty broad concept.</p> <p>This is why we put a lot of emphasis on Linked Data and the <a href="http://dbpedia.org/resource/Data" id="link-id0x1b85fa38">Data</a> Web for now, as this is the emerging game. This is a deliberate choice, not an outside imperative or built-in limitation. More specifically, this means exposing any pre-existing relational data as linked data plus being the definitive <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1f5b4468">RDF</a> store.</p> <p>We can do this because we own our database and <a href="http://dbpedia.org/resource/SQL" id="link-id0x20076468">SQL</a> and data access middleware and have a history of connecting to any <a href="http://dbpedia.org/resource/Relational_database_management_system" id="link-id0x1ffd6f98">RDBMS</a> out there.</p> <p>The principal message we have been hearing from the RDF field is the call for scale of triple storage. This is even louder than the call for relational mapping. We believe that in time mapping will exceed triple storage as such, once we get some real production strength mappings deployed, enough to outperform RDF warehousing.</p> <p>There are also RDF middleware things like RDF-ization and demand-driven web harvesting (i.e, the so-called Sponger). These are <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1316f720">SPARQL</a> options, thus accessed via standard interfaces. We have little desire to create our own languages or APIs, or to tell people how to program. This is why we recently introduced <a href="http://sourceforge.net/projects/sesame/" id="link-id0x20756a68">Sesame</a>- and <a href="http://jena.sourceforge.net/" id="link-id0x1ec01ac0">Jena</a>-compatible APIs to our RDF store. From what we hear, these work. On the other hand, we do not hesitate to move beyond the standards when there is obvious value or necessity. This is why we brought SPARQL up to and beyond SQL expressivity. It is not a case of E3 (Embrace, Extend, Extinguish).</p> <p>Now, this message could be better reflected in our material on the web. This <a href="http://dbpedia.org/resource/Blog" id="link-id0x2027b410">blog</a> is a rather informal step in this direction; more is to come. For now we concentrate on delivering.</p> <p>The conventional communications wisdom is to split the message by target audience. For this, we should split the RDF, relational, and web services messages from each other. We believe that a challenger, like the semantic web technology stack, must have a compelling message to tell for it to be interesting. This is not a question of research prototypes. The new technology cannot lack something the installed technology takes for granted.</p> <p>This is why we do not tend to show things like how to insert and query a few triples: No business out there will insert and query triples for the sake of triples. There must be a more compelling story â for example, turning the whole world into a database. This is why our examples start with things like turning the <a href="http://dbpedia.org/resource/TPC-H" id="link-id0x2051ff98">TPC-H</a> database into RDF, queries and all. Anything less is not interesting. Why would an enterprise that has business intelligence and integration issues way more complex than the rather stereotypical TPC-H even look at a technology that pretends to be all for integration and all for expressivity of queries, yet cannot answer the first question of the entry exam?</p> <p>The world out there is complex. But maybe we ought to make some simple tutorials? So, as a call to the people out there, tell us what a good tutorial would be. The question is more about figuring out what is out there and adapting these and making a sort of compatibility list. Jena and Sesame stuff ought to run as is. We could offer a webinar to all the data web luminaries showing how to promote the data web message with Virtuoso. After all, why not show it on the best platform?</p> <h3>"You are arrogant. When I read your papers or documentation, the impression I get is that you say you are smart and the reader is stupid."</h3> <p>We should answer in multiple parts.</p> <p>For general collateral, like web sites and documentation:</p> <p>The web site gives a confused product image. For the Virtuoso product, we should divide at the top into</p> <ul> <li> Data web and RDF - Host linked data, expose relational assets as linked data;</li> <li> Relational Database - Full function, high performance, open source, Federated/Virtual Relational DBMS, expose heterogeneous RDB assets through one point of contact for integration;</li> <li> Web Services - access all the above over standard protocols, dynamic web pages, web hosting.</li> </ul> <p>For each point, one simple statement. We all know what the above things mean?</p> <p>Then we add a new point about scalability that impacts all the above, namely the Virtuoso version 6 Cluster, meaning that you can do all these things at 10 to 1000 times the scale. This means this much more data or in some cases this much more requests per second. This too is clear.</p> <p>Far as I am concerned, hosting Java or .<a href="http://dbpedia.org/resource/.NET_Framework" id="link-id0x1f297540">NET</a> does not have to be on the front page. Also, we have no great interest in going against <a href="http://dbpedia.org/resource/Apache" id="link-id0x1ea29578">Apache</a> when it comes to a web server only situation. The fact that we have a web listener is important for some things but our claim to fame does not rest on this.</p> <p>Then for documentation and training materials: The documentation should be better. Specifically it should have more of a how-to dimension since nobody reads the whole thing anyhow. About online tutorials, the order of presentation should be different. They do not really reflect what is important at the present moment either.</p> <p>Now for conference papers: Since taking the data web as a focus area, we have submitted some papers and had some rejected because these do not have enough references and do not explain what is obvious to ourselves.</p> <p>I think that the communications failure in this case is that we want to talk about end to end solutions and the reviewers expect research. For us, the solution is interesting and exists only if there is an adequate functionality mix for addressing a specific use case. This is why we do not make a paper about query cost model alone because the cost model, while indispensable, is a thing that is taken for granted where we come from. So we mention RDF adaptations to cost model, as these are important to the whole but do not find these to be the justification for a whole paper. If we made papers on this basis, we would have to make five times as many. Maybe we ought to.</p> <h3>"Virtuoso is very big and very difficult"</h3> <p>One thing that is not obvious from the Virtuoso packaging is that the minimum installation is an executable under 10MB and a config file. Two files.</p> <p>This gives you SQL and SPARQL out of the box. Adding <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id0x20a2e7d0">ODBC</a> and <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0x1e4cceb8">JDBC</a> clients is as simple as it gets. After this, there is basic database functionality. Tuning is a matter of a few parameters that are explained on this blog and elsewhere. Also, the full scale installation is available as an Amazon EC2 image, so no installation required.</p> <p>Now for the difficult side:</p> <p>Use SQL and SPARQL; use stored procedures whenever there is server side business logic. For some time critical web pages, use VSP. Do not use VSPX. Otherwise, use whatever you are used to â <a href="http://dbpedia.org/resource/PHP" id="link-id0x20b03f08">PHP</a> or Java or anything else. For web services, simple is best. Stick to basics. "The engineer is one who can invent a simple thing." Use SQL statements rather than admin UI.</p> <p>Know that you can start a server with no database file and you get an initial database with nothing extra. The demo database, the way it is produced by installers is cluttered.</p> <p>We should put this into a couple of use case oriented how-tos.</p> <p>Also, we should create a network of "friendly local virtuoso geeks" for providing basic training and services so we do not have to explain these things all the time. To all you data-web-ers out there â please sign up and we will provide instructions, etc. Contact Yrjänä Rankka (ghard[at-sign]openlinksw.com), or go through the mailing lists; do not contact me directly.</p> <h3>"OK, we understand that you may be good at the large end of the spectrum but how do you reconcile this with the lightweight or embedded end, like the semantic desktop?"</h3> <p>Now, what is good for one end is usually good for the other. Namely, a database, no matter the scale, needs to have space efficient storage, fast index lookup, and correct query plans. Then there are things that occur only at the high-end, like clustering, but these are separate things. For embedding, the initial memory footprint needs to be small. With Virtuoso, this is accomplished by leaving out some 200 built-in tables and 100,000 lines of SQL procedures that are normally in by default, supporting things such as DAV and diverse other protocols. After all, if SPARQL is all one wants these are not needed.</p> <p>If one really wants to do one's server logic (like web listener and thread dispatching) oneself, this is not impossible but requires some advice from us. On the other hand, if one wants to have logic for security close to the data, then using stored procedures is recommended; these execute right next to the data, and support inline SPARQL and SQL. Depending on the license status of the other code, some special licensing arrangements may apply.</p> <p>We are talking about such things with different parties at present.</p> <h3>"How webby are you? What is webby?"</h3> <p>"Webby means distributed, heterogeneous, open; not monolithic consolidation of everything."</p> <p>We are philosophically webby. We come from open standards; we are after all called OpenLink; our history consists of connecting things. We believe in choice â the user should be able to pick the best of breed for components and have them work together. We cannot and do not wish to force replacement of existing assets. Transforming data on the fly and connecting systems, leaving data where it originally resides, is the first preference. For the data web, the first preference is a federation of independent SPARQL end points. When there is harvesting, we prefer to do it on demand, as with our Sponger. With the immense amount of data out there we believe in finding what is relevant <i>when</i> it is relevant, preferably close at hand, leveraging things like social networks. With a data web, many things which are now siloized, such as marketplaces and social networks, will return to the open.</p> <p>Google-style crawling of everything becomes less practical if one needs to run complex <i>ad hoc</i> queries against the mass of data. For these types of scenarios, if one needs to warehouse, the data cloud will offer solutions where one pays for database on demand. While we believe in loosely coupled federation where possible, we have serious work on the scalability side for the data center and the compute-on-demand cloud.</p> <h3>"How does OpenLink see the next five years unfolding?"</h3> <p>Personally, I think we have the basics for the birth of a new inflection in the <a href="http://dbpedia.org/resource/Knowledge" id="link-id0x2018bd98">knowledge</a> economy. The <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x1ec110d8">URI</a> is the unit of exchange; its value and competitive edge lie in the data it links you with. A name without context is worth little, but as a name gets more use, more <a href="http://dbpedia.org/resource/Information" id="link-id0x1ecfba08">information</a> can be found through that name. This is anything from financial statistics, to legal precedents, to news reporting or government data. Right now, if the SEC just added one line of markup to the XBRL template, this would instantaneously make all SEC-mandated reporting into linked data via GRDDL.</p> <p>The URI is a carrier of brand. An information brand gets traffic and references, and this can be monetized in diverse ways. The key word is <i>context</i>. Information overload is here to stay, and only better context offers the needed increase in productivity to stay ahead of the flood.</p> <p>Semantic technologies on the whole can help with this. Why these should be semantic web or data web technologies as opposed to just semantic is the linked data value proposition. Even smart islands are still islands. Agility, scale, and scope, depend on the possibility of combining things. Therefore common terminologies and dereferenceability and discoverability are important. Without these, we are at best dealing with closed systems even if they were smart. The expert systems of the 1980s are a case in point.</p> <p>Ever since the .com era, the <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id0x1c4c9248">URL</a> has been a brand. Now it becomes a URI. Thus, entirely hiding the URI from the user experience is not always desirable. The URI is a sort of handle on the provenance and where more can be found; besides, people are already used to these.</p> <p>With linked data, information value-add products become easy to build and deploy. They can be basically just canned SPARQL queries combining data in a useful and insightful manner. And where there is traffic there can be monetization, whether by advertizing, subscription, or other means. Such possibilities are a natural adjunct to the blogosphere. To publish analysis, one no longer needs to be a think tank or media company. We could call this scenario the birth of a meshup economy.</p> <p>For OpenLink itself, this is our roadmap. The immediate future is about getting our high end offerings like clustered RDF storage generally available, both on the cloud and for private data centers. Ourselves, we will offer the whole <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x20791bf0">Linked Open Data</a> cloud as a database. The single feature to come in version 2 of this is fully automatic partitioning and repartitioning for on-demand scale; now, you have to choose how many partitions you have.</p> <p>This makes some things possible that were hard thus far.</p> <p>On the mapping front, we go for real-scale data integration scenarios where we can show that SPARQL can unify terms and concepts across databases, yet bring no added cost for complex queries. Enterprises can use their existing warehouses and have an added level of abstraction, the possibility of cross systems interlinking, the advantages of using the same taxonomies and ontologies across systems, and so forth.</p> <p>Then there will be developments in the direction of smarter web harvesting on demand with the Virtuoso <a href="http://virtuoso.openlinksw.com/Whitepapers/html/VirtSpongerWhitePaper.html" id="link-id0x1f27e6d8">Sponger</a>, and federation of heterogeneous SPARQL end points. The federation is not so unlike clustering, except the time scales are 2 orders of magnitude longer. The work on SPARQL end point statistics and data set description and discovery is a good development in the community.</p> <p>Then there will be NLP integration, as exemplified by the Open Calais linked data wrapper and more.</p> <p>Can we pull this off or is this being spread too thin? We know from experience that all this can be accomplished. Scale is already here; we show it with the billion triples set. Mapping is here; we showed it last in the Berlin Benchmark. We will also show some TPC-H results after we get a little quiet after the ISWC event. Then there is ongoing maintenance but with this we have shown a steady turnaround and quick time to fix for pretty much anything.</p>
The Trouble with Labels (Contd.): Data Integration & SOA
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2008-10-12#1457
2008-10-12T18:53:44Z
2008-10-12T18:54:22-04:00
<p>I just stumbled across an post from <a href="http://www.itbusinessedge.com" id="link-id10f82f50">ITBusines Edge</a> titled: <a href="http://www.itbusinessedge.com/item/?ci=48119" id="link-id10f37b90">How Semantic Technology Can Help Companies with Integration</a>. While reading the post I encountered the term: <a href="http://dbpedia.org/resource/Master_Data_Management" id="link-id11055eb8">Master Data Manager (MDM)</a>, and wondered to myself, "what's that?" only to realize it's the very same thing I described as a <a href="http://dbpedia.org/resource/Federated_database_system" id="link-id13985af0">Data Virtualization</a> or <a href="http://dbpedia.org/resource/Virtual_Database" id="link-id1167c720">Virtual Database technology</a> (circa. 1998).</p> <p>Now, if re-labeling can confuse <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id14aaaaf0">me</a> when applied to a realm I've been intimately involved with for eons (<a href="http://dbpedia.org/resource/Internet" id="link-id112042f0">internet</a> time). I don't want to imagine what it does for others who aren't that intimately involved with the important data access and data integration realms. </p> <p>On the more refreshing side, the article does shed some light on the potency of RDF and OWL when applied to the construction of conceptual views of heterogeneous data sources.</p> <blockquote> <cite>"How do you know that data coming from one place calculates net revenue the same way that data coming from another place does? Youâve got people using the same term for different things and different terms for the same things. How do you reconcile all of that? Thatâs really what semantic integration is about." </cite> </blockquote> <p>BTW - I discovered this article via another titled: <a href="http://www.itbusinessedge.com/blogs/mia/?p=485" id="link-id11134098">Understanding Integration And How It Can Help with SOA</a>, that covers SOA and Integration matters. Again, in this piece I feel the gradual realization of the virtues that RDF, OWL, and RDF <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11048740">Linked Data</a> bring to bear in the vital realm of data integration across heterogeneous data silos.</p> <h3>Conclusion</h3> <p>A number of events, at the micro and macro economic levels, are forcing attention back to the issue of productive use of existing IT resources. The trouble with the aforementioned quest is that it ultimately unveils the global IT affliction known as: heterogeneous data silos, and the challenges of pain alleviation, that have been ignored forever or approached inadequately as clearly shown by the rapid build up of SOA horror stories in the data integration realm.</p> <p>Data Integration via conceptualization of heterogenous data sources, that result in concrete conceptual layer data access and management, remains the greatest and most potent application of technologies associated with the "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id10fa5050">Semantic Web</a>" and/or "Linked Data" monikers.</p> <h3>Related</h3> <ul> <li> <a href="http://www.infoworld.com/article/03/05/23/21FEinnovidehen_1.html" id="link-id118c9c00">InforWorld 2003 Innovator article</a> </li> <li> <a href="http://weblog.infoworld.com/udell/2006/04/28.html" id="link-id11057298">2006 Podcast Interview with Jon Udell</a> </li> <li> <a href="http://dbpedia.org/resource/Enterprise_Information_Integration" id="link-id13f89030">Enterprise Information Integration</a> </li> <li>One of <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=data%20integration&type=text&output=html" id="link-id11048b98">several posts</a> about our <a href="http://virtuoso.openlinksw.com" id="link-id10fef0e0">Virtuoso</a> <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id10e5a068">Universal Server</a> and <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com%27s%20BLOG%20%5B127%5D/1406" id="link-id111d5aa8">Conceptual Model based data integration</a> </li> <li> <a href="http://virtuoso.openlinksw.com/wiki/main/Main/VOSHistory" id="link-id11020108">History of Virtuoso</a> </li> <li> <a href="http://www.mkbergman.com/me/" id="link-id1101e7b0">Mike Bergman</a>'s post titled: <a href="http://www.mkbergman.com/?p=459" id="link-id10fdb640">WOA: A New Enterprise Partner for Linked Data</a> </li> </ul>
Transitivity and Graphs for SQL
http://www.openlinksw.com/blog/vdb/blog/?date=2008-09-08#1435
2008-09-08T09:41:24Z
2008-09-08T15:43:07-04:00
<div> <div style="display:none;">Transitivity and Graphs for SQL</div> <h2>Background</h2> <p>I have mentioned on a couple of prior occasions that basic graph operations ought to be integrated into the <a href="http://dbpedia.org/resource/SQL" id="link-id0xa1a18c58">SQL</a> query language.</p> <p>The history of databases is by and large about moving from specialized applications toward a generic platform. The introduction of the DBMS itself is the archetypal example. It is all about extracting the common features of applications and making these the features of a platform instead.</p> <p>It is now time to apply this principle to graph traversal.</p> <p>The rationale is that graph operations are somewhat tedious to write in a parallelize-able, latency-tolerant manner. Writing them as one would for memory-based <a href="http://dbpedia.org/resource/Data" id="link-id0xaf8c730">data</a> structures is easier but totally unscalable as soon as there is any latency involved, i.e., disk reads or messages between cluster peers.</p> <p>The ad-hoc nature and very large volume of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xae41ef0">RDF</a> data makes this a timely question. Up until now, the answer to this question has been to materialize any implied facts in RDF stores. If <i>a</i> was part of <i>b</i>, and <i>b</i> part of <i><a href="http://dbpedia.org/resource/C_(programming_language)" id="link-id0xac9d8790">c</a></i>, the implied fact that <i>a</i> is part of <i>c</i> would be inserted explicitly into the database as a pre-query step.</p> <p>This is simple and often efficient, but tends to have the downside that one makes a specialized warehouse for each new type of query. The activity becomes less ad-hoc.</p> <p>Also, this becomes next to impossible when the scale approaches web scale, or if some of the data is liable to be on-and-off included-into or excluded-from the set being analyzed. This is why with <a href="http://virtuoso.openlinksw.com" id="link-id0xb68f9d0">Virtuoso</a> we have tended to favor inference on demand ("backward chaining") and mapping of relational data into RDF without copying.</p> <p>The SQL world has taken steps towards dealing with recursion with the <code>WITH - UNION</code> construct which allows definition of recursive views. The idea there is to define, for example, a tree walk as a <code>UNION</code> of the data of the starting node plus the recursive walk of the starting node's immediate children.</p> <p>The main problem with this is that I do not very well see how a SQL optimizer could effectively rearrange queries involving <code>JOIN</code>s between such recursive views. This model of recursion seems to lose SQL's non-procedural nature. One can no longer easily rearrange <code>JOIN</code>s based on what data is given and what is to be retrieved. If the recursion is written from root to leaf, it is not obvious how to do this from leaf to root. At any rate, queries written in this way are so complex to write, let alone optimize, that I decided to take another approach.</p> <p>Take a question like "list the parts of products of category <i>C</i> which have materials that are classified as toxic." Suppose that the product categories are a tree, the product parts are a tree, and the materials classification is a tree taxonomy where "toxic" has a multilevel substructure.</p> <p>Depending on the count of products and materials, the query can be evaluated as either going from products to parts to materials and then climbing up the materials tree to see if the material is toxic. Or one could do it in reverse, starting with the different toxic materials, looking up the parts containing these, going to the part tree to the product, and up the product hierarchy to see if the product is in the right category. One should be able to evaluate the identical query either way depending on what indices exist, what the cardinalities of the relations are, and so forth â regular cost based optimization.</p> <p>Especially with RDF, there are many problems of this type. In regular SQL, it is a long-standing cultural practice to flatten hierarchies, but this is not the case with RDF.</p> <p>In Virtuoso, we see <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xb3bdcc0">SPARQL</a> as reducing to SQL. Any RDF-oriented database-engine or query-optimization feature is accessed via SQL. Thus, if we address run-time-recursion in the Virtuoso query engine, this becomes, <i>ipso facto</i>, an SQL feature. Besides, we remember that SQL is a much more mature and expressive language than the current SPARQL recommendation.</p> <h2> SQL and Transitivity </h2> <p>We will here look at some simple social network queries. A later article will show how to do more general graph operations. We extend the SQL derived table construct, i.e., <code>SELECT</code> in another <code>SELECT</code>'s <code>FROM</code> clause, with a <code>TRANSITIVE</code> clause.</p> <p>Consider the data:</p> <blockquote> <pre><code>CREATE TABLE "knows" ("p1" INT, "p2" INT, PRIMARY KEY ("p1", "p2") ); ALTER INDEX "knows" ON "knows" PARTITION ("p1" INT); CREATE INDEX "knows2" ON "knows" ("p2", "p1") PARTITION ("p2" INT); </code> </pre></blockquote> <p>We represent a social network with the many-to-many relation "knows". The persons are identified by integers.</p> <blockquote> <pre><code>INSERT INTO "knows" VALUES (1, 2); INSERT INTO "knows" VALUES (1, 3); INSERT INTO "knows" VALUES (2, 4);</code> </pre> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2" FROM "knows" ) "k" WHERE "k"."p1" = 1;</code></pre></blockquote> <p>We obtain the result:</p> <blockquote> <table width="100"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> </tr> <tr> <td align="center">1</td> <td align="center">3</td> </tr> <tr> <td align="center">1</td> <td align="center">2</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> </tr> </table> </blockquote> <p>The operation is reversible:</p> <blockquote> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2" FROM "knows" ) "k" WHERE "k"."p2" = 4; </code> </pre> <table width="100"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> </tr> <tr> <td align="center">2</td> <td align="center">4</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> </tr> </table> </blockquote> <p>Since now we give <i>p2</i>, we traverse from <i>p2</i> towards <i>p1</i>. The result set states that 4 is known by 2 and 2 is known by 1.</p> <p>To see what would happen if <i>x</i> knowing <i>y</i> also meant <i>y</i> knowing <i>x</i>, one could write:</p> <blockquote> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2" FROM (SELECT "p1", "p2" FROM "knows" UNION ALL SELECT "p2", "p1" FROM "knows" ) "k2" ) "k" WHERE "k"."p2" = 4;</code> </pre> <table width="100"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> </tr> <tr> <td align="center">2</td> <td align="center">4</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> </tr> <tr> <td align="center">3</td> <td align="center">4</td> </tr> </table> </blockquote> <p>Now, since we know that 1 and 4 are related, we can ask how they are related.</p> <blockquote> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2", T_STEP (1) AS "via", T_STEP ('step_no') AS "step", T_STEP ('path_id') AS "path" FROM "knows" ) "k" WHERE "p1" = 1 AND "p2" = 4;</code> </pre> <table width="250"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> <th align="center" width="50">via</th> <th align="center" width="50">step</th> <th align="center" width="50">path</th> </tr> <tr> <td align="center">1</td> <td align="center">4</td> <td align="center">1</td> <td align="center">0</td> <td align="center">0</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> <td align="center">2</td> <td align="center">1</td> <td align="center">0</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> <td align="center">4</td> <td align="center">2</td> <td align="center">0</td> </tr> </table> </blockquote> <p>The two first columns are the ends of the path. The next column is the person that is a step on the path. The next one is the number of the step, counting from 0, so that the end of the path that corresponds to the end condition on the column designated as input, i.e., <i>p1</i>, has number 0. Since there can be multiple solutions, the last column is a sequence number allowing distinguishing multiple alternative paths from each other.</p> <p>For LinkedIn users, the friends ordered by distance and descending friend count query, which is at the basis of most LinkedIn search result views can be written as: </p> <blockquote> <pre><code>SELECT p2, dist, (SELECT COUNT (*) FROM "knows" "c" WHERE "c"."p1" = "k"."p2" ) FROM (SELECT TRANSITIVE t_in (1) t_out (2) t_distinct "p1", "p2", t_step ('step_no') AS "dist" FROM "knows" ) "k" WHERE "p1" = 1 ORDER BY "dist", 3 DESC;</code> </pre> <table width="150"> <tr> <th align="center" width="50">p2</th> <th align="center" width="50">dist</th> <th align="center" width="50">aggregate</th> </tr> <tr> <td align="center">2</td> <td align="center">1</td> <td align="center">1</td> </tr> <tr> <td align="center">3</td> <td align="center">1</td> <td align="center">0</td> </tr> <tr> <td align="center">4</td> <td align="center">2</td> <td align="center">0</td> </tr> </table> </blockquote> <h2>How?</h2> <p>The queries shown above work on Virtuoso v6. When running in cluster mode, several thousand graph traversal steps may be proceeding at the same time, meaning that all database access is parallelized and that the algorithm is internally latency-tolerant. By default, all results are produced in a deterministic order, permitting predictable slicing of result sets.</p> <p>Furthermore, for queries where both ends of a path are given, the optimizer may decide to attack the path from both ends simultaneously. So, supposing that every member of a social network has an average of 30 contacts, and we need to find a path between two users that are no more than 6 steps apart, we begin at both ends, expanding each up to 3 levels, and we stop when we find the first intersection. Thus, we reach 2 * 30^3 = 54,000 nodes, and not 30^6 = 729,000,000 nodes.</p> <p>Writing a generic database driven graph traversal framework on the application side, say in Java over <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0xa8a9ef8">JDBC</a>, would easily be over a thousand lines. This is much more work than can be justified just for a one-off, ad-hoc query. Besides, the traversal order in such a case could not be optimized by the DBMS.</p> <h2>Next</h2> <p>In a future <a href="http://dbpedia.org/resource/Blog" id="link-id0xb526a40">blog</a> post I will show how this feature can be used for common graph tasks like critical path, itinerary planning, traveling salesman, the 8 queens chess problem, etc. There are lots of switches for controlling different parameters of the traversal. This is just the beginning. I will also give examples of the use of this in SPARQL.</p> </div>
Transitivity and Graphs for SQL
http://www.openlinksw.com/weblog/oerling/?date=2008-09-08#1433
2008-09-08T09:20:11Z
2008-09-08T15:43:04.000006-04:00
<h2>Background</h2> <p>I have mentioned on a couple of prior occasions that basic graph operations ought to be integrated into the <a href="http://dbpedia.org/resource/SQL" id="link-id0xb1fe830">SQL</a> query language.</p> <p>The history of databases is by and large about moving from specialized applications toward a generic platform. The introduction of the DBMS itself is the archetypal example. It is all about extracting the common features of applications and making these the features of a platform instead.</p> <p>It is now time to apply this principle to graph traversal.</p> <p>The rationale is that graph operations are somewhat tedious to write in a parallelize-able, latency-tolerant manner. Writing them as one would for memory-based <a href="http://dbpedia.org/resource/Data" id="link-id0x1cb37218">data</a> structures is easier but totally unscalable as soon as there is any latency involved, i.e., disk reads or messages between cluster peers.</p> <p>The ad-hoc nature and very large volume of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1e1850a0">RDF</a> data makes this a timely question. Up until now, the answer to this question has been to materialize any implied facts in RDF stores. If <i>a</i> was part of <i>b</i>, and <i>b</i> part of <i><a href="http://dbpedia.org/resource/C_(programming_language)" id="link-id0xa1a08d38">c</a></i>, the implied fact that <i>a</i> is part of <i>c</i> would be inserted explicitly into the database as a pre-query step.</p> <p>This is simple and often efficient, but tends to have the downside that one makes a specialized warehouse for each new type of query. The activity becomes less ad-hoc.</p> <p>Also, this becomes next to impossible when the scale approaches web scale, or if some of the data is liable to be on-and-off included-into or excluded-from the set being analyzed. This is why with <a href="http://virtuoso.openlinksw.com" id="link-id0xa51bd10">Virtuoso</a> we have tended to favor inference on demand ("backward chaining") and mapping of relational data into RDF without copying.</p> <p>The SQL world has taken steps towards dealing with recursion with the <code>WITH - UNION</code> construct which allows definition of recursive views. The idea there is to define, for example, a tree walk as a <code>UNION</code> of the data of the starting node plus the recursive walk of the starting node's immediate children.</p> <p>The main problem with this is that I do not very well see how a SQL optimizer could effectively rearrange queries involving <code>JOIN</code>s between such recursive views. This model of recursion seems to lose SQL's non-procedural nature. One can no longer easily rearrange <code>JOIN</code>s based on what data is given and what is to be retrieved. If the recursion is written from root to leaf, it is not obvious how to do this from leaf to root. At any rate, queries written in this way are so complex to write, let alone optimize, that I decided to take another approach.</p> <p>Take a question like "list the parts of products of category <i>C</i> which have materials that are classified as toxic." Suppose that the product categories are a tree, the product parts are a tree, and the materials classification is a tree taxonomy where "toxic" has a multilevel substructure.</p> <p>Depending on the count of products and materials, the query can be evaluated as either going from products to parts to materials and then climbing up the materials tree to see if the material is toxic. Or one could do it in reverse, starting with the different toxic materials, looking up the parts containing these, going to the part tree to the product, and up the product hierarchy to see if the product is in the right category. One should be able to evaluate the identical query either way depending on what indices exist, what the cardinalities of the relations are, and so forth â regular cost based optimization.</p> <p>Especially with RDF, there are many problems of this type. In regular SQL, it is a long-standing cultural practice to flatten hierarchies, but this is not the case with RDF.</p> <p>In Virtuoso, we see <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xb4b3ce8">SPARQL</a> as reducing to SQL. Any RDF-oriented database-engine or query-optimization feature is accessed via SQL. Thus, if we address run-time-recursion in the Virtuoso query engine, this becomes, <i>ipso facto</i>, an SQL feature. Besides, we remember that SQL is a much more mature and expressive language than the current SPARQL recommendation.</p> <h2> SQL and Transitivity </h2> <p>We will here look at some simple social network queries. A later article will show how to do more general graph operations. We extend the SQL derived table construct, i.e., <code>SELECT</code> in another <code>SELECT</code>'s <code>FROM</code> clause, with a <code>TRANSITIVE</code> clause.</p> <p>Consider the data:</p> <blockquote> <pre><code>CREATE TABLE "knows" ("p1" INT, "p2" INT, PRIMARY KEY ("p1", "p2") ); ALTER INDEX "knows" ON "knows" PARTITION ("p1" INT); CREATE INDEX "knows2" ON "knows" ("p2", "p1") PARTITION ("p2" INT); </code> </pre></blockquote> <p>We represent a social network with the many-to-many relation "knows". The persons are identified by integers.</p> <blockquote> <pre><code>INSERT INTO "knows" VALUES (1, 2); INSERT INTO "knows" VALUES (1, 3); INSERT INTO "knows" VALUES (2, 4);</code> </pre> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2" FROM "knows" ) "k" WHERE "k"."p1" = 1;</code></pre></blockquote> <p>We obtain the result:</p> <blockquote> <table width="100"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> </tr> <tr> <td align="center">1</td> <td align="center">3</td> </tr> <tr> <td align="center">1</td> <td align="center">2</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> </tr> </table> </blockquote> <p>The operation is reversible:</p> <blockquote> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2" FROM "knows" ) "k" WHERE "k"."p2" = 4; </code> </pre> <table width="100"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> </tr> <tr> <td align="center">2</td> <td align="center">4</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> </tr> </table> </blockquote> <p>Since now we give <i>p2</i>, we traverse from <i>p2</i> towards <i>p1</i>. The result set states that 4 is known by 2 and 2 is known by 1.</p> <p>To see what would happen if <i>x</i> knowing <i>y</i> also meant <i>y</i> knowing <i>x</i>, one could write:</p> <blockquote> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2" FROM (SELECT "p1", "p2" FROM "knows" UNION ALL SELECT "p2", "p1" FROM "knows" ) "k2" ) "k" WHERE "k"."p2" = 4;</code> </pre> <table width="100"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> </tr> <tr> <td align="center">2</td> <td align="center">4</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> </tr> <tr> <td align="center">3</td> <td align="center">4</td> </tr> </table> </blockquote> <p>Now, since we know that 1 and 4 are related, we can ask how they are related.</p> <blockquote> <pre><code>SELECT * FROM (SELECT TRANSITIVE T_IN (1) T_OUT (2) T_DISTINCT "p1", "p2", T_STEP (1) AS "via", T_STEP ('step_no') AS "step", T_STEP ('path_id') AS "path" FROM "knows" ) "k" WHERE "p1" = 1 AND "p2" = 4;</code> </pre> <table width="250"> <tr> <th align="center" width="50">p1</th> <th align="center" width="50">p2</th> <th align="center" width="50">via</th> <th align="center" width="50">step</th> <th align="center" width="50">path</th> </tr> <tr> <td align="center">1</td> <td align="center">4</td> <td align="center">1</td> <td align="center">0</td> <td align="center">0</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> <td align="center">2</td> <td align="center">1</td> <td align="center">0</td> </tr> <tr> <td align="center">1</td> <td align="center">4</td> <td align="center">4</td> <td align="center">2</td> <td align="center">0</td> </tr> </table> </blockquote> <p>The two first columns are the ends of the path. The next column is the person that is a step on the path. The next one is the number of the step, counting from 0, so that the end of the path that corresponds to the end condition on the column designated as input, i.e., <i>p1</i>, has number 0. Since there can be multiple solutions, the last column is a sequence number allowing distinguishing multiple alternative paths from each other.</p> <p>For LinkedIn users, the friends ordered by distance and descending friend count query, which is at the basis of most LinkedIn search result views can be written as: </p> <blockquote> <pre><code>SELECT p2, dist, (SELECT COUNT (*) FROM "knows" "c" WHERE "c"."p1" = "k"."p2" ) FROM (SELECT TRANSITIVE t_in (1) t_out (2) t_distinct "p1", "p2", t_step ('step_no') AS "dist" FROM "knows" ) "k" WHERE "p1" = 1 ORDER BY "dist", 3 DESC;</code> </pre> <table width="150"> <tr> <th align="center" width="50">p2</th> <th align="center" width="50">dist</th> <th align="center" width="50">aggregate</th> </tr> <tr> <td align="center">2</td> <td align="center">1</td> <td align="center">1</td> </tr> <tr> <td align="center">3</td> <td align="center">1</td> <td align="center">0</td> </tr> <tr> <td align="center">4</td> <td align="center">2</td> <td align="center">0</td> </tr> </table> </blockquote> <h2>How?</h2> <p>The queries shown above work on Virtuoso v6. When running in cluster mode, several thousand graph traversal steps may be proceeding at the same time, meaning that all database access is parallelized and that the algorithm is internally latency-tolerant. By default, all results are produced in a deterministic order, permitting predictable slicing of result sets.</p> <p>Furthermore, for queries where both ends of a path are given, the optimizer may decide to attack the path from both ends simultaneously. So, supposing that every member of a social network has an average of 30 contacts, and we need to find a path between two users that are no more than 6 steps apart, we begin at both ends, expanding each up to 3 levels, and we stop when we find the first intersection. Thus, we reach 2 * 30^3 = 54,000 nodes, and not 30^6 = 729,000,000 nodes.</p> <p>Writing a generic database driven graph traversal framework on the application side, say in Java over <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id0xb595050">JDBC</a>, would easily be over a thousand lines. This is much more work than can be justified just for a one-off, ad-hoc query. Besides, the traversal order in such a case could not be optimized by the DBMS.</p> <h2>Next</h2> <p>In a future <a href="http://dbpedia.org/resource/Blog" id="link-id0x1e4d4f18">blog</a> post I will show how this feature can be used for common graph tasks like critical path, itinerary planning, traveling salesman, the 8 queens chess problem, etc. There are lots of switches for controlling different parameters of the traversal. This is just the beginning. I will also give examples of the use of this in SPARQL.</p>
Crunchbase & Semantic Web Interview (Remix - Update 1)
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2008-08-27#1424
2008-08-27T18:16:37Z
2008-08-27T20:35:15-04:00
<p>After reading <a href="http://blog.crunchbase.com/2008/08/26/building-a-semantic-web-interview-with-benjamin-nowack/" id="link-id16b8e0e0">Bengee's interview with CrunchBase</a>, I decided to knock up a quick interview remix as part of my usual attempt to add to the developing discourse.</p> <blockquote> <cite><a href="http://www.crunchbase.com/" id="link-id17c8e7b8">CrunchBase</a>: When we released the <a href="http://www.crunchbase.com/help/api" id="link-id16681f68">CrunchBase API</a>, you were one of the first developers to step up and quickly released a <a href="http://www.openlinksw.com/dataspace/kidehen@openlinksw.com/weblog/kidehen@openlinksw.com's%20BLOG%20%5B127%5D/1395" id="link-id1016d5f0">CrunchBase Sponger Cartridge</a>. Can you explain what a CrunchBase Sponger Cartridge is?</cite> </blockquote> <blockquote> <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id13243300">Me</a>: A Sponger Cartridge is a <a href="http://dbpedia.org/resource/Data">data</a> access driver for <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> Resources that plugs into our <a href="http://virtuoso.openlinksw.com" id="link-id17042f08">Virtuoso</a> <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id1399b588">Universal Server</a> (DBMS and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id137fd188">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id100b23d8">Web</a> Server combo amongst other things). It uses the internal structure of a resource and/or a web service associated with a resource, to materialize an RDF based <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id10418750">Linked Data graph</a> that essentially describes the resource via its properties (Attributes & Relationships). </blockquote> <br /> <img src="http://virtuoso.openlinksw.com/presentations/Creating_Deploying_Exploiting_Linked_Data2/images/ldp4.png" /> <br /> <br /> <br /> <blockquote> <cite>CrunchBase: And what inspired you to create it?</cite> </blockquote> <blockquote> <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id12fa60c0">Me</a>: Bengee built a new space with your data, and we've built a space on the fly from your data which still resides in your domain. Either solution extols the virtues of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id101a8d28">Linked Data</a> i.e. the ability to explore relationships across data items with high degrees of serendipity (also colloquially known as: following-your-nose pattern in <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id14a3ff30">Semantic Web</a> circles).</blockquote> <blockquote> <a href="http://cb.semsol.org/" id="link-id182a0170">Bengee</a> posted a notice to the <a href="http://esw.w3.org/topic/SweoIG/TaskForces/CommunityProjects/LinkingOpenData" id="link-id131e8d10">Linking Open Data Community</a>'s public <a href="http://lists.w3.org/Archives/Public/public-lod/2008Jul/0110.html" id="link-id11dd0720">mailing list announcing his effort</a>. Bearing in mind the fact that we've been using <a href="http://www.openlinksw.com/blog/~kidehen/?id=1144" id="link-id117cf6e8">middleware to mesh the realms of Web 2.0 and the Linked Data Web</a> for a while, it was a no-brainer to knock something up based on the conceptual similarities between <a href="http://wikicompany.org/wiki/Main_Page" id="link-id13b87a68">Wikicompany</a> and CrunchBase. In a sense, a quadrant of orthogonality is what immediately came to mind re. Wikicompany, CrunchBase, Bengee's RDFization efforts, and ours.</blockquote> <blockquote>Bengee created an RDF based <a href="http://dbpedia.org/resource/Linked_Data" id="link-id133c8fc8">Linked Data</a> warehouse based on the data exposed by your API, which is exposed via the <a href="http://cb.semsol.org/" id="link-id1826f928">Semantic CrunchBase</a> <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id102d8890">data space</a>. In our case we've taken the "RDFization on the fly" approach which produces a transient <a href="http://dbpedia.org/resource/Linked_Data" id="link-id16a0b8d0">Linked Data</a> View of the CrunchBase data exposed by your APIs. Our approach is in line with our world view: all resources on the Web are data sources, and the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1668e6c8">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id188e7da0">Web</a> is about incorporating HTTP into the naming scheme of these data sources so that the conventional <a href="http://dbpedia.org/resource/Uniform_Resource_Locator" id="link-id13490710">URL</a> based hyperlinking mechanism can be used to access a structured description of a resource, which is then transmitted using a range negotiable representation formats. In addition, based on the fact that we house and publish a lot of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id169aa568">Linked Data</a> on the Web (e.g. <a href="http://dbpedia.org/resource/DBpedia" id="link-id10af10e8">DBpedia</a>, <a href="http://www.pingthesemanticweb.com/about/" id="link-id10a2b710">PingTheSemanticWeb</a>, and others), we've also automatically meshed Crunchbase data with related data in <a href="http://dbpedia.org/resource/DBpedia" id="link-id1403cd40">DBpedia</a> and Wikicompany data.</blockquote> <br /> <blockquote> <cite>CrunchBase: Do you know of any apps that are using CrunchBase Cartridge to enhance their functionality?</cite> </blockquote> <blockquote> <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id177d24c8">Me</a>: Yes, the <a href="http://ode.openlinksw.com" id="link-id10725ca0">OpenLink Data Explorer</a> which provides CrunchBase site visitors with the option to explore the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id17dedea8">Linked Data</a> in the CrunchBase <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id13f02a00">data space</a>. It also allows them to "Mesh" (rather than "Mash") CrunchBase data with other <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11fb3ba0">Linked Data</a> sources on the Web without writing a single line of code. </blockquote> <br /> <blockquote> <cite>CrunchBase: You have been immersed in the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id12e18a00">Semantic Web</a> movement for a while now. How did you first get interested in the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id15132110">Semantic Web</a>?</cite> </blockquote> <blockquote> <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id0xddaa9c8">Me</a>: We saw the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id188b3330">Semantic Web</a> as a vehicle for standardizing conceptual views of heterogeneous data sources via <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id10350978">context</a> lenses (URIs). In 1998 as part of our strategy to expand our business beyond the development and deployment of <a href="http://dbpedia.org/resource/Open_Database_Connectivity" id="link-id171d6798">ODBC</a>, <a href="http://dbpedia.org/resource/Java_Database_Connectivity" id="link-id138120a0">JDBC</a>, and OLE-DB data providers, we decided to build a <a href="http://dbpedia.org/resource/Virtual_Database" id="link-id13ea6618">Virtual Database</a> Engine (see: <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSHistory" id="link-id11a4fa30">Virtuoso History</a>), and in doing so we sought a standards based mechanism for the conceptual output of the <a href="http://dbpedia.org/resource/Federated_database_system" id="link-id101a1248">data virtualization</a> effort. As of the time of the <a href="http://www.w3.org/DesignIssues/Semantic.html" id="link-id18882cf8">seminal unveiling of the Semantic Web in 1998</a> we were clear about two things, in relation to the effects of the Web and <a href="http://dbpedia.org/resource/Internet" id="link-id12fa2c58">Internet</a> data management infrastructure inflections: 1) Existing DBMS technology had reached it limits 2) Web Servers would ultimately hit their functional limits. These fundamental realities compelled us to develop <a href="http://virtuoso.openlinksw.com" id="link-id102b09a0">Virtuoso</a> with an eye to leveraging the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id11984d98">Semantic Web</a> as a vehicle from completing its technical roadmap.</blockquote> <br /> <blockquote> <cite>CrunchBase: Can you put into laymanâs terms exactly what RDF and <a href="http://dbpedia.org/resource/SPARQL" id="link-id1066dcf0">SPARQL</a> are and why they are important? Do they only matter for developers or will they extend past developers at some point and be used by website visitors as well?</cite> </blockquote> <blockquote>Me: RDF (Resource Description Framework) is a Graph based Data Model that facilitates resource description using the <a href="http://www.eslincanada.com/englishlesson2.html" id="link-id178b94a8">Subject, Predicate, and Object principle</a>. Associated with the core data model, as part of the overall framework, are a number of markup languages for expressing your descriptions (just as you express presentation markup semantics in HTML or document structure semantics in XML) that include: <a href="http://dbpedia.org/resource/RDFa" id="link-id188db0a8">RDFa</a> (simple extension of HTML markup for embedding descriptions of things in a page), N3 (a human friendly markup for describing resources), RDF/XML (a machine friendly markup for describing resources).</blockquote> <blockquote> <a href="http://dbpedia.org/resource/SPARQL" id="link-id188c2030">SPARQL</a> is the query language associated with the RDF Data Model, just as <a href="http://dbpedia.org/resource/SQL" id="link-id13f0ffe0">SQL</a> is a query language associated with the Relational Database Model. Thus, when you have RDF based structured and <a href="http://dbpedia.org/resource/Linked_Data" id="link-id166874d0">linked data</a> on the Web, you can query against Web using <a href="http://dbpedia.org/resource/SPARQL" id="link-id1016cc98">SPARQL</a> just as you would against an <a href="http://dbpedia.org/resource/Oracle_Database" id="link-id101c9708">Oracle</a>/<a href="http://dbpedia.org/resource/SQL" id="link-id11cb0b18">SQL</a> Server/<a href="http://dbpedia.org/resource/IBM_DB2" id="link-id10760ec0">DB2</a>/<a href="http://dbpedia.org/resource/IBM_Informix" id="link-id1066c8c0">Informix</a>/<a href="http://dbpedia.org/resource/Ingres" id="link-id18894f40">Ingres</a>/<a href="http://dbpedia.org/resource/MySQL" id="link-iddc9ebb0">MySQL</a>/etc.. DBMS using <a href="http://dbpedia.org/resource/SQL" id="link-id1030d120">SQL</a>. That's it in a nutshell.</blockquote> <br /> <blockquote> <cite>CrunchBase: On your website you wrote that âRDF and <a href="http://dbpedia.org/resource/SPARQL" id="link-id168e9ad0">SPARQL</a> as productivity boosters in everyday web developmentâ. Can you elaborate on why you believe that to be true?</cite> </blockquote> <blockquote>Me: I think the ability to discern a formal description of anything via its discrete properties is of immense value re. productivity, especially when the capability in question results in a graph of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x179f6328">Linked Data</a> that isn't confined to a specific host operating system, database engine, application or service, programming language, or development framework. RDF <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> is about infrastructure for the true materialization of the "<a href="http://dbpedia.org/resource/Information" id="link-id13e475b8">Information</a> at Your Fingertips" vision of yore. Even though it's taken the emergence of RDF Linked Data to make the aforementioned vision tractable, the comprehension of the vision's intrinsic value have been clear for a very long time. Most organizations and/or individuals are quite familiar with the adage: <a href="http://dbpedia.org/resource/Knowledge" id="link-id13e38a30">Knowledge</a> is Power, well there isn't any <a href="http://dbpedia.org/resource/Knowledge" id="link-id188b7348">knowledge</a> without accessible <a href="http://dbpedia.org/resource/Information" id="link-id140415d0">Information</a>, and there isn't any accessible <a href="http://dbpedia.org/resource/Information" id="link-id11a976e8">Information</a> without accessible Data. The Web has always be grounded in accessibility to data (albeit via compound container documents called Web Pages).</blockquote> <blockquote>Bottom line, RDF based Linked Data is about Open <a href="http://dbpedia.org/resource/Reference_(computer_science)" id="link-id1206bfb8">Data access by reference</a> using URIs (HTTP based <a href="http://dbpedia.org/resource/Entity" id="link-idfaa6ce0">Entity</a> IDs / Data Object IDs / Data Source Names), and as I said earlier, the intrinsic value is pretty obvious bearing in mind the costs associated with integrating disparate and heterogeneous data sources -- across intranets, extranets, and the <a href="http://dbpedia.org/resource/Internet" id="link-id188ecc68">Internet</a>.</blockquote> <br /> <blockquote> <cite>CrunchBase: In his definition of Web 3.0, Nova Spivack proposes that the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id12e2d968">Semantic Web</a>, or Semanti<a href="http://dbpedia.org/resource/C_(programming_language)" id="link-id105744c0">c</a> Web technologies, will be force behind much of the innovation that will occur during Web 3.0. Do you agree with Nova Spivack? What role, if any, do you feel the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id13fa4218">Semantic Web</a> will play in Web 3.0?</cite> </blockquote> <blockquote>Me: I agree with Nova. But I see Web 3.0 as a phase within the <a href="http://dbpedia.org/resource/Semantic_Web" id="link-id188c9000">Semantic Web</a> innovation continuum. Web 3.0 exists because Web 2.0 exists. Both of these Web versions express usage and technology focus patterns. Web 2.0 is about the use of Open Source technologies to fashion Web Services that are ultimately used to drive proprietary Software as Service (SaaS) style solutions. Web 3.0 is about the use of "Smart Data Access" to fashion a new generation of Linked Data aware Web Services and solutions that exploit the federated nature of the Web to maximum effect; proprietary branding will simply be conveyed via quality of data (cleanliness, <a href="http://dbpedia.org/resource/Context_%28language_use%29" id="link-id188d2ef8">context</a> fidelity, and comprehension of privacy) exposed by URIs.</blockquote> <p>Here are some examples of the CrunchBase Linked Data <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id122756f8">Space</a>, as projected via our CruncBase Sponger Cartridge:</p> <ol> <li> <a href="http://demo.openlinksw.com/rdfbrowser2/?uri=http%3A%2F%2Fwww.crunchbase.com%2Fcompany%2Famazon" id="link-id13e0fd18">Amazon.com</a> </li> <li> <a href="http://demo.openlinksw.com/rdfbrowser2/?uri=http%3A%2F%2Fwww.crunchbase.com%2Fcompany%2Fmicrosoft" id="link-id13eef9e0">Microsoft</a> </li> <li> <a href="http://demo.openlinksw.com/rdfbrowser2/?uri=http%3A%2F%2Fwww.crunchbase.com%2Fcompany%2Fgoogle" id="link-id13fe47a0">Google</a> </li> <li> <a href="http://demo.openlinksw.com/rdfbrowser2/?uri=http%3A%2F%2Fwww.crunchbase.com%2Fcompany%2Fapple" id="link-id170c73b8">Apple</a> </li> </ol>
Comments about recent Semantic Gang Podcast
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2008-05-02#1357
2008-05-02T21:44:31Z
2008-05-05T20:06:42.000004-04:00
<p>After listening to the <a href="http://semanticgang.talis.com/2008/05/02/april-2008-the-semantic-web-gang-discuss-a-wikipedia-for-data/" id="link-id1089e218">latest Semantic Web Gang podcast</a>, I found myself agreeing with some of the points made by <a href="http://www.linkedin.com/in/iskold" id="link-id10b91e58">Alex Iskold</a>, specifically: </p> <ul>-- <a href="http://dbpedia.org/resource/Linked_Data" id="link-id106e24e0">Linked Data</a> does not implicitly imply making all your <a href="http://dbpedia.org/resource/Data" id="link-id17ab3d48">data</a> public</ul> <ul>-- <a href="http://dbpedia.org/resource/Linked_Data" id="link-id11fdcef0">Linked Data</a> principles benefit <a href="http://dbpedia.org/resource/Intranet" id="link-id109756e8">Intranet</a> and <a href="http://dbpedia.org/resource/Extranet" id="link-id1099cfd8">Extranet</a> style <a href="http://dbpedia.org/resource/Data" id="link-id10cd25b0">data</a> integration (trumps alternative <a href="http://dbpedia.org/resource/federated_database_system" id="link-id14f29940">distributed database</a> integration approaches any day)</ul> <ul>-- Business exploitation of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0xca51940">Linked Data</a> on the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> will certainly be driven by the correlation of opportunity costs (which is more than likely what Alex meant by "use cases") associated with the lack of URIs originating from the domain of a given business (Tom Heath: also effectively alluded to this via his <a href="http://dbpedia.org/resource/BBC" id="link-id16f33348">BBC</a> and <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id10decf38">URI</a> land grab anecdotes; same applies Georgi's examples)</ul> <ul>-- History is a great tutor, answers to many of today's problems always lie somewhere in plain sight of the past.</ul> <p>Of course, I also believe that <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> serves Web <a href="http://dbpedia.org/resource/Data" id="link-id0x1afebd58">Data</a> Integration across the <a href="http://dbpedia.org/resource/Internet" id="link-id10aa5668">Internet</a> very well too, and the fact that it will be beneficial to businesses in a big way. No individual or organization is an island, I think the <a href="http://dbpedia.org/resource/Internet" id="link-id0xb25fbd0">Internet</a> and Web have done a good job of demonstrating that thus far :-) We're all <a href="http://dbpedia.org/resource/Data">data</a> nodes in a <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id5d8a3a8">Giant Global Graph</a>.</p> <p> <a href="http://myopenlink.net/dataspace/person/danieljohnlewis#this" id="link-id17cac8a0">Daniel lewis</a> did shed light on the read-write aspects of the Linked Data <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id10be8590">Web</a>, which is actually very close to the callout for a Wikipedia for Data. <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id10a810c0">TimBL</a> has been working on this via <a href="http://dig.csail.mit.edu/2005/ajar/release/tabulator/0.8/tab.html" id="link-id184b7108">Tabulator</a> (see <a href="http://dig.csail.mit.edu/2007/tab/tutorial/editing.mov" id="link-id1416f1e8">Tabulator Editing Screencast</a>), <a href="http://bnode.org/about" id="link-id17e33750">Bengamin Nowack</a> also added <a href="http://arc.semsol.org/download/plugins/data_wiki" id="link-id1688cc40">similar functionality to ARC</a>, and of course we support the same <a href="http://dbpedia.org/resource/SPARQL" id="link-id10bff7c8">SPARQL</a> UPDATE into an <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id168ace08">RDF</a> <a href="http://dbpedia.org/resource/Information" id="link-id10641878">information</a> resource via the <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xddb5240">RDF</a> Sink feature of our WebDAV and <a href="http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/OdsBriefcase" id="link-id0x11199310">ODS</a>-Briefcase implementations.</p>
Linked Data and Information Architecture
http://www.openlinksw.com/blog/vdb/blog/?date=2008-04-29#1350
2008-04-29T14:37:22Z
2008-04-29T17:18:21.000048-04:00
<div> <div style="display:none;">Linked Data and Information Architecture</div> <p>We had a workshop on <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1437ac70">Linked Open Data</a> (<a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x1315f788">LOD</a>) last week in <a href="http://www2008.org/" id="link-id0x13737468">Beijing</a>. You can see the papers in <a href="http://events.linkeddata.org/ldow2008/#program" id="link-id10651ab8">the program</a>. The event was a success with plenty of good talks and animated conversation. I will not go into every paper here but will comment a little on the conversation and draw some technology requirements going forward.</p> <p>Tim Berners-Lee showed a read-write version of <a href="http://dig.csail.mit.edu/2005/ajar/release/tabulator/0.8/tab.html" id="link-id0x15633520">Tabulator</a>. This raises the question of updating on the <a href="http://dbpedia.org/resource/Data" id="link-id0x1350a178">Data</a> Web. The consensus was that one could assert what one wanted in one's own space but that others' spaces would be read-only. What spaces one considered relevant would be the user's or developer's business, as in the document web.</p> <p>It seems to me that a significant use case of LOD is an open-web situation where the user picks a broad read-only "data wallpaper" or backdrop of assertions, and then uses this combined with a much smaller, local, writable data set. This is certainly the case when editing data for publishing, as in Tim's demo. This will also be the case when developing mesh-ups combining multiple distinct data sets bound together by sets of SameAs assertions, for example. Questions like, "What is the minimum subset of n data sets needed for deriving the result?" will be common. This will also be the case in applications using proprietary data combined with open data.</p> <p>This means that databases will have to deal with queries that specify large lists of included graphs, all graphs in the store or all graphs with an exclusion list. All this is quite possible but again should be considered when architecting systems for an open <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0xa27bae8">linked data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x155c3f18">web</a>.</p> <p>"There is data but what can we really do with it? How far can we trust it, and what can we confidently decide based on it?"</p> <p>As an answer to this question, <a href="http://zitgist.com/about/" id="link-id0xd447580">Zitgist</a> has compiled the <a href="http://umbel.org/about/" id="link-id0x14735008">UMBEL</a> taxonomy using <a href="http://dbpedia.org/resource/SKOS" id="link-id0x15ab1c48">SKOS</a>. This draws on Wikipedia, Open CYC, Wordnet, and <a href="http://www.mpi-inf.mpg.de/~suchanek/downloads/yago/" id="link-id0x15d5aa88">YAGO</a>, hence the acronym WOWY. UMBEL is both a taxononmy and a set of instance data, containing a large set of <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id0x9fe45d98">named entities</a>, including persons, organizations, geopolitical entities, and so forth. By extracting references to this set of named entities from documents and correlating this to the taxonomy, one gets a good idea of what a document (or part thereof) is about.</p> <p>Kingsley presented this in the Zitgist demo. This is our answer to the criticism about <a href="http://dbpedia.org/resource/DBpedia" id="link-id0xa1920800">DBpedia</a> having errors in classification. DBpedia, as a bootstrap stage, is about giving names to all things. Subsequent efforts like UMBEL are about refining the relationships.</p> <p>"Should there be a global <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0x12cd5290">URI</a> dictionary?"</p> <p>There was a talk by Paolo Bouquet about <a href="http://dbpedia.org/resource/Entity" id="link-id0x12d03400">Entity</a> Name System, a a sort of data DNS, with the purpose of associating some description and rough classification to URIs. This would allow discovering URIs for reuse. I'd say that this is good if it can cut down on the SameAs proliferation and if this can be widely distributed and replicated for resilience, <i>Ã la</i> DNS. On the other hand, it was pointed out that this was not quite in the LOD spirit, where parties would mint their own dereferenceable URIs, in their own domains. We'll see.</p> <p>"What to do when identity expires?"</p> <p>Giovanni of Sindice said that a document should be removed from search if it was no longer available. Kingsley pointed out that resilience of reference requires some way to recover data. The data web cannot be less resilient than the document web, and there is a point to having access to history. He recommended hooking up with the <a href="http://dbpedia.org/resource/Internet" id="link-id0x143e4130">Internet</a> Archive, since they make long term persistence their business. In this way, if an application depends on data, and the URIs on which it depends are no longer dereferenceable or or provide content from a new owner of the domain, those who need the old version can still get it and host it themselves.</p> <p>It is increasingly clear that OWL SameAs is both the blessing and bane of linked data. We can easily have tens of URIs for the same thing, especially with people. Still, these should be considered the same.</p> <p>Returning every synonym in a query answer hardly makes sense but accepting them as input seems almost necessary. This is what we do with <a href="http://virtuoso.openlinksw.com" id="link-id0x15a2a930">Virtuoso</a>'s SameAs support. Even so, this can easily double query times even when there are no synonyms.</p> <p>Be that as it may, SameAs is here to stay; just consider the mapping of DBpedia to Geonames, for example.</p> <p>Also, making aberrant SameAs statements can completely poison a data set and lead to absurd query results. Hence choosing which SameAs assertions from which source will be considered seems necessary. In an open web scenario, this leads inevitably to multi-graph queries that can be complex to write with regular <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x12bb8ce8">SPARQL</a>. By extension, it seems that a good query would also include the graphs actually used for deriving each result row. This is of course possible but has some implications on how databases should be organized.</p> <p>Yves Raymond gave a talk about deriving identity between Musicbrainz and Jamendo. I see the issue as a core question of linked data in general. The algorithm Yves presented started with attribute value similarities and then followed related entities. Artists would be the same if they had similar names and similar names of albums with similar song titles, for example. We can find the same basic question in any analysis, for example, looking at how news reporting differs between media, supposing there is adequate entity extraction.</p> <p>There is basic graph diffing in <a href="http://data.semanticweb.org/conference/iswc-aswc/2007/tracks/research/papers/533/html" id="link-id0x153c1fa8">RDFSync</a>, for example. But here we are expanding the context significantly. We will traverse references to some depth, allow similarity matches, SameAs, and so forth. Having presumed identity of two URIs, we can then look at the difference in their environment to produce a human readable summary. This could then be evaluated for purposes of analysis or of combining content.</p> <p>At first sight, these algorithms seem well parallelizable, as long as all threads have access to all data. For scaling, this means a probably message-bound distributed algorithm. This is something to look into for the next stage of linked data.</p> <p>Some inference is needed, but if everybody has their own choice of data sets to query, then everybody would also have their own entailed triples. This will make for an explosion of entailed graphs if forward chaining is used. Forward chaining is very nice because it keeps queries simple and easy to optimize. With Virtuoso, we still favor backward chaining since we expect a great diversity of graph combinations and near infinite volume in the open web scenario. With private repositories of slowly changing data put together for a special application, the situation is different.</p> <p>In conclusion, we have a real LOD movement with actual momentum and a good idea of what to do next. The next step is promoting this to the broader community, starting with <a href="http://www.linkeddataplanet.com/" id="link-id0x155d1d00">Linked Data Planet</a> in New York in June.</p> </div>
Linked Data and Information Architecture
http://www.openlinksw.com/weblog/oerling/?date=2008-04-29#1347
2008-04-29T12:08:24Z
2008-04-29T17:18:15.000009-04:00
<p>We had a workshop on <a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0x9e7e5098">Linked Open Data</a> (<a href="http://community.linkeddata.org/dataspace/organization/lod#this" id="link-id0xb9e86c8">LOD</a>) last week in <a href="http://www2008.org/" id="link-id0x131a72a0">Beijing</a>. You can see the papers in <a href="http://events.linkeddata.org/ldow2008/#program" id="link-id10651ab8">the program</a>. The event was a success with plenty of good talks and animated conversation. I will not go into every paper here but will comment a little on the conversation and draw some technology requirements going forward.</p> <p>Tim Berners-Lee showed a read-write version of <a href="http://dig.csail.mit.edu/2005/ajar/release/tabulator/0.8/tab.html" id="link-id0x9da015a0">Tabulator</a>. This raises the question of updating on the <a href="http://dbpedia.org/resource/Data" id="link-id0x14813578">Data</a> Web. The consensus was that one could assert what one wanted in one's own space but that others' spaces would be read-only. What spaces one considered relevant would be the user's or developer's business, as in the document web.</p> <p>It seems to me that a significant use case of LOD is an open-web situation where the user picks a broad read-only "data wallpaper" or backdrop of assertions, and then uses this combined with a much smaller, local, writable data set. This is certainly the case when editing data for publishing, as in Tim's demo. This will also be the case when developing mesh-ups combining multiple distinct data sets bound together by sets of SameAs assertions, for example. Questions like, "What is the minimum subset of n data sets needed for deriving the result?" will be common. This will also be the case in applications using proprietary data combined with open data.</p> <p>This means that databases will have to deal with queries that specify large lists of included graphs, all graphs in the store or all graphs with an exclusion list. All this is quite possible but again should be considered when architecting systems for an open <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x9f1a1d8">linked data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0xa5a3e1b0">web</a>.</p> <p>"There is data but what can we really do with it? How far can we trust it, and what can we confidently decide based on it?"</p> <p>As an answer to this question, <a href="http://zitgist.com/about/" id="link-id0xad44dc0">Zitgist</a> has compiled the <a href="http://umbel.org/about/" id="link-id0x9ebde358">UMBEL</a> taxonomy using <a href="http://dbpedia.org/resource/SKOS" id="link-id0xa04a85c0">SKOS</a>. This draws on Wikipedia, Open CYC, Wordnet, and <a href="http://www.mpi-inf.mpg.de/~suchanek/downloads/yago/" id="link-id0x9fdd9018">YAGO</a>, hence the acronym WOWY. UMBEL is both a taxononmy and a set of instance data, containing a large set of <a href="http://dbpedia.org/resource/Named_entity_recognition" id="link-id0xa0b9cb70">named entities</a>, including persons, organizations, geopolitical entities, and so forth. By extracting references to this set of named entities from documents and correlating this to the taxonomy, one gets a good idea of what a document (or part thereof) is about.</p> <p>Kingsley presented this in the Zitgist demo. This is our answer to the criticism about <a href="http://dbpedia.org/resource/DBpedia" id="link-id0xdc5d940">DBpedia</a> having errors in classification. DBpedia, as a bootstrap stage, is about giving names to all things. Subsequent efforts like UMBEL are about refining the relationships.</p> <p>"Should there be a global <a href="http://dbpedia.org/resource/Uniform_Resource_Identifier" id="link-id0xa0aa2cc0">URI</a> dictionary?"</p> <p>There was a talk by Paolo Bouquet about <a href="http://dbpedia.org/resource/Entity" id="link-id0x9e6f4b28">Entity</a> Name System, a a sort of data DNS, with the purpose of associating some description and rough classification to URIs. This would allow discovering URIs for reuse. I'd say that this is good if it can cut down on the SameAs proliferation and if this can be widely distributed and replicated for resilience, <i>Ã la</i> DNS. On the other hand, it was pointed out that this was not quite in the LOD spirit, where parties would mint their own dereferenceable URIs, in their own domains. We'll see.</p> <p>"What to do when identity expires?"</p> <p>Giovanni of Sindice said that a document should be removed from search if it was no longer available. Kingsley pointed out that resilience of reference requires some way to recover data. The data web cannot be less resilient than the document web, and there is a point to having access to history. He recommended hooking up with the <a href="http://dbpedia.org/resource/Internet" id="link-id0xcc203f8">Internet</a> Archive, since they make long term persistence their business. In this way, if an application depends on data, and the URIs on which it depends are no longer dereferenceable or or provide content from a new owner of the domain, those who need the old version can still get it and host it themselves.</p> <p>It is increasingly clear that OWL SameAs is both the blessing and bane of linked data. We can easily have tens of URIs for the same thing, especially with people. Still, these should be considered the same.</p> <p>Returning every synonym in a query answer hardly makes sense but accepting them as input seems almost necessary. This is what we do with <a href="http://virtuoso.openlinksw.com" id="link-id0xa0d28c98">Virtuoso</a>'s SameAs support. Even so, this can easily double query times even when there are no synonyms.</p> <p>Be that as it may, SameAs is here to stay; just consider the mapping of DBpedia to Geonames, for example.</p> <p>Also, making aberrant SameAs statements can completely poison a data set and lead to absurd query results. Hence choosing which SameAs assertions from which source will be considered seems necessary. In an open web scenario, this leads inevitably to multi-graph queries that can be complex to write with regular <a href="http://dbpedia.org/resource/SPARQL" id="link-id0xc89a768">SPARQL</a>. By extension, it seems that a good query would also include the graphs actually used for deriving each result row. This is of course possible but has some implications on how databases should be organized.</p> <p>Yves Raymond gave a talk about deriving identity between Musicbrainz and Jamendo. I see the issue as a core question of linked data in general. The algorithm Yves presented started with attribute value similarities and then followed related entities. Artists would be the same if they had similar names and similar names of albums with similar song titles, for example. We can find the same basic question in any analysis, for example, looking at how news reporting differs between media, supposing there is adequate entity extraction.</p> <p>There is basic graph diffing in <a href="http://data.semanticweb.org/conference/iswc-aswc/2007/tracks/research/papers/533/html" id="link-id0x9fe62620">RDFSync</a>, for example. But here we are expanding the context significantly. We will traverse references to some depth, allow similarity matches, SameAs, and so forth. Having presumed identity of two URIs, we can then look at the difference in their environment to produce a human readable summary. This could then be evaluated for purposes of analysis or of combining content.</p> <p>At first sight, these algorithms seem well parallelizable, as long as all threads have access to all data. For scaling, this means a probably message-bound distributed algorithm. This is something to look into for the next stage of linked data.</p> <p>Some inference is needed, but if everybody has their own choice of data sets to query, then everybody would also have their own entailed triples. This will make for an explosion of entailed graphs if forward chaining is used. Forward chaining is very nice because it keeps queries simple and easy to optimize. With Virtuoso, we still favor backward chaining since we expect a great diversity of graph combinations and near infinite volume in the open web scenario. With private repositories of slowly changing data put together for a special application, the situation is different.</p> <p>In conclusion, we have a real LOD movement with actual momentum and a good idea of what to do next. The next step is promoting this to the broader community, starting with <a href="http://www.linkeddataplanet.com/" id="link-id0xa16319a8">Linked Data Planet</a> in New York in June.</p>
Linked Data Illustrated and a Virtuoso Functionality Reminder
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2008-04-28#1342
2008-04-28T17:32:47Z
2008-04-28T14:47:06.000001-04:00
<a href="http://myopenlink.net/dataspace/person/danieljohnlewis#this" id="link-id156ceb30">Daniel Lewis</a> has put together a nice <a href="http://vanirsystems.com/danielsblog/2008/04/27/linked-data-the-role-of-the-data-server/" id="link-id10456040">collection of Linked Data related posts</a> that illustrate the fundamentals of the <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1033f6f0">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id106fa168">Web</a> and the vital role that <a href="http://virtuoso.openlinksw.com" id="link-id10141c20">Virtuoso</a> plays as a deployment platform. Remember, <a href="http://virtuoso.openlinksw.com" id="link-id10301e38">Virtuoso</a> was architected in 1998 (see <a href="http://virtuoso.openlinksw.com/wiki/main/Main/VOSHistory" id="link-id10c44088">Virtuoso History</a>) in anticipation of the eventual <a href="http://dbpedia.org/resource/Internet" id="link-id1383a1e8">Internet</a>, <a href="http://dbpedia.org/resource/Intranet" id="link-id1028e770">Intranet</a>, and <a href="http://dbpedia.org/resource/Extranet" id="link-id14b07b40">Extranet</a> level requirements for a different kind of Server. At the time of <a href="http://virtuoso.openlinksw.com" id="link-id14ad24a8">Virtuoso</a>'s inception, many thought our desire to build a multi-protocol, multi-model, and multi-purpose, virtual and native <a href="http://dbpedia.org/resource/Data" id="link-id108dac48">data</a> server was sheer craziness, but we pressed on (courtesy of our vision and technical capabilities). Today, we have a very sophisticated <a href="http://dbpedia.org/resource/Virtuoso_Universal_Server" id="link-id14a65d48">Universal Server</a> Platform (in Open Source and Commercial forms) that is naturally equipped to do the following via very simple interfaces: <ul> - Produce <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id11fb1170">RDF</a> <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10871da8">Linked Data</a> from non <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id156ec3d0">RDF</a> <a href="http://dbpedia.org/resource/Data" id="link-id10f0ca38">Data</a> Sources (Heterogeneous <a href="http://dbpedia.org/resource/SQL" id="link-id15133078">SQL</a>, XML, <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> Services)</ul> <ul> - Provide highly scalable <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id10585940">RDF</a> <a href="http://dbpedia.org/resource/Data" id="link-id15151e10">Data</a> Management via a Quad Store (<a href="http://dbpedia.org/resource/DBpedia" id="link-id1530d640">DBpedia</a> is an example of a live demonstration)</ul> <ul> - Sophisticated Deployment of <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10141c80">Linked Data</a> that exploits the power of <a href="http://dbpedia.org/resource/SPARQL" id="link-id1064fa18">SPARQL</a> </ul> <ul> - Powerful WebDAV innovations that simplify read-write mode interaction with <a href="http://dbpedia.org/resource/Linked_Data" id="link-id1396ff68">Linked Data</a> </ul> <ul> - Use Web <a href="http://dbpedia.org/resource/Federated_database_system" id="link-id108256e8">Data Virtualization</a> to address the pain and frustration associated with Web <a href="http://dbpedia.org/resource/Data" id="link-id147e65f8">Data</a> Silos (e.g. <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-idffaf078">OpenLink Data Spaces</a> layer stop <a href="http://virtuoso.openlinksw.com" id="link-id14ae8fe8">Virtuoso</a> that delivers <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id0xa0fb5e40">Personal Data Spaces</a> / Unified Storage in the Clouds) </ul> <ul> - Deliver a <a href="http://dbpedia.org/resource/Linked_Data" id="link-id10869700">Linked Data</a> development and deployment platform to .<a href="http://dbpedia.org/resource/.NET_Framework" id="link-id1514cac0">NET</a> (<a href="http://dbpedia.org/resource/Visual_Basic" id="link-id10c107a8">VB</a>, <a href="http://dbpedia.org/resource/C_(programming_language)" id="link-id101f3c68">C</a>#) , Java, <a href="http://dbpedia.org/resource/PHP" id="link-id106e4710">PHP</a>, <a href="http://dbpedia.org/resource/Ruby_programming_language" id="link-id10277448">Ruby</a>, <a href="http://dbpedia.org/resource/Perl" id="link-id10a75748">Perl</a>, <a href="http://dbpedia.org/resource/Python_programming_language" id="link-id12fdb118">Python</a>, '<a href="http://dbpedia.org/resource/C_(programming_language)" id="link-id10c9d9e0">C</a>', <a href="http://dbpedia.org/resource/C%2B%2B" id="link-id10392400">C++</a>, and other developers </ul> <ul>- More...</ul>
Semantic Web Patterns: A Guide to Semantic Technologies (Update 2)
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2008-03-27#1329
2008-03-27T00:08:13Z
2008-07-16T21:43:36-04:00
<p>For all the one-way feed consumers and aggregators, and readers of the original post, here is a variant equipped hyperlinked phrases as opposed to words. As I stated in the prior post, the post (like most of my posts) was part experiment / dog-fodding of automatic tagging and hyper-linking functionality in <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id0x194f56f0">OpenLink Data Spaces</a>. </p> <p> <a href="http://www.readwriteweb.com" id="link-id0x1bddde00">ReadWriteWeb</a> via <a href="http://alexiskold.wordpress.com/" id="link-id154ae848">Alex Iskold's post</a> have delivered another iteration of their "Guide to Semantic Technologies". </p> <p>If you look at the title of this post (and <a href="http://feeds.feedburner.com/%7Er/readwriteweb/%7E3/257943334/semantic_web_patterns.php" id="link-id10a9a900">their article</a>) they seem to be accurately providing a guide to Semantic Technologies, so no qualms there. If on the other hand, this is supposed to he a guide to the "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0x15ccef28">Semantic Web</a>" as prescribed by <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id0xb94a2d40">TimBL</a> then they are completely missing the essence of the whole subject, and demonstrably so I may add, since the entities: "<a href="http://www.readwriteweb.com">ReadWriteWeb</a>" and "<a href="http://www.linkedin.com/in/iskold" id="link-id0x19960308">Alex Iskold</a>" are only describable today via the attributes of the documents they publish i.e their respective blogs and hosted <a href="http://dbpedia.org/resource/Blog" id="link-id0x1a719968">blog</a> posts.</p> <blockquote> <p>Preoccupation with Literal objects as describe above, implies we can only take what "ReadWriteWeb" and "<a href="http://www.linkedin.com/in/iskold">Alex Iskold</a>" say "Literally" (<a href="http://dbpedia/resource/Grep" id="link-id0xbc8568f8">grep</a>, <a href="http://dbpedia.org/resource/regular_expression" id="link-id0x1d915e70">regex</a>, and <a href="http://dbpedia.org/resource/XPath" id="link-id0xbc617820">XPath</a>/<a href="http://dbpedia.org/resource/XQuery" id="link-id0x150e1c50">Xquery</a> are the only tools for searching deeper in this Literal realm), we have no sense of what makes them tick or where they come from, no history (bar "About Page" blurb), no <a href="http://dbpedia.org/resource/Data">data</a> connections beyond anchored text (more pointers to opaque data sources) in post and blogrolls. The only connection between this post and them is the my deliberate use of the same literal text in the Title of this post.</p> </blockquote> <p><a href="http://www.w3.org/People/Berners-Lee/card#i">TimBL</a>'s vision as espoused via the "<a href="http://dbpedia.org/resource/Semantic_Web">Semantic Web</a>" vision is about the production, consumption, and sharing of Data Objects via HTTP based Identifiers called URIs/IRIs (<a href="http://dbpedia.org/resource/Linked_Data" id="link-id0xb867ced0">Hyperdata</a> Links / <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x3c8f438">Linked Data</a>). It's how we use the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> as a <a href="http://dbpedia.org/resource/federated_database_system" id="link-id0xbcb04f20">Distributed Database</a> where (as <a href="http://www.cs.umd.edu/~hendler/2003/foaf.rdf#jhendler" id="link-id0xb8595f18">Jim Hendler</a> once stated with immense clarity): I can point to records (<a href="http://dbpedia.org/resource/Entity" id="link-id0xbc9c8ab8">entity</a> instances) in your database (aka <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id0x3b911c0">Data Space</a>) from mine. Which is to say that if we can all point to data entities/objects (not just data entities of type "Document") using these Location, Value, and Structure independent Object Identifiers (courtesy of HTTP) we end up with a much more powerful Web, and one that is closer to the "Federated and Open" nature of the Web.</p> <p>As I stated in a prior post, if you or your platform of choice aren't producing de-referencable URIs for your data objects, you may be Semantic (this data model predates the Web), but there is no "<a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id0xbcb968b0">World Wide Web</a>" in what you are doing.</p> <h2>What are the Benefits of the Semantic Web?</h2> <ul> <strong>Consumer</strong> - "Discovery of relevant things" and be being "Discovered by relevant things" (people, places, events, and other things)</ul> <ul> <strong>Enterprise</strong> - ditto plus the addition of enterprise domain specific things such as market opportunities, product portfolios, human resources, partners, customers, competitors, co-opetitors, acquisition targets, new regulation etc..)</ul> <h2>Simple demo:</h2> <blockquote> <p>I am a <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id0x150661b0">Kingsley Idehen</a>, a Person who authors <a href="http://www.openlinksw.com/blog/~kidehen" id="link-id0x3b956d0">this weblog</a>. I also share bookmarks gathered over the years across an array of subjects via <a href="http://myopenlink.net/dataspace/kidehen/bookmark/KingsleyBookmarks" id="link-id0x164fecb0">my bookmark data space</a>. I also subscribe to a number of RSS/Atom/RDF feeds, which I share via my feeds subscription data <a href="http://en.wikipedia.org/wiki/Data_Spaces">space</a>. Of course, all of these data sources have Tags which are collectively exposed via my <a href="http://myopenlink.net/dataspace/kidehen/weblog/MyBlogDataSpace/tagcloud" id="link-id0x15188c50">weblog tag-cloud</a>, feeds subscriptions <a href="http://dbpedia.org/resource/Tag" id="link-id0x5f38b98">tag</a>-cloud, and <a href="http://myopenlink.net/dataspace/kidehen/bookmark/KingsleyBookmarks/tagcloud" id="link-id0xb93c2a50">bookmarks tag-cloud</a> data spaces.</p> <p>As I don't like repeating myself, and I hate wasting my time or the time of others, I simply share <a href="http://myopenlink.net/dataspace/kidehen" id="link-id0x3aeba98">my Data Space</a> (a collection of all of my purpose specific data spaces) via the Web so that others (friends, family, employees, partners, customers, project collaborators, competitors, co-opetitors etc.) can can intentionally or serendipitously discover relevant data en route to creating new <a href="http://dbpedia.org/resource/Information" id="link-id0x14e35d78">information</a> (perspectives) that is hopefully exposed others via the Web.</p> </blockquote> <p>Bottom-line, the Semantic Web is about adding the missing "Open Data Access & Connectivity" feature to the current Document Web (we have to beyond <a href="http://dbpedia.org/resource/regular_expression">regex</a>, <a href="http://dbpedia/resource/Grep">grep</a>, <a href="http://dbpedia.org/resource/XPath">xpath</a>, <a href="http://dbpedia.org/resource/XQuery">xquery</a>, <a href="http://dbpedia.org/resource/Full_text_search" id="link-id0x1c1bf9c8">full text search</a>, and other literal scrapping approaches). The <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x14c9e0e8">Web</a> of de-referencable data object URIs is the critical foundation layer that makes this feasible.</p> <p> Remember, It's not about "Applications" it's about Data and actually freeing Data from the "tyranny of Applications". Unfortunately, application inadvertently always create silos (esp. on the Web) since <a href="http://dbpedia.org/resource/Entity">entity</a> data modeling, open data access, and other database technology realm matters, remain of secondary interest to many application developers.</p> <p>Final comment, RDF facilitates Linked Data on the Web, but all RDF isn't endowed with de-referencable URIs (a major source of confusion and misunderstanding). Thus, you can have RDF Data Source Providers that simply project RDF data silos via Web Services APIs if RDF output emanating from a Web Service doesn't provide out-bound pathways to other data via de-referencable URIs. Of course the same also applies to Widgets that present you with all the things they've discovered without exposing de-referencable URIs for each item.</p> <p>BTW - my final comments above aren't in anyway incongruent with devising successful business models for the Web. As you may or may not know, OpenLink is not only a major platform provider for the Semantic Web (expressed in our UDA, <a href="http://virtuoso.openlinksw.com" id="link-id0xb919b098">Virtuoso</a>, OpenLink Data Spaces, and OAT products), we are also actively seeding Semantic Web (tribe: Linked Data of course) startups. For instance, <a href="http://zitgist.com/about/" id="link-id0x1481b218">Zitgist</a>, which now has <a href="http://community.linkeddata.org/dataspace/person/mkbergman#this" id="link-id0xb869bb18">Mike Bergman</a> as it's CEO alongside <a href="http://fgiasson.com/me/" id="link-id0x1d18fe50">Frederick Giasson</a> as CTO. Of course, I cannot do <a href="http://zitgist.com/about/">Zitgist</a> justice via a footnote in a <a href="http://dbpedia.org/resource/Blog">blog</a> post, so I will expand further in a separate post.</p> <h2>Additional <a href="http://dbpedia.org/resource/Information">information</a> about this blog post: </h2> <ol> <li> I didn't spent hours looking for URIs used in my hyperlinks</li> <li> The post is best viewed via an RDF Linked Data aware user agents (<a href="http://demo.openlinksw.com/rdfbrowser" id="link-id0x19af3468">OpenLink RDF Browser</a>, Zitgist <a href="http://dataviewer.zitgist.com" id="link-id0x13b17138">Data Viewer</a>, <a href="http://www4.wiwiss.fu-berlin.de/rdf_browser" id="link-id0xbc8579e0">DISCO Hyperdata Browser</a>, <a href="http://dig.csail.mit.edu/2005/ajar/release/tabulator/0.8/tab.html" id="link-id0x18ad0ec8">Tabulator</a>).</li> </ol>
Semantic Web Patterns: A Guide to Semantic Technologies (Update 1)
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2008-03-26#1328
2008-03-26T22:44:00Z
2008-07-16T21:43:04-04:00
<p> <a href="http://www.readwriteweb.com" id="link-id11846528">ReadWriteWeb</a> via <a href="http://alexiskold.wordpress.com/" id="link-id154ae848">Alex Iskold</a> have delivered another iteration of their "Guide to Semantic Technologies". </p> <p>If you look at the title of this post (and <a href="http://feeds.feedburner.com/%7Er/readwriteweb/%7E3/257943334/semantic_web_patterns.php" id="link-id10a9a900">their article</a>) they seem to be accurately providing a guide to Semantic Technologies, so no qualms there. If on the other hand, this is supposed to he a guide to the "<a href="http://dbpedia.org/resource/Semantic_Web" id="link-id0xbcb19320">Semantic Web</a>" as prescribed by <a href="http://www.w3.org/People/Berners-Lee/card#i" id="link-id0xb8725878">TimBL</a> then they are completely missing the essence of the whole subject, and demonstrably so I may add, since the entities: "<a href="http://www.readwriteweb.com" id="link-id0x16804040">ReadWriteWeb</a>" and "<a href="http://www.linkedin.com/in/iskold" id="link-id0x13f08538">Alex Iskold</a>" are only describable today via the attributes of the documents they publish i.e their respective blogs and hosted <a href="http://dbpedia.org/resource/Blog" id="link-id0x1850ca98">blog</a> posts. </p> <blockquote> <p>Preoccupation with Literal objects as describe above, implies we can only take what "<a href="http://www.readwriteweb.com">ReadWriteWeb</a>" and "<a href="http://www.linkedin.com/in/iskold">Alex Iskold</a>" say "Literally" (<a href="http://dbpedia/resource/Grep" id="link-id0xb95a6a40">grep</a>, <a href="http://dbpedia.org/resource/regular_expression" id="link-id0x1a719968">regex</a>, and <a href="http://dbpedia.org/resource/XPath" id="link-id0xb89d78b8">XPath</a>/<a href="http://dbpedia.org/resource/XQuery" id="link-id0x1bddde00">Xquery</a> are the only tools for searching deeper in this Literal realm), we have no sense of what makes them tick or where they come from, no history (bar "About Page" blurb), no <a href="http://dbpedia.org/resource/Data">data</a> connections beyond anchored text (more pointers to opaque data sources) in post and blogrolls. The only connection between this post and them is the my deliberate use of the same literal text in the Title of this post.</p> </blockquote> <p><a href="http://www.w3.org/People/Berners-Lee/card#i">TimBL</a>'s vision as espoused via the "<a href="http://dbpedia.org/resource/Semantic_Web">Semantic Web</a>" vision is about the production, consumption, and sharing of Data Objects via HTTP based Identifiers called URIs/IRIs (<a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x150e7be0">Hyperdata</a> Links / <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x18e50818">Linked Data</a>). It's how we use the <a href="http://dbpedia.org/resource/World_Wide_Web">Web</a> as a <a href="http://dbpedia.org/resource/federated_database_system" id="link-id0x194f56f0">Distributed Database</a> where (as <a href="http://www.cs.umd.edu/~hendler/2003/foaf.rdf#jhendler" id="link-id0x17043b38">Jim Hendler</a> once stated with immense clarity): I can point to records (<a href="http://dbpedia.org/resource/Entity" id="link-id0x1476f788">entity</a> instances) in your database (aka <a href="http://en.wikipedia.org/wiki/Data_Spaces" id="link-id0x2621140">Data Space</a>) from mine. Which is to say that if we can all point to data entities/objects (not just data entities of type "Document") using these Location, Value, and Structure independent Object Identifiers (courtesy of HTTP) we end up with a much more powerful Web, and one that is closer to the "Federated and Open" nature of the Web.</p> <p>As I stated in a prior post, if you or your platform of choice aren't producing de-referencable URIs for your data objects, you may be Semantic (this data model predates the Web), but there is no "<a href="http://dbpedia.org/resource/World_Wide_Web" id="link-id0xb860eec8">World Wide Web</a>" in what you are doing.</p> <h2>What are the Benefits of the Semantic Web?</h2> <ul> <strong>Consumer</strong> - "Discovery of relevant things" and be being "Discovered by relevant things" (people, places, events, and other things)</ul> <ul> <strong>Enterprise</strong> - ditto plus the addition of enterprise domain specific things such as market opportunities, product portfolios, human resources, partners, customers, competitors, co-opetitors, acquisition targets, new regulation etc..)</ul> <h2>Simple demo:</h2> <blockquote> <p>I am a <a href="http://myopenlink.net/dataspace/person/kidehen#this" id="link-id0x15394798">Kingsley Idehen</a>, a Person who authors <a href="http://www.openlinksw.com/blog/~kidehen" id="link-id0x2556670">this weblog</a>. I also share bookmarks gathered over the years across an array of subjects via <a href="http://myopenlink.net/dataspace/kidehen/bookmark/KingsleyBookmarks" id="link-id0x142eaa10">my bookmark data space</a>. I also subscribe to a number of RSS/Atom/RDF feeds, which I share via my feeds subscription data <a href="http://en.wikipedia.org/wiki/Data_Spaces">space</a>. Of course, all of these data sources have Tags which are collectively exposed via my <a href="http://myopenlink.net/dataspace/kidehen/weblog/MyBlogDataSpace/tagcloud" id="link-id0x140b8050">weblog tag-cloud</a>, feeds subscriptions <a href="http://dbpedia.org/resource/Tag" id="link-id0x15158d60">tag</a>-cloud, and <a href="http://myopenlink.net/dataspace/kidehen/bookmark/KingsleyBookmarks/tagcloud" id="link-id0xb8652490">bookmarks tag-cloud</a> data spaces.</p> <p>As I don't like repeating myself, and I hate wasting my time or the time of others, I simply share <a href="http://myopenlink.net/dataspace/kidehen" id="link-id0x13b63208">my Data Space</a> (a collection of all of my purpose specific data spaces) via the Web so that others (friends, family, employees, partners, customers, project collaborators, competitors, co-opetitors etc.) can can intentionally or serendipitously discover relevant data en route to creating new <a href="http://dbpedia.org/resource/Information" id="link-id0x14365150">information</a> (perspectives) that is hopefully exposed others via the Web.</p> </blockquote> <p>Bottom-line, the Semantic Web is about adding the missing "Open Data Access & Connectivity" feature to the current Document Web (we have to beyond <a href="http://dbpedia.org/resource/regular_expression">regex</a>, <a href="http://dbpedia/resource/Grep">grep</a>, <a href="http://dbpedia.org/resource/XPath">xpath</a>, <a href="http://dbpedia.org/resource/XQuery">xquery</a>, <a href="http://dbpedia.org/resource/Full_text_search" id="link-id0x15ccef28">full text search</a>, and other literal scrapping approaches). The <a href="http://dbpedia.org/resource/Linked_Data">Linked Data</a> <a href="http://dbpedia.org/resource/Giant_Global_Graph" id="link-id0x1a2810b8">Web</a> of de-referencable data object URIs is the critical foundation layer that makes this feasible.</p> <p> Remember, It's not about "Applications" it's about Data and actually freeing Data from the "tyranny of Applications". Unfortunately, application inadvertently always create silos (esp. on the Web) since <a href="http://dbpedia.org/resource/Entity">entity</a> data modeling, open data access, and other database technology realm matters, remain of secondary interest to many application developers.</p> <p>Final comment, RDF facilitates Linked Data on the Web, but all RDF isn't endowed with de-referencable URIs (a major source of confusion and misunderstanding). Thus, you can have RDF Data Source Providers that simply project RDF data silos via Web Services APIs if RDF output emanating from a Web Service doesn't provide out-bound pathways to other data via de-referencable URIs. Of course the same also applies to Widgets that present you with all the things they've discovered without exposing de-referencable URIs for each item.</p> <p>BTW - my final comments above aren't in anyway incongruent with devising successful business models for the Web. As you may or may not know, OpenLink is not only a major platform provider for the Semantic Web (expressed in our UDA, <a href="http://virtuoso.openlinksw.com" id="link-id0x19e44e80">Virtuoso</a>, <a href="http://dbpedia.org/resource/OpenLink_Data_Spaces" id="link-id0xb8637720">OpenLink Data Spaces</a>, and OAT products), we are also actively seeding Semantic Web (tribe: Linked Data of course) startups. For instance, <a href="http://zitgist.com/about/" id="link-id0x397b940">Zitgist</a>, which now has <a href="http://community.linkeddata.org/dataspace/person/mkbergman#this" id="link-id0x5fabcf0">Mike Bergman</a> as it's CEO alongside <a href="http://fgiasson.com/me/" id="link-id0xb84720f8">Frederick Giasson</a> as CTO. Of course, I cannot do <a href="http://zitgist.com/about/">Zitgist</a> justice via a footnote in a <a href="http://dbpedia.org/resource/Blog">blog</a> post, so I will expand further in a separate post.</p> <h2>Additional <a href="http://dbpedia.org/resource/Information">information</a> about this blog post:</h2> <ol> <li> I didn't spent hours looking for URIs used in my hyperlinks </li> <li> The post is best viewed via an RDF Linked Data aware user agents (<a href="http://demo.openlinksw.com/rdfbrowser" id="link-id0x3ac1b68">OpenLink RDF Browser</a>, Zitgist <a href="http://dataviewer.zitgist.com" id="link-id0x1d8e7ec0">Data Viewer</a>, <a href="http://www4.wiwiss.fu-berlin.de/rdf_browser" id="link-id0x19af3468">DISCO Hyperdata Browser</a>, <a href="http://dig.csail.mit.edu/2005/ajar/release/tabulator/0.8/tab.html" id="link-id0x1532e630">Tabulator</a>).</li> </ol>
RDF Benchmarking, Role, Motives, and Rationale
http://www.openlinksw.com/weblog/oerling/?date=2007-11-21#1274
2007-11-21T14:19:39Z
2008-04-30T14:28:17.000001-04:00
<p>Arising from the recent W3C workshop on <a href="http://www.w3.org/2007/03/RdfRDB/" id="link-id10679c70">mapping relational data to RDF</a>, there is some discussion on <a href="http://www.openlinksw.com/weblog/oerling/?id=1268" id="link-id1258dca8">starting a benchmarking oriented experimental group</a> under the W3C. I'll here make some comments on where this might fit and how this might serve our nascent industry.</p> <p>To the public, basically any recipient of the semantic <a href="http://dbpedia.org/resource/Data" id="link-id0xa203a350">data</a> web message, the benchmarking activity should communicate:</p> <ul> <li> <p>The semantic data web claims to</p> <ol> <li> allow integrating any legacy data from wherever and allow translating this into common, mutually joinable vocabularies, and</li> <li>make the web into a big database capable of answering structured queries on any open data.</li> </ol> </li> <li> <p>The benchmarking activity is to prove that this is not a pipe dream that Gartner Group forecast for 2027. Instead, there exists </p> <ol> <li>an industry, </li> <li>a degree of consensus within the industry concerning what the semantic data web is for, and</li> <li>products that are beyond experimental and can deliver at least some of the claimed benefits of the semantic data web.</li> </ol> </li> </ul> <p>To the general public, the message will be best delivered by the existence of online services that do interesting things with <a href="http://dbpedia.org/resource/Linked_Data" id="link-id0x1404e708">linked data</a>, starting from search and going to more specialized derivative products of structured <a href="http://dbpedia.org/resource/Information" id="link-id0xa1e12cd8">information</a> on the web.</p> <p>To those intending to apply some semantic data web things themselves, the benchmark activity should give a directory of products to look at. The reason why a benchmark suite backed by some industry consortium is useful is that it adds to the end user's confidence that the use case being measured is of somewhat general relevance and not just made to demonstrate any single product's strengths. Besides this, the TPC idea of disclosing scale, throughput, price per throughput and date is fine because it makes for easy tabulation of results. The intricacies in the full disclosure is effectively masked and it is my guess that very few read the actual full disclosures.</p> <p>The inference that an evaluator draws from benchmark results is that some product figuring there consistently is somewhat serious and can be studied further. Being in the running is like a stamp of approval. The benchmarks are complex and the evaluator seldom goes to the trouble of really analyzing performance by individual query or transaction even if these are and must be given. It is a bit like Formula 1 viewers do not generally read the rules on car engine or aerodynamics, let alone understand their finer points.</p> <p>For credibility to be thus given to products and hence the industry, we should just have a couple of well defined and agreed upon benchmarks, just like TPC.</p> <p>The third public is the developer. As a DBMS developer, I am a great fan of TPC. The great benefit I derive from their work is that they give a test suite for measuring effects of code changes on performance. Also, assuming that the TPC workload mix is representative, it also allows ranking what optimizations are more important than others. Lastly, TPC gives a great way of describing results, e.g., changes resulting in x% improvement on throughput of y. In such usage, the benchmarks are pretty much never run by the rules but results obtained are still good for internal comparison.</p> <p>Communication about IS should allow for short, simple messages: Release XX Halves Price per Throughput.</p> <p>The existence of benchmarks is, if not absolutely necessary, then at least a great help for such communication. Besides, people are culturally used to all kinds of racing and sports results so this is even a familiar format.</p> <p>Now the TPC is also not perfect. In the high end, the measured configurations are so large that one does not see them very often in practice. It is like the techno sports of Formula 1 or America's Cup. Interesting for the curiosity value but not immediately relevant to the regular car buyer or weekend yachtsman. Further, sponsoring a by-the-book audited TPC result is not so simple. Not as expensive as putting out an America's Cup challenge but still some trouble and expense.</p> <p>So, for us to benefit by the benchmarking activity, we must find a group that can both agree and be somewhat representative. Then we must put out a simple message: This here is for integration of relational sources and this here for storage and query of <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0xa192c590">RDF</a>.</p> <p>Furthermore, in so far we derive from relational or similar sources, the technology should not do less than the established alternative. This sends the wrong message.</p> <p>Entering the running should not be overly difficult for vendors, hence we should not have too many benchmarks and the ones that there are should be representative and sufficiently varied workloads. The results should be compact and easy to state. One more reason why I like TPC's work is the fact that the benchmarks have an easy to understand, unified use case behind them. Approximately what is done in each becomes clear from a very short and succinct description even though the details can be complex. I suspect this is one side of their appeal. I would venture the guess that a single use case story is easier to sell than a composite metric of disparate tests. Also in the scientific computing world, we have use cases, like NAS for aerodynamics, so having a use case story is quite common and a factor for making a benchmark's relevance understandable.</p> <p>Is this all possible?</p> <p>To play the devil's advocate, I could say that the use cases are not as well settled as the relational ones hence formulating a generally representative benchmark is not possible. Now this is certainly not a message that this community wishes to send. Besides, there exists decades worth of history of the problems of information integration and a great deal of RDF data out there, , even a compilation of dozens of industry use cases by the SWEO, so we are not exactly in the dark here.</p> <p>Can there be political agreement in reasonable time? If we look at the TPC as a precedent, judging by the rate of publication and revision, the process is not exactly quick. Now, for the TPC, it does not have to be. Judging by the frequency of published test results, hardware vendors are happy enough to have a forum to show off and do so at every turn.</p> <p>Now we are not at this stage of maturity yet.</p> <p>Composing a TPC style test spec is possible in a reasonable time for an individual but likely not for a committee. It is quite voluminous but also quite formulaic. While TPC's material is their own, I see no reason that we could not reference or link to it it where applicable.</p> <p>Who would be motivated by such activity? How to pitch the activity to would be participants? I don't think that just talking about what to measure and how is interesting enough. This is covered ground. Vendors want to promote themselves and end users want to have vendors compete at solving their problems. Or so it would be in a simpler world.</p> <p>Personally, I'd like to see a benchmark with a use case story people can relate to emerge in the next few months. Now I am not necessarily holding my breath waiting for this. For purposes of ongoing development, there is the real data out there and we can for example do the social web workload mix I suggested a couple of <a href="http://dbpedia.org/resource/Blog" id="link-id0x1f671bf8">blog</a> posts back on that and it is good enough for us. But that is not good enough for the industry's messaging.</p> <p>I'd say that we have to assume that people play in good faith and simply ask who want to run and get an extra edge by being in on the design of the race track. By good faith I here mean a sincere wish to have the race take place in the first place.</p> <p>The sport is exciting for the players and spectators alike if there is a use case story that they can relate to and an actual tournament. So this is what we should aim for. Because this is so far a niche public, we should not fragment the activity too much and we should consider how understandable and relevant the benchmark activity is to likely semantic data web adopters.</p>
On RDF and Vertical Storage
http://www.openlinksw.com/weblog/oerling/?date=2007-05-23#1197
2007-05-23T14:08:23Z
2008-04-24T09:52:09-04:00
<p>The topic of column-wise storage has not escaped us. We are not convinced that this is good for <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1f98ac20">RDF</a>. There is a point to this for business intelligence <a href="http://dbpedia.org/resource/Data" id="link-id0x1e0d2d30">data</a> warehouses, no doubt, although one could argue that one could get the same IO benefit with suitably selected covering indices but this is more design work. Column storage fits in less space and is more versatile For unexpected workloads.</p> <p>But we can look at the RDF case in specific. You have a quad of G, S, P, O. You have a one part index on each and you have a unique row number for each quad. Given the row number, you must get the G, S, P, and O, and given any one of these, you must get the row numbers where this occurs. If there were multi-part keys, then this would be a row store with covering indices, like <a href="http://virtuoso.openlinksw.com" id="link-id0x1c81d0b0">Virtuoso</a>'s RDF store.</p> <p>Each datum is stored 8 times. What is nice is that one can use any combination of selection criteria with equal ease and in the same working set. With the RDF workload, you end up typically referencing all parts of each quad. It is not like in the business intelligence case where the typical query accesses 4 columns of the 15 column history table. Of the 4 RDF quad keys, at least 2 are generally given. So this becomes a merge intersection of two or three indices and random lookups for the unspecified columns. Complicated control path, even if the engine is meant to do this thing alone.</p> <p>We'll have to try this. We could set up Virtuoso with 4 bitmap indices, each column to row ID and then a table with the 4 columns. Then we'd get bitmap ANDs for multi-column criteria and would have to get the row by row ID. As long as we run in memory, this should perform like a column store, close enough. We get the row with all the columns once, so we compensate for the fact that a column store has a special means for dereferencing the row ID for any column.</p> <p>If we optimized this specially, which would not be so terribly hard, we'd have a column store. The main new thing would be making a special index by row ID that would have the ID just once per index leaf and a bitmap for dense allocation of row IDs. The rest is not too different.</p> <p>For now, we will watch. If this is the next big thing, we can get there in little time.</p>
Semantic Web Data Spaces
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2007-04-13#1185
2007-04-13T21:15:54Z
2007-04-13T18:19:29.000001-04:00
<b>Web Data Spaces</b> <p>Now that broader understanding of the Semantic Data Web is emerging, I would like to revisit the issue of "<a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q='data%20spaces'&type=text&output=html">Data Spaces</a>".</p> <p>A Data Space is a place where Data Resides. It isn't inherently bound to a specific Data Model (<a href="http://en.wikipedia.org/wiki/Network_model">Concept Oriented</a>, <a href="http://en.wikipedia.org/wiki/Relational_model">Relational</a>, <a href="http://en.wikipedia.org/wiki/Hierarchical_database">Hierarchical</a> etc..). Neither is it implicitly an access point to Data, Information, or Knowledge (the perception is purely determined through the experiences of the user agents interacting with the Data Space.</p> <p>A Web Data Space is a Web accessible Data Space.</p> <p>Real world example:</p> <p>Today we increasing perform one of more of the following tasks as part of our professional and personal interactions on the Web:</p> <ol> <li>Blog via many service providers or personally managed weblog platforms</li> <li>Create Event Calendars via <a href="http://upcoming.com">Upcoming.com</a> and <a href="http://eventful.com">Eventful</a> </li> <li>Maintain and participate in Social Networks (e.g. <a href="http://facebook.com">Facebook</a>, <a href="http://orkut.com">Orkut</a>, <a href="http://myspace.com">MySpace</a>)</li> <li>Create and Participate in Discussions (note: when you comment on blogs or wikis for instance, you are participating in, or creating, a conversation)</li> <li>Track news by subscribing to <a href="http://web.resource.org/rss/1.0/">RSS 1.0</a>, <a href="http://cyber.law.harvard.edu/rss/rss.html">RSS 2.0</a>, or <a href="http://en.wikipedia.org/wiki/Atom_(standard)">Atom</a> Feeds</li> <li>Share Bookmarks & Tags via <a href="http://del.icio.us">Del.icio.us</a> and other Services</li> <li>Share Photos via <a href="http://flickr.com">Flickr</a> </li> <li>Buy, Review, or Search for books via <a href="http://amazon.com">Amazon</a> </li> <li>Participates in auctions via <a href="http://ebay.com">eBay</a> </li> <li>Search for data via <a href="http://google.com">Google</a> (of course!)</li> </ol> <p> <a href="http://www.johnbreslin.com/">John Breslin</a> has nice a <a href="http://www.johnbreslin.com/blog/wp-content/20051015a.gif">animation depicting the creation of Web Data Spaces</a> that drives home the point.</p> <b>Web Data Space Silos</b> <p> Unfortunately, what isn't as obvious to many netizens, is the fact that each of the activities above results in the creation of data that is put into some context by you the user. Even worse, you eventually realize that the service providers aren't particularly willing, or capable of, giving you unfettered access to your own data. Of course, this isn't always by design as the infrastructure behind the service can make this a nightmare from security and/or load balancing perspectives. Irrespective of cause, we end up creating our own "Data Spaces" all over the Web without a coherent mechanism for accessing and meshing these "Data Spaces".</p> <b>What are Semantic Web Data Spaces?</b> <p>Data Spaces on the Web that provide granular access to RDF Data.</p> <b>What's OpenLink Data Spaces (ODS) About?</b> <blockquote> <p>Short History</p> <p>In anticipation of this the "Web Data Silo" challenge (an issue that we tackled within internal enterprise networks for years) we commenced the development (circa. 2001) of a distributed collaborative application suite called OpenLink Data Spaces (ODS). The project was never released to the public since the problems associated with the deliberate or inadvertent creation of Web Data silos hadn't really materialized (silos only emerged in concreted form after the emergence of the Blogosphere and Web 2.0). In addition, there wasn't a clear standard Query Language for the RDF based Web Data Model (i.e. the SPARQL Query Language didn't exist).</p> </blockquote> <p> Today, ODS is delivered as a packaged solution (in Open Source and Commercial flavors) that alleviates the pain associated with Data Space Silos that exist on the Web and/or behind corporate firewalls. In either scenario, ODS simply allows you to create Open and Secure Data Spaces (via it's suite of applications) that expose data via SQL, RDF, XML oriented data access and data management technologies. Of course it also enables you to integrates transparently with existing 3rd party data space generators (Blogs, Wikis, Shared Bookmrks, Discussion etc. services) by supporting industry standards that cover:</p> <ol> <li> Content Publishing - Atom, <a href="http://www.sixapart.com/developers/product_documentation/movable_type/">Moveable Type</a>, <a href="http://www.xmlrpc.com/metaWeblogApi">MetaWeblog</a>, Blogger protocols </li> <li> Content Syndication Formats - RSS 1.0, RSS 2.0, Atom, OPML etc. </li> <li> Data Management - <a href="http://en.wikipedia.org/wiki/SQL">SQL</a>, <a href="http://www.w3.org/RDF/">RDF</a>, XML, Free Text </li> <li> Data Access - SQL, <a href="http://www.w3.org/TR/rdf-sparql-query/">SPARQL</a>, GData, Web Services (SOAP or REST styles), WebDAV/HTTP </li> <li> Semantic Data Web Middleware - <a href="http://www.w3.org/2004/01/rdxh/spec">GRDDL</a>, <a href="http://www.w3.org/TR/xslt">XSLT</a>, SPARQL, XPath/XQuery, HTTP (Content Negotiation) for producing RDF from non RDF Data ((X)HTML, Microformats, XML, Web Services Response Data etc). </li> </ol> <p>Thus, by installing ODS on your Desktop, Workgroup, Enterprise, or public Web Server, you end up with a very powerful solution for creating Open Data access oriented presence on the "Semantic Data Web" without incurring any of the typically assumed "RDF Tax".</p> <p>Naturally, ODS is built atop <a href="http://virtuoso.openlinksw.com">Virtuoso</a> and of course it exploits Virtuoso's feature-set to the max. It's also beginning to exploit functionality offered by the OpenLink Ajax Toolkit (<a href="http://demo.openlinksw.com/DAV/JS/demo/index.html">OAT</a>).</p>
Microsoft & Wikipedia Imbroglio
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2007-01-25#1124
2007-01-26T00:10:00Z
2007-01-25T18:47:47.000001-05:00
<p>I tried to post a comment to <a href="http://www.25hoursaday.com/weblog">Dare Obasanjo</a>'s blog post: <a href="http://www.25hoursaday.com/weblog/PermaLink.aspx?guid=0c22a95a-2d81-4f40-bbce-c763d8447468">How Do We Get Rid of Lies on Wikipedia</a>, without success (due to my attempts to add links to the post etc..). Hence a Blog style response instead.</p> <p>Dare:</p> <p>I have been through the Wikipedia fires a few times. If you recall that I actually triggered the early<a href="http://en.wikipedia.org/wiki/Web_2.0"> Web 2.0 Wikipedia article</a>. along the following lines: </p> <ol> <li> Asked one of my staff to start a post with the sole intention of defining Web 2.0 properly </li> <li> I then attempted to edit the initial post </li> <li> I left a typo re. REST </li> <li> Got set on Fire etc... (see very beginning of <a href="http://en.wikipedia.org/w/index.php?title=Talk:Web_2.0&action=history">Wikipedia Web 2.0 history page</a>) </li> </ol> <p>As annoying as the experience above was, I didn't find this inconsistent with the spirit of Wikipedia (i.e. open contribution and discourse). I felt, at the time, that a lot of historical data was being left in place for future reference etc.. In addition, the ultimate aim of creating an evolving Web 2.0 document did commence albeit some distance from "modern man" re. accuracy and meaningfulness as of my last read (today).</p> <p>Even closer to home, I repeated the process above re. <a href="http://en.wikipedia.org/wiki/Virtuoso_Universal_Server">Virtuoso Universal Server</a>. This basically ended up being a live case study on how you handle the Wikipedia NPOV conundurum. Just look at the <a href="http://en.wikipedia.org/wiki/Talk:Virtuoso_Universal_Server">Virtuoso Universal Server Talk Pages</a> to see how the process evolved (the key was Virtuoso's lineage and it's proximity to the very DBMS platform upon which Wikipedia runs i.e <a href="http://en.wikipedia.org/wiki/MySQL">MySQL</a>).</p> <p>Bearing in mind the size and magnitude of Microsoft, there should be no reason why Microsoft's "Microsoft Digital Caucus" ( legions of Staff, MSDN members, Integrators, and other partners) can't simply go into Wikipedia and participate in the edit and discourse process.</p> <p> Truth cannot be surpressed! At best, it can only be temporarily delayed :-) Even more so on the Web!</p>
Microsoft Data Access API Backgrounder: ODBC
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2007-01-03#1106
2007-01-03T18:20:45Z
2007-01-03T13:35:51.000001-05:00
<p>Mike Pizzo has commenced a much needed<a href="http://blogs.msdn.com/data/archive/2006/12/05/data-access-api-of-the-day-part-i.aspx"> 4-part article series covering the history of Microsoft's various Data Access</a> related APIs. Naturally, Part 1 covers: <a href="http://en.wikipedia.org/wiki/Open_Database_Connectivity">Open Database Connectivity</a> (ODBC) which is the first of a series of purpose specific Data Access APIs.</p> <p>Here is a very important excerpt:</p> <blockquote> <cite> ... <p>And then something happened. Visual Basic became popular as a scriptable "automation language". ODBC, being a C-style interface, was not directly consumable from VB. However, some of you clever folks figured out that Microsoft Access supported executing queries against ODBC Datasources, and that Access did support scriptable automation through its Data Access Object (DAO) API. Voila! Now you could write applications against ODBC sources using VB.</p> <p>However, DAO went through <b>Access's internal "Jet" (Joint Engine Technology)</b> database engine, which defaulted to building local keysets for each result in order to do advanced query processing and cursoring against the remote data. This was fine if you needed that functionality, but significant performance overhead and additional round trips when you didn't. </p> <p>Enter the Visual Basic team who, responding to customer demand for better performance against ODBC sources, came up with something called Remote Data Objects (RDO). RDO implemented the same DAO programming patterns directly against ODBC, rather than going through Jet. RDO was extremely popular among VB developers, but the fact that we had two different sets of automation objects for accessing ODBC sources caused confusion.</p> <p> But apparently not enough confusion, because our solution was to introduce "ODBCDirect". Despite its name, ODBCDirect was not a new API; it was just a mode we added to DAO that set defaults in such a way as to avoid the overhead of building keysets and such</p> ... </cite> </blockquote> <p>To this very day (unfortunately!) ODBC has been maligned by the perpetuated misunderstanding of JET's DAO layer that sits atop ODBC providing advanced query processing (i.e. Virtual DBMS functionality) alongside a client-side keyset cursor model implementation.</p>
Web 2.0's Open Data Access Conundrum (Update)
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2006-09-05#1034
2006-09-05T21:02:00Z
2006-11-16T16:11:45-05:00
<p> Open Data Access and Web 2.0 have a very strange relationship that continues to blur the lines of demarcation between where Web 2.0 ends and where Web.Next (i.e Web 3.0, Semantic/Data Web, Web of Databases etc.) starts. But before I proceed, let me attempt to define Web 2.0 one more time: </p> <p style="text-align: center;"> <em>A phase in the evolution web usage patterns that emphasizes Web Services based interaction between âWeb Usersâ and âPoints of Web Presenceâ over traditional âWeb Usersâ and âWeb Sitesâ based interaction. Basically, a transition from visual site interaction to presence based interaction.</em> </p> <p> BTW - Dare Obasanjo also commented about Web usage patterns in his post titled: <a href="http://www.25hoursaday.com/weblog/PermaLink.aspx?guid=929a7fd6-1dfc-43f4-a549-d2c9fa873655">The Two Webs</a>. Where he concluded that we had a dichotomy along the lines of: HTTP-for-APIs (2.0) and HTTP-for-Browsers (1.0). Which <a href="http://weblog.infoworld.com/udell">Jon Udell</a> evolved into: HTTP-Services-Web and HTTP-Intereactive-Web during our recent <a href="http://weblog.infoworld.com/udell/gems/ju_idehen.mp3">podcast conversation</a>. </p> <p> With definitions in place, I will resume my quest to unveil the aforementioned Web 2.0 Data Access Conundrum: </p> <ul> <li>Emphasis on XML's prowess in the realms of Data and Protocol Modeling alongside Data Representation. Especially as SOAP or REST styles of Web Services and various XML formats (RSS 0.92/1.0/1.1/2.0, Atom, OPML, OCS etc.) collectively define the Web 2.0 infrastructure landscape</li> <li>Where a modicum of Data Access appreciation and comprehension does exist it is inherently compromised by business models that mandate some form of âWalled Gardensâ and âData Silosâ</li> <li>Mash-ups are a response to said âWalled Gardensâ and âData Silosâ . Mash-ups by definition imply combining things that were not built for recombination.</li> </ul> <p> As you can see from the above, Open Data access isn't genuinely compatible with Web 2.0. </p> <p> We can also look at the same issue by way of the popular M-V-C (Model View Controller) pattern. Web 2.0 is all about the âVâ and âCâ with a modicum of âMâ at best (data access, open data access, and flexible open data access are completely separate things). The âCâ items represent application logic exposed by SOAP or REST style web services etc. I'll return to this later in this post. </p> <p> What about Social Networking you must be thinking? Isn't this a Web 2.0 manifestation? Not at all (IMHO). The Web was developed / invented by Tim Berners-Lee to leverage the âNetwork Effectsâ potential of the Internet for connecting <a href="http://www.w3.org/History/1989/Image1.gif">People and Data</a>. Social Networking on the other hand, is simply one of several ways by which construct network connections. I am sure we all accept the fact that connections are built for many other reasons beyond social interaction. That said, we also know that through social interactions we actually develop some of our most valuable relationships (we are social creatures after-all). </p> <p> The Web 2.0 Open Data Access impedance reality is ultimately going to be the greatest piece of tutorial and usecase material for the Semantic Web. I take this position because it is human nature to seek Freedom (in unadulterated form) which implies the following: </p> <ul> <li>Access Data from a myriad of data sources (irrespective of structural differences at the database level)</li> <li>Mesh (not Mash) data in new and interesting ways</li> <li>Share the meshed data with as many relevant people as possible for social, professional, political, religious, and other reasons</li> <li>Construct valuable networks based on data oriented connections</li> </ul> <p> Web 2.0 by definition and use case scenarios is inherently incompatible with the above due to the lack of Flexible and Open Data Access. </p> <p> If we take the definition of Web 2.0 (above) and rework it with an appreciation Flexible and Open Data Access you would arrive at something like this: </p> <p style="text-align: center;"> <em>A phase in the evolution of the web that emphasizes interaction between âWeb Usersâ and âWeb Dataâ facilitated by Web Services based APIs and an Open & Flexible Data Access Model â. </em> </p> <p> <em> <br /> </em>In more succinct form: </p> <p style="text-align: center;"> <em>A pervasive network of people connected by data or data connected by people.</em> </p> <p> <em> <br /> </em>Returning to M-V-C and looking at the definition above, you now have a complete of âMâ which is enigmatic in Web 2.0 and the essence of the Semantic Web (Data and Context). </p> <p> To make all of this possible a palatable Data Model is required. The model of choice is the Graph based RDF Data Model - not to be mistaken for the RDF/XML serialization which is just that, a data serialization that conforms to the aforementioned RDF data model. </p> <p> <strong>The Enterprise Challenge</strong> </p> <p> Web 2.0 cannot and will not make valuable inroads into the the enterprise because enterprises live and die by their ability to exploit data. Weblogs, Wikis, Shared Bookmarking Systems, and other Web 2.0 distributed collaborative applications profiles are only valuable if the data is available to the enterprise for meshing (not mashing). </p> <p> A good example of how enterprises will exploit data by leveraging networks of people and data (social networks in this case) is shown in this nice presentation by Accenture's Institute for High Performance Business titled: <a href="http://www.accenture.com/xdoc/en/AccentureSNA.swf">Visualizing Organizational Change</a>. </p> <p> Web 2.0 commentators (for the most part) continue to ponder the use of Web 2.0 within the enterprise while forgetting the congruency between enterprise agility and exploitation of people & data networks (The very issue emphasized in this original <a href="http://www.w3.org/History/1989/proposal.html">Web vision document by Tim Berners-Lee</a>). Even worse, they remain challenged or spooked by the Semantic Web vision because they do not understand that Web 2.0 is fundamentally a Semantic Web precursor due to Open Data Access challenges. Web 2.0 is one of the greatest demonstrations of why we need the Semantic Web at the current time. </p> <p> Finally, juxtapose the items below and you may even get a clearer view of what I am an attempting to convey about the virtues of Open Data Access and the inflective role it plays as we move beyond Web 2.0: </p> <p> <a href="http://www.w3.org/History/1989/proposal.html">Information Management Proposal </a>- <a href="http://www.w3.org/People/Berners-Lee/">Tim Berners-Lee</a> <br /> <a href="http://www.accenture.com/xdoc/en/AccentureSNA.swf">Visualizing Organizational Change</a> - <a href="http://www.accenture.com/Global/High_Performance_Business/default.htm">Accenture Institute of High Performance Business</a> </p>
Web 2.0's Open Data Access Conundrum
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2006-09-02#1032
2006-09-02T16:47:52Z
2006-11-16T15:51:43-05:00
<p> Open Data Access and Web 2.0 have a very strange relationship that continues to blur the lines of demarcation between where Web 2.0 ends and where Web.Next (i.e Web 3.0, Semantic/Data Web, Web of Databases etc.) starts. But before I proceed, let me attempt to define Web 2.0 one more time: </p> <p style="text-align: center;"> <em>A phase in the evolution web usage patterns that emphasizes Web Services based interaction between âWeb Usersâ and âPoints of Web Presenceâ over traditional âWeb Usersâ and âWeb Sitesâ based interaction. Basically, a transition from visual site interaction to presence based interaction.</em> </p> <p> BTW - Dare Obasanjo also commented about Web usage patterns in his post titled: <a href="http://www.25hoursaday.com/weblog/PermaLink.aspx?guid=929a7fd6-1dfc-43f4-a549-d2c9fa873655">The Two Webs</a>. Where he concluded that we had a dichotomy along the lines of: HTTP-for-APIs (2.0) and HTTP-for-Browsers (1.0). Which <a href="http://weblog.infoworld.com/udell">Jon Udell</a> evolved into: HTTP-Services-Web and HTTP-Intereactive-Web during our recent <a href="http://weblog.infoworld.com/udell/gems/ju_idehen.mp3">podcast conversation</a>. </p> <p> With definitions in place, I will resume my quest to unveil the aforementioned Web 2.0 Data Access Conundrum: </p> <ul> <li>Emphasis on XML's prowess in the realms of Data and Protocol Modeling alongside Data Representation. Especially as SOAP or REST styles of Web Services and various XML formats (RSS 0.92/1.0/1.1/2.0, Atom, OPML, OCS etc.) collectively define the Web 2.0 infrastructure landscape</li> <li>Where a modicum of Data Access appreciation and comprehension does exist it is inherently compromised by business models that mandate some form of âWalled Gardensâ and âData Silosâ</li> <li>Mash-ups are a response to said âWalled Gardensâ and âData Silosâ . Mash-ups by definition imply combining things that were not built for recombination.</li> </ul> <p> As you can see from the above, Open Data access isn't genuinely compatible with Web 2.0. </p> <p> We can also look at the same issue by way of the popular M-V-C (Model View Controller) pattern. Web 2.0 is all about the âVâ and âCâ with a modicum of âMâ at best (data access, open data access, and flexible open data access are completely separate things). The âCâ items represent application logic exposed by SOAP or REST style web services etc. I'll return to this later in this post. </p> <p> What about Social Networking you must be thinking? Isn't this a Web 2.0 manifestation? Not at all (IMHO). The Web was developed / invented by Tim Berners-Lee to leverage the âNetwork Effectsâ potential of the Internet for connecting <a href="http://www.w3.org/History/1989/Image1.gif">People and Data</a>. Social Networking on the other hand, is simply one of several ways by which construct network connections. I am sure we all accept the fact that connections are built for many other reasons beyond social interaction. That said, we also know that through social interactions we actually develop some of our most valuable relationships (we are social creatures after-all). </p> <p> The Web 2.0 Open Data Access impedance reality is ultimately going to be the greatest piece of tutorial and usecase material for the Semantic Web. I take this position because it is human nature to seek Freedom (in unadulterated form) which implies the following: </p> <ul> <li>Access Data from a myriad of data sources (irrespective of structural differences at the database level)</li> <li>Mesh (not Mash) data in new and interesting ways</li> <li>Share the meshed data with as many relevant people as possible for social, professional, political, religious, and other reasons</li> <li>Construct valuable networks based on data oriented connections</li> </ul> <p> Web 2.0 by definition and use case scenarios is inherently incompatible with the above due to the lack of Flexible and Open Data Access. </p> <p> If we take the definition of Web 2.0 (above) and rework it with an appreciation Flexible and Open Data Access you would arrive at something like this: </p> <p style="text-align: center;"> <em>A phase in the evolution of the web that emphasizes interaction between âWeb Usersâ and âWeb Dataâ facilitated by Web Services based APIs and an Open & Flexible Data Access Model â. </em> </p> <p> <em> <br /> </em>In more succinct form: </p> <p style="text-align: center;"> <em>A pervasive network of people connected by data or data connected by people.</em> </p> <p> <em> <br /> </em>Returning to M-V-C and looking at the definition above, you now have a complete of âMâ which is enigmatic in Web 2.0 and the essence of the Semantic Web (Data and Context). </p> <p> To make all of this possible a palatable Data Model is required. The model of choice is the Graph based RDF Data Model - not to be mistaken for the RDF/XML serialization which is just that, a data serialization that conforms to the aforementioned RDF data model. </p> <p> <strong>The Enterprise Challenge</strong> </p> <p> Web 2.0 cannot and will not make valuable inroads into the the enterprise because enterprises live and die by their ability to exploit data. Weblogs, Wikis, Shared Bookmarking Systems, and other Web 2.0 distributed collaborative applications profiles are only valuable if the data is available to the enterprise for meshing (not mashing). </p> <p> A good example of how enterprises will exploit data by leveraging networks of people and data (social networks in this case) is shown in this nice presentation by Accenture's Institute for High Performance Business titled: <a href="http://www.accenture.com/xdoc/en/AccentureSNA.swf">Visualizing Organizational Change</a>. </p> <p> Web 2.0 commentators (for the most part) continue to ponder the use of Web 2.0 within the enterprise while forgetting the congruency between enterprise agility and exploitation of people & data networks (The very issue emphasized in this original <a href="http://www.w3.org/History/1989/proposal.html">Web vision document by Tim Berners-Lee</a>). Even worse, they remain challenged or spooked by the Semantic Web vision because they do not understand that Web 2.0 is fundamentally a Semantic Web precursor due to Open Data Access challenges. Web 2.0 is one of the greatest demonstrations of why we need the Semantic Web at the current time. </p> <p> Finally, juxtapose the items below and you may even get a clearer view of what I am an attempting to convey about the virtues of Open Data Access and the inflective role it plays as we move beyond Web 2.0: </p> <p> <a href="http://www.w3.org/History/1989/proposal.html">Information Management Proposal </a>- <a href="http://www.w3.org/People/Berners-Lee/">Tim Berners-Lee</a> <br /> <a href="http://www.accenture.com/xdoc/en/AccentureSNA.swf">Visualizing Organizational Change</a> - <a href="http://www.accenture.com/Global/High_Performance_Business/default.htm">Accenture Institute of High Performance Business</a> </p>
The WWW Proposal and RDF: Then and Now (circa 1999)
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2006-08-28#1029
2006-08-28T10:20:00Z
2006-09-30T16:27:36-04:00
<p>I've just re-read an article penned by Dan Brickley in 1999 titled: <a href="http://www.w3.org/1999/11/11-WWWProposal/thenandnow">The WWW Proposal and RDF: Then and Now</a>, that retains its prescience to this very day. Ironically I stumbled across this timeless piece while revisiting the <a href="http://diveintomark.org/archives/2002/09/06/history_of_the_rss_fork">RSS name imbroglio</a> that gave us a simple syndication format (RSS 2.0) that will ultimately implode (IMHO) since "Simple" is ultimately short lived when dealing with attention challenged end-users that are always assumed to be dumb when in fact they are simply ambivalent.</p> <p>I was compelled to go back to the RSS 2.0 imbroglio when I came across <a href="http://www.scripting.com/dwiner/">Dave Winer</a>'s comments re. "the SEC attempting to reinvent RSS 2.0..." response to <a href="http://weblog.infoworld.com/udell/2006/08/16.html">Jon Udell's recent XBRL article</a>. </p> <p>Although I don't believe in complex entry points into complex technology realms, I do subscribe to the approach where developers deal with the complexity associated with a problem domain while hiding said complexity from ambivalent end-users via coherent interfaces -- which does not always imply User Interface.</p> <p> <a href="http://xml.coverpages.org/xbrl.html">XBRL</a> is a great piece of work that addresses the complex problem domain of Financial Reporting. The only thing it's missing right now is an Ontology that facilitates <a href="http://www.w3.org/TR/rdf-primer/">RDF Data Model</a> based XBRL Schema and Instance Data which ultimately makes XBRL data available to RDF query languages such as <a href="http://www.w3.org/TR/rdf-sparql-query/">SPARQL</a>. This line of thought implies, for instance, an XML Schema to <a href="http://www.w3.org/TR/owl-guide/">OWL Ontology Mapping</a> for Schema Data (as explained in a <a href="http://www.google.com/url?sa=t&ct=res&cd=4&url=http%3A%2F%2Fvsis-www.informatik.uni-hamburg.de%2FgetDoc.php%2Fpublications%2F204%2Ffzt-lxs-04.pdf&ei=4lXzRPLaO8SmaJmgsLgC&sig2=INc-OyDoxj16TW8tb0pNXA#search=%22xml%20schema%20owl%20mapping%22">white paper by the VSIS Group at the university of Hamburg</a>) leaving the Instance Data to be generated in a myriad of ways that includes XML to RDF and/or XML->SQL->RDF.</p> <p>As I stated in an earlier post: <a href="http://www.openlinksw.com/blog/%7Ekidehen/index.vspx?page=&id=1018">we should not mistake ambivalence to lack of intelligence</a>. Assuming "Simple" is always right at all times is another way of subscribing to this profound misconception. You know, assuming the world was flat (as opposed to geoid) was quite palatable at some point in the history of mankind, I wonder what would have happened if we held on to this point of view to this day because of its "Simplicity"?</p>
Value vs Source
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2006-07-29#1020
2006-07-29T22:19:22Z
2006-07-29T18:55:52.000002-04:00
<p>(Via <a href="http://davidwarlick.com/2cents">David Warlick</a>.)</p>: <p> <a href="http://davidwarlick.com/2cents/2006/07/29/value-vs-source/#comments">Value vs Source</a>: "</p> <p>I think weâre all sorta jumping around the same bush. Itâs been a good dance because Iâve learned some things. First of all, nothingâs simple and it isnât getting any simpler. There are no rules any more and as much as Iâd like to come up with some kind of all encompassing unified field theory of ethical research method, I know that smarter people than me have already done a better job, and none of it is perfect.</p> <p>Please allow me to do something kinda strange. I want to look backward for some clues. When I was young, my Dad loved to build things. He was the preeminent do-it-yourselfer. Every weekend, he had a building project, and every Saturday morning he loaded us boys into the station wagon and off we went to the Lowes Hardware Store in Shelby, where he bought the tools and materials he would need for the project. </p> <p>He did not have a list of criterial for selecting his materials, because every project was different â the goal was different. If he had selected everything based on the same criteria, then everything he built would have been made with pine shelving, two-penny finishing nails, and all the work would have been done with a Craftsman common nail hammer. Instead, he selected his building materials and tools based on the goal of the project. To do otherwise would have resulted in a product that did not last long, and that would have been unethical.</p> <p> <img src="http://davidwarlick.com/2cents/wp-content/uploads/2006/07/bill_edwards.jpg" height="309" width="169" border="1" align="right" hspace="4" vspace="4" alt="Bill Edwards" />Years later, I studied under the best teacher I ever had, Mr. Bill Edwards â my industrial arts teacher. His technique was to help us learn industrial arts skills by helping us to build something of value. I built a kayak. Other students built book shelves, stools, and chess boards. Two friends of mine built a life-size replica of a Gemini Space Capsule. Mr. Edwards taught us to set goals and to make decisions based on those goals.</p> <p>This was the perfect way to teach industrial arts skills, since we were in the industrial age. If Edwards had taught us in the same way that my information arts teachers were teaching, he would have put a stack of lumber on our desks and asked us to practice driving nails. But he taught us by putting us in the industry. We should be teaching today by putting students in the industry of information. We need to stop teaching science and start teaching students to be scientists. Stop teaching history, but rather teach to be historians. Stop teaching students to be researchers, and instead, teach them to solve problems and accomplish goals using information.</p> <p>I am certain that there were brands of wood and nails that my father wouldnât buy, because he couldnât depend on them. He swore by Craftsman tools. To build with materials that were unreliable would have been unethical. But his conscious work in finding and selecting materials was based on the goal at hand. All else pointed to that criteria.</p> <p>It is critical to know and understand the source of the information. But what is it about the source that helps you accomplish your goal. Itâs important to understand when the information was generated and published. But what is it about âwhenâ that helps you accomplish your goal. Itâs important to understand what the information is made of, and what it is about its format and how you can use it that helps you accomplish your goal. Itâs important to understand the informationâs cultural, economic, environmental, and emotional context, and what it is about the context that helps you accomplish your goal. All aspects remain critical, but its problem solving and goal achieving that children need to be doing, not just hoop-jumping in their schools. The need to look for the informationâs value as a tool for <span style="text-decoration:underline;">ethically</span> accomplishing their goals.</p> <p> <br /> </p> <p style="text-align:right;font-size:10px;">Technorati Tags: <a href="http://www.technorati.com/tag/librarians" rel="tag">librarians</a>, <a href="http://www.technorati.com/tag/warlick" rel="tag">warlick</a> </p> <p></p> <hr size="1" /> <br /> Portions of this post come from Raw Materials for the Mind ISBN #1-4116-2795-4<br /> <a href="http://www.lulu.com/content/116469"><br /> <img src="http://www.lulu.com/services/buy_now_buttons/images/book_maroon.gif" border="0" alt="Support independent publishing: buy this book on Lulu." /> <br /> </a> "
RDF's History
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2006-07-13#1004
2006-07-13T21:42:57Z
2006-07-13T19:04:36-04:00
<p>We are getting very close to a Semantic Web watershed moment (IMHO). Thus, for the purpose of historic record, I would like to create a public bookmark to Tim Bray's 2003 post titled: <a href="http://www.tbray.org/ongoing/When/200x/2003/05/21/RDFNet">RDF.net</a> Challenge that also contains a nice section about the <a href="http://www.tbray.org/ongoing/When/200x/2003/05/21/RDFNet">History of RDF</a>.</p> <p>Note to Tim:</p> <p> Is the RDF.net domain deal still on? I know it's past 1st Jan 2006, but do bear in mind that the critical issue of a broadly supported RDF Query Language only took significant shape approximately 13 months ago (in the form of SPARQL), and this is all so critical to the challenge you posed in 2003.</p> <p> <a href="http://rdf.net">RDF.net</a> could become a point of semantic-web-presence through which the benefits of SPARQL compliant Triple|Quad Stores, Shared Ontologies, and SPARQL Protocol are unveiled in their well intended glory :-).</p>
Standards as social contracts
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2006-07-04#995
2006-07-04T17:25:51Z
2006-07-04T14:53:48.000001-04:00
<blockquote> <p> <a href="http://www.tnl.net/blog/2006/06/07/standards-as-social-contracts/#comments">Standards as social contracts</a>: "Looking at Dave Winer's efforts in evangelizing OPML, I try to draw some rough lines into what makes a de-facto standard. De Facto standards are made and seldom happen on their own. In this entry, I look back at the history of HTML, RSS, the open source movement and try to draw some lines as to what makes a standard. </p> <p> <a href="http://feeds.tristanlouis.com/~a/TNLnet?a=nXIQUu"><img src="http://feeds.tristanlouis.com/~a/TNLnet?i=nXIQUu" border="0" /> </a> </p> <div class="feedflare"> <a href="http://feeds.tristanlouis.com/~f/TNLnet?a=dklI2jYY"><img src="http://feeds.tristanlouis.com/~f/TNLnet?i=dklI2jYY" border="0" /> </a> <a href="http://feeds.tristanlouis.com/~f/TNLnet?a=HoauA2Ma"><img src="http://feeds.tristanlouis.com/~f/TNLnet?i=HoauA2Ma" border="0" /></a> <a href="http://feeds.tristanlouis.com/~f/TNLnet?a=DxOLN3Br"><img src="http://feeds.tristanlouis.com/~f/TNLnet?i=DxOLN3Br" border="0" /></a> <a href="http://feeds.tristanlouis.com/~f/TNLnet?a=zU2uLdOm"><img src="http://feeds.tristanlouis.com/~f/TNLnet?i=zU2uLdOm" border="0" /></a> </div>" <p>(Via <a href="http://www.tnl.net/blog">Tristan Louis</a>.)</p> </blockquote> <p>I posted a comment to the Tristan Louis' post along the following lines:</p> <p>Analysis is spot on re. the link between de facto standardization and bootstrapping. Likewise, the clear linkage between boostrapping and connected communities (a variation of the social networking paradigm). </p> <p>Dave built a community around a XML content syndication and subscription usecase demo that we know today as the blogosphere. Superficially, one may conclude that Semantic Web vision has suffered to date from a lack a similar bootstrap effort. Whereas in reality, we are dealing with "time and context" issues that are critical to the base understanding upon which a "Dave Winer" style bootstrap for the Semantic Web would occur.</p> <p>Personally, I see the emergence of Web 2.0 (esp. the mashups phenomenon) as the "time and context" seeds from which the Semantic Web bootstrap will sprout. I see shared ontologies such as <a href="http://oplussol5.usnet.private:8893/foaf">FOAF</a> and <a href="http://rdfs.org/sioc/">SIOC</a> leading the way (they are the RSS 2.0's of the Semantic Web IMHO).</p>
My podcast conversation with Jon Udell
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2006-04-28#965
2006-04-28T14:43:12Z
2006-07-21T07:22:41.000001-04:00
<p>Jon and I had a recent chat yesterday that is now available in <a href="http://weblog.infoworld.com/udell/2006/04/28.html#a1437">Podcast</a> form.</p> <blockquote> <cite><p>"In my <a href="http://weblog.infoworld.com/udell/gems/ju_idehen.mp3">fourth Friday podcast</a> we hear from Kingsley Idehen, CEO of <a href="http://openlinksw.com/">OpenLink Software</a>. I wrote about OpenLink's universal database and app server, Virtuoso, back in <a href="http://www.infoworld.com/article/02/04/12/020415plvirtuoso_1.html">2002</a> and <a href="http://www.infoworld.com/article/03/03/21/12virtuoso_1.html">2003</a>. Earlier this month Virtuoso became the first mature SQL/XML hybrid to make the <a href="http://www.openlinksw.com/blog/~kidehen/?id=951">transition to open source</a>. The latest incarnation of the product also adds SPARQL (a semantic web query language) to its repertoire. <b>...</b>"</p> <p>(Via <a href="http://weblog.infoworld.com/udell/">Jon's Radio</a>.)</p> </cite> </blockquote> I would like to make an important clarification re. the GData Protocol and what is popularly dubbed as "<a href="http://jeremy.zawodny.com/blog/archives/006687.html">Adam Bosworth's fingerprints.</a>" I do not believe in a one solution (a simple one for the sake of simplicity) to a deceptively complex problem. Virtuoso supports Atom 1.0 (syndication only at the current time) and Atom 0.3 (syndication and publication which have been in place for years). <blockquote>BTW - the GData Protocol and Atom 1.0 publishing support will be delivered in both the Open Source and Commercial Edition updates to Virtuoso next week (very little work due to what's already in place).</blockquote> <p>I make the clarification above to eliminate the possibility of assuming mutual exclusivity of my perspective/vison and Adam's (Jon also makes this important point when he speaks about our opinions being on either side of a spectrum/continuum). I simply want to broaden the scope of this discussion. I am a profound believer in the Semantic Web / Data Web vision, and I predict that we will be querying the Googlebase via SPARQL in the not to distant future (this doesn't mean that netizens will be forced to master SPARQL, absolutely not! But there will be conduit technologies that deal with matter).</p> <p>Side note: I actually last spoke with Adam at the NY Hilton in 2000 (the day I unveiled Virtuoso to the public for the first time, in person). We bumped into each other and I told him about Virtuoso (at the time the big emphasis was SQL to XML and the vocabulary we had chosen re. SQL extension...), and he told me about his departure from Microsoft and the commencement of his new venture (CrossGain prior to his stint at BEA), what struck me even more was his interest in Linux and Open Source (bearing in mind this was about 3 or so week after he departed Microsoft.)</p> <p>If you are encountering Virtuoso for the first time via this post or Jon's, please make time to read the <a href="http://virtuoso.openlinksw.com/wiki/main/Main/VOSHistory/">product history</a> article on the <a href="http://virtuoso.openlinksw.com/wiki/main/Main/">Virtuoso Wiki</a> (which is one of many Virtuoso based applications that make up our soon to be released OpenLink DataSpace offering).</p> <p>That said, I better go listen to the podcast :-)</p>
My podcast conversation with Jon Udell
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2006-04-28#993
2006-04-28T14:43:12Z
2006-06-29T10:14:44.000001-04:00
<p>Jon and I had a recent chat yesterday that is now available in <a href="http://weblog.infoworld.com/udell/2006/04/28.html#a1437">Podcast</a> form.</p> <blockquote> <cite></cite> <p>"In my <a href="http://weblog.infoworld.com/udell/gems/ju_idehen.mp3">fourth Friday podcast</a> we hear from Kingsley Idehen, CEO of <a href="http://openlinksw.com/">OpenLink Software</a>. I wrote about OpenLink's universal database and app server, Virtuoso, back in <a href="http://www.infoworld.com/article/02/04/12/020415plvirtuoso_1.html">2002</a> and <a href="http://www.infoworld.com/article/03/03/21/12virtuoso_1.html">2003</a>. Earlier this month Virtuoso became the first mature SQL/XML hybrid to make the <a href="http://www.openlinksw.com/blog/%7Ekidehen/?id=951">transition to open source</a>. The latest incarnation of the product also adds SPARQL (a semantic web query language) to its repertoire. <b>...</b>"</p> <p>(Via <a href="http://weblog.infoworld.com/udell/">Jon's Radio</a>.)</p> </blockquote> I would like to make an important clarification re. the GData Protocol and what is popularly dubbed as "<a href="http://jeremy.zawodny.com/blog/archives/006687.html">Adam Bosworth's fingerprints.</a>" I do not believe in a one solution (a simple one for the sake of simplicity) to a deceptively complex problem. Virtuoso supports Atom 1.0 (syndication only at the current time) and Atom 0.3 (syndication and publication which have been in place for years). <blockquote>BTW - the GData Protocol and Atom 1.0 publishing support will be delivered in both the Open Source and Commercial Edition updates to Virtuoso next week (very little work due to what's already in place).</blockquote> <p>I make the clarification above to eliminate the possibility of assuming mutual exclusivity of my perspective/vison and Adam's (Jon also makes this important point when he speaks about our opinions being on either side of a spectrum/continuum). I simply want to broaden the scope of this discussion. I am a profound believer in the Semantic Web / Data Web vision, and I predict that we will be querying the Googlebase via SPARQL in the not to distant future (this doesn't mean that netizens will be forced to master SPARQL, absolutely not! But there will be conduit technologies that deal with matter).</p> <p>Side note: I actually last spoke with Adam at the NY Hilton in 2000 (the day I unveiled Virtuoso to the public for the first time, in person). We bumped into each other and I told him about Virtuoso (at the time the big emphasis was SQL to XML and the vocabulary we had chosen re. SQL extension...), and he told me about his departure from Microsoft and the commencement of his new venture (CrossGain prior to his stint at BEA), what struck me even more was his interest in Linux and Open Source (bearing in mind this was about 3 or so week after he departed Microsoft.)</p> <p>If you are encountering Virtuoso for the first time via this post or Jon's, please make time to read the <a href="http://virtuoso.openlinksw.com/wiki/main/Main/VOSHistory">product history</a> article on the <a href="http://virtuoso.openlinksw.com/wiki/main/">Virtuoso Wiki</a> (which is one of many Virtuoso based applications that make up our soon to be released OpenLink DataSpace offering).</p> <p>That said, I better go listen to the podcast :-)</p>
Virtuoso is Officially Open Source!
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2006-04-11#951
2006-04-11T18:01:44Z
2006-07-21T07:22:20.000001-04:00
<p>I am pleased to unveil (officially) the fact that <a href="http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104&STORY=/www/story/04-11-2006/0004338324&EDATE=">Virtuoso is now available in Open Source form</a>.</p> <p></p> <h4>What Is Virtuoso?</h4> <p>A powerful next generation server product that implements otherwise distinct server functionality within a single server product. Think of Virtuoso as the server software analog of a dual core processor where each core represents a traditional server functionality realm.</p> <p></p> <h4>Where did it come from?</h4> <p>The <a href="http://virtuoso.openlinksw.com/wiki/main/Main/VOSHistory">Virtuoso History page</a> tells the whole story.</p> <p></p> <h4>What Functionality Does It Provide?</h4> The following: <ul> 1. Object-Relational DBMS Engine (ORDBMS like PostgreSQL and DBMS engine like MySQL) </ul> <ul> 2. XML Data Management (with support for XQuery, XPath, XSLT, and XML Schema) </ul> <ul> 3. RDF Triple Store (or Database) that supports SPARQL (Query Language, Transport Protocol, and XML Results Serialization format) </ul> <ul> 4. Service Oriented Architecture (it combines a BPEL Engine with an ESB) </ul> <ul> 5. Web Application Server (supports HTTP/WebDAV) </ul> <ul> 6. NNTP compliant Discussion Server </ul> And more. (see: <a href="http://virtuoso.openlinksw.com">Virtuoso Web Site</a>) <p> 90% of the aforementioned functionality has been available in Virtuoso since 2000 with the RDF Triple Store being the only 2006 item.</p> <p></p> <h4>What Platforms are Supported</h4> <p> The Virtuoso build scripts have been successfully tested on Mac OS X (Universal Binary Target), Linux, FreeBSD, and Solaris (AIX, HP-UX, and True64 UNIX will follow soon). A Windows Visual Studio project file is also in the works (ETA some time this week).</p> <p></p> <h4>Why Open Source?</h4> <p>Simple, there is no value in a product of this magnitude remaining the "best kept secret". That status works well for our competitors, but absolutely works against the legions of new generation developers, systems integrators, and knowledge workers that need to be aware of what is actually achievable today with the right server architecture.</p> <p></p> <h4>What Open Source License is it under?</h4> <p>GPL version 2.</p> <p></p> <h4>What's the business model?</h4> <p>Dual licensing.</p> <p>The Open Source version of Virtuoso includes all of the functionality listed above. While the Virtual Database (distributed heterogeneous join engine) and Replication Engine (across heterogeneous data sources) functionality will only be available in the commercial version. </p> <p></p> <h4>Where is the Project Hosted?</h4> <p>On <a href="http://sourceforge.net/projects/virtuoso">SourceForge.</a> </p> <p></p> <h4>Is there a product Blog?</h4> <p>Of course! </p> <p>Up until this point, the <a href="http://virtuoso.openlinksw.com/blog/">Virtuoso Product Blog</a> has been a covert live demonstration of some aspects of Virtuoso (Content Management). My Personal Blog and the Virtuoso Product Blog are actual Virtuoso instances, and have been so since I started blogging in 2003.</p> <p>Is There a product Wiki?</p> <p>Sure! <a href="http://virtuoso.openlinksw.com/wiki/main/">The Virtuoso Product Wiki</a> is also an instance of Virtuoso demonstrating another aspect of the Content Management prowess of Virtuoso.</p> <p></p> <h4>What About Online Documentation?</h4> <p>Yep! <a href="http://docs.openlinksw.com/virtuoso/">Virtuoso Online Documentation</a> is hosted via yet another Virtuoso instance. This particular instance also attempts to demonstrate Free Text search combined with the ability to repurpose well formed content in a myriad of forms (Atom, RSS, RDF, OPML, and OCS).</p> <p></p> <h4>What about Tutorials and Demos?</h4> <p>The <a href="http://demo.openlinksw.com/tutorial/">Virtuoso Online Tutorial</a> Site has operated as a live demonstration and tutorial portal for a numbers of years. During the same timeframe (circa. 2001) we also assembled a few Screencast style demos (their look feel certainly show their age; updates are in the works).</p> <p>BTW - We have also updated the <a href="http://virtuoso.openlinksw.com/FAQ/">Virtuoso FAQ</a> and also released a number of missing <a href="http://virtuoso.openlinksw.com/Whitepapers/">Virtuoso White Papers</a> (amongst many long overdue action items).</p>
History of Programming Languages
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2006-03-15#940
2006-03-15T04:12:00Z
2006-06-22T08:56:58-04:00
<a href="http://www.oreilly.com/news/graphics/prog_lang_poster.pdf">History of Programming Languages Poster</a>.
Ted Nelson's Perspective on Technology Lock-in
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2006-02-15#935
2006-02-15T19:50:41Z
2006-06-22T08:56:58-04:00
<p> <a href="http://www.invisiblerevolution.net/ted-bar-it/top-level.html">Ted Nelson expresses technology lock-in dislike</a>. This applies to Operating System, Programming Language, Database, or any other forms. </p> Amen! <a href="http://www.openlinksw.com:8889/index.vspx?tag=zigzag" rel="tag" style="display:none;">zigzag</a><a href="http://www.openlinksw.com:8889/index.vspx?tag=xanadu" rel="tag" style="display:none;">xanadu</a><a href="http://www.openlinksw.com:8889/index.vspx?tag=semantic_web" rel="tag" style="display:none;">semantic_web</a><a href="http://www.openlinksw.com:8889/index.vspx?tag=semweb" rel="tag" style="display:none;">semweb</a><a href="http://www.openlinksw.com:8889/index.vspx?tag=visionary" rel="tag" style="display:none;">visionary</a><a href="http://www.openlinksw.com:8889/index.vspx?tag=history" rel="tag" style="display:none;">history</a><a href="http://www.openlinksw.com:8889/index.vspx?tag=hypertext" rel="tag" style="display:none;">hypertext</a><a href="http://www.openlinksw.com:8889/index.vspx?tag=hyperlink" rel="tag" style="display:none;">hyperlink</a>
Video: Tribute to Innovation (featuring: Doug Engelbart)
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2006-02-15#934
2006-02-15T19:08:55Z
2006-07-21T07:22:48.000001-04:00
A really nice <a href="http://www.invisiblerevolution.net/index-video-web.html">video tribute to Doug Engelbart</a> and the fundamental challenges of seeing way ahead of your time (aka. Prescience) :-) <a href="http://www.openlinksw.com:8889/index.vspx?tag=semantic_web" rel="tag" style="display:none;">semantic_web</a><a href="http://www.openlinksw.com:8889/index.vspx?tag=semweb" rel="tag" style="display:none;">semweb</a><a href="http://www.openlinksw.com:8889/index.vspx?tag=visionary" rel="tag" style="display:none;">visionary</a><a href="http://www.openlinksw.com:8889/index.vspx?tag=history" rel="tag" style="display:none;">history</a><a href="http://www.openlinksw.com:8889/index.vspx?tag=hypertext" rel="tag" style="display:none;">hypertext</a><a href="http://www.openlinksw.com:8889/index.vspx?tag=hyperlink" rel="tag" style="display:none;">hyperlink</a>
A Sketch of Database History
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2005-11-04#893
2005-11-04T21:14:55Z
2006-06-22T08:56:58-04:00
I just stumbled across a 2003 article titled: <a href="http://math.hws.edu/vaughn/cpsc/343/2003/history.html">A Sketch of Database History</a>. A pretty good read for those interested in this very important technology.
Clone the Google APIs: Kill That Noise
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2005-11-03#892
2005-11-03T22:44:04Z
2006-06-22T08:56:58-04:00
<p>I am kinda scratching my head a little re. the "Clone Google APIs" call; especially as Amazon's <a href="http://opensearch.a9.com/">A9</a> already provides <a href="http://opensearch.a9.com/docs/howto.jsp">infrastructure for generic search</a>. A9 is open at both ends; you can consume search services via a RESTian API or plug your search engine into A9 (playing the role of A9 search service provider). </p><p>Quick Example using my blog: </p><ul>1. <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127">My Blog's Search Page</a> (note it support Full Text and XPath/XQuery)</ul><ul>2. <a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=#39web%202.0#39&type=text&output=html">Search on pattern 'Web 2.0'</a> via my Blog's Search Engine</ul><ul>3. <a href="http://en.wikipedia.org/wiki/Hacktivism" xmlns:n0="http" n0:="http:" a9.com="a9.com" search="search" morecolumns.jsp="morecolumns.jsp" a="a">Hactivism</a>" regarding this matter. Certainly worth a full-post-scrape for my ongoing content annotation efforts (see <a href="http://www.openlinksw.com/blog/~kidehen/index.vspx?page=linkblog">Linkblog</a> and <a href="http://www.openlinksw.com/blog/~kidehen/index.vspx?page=summary">BlogSummary</a>). <p>Digest the rest of Dare's post:</p><blockquote><p><a href="http://www.25hoursaday.com/weblog/PermaLink.aspx?guid=3faf48bb-cf43-4fad-9145-cd749bd0288e">Clone the Google APIs: Kill That Noise</a>: "</p><p> Yesterday Dave Winer wrote in a post about <a href="http://www.scripting.com/2005/11/02.html#When:2:31:38PM">cloning the Google API</a> Dave Winer wrote </p><blockquote><i>Let's make the <a href="http://www.clonethegoogleapi.com/">Google API an open standard</a>. Back in 2002, Google took a bold first step to enable open architecture search engines, by creating an API that allowed developers to build applications on top of their search engine. However, there were severe limits on the capacity of these applications. So we got a good demo of what might be, now three years later, it's time for the real thing.<br /><br /></i></blockquote>and earlier that <br /><blockquote><i>If you didn't get a chance to hear <a href="http://www.scripting.com/2005/11/01.html#When:12:26:58AM">yesterday's podcast</a>, it recommends that Microsoft clone the <a href="http://davenet.scripting.com/2002/04/13/whatsNextAfterTheGoogleApi">Google API</a> for search, without the keys, and without the limits. When a developer's application generates a lot of traffic, buy him a plane ticket and dinner, and ask how you both can make some money off their excellent booming application of search. This is something Google can't do, because search is their cash cow. That's why Microsoft should do it. And so should Yahoo. Also, there's no doubt Google will be competing with Apple soon, so they should be also thinking about ways to devalue Google's advantage.</i></blockquote><blockquote></blockquote><p> This doesn't seem like a great idea to me for a wide variety of reasons but first, let's start with a history lesson before I tackle this specific issue </p><p><b>A Trip Down Memory Lane</b><br /> This history lesson <strike>used to be in</strike> is in a post entitled <a href="http://web.archive.org/web/20041011135623/http://www.evhead.com/archives/2003_05_10_archive_default.asp">The Tragedy of the API</a> by <a href="http://www.evhead.com/">Evan Williams</a> <strike>but seems to be gone now</strike>. Anyway, back in the early days of blogging the folks at Pyra [which eventually got bought by Google] created the <a href="http://www.blogger.com/developers/api/1_docs/">Blogger API</a> for their service. Since Blogspot/Blogger was a popular service, a the number of applications that used the API quickly grew. At this point Dave Winer decided that since the Blogger API was so popular he should implement it in his weblogging tools but then he decided that he didn't like some aspects of it such as application keys (sound familiar?) and did without them in his version of the API. Dave Winer's version of the Blogger API became the <a href="http://www.xmlrpc.com/metaWeblogApi">MetaWeblog API</a>. These APIs became de facto standards and a number of other weblogging applications implemented them. </p><p> After a while, the folks at Pyra decided that their API needed to evolve due to various flaws in its design. As Diego Doval put it in his post <a href="http://www.dynamicobjects.com/d2r/archives/001921.html">a review of blogging APIs</a>, <i>The Blogger API is a joke, and a bad one at that</i>. This lead to the creation of the <a href="http://www.blogger.com/developers/api/documentation20.html">Blogger API 2.0</a>. At this point a heated debate erupted online where Dave Winer berated the Blogger folks for deviating from an industry standard. The irony of flaming a company for coming up with a v2 of their own API seemed to be lost on many of the people who participated in the debate. Eventually the Blogger API 2.0 went nowhere. </p><p> Today the blogging API world is a few de facto standards based on a hacky API created by a startup a few years ago, a number of site specific APIs (<a href="http://www.livejournal.com/doc/server/ljp.csp.xml-rpc.protocol.html">LiveJournal API</a>, <a href="http://www.sixapart.com/movabletype/docs/mtmanual_programmatic.html">MovableType API</a>, etc) and a number of inconsistently implemented versions of the <a href="http://bitworking.org/projects/atom/">Atom API</a>.<br /></p><p><b>On Cloning the Google Search API</b><br /> To me the most salient point in the hijacking of the Blogger API from Pyra is that it didn't change the popularity of their service or even make Radio Userland (Dave Winer's product) catch up to them in popularity. This is important to note since this is Dave Winer's key argument for Microsoft cloning the Google API. </p><p> Off the top of my head, here are my top three technical reasons for Microsoft to ignore the calls to clone the Google Search APIs<br /></p><ol><li><p><u>Difference in Feature Set:</u> The features exposed by the API do not run the entire gamut of features that other search engines may want to expose. Thus even if you implement something that looks a lot like the Google API, you'd have to extend it to add the functionality that it doesn't provide. For example, compare the <a href="http://www.google.com/apis/reference.html">features provided by the Google API</a> to the <a href="http://developer.yahoo.net/search/">features provided by the Yahoo! search API</a>. I can count about half a dozen features in the Yahoo! API that aren't in the Google API. </p></li><li><p><u>Difference in Technology Choice:</u> The Google API uses SOAP. This to me is a phenomenally bad technical decision because it raises the bar to performing a basic operation (data retrieval) by using a complex technology. I much prefer Yahoo!'s approach of providing a RESTful API and <strike>MSN</strike> Windows Live Search's approach of providing RSS search feeds and a SOAP API for the folks who need such overkill. <br /></p></li><li><u>Unreasonable Demands:</u> A number of Dave Winer's demands seem contradictory. He asks companies to not require application keys but then advises them to contact application developers who've built high traffic applications about revenue sharing. Exactly how are these applications to be identified without some sort of application ID? As for removing the limits on the services? I guess Dave is ignoring the fact that providing services costs money, which I seem to remember is why <a href="http://www.kottke.org/05/10/weblogscom-sold-to-verisign">he sold weblogs.com to Verisign for a few million dollars</a>. I do agree that some of the limits on existing search APIs aren't terribly useful. The Google API limit of 1000 queries a day seems to guarantee that you won't be able to power a popular application with the service. <br /></li><li><p><u>Lack of Innovation:</u> Copying Google sucks. <br /></p></li></ol><p>(Via <a href="http://www.25hoursaday.com/weblog/">Dare Obasanjo aka Carnage4Life</a>.)</p></blockquote></ul>
Rise of Relational Databases
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2005-10-29#889
2005-10-29T20:33:43Z
2006-06-22T08:56:58-04:00
I suspect the subject of this post triggers the following questions: <ul>1. Don't you mean the fall/death of Relational Databases?</ul><ul>2. Does anyone use these anymore?</ul><ul>3. What are these?</ul> Relational Database Management Systems (RDBMS) are alive and kicking as expressed eloquently in this excerpt from a book titled "Funding A Revolution": <blockquote><cite></cite><p> Large-scale computer applications require rapid access to large amounts of data. A computerized checkout system in a supermarket must track the entire product line of the market. Airline reservation systems are used at many locations simultaneously to place passengers on numerous flights on different dates. Library computers store millions of entries and access citations from hundreds of publications. Transaction processing systems in banks and brokerage houses keep the accounts that generate international flows of capital. World Wide Web search engines scan thousands of Web pages to produce quantitative responses to queries almost instantly. Thousands of small businesses and organizations use databases to track everything from inventory and personnel to DNA sequences and pottery shards from archaeological digs.</p><p>Thus, databases not only represent significant infrastructure for computer applications, but they also process the transactions and exchanges that drive the U.S. economy. </p></blockquote><p>My only addition to the excerpt above is that the impact of databases extends beyond the U.S. economy. We are talking about the global economy. And this will be so for all of time!</p><p>I came across this page while enriching the links in one of my earlier "<a href="http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=history&type=text&output=html">history</a>" related posts about <a href="http://www.openlinksw.com/weblog/kidehen@openlinksw.com/127/index.vspx?page=&id=266">Relational Database Technology pioneers</a>. During this effort I also stumbled across another historic document titled: "<a href="http://www.mcjones.org/System_R/SQL_Reunion_95/sqlr95.html">1995 SQL Reunion</a>".</p>
Breaking the Web Wide Open!
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2005-10-26#882
2005-10-26T19:28:47Z
2006-06-22T08:56:58-04:00
<p><a href="http://marc.blogs.it/">Marc Canter</a>'s <a href="http://marc.blogs.it/archives/2005/10/breaking_the_we.html">Breaking the Web Wide Open! </a> article is something I found pretty late (by my normal discovery standards). This was partly due to the pre- and post- Web 2.0 event noise levels that have dumped the description of an important industry inflection into the "Bozo Bin" of many. Personally, I think we shouldn't confuse the Web 2.0 traditional-pitch-fest conference with an attempt to identify an important industry inflection).</p><p> Anyway, Marc's article is a very refreshing read because it provides a really good insight into the general landscape of a rapidly evolving Web alongside genuine appreciation of our broader timeless pursuit of "Openness". </p><p>To really help this document provide additional value have scrapped the content of the original post and dumped it below so that we can appreciate the value of the links embedded within the article (note: thanks to Virtuoso I only had to paste the content into my blog, the extraction to my <a href="http://www.openlinksw.com/blog/~kidehen/index.vspx?page=linkblog">Linkblog</a> and <a href="http://www.openlinksw.com/blog/~kidehen/index.vspx?page=summary">Blog Summary</a> Pages are simply features of my <a href="http://www.openlinksw.com/virtuos">Virtuoso </a>based Blog Engine):</p><blockquote><h3 class="hed2" style="padding-bottom: 10px">Breaking the Web Wide Open! (complete story)</h3><p>Even the web giants like AOL, Google, MSN, and Yahoo need to observe these open standards, or they'll risk becoming the "walled gardens" of the new web and be coolio no more.</p><p class="byline"><b><a href="http://community.alwayson-network.com/cgi-bin/WebObjects/AlwaysOn.woa/wa/display?id=9254:Person">Marc Canter</a></b> [<a href="http://community.alwayson-network.com/cgi-bin/WebObjects/AlwaysOn.woa/wa/display?id=9254:Person"><b>Broadband Mechanics, Inc.</b></a>] | POSTED: 09.26.05 @12:00</p><table width="100%" border="0" cellspacing="0" cellpadding="0"><tr><td valign="TOP" class="copy1"><img src="http://community.alwayson-network.com/ao/images/thumb/19433429363e7cd6b1ecfb7.jpg" align="LEFT" border="0" width="80" style="margin: 0px 10px 5px 0px" alt="" /><i><b>Editorial Note:</b> Several months ago, AlwaysOn got a personal invitation from Yahoo founder Jerry Yang "to see and give us feedback on our new social media product, y!360." We were happy to oblige and dutifully showed up, joining a conference room full of hard-core bloggers and new, new media types. The geeks gave Yahoo 360 an overwhelming thumbs down, with comments like, "So the only services I can use within this new network are Yahoo services? What if I don't use Yahoo IM?" In essence, the Yahoo team was booed for being "closed web," and we heartily agreed. With Yahoo 360, Yahoo continues building its own "walled garden" to control its 135 million customersÂan accusation also hurled at AOL in the early 1990s, before AOL migrated its private network service onto the web. As the</i> <a href="http://bernardmoon.blogspot.com/2005/08/yahoos-personality-crisis.html" target="_blank">Economist<i> recently noted</i></a>, "Yahoo, in short, has old media plans for the new-media era."<br /><br />The irony to our view here is, of course, that today's AO Network is also a "closed web." In the end, Mr. Yang's thoughtful invitation and our ensuing disappointment in his new service led to the assignment of this article. It also confirmed our existing plan to completely revamp the AO Network around open standards. To tie it all together, we recruited the chief architect of our new site, <a href="http://www.corante.com/amateur/articles/20030211-3564.html" target="_blank">the notorious Marc Canter</a>, to pen this piece. We look forward to our reader feedback.<br /><br /><b>Breaking the Web Wide Open!</b><br />By Marc Canter<br /><br />For decades, "walled gardens" of proprietary standards and content have been the strategy of dominant players in mainframe computer software, wireless telecommunications services, and the World Wide WebÂit was their successful lock-in strategy of keeping their customers theirs. But like it or not, those walls are tumbling down. Open web standards are being adopted so widely, with such value and impact, that the web giantsÂAmazon, AOL, eBay, Google, Microsoft, and YahooÂare facing the difficult decision of opening up to what they don't control.<br /><br />The online world is evolving into a new open web (sometimes called the Web 2.0), which is all about being personalized and customized for each user. Not only open source software, but <i>open standards</i> are becoming an essential component. <br /><br />Many of the web giants have been using open source software for years. Most of them use at least parts of the <a href="http://www.onlamp.com/pub/a/onlamp/2001/01/25/lamp.html" target="_blank">LAMP</a> (Linux, Apache, MySQL, Perl/Python/PHP) stack, even if they aren't well-known for giving back to the open source community. For these incumbents that grew big on proprietary web services, the methods, practices, and applications of open source software development are difficult to fully adopt. And the next open source movementsÂwhich will be as much about open standards as about codeÂwill be a lot harder for the incumbents to exploit.<br /><br />While the incumbents use cheap open source software to run their back-ends systems, their business models largely depend on proprietary software and algorithms. But our view a new slew of open software, open protocols, and open standards will confront the incumbents with the classic <i><a href="http://www.businessweek.com/chapter/christensen.htm" target="_blank">Innovator's Dilemma</a></i>. Should they adopt these tools and standards, painfully cannibalizing their existing revenue for a new unproven concept, or should they stick with their currently lucrative model with the risk that eventually a bunch of upstarts eat their lunch? <br /><br />Credit should go to several of the web giants who have been making efforts to "open up." Google, Yahoo, eBay, and Amazon all have Open APIs (Application Programming Interfaces) built into their data and systems. Any software developer can access and use them for whatever creative purposes they wish. This means that the API provider becomes an open platform for everyone to use and build on top of. This notion has expanded like wildfire throughout the blogosphere, so nowadays, Open APIs are pretty much required.<br /><br />Other incumbents also have open strategies. AOL has got the RSS religion, <a href="http://www.siliconbeat.com/entries/2005/07/27/aol_gets_rss_religion_with_my_aoland_feedsters_help.html" target="_blank">providing a feedreader and RSS search</a> in order to escape the "walled garden of content" stigma. <a href="http://www.apple.com/podcasting/" target="_blank">Apple now incorporates podcasts</a>, the "personal radio shows" that are latest rage in audio narrowcasting, into iTunes. Even Microsoft is supporting open standards, for example <a href="http://www.microsoft.com/technet/prodtechnol/winxppro/plan/rtcprot.mspx#EKAA" target="_blank">by endorsing SIP (Session Initiation Protocol) for internet telephony and conferencing</a> over Skype's proprietary format or one of its own devising.<br /><br />But new open standards and protocols are in use, under construction, or being proposed every day, pushing the envelope of where we are right now. Many of these standards are coming from startup companies and small groups of developers, not from the giants. Together with the Open APIs, those new standards will contribute to a new, open infrastructure. Tens of thousands of developers will use and improve this open infrastructure to create new kinds of web-based applications and services, to offer web users a highly personalized online experience.<br /><br /><b>A Brief History of Openness</b><br /><br />At this point, I have to admit that I am not just a passive observer, full-time journalist or "just some blogger"Âbut an active evangelist and developer of these standards. It's the vision of "open infrastructure" that's driving <a href="http://www.broadbandmechanics.com/bbm2005.htm" target="_blank">my company </a> and the reason why I'm writing this article. This article will give you some of the background behind on these standards, and what the evolution of the next generation of open standards will look like.<br /><br />Starting back in the 1980s, establishing a software standard was a key strategy for any software company. My former company, MacroMind (which became Macromedia), achieved this goal early on with Director. As <a href="http://webmonkey.wired.com/webmonkey/99/27/index3a_page6.html?tw=multimedia" target="_blank">Director evolved into Flash</a>, the world saw that other companies besides Microsoft, Adobe, and Apple could establish true cross-platform, independent media standards.<br /><br />Then <a href="http://www.w3.org/People/Berners-Lee/" target="_blank">Tim Berners-Lee</a> and <a href="http://www.ibiblio.org/pioneers/andreesen.html" target="_blank">Marc Andreessen</a> came along, and changed the rules of the software business and of entrepreneurialism. No matter how entrenched and "standardized" software was, the rug could still get pulled out from under it. <a href="http://geekphilosopher.com/MainPage/WebBrowserWars.htm?q=Stocks" target="_blank">Netscape did it to Microsoft, and then Microsoft did it <i>back</i> to Netscape</a>. The web evolved, and lots of standards evolved with it. The leading open source standards (such as the LAMP stack) became widely used alternatives to proprietary closed-source offerings. <br /><br />Open standards are more than just technology. Open standards mean sharing, empowering, and community support. Someone floats a new idea (or <a href="http://en.wikipedia.org/wiki/Meme" target="_blank">meme</a>) and the community runs with it â with each person making their own contributions to the standard â evolving it without a moment's hesitation about "giving away their intellectual property."<br /><br />One good example of this was <a href="http://www.sifry.com/alerts/" target="_blank">Dave Sifry</a>, who built the Technorati blog-tracking technology inspired by the <a href="http://www.myelin.co.nz/ecosystem/" target="_blank">Blogging Ecosystem</a>, a weekend project by young hacker <a href="http://marc.blogs.it/archives/2005/07/phil_pearson_jo.html" target="_blank">Phil Pearson</a>. Dave liked what he saw and he ran with itÂturning Technorati into what it is today.<br /><br /><a href="http://en.wikipedia.org/wiki/Dave_Winer" target="_blank">Dave Winer</a> has contributed enormously to this area of open standards. He defined and personally created several open standards and protocolsÂsuch as RSS, OPML, and XML-RPC. Dave has also <a href="http://newhome.weblogs.com/historyOfWeblogs" target="_blank">helped build</a> the blogosphere through his enthusiasm and passion.<br /><br />By 2003, hundreds of programmers were working on creating and establishing new standards for almost everything. The best of these new standards have evolved into compelling web services platforms â such as <a href="http://del.icio.us/" target="_blank">del.icio.us</a>, <a href="http://webjay.org/about" target="_blank">Webjay</a>, or <a href="http://www.flickr.com/photos/tags/ao2005/" target="_blank">Flickr</a>. Some have even spun off formal standards â like XSPF (a standard for playlists) or instant messaging standard XMPP (also known as Jabber).<br /><br />Today's Open APIs are complemented by standardized SchemasÂthe structure of the data itself and its associated meta-data. Take for example a <a href="http://www.ipodder.org/whatIsPodcasting" target="_blank">podcasting feed</a>. It consists of: a) the radio show itself, b) information on who is on the show, what the show is about and how long the show is (the meta-data) and also c) API calls to retrieve a show (a single feed item) and play it from a specified server. <br /><br />The combination of Open APIs, standardized schemas for handling meta-data, and an industry which agrees on these standards are breaking the web wide open right now. So what new open standards should the web incumbentsÂand youÂbe watching? Keep an eye on the following developments:<br /><br /><b>Identity<br />Attention<br />Open Media<br />Microcontent Publishing<br />Open Social Networks<br />Tags<br />Pinging <br />Routing<br />Open Communications<br />Device Management and Control</b><br /><br /><br /><b>1. Identity</b><br /><br />Right now, you don't really control your own online identity. At the core of just about every online piece of software is a membership system. Some systems allow you to browse a site anonymouslyÂbut unless you register with the site you can't do things like search for an article, post a comment, buy something, or review it. The problem is that each and every site has its own membership system. So you constantly have to register with new systems, which cannot share dataÂeven you'd want them to. By establishing a <a href="http://www.wired.com/news/privacy/0,1848,68329-2,00.html?tw=wn_story_page_next1" target="_blank">"single sign-on" standard</a>, disparate sites can allow users to freely move from site to site, and let them control the movement of their personal profile data, as well as any other data they've created. <br /><br />With <a href="http://www.thehindubusinessline.com/2005/01/03/stories/2005010301440200.htm" target="_blank">Passport, Microsoft unsuccessfully attempted</a> to force its proprietary standard on the industry. Instead, a world is evolving where most people assume that users want to control their own data, whether that data is their profile, their blog posts and photos, or some collection of their past interactions, purchases, and recommendations. As long as users can control their digital identity, any kind of service or interaction can be layered on top of it.<br /><br /><a href="http://www.identity20.com/media/OSCON2005/" target="_blank">Identity 2.0</a> is all about users controlling their own profile data and becoming their own agents. This way the users themselves, rather than other intermediaries, will profit from their ID info. Once developers start offering single sign-on to their users, and users have trusted places to store their dataÂwhich respect the limits and provide access controls over that data, users will be able to access personalized services which will understand and use their personal data.<br /><br />Identity 2.0 may seem like some geeky, visionary future standard that isn't defined yet, but by putting each user's digital identity at the core of all their online experiences, Identity 2.0 is becoming the cornerstone of the new open web. <br /><br /><b>The Initiatives:</b><br />Right now, Identity 2.0 is under construction through various efforts from Microsoft (the <a href="http://msdn.microsoft.com/webservices/webservices/understanding/advancedwebservices/default.aspx?pull=/library/en-us/dnwebsrv/html/identitymetasystem.asp" target="_blank">"InfoCard" component built into the Vista operating system</a> and its "<a href="http://garage.docsearls.com/node/605" target="_blank">Identity Metasystem</a>"), <a href="http://sxip.com" target="_blank">Sxip Identity</a>, <a href="http://www.identtycommons.net" target="_blank">Identity Commons</a>, <a href="http://www.projectliberty.org/" target="_blank">Liberty Alliance</a>, <a href="http://lid.netmesh.org/" target="_blank">LID</a> (NetMesh's Lightweight ID), and SixApart's <a href="http://openid.net/" target="_blank">OpenID</a>.<br /><br /><b>More Movers and Shakers:</b><br />Identity Commons and <a href="http://www.identitywoman.net" target="_blank">Kaliya Hamlin</a>, Sxip Identity and <a href="http://blame.ca/dick/" target="_blank">Dick Hardt</a>, the <a href="http://www.identitygang.org/" target="_blank"> Identity Gang</a> and <a href="http://www.searls.com/dochome.html#Bio" target="_blank">Doc Searls</a>, Microsoft's <a href="http://www.identityblog.com/" target="_blank">Kim Cameron</a>, <a href="http://www.craigburton.com/" target="_blank">Craig Burton</a>, <a href="http://phil.windley.org/" target="_blank">Phil Windley</a>, and <a href="http://slashdot.org/article.pl?sid=05/07/05/2020221&from=rss" target="_blank">Brad Fitzpatrick</a>, to name a few.<br /><br /><br /><b>2. Attention</b><br /><br />How many readers know what their online attention is worth? If you don't, Google and Yahoo doÂthey make their living off our attention. They know what we're searching for, happily turn it into a keyword, and sell that keyword to advertisers. They make money off our attention. We don't. <br /><br />Technorati and friends proposed <a href="http://blogs.zdnet.com/Gillmor/index.php?p=74" target="_blank">an attention standard, Attention.xml</a>, designed to "help you keep track of what you've read, what you're spending time on, and what you should be paying attention to." <a href="http://attentiontrust.org/" target="_blank">AttentionTrust</a> is an effort by <a href="http://blogs.zdnet.com/Gillmor/?p=132" target="_blank">Steve Gillmor</a> and <a href="http://majestic.typepad.com/seth/2005/07/attentiontrusto.html" target="_blank">Seth Goldstein </a>to standardize on how captured end-user performance, browsing, and interest data are used. <br /><br />Blogger <a href="http://worcester.typepad.com/pc4media/2005/07/attentiontrusto_1.html" target="_blank">Peter Caputa gives a good summary</a> of AttentionTrust: <blockquote>"As we use the web, we reveal lots of information about ourselves by what we pay attention to. Imagine if all of that information could be stored in a nice neat little xml file. And when we travel around the web, we can optionally share it with websites or other people. We can make them pay for it, lease it ... we get to decide who has access to it, how long they have access to it, and what we want in return. And they have to tell us what they are going to do with our Attention data."</blockquote><br />So when you give your attention to sites that adhere to the AttentionTrust, your attention rights (<i>you own your attention, you can move your attention, you can pay attention and be paid for it</i>, and <i>you can see how your attention is used</i>) are guaranteed. Attention data is crucial to the future of the open web, and Steve and Seth are making sure that no one entity or oligopoly controls it. <br /><br /><b>Movers and Shakers:</b><br /><a href="http://blogs.zdnet.com/Gillmor/" target="_blank">Steve Gillmor</a>, <a href="http://majestic.typepad.com/about.html" target="_blank">Seth Goldstein</a>, <a href="http://www.sifry.com/alerts/" target="_blank">Dave Sifry</a> and the <a href="http://developers.technorati.com/wiki/attentionxml" target="_blank">other Attention.xml folks</a>. <br /><br /><br /><b>3. Open Media</b><br /><br />Proprietary media standardsÂFlash, Windows Media, and QuickTime, to name a few Âhelped liven up the web. But they are proprietary standards that try to keep us locked in, and they weren't created from scratch to handle today's online content. That's why, for many of us, an Open Media standard has been a holy grail. Yahoo's new Media RSS standard brings us one step closer to achieving open media, as do <a href="http://www.vorbis.com/faq/#what" target="_blank">Ogg Vorbis</a> audio codecs, <a href="http://webjay.org/" target="_blank">XSPF playlists</a>, or <a href="http://musicbrainz.org/" target="_blank">MusicBrainz</a>. And several sites offer digital creators not only a place to store their content, but also to sell it. <br /><br /><a href="http://search.yahoo.com/mrss" target="_blank">Media RSS </a>(being developed by Yahoo with help from the community) extends RSS and combines it with "RSS enclosures" Âadds metadata to any media itemÂto create a comprehensive solution for media "narrowcasters." To gain acceptance for Media RSS, Yahoo knows it has to work with the community. As an active member of this community, I can tell you that we'll create Media RSS equivalents for <a href="http://www.xml.com/pub/a/2001/01/24/rdf.html" target="_blank">rdf</a> (an alternative subscription format) and <a href="http://www.atomenabled.org/" target="_blank">Atom</a> (yet <i>another</i> subscription format), so no one will be able to complain that Yahoo is picking sides in format wars.<br /><br />When Yahoo announced the purchase of Flickr, Yahoo founder Jerry Yang insinuated that Yahoo is acquiring "open DNA" to turn Yahoo into <a href="http://www.flickr.com/services/api/" target="_blank">an open standards player</a>. Yahoo is showing what happens when you take a multi-billion dollar company and make openness one of its core valuesÂso Google, beware, even if Google does have more research fellows and Ph.D.s. <br /><br />The open media landscape is far and wide, reaching from game machine hacks and mobile phone downloads to PC-driven bookmarklets, players, and editors, and it includes many other standardization efforts. <a href="http://www.xspf.org/" target="_blank">XSPF</a> is an open standard for playlists, and MusicBrainz is an alternative to the proprietary (and originally effectively stolen) database that <a href="http://en.wikipedia.org/wiki/Gracenote" target="_blank">Gracenote</a> licenses. <br /><br /><a href="http://www.ourmedia.org/" target="_blank">Ourmedia.org</a> is a community front-end to Brewster Kahle's <a href="http://www.archive.org" target="_blank">Internet Archive</a>. Brewster has promised free bandwidth and free storage forever to any content creators who choose to share their content via the Internet Archive. Ourmedia.org is providing an easy-to-use interface and community to get content in and out of the Internet Archive, giving ourmedia.org users the ability to share their media anywhere they wish, without being locked into a particular service or tool. Ourmedia plans to offer open APIs and an open media registry that interconnects other open media repositories into a DNS-like registry (just like the www domain system), so folks can browse and discover open content across many open media services. Systems like <a href="http://www.brightcove.com/" target="_blank">Brightcove</a> and <a href="http://www.evhead.com/2005/02/how-odeo-happened.asp" target="_blank">Odeo</a> support the concept of an open registry, and hope to work with digital creators to sell their work to fulfill the financial aspect of <a href="http://en.wikipedia.org/wiki/The_Long_Tail" target="_blank">the "Long Tail."</a><br /><br /><b>More Movers and Shakers:</b><br /><a href="http://creativecommons.org/about/people" target="_blank">Creative Commons</a>, the <a href="http://www.omn.org/" target="_blank">Open Media Network</a>, <a href="http://www.momentshowing.net/about.html" target="_blank">Jay Dedman</a>, <a href="http://ryanedit.blogspot.com/" target="_blank">Ryanne Hodson</a>, <a href="http://michaelverdi.com/index.php" target="_blank">Michael Verdi</a>, <a href="http://www.chapmanlogic.com/blog/aboutEli.html" target="_blank">Eli Chapman</a>, <a href="http://www.unmediated.org/" target="_blank">Kenyatta Cheese</a>, <a href="http://www.itconversations.com/about.html" target="_blank">Doug Kaye</a>, <a href="http://www.wired.com/wired/archive/13.09/yahoo.html" target="_blank">Brad Horowitz</a>, <a href="http://webjay.org/about#colophon" target="_blank">Lucas Gonze</a>, <a href="http://musicbrainz.org/wd/MusicBrainzBio" target="_blank">Robert Kaye</a>, <a href="http://www.lifewithalacrity.com/" target="_blank">Christopher Allen</a>, <a href="http://en.wikipedia.org/wiki/Brewster_Kahle" target="_blank">Brewster Kahle</a>, <a href="http://www.newmediamusings.com/" target="_blank">JD Lasica</a>, and indeed, <a href="http://www.corante.com/amateur/articles/20030211-3564.html" target="_blank">Marc Canter</a>, among others.<br /><br /><br /><b>4. Microcontent Publishing</b><br /><br />Unstructured content is cheap to create, but hard to search through. Structured content is expensive to create, but easy to search. <a href="http://developers.technorati.com/wiki/MicroFormats" target="_blank">Microformats</a> resolve the dilemma with simple structures that are cheap to use and easy to search.<br /><br />The first kind of widely adopted microcontent is blogging. Every post is an encapsulated idea, addressable via a URL called a permalink. You can syndicate or subscribe to this microcontent using RSS or an RSS equivalent, and news or blog aggregators can then display these feeds in a convenient readable fashion. But a blog post is just a block of unstructured textânot a bad thing, but just a first step for microcontent. When it comes to<i>structured</i> data, such as personal identity profiles, product reviews, or calendar-type event data, RSS was not designed to maintain the integrity of the structures. <br /><br />Right now, blogging doesn't have the underlying structure necessary for full-fledged microcontent publishing. But that will change. Think of local information services (such as movie listings, event guides, or restaurant reviews) that any college kid can access and use in her weekend programming project to create new services and tools.<br /><br />Today's blogging tools will evolve into microcontent publishing systems, and will help spread the notion of structured data across the blogosphere. New ways to store, represent and produce microcontent will create new standards, such as <a href="http://structuredblogging.org/" target="_blank">Structured Blogging</a> and <a href="http://microformats.org/" target="_blank">Microformats</a>. Microformats differ from RSS feeds in that you can't subscribe to them. Instead, Microformats are embedded into webpages and discovered by search engines like Google or Technorati. Microformats are creating common definitions for "What is a review or event? What are the specific fields in the data structure?" They can also specify what we can do with all this information.<a href="http://www.opml.org/spec" target="_blank">OPML (Outline Processor Markup Language)</a> is a hierarchical file format for storing microcontent and structured data. It was developed by <a href="http://en.wikipedia.org/wiki/Dave_Winer" target="_blank">Dave Winer</a> of RSS and podcast fame.<br /><br />Events are one popular type of microcontent. <a href="http://www.openevents.com" target="_blank">OpenEvents</a> is already working to create shared databases of standardized events, which would get used by a new generation of event portalsâsuch as <a href="http://eventful.com/gotevents/" target="_blank">Eventful/EVDB</a>, <a href="http://upcoming.org/" target="_blank">Upcoming.org</a>, and <a href="http://www.whizspark.com/" target="_blank">WhizSpark</a>. The idea of OpenEvents is that event-oriented systems and services can work together to establish shared events databases (and associated APIs) that any developer could then use to create and offer their own new service or application. <a href="http://marc.blogs.it/archives/2005/04/rvw_redux_openr.html" target="_blank">OpenReviews</a> is still in the conceptual stage, but it would make it possible to provide open alternatives to closed systems like Epinions, and establish a shared database of local and global reviews. Its shared open servers would be filled with all sorts of reviews for anyone to access. <br /><br />Why is this important? Because I predict that in the future, 10 times more people will be writing reviews than maintaining their own blog. The list of possible microcontent standards goes on: OpenJobpostings, OpenRecipes, and even OpenLists. Microsoft <a href="http://www.reallysimplesyndication.com/2005/06/22" target="_blank">recently revealed</a> that it has been working on an important new kind of microcontent: Listsâso OpenLists will attempt to establish standards for the <i>kind</i> of lists we all use, such as lists of Links, lists of To Do Items, lists of People, Wish Lists, etc.<br /><br /><b>Movers and Shakers:</b><br /><a href="http://tantek.com/log/2005/09.html" target="_blank">Tantek Ãelik</a> and <a href="http://en.wikipedia.org/wiki/Kevin_Marks" target="_blank">Kevin Marks</a> of <a href="http://developers.technorati.com/wiki/MicroFormats" target="_blank">Technorati</a>, <a href="http://dannyayers.com/" target="_blank">Danny Ayers</a>, <a href="http://www.meyerweb.com/" target="_blank">Eric Meyer</a>, <a href="http://photomatt.net/" target="_blank">Matt Mullenweg</a>, <a href="http://zlab.commerce.net/" target="_blank">Rohit Khare</a>, <a href="http://ifindkarma.typepad.com/relax/" target="_blank">Adam Rifkin</a>, <a href="http://www.sivas.com/aleene/" target="_blank">Arnaud Leene</a>, <a href="http://radio.weblogs.com/0110772/" target="_blank">Seb Paquet</a>, <a href="http://hublog.hubmed.org/" target="_blank">Alf Eaton</a>, <a href="http://www.myelin.co.nz/post/" target="_blank">Phil Pearson</a>, <a href="http://www.joereger.com/" target="_blank">Joe Reger</a>, <a href="http://bobwyman.pubsub.com/" target="_blank">Bob Wyman</a> among others.<br /><br /><br /><b>5. Open Social Networks</b><br /><br />I'll never forget the first time I met <a href="http://www.jabrams.com/" target="_blank">Jonathan Abrams</a>, the founder of Friendster. He was arrogant and brash and he claimed he "<i>owned</i>" all his users, and that he was going to monetize them and make a fortune off them. This attitude robbed Friendster of its momentum, letting MySpace, Facebook, and other social networks take Friendster's place.<br /><br />Jonathan's notion of social networks as a way to control users is typical of the Web 1.0 business model and its attitude towards users in general. Social networks have become one of the battlegrounds between old and new ways of thinking. Open standards for Social Networking will define those sides very clearly. Since meeting Jonathan, I have been working towards finding and establishing open standards for social networks. Instead of closed, centralized social networks with 10 million people in them, the goal is making it possible to have 10 million social networks that each have 10 people in them.<br /><br />FOAF (which stands for Friend Of A Friend, and describes people and relationships in a way that computers can parse) is a schema to represent not only your personal profile's meta-data, but your social network as well. Thousands of researchers use the <a href="http://www.foaf-project.org/" target="_blank">FOAF schema</a> in their "Semantic Web" projects to connect people in all sorts of new ways. <a href="http://gmpg.org/xfn/" target="_blank">XFN</a> is a microformat standard for representing your social network, while <a href="http://www.imc.org/pdi/" target="_blank">vCard</a> (long familiar to users of contact manager programs like Outlook) is a microformat that contains your profile information. Microformats are baked into any xHTML webpage, which means that<i>any</i> blog, social network page, or any webpage in general can "contain" your social network in itÂand be used by<i>any</i> compatible tool, service or application. <br /><br />PeopleAggregator is an earlier project now being integrated into <a href="http://drupal.org/" target="_blank">open content management framework Drupal</a>. The <a href="http://www.broadbandmechanics.com/PeopleAggregator/" target="_blank">PeopleAggregator APIs</a> will make it possible to establish relationships, send messages, create or join groups, and post between different social networks. (Sneak preview: this technology will be available in the upcoming GoingOn Network.) <br /><br />All of these open social networking standards mean that inter-connected social networks will form a mesh that will parallel the blogosphere. This vibrant, distributed, decentralized world will be driven by open standards: personalized online experiences are what the new open web will be all aboutÂand what could be more personalized than people's networks?<br /><br /><b>Movers and Shakers:</b><br /><a href="http://esigler.2nw.net/" target="_blank">Eric Sigler</a>, <a href="http://lucifer.intercosmos.net/index.php?view=about" target="_blank">Joel De Gan</a>, <a href="http://crschmidt.net/" target="_blank">Chris Schmidt</a>, <a href="http://voidstar.com/" target="_blank">Julian Bond</a>, <a href="http://people.tribe.net/paul?_click_path=Application%5Btribe%5D.Person%5Bf2232c95-e123-43a3-b48d-24a5f11f09dc%5D&r=10535" target="_blank">Paul Martino</a>, <a href="http://napsterization.org/stories/archives/000513.html" target="_blank">Mary Hodder</a>, <a href="http://public.2idi.com/=Drummond.Reed" target="_blank">Drummond Reed</a>, <a href="http://danbri.org/" target="_blank">Dan Brickley</a>, <a href="http://360.yahoo.com/profile-9lciejI3aafX1stHPoIRNmkmv4EowQ--" target="_blank">Randy Farmer</a>, and <a href="http://www.kaliyasblogs.net/Iwoman/" target="_blank">Kaliya Hamlin</a>, to name a few.<br /><br /><br /><b>6. Tags</b><br /><br />Nowadays, no self-respecting tool or service can ship without <a href="http://www.salon.com/tech/feature/2005/02/08/tagging/index_np.html" target="_blank">tags</a>. Tags are keywords or phrases attached to photos, blog posts, URLs, or even video clips. These user- and creator-generated tags are an open alternative to what used to be the domain of librarians and information scientists: categorizing information and content using taxonomies. Tags are instead creating <a href="http://www.wired.com/wired/archive/13.04/view.html?pg=4" target="_blank">"folksonomies."</a><br /><br />The recently proposed OpenTags concept would be an open, community-owned version of the popular <a href="http://www.technorati.com/tag/" target="_blank">Technorati Tags service</a>. It would aggregate the usage of tags across a wide range of services, sites, and content tools. In addition to Technorati's current tag features, OpenTags would let groups of people share their tags in "<a href="http://www.zeldman.com/daily/0405d.shtml/" target="_blank">TagClouds</a>." Open tagging is likely to include some of the open identity features discussed above, to create a tag system that is resilient to spam, and yet trustable across sites all over the web.<br /><br />OpenTags owes a debt to earlier versions of shared tagging systems, which include <a href="http://www.topicexchange.com/" target="_blank">Topic Exchange</a> and something called the <a href="http://www.evectors.com/itkcollector/" target="_blank">k-collector</a>Âa knowledge management tag aggregatorÂfrom Italian company eVectors. <br /><br /><b>Movers & Shakers:</b><br /><a href="http://www.myelin.co.nz/notes/" target="_blank">Phil Pearson</a>, <a href="http://matt.blogs.it/" target="_blank">Matt Mower </a>, <a href="http://paolo.evectors.it/" target="_blank">Paolo Valdemarin</a>, and <a href="http://marc.blogs.it/archives/2005/03/opentopics.html" target="_blank">Mary Hodder</a> and <a href="http://www.equalsdrummond.name/index.php?p=39" target="_blank"> Drummond Reed</a> again, among others.<br /><br /><br /><b>7. Pinging</b><br /><br />Websites used to be mostly static. Search engines that <a href="http://en.wikipedia.org/wiki/Web_crawler" target="_blank">crawled</a> (or "spidered") them every so often did a good enough job to show reasonably current versions of your cousin's homepage or even <i>Time</i> magazine's weekly headlines. But when blogging took off, it became hard for search engines to keep up. (Google has only <a href="http://searchenginewatch.com/searchday/article.php/3548411" target="_blank">just managed</a> to offer <a href="http://www.google.com/help/about_blogsearch.html" target="_blank">blog-search functionality</a>, despite <a href="http://www.alwayson-network.com/comments.php?id=325_0_2_0_C" target="_blank">buying Blogger</a> back in early 2003.)<br /><br />To know what was new in the blogosphere, users couldn't depend on services that spidered webpages once in a while. The solution: a way for blogs themselves to automatically notify blog-tracking sites that they'd been updated. <a href="http://weblogs.com/" target="_blank">Weblogs.com</a> was the first blog "ping service": it displayed the name of a blog whenever that blog was updated. Pinging sites helped the blogosphere grow, and <a href="http://blo.gs/" target="_blank">more tools</a>, services, and portals started using pinging in new and different ways. Dozens of pinging services and sitesÂmost of which can't talk to each otherÂsprang up. <br /><br />Matt Mullenweg (the creator of open source blogging software WordPress) decided that a one-stop service for pinging was needed. He created <a href="http://pingomatic.com/" target="_blank">Ping-o-Matic</a>Âwhich aggregates ping services and simplifies the pinging process for bloggers and tool developers. With Ping-o-Matic, any developer can alert all of the industry's blogging tools and tracking sites at once. This new kind of open standard, with shared infrastructure, is a critical to the scalability of Web 2.0 services.<br /><br />As <a href="http://pingomatic.com/about/" target="_blank">Matt said</a>:<br /><blockquote>There are a number of services designed specifically for tracking and connecting blogs. However it would be expensive for all the services to crawl all the blogs in the world all the time. By sending a small ping to each service you let them know you've updated so they can come check you out. They get the freshest data possible, you don't get a thousand robots spidering your site all the time. Everybody wins.</blockquote><br /><b>Movers and Shakers:</b><br /><a href="http://photomatt.net/about/" target="_blank">Matt Mullenweg</a>, <a href="http://trainedmonkey.com/entry/2251" target="_blank">Jim Winstead</a>, <a href="http://newhome.weblogs.com/faq" target="_blank">Dave Winer</a><br /><br /><br /><b>8. Routing</b><br /><br />Bloggers used to have to manually enter the links and content snippets of blog posts or news items they wanted to blog. Today, some RSS aggregators can send a specified post directly into an associated blogging tool: as bloggers browse through the feeds they subscribe to, they can easily specify and send any post they wish to "<a href="http://www.microsoftmonitor.com/archives/010209.html" target="_blank">reblog</a>" from their news aggregator or feed reader into their blogging tool. (This is usually referred to as "<a href="http://help.blogger.com/bin/answer.py?answer=152&topic=17" target="_blank">BlogThis</a>.") As structured blogging comes into its own (see the section on Microcontent Publishing), it will be increasingly important to maintain the structural integrity of these pieces of microcontent when reblogging them. <br /><br />Promising standard <a href="http://redirectthis.com/" target="_blank">RedirectThis</a> will combine a "BlogThis"-like capability while maintaining the integrity of the microcontent. RedirectThis will let bloggers and content developers attach a simple "PostThis" button to their posts. Clicking on that button will send that post to the reader/blogger's favorite <a href="http://ecto.kung-foo.tv/archives/000990.php" target="_blank">blogging tool</a>. This favorite tool is specified at the RedirectThis web service, where users register their blogging tool of choice. RedirectThis also helps maintain the integrity and structure of microcontentÂthen it's just up to the user to prefer a blogging tool that also attains that lofty goal of microcontent integrity. <br /><br />OutputThis is another nascent web services standard, to let bloggers specify what "destinations" they'd like to have as options in their blogging tool. As new destinations are added to the service, more checkboxes would get added to their blogging toolÂallowing them to route their published microcontent to additional destinations.<br /><br /><b>Movers and Shakers:</b><br /><a href="http://reblog.org/" target="_blank">Michael Migurski</a>, <a href="http://www.gonze.com/about" target="_blank">Lucas Gonze</a><br /><br /><br /><b>9. Open Communications</b><br /><br />Likely, you've experienced the joys of finding friends on AIM or Yahoo Messenger, or the convenience of Skyping with someone overseas. Not that you're about to throw away your mobile phone or BlackBerry, but for many, also having access to Instant Messaging (IM) and Voice over IP (VoIP) is crucial. <br /><br />IM and VoIP are mainstream technologies that already enjoy the benefits of open standards. Entire industries are bornÂright this secondÂbased around these open standards. <a href="http://www.jabber.org/" target="_blank">Jabber</a> has been an open IM technology for yearsÂin fact, <a href="http://www.xmpp.org/history.html" target="_blank">as XMPP</a>, it was officially dubbed a standard by <a href="http://www.ietf.org/overview.html" target="_blank">the IETF</a>. Although becoming an <a href="http://en.wikipedia.org/wiki/IETF" target="_blank">official IETF standard</a> is usually the kiss of death, Jabber looks like it'll be around for a while, as entire generations of collaborative, work-group applications and services have been built on top of its messaging protocol. For VoIP, <a href="http://skype.com/helloagain.html" target="_blank">Skype</a> is clearly the leading standard todayÂthough one could <a href="http://socialsoftware.weblogsinc.com/entry/1234000923058521/" target="_blank">argue just how "open" it is</a> (and defenders of the IETF's <a href="http://www.cs.columbia.edu/sip/" target="_blank">SIP standard</a> often do). But it is free and user-friendly, so there won't be much argument from <i>users</i> about it being insufficiently open. Yet there may be a cloud on Skype's horizon: web behemoth Google recently released a beta of <a href="http://www.google.com/talk/developer.html" target="_blank">Google Talk, an IM client committed to open standards</a>. It currently <a href="http://radar.oreilly.com/archives/2005/08/google_talk_rel.html" target="_blank">supports XMPP, and will support SIP</a> for VoIP calls.<br /><br /><b>Movers and Shakers:</b><br /><a href="http://www.jabber.org/people/jer.shtml" target="_blank">Jeremie Miller</a>, <a href="http://www.cs.columbia.edu/~hgs/" target="_blank">Henning Schulzrinne</a>, <a href="http://www.von.com/schedule_eos11114704148.html" target="_blank">Jon Peterson</a>, <a href="http://www.pulver.com/jeff/" target="_blank">Jeff Pulver</a><br /><br /><br /><b>10. Device Management and Control</b><br /><br />To access online content, we're using more and more devices. BlackBerrys, iPods, Treos, you name it. As the web evolves, more and more different devices will have to communicate with each other to give us the content we want when and where we want it. No-one wants to be dependent on one vendor anymoreÂlike, <a href="http://www.alwayson-network.com/comments.php?id=P9409_0_6_0_C" target="_blank">say, Sony</a>Âfor their laptop, phone, MP3 player, PDA, and digital camera, so that it all works together. We need fully interoperable devices, and the standards to make that work. And to fully make use of how content is moving online content and innovative web services, those standards need to be open.<br /><br /><a href="http://en.wikipedia.org/wiki/Midi" target="_blank">MIDI (musical instrument digital interface)</a>, one of the very first open standards in music, connected disparate vendors' instruments, post-production equipment, and recording devices. But MIDI is limited, and <a href="http://www.oreillynet.com/pub/wlg/8015" target="_blank">MIDI II has been very slow to arrive</a>. Now a new standard for controlling musical devices has emerged: <a href="http://www.cnmat.berkeley.edu/OpenSoundControl/" target="_blank">OSC (Open SoundControl)</a>. This protocol is optimized for modern networking technology and inter-connects music, video and controller devices with "other multimedia devices." OSC is used by a wide range of developers, and is being taken up in the mainstream MIDI marketplace.<br /><br />Another open-standards-based device management technology is <a href="http://www.zigbee.org" target="_blank">ZigBee</a>, for building wireless intelligence and network monitoring into all kinds of devices. ZigBee is supported by many networking, consumer electronics, and mobile device companies.<br /><br /><br />   · · · · · ·   <br /><br /><b>The Change to Openness</b><br /><br />The rise of open source software and its "<a href="http://www.oreillynet.com/pub/a/oreilly/tim/articles/architecture_of_participation.html" target="_blank">architecture of participation</a>" are completely shaking up the old proprietary-web-services-and-standards approach. Sun MicrosystemsÂwhose proprietary Java standard helped define the Web 1.0Âis opening its Solaris OS and has even announced the apparent paradox of an <a href="http://blogs.zdnet.com/open-source/?p=418" target="_blank">open-source Digital Rights Management</a> system.<br /><br />Today's incumbents will have to adapt to the new openness of the Web 2.0. If they stick to their <a href="http://www.gartner.com/DisplayDocument?doc_cd=131038" target="_blank">proprietary standards</a>, code, and content, they'll become the new walled gardensÂplaces users visit briefly to retrieve data and content from enclosed data silos, but not where users "live." The incumbents' revenue models will have to change. Instead of "owning" their users, users will know they own themselves, and will expect a return on their valuable identity and attention. Instead of being locked into incompatible media formats, users will expect easy access to digital content across many platforms. <br /><br />Yesterday's web giants and tomorrow's users will need to find a mutually beneficial new balanceÂbetween open and proprietary, developer and user, hierarchical and horizontal, owned and shared, and compatible and closed. <br /><br /><br /><i>Marc Canter is an active evangelist and developer of open standards. Early in his career, Marc founded MacroMind, which became Macromedia. These days, he is CEO of Broadband Mechanics, a founding member of the Identity Gang and of ourmedia.org. Broadband Mechanics is currently developing the <a href="http://www.alwayson-network.com/comments.php?id=11262_0_1_0_C" target="_blank">GoingOn Network</a> (with the AlwaysOn Network), as well as an open platform for social networking called the PeopleAggregator.</i><br /><br />A version of the above post appears in the Fall 2005 issue of AlwaysOn's quarterly print blogozine, and ran as <a href="http://www.alwayson-network.com/comments.php?id=12063_0_1_0_C" target="_blank">a four-part series</a> on the AlwaysOn Network website.</td></tr></table><br /><p>(Via <a href="http://marc.blogs.it/">Marc's Voice</a>.)</p></blockquote>
Yet Another RSS History
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2005-10-25#880
2005-10-25T22:23:48Z
2006-06-22T08:56:58-04:00
<p><a href="http://dannyayers.com/archives/2005/10/24/yet-another-rss-history/">Yet Another RSS History</a>: "</p><p><em>[You donât expect me to work out the CSS right after making it semantic, do you?] </em></p><p>Shift to another universe. Itâs sometime in the late 1990âs. <a href="http://www.guha.com/cv.html">Ramanathan Guha</a>, <a href="http://www.tbray.org/ongoing/">Tim Bray</a>, <a href="http://scripting.com">Dave Winer</a>, <a href="http://tantek.com">Tantek Ãelik</a>, <a href="http://dan.libby.com/">Dan Libby</a> and <a href="http://www.w3.org/People/Connolly/">Dan Connolly</a> are sharing a jacuzzi*. As they sip Marghueritas, their conversation goes like this: </p><ul><li><cite>DanL</cite><br /><blockquote><p>So, weâve got this idea for publishing content thatâs a bit like <a href="http://www.w3.org/TR/NOTE-CDFsubmit.html">CDF</a>, but weâve made the system more of a service than just a desktop thing.</p></blockquote></li><li><cite>Guha</cite><br /><blockquote><p>Sounds cool. Might be a good fit with this RDF thing Iâve been working on.</p></blockquote></li><li><cite>Dave</cite><br /><blockquote><p>Hmm, Danâs stuff does sound cool, but with all due respect dude, RDF does seem a bit complicated. I really donât think the folks out in userland would get it. And they majored in graphs.</p></blockquote></li><li><cite>Tim</cite><br /><blockquote><p>Maybe we could make it a bit more straightforward, you know, like put pointy brackets around it?</p></blockquote></li><li><cite>Dave</cite><br /><blockquote><p>Straightforwardâs good. Better still, simple. They like simple.</p></blockquote></li><li><cite>Tantek</cite><br /><blockquote><p>But what about the rest of the Web, you know, like HTML?</p></blockquote></li><li><cite>DanL</cite><br /><blockquote><p>Hmm, but how do we do the timestamping kind of thing, and wrap it up in a âmicropostyâ way, the things that makes this distribution mode work?</p></blockquote></li><li><cite>Guha</cite><br /><blockquote><p>Yeah, metadata is cool. Keep the metadata.</p></blockquote></li><li><cite>Tim</cite><br /><blockquote><p>Not cheap though. The Web must be cheap. Did Andreesen show you his pictures..?</p></blockquote></li><li><cite>Dave</cite><br /><blockquote><p>â¦âMicropostyâ? you mean like my newsletter thing, but on the Web?</p></blockquote></li><li><cite>DanL</cite><br /><blockquote><p>Yep, like Cool Diary Entry of the Day</p></blockquote></li><li><cite>Tim</cite><br /><blockquote><p>But do we really need 1000 pages of spec for that?</p></blockquote></li><li><cite>Tantek</cite><br /><blockquote><p>â¦Incidentally, did you see my <a href="http://tantek.com/CSS/Examples/boxmodelhack.html">Box Model Hack?</a></p></blockquote></li><li><cite>Guha</cite><br /><blockquote><p>Yup.</p></blockquote></li><li><cite>DanL</cite><br /><blockquote><p>Yup.</p></blockquote></li><li><cite>Tim</cite><br /><blockquote><p>Yup.</p></blockquote></li><li><cite>Dave</cite><br /><blockquote><p>Yup. I explained that on DaveNet last year.</p></blockquote></li><li><cite>MarcC</cite><br /><blockquote><p>Hey! Iâve got it: âMyDigitalCocktailâ..?</p></blockquote></li><li><cite>DanC</cite><br /><blockquote><p>Hang on, that gives me <a href="http://lists.w3.org/Archives/Public/www-rdf-interest/2000Mar/0103">an idea</a>â¦</p></blockquote></li></ul><p><em>There was a tangible outcome to this conversation: a document format which supports content and unambiguous, explicit, data and metadata, timestamping and much, much more. Itâs viewable in a regular browser. Can be syndicated; can be aggregated. Unlike forgetful RSS, archives are almost always retrievable using regular HTTP methods. In this universe there was no RSS. No syndication wars. No talking-at-cross-purposes conflict between docheads and dataheads, syntax fans and model fans. No-one had to publish simple data in Byzantine RDF/XML. No-one had to deal with doubly-escaped content and silent data loss. There was no need for any new format for business cards, calendars, blogs, link lists, reviews, pet profiles. XHTML with CSS was more than enough. DanL got the MyNetscape he wanted. Tim got the simple, tight format he wanted. Guha got the AI. Tantek got to do presentations in a cool black raincoat. DanC finally got his schedule on his Palm Pilot. Dave got the credit. MarcC got the parasols and a grass skirt none of the others would admit to having brought. </em></p><p>Shift back to this universe. Check out <a href="http://microformats.org/wiki/hatom">hAtom</a>. Itâs not finished yet, but <a href="http://blogmatrix.com">David</a>âs been methodically working through the (utterly sound) <a href="http://microformats.org/wiki/process">microformats process</a>. Looks good to me. </p><p><em>* apologies for the imagery, but how else do think Silicon Valley might seem to someone raised in the cowpat-coated hills of Derbyshire?</em></p><p>PS. Apologies to everyone mentioned. And before you suggest it, blogging *is* therapy.</p>" <p>(Via <a href="http://dannyayers.com">Raw</a>.)</p>
Microsoft Gadgets, Start.com and Innovation
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2005-09-16#868
2005-09-16T17:54:52Z
2006-06-22T08:56:58-04:00
<blockquote><p><a href="http://www.25hoursaday.com/weblog/PermaLink.aspx?guid=88270766-b9e1-407b-937f-ab41edce97de">Microsoft Gadgets, Start.com and Innovation</a>: "</p><p> A lot of <a href="http://microsoftgadgets.com/blogs/gadgetnews/archive/2005/09/13/3.aspx#comments">the comments in the initial post on the Microsoft Gadgets blog</a> are complaints that the Microsoft is copying ideas from <a href="http://www.apple.com/macosx/features/dashboard/">Apple's dashboard</a>. First of all, people should give credit where it is due and acknowledge that <a href="http://www.konfabulator.com/">Konfabulator</a> is the real pioneer when it comes to desktop widgets. More importantly, the core ideas in Microsoft Gadgets were pioneered by Microsoft not Apple or Konfabulator. </p><p> From the post <a href="http://microsoftgadgets.com/blogs/gadgetnews/archive/2005/09/15/181.aspx">A Brief History of Windows Sidebar</a> by Sean Alexander </p><blockquote dir="ltr" style="MARGIN-RIGHT: 0px"><p class="MsoNormal"><b><span><?XML:NAMESPACE PREFIX = O ?>Microsoft 'Sideshow*' Research Project (2000-2001)</span></b></p><p xmlns="o"></p><p class="MsoNormal"><span>While work started prior, in September 2001, a team of Microsoft researchers <a href="http://research.microsoft.com/research/pubs/view.aspx?tr_id=488">published a paper</a> entitled, 'Sideshow: Providing peripheral awareness of important information' including findings of their project. </span><br /> ...<br /><span>The research paper provides screenshots that bear a striking resemblance to the Windows Sidebar. The paper is a good read for anyone thinking about Gadget development. For folks who have visited Microsoft campuses, you may recall the posters in elevator hallways and Sidebar running on many employees desktops. Technically one of the first teams to implement this concept </span></p><span><p class="MsoNormal"><i><span>*Internal code-name, not directly related to the official, âÂÂWindows SideShowâ¢â auxiliary display feature in Windows Vista.</span></i>></p><p class="MsoNormal"><b><span>Microsoft âÂÂLonghornâ Alpha Release (2003) </span></b></p><p xmlns="o"></p></span><p class="MsoNormal"><span>In 2003, Microsoft unveiled a new feature called, 'Sidebar' at the Microsoft Professional DeveloperâÂÂs Conference. This feature took the best concepts from Microsoft Research and applied them to a new platform code-named, 'Avalon', now formally known as Windows Presentation Foundation... </span></p><p xmlns="o"></p><p class="MsoNormal"><span></span></p><p xmlns="o"> </p><b><span>Microsoft Windows Vista PDC Release (2005)<p xmlns="o"></p></span></b><p class="MsoNormal"><span>While removed from public eye during the Longhorn plan change in 2004, a small team was formed to continue to incubate Windows Sidebar as a concept, dating back to its roots in 2000/2001 as a research exercise. Now Windows Sidebar will be a feature of Windows Vista. Feedback from customers and hardware industry dynamics are being taken into account, particularly adding support for DHTML-based Gadgets to support a broader range of developer and designer, enhanced security infrastructure, and better support for Widescreen (16:10, 16:9) displays. Additionally a new feature in Windows Sidebar is support for hosting of Web Gadgets which can be hosted on sites such as Start.com or run locally. Gadgets that run on the Windows desktop will also be available for Windows XP customers â more details to be shared here in the future.</span></p></blockquote><p class="MsoNormal" dir="ltr"><span>So the desktop version of 'Microsoft Gadgets' is the shipping version of Microsoft Research's 'Sideshow' project. Since the research paper was published a number of parties have shipped products inspired by that research including <a href="http://www.activewin.com/reviews/software/apps/msn/msn8/interface.shtml">MSN Dashboard</a>, <a href="http://desktop.google.com/features.html#sidebar">Google Desktop</a> and <a href="http://www.desktopsidebar.com/">Desktop Sidebar</a> but this doesn't change the fact that the Microsoft is the pioneer in this space. </span></p><p class="MsoNormal" dir="ltr"><span>From the post <a href="http://microsoftgadgets.com/blogs/gadgetnews/archive/2005/09/15/177.aspx">Gadgets and Start.com</a> by Sanaz Ahari </span></p><blockquote dir="ltr" style="MARGIN-RIGHT: 0px"><span><p><a href="http://start.com/">Start.com </a>was initially released on February 2005, on <a href="http://start.com/1">start.com/1</a> â since then weâÂÂve been innovating regularly (<a href="http://start.com/2">start.com/2</a>, <a href="http://start.com/3">start.com/3</a>, <a href="http://start.com/">start.com </a>and <a href="http://start.com/pdc">start.com/pdc</a>) working towards accomplishing our goals: </p><ul><li> To bring the webâÂÂs content to users through: <ul><li> Rich DHTML components (Gadgets) </li><li> RSS and behaviors associated with RSS </li><li> High customizability and personalization</li></ul></li><li> To enable developers to extend their start experience by building their own Gadgets</li></ul><p> Yesterday marked a humble yet significant milestone for us â we opened our 'Atlas' framework enabling developers to extend their start.com experience. You can read more it here: <a href="http://start.com/developer">http://start.com/developer</a>. The key differentiators about our Gadgets are: </p><ul><li> Most web applications were designed as closed systems rather than as a web platform. For example, most customizable 'aggregator' web-sites consume feeds and provide a fair amount of layout customization. However, the systems were not extensible by developers. With start.com, the experience is now an integrated and extensible application platform. </li><li> We will be enriching the gadgets experience even further, enabling these gadgets to seamlessly work on Windows Sidebar</li></ul></span></blockquote><p class="MsoNormal" dir="ltr"><span>The Start.com stuff is really cool. Currently with traditional portal sites like <a href="http://my.msn.com/">MyMSN</a> or <a href="http://my.yahoo.com/">MyYahoo</a>, I can customize my data sources by subscribing to RSS feeds but not how they look. Instead all my RSS feeds always look like a list of headlines. These portal sites usually use different widgets for display richer data like stock quotes or weather reports but there is no way for me to subscribe to a stock quote or weather report feed and have it look the same as the one provided by the site. <a href="http://www.start.com/developer">Start.com</a> fundamentally changes this model by turning it on its head. I can create a custom RSS feed and specify how it should render in <a href="http://www.start.com/">Start.com</a> using JavaScript which basically makes it a <a href="http://www.start.com/">Start.com</a> gadget, no different from the default ones provided by the site. </span></p><p class="MsoNormal" dir="ltr"><span>From my perspective, we're shipping really innovative stuff but because of branding that has attempted to cash in on the 'widgets' hype, we end up looking like followers and copycats. </span></p><p class="MsoNormal" dir="ltr"><span>Marketing sucks. </span></p>" <p>(Via <a href="http://www.25hoursaday.com/weblog/">Dare Obasanjo aka Carnage4Life</a>.)</p></blockquote> Posted for historic annotation purposes (re. Widgets as Microsoft didn't copy Apple here at all; Apple just packaged this better at the expense of Konfabulator as already noted above). And yes, Marketing sucks big time!!
Bill Gates: Cell Phones Will Overtake MP3 Players, Calls iPod 'Unsustainable'
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2005-05-13#837
2005-05-13T03:53:20Z
2006-06-22T08:56:58-04:00
<blockquote dir="ltr" style="MARGIN-RIGHT: 0px"> <p><a href="http://www.macobserver.com/article/2005/05/12.12.shtml">Bill Gates: Cell Phones Will Overtake MP3 Players, Calls iPod 'Unsustainable'</a> Microsoft's chairman draws on computing history to make his proclamation that the iPod phenomenon won't... </p></blockquote> <div align="right">[via <a href="http://www.macobserver.com/">The Mac Observer</a>]</div> <div align="left">Hmm..!</div> <div align="left"> </div> <div align="left">I think this one speaks for itself! Kind of reminds me of the ominous round during the <a href="http://www.eastsideboxing.com/news.php?p=2100&more=1">rumble in the jungle</a> when Ali asked Foreman: "Is that all you got George!".</div> <div align="left"> </div> <div align="left">Again, Mac OS X vs Windows is a rendition of Ali vs Foreman (circa 1974) as stated in an earlier <a href="http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/index.vspx?id=793">post</a>; very much in line with the essence of the post fight analysis expressed below:</div> <blockquote dir="ltr" style="MARGIN-RIGHT: 0px"> <div align="left">"Why did Foreman lose to Ali? The fact is Ali beat Foreman because he was tougher and stronger than he's ever given credit for. Ali didn't box Foreman! He went to the ropes and allowed Foreman to hit on him, is that boxing? What if Foreman had knocked him out while he was stationary against the ropes. It would've been said for the rest of time, why did Ali remain stationary letting Foreman get off on him? How come he didn't use the ring and box? Which is exactly what those watching the fight were thinking and saying during rounds two through eight. That's not boxing, that's being forced to fight because your opponent will not allow you to box."</div> <div align="left"> </div></blockquote> <div align="left" dir="ltr">The point I am trying to make here is simple: Bill's comments are more about hope than facts. The iPod does not define Apple, the company's future isn't inextricably linked to the iPod. The company's future (as I see it) isn't solely about Desktop Computing (the battle Microsoft won many years ago) or the use of the iPod to ramp up its future growth in this realm. Apple is clearly focused on "Digital Life Style", a broader incarnation of what Bill <a href="http://alia.org.au/advocacy/alw/1998/gates.response.html">described</a> as "Web Life Style" in the late 90's.</div> <div align="left" dir="ltr"> </div> <div align="left" dir="ltr">Apple clearly understands that the Internet is the new Operating System (OS). It also understands that this OS isn't solely about personal Desktop Computing. Most important of all, it understands that it cannot own this OS (so it won't repeat the fatal mistake of not licensing it to potential partners :-) ). In a sense, the new OS protects Apple from itself (I see certainly understand Bill's <a href="http://www.openlinksw.com/blog/search.vspx?blogid=127&q=apple+ipod%0D%0A%0D%0A&type=text&output=html">point</a> if Apple was just about the iPod).</div> <div align="left" dir="ltr"> </div> <div align="left" dir="ltr">Apple is using its significant prowess in technology, aesthetics and user experience innovation to provide great solutions (hardware and software) that empower users of this new OS. Tiger (nee. <a href="http://binarybonsai.com/archives/2005/01/29/jobs-nextstep-os/">OpenStep / NeXTSTEP OS</a>; the platform on which the first Web Browser was <a href="http://www.w3.org/People/Berners-Lee/WorldWideWeb.html">created</a>) is a great example, what a <a href="http://mlagazine.com/modules.php?op=modload&name=News&file=article&sid=142">journey</a>! </div> <div align="left" dir="ltr"> </div>
A History Of Communications
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2005-04-27#813
2005-04-27T21:17:55Z
2006-06-22T08:56:58-04:00
<a href="http://www.nixlog.com/archives/2005/03/20_a_history_of_communications.php">A History Of Communications</a> <a href="http://www.nathan.com/projects/current/comtimeline.html">A History of Communications Timeline</a> (via <a href="http://www.xplane.com/xblog/">xBlog</a>) <div align="right">[via <a href="http://www.nixlog.com/">nixlog</a>]</div>
Condemned To Repeat The Past?
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2005-04-13#779
2005-04-13T16:05:11Z
2006-06-22T08:56:58-04:00
<a href="http://techdirt.com/articles/20050413/0054217_F.shtml">Condemned To Repeat The Past?</a> Last week, I mentioned one of the <a href="http://www.techdirt.com/books/20050407/022236_F.shtml">lessons learned</a> from Andy Kessler's newest book, <a href="http://www.amazon.com/exec/obidos/ASIN/0060840978/techdirtcom/"><i>How We Got Here: A Slightly Irreverent History of Technology and Markets</a></i>. I've just finished reading it, and just in time, as well, since you can now check it out for free. While the hard copy version doesn't come out until June, Kessler is releasing the book for <a href="http://andykessler.com/hwgh.html">free download</a> off his site in e-book form. If you read his previous book, <a href="http://www.amazon.com/exec/obidos/ASIN/0060740647/techdirtcom/"><i>Running Money</i></a>, you might even recognize a few short passages in the new book. In <i>Running Money</i>, Kessler goes through his own experience figuring out the mental model that guided his investment philosophy in technology -- and part of that included a brief history lesson in the start of the industrial revolution. HWGH is basically an extended version of that history lesson, written in the same light tone -- designed to be the basic, quick history manual that anyone in the tech world (in just about any capacity, from engineer to business to investing) should read. Indeed, we're already seeing startups and companies making business decisions that seem like they're following the same bad footsteps companies took only five years ago. Is it any surprise that some are repeating the mistakes of 200 years ago as well? One of the worst things in the tech and business world these days is that many people can't view trends out past a quarter (or they simply extract one single trend, without recognizing how others impact them). Any intelligent business person needs to recognize how trends play out in the long term, and how they interact with each other. HWGH gives you plenty of trends from the past few <i>centuries</i> to help guide you into that longer term thinking. <div align="right">[via <a href="http://www.techdirt.com/">Techdirt</a>]</div>
IDMS and its role in general DBMS History
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2005-03-28#772
2005-03-28T16:34:21Z
2006-06-22T08:56:58-04:00
<p>A great piece of DBMS history conveyed through the <a href="http://users.senet.com.au/~cherlet/idmshist.html">story of IDMS</a>.</p>
The Lost 1984 Mac Video
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2005-03-22#760
2005-03-22T20:20:47Z
2006-06-22T08:56:58-04:00
<p>A great piece that reminds us of <a href="http://en.wikipedia.org/wiki/Apple_Computer">Apple Computer's</a> contributions to <a href="http://www.industrial-technology-and-witchcraft.de/1984.html">desktop computing</a> history. </p>
Longhorn: Fixing Your Own Mess?
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2005-03-08#745
2005-03-08T17:42:00Z
2006-06-22T08:56:58-04:00
<p>Via the <a href="http://www.alwayson-network.com/">always-on</a> network I stumbled across a great <a href="http://www.alwayson-network.com/comments.php?id=9035_0_11_0_C">article</a> by Pip Coburn that posed the following question: "should Microsoft benefit from the mess it helped create?".</p> <p>The article discusses most of the key issues, but it should also have included and discussed he following question: "should Microsoft benefit from the mess that we let them create?". By "we" I mean the extensive pool of Microsoft product consumers, developers, and partners etc.</p> <blockquote dir="ltr" style="MARGIN-RIGHT: 0px"> <p><em>I have worked with Microsoft products (as a developer and user) for more years than I would like to remember; I have personally experienced the journey from Windows 2.0 to Windows XP (and played around with Longhorn).</em> </p></blockquote> <p>I added my question to this dialog as without it's resultant perspective, history will simply repeat itself. If IT technology decision makers don't change their product selection and acquisition habits, then why should Microsoft or any other vendor change their ways? Especially when a perpetual promise-under deliver-repromise cycle works absolutely fine. This isn't rocket science, it basic common sense (but we know that common sense ain't that common).</p> <p>Microsoft like most software companies seek significant portions of their revenue growth from product upgrades. In a sense, it inherently implies that these products will always be millions of miles away from the "silver bullet" promises espoused in the pre product release marketing and PR hype. Sadly, there was a time when Marketing and PR hype used to be about new features; a time when there was a clear line between a new feature and a fundamental product bug. </p> <p>Buying products from any company simply because they have the largest market share is dumb! All it does is encourage other vendors to focus on product market share rather than product quality, which ultimately results in the following:</p> <ol> <li>You basically end up paying (rather than at least being credited) for opportunity costs arising from all the time lost now your PC now works slower than you do. <br></li> <li>You pay for bug fixes and architectural flaws instead of new features</li></ol> <p>Microsoft isn't a unique source of this problem, but hey! They are the largest Software Company (the one with the vital market share), and their software products are on some 80-90% of desktops on this planet, and the planet isn't at its most productive at the current time, and no matter how you look at it, this loss of productivity has something to do with the increased nuisance of desktop computing. </p> <p>If Microsoft could just focus on its core competence (BTW - I can't quite pint point this anymore since they are in every software market that exists today), it would have at least have an iota of a chance in hell of cleaning up this mess.</p> <p> </p>
OpenSolaris: Great Business Strategy or Dumb Luck?
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2005-03-03#725
2005-03-03T18:46:13Z
2006-06-22T08:56:58-04:00
<blockquote> <p><a href="http://blogs.sun.com/roller/page/jimgris/20050302#great_business_strategy_or_dumb">Great Business Strategy or Dumb Luck</a> Interesting read here today at ZDNet -- <a href="http://news.zdnet.com/2100-9590_22-5596710.html">Open Solaris and strategic consequences</a>. Here's a bit of the conclusion:<br><br></p> <div style="MARGIN-LEFT: 40px"><span style="COLOR: rgb(153,0,0)">Open Solaris may go down in history as one the finest examples of business strategy ever -- unless, of course, it's just dumb luck.</span><br></div><br>So, we are so brilliant -- <span style="FONT-STYLE: italic">to the extreme</span> -- that when <a href="http://www.opensolaris.org/">OpenSolaris</a> succeeds it will be characterized as "one of the finest examples of business strategy <span style="TEXT-DECORATION: underline">ever</span>." Ever? That would be quite an achievement. But even if we are successful -- <span style="FONT-STYLE: italic">shifting to the extreme polar opposite now</span> -- we could just as easily be considered "dumb" and that our achievement was "just lucky." What? Why the extremes? Sorry. I just can't factory that. I realize I'm a pretty simple guy, but this makes no sense to me. Why do people look at issues this way? I think this is why some conversations are so confusing. People argue to the extremes. Why can't Sun's <a href="http://www.technorati.com/tag/OpenSolaris">open sourcing</a> of <a href="http://www.technorati.com/tag/Solaris">Solaris</a> be seen as simply the natural evolution of a company, a development team, a product, and a market? Or the genuine attempt of the <a href="http://www.sun.com/software/solaris/">Solaris</a> <a href="http://www.samag.com/documents/s=9427/sam0414a/0414a.htm">kernel engineers</a> to engage with external developers in a community co-development model to improve the system for everyone involved? Why can't it be that simple? What am I missing here?</blockquote> <div align="right">[via <a href="http://blogs.sun.com/roller/page/jimgris">Jim Grisanzio</a>]</div> <div align="left"> </div> <div align="left">Jim makes a great point!</div> <div align="left"> </div> <div align="left">Also note that Open Source Solaris is a huge contribution to the Open Source community from a company (that IMHO) has actually been one of the largest Open Source contributors in history period. We just don't track history very well these days thanks to the kind of zealotry written about <a href="http://caustictech.typepad.com/caustictech/2004/06/the_open_source.html">here</a> (*strong language*), and <a href="http://neopoleon.com/blog/posts/4343.aspx">here</a> (in this case by <a href="http://neopoleon.com/blog/">Rory Blythe</a>).</div> <div align="left"> </div> <div align="left">BTW - <a href="http://www.openlinksw.com/blog/search.vspx?blogid=127&q=open+source%0D%0A&type=text&output=html">Here</a> are some of my previous posts on the subject of Open Source.</div> <div align="left"> </div>
Bloglines
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2004-07-26#600
2004-07-26T19:02:19Z
2006-06-22T08:56:58-04:00
<a href="http://weblog.infoworld.com/udell/2004/07/25.html#a1047">Bloglines</a> <p>Since last fall, I've been recommending <a href="http://www.blogines.com/">Bloglines</a> to first-timers as the fastest and easiest introduction to the subscription side of the blogosphere. Remarkably, this same application also meets the needs of some of the most <a href="http://www.intertwingly.net/blog/1716.html">advanced</a> <a href="http://jeremy.zawodny.com/blog/archives/001829.html">users</a>. I've now added myself to that list. Hats off to <a href="http://www.wingedpig.com/">Mark Fletcher</a> for putting all the pieces together in such a masterful way. </p> <p>What goes around comes around. Five years ago, centralized feed aggregators -- my.netscape.com and my.userland.com -- were the only game in town. Fat-client feedreaders only arrived on the scene later. Because of the well-known rich-versus-reach tradeoffs, I never really settled in with one of those. Most of the time I've used the Radio UserLand reader. It is browser-based, and it normally points to localhost, but I've been parking Radio UserLand on a secure server so that I can read the feeds it aggregates for me from anywhere. </p> <p>Bloglines takes that idea and runs with it. Like the Radio UserLand reader, it supports the all-important (to me) consolidated view of new items. But its two-pane interface also shows me the list of feeds, highlighting those with new entries, so you can switch between a linear of scan of all new items and random access to particular feeds. Once you've read an item it vanishes, but you can recall already-read items like so: </p> <p align="center"> </p><form action="" method="get"><input name="sub" type="hidden" /> Display items within the last <select name="since"><option value="0">Session</option><option value="1">1 Hour</option><option value="2">6 Hours</option><option value="3">12 Hours</option><option value="4">24 Hours</option><option value="5">48 Hours</option><option value="6">72 Hours</option><option value="7">Week</option><option value="8">Month</option><option value="9">All Items</option></select> <input name="Display" type="submit" value="Display" /> </form> <p>If a month's worth of some blog's entries produces too much stuff to easily scan, you can switch that blog to a titles-only view. The titles expand to reveal all the content transmitted in the feed for that item. </p> <p>I haven't gotten around to organizing my feeds into folders, the way <a href="http://www.bloglines.com/public/yoz">other</a> <a href="http://www.bloglines.com/public/marccanter">users</a> of Bloglines do, but I've poked around enough to see that Bloglines, like Zope, handles foldering about as well as you can in a Web UI -- which is to say, well enough. With an intelligent local cache it could be really good; more on that later. </p> <p>Bloglines does two kinds of data mining that are especially noteworthy. First, it counts and reports the number of Bloglines users subscribed to each blog. In the case of <a href="http://www.bloglines.com/preview?siteid=297235">Jonathan Schwartz's weblog</a>, for example, there are (as of this moment) <a href="http://www.bloglines.com/userdir?siteid=297235">253 subscribers</a>. </p> <p>Second, Bloglines is currently managing references to items more effectively than the competition. I was curious, for example, to gauge the reaction to the latest salvo in Schwartz's ongoing campaign to turn up the heat on Red Hat. Bloglines reports <a href="http://www.bloglines.com/citations?siteid=297235&itemid=14" target="_blank" title="References To This Item From Other Blogs">10 References</a>. In this case, the comparable query on Feedster yields a <a href="http://feedster.net//links.php?url=http%3A//blogs.sun.com/roller/page/jonathan/20040721%23competing_against_a_social_movement">comparable result</a>, but on the whole I'm finding Bloglines' assembly of conversations to be more reliable than Feedster's (which, however, is still marked as 'beta'). Meanwhile Technorati, though it casts a much wider net than either, is <a href="http://www.technorati.com/cosmos/search.html?url=http://blogs.sun.com/roller/page/jonathan/20040721#competing_against_a_social_movement">currently struggling</a> with conversation assembly. </p> <p>I love how Bloglines weaves everything together to create a dense web of information. For example, the list of <a href="http://www.bloglines.com/userdir?siteid=297235">subscribers to the Schwartz blog</a> includes: <i><a href="http://www.bloglines.com/public/judell">judell</a> - subscribed since July 23, 2004</i>. Click that link and you'll see my Bloglines subscriptions. Which you can <a href="http://www.bloglines.com/export?id=judell">export</a> and then -- if you'd like to see the world through my filter -- turn around and import. </p> <p>Moving my 265 subscriptions into Bloglines wasn't a complete no-brainer. I imported my <a href="http://weblog.infoworld.com/udell/gems/mySubscriptions.opml">Radio UserLand-generated OPML file</a> without any trouble, but catching up on unread items -- that is, marking all of each feed's sometimes lengthy history of items as having been read -- was painful. In theory you can do that by clicking once on the top-level folder containing all the feeds, which generates the consolidated view of unread items. In practice, that kept timing out. I finally had to touch a number of the larger feeds, one after another, in order to get everything caught up. A <b>Catch Up All Feeds</b> feature would solve this problem. </p> <p>Another feature I'd love to see is <b>Move To Next Unread Item</b> -- wired to a link in the HTML UI, or to a keystroke, or ideally both. </p> <p>Finally, I'd love it if Bloglines cached everything in a local database, not only for offline reading but also to make the UI more responsive and to accelerate queries that reach back into the archive. </p> <p>Like Gmail, Bloglines is the kind of Web application that surprises you with what it can do, and makes you crave more. Some argue that to satisfy that craving, you'll need to abandon the browser and switch to RIA (rich Internet application) technology -- Flash, Java, Avalon (someday), whatever. Others are concluding that perhaps the 80/20 solution that the browser is today can become a 90/10 or 95/5 solution tomorrow with some incremental changes. </p> <p>Dare Obasanjo wondered, over the weekend, "What is Google building?" He wrote: </p><blockquote class="personQuote DareObasanjo">In the past couple of months Google has hired four people who used to work on Internet Explorer in various capacities [especially its XML support] who then moved to BEA; <a href="http://davidbau.com/about/david_bau.html">David Bau</a>, <a href="http://www.oreillynet.com/pub/au/1303">Rod Chavez</a>, <a href="http://gary.burd.info/">Gary Burd</a> and most recently <a href="http://www.eweek.com/article2/0,1759,1627319,00.asp">Adam Bosworth</a>. A number of my coworkers used to work with these guys since our team, the Microsoft XML team, was once part of the Internet Explorer team. It's been interesting chatting in the hallways with folks contemplating what Google would want to build that requires folks with a background in building XML data access technologies both on the client side, Internet Explorer and on the server, BEA's WebLogic. [<a href="http://www.25hoursaday.com/weblog/PermaLink.aspx?guid=1524b97e-f8b1-4e42-ac07-455337f299b4">Dare Obasanjo</a>] </blockquote>It seems pretty clear to me. Web applications such as Gmail and Bloglines are already hard to beat. With a touch of <a href="http://weblog.infoworld.com/udell/2004/06/15.html#a1023">alchemy</a> they just might become unstoppable. <p></p> <div align="right">[via <a href="http://weblog.infoworld.com/udell/">Jon's Radio</a>]</div>
History leading up to today's IE Security and Backwardness Debacle
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2004-07-07#584
2004-07-07T19:30:28Z
2006-06-22T08:56:58-04:00
<p>Another <a href="http://reviews.cnet.com/4520-3513_7-5142439-1.html">insightful piece</a> on the same painful subject of IE and the costs of vendor monoculture. The IE debacle is an important forebearer of what's to come for those who hope this is simply a storm in a tea cup. It isn't inconceivable that a Longhorn upgrade wouldn't be pitched as the way out of this deepening dysfunctional-web quagmire. Unfortunately many will bite if history and current mindset is a barometer :-(</p>
XML, the New Database Heresy
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2004-06-04#555
2004-06-04T04:04:48Z
2006-06-22T08:56:58-04:00
<p dir="ltr">A great <a href="http://www.25hoursaday.com/weblog/PermaLink.aspx?guid=d28ce1fb-7b27-407d-b1a3-0b9a34831ca1">post </a>by Dare, especially his bringing into context the essence of this matter refrred to by C.J. Date as "XML the New Database Heresy".</p> <p dir="ltr">I have little to add to this matter as our understanding and vision is aptly expressed via the architecture and feature set of <a href="http://www.openlinksw.com/virtuoso">Virtuoso</a> (this area was actually addressed circa 1999).</p> <p dir="ltr">We are heading into a era of multi-model databases, these are single database engines that are capable of effectively serving the requirements of the Hierarchical, Network, Relational, and Object database <a href="http://www.web-dictionary.org/encyclopedia/db/DBMS.html#Navigational_databases">models</a> . As we get closer to the unravelling of universal storage, hopefully this will get clearer.</p> <p dir="ltr">Back to Dare's commentary:</p> <blockquote dir="ltr" style="MARGIN-RIGHT: 0px"> <p><a href="http://www.informatik.uni-trier.de/~ley/db/indices/a-tree/d/Date:C=_J=.html">C.J. Date</a>, one of the most influential names in the relational database world, had some harsh words about XML's encroachment into the world of relational databases in a recent article entitled <a href="http://searchdatabase.techtarget.com/originalContent/0,289142,sid13_gci962948,00.html">Date defends relational model </a> that appeared on SearchDatabases.com. Key parts of the article are excerpted below </p> <blockquote dir="ltr" style="MARGIN-RIGHT: 0px"> <p>Date reserved his harshest criticism for the competition, namely object-oriented and XML-based DBMSs. Calling them "the latest fashions in the computer world," Date said he rejects the argument that relational DBMSs are yesterday's news. Fans of object-oriented database systems "see flaws in the relational model because they don't fully understand it," he said. </p> <p>Date also said that XML enthusiasts have gone overboard. </p> <p>"XML was invented to solve the problem of data interchange, but having solved that, they now want to take over the world," he said. "With XML, it's like we forget what we are supposed to be doing, and focus instead on how to do it." </p> <p>Craig S. Mullins, the director of technology planning at BMC Software and a SearchDatabase.com expert, shares Date's opinion of XML. It can be worthwhile, Mullins said, as long as XML is only used as a method of taking data and putting it into a DBMS. But Mullins cautioned that XML data that is stored in relational DBMSs as whole documents will be useless if the data needs to be queried, and he stressed Date's point that XML is not a real data model. </p></blockquote> <p dir="ltr">Craig Mullins points are more straightforward to answer since his comments don't jibe with the current state of the art in the XML world. He states that you can't query XML documents stored in databases but this is untrue. Almost three years ago, I was writing articles about <a href="http://features.slashdot.org/article.pl?sid=01/10/29/0725214&mode=thread&tid=156">querying XML documents stored in relational databases</a>. Storing XML in a relational database doesn't mean it has to be stored in as an opaque binary BLOB or as a big, bunch of text which cannot effectively be queried. The next version of SQL Server will have extensive capabilities for querying XML data in relational database and doing joins across relational and XML data, a lot of this functionality is described in the article on <a href="http://msdn.microsoft.com/xml/default.aspx?pull=/library/en-us/dnsql90/html/sql2k5xml.asp">XML Support in SQL Server 2005</a>. As for XML not having a data model, I beg to differ. There is a data model for XML that many applications and people adhere to, often without realizing that they are doing so. This data model is the <a href="http://www.w3.org/TR/1999/REC-xpath-19991116#data-model">XPath 1.0 data model</a>, which is being updated to handled typed data as the <a href="http://www.w3.org/TR/2003/WD-xpath-datamodel-20031112/">XQuery and XPath 2.0 data model</a>. </p> <p dir="ltr">Now to tackle the meat of C.J. Date's criticisms which is that XML solves the problem of data interchange but now is showing up in the database. The thing first point I'd like point out is that there are two broad usage patterns of XML, it is used to represent both rigidly structured tabular data (e.g., relational data or serialized objects) and semi-structured data (e.g., office documents). The latter type of data will only grow now that office productivity software like <a href="http://www.microsoft.com/office">Microsoft Office</a> have enabled users to save their documents as XML instead of proprietary binary formats. In many cases, these documents cannot simply shredded into relational tables. Sure you can shred an Excel spreadsheet written in spreadsheetML into relational tables but is the same really feasible for a Word document written in WordprocessingML? Many enterprises would rather have their important business data being stored and queried from a unified location instead of the current situation where some data is in document management systems, some hangs around as random files in people's folders while some sits in a database management system. </p> <p dir="ltr">As for stating that critics of the relational model don't understand it, I disagree. One of the major benefits of using XML in relational databases is that it is a lot easier to deal with fluid schemas or data with sparse entries with XML. When the shape of the data tends to change or is not fixed the relational model is simply not designed to deal with this. Constantly changing your database schema is simply not feasible and there is no easy way to provide the extensibility of XML where one can say "after the <font face="Courier New">X </font>element, any element from any namespace can appear". How would one describe the capacity to store âany dataâ in a traditional relational database without resorting to an opaque blob? </p> <p dir="ltr">I do tend to agree that some people are going overboard and trying to model their data hierarchically instead of relationally which experience has thought us is a bad idea. Recently on the XML-DEV mailing list entitled <a href="http://lists.xml.org/archives/xml-dev/200405/msg00216.html">Designing XML to Support Information Evolution </a>where Roger L. Costello described his travails trying to model his data which was being transferred as XML in a hierarchical manner. Micheal Champion accurately described the process Roger Costello went through as having "rediscovered the relational model". In a response to that thread I wrote "Hierarchical databases failed for a reason". </p> <p dir="ltr">Using hierarchy as a primary way to model data is bad for at least the following reasons </p> <ol dir="ltr"> <li> <div>Hierarchies tend to encourage redundancy. Imagine I have a <Customer> element who has one or more <ShippingAddress> elements as children as well as one or more <Order> elements as children as well. Each order was shipped to an address, so if modelled hierarchically each <Order> element also will have a <ShippingAddress> element which leads to a lot of unnecessary duplication of data. </div></li> <li> <div>In the real world, there are often multiple groups to which a piece of data belongs which often cannot be modelled with a single hierarchy. </div></li> <li> <div>Data is too tightly coupled. If I delete a <Customer> element, this means I've automatically deleted his entire order history since all the <Order> elements are children of <Customer>. Similarly if I query for a <Customer>, I end up getting all the <Order> information as well. </div></li></ol> <p>To put it simply, experience has taught the software world that the relational model is a better way to model data than the hierarchical model. Unfortunately, in the rush to embrace XML many a repreating the mistakes from decades ago in the new millenium. </p></blockquote> <div align="right">[via <a href="http://www.25hoursaday.com/weblog/">Dare Obasanjo aka Carnage4Life</a>]</div>
Essay about current and past trends -- Joi Ito
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2004-04-24#528
2004-04-24T21:27:24Z
2006-06-22T08:56:58-04:00
<p>Here are some thoughts on where I think things are going in the mobile and content space.</p> <p><em>I wrote this essay before reading <a href="http://en.wikipedia.org/wiki/Free_Culture">Free Culture</a> so I'm saying a lot of stuff that <a href="http://www.lessig.org/">Larry</a> says better...</em></p> <p>Several crucial shifts in technology are emerging that will drastically affect the relationship between users and technology in the near future. Wireless Internet is becoming ubiquitous and economically viable. Internet capable devices are becoming smaller and more powerful.</p> <p>Alongside technological shifts, new social trends are emerging. Users are shifting their attention from packaged content to social information about location, presence and community. Tools for identity, trust, relationship management and navigating social networks are becoming more popular. Mobile communication tools are shifting away from a 1-1 model, allowing for increased many-to-many interactions; such a shift is even being used to permit new forms of democracy and citizen participation in global dialog.</p> <p>While new technological and social trends are occurring, it is not without resistance, often by the developers and distributors of technology and content. In order to empower the consumer as a community member and producer, communication carriers, hardware manufacturers and content providers must understand and build models that focus less on the content and more on the relationships. </p> <p><strong>Smaller faster</strong></p> <p>Computing started out as large mainframe computers, software developers and companies “time sharing” for slices of computing time on the large machines. The mini-computer was cheaper and smaller, allowing companies and labs to own their own computers. The mini computer allowed a much greater number of people to have access to computers and even use them in real time. The mini computer lead to a burst in software and networking technologies. In the early 80’s, the personal computer increased the number of computers by an order of magnitude and again, led to an explosion in new software and technology while lowering the cost even more. Console gaming companies proved once again that unit costs could be decreased significantly by dramatically increasing the number of units sold. Today, we have over a billion cell phones in the market. There are tens of millions camera phones. The incredible number of these devices has continued to lower the unit cost of computing as well as devices imbedded in these devices such as small cameras. High end phones have the computing power of the personal computers of the 80’s and the game consoles of the 90’s.</p> <p><strong>History repeats with WiFi</strong></p> <p>There are parallels in the history of communications and computing. In the 1980’s the technology of packet switched networks became widely deployed. Two standards competed. X.25 was a packet switched network technology being promoted by CCITT (a large, formal international standards body) and the telephone companies. It involved a system run by telephone companies including metered tariffs and multiple bilateral agreements between carriers to hook up.</p> <p>Concurrently, universities and research labs were promoting TCP/IP and the Internet opportunity for loosely organized standards meetings being operated with flat rate tariffs and little or no agreements between the carriers. People just connected to the closest node and everyone agreed to freely carry traffic for others.</p> <p>There were several “free Internet” services such as “The Little Garden” in San Francisco. Commercial service providers, particularly the telephone company operators such as SprintNet tried to shut down such free services by threatening not to carry this free traffic.</p> <p>Eventually, large ISPs began providing high quality Internet connectivity and finally the telephone companies realized that the Internet was the dominant standard and shutdown or acquired the ISPs.</p> <p>A similar trend is happening in wireless data services. GPRS is currently the dominant technology among mobile telephone carriers. GPRS allows users to transmit packets of data across the carrier network to the Internet. One can roam to other networks as long as the mobile operators have agreements with each other. Just like in the days of X.25, the system requires many bilateral agreements between the carriers; their goal is to track and bill for each packet of information.</p> <p>Competing with this standard is WiFi. WiFi is just a simple wireless extension to the current Internet and many hotspots provide people with free access to the Internet in cafes and other public areas. WiFi service providers have emerged, while telephone operators –such as a T-Mobile and Vodaphone- are capitalizing on paid WiFi services. Just as with the Internet, network operators are threatening to shut down free WiFi providers, citing a violation of terms of service. </p> <p>Just as with X.25, the GPRS data network and the future data networks planned by the telephone carriers (e.g. 3G) are crippled with unwieldy standards bodies, bilateral agreements, and inherently complicated and expensive plant operations.</p> <p>It is clear that the simplicity of WiFi and the Internet is more efficient than the networks planned by the telephone companies. That said, the availability of low cost phones is controlled by mobile telephone carriers, their distribution networks and their subsidies.</p> <p><strong>Content vs Context</strong></p> <p>Many of the mobile telephone carriers are hoping that users will purchase branded content manufactured in Hollywood and packaged and distributed by the telephone companies using sophisticated technology to thwart copying.</p> <p>Broadband in the home will always be cheaper than mobile broadband. Therefore it will be cheaper for people to download content at home and use storage devices to carry it with them rather than downloading or viewing content over a mobile phone network. Most entertainment content is not so time sensitive that it requires real time network access.</p> <p>The mobile carriers are making the same mistake that many of the network service providers made in the 80s. Consider Delphi, a joint venture between IBM and Sears Roebuck. Delphi assumed that branded content was going to be the main use of their system and designed the architecture of the network to provide users with such content. Conversely, the users ended up using primary email and communications and the system failed to provide such services effectively due to the mis-design.</p> <p>Similarly, it is clear that mobile computing is about communication. Not only are mobile phones being used for 1-1 communications, as expected through voice conversations; people are learning new forms of communication because of SMS, email and presence technologies. Often, the value of these communication processes is the transmission of “state” or “context” information; the content of the messages are less important.</p> <p><strong>Copyright and the Creative Commons</strong></p> <p>In addition to the constant flow of traffic keeping groups of people in touch with each other, significant changes are emerging in multimedia creation and sharing. The low cost of cameras and the nearly television studio quality capability of personal computers has caused an explosion in the number and quality of content being created by amateurs. Not only is this content easier to develop, people are using the power of weblogs and phones to distribute their creations to others. </p> <p>The network providers and many of the hardware providers are trying to build systems that make it difficult for users to share and manipulate multimedia content. Such regulation drastically stifles the users’ ability to produce, share and communicate. This is particularly surprising given that such activities are considered the primary “killer application” for networks.</p> <p>It may seem unintuitive to argue that packaged commercial content can co-exist alongside consumer content while concurrently stimulating content creation and sharing. In order to understand how this can work, it is crucial to understand how the current system of copyright is broken and can be fixed.</p> <p>First of all, copyright in the multimedia digital age is inherently broken. Historically, copyright works because it is difficult to copy or edit works and because only few people produce new works over a very long period of time. Today, technology allows us to find, sample, edit and share very quickly. The problem is that the current notion of copyright is not capable of addressing the complexity and the speed of what technology enables artists to create. Large copyright holders, notably Hollywood studios, have aggressively extended and strengthened their copyright protections to try to keep the ability to produce and distribute creative works in the realm of large corporations.</p> <p>Hollywood asserts, “all rights reserved” on works that they own. Sampling music, having a TV show running in the background in a movie scene or quoting lyrics to a song in a book about the history of music all require payment to and a negotiation with the copyright holder. Even though the Internet makes available a wide palette of wonderful works based on content from all over the world, the current copyright practices forbid most of such creation.</p> <p>However, most artists are happy to have their music sampled if they receive attribution. Most writers are happy to be quoted or have their books copied for non-commercial use. Most creators of content realize that all content builds on the past and the ability for people to build on what one has created is a natural and extremely important part of the creative process.</p> <p>Creative Commons tries to give artists that choice. By providing a more flexible copyright than the standards “all rights reserved” copyright of commercial content providers, Creative Commons allows artists to set a variety of rights to their works. This includes the ability to reuse for commercial use, copy, sample, require attribution, etc. Such an approach allows artists to decide how their work can be used, while providing people with the materials necessary for increased creation and sharing. </p> <p>Creative Commons also provides for a way to make the copyright of pieces of content machine-readable. This means that a search engine or other tool to manipulate content is able to read the copyright. As such, an artist can search for songs, images and text to use while having the information to provide the necessary attribution.</p> <p>Creative Commons can co-exist with the stringent copyright regimes of the Hollywood studios while allowing professional and amateur artists to take more control of how much they want their works to be shared and integrated into the commons. Until copyright law itself is fundamentally changed, the Creative Commons will provide an essential tool to provide an alternative to the completely inflexible copyright of commercial content. </p> <p>Content is not like some lump of gold to be horded and owned which diminishes in value each time it is shared. Content is a foundation upon which community and relationships are formed. Content is the foundation for culture. We must evolve beyond the current copyright regime that was developed in a world where the creation and transmission of content was unwieldy and expense, reserved to those privileged artists who were funded by commercial enterprises. This will provide the emerging wireless networks and mobile devices with the freedom necessary for them to become the community building tools of sharing that is their destiny.<br /></p> <div align="right">[via <a href="http://joi.ito.com/">Joi Ito's Web</a>]</div>
Enterprise Databases get a grip on XML
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2004-01-06#442
2004-01-06T23:17:07Z
2006-06-22T08:56:58-04:00
<blockquote style="margin-right: 0px;" dir="ltr"> <p><a class="listLinkLrg" title="http://newsletter.infoworld.com/t?ctl=4FEDB6:1F3948D" href="http://newsletter.infoworld.com/t?ctl=4FEDB6:1F3948D" target="_new"><strong><font face="Verdana">Databases get a grip on XML</font></strong></a><br /><font size="2"></font><font face="Verdana">From <a href="http://newsletter.infoworld.com/t?ctl=4FEDB6:1F3948D">Inforworld</a>.</font><br /><br /><font face="Verdana,Geneva,Arial,sans-serif" size="2">The next iteration of the SQL standard was supposed to arrive in 2003. But SQL standardization has always been a glacially slow process, so nobody should be surprised that SQL:2003 ? now known as SQL:200n ? isn?t ready yet. Even so, 2003 was a year in which XML-oriented data management, one of the areas addressed by the forthcoming standard, showed up on more and more developers? radar screens.ÃÂ <a title="http://newsletter.infoworld.com/t?ctl=4FEDB6:1F3948D" href="http://newsletter.infoworld.com/t?ctl=4FEDB6:1F3948D" target="_blank">>> READ MORE</a></font></p></blockquote> <p dir="ltr"><font face="Verdana" size="2">This article rounds up product for 2003 in the critical area of Enterprise Database Technology. It's certainly provides an apt reflection of how Virtuoso compares with offerings from some the larger (but certainly slower to implement) database vendors in this space. As usual Jon Udell's quote pretty much sums this up:</font></p> <blockquote style="margin-right: 0px;" dir="ltr"> <p dir="ltr"><!--StartFragment --><span class="artText"><em>"While the spotlight shone on the heavyweight contenders, a couple of agile innovators made noteworthy advances in 2003. </em><a class="regularArticleU" href="http://www.infoworld.com/699"><em>OpenLink Software?s Virtuoso 3.0</em></a><em>, which we reviewed in March, stole thunder from all three major players. Like Oracle, it offers a WebDAV-accessible XML repository. Like DB2 Information Integrator, it functions as database middleware that can perform federated ?joins? across SQL and XML sources. And like the forthcoming Yukon, it embeds the .Net CLR (Common Language Runtime), or in the case of Linux, Novell/Ximian?s Mono."</em></span> </p></blockquote> <p dir="ltr"><font face="Verdana" size="2">Albeit still somewhat unknown to the broader industry we have remained true our "innovator" discipline, which still remains our chosen path to market leadership. Thus, its worth a quick Virtuoso release history, and featuresÃÂ recap as we get set to up the ante even further in 2004:</font></p> <p dir="ltr"><font face="Verdana" size="2"><a href="http://www.openlinksw.com/press/virtuoso.htm">1998 - Virtuoso's initial public beta</a> release with functional emphasis on Virtual Database Engine for ODBC and JDBC Data Sources.</font></p> <p dir="ltr"><font face="Verdana" size="2"><a href="http://www.openlinksw.com/press/virtuoso1.htm">1999 - Virtuoso's official commercial</a> release, with emphasis stillÃÂ on Virtual Database functionality for ODBC, JDBC accessible SQL Databases.</font></p> <p dir="ltr"><font face="Verdana" size="2"><a href="http://www.openlinksw.com/press/v2releas.htm">2000 - Virtuoso 2.0</a> adds XML Storage, XPath, XML Schema, XQuery, XSL-T, WebDAV, SOAP, UDDI, HTTP, Replication, Free Text Indexing (*feature update*), POP3, and NNTP support.</font></p> <p dir="ltr"><font face="Verdana" size="2"><a href="http://www.openlinksw.com/press/v27releas.htm">2002 - Virtuoso 2.7</a> extends Virtualization prowess beyond data access via enhancements to its Web Services protocol stack implementation by enabling SQL Stored Procedures to be published as Web Services. It also debutsÃÂ its Object-Relational engine enhancements that include theÃÂ incorporation of Java and Microsoft .NET Objects into its User Defined Type, User Defined Functions, and Stored ProcedureÃÂ offerings.</font></p> <p dir="ltr"><font face="Verdana" size="2"><a href="http://www.openlinksw.com/press/virt3beta.htm">2003 - Virtuoso 3.0</a> extends data and application logic virtualization into the Application Server realm (basically a Virtual Application server too!), by adding support for ASP.NET, PHP, Java Server Pages runtime hosting (making applications built using any of these languages deployable using Virtuoso across all supported platforms).</font></p> <p dir="ltr"><font face="Verdana" size="2">Collectively each of these releases have contributed to a very premeditated architecture and vision that will ultimately unveil the inherent power of critical I.S infrastructure virtualizationÃÂ along the following lines; data storage, data access , and application logic via coherent integration of SQL, XML, Web Services, and Persistent Stored Modules (.NET, Java, and other object based component building blocks).</font></p> <p dir="ltr"><font face="Verdana"></font>ÃÂ </p>
Borland's Early Years: A Wild Ride
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2003-11-05#415
2003-11-05T17:31:38Z
2006-06-22T08:56:58-04:00
<A href="http://weblogs.asp.net/ssivakumar/posts/35896.aspx">Borland's Early Years: A Wild Ride</A> <FONT face=Verdana size=2>A nice article about history of Borland </FONT><A href="http://www.eweek.com/article2/0,4149,1370757,00.asp"><FONT face=Verdana size=2>http://www.eweek.com/article2/0,4149,1370757,00.asp</FONT></A><FONT face=Verdana size=2> </FONT> <DIV align=right>[via <A href="http://weblogs.asp.net/">WebLogs @ ASP.NET</A>]</DIV>
The Nigerian SCO Connection
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2003-10-01#888
2003-10-01T22:44:03Z
2006-06-22T08:56:58-04:00
<p><span style="font-size: 10pt; font-family: Arial;"><span style="font-size: 10pt; font-family: Arial;">I am a Nigerian reminiscing as my country that turns 43 today (as a post-colonial independent nation). </span></span></p><p xmlns="o"></p><p xmlns="o"></p> <p><span style="font-size: 10pt; font-family: Arial;">October the 1st is an emotional day for many Nigerians, especially those of us in the Diaspora. Our country remains a paradox as the excerpts below attest:</span></p><p xmlns="o"></p> <p><span style="font-size: 10pt; font-family: Arial;">The more popular view of Nigerians as a result of the proliferation of 419 scams (the mangled by-product of misdirected intellectual prowess and the boundless depths of greed -- which applies to perpetrators and victims alike).</span></p><p xmlns="o"></p> <p style="margin-left: 0.5in;"><span style="font-size: 10pt; font-family: Arial;"><a href="http://www.beblogging.com/blog/20031001-214515"><span style="font-size: 12pt; font-family: 'Times New Roman';"><font face="Arial" size="2">The Nigerian SCO Connection</font></span></a> "I AM MR. DARL MCBRIDE CURRENTLY SERVING AS THE PRESIDENT AND CHIEF EXECUTIVE OFFICER OF THE SCO GROUP ..." [via <a href="http://www.beblogging.com/blog/"><span style="font-size: 12pt; font-family: 'Times New Roman';"><font face="Arial" size="2">Be Blogging</font></span></a>]</span></p><p xmlns="o"></p> <p><span style="font-size: 10pt; font-family: Arial;">Funny! But many a truth is told in jest (I think that's how the quote goes); this one is pretty damned poignant. </span></p> <p><span style="font-size: 10pt; font-family: Arial;">Unbeknownst to many, there are other views of Nigeria (unfortunately these aren't the norm).</span></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style="font-size: 10pt; font-family: Arial;">The call for optimism by our president (he doesn't support or condone the 419 nonsense):</span></p><p xmlns="o"></p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style="font-size: 10pt; font-family: Arial;"></span></p><p xmlns="o"> </p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style="font-size: 10pt; font-family: Arial;">President Olusegun Obasanjo urged Nigerians <a href="http://odili.net/news/source/2003/oct/1/40.html">to change their ways and be optimistic about the future</a> as <country-region xmlns="st1" xmlns:n0="w" n0:st="on">Nigeria</country-region> marks its 43rd <place xmlns="st1" xmlns:n0="w" n0:st="on"><city n0:st="on">Independence</city></place> anniversary. Read on </span></p><p xmlns="o"></p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style="font-size: 10pt; font-family: Arial;">[via <a href="http://nigeriaworld.com/"><span style="font-size: 12pt; font-family: 'Times New Roman';"><font face="Arial" size="2">Odili.net </font></span></a>� this site desperately needs RSS!]</span></p><p xmlns="o"></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style="font-size: 10pt; font-family: Arial;"></span></p><p xmlns="o"> </p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style="font-size: 10pt; font-family: Arial;">There is an increasing pool of key high-tech players of Nigerian decent (and nationality) making constructive impact on the high-tech industry (making it <span style="font-size: 12pt; font-family: 'Times New Roman';"><font face="Arial" size="2"><a href="http://www.infoworld.com/article/03/05/23/21FEinnovidehen_1.html">less lonely for myself</a> </font></span>and other Nigerians in the high-tech arena):</span></p><p xmlns="o"></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style="font-size: 10pt; font-family: Arial;"></span></p><p xmlns="o"> </p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style="font-size: 10pt; font-family: Arial;"><a href="http://www.kuro5hin.org/user/Carnage4Life/diary"><span style="font-size: 12pt; font-family: 'Times New Roman';"><font face="Arial" size="2">Dare Obasanjo</font></span></a> is a member of Microsoft's WebData team, which among other things develops the components within the System.Xml and System.Data namespace of the .NET Framework, Microsoft XML Core Services (MSXML), and Microsoft Data Access Components (MDAC). More of Dare's writings on XML can be found on his <a href="http://msdn.microsoft.com/voices/xml.asp"><span style="font-size: 12pt; font-family: 'Times New Roman';"><font face="Arial" size="2">Extreme XML column</font></span></a> on MSDN. </span></p><p xmlns="o"></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style="font-size: 10pt; font-family: Arial;"> </span></p><p xmlns="o"></p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style="font-size: 10pt; font-family: Arial;"><a href="http://uche.ogbuji.net/uche.ogbuji.net/caramusis/"><!--StartFragment --><span style="font-size: 12pt; font-family: 'Times New Roman';"><font face="Arial" size="2">Uche Ogbuji</font></span></a> is a consultant and co-founder of Fourthought Inc., a consulting firm specializing in XML solutions for enterprise knowledge management applications. Fourthought develops 4Suite, the open source platform for XML middleware. Mr. Ogbuji is a Computer Engineer and writer born in <country-region xmlns="st1" xmlns:n0="w" n0:st="on">Nigeria</country-region>, living and working in <place xmlns="st1" xmlns:n0="w" n0:st="on"><city n0:st="on">Boulder</city>, <state n0:st="on">Colorado</state>, <country-region n0:st="on">USA</country-region></place>. <br /><b>Website</b>: <a href="http://www.fourthought.com/"><span style="font-size: 12pt; font-family: 'Times New Roman';"><font face="Arial" size="2">http://www.fourthought.com/</font></span></a> </span></p><p xmlns="o"></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style="font-size: 10pt; font-family: Arial;"> </span></p><p xmlns="o"></p> <p class="MsoNormal" style="margin: 0in 0in 0pt 0.5in;"><span style="font-size: 10pt; font-family: Arial;"><a href="http://www.emeagwali.com/index.shtml"><!--StartFragment --><!--StartFragment --><span style="font-size: 12pt; font-family: 'Times New Roman';"><font face="Arial" size="2">Philip Emeagwali</font></span></a>, a computer scientist, is one of the <a href="http://emeagwali.com/history/internet/index.html" _base_href="http://radioafrica.biz"><span style="font-size: 12pt; font-family: 'Times New Roman';"><font face="Arial" size="2">fathers of the Internet</font></span></a> and a <a href="http://www.emeagwali.com/printed-articles/upstream/natures-own-numbers-man_upstream_january-27-1997.html" target="new" _base_href="http://radioafrica.biz"><span style="font-size: 12pt; font-family: 'Times New Roman';"><font face="Arial" size="2">trailblazer in petroleum extraction</font></span></a>," as quoted by <a href="http://fyi.cnn.com/fyi/interactive/specials/bhm/story/black.innovators.html" target="new" _base_href="http://radioafrica.biz"><i><span style="font-size: 12pt; font-family: 'Times New Roman';"><font face="Arial" size="2">CNN</font></span></i></a>. </span></p><p xmlns="o"></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style="font-size: 10pt; font-family: Arial;"></span></p><p xmlns="o"> </p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style="font-size: 10pt; font-family: Arial;">Philip leaves all Nigerians with this <a href="http://emeagwali.com/speeches/nigeria/43rd-independence-anniversary-message/index.html">important message</a> on this special day (key excerpt below):</span></p><p xmlns="o"></p> <p style="margin-left: 0.5in;"><span style="font-size: 10pt; font-family: Arial;">"Our investments in education and technology will be our legacy to our children. They are investments that will bring the best out of the next generation of Nigerians and enable us to reach our potential as individuals, as communities, as a nation." </span></p><p xmlns="o"></p> <p class="MsoNormal" style="margin: 0in 0in 0pt;"><span style="font-size: 10pt; font-family: Arial;">Happy Birthday dear motherland!</span></p> <a href="index.vspx?tag=Africa" rel="tag" style="display:none;">Africa</a><a href="index.vspx?tag=Nigeria" rel="tag" style="display:none;">Nigeria</a><a href="index.vspx?tag=xml" rel="tag" style="display:none;">xml</a>
The Future of Weblogging
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2003-09-18#250
2003-09-19T02:21:02Z
2006-06-22T08:56:58-04:00
<P>Nico MacDonald: <A href="http://www.spy.co.uk/Articles/Spiked/Weblogging/">The Future of Weblogging</A>. </P> <DIV class=SpyStandfirst>Nico Macdonald puts Weblogging in the context of the history of online publishing, explaining its novelty and value, and indicating where it needs to innovate. He concludes with a proposal encouraging publishers to properly embrace the Weblogging model. </DIV> <DIV class=SpyStandfirst> </DIV> <DIV class=SpyStandfirst><A href="http://www.spy.co.uk/Articles/Spiked/Weblogging/">More.</A></DIV> <DIV align=right>[via <A href="http://www.scripting.com/">Scripting News</A>] <DIV></DIV></DIV>
RSS: INJAN (It's not just about news)
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2003-08-21#241
2003-08-21T15:41:25Z
2006-06-22T08:56:58-04:00
<p><span style="font-size: 10pt; font-family: Arial;">When Virtuoso first unleashed support for XML (in-built XSL, Native XML Storage, Validating XML Parser, XPath, and XQuery) the core message was the delivery of a single server solution that would address the challenges of creating XML data.</span></p><p xmlns="o"></p> <p><span style="font-size: 10pt; font-family: Arial;">In the year 2000 the question of the shape and form of XML data was unclear to many, and reading the article below basically took me back in time to when we released <a href="http://www.it-director.com/article.php?articleid=916">Virtuoso 2.0</a> (we are now at <a href="http://www.openlinksw.com/virtuoso">release 3.0</a> commercially with a <a href="http://www.openlinksw.com/press/virt32_wwdc1.htm">3.2 beta </a>dropping any minute).</span></p><p xmlns="o"></p> <p><span style="font-size: 10pt; font-family: Arial;">RSS is a great XML application, and it does a great job of demonstrating how XML --the new data access foundation layer-- will galvanize the next generation Web (I refer to this as Web 2.0.). </span></p> <blockquote dir="ltr" style="margin-right: 0px;"><span style="font-size: 10pt; font-family: Arial;"> <p><a href="http://jena.hpl.hp.com:3030/blojsom-hp/blog/technologies/blogging/metadata/?permalink=1214847A10C1966396472E816A7A4243.textile">RSS: INJAN (It's not just about news)</a> </p> <p><span class="caps">RSS</span> is not just about news, according to <a href="http://groups.yahoo.com/group/rss-dev/message/5764">Ian Davis on rss-dev</a>.<br />He presents a nice list of alternatives, which I reproduce here (and to which I�d add, of course, bibliography management)</p> <ul> <li>Sitemaps: one of the S�s in <span class="caps">RSS</span> stands for summary. A sitemap is a summary of the content on a site, the items are pages or content areas. This is clearly a non-chronological ordering of items. Is a hierarchy of <span class="caps">RSS</span> sitemaps implied here � how would the linking between them work? How hard would it be to hack a web browser to pick up the <span class="caps">RSS</span> sitemap and display it in a sidebar when you visit the site?</li> <li>Small ads: also known as classifieds. These expire so there�s some kind of dynamic going on here but the ordering of items isn�t necessarily chronological. How to describe the location of the seller, or the condition of the item or even the price. Not every ad is selling something � perhaps it�s to rent out a room.</li> <li>Personals: similar model to the small ads. No prices though (I hope). Comes with a ready made vocabulary of terms that could be converted to an <span class="caps">RDF</span> schema. Probably should do that just for the hell of it anyway � gsoh</li> <li>Weather reports: how about a week�s worth of weather in an <span class="caps">RSS</span> channel. If an item is dated in the future, should an aggregator display it before time? Alternate representations include maps of temperature and pressure etc.</li> <li>Auctions: again, related to small ads, but these are much more time limited since there is a hard cutoff after which the auction is closed. The sequence of bids could be interesting � would it make sense to thread them like a discussion so you can see the tactics?</li> <li>TV listings: this is definitely chronological but with a twist � the items have durations. They also have other metadata such as cast lists, classification ratings, widescreen, stereo, program type. Some types have additional information such as director and production year.</li> <li>Top ten listings: top ten singles, books, dvds, richest people, ugliest, rear of the year etc. Not chronological, but has definate order. May update from day to day or even more often.</li> <li>Sales reporting: imagine if every department of a company reported their sales figures via <span class="caps">RSS</span>. Then the divisions aggregate the departmental figures and republish to the regional offices, who aggregate and add value up the chain. The chairman of the company subscribes to one super-aggregate feed.</li> <li>Membership lists / buddy lists: could I publish my buddy list from Jabber or other instant messengers? Maybe as an interchange format or perhaps could be used to look for shared contacts. Lots of potential overlap with <span class="caps">FOAF</span> here.</li> <li>Mailing lists: or in fact any messaging system such as usenet. There are some efforts at doing this already (e.g. yahoogroups) but we need more information � threads; references; headers; links into archives.</li> <li>Price lists / inventory: the items here are products or services. No particular ordering but it�d be nice to be able to subscribe to a catalog of products and prices from a company. The aggregator should be able to pick out price rises or bargains given enough history.</li> <div align="right">[via <a href="http://jena.hpl.hp.com:3030/blojsom-hp/blog/">Semantic Blogging Demonstrator</a>] </div></ul></span></blockquote> <p><span style="font-size: 10pt; font-family: Arial;">Thus, if we can comprehend RSS (the blog article below does a great job) we should be able to see the fundamental challenges that are before any organization seeking to exploit the potential of the imminent Web 2.0 inflection; how will you cost-effectively create XML data from existing data sources? Without upgrading or switching database engines, operating systems, programming languages? Put differently how can you exploit this phenomenon without losing your ever dwindling technology choices (believe me choices are dwindling fast but most are oblivious to this fact).</span></p><p xmlns="o"></p> <p> </p> <a href="index.vspx?tag=xml" rel="tag" style="display:none;">xml</a><a href="index.vspx?tag=rss" rel="tag" style="display:none;">rss</a><a href="index.vspx?tag=syndication" rel="tag" style="display:none;">syndication</a>
Interesting Database History: INFORMIX
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2003-06-26#192
2003-06-26T23:45:45Z
2006-06-22T08:56:58-04:00
<!--StartFragment --><A href="http://www.wikipedia.org/wiki/Informix">Interesting Database History: INFORMIX</A> <P class=subtitle>From <A href="http://www.wikipedia.org/">Wikipedia</A>, the free encyclopedia. </P> <P><STRONG>Informix</STRONG> is a <A class=internal title="Relational database" href="http://www.wikipedia.org/wiki/Relational_database">relational database</A> and for almost 20 years was also the name of the company who developed it. Informix DBMS was a development of the pioneering <A class=internal title=Ingres href="http://www.wikipedia.org/wiki/Ingres">Ingres</A> system that also led to <A class=internal title=Sybase href="http://www.wikipedia.org/wiki/Sybase">Sybase</A> and <A class=internal title="SQL Server" href="http://www.wikipedia.org/wiki/SQL_Server">SQL Server</A>, and was the #2 database system behind <A class=internal title=Oracle href="http://www.wikipedia.org/wiki/Oracle">Oracle</A> for some time in the 1990s. Their brush with success was surprisingly short-lived however, and by 2000 a series of management blunders had all but destroyed the company. In <A class=internal title=2001 href="http://www.wikipedia.org/wiki/2001">2001</A> they were purchased by <A class=internal title=IBM href="http://www.wikipedia.org/wiki/IBM">IBM</A> in order to gain access to Informix's existing market share and customer base. Long term plans to merge Informix technology with <A class=internal title=DB2 href="http://www.wikipedia.org/wiki/DB2">DB2</A> are in place, since the Informix Arrowhead project is now called DB2 Arrowhead. <A class=internal title=IBM href="http://www.wikipedia.org/wiki/IBM">IBM</A> is also commited in supporting older versions. </P> <P><A href="http://www.wikipedia.org/wiki/Informix">Read on.</A></P>
How Databases Changed The World
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2003-06-09#113
2003-06-09T09:28:17Z
2006-06-22T08:56:58-04:00
<a href="http://searchdatabase.techtarget.com/bestWebLinks/0,289521,sid13_tax281575,00.html"><b>How Databases Changed The World</b></a> by Tim DiChiara, Site Editor (<a href="http://www.searchdatabase.com">SearchDatabase.com</a>) How did the database industry get started? How has it changed the face of business? What were the key milestones, the big obstacles and the lessons learned? I recently came across an interesting panel discussion addressing these very issues, featuring many of the database pioneers and leaders of the last 30 years: Chris Date, Herb Edelstein, Bob Epstein, Ken Jacobs, Pat Selinger, Roger Sippl and Michael Stonebraker. It's available via streaming <a href="http://www.computerhistory.org/events/lectures/db_02102003/">video</a> and was recorded in February at the Computer History Museum in Mountain View, California. After a chatty and lengthy (45 minutes!) introduction only interesting to hardcore insiders, you can see Chris Date waxing eloquent about Ted Codd (complete with quotes from Shakespeare, no less), Herb Edelstein waxing eloquent about Chris Date, and Michael Stonebraker at his geeky best. There's also interesting trivia about the beginnings of SQL, the role of INGRES, why the relational model will stand the test of time and some friendly Oracle and IBM bashing (and Microsoft and Sybase and...). I urge all you data management pros interested in broadening your knowledge of the field to check it out! If you're still not satiated, don't forget about our collection of backgrounders about the DBMS and the data management industry.
How Databases Changed The World
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2003-06-09#266
2003-06-09T09:28:17Z
2006-06-22T08:56:58-04:00
<p><a href="http://searchdatabase.techtarget.com/bestWebLinks/0,289521,sid13_tax281575,00.html"><b>How Databases Changed The World</b></a> by Tim DiChiara, Site Editor (<a href="http://www.searchdatabase.com">SearchDatabase.com</a>) </p><p>How did the database industry get started? How has it changed the face of business? What were the key milestones, the big obstacles and the lessons learned? I recently came across an interesting panel discussion addressing these very issues, featuring many of the database pioneers and leaders of the last 30 years:</p><a href="http://en.wikipedia.org/wiki/Chris_Date">Chris Date</a><br /><a href="http://www.computerhistory.org/events/lectures/db_02102003/edelstein/">Herb Edelstein</a> <br /><a href="http://www.computerhistory.org/events/lectures/db_02102003/epstein/">Bob Epstein</a> (<a href="http://en.wikipedia.org/wiki/Sybase">Sybase</a> who shared code with Microsoft for remarketing on SQL Server on OS/2 which inevitably lead to the <a href="http://en.wikipedia.org/wiki/Microsoft_SQL_Server">Microsoft SQL Server</a> we know today)<br /><a href="http://www.oracle.com/corporate/pressroom/html/kjacobs.html">Ken Jacobs</a> (<a href="http://en.wikipedia.org/wiki/Oracle_database">Oracle</a>'s Dr. DBA)<br /><a href="http://www.witi.com/center/witimuseum/halloffame/2004/pselinger.php">Pat Selinger </a> (<a href="http://en.wikipedia.org/wiki/DB2">DB2</a> precursor called System R) <br /><a href="http://en.wikipedia.org/wiki/Informix">Roger Sippl</a> (<a href="http://en.wikipedia.org/wiki/Informix">Informix</a>)<br /><a href="http://en.wikipedia.org/wiki/Michael_Stonebraker">Michael Stonebraker</a> (<a href="http://en.wikipedia.org/wiki/Ingres">Ingres</a>, <a href="http://en.wikipedia.org/wiki/PostgreSQL">Postgres</a>, and <a href="http://mariposa.cs.berkeley.edu/about.html">Mariposa</a>)<br /><br />The event is available via streaming <a href="http://www.computerhistory.org/events/lectures/db_02102003/">video</a> and was recorded in February at the Computer History Museum in Mountain View, California. After a chatty and lengthy (45 minutes!) introduction only interesting to hardcore insiders, you can see Chris Date waxing eloquent about Ted Codd (complete with quotes from Shakespeare, no less), Herb Edelstein waxing eloquent about Chris Date, and Michael Stonebraker at his geeky best. There's also interesting trivia about the beginnings of SQL, the role of INGRES, why the relational model will stand the test of time and some friendly Oracle and IBM bashing (and Microsoft and Sybase and...). I urge all you data management pros interested in broadening your knowledge of the field to check it out! If you're still not satiated, don't forget about our collection of backgrounders about the DBMS and the data management industry. <a href="index.vspx?tag=sql" rel="tag" style="display:none;">sql</a><a href="index.vspx?tag=rdbms" rel="tag" style="display:none;">rdbms</a><a href="index.vspx?tag=database" rel="tag" style="display:none;">database</a>
The Market For Money
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2003-06-05#254
2003-06-06T03:06:53Z
2006-06-22T08:56:58-04:00
<a href="http://www.ventureblog.com/articles/indiv/2003/000119.html">The Market For Money</a> <p>I just read this on <a href="http://www.loftesness.com/radio/">Scott Loftesness's blog</a> and thought it was worth sharing. <a href="http://www.loftesness.com/career.html">Scott</a> was an EVP at Visa in the early 90's and his blog is an unbelievably comprehensive discussion of the payments space. Here is his discussion of Visa's recent announcement that this Visa system had reached $1 Trillion in annual United States transaction volume. It is an amazing growth curve and reminds me of a comment Peter Thiel, the former CEO of PayPal, made to me one day when we were having lunch -- he said that when he was pitching PayPal to VCs he was tempted to describe his market opportunity as the "market for money." Visa's numbers prove that that is precisely their market. VC's are always looking for big markets to penetrate and the market for money certainly qualifies. Here are Scott's thoughts:</p> <blockquote>Visa USA announced this morning that, for the first time, its annual sales volume exceeded $1 trillion. </blockquote> <blockquote>The record usage means that an average of $32,000 went through the Visa system every second of every day over the 12-month period that ended March 31 - or nearly 10 percent of the 2002 U.S. Gross Domestic Product. </blockquote> <blockquote>"One trillion dollars is an almost incomprehensible number, but it represents clear evidence of the silent revolution we're witnessing in the way consumers pay for goods and services. It means $12 of every $100 consumers spent in the U.S. is spent using a Visa card," said Carl Pascarella, president and CEO of Visa USA. "This is an important milestone in the history of U.S. commerce. Clearly, more and more people rely upon the security and convenience of Visa credit, debit and other payment products. To put it into context, $1 trillion could buy 162,000 Harley-Davidson motorcycles every day for a year."</blockquote> <blockquote>By comparison, $1 trillion is greater than the combined volume of all other U.S. payment organizations, a field that includes MasterCard, American Express, Discover and others.</blockquote> <blockquote>Just before I left Visa in 1994, I remember having a discussion with a colleague about growth in sales volume. 1993 had just ended with $500 billion in annual Visa sales on an international basis. We were focused on that total growing to $1 trillion globally over the next five years. As I recall, the US in 1993 was about 40+% of the global total -- so the growth in US volume over the last nine years has been pretty amazing. Of course, this is also one of those statistics that has a nice built-in inflation hedge too (the numbers just keep growing!). $32,000 a second -- at a $50 average ticket that works out to an average of 640 Visa transactions per second.[via <a href="http://www.ventureblog.com/">VentureBlog</a>]</blockquote> <div></div>
Who's handing out the crack at Microsoft?
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2003-06-03#357
2003-06-03T19:22:15Z
2006-06-22T08:56:58-04:00
<A href="http://www.surfmind.com/musings///2003/05/31/">Who's handing out the crack at Microsoft?</A> IE6 is the last non-OS based release of the browser? "Futher improvements to IE will require enhancements to the underlying OS". <A href="http://www.microsoft.com/technet/treeview/default.asp?url=/technet/itcommunity/chats/trans/ie/ie0507.asp">source</A>, via <A href="http://techno-weenie.com/archives/2003/05/30/003134.php">techno-weenie</A>.<BR><BR>You gotta be kidding. This is great news for Mozilla, even give the recent <A href="http://www.mozillazine.org/talkback.html?article=3226">AOL prostitution</A>. There are huge strides left to be made in the browser UI -- and they have huge potential impact. There is no other software paradigm in the history of computers that's used and usable by as many people.<BR><BR>Others in the blogdom have questioned the economic payoff of improving IE, given a 88%+ market share. The abandonment of standards based progress by MSoft is reprehensible. [via <A href="http://surfmind.com/musings/">Surf*Mind*Musings</A>] <DIV></DIV>
Who's handing out the crack at Microsoft?
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2003-06-03#87
2003-06-03T19:22:15Z
2006-06-22T08:56:58-04:00
<A href="http://www.surfmind.com/musings///2003/05/31/">Who's handing out the crack at Microsoft?</A> IE6 is the last non-OS based release of the browser? "Futher improvements to IE will require enhancements to the underlying OS". <A href="http://www.microsoft.com/technet/treeview/default.asp?url=/technet/itcommunity/chats/trans/ie/ie0507.asp">source</A>, via <A href="http://techno-weenie.com/archives/2003/05/30/003134.php">techno-weenie</A>.<BR><BR>You gotta be kidding. This is great news for Mozilla, even give the recent <A href="http://www.mozillazine.org/talkback.html?article=3226">AOL prostitution</A>. There are huge strides left to be made in the browser UI -- and they have huge potential impact. There is no other software paradigm in the history of computers that's used and usable by as many people.<BR><BR>Others in the blogdom have questioned the economic payoff of improving IE, given a 88%+ market share. The abandonment of standards based progress by MSoft is reprehensible. [via <A href="http://surfmind.com/musings/">Surf*Mind*Musings</A>] <DIV></DIV>
<a href="http://blogs.law.harvard.edu/isItTheSyntax">My thoughts</a>
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2003-05-23#327
2003-05-23T15:39:41Z
2006-06-22T08:56:58-04:00
<P><A href="http://blogs.law.harvard.edu/isItTheSyntax">My thoughts</A> re Tim Bray's thread on RDF. [via <A href="http://www.scripting.com/">Scripting News</A>]</P> <P><EM>Key excerpt of relevance to us (as potential providers of an application that demonstrates RDFs value prop.):</EM></P> <P>It's not the syntax that makes the difference, it's the app. History supports this view. How many people tried to pry apart the <A href="http://dictionary.reference.com/search?q=obscure"><STRONG><FONT color=#920011>obscure</FONT></STRONG></A> Excel file format on the Mac? Or the Lotus file format on the PC? Name all the market leaders of the past, and only the Web had both the killer app and a transparent format. Maybe the relationship is multiplicative. Maybe Excel would have <I>been the Web</I> if it had used an open file format that anyone could understand. What if you could have created a spreadsheet with BBEdit or a HyperTalk script? The mind boggles at the possibilities (it never happened, of course).</P> <P><EM>Even in Office 2003 there is a failure to really open things up.</EM><BR><BR>An aside, <A href="http://scriptingnews.userland.com/weblogsearch/?q=paoli"><STRONG><FONT color=#920011>Jean Paoli</FONT></STRONG></A> rushes into the room, jumping up and down and saying "That's what I'm doing that's what I'm doing."<BR><BR>Anyway, I don't see any killer apps in the RDF crowd. I see lots of people with strong opinions and not much software. Killer apps are not something you wish into existence. Lots of people have said that RDF models a relational database. Okay that tells me something important, the killer app is a relational database. </P> <P><EM>Ha Ha!</EM></P> <P>But we already have relational databases. They were new when I was a grad student, and that was a <I>long</I> time ago. <IMG src="http://static.userland.com/shortcuts/images/qbullets/sidesmiley.gif"><BR><BR><EM>Yeah, but what we don't have is a relational databases that incorporate RDF as part of the database technology evolution roadmap. Of course many will get it (and FUD-emulate) when we unveil something via Virtuoso.</EM></P> <DIV></DIV>
<a href="http://blogs.law.harvard.edu/isItTheSyntax">My thoughts</a>
http://www.openlinksw.com/blog/kidehen@openlinksw.com/blog/?date=2003-05-23#55
2003-05-23T15:39:41Z
2006-06-22T08:56:58-04:00
<p><a href="http://blogs.law.harvard.edu/isItTheSyntax">My thoughts</a> re Tim Bray's thread on RDF. [via <a href="http://www.scripting.com/">Scripting News</a>]</p> <p><em>Key excerpt of relevance to us (as potential providers of an application that demonstrates RDFs value prop.):</em></p> <p>It's not the syntax that makes the difference, it's the app. History supports this view. How many people tried to pry apart the <a href="http://dictionary.reference.com/search?q=obscure"><strong><font color="#920011">obscure</font></strong></a> Excel file format on the Mac? Or the Lotus file format on the PC? Name all the market leaders of the past, and only the Web had both the killer app and a transparent format. Maybe the relationship is multiplicative. Maybe Excel would have <i>been the Web</i> if it had used an open file format that anyone could understand. What if you could have created a spreadsheet with BBEdit or a HyperTalk script? The mind boggles at the possibilities (it never happened, of course).</p> <p><em>Even in Office 2003 there is a failure to really open things up.</em><br /><br />An aside, <a href="http://scriptingnews.userland.com/weblogsearch/?q=paoli"><strong><font color="#920011">Jean Paoli</font></strong></a> rushes into the room, jumping up and down and saying "That's what I'm doing that's what I'm doing."<br /><br />Anyway, I don't see any killer apps in the RDF crowd. I see lots of people with strong opinions and not much software. Killer apps are not something you wish into existence. Lots of people have said that RDF models a relational database. Okay that tells me something important, the killer app is a relational database. </p> <p><em>Ha Ha!</em></p> <p>But we already have relational databases. They were new when I was a grad student, and that was a <i>long</i> time ago. <img src="http://static.userland.com/shortcuts/images/qbullets/sidesmiley.gif" /><br /><br /><em>Yeah, but what we don't have is a relational databases that incorporate RDF as part of the database technology evolution roadmap. Of course many will get it (and FUD-emulate) when we unveil something via Virtuoso.</em></p> <div></div>