OpenLink Virtuoso (Product Blog)
2012-02-11T04:29:06Z
Virtuso Data Space Bot
kidehen@openlinksw.com
About
<p>We have just added a geometry <a href="http://dbpedia.org/resource/Data" id="link-id0x1c0e02b0">data</a> type and corresponding <a href="http://dbpedia.org/resource/R-tree" id="link-id0x1e093220">R</a>-tree index to <a href="http://virtuoso.openlinksw.com" id="link-id0x1ddccfe8">Virtuoso</a>. This follows the general scheme of <a href="http://dbpedia.org/resource/SQL" id="link-id0x1b88a580">SQL</a>/MM, as is implemented by <a href="http://dbpedia.org/resource/PostGIS" id="link-id0x1d271a90">PostGIS</a> and many others. We have all the engine-side stuff, including optimizer support for geometry cardinality sampling and good execution plans for combinations of spatial and other joins. We have however not yet implemented all the different geometry types and library function support for them, like shortest distance between two arbitrary shapes.</p> <p>The geometry support is for both SQL and <a href="http://dbpedia.org/resource/SPARQL" id="link-id0x1b8d4ca8">SPARQL</a>. On the SQL side, it works with the ISO/IEC 13249 SQL/MM API; with <a href="http://dbpedia.org/resource/Resource_Description_Framework" id="link-id0x1ed69318">RDF</a>, a geometry can occur as the object of a quad. If the object is a typed-literal of the <code>virtrdf:Geometry</code> type, it gets indexed in a geometry index over all geometries in quads; no special declarations are needed. After this, SQL MM predicates and functions can be used with SPARQL, like this:</p> <blockquote> <pre><code> PREFIX geo: <<a href="http://dbpedia.org/resource/Hypertext_Transfer_Protocol" id="link-id0x1d2d0ae0">http</a>://www.w3.org/2003/01/geo/wgs84_pos#> SELECT ?class COUNT (*) WHERE { ?m geo:geometry ?geo . ?m a ?class . FILTER ( <bif:st_intersects> ( ?geo, <bif:st_point> (0, 52), 100 ) ) } GROUP BY ?class ORDER BY DESC 2 </code> </pre></blockquote> <p>This returns the counts of objects of each class occurring within 100 km of (0, 52), a point near London.</p> <p>For any data set with <a href="http://dbpedia.org/resource/World_Geodetic_System" id="link-id0x1ec00578">WGS 84</a> <code>geo:long</code> and <code>geo:lat</code> values, a simple SQL function makes a point geometry for each such coordinate pair and adds it as the <code>geo:geometry</code> property of the subject with the long/lat. This then enables fast spatial access to arbitrary location data in RDF.</p> <p>Right now, we hardly see any geometries other than points in RDF data, even though there are some efforts for vocabularies for more complex entities. As these get adopted we will support them.</p> <p>For scalability, we tried the implementation with <a href="http://www.openstreetmap.org/" id="link-id0x1c781e68">OpenStreetMap</a>'s 350 million or so points. The geometry implementation partitions well over a cluster, similarly to a full text index, i.e., every server has its slice of the geometries, partitioned by the geometry object's key, thus not by range of coordinates or such. Like this, the items are evenly spread even though the coordinate distribution is highly uneven.</p> <p>We can do spatial joins like —</p> <blockquote> <pre><code> SELECT ?s ( <sql:num_or_null> (?p) ) COUNT (*) WHERE { ?s <http://<a href="http://dbpedia.org/resource/DBpedia" id="link-id0x1f885868">dbpedia</a>.org/ontology/populationTotal> ?p . FILTER ( <sql:num_or_null> (?p) > 1000000 ) . ?s geo:geometry ?geo . FILTER ( <bif:st_intersects> ( ?pt, ?geo, 5 ) ) . ?xx geo:geometry ?pt } GROUP BY ?s ( <sql:num_or_null> (?p) ) ORDER BY DESC 3 LIMIT 20 </code> </pre></blockquote> <p>This takes the DBpedia subjects that have a population over 1 million and a geometry. We then count all the geometries within 5 km of the point location of the first geometry. With DBpedia (about 5 million points), <a href="http://www.geonames.org/" id="link-id0x1d4279b0">GeoNames</a> (7 million points), and OpenStreetMap (350 million points), we get the result:</p> <blockquote> <pre><code>http://dbpedia.org/resource/Munich 1356594 117280 http://dbpedia.org/resource/London 7355400 81486 http://dbpedia.org/resource/Davao_City 1363337 58640 http://dbpedia.org/resource/Belo_Horizonte 2412937 58640 http://dbpedia.org/resource/Chengde 3610000 58640 http://dbpedia.org/resource/Hamburg 1769117 51664 http://dbpedia.org/resource/San_Diego%2C_California 1266731 47685 http://dbpedia.org/resource/Bursa 1562828 47685 http://dbpedia.org/resource/Port-au-Prince 1082800 47685 http://dbpedia.org/resource/Oakland_County%2C_Michigan 1194156 45636 http://dbpedia.org/resource/Sana%27a 1747627 40923 http://dbpedia.org/resource/Milan 1303437 40923 http://dbpedia.org/resource/Campinas 1059420 40923 http://dbpedia.org/resource/Hohhot 2580000 40923 http://dbpedia.org/resource/Brussels 1031215 40923 http://dbpedia.org/resource/Bogra_District 2988567 40923 http://dbpedia.org/resource/Cort%C3%A9s_Department 1202510 40923 http://dbpedia.org/resource/Berlin 3416300 35668 http://dbpedia.org/resource/New_York_City 8274527 30810 http://dbpedia.org/resource/Los_Angeles%2C_California 3849378 25614<br /> 20 Rows. -- 1733 msec.<br /> Cluster 8 nodes, 1 s. 358 m/s 1596 KB/s 664% <a href="http://dbpedia.org/resource/Central_processing_unit" id="link-id0x1e6403b0">cpu</a> 2% read 16% clw threads 1r 0w 0i buffers 1124351 0 d 0 w 0 pfs </code></pre></blockquote> <p>This takes 1.7 seconds on a Virtuoso Cluster configured with 8 processes on a single dual-Xeon 5520 box, running at about 664% CPU with warm <a href="http://dbpedia.org/resource/Cache" id="link-id0x1e81f610">cache</a>. Fair enough for a first crack, this can obviously be optimized further. Still, the geo part of the processing is already as good as instantaneous.</p> <p>We will shortly have the geography features installed on DBpedia and the other data sets we host. As these come online we will show more demo queries.</p> <p>For more about SQL/MM, you can look to a couple of PDFs:</p> <ul> <li> <a href="http://www.fer.hr/_download/repository/SQLMM_Spatial-_The_Standard_to_Manage_Spatial_Data_in_Relational_Database_Systems.pdf" id="link-id133775f0">SQL/MM Spatial: The Standard to Manage Spatial Data in Relational Database Systems</a> by Knut Stolze</li> <li> <a href="http://www.sigmod.org/record/issues/0112/standards.pdf" id="link-id1433c5e0">SQL Multimedia and Application Packages (SQL/MM)</a> by Jim Melton and Andrew Eisenberg</li> </ul>
RDF Geography With Virtuoso
2010-02-01T14:14:29Z
2010-02-01T14:14:29Z
2010-02-01T09:14:29.000012-05:00
Virtuso Data Space Bot
kidehen@openlinksw.com