Location: / Dashboard / Main / VirtBulkRDFLoader

Bulk Loading RDF Source Files into one or more Graph IRIs

This document details how large RDF data set files can be bulk-loaded into Virtuoso. The data sets may consist of multiple files, which may be loaded into one or several graphs.

Prerequisites
Bulk loading process
Stopping the bulk load process
Checking bulk load status
Cluster-specific details
Related

Prerequisites

If your Virtuoso release is prior to the commercial 06.02.3129 or open source 6.1.3 releases, then the Virtuoso Bulk Loader functions need to be loaded manually.
The directory containing the data set files must be included in the DirsAllowed parameter defined in the virtuoso INI file, after which the Virtuoso server must be restarted.
The Virtuoso Server should be appropriately configured to use sufficient memory and other system resources as detailed in the Virtuoso RDF Performance Tuning Guide, or the load may take an unacceptably long time, approaching forever.

Bulk loading process

The name of the RDF graph into which the data set(s) should be loaded can be specified through a text file placed in the same source directory as the source data files. This will override the graph name specified in the ld_dir() or ld_dir_all() function call. The content of a file with the same name as a data file plus the .graph filename extension will be used for that data file (e.g., my_data.n3.graph will be used with my_data.n3). The content of a file named global.graph will be used for any and all other data files in that directory. Note: if the third parameter (graph_iri) of ld_dir() or ld_dir_all() is NULL, any data files that do not have a corresponding .graph file will not be loaded.
```
<source-file>.<ext>
<source-file>.<ext>.graph
global.graph
```
— e.g., —
```
myfile.n3          ;; RDF data
myfile.n3.graph    ;; Contains Graph IRI name into which RDF data from myfile.n3 will be loaded
global.graph       ;; Contains Graph IRI name into which RDF data from any files that do not have a specific graph name file will be loaded
```
Place the graph IRI, , e.g., http://dbpedia.org, in the *.graph file.
Use isql to register the file(s) to be loaded by running the appropriate function, e.g. --
```
SQL> ld_dir ('/path/to/files', '*.n3', 'http://dbpedia.org');
```
- ld_dir() to load only from the specified directory, excluding any subdirectories --
```
SQL> ld_dir ('<source-filename-or-directory>', '<file name pattern>', 'graph iri');
```
- ld_dir_all() to load from the specified directory, including any and all subdirectories --
```
SQL> ld_dir_all ('<source-filename-or-directory>', '<file name pattern>', 'graph iri');
```

The table DB.DBA.load_list can be used to check the list of data sets registered for loading, and the graph IRIs into which they will be or have been loaded. The ll_state field can have three values: 0 indicating the data set is to be loaded; 1 the data set load is in progress; or 2 the data set load is complete:


SQL> select * from DB.DBA.load_list;
ll_file               ll_graph        ll_state    ll_started              ll_done                 ll_host    ll_work_time    ll_error
VARCHAR NOT NULL      VARCHAR         INTEGER     TIMESTAMP               TIMESTAMP               INTEGER    INTEGER         VARCHAR
_____________________________________________________________________________________________________________________________________

./dump/d1/file1.n3    http://file1    2           2010.10.20 9:21.18 0    2010.10.20 9:21.18 0    0          NULL            NULL
./dump/d2/file2.n3    http://file2    2           2010.10.20 9:21.18 0    2010.10.20 9:21.18 0    0          NULL            NULL
./dump/file.n3        http://file     2           2010.10.20 9:21.18 0    2010.10.20 9:21.18 0    0          NULL            NULL

3 Rows. -- 1 msec.
SQL>

Finally, perform the bulk load of all data by executing the rdf_loader_run() function:
```
SQL> rdf_loader_run();
```
- Note: the rdf_loader_run() function prototype is:
```
rdf_loader_run (in max_files integer := null, in log_enable int := 2)
```
  One of the side effects of the default log_enable = 2 setting is that triggers are not enabled to speed the loading of data. If triggers are required for RDF Graph replication between nodes etc. then the log_enable mode should be set to 3 when calling the rdf_loader_run() function as follows:
```
rdf_loader_run (log_enable=>3);
```

Stopping the bulk load process

All RDF loader threads can be stopped using the command rdf_load_stop(), at which point all currently running threads will be allowed to complete and then exit:
```
SQL> rdf_load_stop();
```

Checking bulk load status

Once the rdf_loader_run() is complete, you can check the DB.DBA.load_list to confirm all data sets were loaded successfully. This is indicated by an ll_state value of 2 and an ll_error value of NULL.

Cluster-specific details

On a Virtuoso Clustered Server the "cl_exec('rdf_ld_srv(log_enable)')" commands (where log_enable is 2 or 3, as with the rdf_loader_run() function) can be used to invoke a single "rdf_loader_run()" on each node of the cluster:
```
SQL> cl_exec('rdf_ld_srv()'); 

Done. -- 265956 msec.
SQL>
```

Feeds