The Data Space PhilosophyBarCampOxford Announcements

# Ways to get Data into Virtuoso

semweb @ 06 February 2008

The many different ways to get data into Virtuoso:

[Warning]Contains quite technical information, its suggested that you know a little bit about the Semantic Web before you read. See the Semantic Web FAQ document for more information about the Semantic Web.[/Warning]

There are many different ways of getting data into Virtuoso, I’ll give you a more detailed run through but to get you started here is a list:

  • RDF Sink Folders
  • HTTP Put
  • HTTP Post
  • SPARQL Insert via /sparql endpoint
  • SPARQL Insert via HTTP Post using Content-Type: application/sparql-query
  • Using the Virtuoso Crawler
  • Using a Sponger with a SPARQL Query
  • Using Virtuoso PL APIs
  • Using the SIMILE RDF Bank API
  • Using RDF NET

RDF Sink Folders

An RDF Sink folder is a special folder that is part of Virtuosos WebDAV system.

  1. When a new user is created, then a new WebDAV folder with name rdf_sink is created in the user’s home directory.
  2. The folder has two properties - virt:rdf_graph and virt:rdf_sponger. Names of the WebDAV properties are prefixed with virt:. The properties of the other DET folders are also prefixed in the same way. Properties prefixed with virt: are not shown as resource/collection properties in the ODS Briefcase - they can be seen only in the Conductor. Uploaded resources in the rdf_sink folder don’t have any special properties.
    1. The property virt:rdf_graph contains the name of the graph for uploads.
    2. The property virt:rdf_sponger points to the upload method. If it is “on” then the content of every file uploaded to that folder is added to RDF store using the Sponger method. If this value is “off” then only files with special mime types are added to RDF store. After the first upload into the folder a new file is created too with a name based on rdf graph name - this file is dynamic and contains all uploaded triplets;
  3. For the existing users is added a procedure for automatic creation of the rdf_sink folder in their home directory;
  4. The Conductor add users page is changed so that when a new user is created with option for creating WebDAV home directory, then the rdf_sink folder is created too;

From the Conductor UI an user may see properties for this folder using properies button at the right of the folder name. The opened page shows two WebDAV properties:

  1. virt:rdf_graph: for ex. https://demo.openlinksw.com/DAV/home/demo/rdf_sink
  2. virt:rdf_sponger: for ex. “on”.

Example:
File: myfoaf.rdf
Destination Server: demo.openlinksw.com
curl -T myfoaf.rdf https://demo.openlinksw.com/DAV/home/demo/rdf_sink/myfoaf.rdf -u demo:demo
Resource /DAV/home/demo/rdf_sink/ myfoaf.rdf has been created.

HTTP Put

Using this approach we can upload RDF resources into Virtuoso’s WebDAV repository and then automatically upload the triples within the resource into the Virtuoso Quad store (if you so choose via Virtuoso’s special RDF Sink Folder feature of WebDAV ).

Example:
File: myfoaf.rdf
Destination Server: demo.openlinksw.com
curl -T myfoaf.rdf https://demo.openlinksw.com/DAV/home/demo/rdf_sink/myfoaf.rdf -u demo:demo

Resource /DAV/home/demo/rdf_sink/ myfoaf.rdf has been created.

HTTP POST

Virtuoso’s support for HTTP content negotiation and the SPARQL Protocol enables the execution of SPARUL as part of an HTTP POST payload using Content-Type: application/sparql-query.

1. Create a DAV collection and not to request for authentication set RW permissions for public. For ex. the collection path could be /DAV/xx

2. Execute the following command:
curl -i -d "INSERT {
}"
-H "Content-Type: application/sparql-query" https://localhost:8890/DAV/xx/yy

3. The response should be:
HTTP/1.1 201 Created
Server: Virtuoso/05.00.3023 (Win32) i686-generic-win-32 VDB
Connection: Keep-Alive
Content-Type: text/html; charset=ISO-8859-1
Date: Fri, 28 Dec 2007 12:50:12 GMT
Accept-Ranges: bytes
MS-Author-Via: SPARQL
Content-Length: 0

4. the result in the DAV/xx location will be a new WebDAV resource with name “yy” containing the following:
# if opened with Conductor:
CONSTRUCT { ?s ?p ?o } FROM WHERE { ?s ?p ?o }
# if opened with HTTP GET, then the content will be RDF representation of what was inserted into the graph, i.e:
<?xml version="1.0" encoding="utf-8" ?>
<rdf:RDF xmlns:rdf="https://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:rdfs="https://www.w3.org/2000/01/rdf-schema#">
<rdf:Description
rdf:about="https://www.openlinksw.com/dataspace/[email protected]#this">
<ns0pred:type xmlns:ns0pred="https://www.w3.org/1999/02/22-rdf-syntax-ns#"
rdf:resource="https://rdfs.org/sioc/ns#User"/>
</rdf:Description>
</rdf:RDF>

SPARQL Insert via /sparql endpoint

SPARQL INSERT operation can be sent to a web service endpoint as a single statement and executed in sequence.

Example:

Using the Virtuoso ISQL tool or using the /sparql UI, for ex. at https://localhost:8890/sparql, execute the following:

1. Insert into graph https://BookStore.com 3 triples:
sparql insert in graph <https://BookStore.com>
{ <https://www.dajobe.org/foaf.rdf#i> <https://purl.org/dc/elements/1.1/date> <1999-04-01T00:00:00> .
<https://www.w3.org/People/Berners-Lee/card#i> <https://purl.org/dc/elements/1.1/date> <1998-05-03T00:00:00> .
<https://www.w3.org/People/Connolly/#me> <https://purl.org/dc/elements/1.1/date> <2001-02-08T00:00:00> };
2. As result will be shown the message:
Insert into <https://BookStore.com>, 3 triples -- done

3. Next we will select all triples from the graph https://BookStore.com:
sparql select * from <https://BookStore.com> where {?s ?p ?o};
4. As result will be shown:
s p o
VARCHAR VARCHAR VARCHAR
_______________________________________________________________________________

https://www.w3.org/People/Berners-Lee/card#i https://purl.org/dc/elements/1.1/date 1998-05-03T00:00:00
https://www.w3.org/People/Connolly/#me https://purl.org/dc/elements/1.1/date 2001-02-08T00:00:00
https://www.dajobe.org/foaf.rdf#i https://purl.org/dc/elements/1.1/date 1999-04-01T00:00:00

3 Rows. -- 0 msec.
5. Now let’s insert into graph another https://NewBookStore.com graph’s values:
sparql
PREFIX dc: <https://purl.org/dc/elements/1.1/>
PREFIX xsd: <https://www.w3.org/2001/XMLSchema#>
INSERT INTO GRAPH <https://NewBookStore.com> { ?book ?p ?v }
WHERE
{ GRAPH <https://BookStore.com>
{ ?book dc:date ?date
FILTER ( xsd:dateTime(?date) < xsd:dateTime("2000-01-01T00:00:00")).
?book ?p ?v.
}
};
6. As result will be shown:
callret-0
VARCHAR
_______________________________________________________________________________

Insert into <https://NewBookStore.com>, 2 triples -- done
7. Finally we will check the triples from the graph NewBookStore.com:
SQL> sparql select * from <https://NewBookStore.com> where {?s ?p ?o};
8. As result will be shown:
s p o
VARCHAR VARCHAR VARCHAR
_______________________________________________________________________________

https://www.w3.org/People/Berners-Lee/card#i https://purl.org/dc/elements/1.1/date 1998-05-03T00:00:00
https://www.dajobe.org/foaf.rdf#i https://purl.org/dc/elements/1.1/date 1999-04-01T00:00:00

2 Rows. -- 10 msec.
SPARQL Insert via HTTP Post Using Content-Type: application/sparql-query and ODS Wiki

  • With HTTP Post and ODS wiki can be written an rdf document and respectively to be performed over it INSERT/UPDATE action.
  • You can write to a file using SIOC terms for ODS-Wiki
  • You can check with sparql the inserted / updated triples in the Quad Store.

Example:

1. Suppose there is ODS user test3 with ODS password 1, which has testWiki wiki instance.

2. Execute the following:
curl -i -d "INSERT {<https://localhost:8890/dataspace/test3/wiki/testWiki> <https://atomowl.org/ontologies/atomrdf#contains> <https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest> . <https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest> <https://rdfs.org/sioc/ns#has_container> <https://localhost:8890/dataspace/test3/wiki/testWiki> . <https://localhost:8890/dataspace/test3/wiki/testWiki> <https://atomowl.org/ontologies/atomrdf#entry> <https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest> . <https://localhost:8890/dataspace/test3/wiki/testWiki> <https://rdfs.org/sioc/ns#container_of> <https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest> . <https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest> <https://rdfs.org/sioc/ns#topic> <https://localhost:8890/dataspace/test3/wiki/testWiki> . <https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest> <https://atomowl.org/ontologies/atomrdf#source> <https://localhost:8890/dataspace/test3/wiki/testWiki> . <https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest> <https://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://rdfs.org/sioc/types#Comment> . <https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest> <https://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://atomowl.org/ontologies/atomrdf#Entry> . <https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest> <https://www.w3.org/2000/01/rdf-schema#label> 'MyTest' . <https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest> <https://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://atomowl.org/ontologies/atomrdf#Link> . <https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest> <https://rdfs.org/sioc/ns#content> <test>}" -u "test3:1" -H "Content-Type: application/sparql-query" https://localhost:8890/DAV/home/test3/wiki/testWiki/MyTest
3. As result we should have 2 files created:

3.1 In the user DAV folder “DAV/home/test3/wiki/testWiki/” will be created a file “MyTest?” with type “application/sparql-query”.

3.1.1 You can view the content of this file from from the Conductor UI or from the user’s Briefcase UI, path “DAV/home/test3/wiki/testWiki”:

3.1.2 It content will be:
<?xml version="1.0" encoding="utf-8" ?>
<rdf:RDF xmlns:rdf="https://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="https://www.w3.org/2000/01/rdf-schema#">
<rdf:Description rdf:about="https://localhost:8890/dataspace/test3/wiki/testWiki"><ns0pred:entry xmlns:ns0pred="https://atomowl.org/ontologies/atomrdf#" rdf:resource="https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest"/></rdf:Description>
<rdf:Description rdf:about="https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest"><ns0pred:label xmlns:ns0pred="https://www.w3.org/2000/01/rdf-schema#">MyTest</ns0pred:label></rdf:Description>
<rdf:Description rdf:about="https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest"><ns0pred:type xmlns:ns0pred="https://www.w3.org/1999/02/22-rdf-syntax-ns#" rdf:resource="https://atomowl.org/ontologies/atomrdf#Link"/></rdf:Description>
<rdf:Description rdf:about="https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest"><ns0pred:type xmlns:ns0pred="https://www.w3.org/1999/02/22-rdf-syntax-ns#" rdf:resource="https://rdfs.org/sioc/types#Comment"/></rdf:Description>
<rdf:Description rdf:about="https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest"><ns0pred:type xmlns:ns0pred="https://www.w3.org/1999/02/22-rdf-syntax-ns#" rdf:resource="https://atomowl.org/ontologies/atomrdf#Entry"/></rdf:Description>
<rdf:Description rdf:about="https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest"><ns0pred:has_container xmlns:ns0pred="https://rdfs.org/sioc/ns#" rdf:resource="https://localhost:8890/dataspace/test3/wiki/testWiki"/></rdf:Description>
<rdf:Description rdf:about="https://localhost:8890/dataspace/test3/wiki/testWiki"><ns0pred:container_of xmlns:ns0pred="https://rdfs.org/sioc/ns#" rdf:resource="https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest"/></rdf:Description>
<rdf:Description rdf:about="https://localhost:8890/dataspace/test3/wiki/testWiki"><ns0pred:contains xmlns:ns0pred="https://atomowl.org/ontologies/atomrdf#" rdf:resource="https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest"/></rdf:Description>
<rdf:Description rdf:about="https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest"><ns0pred:content xmlns:ns0pred="https://rdfs.org/sioc/ns#">test</ns0pred:content></rdf:Description>
<rdf:Description rdf:about="https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest"><ns0pred:topic xmlns:ns0pred="https://rdfs.org/sioc/ns#" rdf:resource="https://localhost:8890/dataspace/test3/wiki/testWiki"/></rdf:Description>
<rdf:Description rdf:about="https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest"><ns0pred:source xmlns:ns0pred="https://atomowl.org/ontologies/atomrdf#" rdf:resource="https://localhost:8890/dataspace/test3/wiki/testWiki"/></rdf:Description>
</rdf:RDF>
3.2 To the user’s wiki instance will be added a new WikiWord “MyTest” with content the value of the SIOC term attribue “content”:
https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest> <https://rdfs.org/sioc/ns#content> <test>
i.e. the content will be "test".
3.2.1 You can view this file in path “DAV/home/test3/wiki/testWiki/” and its name will be “MyTest.txt”.

3.2.2 Using the ODS wiki UI you can:

3.2.2.1 Go to Index link

3.2.2.2 As result the MyTest article will be shown in the given list. Click on its link.

3.2.2.3 As result the content of the article will be shown.

4. Now lets check what data was inserted in the Quad Store:

4.1 Go to the sparql endpoint, i.e. for ex. to https://localhost:8890/sparql

4.2 Enter for Default Graph URI:
https://localhost:8890/DAV/home/test3/wiki/testWiki/MyTest
4.3 Enter for Query text:
SELECT * WHERE {?s ?p ?o}
4.4 Click the “Run Query” button.4.5 The result will show triples:
s p o
https://localhost:8890/dataspace/test3/wiki/testWiki https://rdfs.org/sioc/ns#container_of https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest
https://localhost:8890/dataspace/test3/wiki/testWiki https://atomowl.org/ontologies/atomrdf#entry https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest
https://localhost:8890/dataspace/test3/wiki/testWiki https://atomowl.org/ontologies/atomrdf#contains https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest
https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest https://www.w3.org/1999/02/22-rdf-syntax-ns#type https://rdfs.org/sioc/types#Comment
https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest https://www.w3.org/1999/02/22-rdf-syntax-ns#type https://atomowl.org/ontologies/atomrdf#Entry
https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest https://www.w3.org/1999/02/22-rdf-syntax-ns#type https://atomowl.org/ontologies/atomrdf#Link
https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest https://www.w3.org/2000/01/rdf-schema#label MyTest
https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest https://rdfs.org/sioc/ns#has_container https://localhost:8890/dataspace/test3/wiki/testWiki
https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest https://rdfs.org/sioc/ns#content test
https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest https://rdfs.org/sioc/ns#topic https://localhost:8890/dataspace/test3/wiki/testWiki
https://localhost:8890/dataspace/test3/wiki/testWiki/MyTest https://atomowl.org/ontologies/atomrdf#source https://localhost:8890/dataspace/test3/wiki/testWiki
[Warning]This document and its examples are correct at the time of writing (6th February 2008)[/Warning]

Human (Document-Web) Links:

  • Semantic Web Frequently Asked Questions (by the Semantic Web Education and Outreach Interest Group)
  • OpenLink Software Website
  • OpenLink Virtuoso Website
  • RDF Specification
  • HTTP Method Definition Documentation
  • SPARQL Specification
  • Virtuoso PL
  • SIMILE
  • RDF NET API Submission
  • WebDAV Working/Interest Group
  • Virtuoso Multipart Form PUT documentation
  • The SPARQL/Update (aka SPARUL) Specification by Andy Seaborne and Geetha Manjunath of HP Labs
  • Virtuoso Documentation for managing RDF Storage using SPARUL
  • OpenLink Data Spaces
  • SIOC Project Website

Folksonomy Stuff:

Technorati Tags: openlink, virtuoso, data, input, semanticweb, sparql

Leave a Reply