I’m writing this up as an example of one way — a relatively easy way — to publish Linked Data off the back of some existing API. I hope that some other libraries, archives, and museums with their own API will adopt this approach and start publishing their data in a standard Linked Data style, so it can be linked up with the wider web of data.

Two ways to skin a cat

There are two basic ways you can take an existing API and turn it into a Linked Data service.

One way is the “metadata aggregator” approach. In this approach, an “aggregator” periodically downloads (or “harvests”) the data from the API, in bulk, converts it all into RDF, and stores it in a triple-store. Then another application — a Linked Data publishing service — can read that aggregated data from the triple store using a SPARQL query and expose it in Linked Data form. The tricky part here is that you have to create and maintain your own copy (a cache) of all the data which the API provides. You run the risk that if the source data changes, then your cache is out of date. You need to schedule regular harvests to be sure that the copy you have is as up to date as you need it to be. You have to hope that the API can tell you which particular records have changed or been deleted, otherwise, you may have to download every piece of data just to be sure.

But this blog post is about another way which is much simpler: the “proxy” approach. A Linked Data proxy is a web application that receives a request for Linked Data, and in order to satisfy that request, makes one or more requests of its own to the API, processing the results it receives, and formatting them as Linked Data, which it then returns. The advantage of this approach is that every response to a request for Linked Data is freshly prepared. There is no need to maintain a cache of the data. There is no need for harvesting or scheduling. It’s simply a translator that sits in front of the API and translates what it says into Linked Data.

This is an approach I’ve been meaning to try out for a fair while, and in fact I gave a very brief presentation on the proxy idea at the recent LODLAM Summit in Sydney. All I needed was a good API to try it out with.

Museum Victoria Collection API

The Museum Victoria Collection API was announced on Twitter by Ely Wallis on August 25th:

The cat's out of the bag – @museumvictoria has a new collections online. We hope you like it! http://t.co/c0oNTWDk5t

— Ely Wallis (@elyw) August 25, 2015

Well as it happened I did like it, so I got in touch. Since it’s so new, the API’s documentation is a bit sparse, but I did get some helpful advice from the author of the API, Museum Victoria’s own Michael Mason, including details of how to perform searches, and useful hints about the data structures which the API provides.

In a nutshell, the Museum Victoria API provides access to data about four different sorts of things:

• Items (artefacts in the Museum’s collections),
• Specimens (biological specimens),
• Species (which the specimens belong to), and
• Articles (which document the other things)

There’s also a search API with which you can search within all of those categories.

Armed with this knowledge, I used my trusty XProc-Z proxy software to build a Linked Data proxy to that API.

Linked Data

Linked Data is a technique for publishing information on the web in a common, machine-understandable way.

The central principle of Linked Data is that all items of interest are identified with an HTTP URI (“Uniform Resource Identifier”). And “Resource” here doesn’t just mean web pages or other electronic resources; anything at all can be a “Resource”: people, physical objects, species of animal, days of the week … anything. If you take one of these URIs and put it into your browser, it will deliver you up some information which relates to the thing identified by the URI.

Because of course you can’t download a person or a species of animal, there is a special trick to this: if you send a request for a URI which identifies one of these “non-information resources”, such as a person, the server can’t simply respond by sending you an information resource (after all, you asked for a person, not a document). Instead it responds by saying “see also” and referring you to a different URL. This is basically saying “since you can’t download the resource you asked for (because it’s not an information resource), here is the URL of an information resource which is relevant to your request”. Then when your browser makes a request from that second URL, it receives an information resource. This is why, when browsing Linked Data, you will sometimes see the URI in your browser’s address bar change: first it makes a request for one URI and then is automatically redirected to another.

That information also needs to be encoded as RDF (“Resource Description Framework”). The RDF document you receive from a Linked Data server consists of a set of statements (called “triples”) about various things,including the original “resource” which your original URI identified, but usually also other things as well. Those statements assign various properties to the resources, and also link them to other resources. Since those other resources are also identified by URIs, you can follow those links and retrieve information about those related resources, and resources that are related to them, and so on.

Linked Data URIs as proxies for Museum Victoria identifiers

So one of the main tasks of the Linked Data proxy is to take any identifiers which it retrieves from the Museum Victoria API, and convert them into full HTTP URIs. That’s pretty easy; it’s just a matter of adding a prefix like “http://example/something/something/”. When the proxy receives a request for one of those URIs, it has to be able to turn it back into the form that Museum Victoria’s API uses. That basically involves trimming the prefix back off. Because many of the things identified in the Museum’s API are not information resources (many are physical objects), the proxy makes up two different URIs, one to denote the thing itself, and one to refer to the information about the thing.

The conceptual model (“ontology”)

The job of the proxy is to publish the data in a standard Linked Data vocabulary. There was an obvious choice here; the well-known museum ontology (and ISO standard) with the endearing name “CIDOC-CRM”. This is the Conceptual Reference Model produced by the International Committee for Documentation (CIDOC) of the International Council of Museums. This abstract model is published as an OWL ontology (a form that can be directly used in a Linked Data system) by a joint working group of computer scientists and museologists in Germany.

This Conceptual Reference Model defines terms for things such as physical objects, names, types, documents, and images, and also for relationships such as “being documented in”, or “having a type”, or “being an image of”. The proxy’s job is to translate the terminology used in Museum Victoria’s API into the terms defined in the CIDOC-CRM. Unsurprisingly, much of that translation is pretty easy, because there are long-standing principles in the museum world about how to organise collection information, and both the Museum Victoria API and the CIDOC-CRM are aligned to those principles.

As it happened I already knew the CIDOC-CRM model pretty well, which was one reason why a museum API was an attractive subject for this exercise.

Progress and prospects

At this stage I haven’t yet translated all the information which the Museum’s API provides; most of the details are still simply ignored. But already the translation does include titles and types, as well as descriptions, and many of the important relationships between resources (it wouldn’t be Linked Data without links!). I still want to flesh out the translation some more, to include more of the detailed information which the Museum’s API makes available.

This exercise was a test of my XProc-Z software, and of the general approach of using a proxy to publish Linked Data. Although the result is not yet a complete representation of the Museum’s API, I think it has at least proved the practicality of the approach.

At present my Linked Data service produces RDF in XML format only. There are many other ways that the RDF can be expressed, such as e.g. JSON-LD, and there are even ways to embed the RDF in HTML, which makes it easier for a human to read. But I’ve left that part of the project for now; it’s a very distinct part that will plug in quite easily, and in the meantime there are other pieces of software available that can do that part of the job.

See the demo

The proxy software itself is running here on my website conaltuohy.com, but for ease of viewing it’s more convenient to access it through another proxy which converts the Linked Data into an HTML view.

Here is an HTML view of a Linked Data resource which is a timber cabinet for storing computer software for an ancient computer: http://conaltuohy.com/xproc-z/museum-victoria/resource/items/1411018. Here is the same Linked Data resource description as raw RDF/XML: http://conaltuohy.com/xproc-z/museum-victoria/data/items/1411018 — note how, if you follow this link, the URL in your browser’s address bar changes as the Linked Data server redirects you from the identifier for the cabinet itself, to an identifier for a set of data about the cabinet.

The code

The source code for the proxy is available in the XProc-Z GitHub repository: I’ve packaged the Museum Victoria Linked Data service as one of XProc-Z’s example apps. The code is contained in two files:

• museum-victoria.xpl which is a pipeline written in the XProc language, which deals with receiving and sending HTTP messages, and converting JSON into XML, and
• museum-victoria-json-to-rdf.xsl, which is a stylesheet written in the XSLT language, which performs the translation between Museum Victoria’s vocabulary and the CIDOC-CRM vocabulary.

One especially nice feature of Zotero is that you can use it to collaborate with a group of people on a shared library of data which is stored in the cloud and synchronized to the devices of the group members.

Getting data out of Zotero’s web API

If you then want to get the data out of Zotero to do other things with it, you have a number of options. Zotero supports many standard export formats, but the problem I found was that none of those export formats exposed the full richness of your data. Some formats don’t include the “Tags” that you can apply to items in your library; some don’t reflect the hierarchical structure of ‘Collections’ in your library; and so on. It seems the only way to get the full story is to use Zotero’s web API.

Like any web API, this API is a great thing; it makes it possible to use Zotero as a platform for building all kinds of other web applications and systems. The nice thing about a web API is that it’s open to being accessed by any other kind of software. You don’t need to write your software in Javascript or in PHP (the Zotero data server is written in PHP). To access a web API you only need to be able to make HTTP requests, so you’re not tied to any particular platform.

Zotero’s web API is pretty good as web APIs go, though it does have a weakness which is common to many “web APIs”. The weakness is that it’s not obvious how to interpret the data which Zotero provides, and this is a practical barrier to the use of the API. It certainly was for me.

REST

Zotero’s API documentation makes mention of the buzzword “REST”, which is an acronym for “Representational State Transfer”. REST is the name for a style of network communications, defined by a set of design principles or guidelines. A network protocol or web API that conforms to those guidelines is said to be “RESTful”. However, in practice a great many “RESTful” web APIs fail to conform to one or more of the principles, commonly the principle of the “Uniform Interface”, one corollary of which is that the packets of information sent back and forth must be “self-descriptive”.

Self-descriptive messages

To get it right, a RESTful web API needs to provide self-descriptive information; the information it sends you must describe itself sufficiently that you could work out what to do with it. Often the publishers of APIs rely on providing documentation of the different data formats their API provides, and they expect you to have found and read that documentation before you use their API, and to already know what kind of response you will get from the various different parts of their API. But if an API relies on you already knowing what kind of information it provides, then it’s not RESTful. This unfortunately is the case with Zotero.

So how should an API publish “self-descriptive” data?

The HTTP Content-Type header

The main mechanism a web server uses to publish self-descriptive data is to include along with the data a Content-Type header which explicitly declares the format of that information using a code called an “Internet Media Type”. There are a zillion of these Internet Media Types, including image/jpeg and image/png for images, text/html for web pages, application/xml for generic XML documents, or application/json for generic JSON data objects. Of these examples, the last two stand out as different because they are not very specific. What does an XML document mean? What does a JSON object mean? They could mean anything at all, because XML and JSON are generic data formats which can be used to transmit all different kinds of information. It’s possible to be more specific about what kind of XML or JSON you are producing, by saying for instance application/rdf+xml (RDF data encoded in XML) or application/ld+json (Linked Data encoded in JSON). But if you only give a more generic Content-Type, then a client will need to look inside the data package itself to determine what it means.

If Zotero were to publish its data as application/zotero+json, that would be an improvement. It would mean that Zotero data in this format could be exchanged around in other systems, and still be understandable. As it stands, Zotero’s application/json data can only reliably be understood if you have just read it from Zotero.

Here’s an example of the JSON data you can read from Zotero’s API: https://api.zotero.org/groups/300568/items?v=3&format=json

Namespaces

One of the nice features of XML is the concept of “namespaces”. These are distinct vocabularies with globally unique names, which allow you to unambiguously identify what kind of XML data you are looking at. If a piece of software can recognise the namespace or namespaces that a document uses, then it’s in a position to understand what it means, and to process it usefully. Otherwise a human is going to have to look at the XML and try to make some sense of it. JSON doesn’t have an equivalent to XML Namespaces (although JSON-LD does), so that means that information served up as application/json can’t be considered very self-descriptive.

Another interesting point about XML Namespaces is that each of these vocabularies is uniquely identified by a URI; that is, the URI is the name of the vocabulary. This has the nice feature that you can open an XML file, find the namespace URI, plug that namespace URI into your browser, and magically be presented with some useful information about that vocabulary. In other words, any data in this format will always contain a hyperlink to its own documentation (called a “Namespace Document“).

If Zotero were to publish its data in XML, and use a “Zotero” namespace to label all the terms in its vocabulary, then that would be another improvement. Any XML documents of that type could be downloaded from Zotero and stored in any other kind of system, and because they would contain that identifier, they would still be identifiably Zotero-flavoured, even after they had long been detached from Zotero itself.

Formalised data formats

Although it is a problem that Zotero’s JSON data format doesn’t have its own formal name by which it can identify itself, the more critical issue for me in attempting to understand the Zotero data was that the data format exposed by the API is barely documented at all.

If you read the JSON data, you will see names such as publicationTitle, itemType, dateAdded, etc, and you can have a guess at what they mean, but it shouldn’t be necessary to guess what they mean, or to understand the relationships between them. I had to spend hours analysing the dataset I had extracted from the web API, before I could seriously attempt to convert it to some other form. There is some documentation scattered about here and there, but no authoritative description of the data format. Compare this to the situation with the more formalised formats which Zotero can export: TEI, RIS, MODS, etc, which have formal specifications defining all the terms in their vocabulary.

Is this something that Zotero could do? It’s hard to say; it would require some technical changes to the Zotero data server code, but probably more signficantly it would involve a change in collective mindset by the developers involved; to see Zotero’s data model as an abstraction independent of Zotero’s data server application; a genuine public language for communicating between arbitrary bibliographic systems, not merely a kind of window into the internal workings of a particular software system.

This is a common situation in web applications which offer an API: the application developers are focused intellectually on the application itself; its own internal workings and the functionality it provides, and they naturally tend to see the API as merely an aspect of that system. The idea that the data format of the API might have a life independent of their software, or that it might even outlive their software altogether, is a stretch. But if the data which their system works with is important, then it is surely important enough to accord some formal status: to give it a name; an Internet Media Type, even a namespace URI, to constrain it with a schema, and to explain it with formal documentation.

Next steps

As usual, the code I’m writing is published on GitHub; in the first instance this XProc pipeline to convert a Zotero library to EAD. But this was really a first stab at the problem; where I’d like to go is to try to formalise and specify the Zotero data format itself; to give it an XML encoding with a formal definition, and then to build other systems, such as Linked Data systems, on top of that formalised format.