Open Data Problems

If you or your organisation have any experiences in providing Open Data (or if you’re thinking about it), then you will have come across the 5-Star scheme for Open Data (for the original 5-star data proposal see here, a nice write-up is here) – the more stars, the more useful and connected it is. Publishing 1-star data (just put it online) to 3-star data (use a non-proprietary format) is relatively simple and straightforward. However, when it comes to 4- and 5-star data, things can become quickly become a bit more complex: If you’re new to it, the world of Linked Data and URIs can seem daunting and difficult to understand. However, even with some experience, there can often be issues of figuring out and mastering the right approach, such as deciding how to model your data, how to structure your URIs, which other data to link to and how, which vocabularies to use (or maybe you need to develop your own), etc. Other issues that can arise are more technical in nature, such as deciding which hosting platform to choose, which software to use for modelling, conversion and data maintenance, whether to set up your own infrastructure or use an external service, etc.

What are your Experiences?

If you or your organisation have encountered any of these or any other problems in the process of publishing data, we’re interested in hearing from you! We would love to learn about your data publishing experiences (both the good and bad), and the reasons you embarked on doing this. Also, Talis are currently offering a 1/2 day review of data that has been published. If you would like me (or one of my colleagues) to review the data you have published, feel free to contact me at knud.moeller@talis.com.

Labyrinth picture by rosmary on Flickr, licensed under CC BY 2.0.

Talis unveiled their software thought piece encouraging cities to engage with communities to explore which data would be of most interest.

Leigh Dodds’ presentation is now available to watch:

Photo credit: Tim Hodson

At the Open Data Cities Conference in Brighton, Talis unveiled their demonstrator app (link below) which shows how a city might begin to engage with it’s citizens and promote digital economy innovation.

The demo is designed to highlight the ways in which a city and its citizens might be brought together in an information marketplace. The demo is designed to trigger questions around how cities might use an interactive information marketplace to measure social impact. The demo is the software equivalent of a thought piece, allowing us to talk about the things that might change the way people in your city think about engaging with each other in social enterprise.

Talis have been exploring ways in which a data marketplace might add value to individual datasets, and have built Kasabi which allows anyone to publish their data easily, and then harness the power of multiple data access channels.

Key demonstrator themes:

• citizens can request data about their local area
• citizens can use data, from the city and local businesses, to build apps
• the city might fund the building of apps that are in demand
• citizens can share apps they have built
• business can use the marketplace, to publish the data that will power other applications.
• cities can easily publish data about anything
• citizens able to add data to existing datasets
• developers have several tools for accessing indexed and structured data
• all data added to the site is indexed as it arrives and becomes available to applications within a very short time
• the information marketplace is a data hub providing a revenue share opportunity

Behind the lightweight demonstrator sits a technology stack that provides data hosting and integration. The simple datasets used as examples in the demo can be explored by both developers who understand working with data, and citizens with no programming background.

I could throw the names of some technologies at you, such as graph databases, geo indexes, full-text indexes and application programming interfaces using a variety of protocols, but it is the self service nature of kasabi combined with the interactive and social aspects of our demonstrator that we think will make the difference to your city.

As a city we think you probably know your citizens quite well, however I am sure that there are ways that they can surprise you. Maybe it is a loosely organised not for profit company that sets itself the mission of providing the best quality data about where to park in your city. Maybe they take data that you provide about where the parking spaces are and how often they are used and combine it with a calendar of city wide events sourced from several other data providers. Maybe they built an indispensable app that helps people to choose the best parking site in the city. Maybe it even integrates with an existing drive-sharing scheme to provide parking booking services for commuters and tourists alike.

An idea like that is only possible if the people wanting to build a data driven application have easy access to data.

Of course there is no reason why that access should be free. A car parking app might charge a small fee for the provision of the service, and that fee might be shared with the data providers and the city playing host to the data in a marketplace. Everyone gets to have a share in the success of the idea.

For cities that might have a perceived poor parking experience, an app like this might improve the imgae of the city and reposition it as an easy place to find parking. It might even change people’s parking behaviour to the better, a social impact that becomes easier to measure.

At Talis, we are keen to work with you to explore how your city data and your citizen’s data might be brought together in a marketplace that allows new business to start and thrive.

See the demo >>

Talk to us

We found three applications which excited us, and offered them financial backing to get off the ground. For the last few months, the teams have been building up and testing out their applications, and we are watching as they get them going!

I’ll post an original roundup of the app projects below, and there will be follow-up posts here and on the Kasabi blog.

If you work on a similar project, or would like to share your ideas with us, please get in touch!

Exploring Botanical Gardens from a Smart Phone

StrongSteam: Ian Ozsvald and Kyran Dale

The StrongSteam team are working on an iPhone app that opens up new levels of exploration for visitors to botanical gardens. The app will let people access tons of information about the plants they find by taking a photo of the label. The app uses advanced character recognition to read the Latin name from descriptive labels, and pulls in data from a variety of sources to tell the user far more about the plant than could be available on signs.

They’re using the StrongSteam datamining API for matching plant labels and IDs, then using datasets in Kasabi (GeoSpecies, DBPedia and BBC Wildlife for example) to extract detailed information about plant species. The user will then be given facts, figures and other pieces of information, letting them learn far more about the plants they find interesting.

FixMyStreet (Latest from MySociety Here)

My Society: Paul Lenz and Myfanwy Nixon

Through the popular app, FixMyStreet, My Society has been giving people the ability to report damaged infrastructure to their local authorities for a few years. Using smartphones, people have been highlighting things like potholes and broken streetlights across the UK since 2008. The app is now going through a complete overhaul, upgrading to a more sophisticated, HTML5-based service. The new FixMyStreet is a more powerful, responsively designed mobile-web version of the older native apps, and uses Kasabi to store a continuously-updated list of new problem reports. The new dataset includes information about councils, kinds of damage, timestamp and status of repairs along with detailed lat/long locations.

John Peel Time Machine

Storm: Dave Kelly, Mike Ellis, and Paul Leader

Developers from Storm are putting together a time machine travelling back through some of the greatest musical events of the 20th century under the watchful eyes of the legendary BBC radio DJ, the late John Peel. Building on the dataset of John Peel Sessions, the web app will guide users’ journeys on their search for artists who appeared on the live recordings of John Peel’s long-running show.

The Time Machine will work on a timeline, giving a high-level view of the Peel sessions by year, and highlight some of the relationships amongst musical artists. Where it can, it will link to recordings of the live sessions, and provide biographical information about the artists. The time machine will also provide information about the albums and tracks featured, and point users towards playlists of sessions, which they can purchase or listen to via the likes of iTunes or Spotify.

This means we’re part of the network of suppliers created to make finding cloud-based for public work a lot easier. Francis Maude, the Minister for the Cabinet Office summed up the setup:

“Simply stated, purchasing services from CloudStore will be quicker, easier, cheaper and more transparent for the public sector and suppliers alike.

GCloud is a list of selected suppliers, and has been built to work like a shop front (a “CloudStore”) for government group to search for solutions to problems or offering ideas that enhance their public service. Suppliers from the GCloud store would still work with transparent tendering, but the processes have been sped up to make it quicker to find a provider. It’s also aimed at helping various governmental bodies to get the best out of small/medium businesses (like Talis).

For Talis, this means it is now easier for us to work on exciting data projects with public-sector ‘sets for local and national government. It should also be quicker for any projects that would benefit from us hosting data to get off the ground—and into the cloud (sorry, had to.) So, for your public-sector project, it is a simpler process to work with us—email Alison to learn more.

If you are curious, you can read more about GCloud on the Civil Service Site.

Last weekend, the National Archives held a Hackathon in the reading room at Kew. Around 40 developers and interested people took data from the National Archives and played with it.  There were new mobile interfaces for the NRA discovery API; collections of tweets mined for the data and PDFs they contained; stats on historical participation in the olympics pulled from the archives and shown on interactive maps. In all it was a fun weekend with lots of smart people in the room and very quiet but rapid typing on keyboards to get something finished by the 4pm Sunday deadline.

Prizes were:

• 1st – Jonathan Tweed and Kai En Ong (ably assisted by Michael Smethurst, Faith Mowbray and Paul Rissen). A hack that pulls out data surrounding people & places in documents tweeted by @ukwarcabinet (and which – for a hack – is beautifully presented!).
• 2nd – Jamie Mahoney - Debtors & creditors dataset hack maps the most popular lenders & shows who’s borrowing from where – Show me the money.
• 2nd – Tim Hodson – A hack showing who wrote to whom in the middle ages.
• 3rd – Crystal and Steven Hirschorn – A hack showing participation in the Olympics on an interactive map.

You can read more about these entries on the National Archives blog.

I hope you’ll forgive my showing off of my joint second prize winning contribution to the pizza and jelly baby fuelled hack fest.

I took a suggestion from Paul Risson as a personal challenge, and started puling the data that I wanted into a new CSV file.  I then converted that CSV file to a rudimentary RDF based model of the letters and people that the data described.  I now had a graph dataset which captured – in the way only a graph can – the network of relationships between people who are corresponding. It was then a case of finding a suitable javascript library to render my graph as a visual and to allow people to find out about who wrote to whom without cluttering up the graph diagram.

Our recent webinars sought to dispel any such myths, and provide you with a project outline that would work for your teams.  We know it works because this is how we run our projects to help organisations manage the transition to publishing their data in a new way.

Talis have the tools and experience to help you get up and running in months rather than years. Now you can watch this recording of the webinar to find out more.

Tim Berners-Lee took a pragmatic approach by simplifying the problem into bite size chunks with an inbuilt mark of quality. The famous 5 Stars are not a new thing, they have been around since 2010, and we have asked if your data is 5 star before.

The 5 stars of open data publishing are clear simple steps that you can take to get your data published openly. Talis have helped the likes of the Ordnance Survey, Office of National Statistics, British Library, Data.Gov.Uk and the department for Business Innovation and skills to publish their data openly.

But how did they do it?

Our experience, gained by working with the likes of the Ordnance Survey, has shown us that providing a hosted platform for publishing data means that organisations can concentrate on data quality and utility without having to find funding for infrastructure and maintenance of specialist hardware. As John Goodwin, Research Scientist at the Ordnance Survey said:

“We decided to let Talis take care of the hosting and serving, so all we had to do was worry about making the data available.”

By pushing the infrastructure costs outside the organisation, data publishers can get on with making their local data link to a wider network of global data and gain their 5 star rating.

If you need help making open data achievable in your organisation, talk to us.

Their discussions ranges from Åke‘s background in Swedish libraries, including the creation of a new library, through to his engagement with several digital initiatives, before exploring the themes of his session.

MARC is Dead! – RDA made irrelevant! – cries that can be heard rattling around the bibliographic blogo-twittersphere.   My opinion is that this is an inevitable move based upon serious consideration, and has been building on several initiatives that have been brewing for many months.

Bold though – very bold.  I am sure that there are many in the library community, who have invested much of their careers in MARC and its slightly more hip cousin RDA, who are now suffering from vertigo as they feel the floor being pulled from beneath their feet.

The Working Group of the Future of Bibliographic Control, as it examined technology for the future, wrote that the Library community’s data carrier, MARC, is “based on forty-year-old techniques for data management and is out of step with programming styles of today.”

Many of the libraries taking part in the test [of RDA] indicated that they had little confidence RDA changes would yield significant benefits…

And on a more positive note:

The Library of Congress (LC) and its MARC partners are interested in a deliberate change that allows the community to move into the future with a more robust, open, and extensible carrier for our rich bibliographic data….
….The new bibliographic framework project will be focused on the Web environment, Linked Data principles and mechanisms, and the Resource Description Framework (RDF) as a basic data model.

There is still a bit of confusion there between a data carrier and a framework for describing resources.  Linked Data is about linking descriptions of things, not necessarily transporting silos of data from place to place.  But maybe I quibble a little too much at this early stage.

So now what:

The Library of Congress will be developing a grant application over the next few months to support this initiative.  The two-year grant will provide funding for the Library of Congress to organize consultative groups (national and international) and to support development and prototyping activities.  Some of the supported activities will be those described above:  developing models and scenarios for interaction within the information community, assembling and reviewing ontologies currently used or under development, developing domain ontologies for the description of resources and related data in scope, organizing prototypes and reference implementations.

I know that this is the way that LoC and the library community do things, but do I hope that this doesn’t mean that they will disappear into an insular huddle for a couple of years to re-emerge with something that is almost right yet missing some of the evolution that is going on around them over that period.

As per other recent announcements, such as the vote to openly share European Libraries’ data, the report from the W3C’s Library Linked Data Incubator Group, and now the report from the Stanford Linked Data Workshop.  I welcome these developments. However I warn those involved that these are great opportunities [to enable the valuable resources catalogued and curated by libraries over decades to become foundational assets of the future web] that can be easily squandered by not applying the open thinking that characterise successes in the web of data.

One very relevant example of the success of applying open thinking and approach to the bibliographic word using Linked Data is the open publishing of the British National Bibliography (BnB).  Readers of this blog will know that we at Talis have worked closely with the team at the BL in their ground breaking work.   The data model they produced is an example of one of those things that may induce that feeling of vertigo that I mentioned.  It doesn’t look much like a MARC record!  I can assure the sceptical that although it may be very different to what you are used to, it is easy to get your head around.  (Drop us a line if you want some guidance).

As we host the BnB Linked Data for the BL, I can testify to the success of this work – only launched in mid July.  It’s use is growing rapidly, receiving just short of 2 million hits in the last month alone.

With the British Library, along with the National Libraries of Canada and Germany, being quoted as partners with the LoC in this initiative, plus their work being referenced as an exemplar in the other reports I mention, I hold out a great hope that things are headed in the right direction.

As comments to some of my previous posts attest, there is concern from some in the community of domain experts, that this RDF stuff is too simple and light-weight and will not enable them capture the rich detail that they need.  They are missing a couple of points.  Firstly, it is this simplicity that will help non-domain experts to understand, reference and link to their rich resources.  Secondly, RDF is more than capable of describing the rich detail that they require – using several emerging ontologies including the RDA ontology, FRBR, etc.  Finally and most importantly, it is not a binary choice between widely comprehended simplicity and and domain specific detailed description.   The RDF for a resource can, and probably should, contain both.

So Library of Congress, I welcome your announcement and offer a friendly reminder that you not only need to draw expertise from the forward thinking library community, but also from the wider Linked Data world.  I am sure your partners from the British Library will reinforce this message.