Category: Government

The Open Data Cities conference in Brighton was well attended  by around 150 people interested in how cities can grow their use of open data.  The thought provoking speakers touched on subjects ranging from why cities should invest in making their data open, to how they can make that achievable.

Talis unveiled their software thought piece encouraging cities to engage with communities to explore which data would be of most interest.

Leigh Dodds’ presentation is now available to watch:

Photo credit: Tim Hodson

At the Open Data Cities Conference in Brighton, Talis unveiled their demonstrator app (link below) which shows how a city might begin to engage with it’s citizens and promote digital economy innovation.

The demo is designed to highlight the ways in which a city and its citizens might be brought together in an information marketplace. The demo is designed to trigger questions around how cities might use an interactive information marketplace to measure social impact. The demo is the software equivalent of a thought piece, allowing us to talk about the things that might change the way people in your city think about engaging with each other in social enterprise.

Talis have been exploring ways in which a data marketplace might add value to individual datasets, and have built Kasabi which allows anyone to publish their data easily, and then harness the power of multiple data access channels.

Key demonstrator themes:

• citizens can request data about their local area
• citizens can use data, from the city and local businesses, to build apps
• the city might fund the building of apps that are in demand
• citizens can share apps they have built
• business can use the marketplace, to publish the data that will power other applications.
• cities can easily publish data about anything
• citizens able to add data to existing datasets
• developers have several tools for accessing indexed and structured data
• all data added to the site is indexed as it arrives and becomes available to applications within a very short time
• the information marketplace is a data hub providing a revenue share opportunity

Behind the lightweight demonstrator sits a technology stack that provides data hosting and integration. The simple datasets used as examples in the demo can be explored by both developers who understand working with data, and citizens with no programming background.

I could throw the names of some technologies at you, such as graph databases, geo indexes, full-text indexes and application programming interfaces using a variety of protocols, but it is the self service nature of kasabi combined with the interactive and social aspects of our demonstrator that we think will make the difference to your city.

As a city we think you probably know your citizens quite well, however I am sure that there are ways that they can surprise you. Maybe it is a loosely organised not for profit company that sets itself the mission of providing the best quality data about where to park in your city. Maybe they take data that you provide about where the parking spaces are and how often they are used and combine it with a calendar of city wide events sourced from several other data providers. Maybe they built an indispensable app that helps people to choose the best parking site in the city. Maybe it even integrates with an existing drive-sharing scheme to provide parking booking services for commuters and tourists alike.

An idea like that is only possible if the people wanting to build a data driven application have easy access to data.

Of course there is no reason why that access should be free. A car parking app might charge a small fee for the provision of the service, and that fee might be shared with the data providers and the city playing host to the data in a marketplace. Everyone gets to have a share in the success of the idea.

For cities that might have a perceived poor parking experience, an app like this might improve the imgae of the city and reposition it as an easy place to find parking. It might even change people’s parking behaviour to the better, a social impact that becomes easier to measure.

At Talis, we are keen to work with you to explore how your city data and your citizen’s data might be brought together in a marketplace that allows new business to start and thrive.

See the demo >>

Talk to us

Following on from work with UK governmental agencies (such as the BIS Research Funding Explorer and Ordnance Survey), Talis has joined the UK Government’s GCloud supplier community. We have been awarded an agreement within the framework to provide the UK government with Software as a Service products and services.

This means we’re part of the network of suppliers created to make finding cloud-based for public work a lot easier. Francis Maude, the Minister for the Cabinet Office summed up the setup:

“Simply stated, purchasing services from CloudStore will be quicker, easier, cheaper and more transparent for the public sector and suppliers alike.

GCloud is a list of selected suppliers, and has been built to work like a shop front (a “CloudStore”) for government group to search for solutions to problems or offering ideas that enhance their public service. Suppliers from the GCloud store would still work with transparent tendering, but the processes have been sped up to make it quicker to find a provider. It’s also aimed at helping various governmental bodies to get the best out of small/medium businesses (like Talis).

For Talis, this means it is now easier for us to work on exciting data projects with public-sector ‘sets for local and national government. It should also be quicker for any projects that would benefit from us hosting data to get off the ground—and into the cloud (sorry, had to.) So, for your public-sector project, it is a simpler process to work with us—email Alison to learn more.

If you are curious, you can read more about GCloud on the Civil Service Site.

Government organisations have a remit for publishing some of their core data as open data.  This remit sometimes seems too difficult to achieve.

Tim Berners-Lee took a pragmatic approach by simplifying the problem into bite size chunks with an inbuilt mark of quality. The famous 5 Stars are not a new thing, they have been around since 2010, and we have asked if your data is 5 star before.

The 5 stars of open data publishing are clear simple steps that you can take to get your data published openly. Talis have helped the likes of the Ordnance Survey, Office of National Statistics, British Library, Data.Gov.Uk and the department for Business Innovation and skills to publish their data openly.

But how did they do it?

Our experience, gained by working with the likes of the Ordnance Survey, has shown us that providing a hosted platform for publishing data means that organisations can concentrate on data quality and utility without having to find funding for infrastructure and maintenance of specialist hardware. As John Goodwin, Research Scientist at the Ordnance Survey said:

“We decided to let Talis take care of the hosting and serving, so all we had to do was worry about making the data available.”

By pushing the infrastructure costs outside the organisation, data publishers can get on with making their local data link to a wider network of global data and gain their 5 star rating.

If you need help making open data achievable in your organisation, talk to us.

In August of 2011 the Cabinet Office and Department for Business, Innovation and Skills issued two public consultation papers, one entitled Making Open Data Real: A Public Consultation and the second entitled A Consultation on Data Policy for a Public Data Corporation.

Talis has responded to both and we wanted to share our responses with you and welcome discussion on the comments here.

Some folks have been more concerned about the second consultation, that dealing with the creation of a Public Data Corporation or PDC.

We believe that the creation of a PDC has the potential to significantly simplify access to government data and make it possible for many more individuals and companies to make use of it. In that, however, there is risk. By making the PDC and its parts accountable for establishing “sustainable business models” we risk continuing the status quo in which licensing fees, restrictive licenses and lengthy processes make it impossible to innovate on this data.

We hope that these consultations will continue a discussion about the potential benefits of opening up government data and encourage you to comment below and to link to your own responses if you responded to the consultations also.

We’ve also been contributing to a response from the Linked Data community. I’ll update with a link to that once it’s published.

Here are the Talis Group responses in full:

Talis Group Response to Making Open Data Real A Public Consultation (PDF)

Talis Group Response to A Consultation on Data Policy for a Public Data Corporation (PDF)

A guest post by Lawrence Serewicz, Principal Information Management Officer, Durham County Council

I came across the following reference to time within the retail sector and it made me consider how my world of local government, or any business for that matter, thinks about time.

An old saying in the retail industry is that: ‘If information is available monthly, then decisions taken will take 6 months to have an effect. If it is available weekly, then decisions take a month to influence outcomes; if daily, it takes a week; and if hourly, the decisions can have an impact the next day’ (p.13)

(Source : Valuing Information as an Asset http://www.sas.com/reg/gen/uk/valuing-information )

How often do we collect data? In many organisations, there are quarterly returns, but is that enough for today’s services? In some cases, councils collect real-time data, but are their reporting systems ready for it? For example, management or cabinet committee, meetings may be once a month, but is that enough to have a strategic view of what is happening within an organisation?

At one level, the timeframe for Council Members is different because their work is strategic, they are trying to shape the organisation’s future and where it will be over the long term and not determining if the recent refuse collection achieved 99% or 98% effectiveness. Even if we discount the member’s need to have real time data (at least a strategic level) and focus on the manager/officer’s role, we still see the tyranny to time.

How often do we see, use, or for that matter, analyse, real-time data? Do our performance management systems display a disconnect between the timescale within which they are collected and reported? We may have refuse bin collection rates measured every day, but if our performance reporting within the organisation is quarterly, how well does that serve the organisation? At the same time, is that performance information available to the services, such as customer service desks?

In this example, if the real-time performance is being reported to the customer service desk, they can see that the bin collection rate on a snowy day (for example) is lagging in some areas, but is still robust in other areas. Thus, a call from an area with good collection (say 99%) is going to be a different issue than a missed collection in an area with an 80% collection rate because of the snow conditions. Yet, how many performance management systems or performance information systems are designed to capture and analyse real time data. Even weekly data, can be considered real time data depending on the service, so it raises the point at the start. If the information is only available quarterly, what is the impact rate? If you collect each quarter, is the final impact seen yearly or in two years? If that is the timescale, is it going to be effective?

What does this have to do with open data? If data is being collected and made available to customers and the public, are they getting real time data or is there an organisation influenced lag effect on the data? One of the main themes within the UK  government’s open data collection consultation (http://data.gov.uk/opendataconsultation) as well as its overall transparency agenda is to open service performance information to the public.

The service performance information will inform their choice about services but also to hold it to account. Yet, if there is a lag effect, between when service information is collected and published can the public hold a local authority to account effectively? How much and when the information is released can have a large influence on whether an organisation is accountable. If it only has to report once a year, how much accountability can be achieved? If a change in performance is required, how will it be demonstrated in such a long reporting cycle?

If, however, real time data is released, will that have a destabilizing effect on the political process? If the political process is relying on quarterly performance reporting and the public are getting the information in real time, how will elected members be able to respond? Moreover, if members, as residents, are consuming the information in real time as well, what is the role of a quarterly performance reporting system?  To be sure there will be different reports for different issues, but the underlying question is how to make open data respond to real time demand.  Do I need to know the car park was full last week if I am trying to get parked now?

The issue of time is also about how and where information is released. If an organisation releases its performance statistics in a paper report, and not as a spreadsheet, can external scrutiny be achieved?  In that sense, the format for publication will show the timescales. Such reporting has an immediate and direct effect on the ability of the public, and members, to hold the organisation to account.

At the same time, there is the question of whether real time reporting fits your strategy. If one company is working on a the day to day reporting and another is taking a ten year strategy to grow they will have different understandings of time.  Moreover, their reporting mechanisms will be different.  Yet, can the 10 year plan work without taking care of the day to day? In that sense, can anyone escape the tyranny of time?  The more your competitors harness, the more you will need to adapt or adopt.

From an accountability perspective, the issue may be simply finding a way to reconcile that with monthly or quarterly performance reporting to the real time data.

What effect this will have on the way we operate in the public and private sectors?  Only time will tell.

National Archives announced today UK government licensing policy extended to make more public sector information available:

Building on the success of the Open Government Licence, The National Archives has extended the scope of its licensing policy, encouraging and enabling even easier re-use of a wider range of public sector information.

The UK Government Licensing Framework (UKGLF), the policy and legal framework for the re-use of public sector information, now offers a growing portfolio of licences and guidance to meet the diverse needs and requirements of both public sector information providers and re-user communities.

On the surface this is move is to to be welcomed.  Providing, amongst other things, licensing choices and guidance for re-using information free of charge for non-commercial purposes – the Non-Commercial Government Licence; guidance to licensing where charges apply and for the licensing of software and source code.

All this is available from the UK Government Licensing Framework area of the National Archives site, along with FAQs and other useful supporting information, including machine readable licenses.

As the press release says, the extensions are building on the success of the Open Government License(OGL) and are designed to cover what the OGL can not. 

So the [data publishers] thought process should be to try to publish under the OGL and then, only if ownership/licensing/cost of production provide an overwhelming case to be more restrictive, utilise these extensions and/or guidance.

My concern, having listened to many questions at conferences from what I would characterise as government conservative traditionalists, is that many will start at the charge-for/non-commercial use end of this licensing spectrum because of the fear/danger of opening up data too openly.  I do hope my concerns are unfounded and that the use of these extensions will be the exception, with the OGL being the de facto licence of choice for all public sector data.

A couple of weeks back UK Prime Minister David Cameron announced the broadening of the publicly available government data with the publishing of key data on the National Health Service, schools, criminal courts and transport.

The background to the announcement was a celebration of the preceding year of activity in the areas of transparency and open data, with many core government data sets being published. Too many to list here, but the 7,200+ listed on data.gov.uk gives you an insight.  The political guide to this is undeniable, as Mr Cameron makes clear in his YouTube speech for the announcement “Information is power because it allows people to hold the powerful to account”

His “I believe it will also drive economic growth as companies can use this new data to build web sites or apps that allow people to access this information in creative ways” statement also gives an indication of the drivers for the way forward.

To be successful in either of these ambitions, the people and the companies have to have access to information in an easy and reliable way that gives them confidence to build their opinions and their business models upon.  What do we measure that ease and reliability against – is it the against the world of audited business practice, where the legal eagles and armchair auditors strive towards perfection, or is it the web world in which a lack of perfection is accepted with the norm of good-enough is good enough?  I believe that with government data, on the web, we should still accept that it will not be perfect, but the good-enough hurdle should be set higher than we would expect from the likes of Wikipedia and some other oft used data sources.

There are two mentions, in the words that accompany the announcement, that appear to recognise this. Firstly in the announcement itself on the Number 10 website, this: All of the new datasets will be published in an open standardised format so they can be freely re-used under the Open Government Licence by third parties. What ‘open standardised format’ actually means is something we need to delve in to, but previous data.gov.uk work towards Linked Data and shared reliable identifiers for things [such as postcodes, schools, and stations] bodes well.   Secondly in Mr Cameron’s letter to his Cabinet we get a section on improving data quality, including things like plain English descriptions of scope and purpose, introducing unique identifiers to help tracking of interactions with companies, and an action plan for improving quality and comparability of data.

So where are we now?  Some of the new data is not perfect, as this thread on the UK Government Data Developers Google Group, shows.  William Waltes, identifies that the [government] reporting of transactions with the Open Knowledge Foundation, do not match the transactions in the OKF’s own books, therefore calling in to question how reliable those [government] figures are.  In my opinion, this is an example where we should applaud the release of such new data but, with conversations such as the one William started, help those who are publishing the data to improve the quality, reliability and comparability of their output.  Of course by definition it means that the publishers are prepared and ready to listen – and are listening.

What we shouldn’t do is throw our hands in the air in despair because the first publishing of data by some departments is not up to what we would expect, or decry the move towards shared [URI based] identifiers because they look confusing in a csv file.  Data publishers will get better at it with helpful criticism.  I am also convinced that sharing well known reliable identifiers for things across desperate, government and non-government, data will in the medium term have a far greater benefit than most [including enthusiasts for Linked Data like me] can currently envisage.

I recently attended an LGID Knowledge Hub workshop that packed into one day a plethora of discussion around approaches for fostering and curating an environment in which local government officers can share their data, and their interpretation of that data, as applications. Broadly the discussions were split into two areas. Apps and data.

The area that (in my opinion rightly) got the most attention was data. Any app’s functionality will be dependant on the data available to it. Data forms the ingredients that can be mixed to provide rich input for understanding new problems in new ways.

But data is only as good as… well, the data curator.

Good questions around data curation were raised, including: What does this data look like? How do I know that data can be of use to me? Where does the data come from? Can I trust this data? Will the data be updated? Will the data be available tomorrow? Will the data be in a form that allows me to work with it easily? Will the data mesh with some other data?

We talk about some of these issues through our training and consulting activities, because in every project these are issues that will need to be addressed. In the context of the LGID use case, the way the Knowledge Hub is seen and used by it’s proposed audience will rest on how useful the data is to a local government officer in their daily workflow.

Lets take local government spending data as an example of how an officer might need to be able to reconcile two datasets. If dataset A contains a column ‘spend’, are the values directly comparable with a similarly named column in dataset B? Do authority A and authority B define spend in the same way? Is it possible that one column includes VAT while the other does not? How does the officer get the information they need to make judgements as to whether the spend columns are talking about the same thing?

Ideally the Knowledge Hub should provide rich descriptions of the data. We start to wonder where these descriptions would come from, and hopefully, from what I have said so far, you answered that and said ‘the data curator’ or ‘the person who knows most about the data’. It was suggested in the workshop that that people should be encouraged to describe their data in as much detail as possible. To my mind, this means that people should be encouraged to describe their data using a descriptive schema that allows the data to be self describing.

I am of course suggesting that the data, described as Linked Data, would allow for easiest usability and greatest re-usability. This would mean the Knowledge Hub would need to encourage users to go the extra mile by leading them through a series of questions to tease out the nuances of the data. Not an easy thing to do if the data that is being described is going to be heterogeneous and the people entering the data are not used to formally describing their data.

I look with interest to see how Knowledge Hub will solve this problem.

Friday night – nothing on the TV – I know! I’ll browse through the Protection of Freedoms Bill, currently passing through the UK Parliament. Sad I know, but interesting.

Lets scroll back in time a bit to November 19th 2010 and a government press conference introduced by a video from Prime Minister David Cameron.  The headline story was about the publishing of government spending and contract data, but towards the end of this 109 second short he said the following:

… the most exciting is a new right to data. Which will let people request streams of government information and use it for social or commercial purposes.  Take all this together and we really can make this one of the most open, accountable and transparent governments there is.  Let me end by saying this. You are going to have so much information about what we do, how much of your money we spend doing it, and what the outcome is.  So use it, exploit it, hold us to account.  Together we can set a great example of what a modern democracy aught to look like. (my emphasis)

Obviously to realise this Right to Data there needs to be some legislation, which brings me to the Protection of Freedoms Bill.  This is one of those bills which covers all sorts of issues, from rules for destruction of fingerprints and DNA profiles, CCTV camera regulations, detention of terrorist suspects, to freedom of information and data protection.  Zooming in on the bits on the topic of the release and publication of datasets held by public authorities, we find a set of clauses that amend the Freedom of Information Act 2000. 

Re-use

After some amendments which allow for datasets and provision in electronic form we get this: “the public authority must, so far as reasonably practicable, provide the information to the applicant in an electronic form which is capable of re-use.”  Unfortunately there is no definition of the term re-use.  It could be argued that a pdf of some tables in a MS Word document could be re-used, where as I believe the spirit of the legislation should be made more explicit to by identifying non-proprietary data formats.  I know this would be a tricky job for the parliamentary draftsmen, as we would not want to restrict it to things, such as XML and csv, that could age and be replaced by something better which then could not be used as it had not been mentioned in the legislation, but I believe that just using the term ‘re-use’ is far too woolly and open to [mis]interpretation.

What is [not] a dataset

This is one of the areas that raises most concern for me. Checkout this wording from the Bill: I am OK with (a) – data collected as part of an authority doing it’s job – and (c) – don’t change the data you have collected – publishing that raw data is important.  However (b) specifically excludes data that is the product of analysis.  Presumably analysis of collected data is one significant way that an authority measures the outcomes of its efforts.  Understanding that analysis will help understand the subsequent decisions and actions they make and take.  I assume that there may be some specific reasons that underpin this blanket exclusion of analysis data.  If there are, they should be identified, instead of generally throttling the output of useful data that will go a long way to helping with Mr Cameron’s stated ambition for us to be able to see “what the outcome is” of the spending of public money.

Release of datasets for re-use

This is a whole new section (11A)  to be added to the 2000 act to cover the release of datasets. It covers ownership, copyright, and/or database right of the information to be published and states that it should be published under “the licence specified by the Secretary of State in a code of practice issued under section 45”. Section 45 basically puts in to the hands of the Secretary of State the definition of the license(s) data should be published under.  As of today the Open Government Licence for public sector information is what is wanted to keep the publishing of information open.  However, what is there to stop a future Secretary of State, who has a less open outlook in replacing it with far more restrictive licences?  Do we not need some form of presumption of openness being attached to the Secretary of States powers as part of this change in legislation?

On the topic of presumptions of openness, the wording of this bill contains phrases such as “unless the authority is satisfied that it is not appropriate for the dataset to be published” and “where reasonably practicable”.  It is clear that many in the public sector are not as enthusiastic about publishing data as the current government position and such vague phrases as these may well be unreasonably used by some in justifying a throttling of the stream of information.   They could easily be used to build in a bureaucratic decision hurdle for each dataset to have to jump, proving its appropriateness and practicality, before publication.  I am sure that it would not be beyond a parliamentary draftsman’s skill to produce wording that means that all will be published, unless a specific objection is raised for an individual dataset, for reasons of excessive effort or data protection reasons.

Up-dated data

Data published by an authority should be published under a scheme, the following applies here:How should we interpret “any up-dated version held by the authority of such a dataset”? My interpretation is that once a dataset has been published is shall continue to be published as it changes.  The precedent for this is spending data – having published authority spending for January 2011, authorities should be automa
tically publishing it for February and following months.  But what if, in response to a request, an authority publishes the contents of a spreadsheet used to track the amount of salt applied to roads in its area during winter 2010-11 and then uses a different spreadsheet for the following winter.  Does the output of that new spreadsheet constitute a new dataset, or an up-date to it’s predecessor?  From the wording in the Bill it is not clear.

Who does it cover?

I probably need a bit of help here from those that understand the public sector better than I do, but I am suspicious that references to the organisations listed in Schedule 1 and “the wider public sector”, do not take the net wide enough to cover some of the data that is relevant to our daily lives but is delivered on behalf of some authorities by third parties.  For example I am aware that recently a large city was not able to inform citizens of their rubbish collection schedules because that data was considered as commercially restricted by their service provider.

So in summary, I welcome the commitment to a right to data being realised by streams of government information about what we do, how much of our money is spend doing it, and what the outcomes are.  However, I am sceptical as to how effective the measures in the current Protection of Freedoms Bill will be in delivering them.  Especially in the light of very recent comments made by the Prime Minister highlighting the “enemies of enterprise” in Whitehall and town halls across the country, attacking what he called the “mad” bureaucracy that holds back entrepreneurs.  Those enemies are just the people who might take the wording of this bill as ammunition in their cause.

Whilst being concerned about this topic, I have been wondering why few are commenting on it.  Are the majority just taking the press conference statements by David Cameron, and his fellow Ministers, as indications of a battle won, or am I missing something?  I promote Sir Tim Berners-Lee’s 5 Star Data as the steps towards a Web of Linked Data – if we don’t get the publishing of public sector data to at least 3 star standard (Available as machine-readable structured data – in non-proprietary format), many of the current ambitions may remain just that, ambitions.  That would be a massive missed opportunity. 

So are we getting a right to data? – or just some provisions to extend the Freedom of Information Act a bit further in the dataset direction?  I’m not sure.

Personal note: As you may tell from the above, I am no expert on the interpretation of parliamentary legislation, and I have left several unanswered questions hanging in this post.  Any help in clarifying my thinking, confirming or disproving my assumptions, or answering some of those questions, will be gratefully received in comments to this post or your own posted thoughts.