The state of Open Data 2014

I was reading an old blog post I wrote in 2011 about the state of Open Data in BC ( http://www.unrest.ca/2011-state-of-open-data ) and thought I’d pen another annual update. I should do this every year, but sadly, I’ve not had the time to really blog lately.

In 2011, I highlighted rights and privacy clearing, cost recovery, scaling and license diversity as major failures and opportunities for course correction in the emerging open data field — and I’m sorry to say, many of these problems materialized.

But we’ve also had a lot of successes since 2011 — the Open Data Society of BC (disclosure: I’m a member of the board of directors) has held two highly successful Open Data Summits that have convened hundreds of people from across Canada and even the world to talk about Open Data. My favourite memories of these events were the edgy talks, like Pete Forde’s entitled ‘People are dying to get the data’ ( https://www.youtube.com/watch?v=s7rpKYSZUDo ) because they really bridge a gap between the civil service and the data activism that is occurring all over the web today. These events help bring together people who would otherwise never meet, and invite them to learn from each other.

The Data BC group of the provincial government has been doing a great job with what limited resources they have — in the last couple year’s they’ve facilitated the publishing of unprecedented transparency/accountability information in the form of fiscal data, the personal information directory and geographic data that has been hugely helpful to a number of stakeholders.  They’ve done considerable work on licensing and on trying to source data — even where it doesn’t exist. I’ve come to like and respect the work they’ve done for BC in a challenging environment.

But there’s a problem in the foundation of this group as well — they don’t have a budget to replace funding for datasets that are currently being charged for (the cost recovery problem), they don’t have the statutory ability to command data release from other ministries, and they don’t have the resources needed to implement the commitments made in the G8 Open Data Charter — especially the transformative commitment to an ‘Open By Default Policy’. This fix will have to come from cabinet, take the form of significant budget increases and involve the creation of a legislative framework. Moreover, the architecture of data release will have to change — a central team fetching data for a portal wont scale. Data release has to be federated within each ministry, and just as each ministry has an officer responsible for handling FOI requests, so too should they have one to handle data requests. Its 2014, its time to make data exchange as seamless and as common as email in the public sector.

The lawyers are also hurting the economics of open data — while much progress has been achieved on licensing, there are still very real debates about conformance to the Open Definition and serious problems with the legal disclaimers of liability for intellectual property and privacy rights clearing. It is my belief that these issues are hurting commercial investment in Open Data.

Across the country, other groups are also making positive progress — the Federal Government included a large funding commitment for Open Data in their 2014 budget, they’re hosting hackathons (which they misspell as appathon [because hackers = bad of course]) and their MP Tony Clement is taking every opportunity to talk about the benefits of open data and the future promise of a more transparent public service. Major wins with digitally filing access-to-information requests, and citizen consultation exists in this area. The publication of valuable datasets like the Canada Land Inventory and Border Wait Times are also impressive.

There too, there are big failures. Canada Post is suing innovators over their use of Postal Codes ( CBC Story ) and DFO’s hydrographic data remains closed and mostly collecting dust. The government seems to be ignoring responsibility for Canada Post’s behaviour, but most will point out that they have jurisdiction over the Canada Post Corporation Act and could make a simple and common sense legislative change to resolve this embarrassment to our federal open data commitments.

We’re making progress municipally — the City of Vancouver has made amazing strides in digital archiving, making digitized archives available on the local intranet in a unique and groundbreaking way that deals with intellectual property concerns. The City of Victoria has embraced open data, they launched VicMap (making their cadastral data open), began webcasting council meetings and published an open data portal. They even hosted a hackathon with Mayor Dean Fortin and Councillor Marianne Alto helping the Mozilla Webmaker folks teach children about digital literacy and creating the web [ link ]. The City of Nanaimo continues to lead the pack with realtime feeds, bid opportunities, maps of city owned fibre resources, digitally accessible FOI processes and so much more.

In the private sector and ngo space there are so many notable projects — the GSK backed Open Source Malaria project being my favourite. There are also successes like Luke Closs' and David Eaves’ recollect.net in the civic app space.

The hacker space is also seeing some success with proof-of-concept prototype applications developed by citizens at hackathons going on to inspire civil servants to create their own versions and publish them widely. The BC Health Service Locator App and the Victoria Police Department App both get credit for listening to citizen input.  Other apps have been created and have seen little to no uptake, like those developed to help citizens better understand freshwater fishing regulations (mycatch), or storefront display apps to help the homeless find shelters with available space (VanShelter). The next steps here are clearly to create bidirectional projects that allow both civil servants and citizens to work collaboratively on applications together using the open source methodology. (Who wants to be the first to get the Queen in right of British Columbia into using GitHub?)… Other projects have failed to find traction due to lack of, or bad quality data. My OpenMoonMap.org site is failing, due to unreliable WMS-only access to data from NASA which is down more often than it is up… the lesson here, online services are no replacement for downloadable primary source data. mycelium.ca (My house of commons video service) is in its 7th year of operation, and continues to prove that even simple prototype apps can be useful and long-lived, drive policy change (House of Commons Guidelines) and find feature uptake (Hansard now has video clips embedded). Hopefully same-day recording download, clipping and linking will be added to ParlVU and this app will no longer be useful.

For the coming year the Open Data Society of BC is crowdsourcing its agenda and I’d encourage you to participate in that discussion and to join or support the society.  via OpenDataBC-Society

I know I missed some people and agencies who are doing great things, so please leave comments if I missed you. (tweet me @kevinsmcarthur for an account as I dont monitor this site’s admin pages often)