- Identify datasets that would make interesting open data
- Help to create the first open data API
- Provide a platform ready for #EULife AppChallenge
We worked closely with the UK Data Service to identify three datasets that we felt could be interesting to developers if they were made available as open data. Eventually we narrowed this down to working with Eurofound in Dublin, Ireland to open up the European Quality of Life Survey (2007 & 2011) via an API which we designed with the UK Data Service team.
The data that we’ve opened up shows how people across Europe in 2007 and 2011 felt about:
- Well Being e.g. satisfaction and happiness
- Living Standards e.g. financial situation and standard of living
- Work-Life Balance
- Family & Social Life e.g. Satisfaction, sources of support and household structure
- Housing & Local Environment e.g. Affordability
- Public Services e.g. Healthcare and Education
- Quality of Society e.g. Trust in people
The process of providing this as an API involved multiple expert teams at the UK Data Service and was not straight forward. First we had to make decisions about disclosure. This is where, if you’re not careful, it’s possible to figure out who a person is due to a low volume of data. A classic example would be someone who votes for a right wing party, lives in a particular post code and works as a doctor. You could start to figure out who that person is so you have to automate the process of checking the data. Secondly the data needs checking for errors and decisions need to be made about which fields/variables (or questions in this case) are to be kept in. Next, weighting is particularly tricky. This is a common method used by data scientists to make up for a lack of sample size in a particular demographic. For example, young white males are notoriously difficult to survey, so normally if the sample size of young white males is particularly low, the data scientist will extrapolate the findings from this group. How this is done is not straight forward. We made a decision that we couldn’t leave this up to developers. At the same time our data has to produce similar results to what the Office of National Statistics would expect so weighting becomes a key issue.
Next was the technical infrastructure. In general we prefer the cloud, but this isn’t always possible. There are issues like where the data is stored (there are now EU cloud services that are certified by the UK Government) and how intensive the API operations are. The decision was made by the UKDS to keep the API “in house” and run the infrastructure at the UK Data Archive. This required design, build and testing skills.
The AppChallenge team provided valuable design guidance and testing feedback to help the development of the API in preparation for the next phase, which is to run a crowdsourcing contest to get developers to use the data.
Visit https://ukdataservice.3scale.net to use the Open Data API