Open data on the cheap

Jimmy Leach, Head of Digital Diplomacy at the Foreign & Commonwealth Office blogged earlier about a quiet little project his team have rolled out, using extended RSS 2.0 feeds to provide access to the FCO’s travel advice data.

As he says, the key thing for publicly-funded organisations is to get the information out there, which is why the corporate platform, and even corporate social media channels, are just the beginning. RSS feeds are a tried-and-tested technique for ensuring content can reach a wider audience:

Anyone can follow the latest alerts and changes using our travel advice RSS feeds in a standard reader like Google Reader or Netvibes. But you can consider this a call to developers to use our feeds as they want, to make our data useful, to add relevant information, to create visualisations, mobile apps or map-based viewers, incorporating extra machine-readable data about locations, contact details, reviews, ticket booking, all sorts of information and services that you wouldn’t expect a government department to provide, and, hopefully, pulled together in clever and innovative ways that you wouldn’t expect from a civil servant.

The existing travel advice feed on the site contains the alerts, the advice, the news and the embassy details all rolled into one. It does the job. It’s useful and relevant but it is also blunt and we know there are ways we can do it better.

We have started the ball rolling by creating some test feeds containing additional custom elements, so that each element in the feed is generated from a single field from our database of travel advice.

Inspired by Matthew Somerville’s use of iUI to fake an iPhone look and feel to his Train Times application, I’ve put together a little demo of how the FCO’s new feeds might be repurposed, with a little app optimised for iPhones which takes the latest alerts, visualises them on a map, and enables you to get the phone number and opening times for an embassy if you need it. It’s a fairly silly little proof of concept, but hopefully it shows that RSS feeds don’t just have to live in newsreaders. And it’s what Andrea DiMaio has decreed.

The bigger point here is about open data and cost. Most enterprise CMSes can generate RSS feeds, and it’s a technology that almost all developers and webbies feel comfortable with. So without the cost and complexity of building and mantaining a full API to their database, a corporate public sector organisation has been able to support reuse in a quick and simple way. Jimmy has asked for thoughts from developers and others on how the feeds might be cleaned up and made more useful, so do give him your ideas.

Newsroom: the backstory

Cast your mind back if you will to chilly February, amid the growing crescendo/death spiral of pre-election communications. Neil and his team were finishing off the new corporate website, having shunned friends and family for weekends on end. A member of the senior management team came bounding back from a cross government meeting where they had been shown this, and, in a nutshell, they wanted one too.

The brief was helpfully loose: make it easier for the media to access the information they needed via simple link in the bottom of a press notice, without generating a load of extra work for Press Officers. From the Digital team’s perspective, we wanted to increase visibility of our YouTube and Flickr content for media, ensuring that these channels get promoted in every news release. Oh, and the kicker: make something technology independent, that could survive the imminent move from WordPress to SiteCore, without incurring external costs. So we set out to develop something based largely in client-side technologies (i.e. Javascript and CSS) which usefully aggregated corporate announcements, multimedia output and press office contacts for mainstream media and bloggers in a single place – frankly, more of a technical and design challenge than a strategic one, but a fun one nonetheless.

There were half a dozen or so information sources to play with*:

• Press Releases, ministerial speeches (RSS feed)
• Tweets from corporate accounts (RSS feeds)
• Videos on YouTube (RSS feed with multimedia enclosures)
• Flickr photos (API)
• Podcasts on SoundCloud (added by the team later, again, RSS feed)
• Contact details for Press Officers & key facts on policies (static text)
• Email alerts for media to sign up to via GovDelivery

*We also had a plan to add a couple of extras which were built but not yet used. Case studies published elsewhere online were to be tagged using a corporate Delicious account and imported into the newsroom using the RSS feed for the tag. Urgent statements or rebuttals put out by a Press Officer out of hours sometimes aren’t issued as Press Notices in the normal way, so we set up a private Tumblr site to which these could be emailed, which could be embedded or imported into the Newsroom, again via RSS.

The primary tool in our arsenal was the wonderous Feed2JS, which takes an RSS feed and gives you a snippet of Javascript to embed which will render it for you in HTML. It’s free and awesome (and you can even self-host it if you want). This little tool helps single-handedly render the majority of the Newsroom content, the code snippet tweaked slightly to ensure the <noscript> alternative ensures the site degrades fairly gracefully for non-Javascript enabled browsers.

I also developed a couple of code snippets to render the content of a Flickr account or set as an RSS, HTML or Javascript snippet, and likewise with YouTube – feel free to grab the code from those links if that kind of thing is of use to you.

• Version 0.1 (click the image to enlarge) was a good proof of concept, built in an empty page template on our old WordPress site. But there was too much to take in for a notoriously lazy audience.
• Version 0.2 was an improvement, splitting the content into more manageable chunks with a natty Apple-style navigation bar and some concertina sections done in Javascript – but it still felt hard to differentiate the content types on the page
• Version 0.3 was almost there, introducing some nice little icons for the different content types, using CSS to help visually distinguish the lists, and losing the unnecessary mission statement with some DOM-rewriting to save valuable pixels for this audience. And then we moved to SiteCore and purdah struck, so…
• …Version 1.0, which you can now see in all its glory transferred the code into a new CMS and migrated across a stylesheet. The team added SoundCloud podcasts using its RSS feed, in the same way as the other media types.

Early feedback on the prototype from journalists was positive, the Press Office got a nice-looking tool which required literally zero additional work beyond emailing over their contact list, and Neil got one of his much-loved quick wins – and within SiteCore too. Props for this one to Rhys and Ian in the BIS Digital Communications team.

Photo credit: Victoria Peckham

Public Appointments by RSS

In the words of Directgov:

A public appointment is an appointment to the board of a public body or to a government committee. Around 18,500 men and women hold a public appointment.

The public bodies involved are quite important, including health trusts, museum boards and regulators, some demanding specialist skills in law or social work, but many requiring general common sense and broad experience. So it’s important that the people who fill these posts are of the right calibre and reflect the diversity of our society.

The Cabinet Office has recently revamped its Public Appointments system, and you can now sign up to sophisticated email alerts about public appointments vacancies you might be interested in. As a publisher of vacancies, the central system also has an excellent API, enabling you to extract data feeds from the vacancy database to republish on your own site. There’s even some RDFa in the output should you wish to use that to mark-up the vacancy descriptions.

I’ve just created and added a dead simple RSS feed for the BIS-related public appointments to our homepage. But anyone can grab the code and set it up to generate their own feed, or indeed re-publish the vacancy data far and wide in any format compliant with its licence, in order to help spread the word about the interesting and varied positions available.

Hurrah for open data and APIs, and above all, hurrah to the Cabinet Office for building one in this case. Thanks chaps.

Adding RDFa to a consultation

Recently, I’ve been involved in a project to ensure our consultations support RDFa markup, to make them indexable and reusable by third parties, including Directgov. Without duplicating the quite accessible and useful COI guidance, I thought I’d summarise here the process involved from the perspective of implementing the standard with minimal prior knowledge of the whys and wherefores.

Why bother?

As of Jan 1st 2010, it’s now a mandatory requirement for government sites. But more importantly than that, it’s a Jolly Good Idea to provide a low-maintenance way of enabling other systems and services to grab a list of consultations from your site, and identify the important metadata about them, including the closing date and how to respond. Short term, it will make services like TellThemWhatYouThink and Directgov more useful, but in terms of the bigger picture, it will expose the opportunity to get involved with policymaking to a wider audience, and reduce the hassle for those who are already part of our regular stakeholder group (by making possible new services such as auto email alerts, RSS feeds, cross-government updates and so on).

What’s involved?

RDFa offers a simple way to add meaningful information to existing web pages, which can be extracted easily by software (as opposed to hit-and-miss ‘scraping’ of regular web pages). As a lay person, I’d say there are three key principles which I can articulate:

1. Be unobtrusive and minimalistic: taking this approach lets you add extra items to pages which aren’t seen by regular browsing visitors, but which are accessible to software robots looking for them. It’s also not ‘an extra thing’ to maintain and serve like an RSS feed, so reduces risk, in theory.
2. Offer clean data: through being consistent in how data about the consultation is described, the idea is that RDFa helps to extract very clean information about the consultation – for example, an unambiguous closing date, a response email address, an exact postcode, all in formats which can then be used in other ways (plotted on a map, listed on a calendar, turned into a mailform on a website etc)
3. Extend existing conventions: the most complicated aspect of implementing this particular specification is that the authors have gone out of their way to find existing wheels rather than reinvent their own. So they use Dublin Core metadata to describe authors and organisations; vCard to describe response contact information; plus nods to DBPedia and FOAF (Friend Of A Friend) to support these major semantic web initiatives. Only for the  gaps where specific consultation information needs to be marked up is there a new standard introduced, using the namespace (prefix) argot.

In a nutshell, the process involves tweaking the template for your consultation pages, adding extra metadata elements and attributes. This is only as easy or hard as your CMS makes it. It’s important that it’s right though – even a few ‘broken bits’ could render the page useless to a software robot trying to extract data from it.

How to do it

Read the COI guidance (and give it to your developer), which is the most comprehensive guide, with useful illustrated examples. There’s also a worked up HTML page showing how this works, and of course you’re welcome to look at ours (which I *think* are right, based on feedback from the gurus).

As an example (but again, you should read the official guidance) I found I needed to work through the following:

• ensure we have a single page per per consultation
• amend the DOCTYPE, if you’re using something like the standard XHTML strict/transitional version. Needs to tell requesters of the page that it contains RDFa
• add some attributes to the <html> element, highlighting the namespaces (vocabularies) you’re referencing in the document
• add Dublin Core metadata elements/attributes to your page <head> element if they’re not there already
• ensure we have a wrapper <div> around the consultation information which again references the namespaces (vocabularies) you’re using. This also identifies the name of the organisation publishing the document
• add some Dublin Core metadata attributes as <spans> within this <div> identifying this as a consultation
• add some Dublin Core attributes to key bits of the HTML, such as the consultation title, start date, closing date and description, marking these as such – and in the case of dates, ensuring there’s a machine-readable data format value in the attribute. Also add a unique identifier – a reference number – to each consultation (not something we’d done routinely before)
• ensure the contact details for responses is carefully structured using vCard format, with separate ‘Full Name’, ‘Street Address’, ‘Locality’ and ‘Post Code’ elements, suitably marked-up with attributes. Since vCard doesn’t cover the specific case of a consultation with an email reply address, for example, these elements are marked up with the new argot: namespace attributes
• add Dublin Core-based attributes describing the file attachments – the consultation document itself, and any related ones such as appendices or Impact Assessments

UPDATE: in retrospect, it was foolish to attempt a blog post about code without some code examples. I’ve tried and failed to find a half-decent code syntax highlighter plugin for WordPress, but the following couple of screenshots hopefully illustrate the before and after situations for the contact information part of a consultation:

Before, plain HTML:

After, with RDFa added (and marked up more semantically as a list item within the consultation metadata)

What help is available?

I worked from the examples given in the COI guidance and the pioneers in this at the Ministry of Justice. The COI Digigov team are your allies in helping to implement this, and should be able to answer queries and/or direct you to sources of further implementation advice and support.

In terms of online tools, you can see whether your RDFa is visible to suitably-equipped applications using Mark Birbeck’s tool or bookmarklet, if you prefer (and he should know; he invented RDFa).

Good luck!

P.S. If you Know About This Stuff and feel I’m giving duff advice here, please drop me a line in the comments or via the contact form and I’ll correct. Thanks.

Should you learn to code?

I was never a born project manager. I didn’t have the organisational skills, the discipline or indeed a sufficient dislike of my colleagues to want to inflict upon them the highlight reports, gantt charts and benefits realisation plans needed for Proper Projects. But in my fairly brief stint as formal Project Manager, I did have one knack, and that was getting on quite well with developers. I can only think that the reason for this was that I can relate to the work they do, have an idea of what is easy and what is hard, and respect the elegance of the craft – because I dabble in code myself.

My ears pricked up when Alistair pointed me to Mercedes Bunz of The Guardian asking: ‘Will journalists of the future need to know how to code?’:

Up until now, as a journalist you worked with information, researching facts and figures which then you passed on to the reader. However, in a digital world there are more platforms you can use to convey that information – think of maps or mobile applications, augmented reality. And to be able to do that you will have know how to code.

I think it’s an interesting thesis, even if the scenario of journalists learning Python to develop their own Google-esque apps is pretty hardcore. But I don’t think it just applies to journalists – almost regardless of your role, I think it’s worth learning a bit of code, especially if your academic training has been in hand-wavy social sciences like me.

1. It helps you think about how everyday processes work: there’s nothing like building your own applications to make you think  logically about how people behave online, and the hidden sophistication of seemingly simple systems like cash machines or website subscription services.
2. It’s good for your attention to detail and organisational skills: you can be sloppy about how you capitalise words or use punctuation in the real world, but the world of code makes you a more organised, consistent person (n.b. those who know me will laugh at this hubris)
3. It gives you an insight into why websites work the way they do, and why they break: as a webbie or even just a web user, coding for yourself helps you understand the anatomy of websites, the technologies which come together to deliver them, and gives you some explanations for why they’re ‘being a bit funny today’.
4. It lets you translate ideas into prototypes: talk is cheap, but if you can turn it into a prototype, you’re already a step ahead – and you can refine your thinking as you build it and get feedback on something tangible, rather than just a brainwave.
5. It opens up a new world of lifehacks you can build for yourself: whether it’s a way to backup your Twitter account or a to-do list application that actually reflects how you work, being able to write bits of code to save yourself time is neat.
6. It’s a social* thing: fifteen years ago, I was getting little applications on computer magazine cover disks, and receiving letters back from all over the world via the school register. Now, when I release code I get feedback instantly, along with help, suggestions and improvements, and feel part of something energetic and positive. (n.b. I say ‘social’, but not necessarily family friendly. I’m still squaring that circle
7. It’s creative and relaxing: I don’t actually get paid to code, so for me there’s something relaxing and challenging in sitting down of an evening to make a new tool or improve something.
8. It’s good for the career: maybe a bit obvious, but even a small amount of coding capability (real; not just puffed-up for CV purposes) helps you do your job, and get noticed for doing it, generally without antagonising your colleagues. Frankly, bosses like clever bits of digital innovation: it’s worked for me in pretty much every job I’ve ever had, particularly the non-digital ones.

It’s worth putting two caveats on that list of benefits:

1. Know your limits: the old cliche ‘a little knowledge is a dangerous thing’ is a double-edged sword when it comes to coding. If you believe it, then you’ll never start learning anything. But if you ignore it, you’ll find yourself in dangerous territory (exposed to hackers, losing friends’ data, costing yourself money etc). Strike a balance between the courage to learn, and the humility to ask for help or say you don’t know.
2. It’s a long way to the summit: ‘coding’ as I’m describing it here is a shorthand for knowledge of a whole range of technologies – all of which are changing over time – which you’ll find you want to develop at least some familiarity with. Of course, you can do some things with just a little practice and knowledge, but unless you focus very narrowly, I don’t think you ever reach a plateau of knowledge – there’s always an infinite amount more to know and potentially keep up with. You’ll be learning forever.

I hear the other side of it, of course: do what you’re good at, and leave the heavy lifting to the professionals, like you would car maintenance or central heating. I think that view gets too much unthinking acceptance, for the reasons above and more. Be proud to be a jack-of-all-trades, master-of-none, I say.