Mashup my blog - from blog to maps

A while back I was playing with Yahoo Pipes after Geo-ripping Wikipedia. This was all part of my on going fight against just doing Word, Powerpoint and whiteboard consulting and generally making sure that I still understand what I am talking about.

So the next challenge was obvious, take the RSS feed and put it up on a Google Map, hey its a mashup and we all know that they MUST use Google Maps :) For this next challenge I decided to abandon Yahoo Pipes for a while (mainly because it absolutely NAILS the server if you are looping through a big blog feed, and do it in the old "traditional" way of processing data in the single lump it comes in (the RSS feed).

My task was as follows

• Put the Geo tagged feed onto a map with the Google Maps API
  
• Generate a KML file for Google Earth
• Use Ajax in someway

I then added in another one (why not) which was to use the Google Maps API to actually do a search on those places I hadn't managed to rip from Wikipedia. Interestingly Google Maps don't enable you to search on the UK so having the Wikipedia information actually helped there.

So anyway after a few days of "downtime" working and lots of cursing about javascript I managed to get it all up and running. The Map your blog page needs an RSS or Atom feed that consists of posts that have (at this stage) just the placename in the title. It will then (after a shortish wait) display the Map and the summary of the trip information (distance between blog items) and the carbon footprint of that based on the mode of travel (again a single tag for the transport type at this stage). The final challenge I set myself was that all of the display elements would be done in a single page, no round-trip to the server.... no real reason I just wanted to try it.

To get you going try this simple UK to Wengen trip or there is a much messier one that goes all over the globe, that later one takes quite a while for Google to render. If you don't see it after 30 seconds just hit the button, I haven't done a "waiting for Google" bit yet, if you get bored just hit the button again.... Over on the right are a bunch of links including one to the KMZ file which should pop up directly in Google Earth (if you haven't got it download it).

I have to say it was really rather easy, apart from the Javascript issues. I used the Yahoo Javascript libraries to do the initially minor bit of Ajax in there, and Google Maps is fine. The paticular piece I liked was turning a KML file into a KMZ file in Java, worth a quick post on its own because of how easy it was.

Now I feel I can safely attack powerpoint for another couple of weeks.

Technorati Tags: SOA, Service Architecture

Geo-ripping Wikipedia

As part of my on going quest to stop my drift into senility and powerpoint (the difference is marginal) and make sure that when I recommend things to clients that they actually work I went in search of some Web Services to play with. Now there used to be a useful bunch over a Capescience, now they've just got the Global Weather one which is okay, but I could do with more than one (and I hate stock quote examples). I also wanted to see what could be done to get some interesting information out of wikipedia, so I hatched a plan.

The idea was to create a very simple Web Service which took the Wikipedia Backup file and then extracted from it the geolocation information that now exists on lots of the pages.

Stage one was doing the georip, this was very simple. I elected to use a StAX parser (the reference implementation infact) as the file is excessively large. Using StAX was certainly quick (it takes Windows longer to copy the file somewhere than it does StAX to do the parse) but there are a few odd elements in using it (which I'll probably put a separate post on). That gave me a simple database schema (created using Hibernate 3 as an EJB 3 implementation and using inheritance in the objects mapped down onto the tables, very nice implementation by the Hibernate folks).

Next up was the bit that I thought should be simple, after all I had my domain model, I had my queries, I had even built some service facades, so how long could it possibly take to get this onto a server? The answer was far too long, particularly if you try and use Axis2 v1.0 (which I just couldn't get to work), switching to Axis 1.4 picked up the pace considerably, and thanks to some server coaching by Mr Hedges its now up and running.

There are around 70,000 geolocations that I managed to extract from Wikipedia. Some of these aren't accurate for several reasons
1) They just aren't accurate in Wikipedia
2) There were multiple co-ordinates in the page, so I just picked the first one
3) There are 8 formats that I identified, there could be more which would be parsed wrong.

So there it is, extracting 70,000 locations out of unstructured information and repurposing it via a Web Service for external access. Couple of notes

1) The Wikipedia -> DB translation is done offline as a batch
2) Name based queries are limited to ten returns
3) Any issues let me know
4) Don't rely on any information in this for anything serious.

The next stage is to get some Web Services up that run WS-Security and some other WS-* elements so there are more public testing services for people to use.

Technorati Tags: SOA, Wikipedia, geo, geo ripping, Web Service, Service Architecture