Mashup my blog - from blog to maps

A while back I was playing with Yahoo Pipes after Geo-ripping Wikipedia. This was all part of my on going fight against just doing Word, Powerpoint and whiteboard consulting and generally making sure that I still understand what I am talking about.

So the next challenge was obvious, take the RSS feed and put it up on a Google Map, hey its a mashup and we all know that they MUST use Google Maps :) For this next challenge I decided to abandon Yahoo Pipes for a while (mainly because it absolutely NAILS the server if you are looping through a big blog feed, and do it in the old "traditional" way of processing data in the single lump it comes in (the RSS feed).

My task was as follows

• Put the Geo tagged feed onto a map with the Google Maps API
  
• Generate a KML file for Google Earth
• Use Ajax in someway

I then added in another one (why not) which was to use the Google Maps API to actually do a search on those places I hadn't managed to rip from Wikipedia. Interestingly Google Maps don't enable you to search on the UK so having the Wikipedia information actually helped there.

So anyway after a few days of "downtime" working and lots of cursing about javascript I managed to get it all up and running. The Map your blog page needs an RSS or Atom feed that consists of posts that have (at this stage) just the placename in the title. It will then (after a shortish wait) display the Map and the summary of the trip information (distance between blog items) and the carbon footprint of that based on the mode of travel (again a single tag for the transport type at this stage). The final challenge I set myself was that all of the display elements would be done in a single page, no round-trip to the server.... no real reason I just wanted to try it.

To get you going try this simple UK to Wengen trip or there is a much messier one that goes all over the globe, that later one takes quite a while for Google to render. If you don't see it after 30 seconds just hit the button, I haven't done a "waiting for Google" bit yet, if you get bored just hit the button again.... Over on the right are a bunch of links including one to the KMZ file which should pop up directly in Google Earth (if you haven't got it download it).

I have to say it was really rather easy, apart from the Javascript issues. I used the Yahoo Javascript libraries to do the initially minor bit of Ajax in there, and Google Maps is fine. The paticular piece I liked was turning a KML file into a KMZ file in Java, worth a quick post on its own because of how easy it was.

Now I feel I can safely attack powerpoint for another couple of weeks.

Technorati Tags: SOA, Service Architecture

The Blog based, RSS using, Yahoo pipe executed Carbon footprint calculator

I've had an idea for a while which started when I did the Geo ripping Wikipedia thing that created a web service. The idea was to have an easy way for me to calculate my carbon footprint, not the one for the house and those things but one that actually took into account how much I travel.

I also set myself another task and that was to achieve this end using communication technologies I hadn't used in that way before. So this ruled out using WS, and led me straight towards the world of POX... or is it REST, I'm not sure I can tell :)

Now taking my own advice and separating "Service" from invocation using a Facade to enable WS and REST on the same code base I quickly knocked up a servlet that takes in a place name (or a geo coord) and returns it as an RSS encoded entry. It also takes in a transport type (more of that later) which is basically to enable pass through. The Place search Pipe uses the same backend service as the WS service, with information ripped from Wikipedia.

Next up is to then have a source of information that will make this all useful. Welcome to Blogger as a data store: My trip blog is just test data (except the last destination for which I leave today). This gave me an input RSS feed that could then be consumed by another pipe, the next pipe is a bit more complex. Firstly it takes in a feed and then orders by date (the opposite way around to most feeds) and then it uses that information to call the first pipe to obtain a geo-tagged set of information. This Geotagging Pipe turns the blog feed into a set of points.

The final pipe in this stage is then one that is meant for cloning by other people if they want to do the same (thinking of the consumer's view) that one can be found here (NB you need to look at the RSS to get the real info).

Stage two of this exercise was then to have something that would take that RSS feed and do all the calculations. Again a quick bit of development followed by fronting using Rome and the next stage was ready. Using the JCoord library to do the distance calculations and taking the carbon numbers from The National Energy Foundation the next servlet was soon deployed. So back into Yahoo Pipes it was, this time taking the URI of the generated feed which is then passed to the new servlet which then returns its resuts as another RSS feed. The carbon calculation Pipe takes an input (as text because Yahoo pipes kept trying to be clever when I used a URL builder) of the Pipe generated RSS feed and then dumps it to the servlet. A Consumer focused (clonable) implementation for my feed is then available here.

Okay so there you have it a few hours of working and its all working, taking information geo ripped from Wikipedia, a blog feed and a bunch of servlets all done using RSS.

Now there is one slight issue here. I've got the carbon calculations for "train", "plane" and "car" but for some reason when you debug the feeds it shows the categories correctly but the deployed pipe is devoid of any category information, so at the moment this means all journeys are calculated based on flying. Other issues are that the database isn't complete (I need to update my regexp stuff to catch some more entries) and the search is currently "first" rather than "best" so London = London borough of Croydon.

Next step (obviously) is to take that feed and turn it into a GoogleMap.

Now one question I have is... did I "do" REST?

If I have time at the station I might lob up a picture of how it all works.

Technorati Tags: SOA, Yahoo Pipes, RSS

Geo-ripping Wikipedia

As part of my on going quest to stop my drift into senility and powerpoint (the difference is marginal) and make sure that when I recommend things to clients that they actually work I went in search of some Web Services to play with. Now there used to be a useful bunch over a Capescience, now they've just got the Global Weather one which is okay, but I could do with more than one (and I hate stock quote examples). I also wanted to see what could be done to get some interesting information out of wikipedia, so I hatched a plan.

The idea was to create a very simple Web Service which took the Wikipedia Backup file and then extracted from it the geolocation information that now exists on lots of the pages.

Stage one was doing the georip, this was very simple. I elected to use a StAX parser (the reference implementation infact) as the file is excessively large. Using StAX was certainly quick (it takes Windows longer to copy the file somewhere than it does StAX to do the parse) but there are a few odd elements in using it (which I'll probably put a separate post on). That gave me a simple database schema (created using Hibernate 3 as an EJB 3 implementation and using inheritance in the objects mapped down onto the tables, very nice implementation by the Hibernate folks).

Next up was the bit that I thought should be simple, after all I had my domain model, I had my queries, I had even built some service facades, so how long could it possibly take to get this onto a server? The answer was far too long, particularly if you try and use Axis2 v1.0 (which I just couldn't get to work), switching to Axis 1.4 picked up the pace considerably, and thanks to some server coaching by Mr Hedges its now up and running.

There are around 70,000 geolocations that I managed to extract from Wikipedia. Some of these aren't accurate for several reasons
1) They just aren't accurate in Wikipedia
2) There were multiple co-ordinates in the page, so I just picked the first one
3) There are 8 formats that I identified, there could be more which would be parsed wrong.

So there it is, extracting 70,000 locations out of unstructured information and repurposing it via a Web Service for external access. Couple of notes

1) The Wikipedia -> DB translation is done offline as a batch
2) Name based queries are limited to ten returns
3) Any issues let me know
4) Don't rely on any information in this for anything serious.

The next stage is to get some Web Services up that run WS-Security and some other WS-* elements so there are more public testing services for people to use.

Technorati Tags: SOA, Wikipedia, geo, geo ripping, Web Service, Service Architecture