Killing a cloud - MandelMap

One of the things talked about with cloud computing is "horizontal scalability" which basically means being able to do lots of small things that are independent of each other. Now some tasks break down horizontally pretty well, others are a bit more of a chore but the challenge is normally how to re-combine those pieces into the overall set of results.

Horizontal scaling in the cloud also requires you to know when something is reaching its peak so you can add new servers. This could be done by the cloud provider, namely they see you are running out of space and they spin up another instance and ensure that it all runs across the same shared data store. This is fine, unless you are paying for those new instances and suddenly find yourself a few thousand dollars a minute down as a result of some coding problems.

Now the Google cloud forces you down this path even more as the last set of Performance metrics showed. Its focus is really short lived transactions which gave a big issue for the Mandelbrot set which is very computationally intensive. Even the little gadget on the right hand side is running at 1.3x quota.

So the solution is then to break the problem down further. Fortunately the Mandelbrot set is perfect for that as each individual point has a known value. So you can break the problem down as far as you want. I went for a series of 54x54 (the gadget is 200x200 so its about 1/13th the number of calculations) squares to create a bigger image. Then I got a little bit silly, thanks to GWT and decided to build a "Mandel Map" in other words use some of the infinite scrolling bits of Google Maps but applied to a Mandelbrot set.



Double click to zoom, shift and double click to zoom out. Drag to scroll around the map. Now the advantage of using the small bits becomes apparent, you can zoom in for miles and still remain within the quota set by Google but the actual effect is that you have a bigger image being displayed. Effectively here the browser is being used as a "late integration" approach to recombine the information at the point of display and act as the point of division for the problem into its smaller chunks. Now this is possible in part thanks to clearly parameterised URIs which can be used by clients to breakdown the problem. This puts a degree of control into the client but has the knock-on effect of more tightly coupling the two together.

Part of the reason for this is to see how Google copes with having a real horizontal hammering (the grid is 5x5x(5x5) so 625 requests for the basic image) and to provide a set of base data for the next challenge which is around how different types of caching can impact performance and how you actually need to design things with caching in mind. The test here is to see how well the standard load (that gadget at the side) copes while people are playing around with the map. This gives us an idea of what the peak scaling of a given image is and whether Google are doing something for us behind the scenes or not.

Yes its not perfect and it could be done in smaller chunks (etc etc) and maybe if I can be bothered I'll tune it. But its just to demonstrate a principle and took a whole couple of hours, if you want the full screen version, well here is MandelMap.

Technorati Tags: SOA, Service Architecture