Monday, June 30, 2008

Google App Engine performance - Part 3

Okay so the last piece was just when does it cut off in pure times perspective?

Shows the various daily peaks and having done a bit more detailed testing the longest has been 9.27 seconds with several going above the 9 seconds mark. These are all massively over the CPU limit but it appears that the only real element that gets culled is the raw time. Doing some more work around the database code at the moment and it appears that long queries there are also a pretty big issue, especially when using an iterator. The best bet is to do a fetch on the results and then use those results to form the next query rather than moving along the offsets, in other words if you are ordering by date (newest to oldest) then do a fetch(20) and take the time of the last element in the results and on the next query say "date>last.date". Fetch is certainly your friend in these scenarios.

So what does this mean? Well Google aren't culling at the CPU limit straight away but are consistent around the time limit, the performance doesn't have peaks and troughs through the day and there doesn't seem to be any swapping out of CPU intensive tasks. All in all its a solid base.

Finally however I just had to lob on something from Google Spreadsheets that brings how the sort of thing that people can do when they have access to real time data and decent programming frameworks.

This just shows the progression of the "default" render over time, if you go by "average" then it will show you the stability that occurs and if you go by count then it shows the amount of calculations that all of these stats have been gleaned form and will help you think whether its a decent enough set of data to draw conclusions from.


Technorati Tags: ,

Thursday, June 26, 2008

Temporalisation or why location is important

The folks at ZapThink have written a report on what they think the "next big thing" will be and they call it location independence and being about the merger of the Web/SOA and mobile. As so often with bold predictions in IT... Bill Joy got there first... almost 12 years ago in fact. In 2006 he presented at MIT on "Six Webs: 10 years on" in particular he coined a term the "here" web which is all about the web that you carry around with you and via which you interact when you are on the move.

One of the bits that I think that doesn't get enough focus right now is temporalization, a sort of made up term that I take to mean "giving something a relevance in time and space". Most applications today are "location independent" and "location unaware" and the closest they get to understanding anything about time is "is it Tuesday" or lobbing something on a log-file.

If I was going to say what the next big thing would be I think it would be much more about applications becoming temporalized than it will be about them becoming location independent. If I'm a shop-owner with special offers, why would I want to put the application anywhere? What I want to do is deploy it to the cloud and have the cloud determine where to put it (for instance in the nearest mobile cell) so its then available as a service only in that location... in other words its only available "here".

This basically means that rather than applications becoming large and learning everything about geo-data like trying to work out where all the war criminals in Washington DC (A wonderful example of how searching can produce comedy) and having every application know all the other option is to go small. So a small application (e.g. an ad server) could be configured and deployed to a local market. This means that when you search for "shops" you have not only the traditional centralised approach but also the ability to "geo search" by which I mean make a query for services available at a given location. This federated model enables greater targeting but has issues in terms of service comprehension that would need to be addressed.

The final piece in the puzzle is then time. "Here" Services would need to know not only where "here" is but they would also need to know when "here" is. I don't want to be told about a show that finished yesterday or about one that is six months away. If its 11pm tell me about the Kebab shop, if its 9am tell me where to get Asprin.

So I think that Zapthink are on to something from an infrastructure perspective but I think that location and time awareness and in particular the federation of services to the "here" could really be the next big thing. Of course what I really want to know is what Bill Joy thinks today is the next big thing so I can understand what I'll be doing in 2020!




Technorati Tags: ,

Wednesday, June 25, 2008

Google App Engine performance - Part 2

So the first analysis was to look at the gadget performance with 40,000 pixels which gives a fair old number of calculations (its 16 iterations for those that want to know). My next consideration was what would happen to a larger image that was further over the threshold. Would that see more issues?



Again its that cliff (I need to go and look at the code history) but again its remarkably stable after that point.


I know I shouldn't be surprised but this is several times over the CPU Quota limit (about 5 times in fact) so I was expecting to see a bit more variation as it caned the processor.



Now this shows just how consistent the processing is. Its important to note here that this isn't what Google App Engine is being pitched at right now. Given they've pitched it at data read intensive apps I'm impressed at just how level the capacity is. Having a 2 x standard deviation that is sitting around +/- 2% and even the "exceptional" items only bumping up around the 5% mark is indication either of a load of spare capacity and therefore not much contention or some very clever loading.

The penultimate bit I wanted to see was whether the four fold increase in calculations resulted in a linear increase in time.



What this graph shows is the raw performance and then the weighting (i.e. blog performance divided by 4). Zooming in on comparing the blog (160000 pixel) weighted against the straight off 40000 pixel gadget we get

Which very impressively means that there is a slight (that really is a fraction of 1% not 0.5 = 50%) performance gain through doing more calculations. Its not enough to be significant but it is enough to say that the performance is pretty linear even several times above the performance quota. The standard deviations are also pretty much in line which indicates a decent amount of stability at this level.

So while this isn't a linear scalability test in terms of horizontal scaling it does indicate that you are pretty much bound to 1 CPU and I'm not seeing much in the way of swapping out (you'd expect the Max/Min and the std dev on the larger one to be higher if swapping was a problem). So either Google have a massive bit of spare capacity or are doing clever scheduling with what they have.

The final question is what is the cut off point....

Technorati Tags: ,

Google App Engine performance - Part 1

Okay its been a few weeks now and I thought I'd take a look at the performance and how spiky it was. The highest hit element was the gadget with a few thousand hits so I took the data from the "default" image (-2,-1), (1,1) and looked at the performance over time.



Are all of the results over time. I'm not quite sure if that cliff is a code optimisation or is Google doing a step-fold increase but the noticeable element is that after the spike its pretty much level. Delving into the results further the question was whether there was a notable peak demand time on the platform which impacted the application.



The answer here looks like its all over the place on the hours, but the reality is that its the first few days that are the issue and in reality the performance is within a fairly impressive tolerance

Beyond the cliff therefore the standard deviation and indeed the entire range is remarkably constrained within about +/- 5% for min/max of average and two standard devs being +/- 2% of the average. These performances are the ones that within 1.2 times of the Google App Engine Quota Limit in terms of CPU so you wouldn't expect to see much throttling. The next question therefore is what happens under higher load...

Technorati Tags: ,

Wednesday, June 18, 2008

Google App Engine - Quota limits

With my first Google App Engine application I deliberately decided to pick on something that is CPU heavy but which when threading is supported will go horizontally to help see how "wide" the cloud is.

Or maybe not.

Checking my results on usage I've come to the conclusion that a thread gets killed at around 9.5 seconds, so its on the deep calculation ones or a very big image. However this isn't the only quota that exists.


So to be clear this is the smallest mandelbrot set, its 200x200 and at this stage its a max of 16 iterations. This still means a potential for a lot of calculations but it really does indicate that Google are aiming at the data access end of the market rather than a commodity platform for doing heavy calculations (sort of a small problem competitor to IBM's Blue Gene.

In terms of what this means well on my MacBook Pro a "time python mandelbrot.py" request (which has to include starting python (about 0.2 seconds appears about right) which creates the "default" image takes the following

real 0m3.721s
user 0m3.436s
sys 0m0.097s

So with the python engine piece take off we have around 3.3 of user + sys and according to the logs this same request via the web is 9.4 times over quota. This means that the request quota is about 0.35 seconds of raw grunt but it will let you hang around in the data access for up to that 9.5 seconds.


Now right now Google are playing very nice with me given that pretty much every request passes or smashes the quota. Some go a LONG way past


The next zoom in then failed (at a recorded 31.7 times over quota! but around 9.5 seconds total time).

So fair play on Google for letting me abuse their infrastructure and it will be interesting to see what they do around CPU heavy tasks especially those that could go horizontally. I could see a real market for companies who effectively want 10,000 CPUs for a short period of time to calculate a forecast or similar where right now the cost/benefit analysis does stack up for a full hardware purchase but buying a whole load of Gigacycles (horizontally scaled) would be a great fit.



Technorati Tags: ,

Amazon's cloud goes dull

Now you can get JBoss on EC2 cloud computing is clearly trying to ditch the sexy image and move into the old dull world of profitable IT. Having an OS on a cloud wasn't a big deal as after all its not the OS that people are after its the apps. This however will be interesting particularly if Red Hat make the investment to develop effective dynamic clustering and scaling (up and down) approaches which can take advantage of a cloud environment.

With Google playing with Python and Salesforce wanting you to use their own proprietary dev environment it appears that Amazon are most clearly going for the traditional data centre. With Google (as it is now) and Salesforce you have to make a decision that the cloud is the thing and then develop for that given cloud environment. With this move Amazon & JBoss have basically announced a strategy where you can take your existing applications and move them into a cloud environment.

The operating system play was mildly interesting but saying to a CIO "move your Linux OS onto a could" is a sort of pointless play, saying "shift your Java App onto the cloud" make much more sense and critically doesn't sound as scary. After all you deploy the app remotely anyway so why not just deploy to a cloud.

This won't be for everyone but it does mean that cloud computing now has a data centre offer that will make it very interesting for people in the SMB hosting market.

Technorati Tags: ,

Tuesday, June 17, 2008

Web 2.0 Administration - let the users help

People might not see National Rail (The UK body responsible for the track and other infrastructure stuff on the railways) as at the forefront of Web 2.0 trends. But this morning their Website really pushed new barriers of Web 2.0 collaborative working. Not happy with users collaborating around ideas and content they moved to allow users to actually administer the site themselves.


Unfortunately as is often the way they forgot to tell everyone the username/password combination but I'm sure that was just a minor oversight in the move towards true Web 2.0 administration ;)

Technorati Tags: ,

Wednesday, June 11, 2008

Learning Python

Okay so a week or go I spent about 12 hours in learning Python and the Google App Engine. I decided to break down the time that I spent doing various tasks: design, research, coding, debugging, deploying, testing.

First off I decided not to build a unit test set-up as we are talking about writing a single class (to do the mandelbrot) and the web handlers, in all its just under 5,000 lines of written code and comments. So how did I get on? The timings are of course approximate, all I did was note minutes of time spent


So what did this tell me? Well first off the things I like

Python has a Complex number type that made the mandelbrot code easier
The Templating engine that Google have included is easy to use

Otherwise this was a poor language learning experience for me. Sure there wasn't the complexity and raw anger that C seemed to demonstrate when I had 7 days to justify claiming in a stage 1 interview that I knew C. Python just seemed to have a lazy disdain for what I was doing. I was using the SPE editor and quite frankly it was less advanced than Emacs at giving decent language support. It doesn't do any checking of the code you are writing against the variables and functions that exist and doesn't do autocomplete on functions unless you type it in yourself. My person favourite was something called PyCheck which I assumes was meant to help me improve my code.... nice idea but probably a better one if it didn't have an error in it


Traceback (most recent call last):
File "/Applications/SPE.app/Contents/Resources/_spe/plugins/Pycheck.py", line 121, in OnProcessEnded
del self.list[0]
IndexError: list assignment index out of range

I'm sure there must be better editors out there, but frankly SPE was like coding in VI but without the performance advantages.

The common errors early on were around scoping. Having to do self."variable" was a bit unusual in an OO language but I soon got used to it. The truly awful thing however about Python is that it is an indentation sensitive language, which was a trick employed in the 1970s to make compiler writing easier and something that people came to dread with makefiles.

Thanks to a combination of the language and an editor that didn't help overcoming the language deficiencies the cycle was "write, run, check error log" the difference between this and the Java "write, auto-compile" and the "write, auto-highlight-error" approaches that I've been used to felt like a leap backwards into the land of C.

It wouldn't be fair for me to write a similar test using Java as I've been doing that for over ten years but what I would say is that after 7 days of learning C I then had 5 hours in which to write a fairly complex address book application. My first C++ programme was a distributed CORBA system and my first Java programme was an interactive VRML app. Python's productivity I would rate as worse than C++ as a development cycle and only above C++ overall due to its prevention of segfaults via a fairly reasonable exception and pointer strategy.

Before people leap forwards, the libraries are the things that make Python better for C++ for the web. The templating library Django was a better development environment that Python itself, I especially like they way they've limited it to only those things that make sense in a templating language thus making sure that you split out the view bits and the code bits. Even down to using "end" markers for loops and ifs because it can't rely on indentation just made it feel better and more stable than the programming language. This is certainly the first time I've ever felt that the templating environment was better than the coding one.

Python is a toy language. Its quite good to knock things up with and probably rewards people who are less structured in their approach. For myself however it was certainly not a productivity gain and I became thoroughly annoyed at a sequence of typo errors wasting minutes of time. Some people will leap up and say "well you can see the changes straight away" because as a scriptoid language Python is "dynamic" and the code changes are "automatically" applied. Errr but I do that in Java as well via an ant script so the fact that I don't actually have to run a bloody test to check a syntax or semantic error is a bit of an advantage.

Its really hard to see what the fanfare is around Python, its a scripting language, it uses indentation to indicate containment. With the largest proportion of my time spent in a break-fix cycle, most of which a decent compiler would have caught, this really doesn't scream to me a language of high productivity.

A better editor might have made the experience better and more effective, so any recommendations would be appreciated. I'm still playing with the App Engine so I can revise my thoughts.

Technorati Tags: ,

Monday, June 02, 2008

Google Apps Engine - is cloud working for space cadets?

As part of my on going quest to not become totally subsumed in Powerpoint I decided to have a crack at Google Apps Engine which is the Google bit of cloud computing. This first release is pretty limited in that there is no multi-threading allowed (a bit of a bugger for async) and you have to use Python. What I was interested in however was the efficiency of the Google Engine so I thought I'd come up with a test that works today but that will scale to multi-threading if it becomes available. This did mean I had to learn how to programme in Python however and would once again put my dislike of "dynamic" aka "code in haste repent at leisure" languages.

When learning new languages there are a couple of things that I do and for this one I went right back to my earliest coding experiences....

The above works just by clicking on where you want to zoom in and then clicking on "zoom out" well that zooms out. I've also added a gadget onto the sidebar.

The reason for picking the Mandelbrot generator is that its pretty simple code, which makes it good for a new language, but it also stresses out the processor a bit with the maths. The basic set up is that there are two Google App Engines, the first one does the calculation and then does a "POST" to the second with the results this gives the obvious 2nd stage which is to start profiling the results to see how the App Engine performs.

Some things I've found out so far. The basic image generator can generate images of "gadget", "blog", "medium" and "large" sizes. But the time taken for Google App Engine to generate the large image is too long so it times out every time I've tried it. This suggests to me that Google's time enforcement on requests is purely at the in/out level rather than being around CPU load which does mean that at times of higher load on the Engine cloud you will see more rejected requests.

The next thing I've found out is that in raw time terms (obviously I can't do "time" on the App Engine) it returns about 30% faster than my MacBook Pro which isn't too bad but certainly seems to indicate its indeed standard CPUs lurking under the covers rather than anything super special, Mandelbrot is quite a good test here as its mainly CPU bound with a bit of memory a faster CPU, or the ability to multi-thread, makes a big difference.

The next piece I've found out is that the offline version and the online version do not appear to work the same. Right now the reason why I'm not giving public access to the results is that I have two different problems being reported between the offline version and the online version. This is a real pain in the arse as it means debugging is effectively a game of Russian roulette. I tracked down the problem and it was a user error in storing and retrieving information, but the reported errors were not very helpful and were different between offline and online.

So the 2nd bit is the information reporting and I've decided to further my REST experience, hell you can't slag stuff off until you've really seen how little it offers ;) This gives the results. In true REST style it currently does bugger all but I will add new features over time as I learn more stuff to do. I'm using Atom and the standard Google output from the data model rather than writing any fancy XML stuff myself at this stage. The server URI takes a parameter of "server" which is the server name (e.g. localhost:80 for my local stuff and georipper.appspot.com for the cloud stuff)

The timings are taken from the request to the point where the image is created. To create the image I needed an all python image library and the rest is pretty standard algorithm stuff.

So what I've found out so far in the approximately 12 hours of playing with Google Apps Engine is that it works like any remote server. Power wise its nothing special and the lack of multi-threading kills it for lots of async tasks. I can see why Google are doing this but a limited threadpool or even green threads would be nice. The kill threshold also doesn't seem to be very high and once I've got some raw info I'll start doing analysis

Its not for space cadets therefore and it isn't (IMO) something that lends itself to overly complex tasks its sweetspot (and ironically the bit I had most problems with) appears to be around data access and retrieval and also acting as a platform for all those weird social apps that explode and die over the course of a month. In other words its part of that further commoditisation of technology and the on-going reduction in costs for basic web applications.

I'll write a bit more in detail about my experiences with Python and the Engine, but the first bit was to start getting some data. It appears that response times are pretty time dependent, sometimes even a single level click on the top level image doesn't render in time.

Technorati Tags: , ,