Or maybe not.
Checking my results on usage I've come to the conclusion that a thread gets killed at around 9.5 seconds, so its on the deep calculation ones or a very big image. However this isn't the only quota that exists.
So to be clear this is the smallest mandelbrot set, its 200x200 and at this stage its a max of 16 iterations. This still means a potential for a lot of calculations but it really does indicate that Google are aiming at the data access end of the market rather than a commodity platform for doing heavy calculations (sort of a small problem competitor to IBM's Blue Gene.
In terms of what this means well on my MacBook Pro a "time python mandelbrot.py" request (which has to include starting python (about 0.2 seconds appears about right) which creates the "default" image takes the following
real 0m3.721s
user 0m3.436s
sys 0m0.097s
So with the python engine piece take off we have around 3.3 of user + sys and according to the logs this same request via the web is 9.4 times over quota. This means that the request quota is about 0.35 seconds of raw grunt but it will let you hang around in the data access for up to that 9.5 seconds.
Now right now Google are playing very nice with me given that pretty much every request passes or smashes the quota. Some go a LONG way past
The next zoom in then failed (at a recorded 31.7 times over quota! but around 9.5 seconds total time).
So fair play on Google for letting me abuse their infrastructure and it will be interesting to see what they do around CPU heavy tasks especially those that could go horizontally. I could see a real market for companies who effectively want 10,000 CPUs for a short period of time to calculate a forecast or similar where right now the cost/benefit analysis does stack up for a full hardware purchase but buying a whole load of Gigacycles (horizontally scaled) would be a great fit.
Technorati Tags: SOA, Service Architecture
3 comments:
Not to state the obvious, but have you looked at Amazon EC2 for these kinds of calculations? It is, in fact, EC2's fastest growning market--namely financials, pharmas and scitechs using EC2 to run compute intensive jobs--at a fraction of the cost of owning the infrastructure themselves.
Google AppEngine is meant for one thing and one thing only: building and hosting short transaction web applications (such as Web 2.0 apps). The stuff you are trying is explicitly outside of their target market, thus the quota issues you are quickly finding.
This is a simple case of the right tool for the job.
James,
The issue with EC2 isn't that it won't do the job its that I'd have to pay for it to do the job and a couple of large renders could really rack up the numbers!
At this stage you are right, EC2 is a much stronger distributed compute platform, my question and tracking is more about where Google might go with this rather than where they are today.
I'd say it would be brave to say that App Engine will never move beyond short transaction web stuff.
This confirms what I found (it looked like approx 9 seconds for me before being killed). But I didn't try to separate the CPU time from the wall clock time, as you did. Thanks for the data.
I wrote a quick prototype of a way to get more quota by chaining requests (with a redirect) that you may be able to make use of if you're interested:
Some breathing room for Google App Engine requests (http://stage.vambenepe.com/archives/213)
Post a Comment