Requirements Landfill - the challenge of central services

Enterprise Services of any type from Business Services like procurement through to technical infrastructure services such as Identity management are all at risk from a threat of pollution. I call this threat the requirements landfill.

One of the key principles of SOA in operation has been the ability to have services which are used across the enterprise. These single services represent a single point of truth and a single functional boundary that ensures enterprise consistency, it is their goal to shift away from fragmented sytems towards Services which represent the single way of accomplishing a technical or business objective.

This is great and good but I've been seeing something more and more over the past few years. Its that these enterprise class services, particuarly when successful, suffer from being the easy place to add new requirements rather than always being the right place to add those requirements. The basic problem is simple

These services sit at the junction between multiple elements, some are nice new service programmes, some are hacky service programmes, some are old legacy systems and others are just systems owned by people who don't have much budget or are rather grumpy.

So what happens is that you get a big and complex requirement. About 50% of this requirement really does belong in the central service but the other 50% is a combination of elements

1. 20% requirements that should be done in the originating systems
2. 20% requirements that are around data storage and access that should be in another system
3. 10% requirements related to business process that are completely beyond the scope of the central service

Now the problem is that to implement this new set of requirements therefore requires a much broader set of changes if it is to be done properly. Initially the discussions might start out talking about doing it the right way but then someone will say the immortal line

It would be easier if we just put all of this is the enterprise service

While the enterprise service team will scream at the prospect this will make perfect sense to everyone else as they won't be doing the work. Unless there is a set of concrete governance processes around then the normal "democratic" decision process will result in the compromise decision being made. The first time this gets done its seen as a compromise that just had to be made. The second time onwards its just replicating the same approach as that is what we normally do. Rapidly therefore the enterprise service goes from being a single clearly defined entity with a defined business purpose to being the requirements landfill for all those requirements that people are either too lazy or too scared to try and do properly. This leads to the service getting ever slower to react to change and eventually being seen as a failure due to its transformation from a business service into a traditional old style application.

The point here is that governance and escalation are essential to maintain a clear set of enterprise wide services and to ensure that requirements are not simply dumped into areas due to them having budget, talented people or just because they are at a central point.

Most of the time I see requirements landfills its because IT owns and manages the services and thus they have no direct link to a specific business owner. This means that business people don't care about keeping them clean as it isn't their problem. One of the first steps in a solution therefore is to ensure that your enterprise services are clearly aligned to the business areas where they matter and thus ensure that they have an in-built desire to keep the services aligned to their area and not blurred across technical and organisational boundaries.

Favourites for requirements landfills are also centrally provisioned IT "enabling" solutions such as ESBs which almost stand up to the business and say "dump crap here and then point the finger of blame here when it goes wrong".

So have you got a decent policy to prevent requirements dumping? Do you have a refactoring and recycling programme to take dumped requirements out of the landfill and put them back where they should be? Or are you just hoping that you'll be able to convince people that its not your fault that you allowed the illegal dumping?

Technorati Tags: SOA, Service Architecture

Data Accuracy isn't always important

Now while Amex really should have understood the term minimum there are examples where it really isn't an issue if someone gets it wrong in displaying the information to you. Sometimes this indicates that a prediction has been incorrect or that "approximately" is good enough for this scenario.
Is a good example of this, the current temperature on the Sydney Morning Herald site is listed as one degree higher than the maximum for the day. Does this matter? Well no and for two reasons. Firstly a weather forecast is accepted as being approximate information, its a chaotic system and so by definition can't be predicted exactly. Secondly the Max number is only a prediction and the current temperature is indicating that it was an incorrect prediction. So by having an incorrect piece of information we actually have more information as it re-enforces the concept that weather forecasts cannot be 100% accurate.
Now when the next day rolls around then looking back you should clearly be recording the actual maximum achieved rather than the prediction. This is because the information has gone from being a record of a prediction into a record of fact. The only question therefore is at what point you should update the maximum. Do you change it dynamically or on a daily basis when reporting historical information. For the Sydney Morning Herald site the answer is simple changing the daily maximum as it increases during the day would defeat the purpose of the "max" level which is what the paper predicted it would be at the start of the day. Its a free news story if it goes well beyond the prediction "Sydney Weather was bonza today with max temperatures 5 degrees higher than expected".

So the point is that when you look at data do think about what level of accuracy is important. If reporting a bank balance then spot on is the only option, if reporting the number of customers who bought cheese with wine as a percentage of overall cheese buyers then you can probably get away with 1 decimal place or less. This sort of view applies even more when looking at forecasting and other predictive data sets, the effort of increasing accuracy by 1% might be pointless due to the extra time it takes.

Data Accuracy isn't an absolute, be clear about what matters to you.

Technorati Tags: SOA, Service Architecture

When you know it isn't a cloud

Following up on the previous post I thought I'd do the REALLY obvious ones that indicate it isn't a cloud. James' list wasn't nearly cynical enough in light of the things that are claimed to be a cloud.
So here goes

1. If its just a single website with no backups storing stuff to disk then its just a standard WEBSITE not a cloud (Hello Sidekick)
2. If its a physical box that is meant to help you manage a virtual environment... then its not a cloud (hello Cloudburst appliance)
3. Seriously if you are just a single website doing stuff with a database you aren't a cloud (hello Soonr)
   
4. No seriously if about buying a physical box it isn't a cloud (like HP's spin though that they are just cloud enabling... nice weasel room)

And I could go on. The point is that cloud is an infrastructure thing, it is IaaS in the "aaS" hierarchy. PaaS can have a go at being cloud but SaaS is "just" something that might be deployed to a cloud. Having a website (SaaS solution) that runs on Amazon doesn't make that SaaS solution "a cloud" it makes it a SaaS solution hosted on a cloud.

The hardware point is that making capital expenditure is exactly what a cloud isn't about and physicality is exactly what a cloud isn't about. You want virtual compute and storage that you pay as a utility. This is the economic model of cloud.

So in the words of Kryton. I know that strictly speaking I've only identified two things, but they were such big things I thought I'd say them twice.

Technorati Tags: SOA, Service Architecture

How you know a phone is rubbish

My ever perceptive wife Heather spotted something the other day about mobile phone adverts. They basically come in two flavours

1. iPhone ads
2. The rest

The difference between them is simple here are the apple ones, first from when it launched

and an "there is an app for that" one



Now for the competition: Samsung

Microsoft

and (with the Omnia)

and Google's Android (via HTC)

See the difference? The Mrs did very quickly: In one set of adverts you see someone using the phone, in the others you appear to be watching a film trailer that is starring a phone. Her point was that the reason she likes her iPhone is what it is like to use and she won't switch to these other phones because you have no idea how they work just that they are in an advert with as much depth as a perfume ad.

Now I'm no advertising genius but I can't help thinking that this means one of two things

1. The other phones are a pain to use
2. The advertising agencies doing the other adverts are rubbish

Now it could of course be both of these things but my betting is on the first one. What appears odd therefore is that in the time since the iPhone has been released that no-one has come close and had the balls to actually advertise people using their phone.

The Windows 6.5 advert that is on TV right now has a bloke holding a phone while people in FOAM COSTUMES with windows Icons on them look sad and then become happy because he has a new Windows 6.5 phone... I mean how bad is that phone to use if you won't even show screenshots but just foam icons?

There are lots of reasons why the iPhone is doing well but top of the list is usability. Interestingly other mobile phone companies keep trying to compete on things like "better camera" or "better windows integration"(!) rather than on the feature that people actually want: USABILITY.

So how do you know a phone is rubbish? If an advert does everything in its power to NOT to show you someone using it.

Technorati Tags: SOA, Service Architecture

Not a cloud? Then what is it?

Redmonk are one of those smaller analyst companies who make up for a lack of numbers with a refreshing depth and honesty. James Governor's latest, and I assume light hearted, view of "15 Ways to Tell Its Not Cloud Computing" however does a bit of a disservice to the debate around clouds. Mostly right but with a few glaring pieces I felt I had to disagree with.

1. If you peel back the label and its says “Grid” or “OGSA” underneath… its not a cloud.
1. Fair enough if its talk about people selling last years technology with this years sticker but.....
   
2. If its a question of doing a deep dive and finding that underneath there is a "Grid" but that you don't care about it then I don't think this discounts it.
   
2. If you need to send a 40 page requirements document to the vendor then… it is not cloud.
1. I'll go with this one... with the caveat of governments can turn anything into 40 pages ;)
   
3. If you can’t buy it on your personal credit card… it is not a cloud
1. Nope I can't accept this. If I'm a fortune 500 company and I'm buying millions of dollars a month in dynamic capacity then I want a professional invoicing and billing approach. When governments build their own clouds they won't be billing to credit cards and for most companies this is an irrelevance.
   
4. If they are trying to sell you hardware… its not a cloud.
1. Absolutely with it
   
5. If there is no API… its not a cloud.
1. This is really to enable 3rd party tools integration and its a good thing. Fair enough
   
6. If you need to rearchitect your systems for it… Its not a cloud.
1. Very very wrong. For a simple reason, shifting boxes into the cloud and doing the same thing you've done before is easy. having a software application that can actually dynamically scale up and down and handle scalable data stores is harder.
   
2. To take best advantage of the cloud you need systems that can scale down and up very quickly, LOTS of systems today do not get the full value out of the cloud (as opposed to just virtual infrastructure) and will require re-architecting to take advantage of the cloud.
   
7. If it takes more than ten minutes to provision… its not a cloud.
1. Depends what we call provisioning. I've got 5TB of data to process that needs pre-loading into the database image. Does this count as provisioning as its going to take more than 10 minutes.
2. If it means 10 minutes to get a new compute instance for an existing system then fair enough but that isn't the same as provisioning a whole system in the cloud.
   
8. If you can’t deprovision in less than ten minutes… its not a cloud.
1. As an IT manager once told me "I can turn any system off in 5 seconds if I have to"... "just kick out the UPS and pull the plugs"
2. Fair enough point though in that it would at least be managed in a cloud.
   
9. If you know where the machines are… its not a cloud.
1. Really? So Amazon don't have a cloud as I know that some of my instances are in the EU?
2. If you mean "don't know exactly physically where a given compute instance is" then fair enough, but most companies don't even have a clue where their SAP systems are physically running.
3. Also against this one is the government cloud and security requirements. I need to know that a given instance is running in a secure environment in a specific country. This doesn't stop it being a cloud it just means that my non-functional requirements have geographical specifications in them.
   
10. If there is a consultant in the room… its not a cloud.
1. Cheap gag. You could add "if a vendor says it is... it is not a cloud"
   
11. If you need to specify the number of machines you want upfront… its not a cloud.
1. Fair enough
   
12. If it only runs one operating system… its not a cloud.
1. Why does this matter? Why can't I have a Linux cloud or a Windows cloud? Why is OS independence critical to a cloud?
   
13. If you can’t connect to it from your own machine… its not a cloud.
1. Non functionals (e.g. Government) might specify this. It depends what connection means. I could connect to the provisioning element without being able to connect to the running instance.
   
14. If you need to install software to use it… its not a cloud.
1. Server or client side? If its the later then I'd disagree, how will you use something like Amazon without installing a browser or the tools to construct an AMI?
   
2. If its the former.... I take it that it isn't the former
   
15. If you own all the hardware… its not a cloud.
1. Or you own the cloud and are selling it. What this would mean would that a mega-corp couldn't turn its current infrastructure into a cloud, and I don't see why they can't.
   
16. If it takes 20 slides to explain…. its not a cloud
1. Fair enough again. As long as this is the concepts rather than a code review!

So pretty much I agree with 50% and disagree with the remainder. The point is that cloud is still arbitrary and there are some fixed opinions. Utility pricing is clearly a given, but credit cards aren't (IMO) required.

One big way to tell its not a cloud is of course if you can see the flashing lights.

Technorati Tags: SOA, Service Architecture

Alternative Engineering

Dan Creswell pointed me towards an interesting blog on Cargo Cults and Computing on the same day I looked at this video on YouTube...

Then yesterday I was talking with someone who is helping dig out a project that has been driven into the ground by some people with very firm beliefs about how things should be done these were people who had taken pieces from agile, from waterfall, from TOGAF, from lots and lots of different places and combined it in their own approach.

This is alternative engineering in practice. The approaches were often contradictory, so they had a waterfall process but they didn't do full requirements up-front as the development would be agile. The Architecture wasn't complete and certainly didn't include the principles or non-functionals, but did include the hardware infrastructure that the solution should be deployed on.

When challenged on this it was described as taking "best practices" from lots of areas and combining it in a methodology which "best suited the company". This wasn't however like the start of a RUP project where you decide which RUP artefacts are required, it was just a complete and utter hack by people who wanted to do certain things, mainly the easy or interesting things, without doing the difficult or boring bits.

Around the industry I see Alternative Engineering practised a lot. Sometimes it evolves and is tested, made robust and clearly documented and becomes Engineering (SCRUM for instance), other times it remains vague in the vast majority of cases with people referring to a very limited set of successes and ignoring a huge amount of failures (step forwards XP).

The point is that Software Engineering does move on. The Spiral killed the Waterfall, at the stage that this happened then Waterfall became alternative engineering, in the same way as people thought that "nice smells" would ward off plague became demonstrably false so Waterfall for the vast majority of software development programmes became nonsense.

Iterative development then evolved from the Spiral and became undoubtedly the best way to deliver software. Agile added some dressing around the edges, some good some bad but the basic principle was still iterative development.

Alternative Engineering is what most IT organisations appear to practice. They don't go and look at what has been proven to work and they certainly don't accurately measure their own processes and ensure that they are working effectively. Their ability to learn is practically zero but their ability to have faith in what they are doing being the right way knows no bounds.

Cargo Cults are at least trying to copy people smarter than them, they are clearly practising alternative medicine but they are doing it by pretending to be proper doctors. Most people in IT are nothing more than quacks who don't, can't and won't prove they are working in better ways and hold on faith that their unique "blended" view and methodology which they are the only people on planet earth using is clearly the best way.

Sometimes these organisations can be broken out of the stupor and made to work in new ways which actually work. Its then always impressive watching the level of religious fervour that can develop as they turn on their previous ways and driving whole scale change. Unfortunately they then often apply this new way to places that it should not be used (Agile on Package implementations for instance). The Alternative Engineering practice then kicks in again and the cycle repeats.

The point is that if you aren't actually measuring your performance and understanding how a given requirement will be implemented and how over time your will reduce the time for that implementation then you are also practising Alternative Engineering. If you aren't looking robustly at your development and delivery processes and verifying that they work and that they are proven to work then you are practising Alternative Engineering.

Alternative Engineering is bunk and its part of the problems of Art v Engineering which also underpins the Cargo Cult phenomena. There are a vanishingly small number of people who can practice software as an Art. In my entire career and with everyone I've ever met its probably no more than 1% of people who can do that properly. The rest need to be doing proper Engineering based on the measured implementation of processes that are proven to work.

Surely in an industry that has above average scepticism for "alternative" therapies we should apply the same rigour to our day jobs?

Updated: Found a quote from Fred Brooks that just sums it all up

“I know of no field of engineering where people do less study of each other’s work, where they do less study of precedents, where they do less study of classical models. I think that that is a very dangerous thing”

The man is a genius.

American Express - Which number is greater?

Sometimes you just sit there and wonder about how the code managed to get to a certain end point... take my current Amex online bill...

Now it could be the surprise that its the lowest monthly bill I've had since I got the card, but I don't quite think that "surprise" is something that should be built into a financial system. But some how the amount that I have to pay (the last bill) is now less than the minimum payment, which includes stuff for which the bill hasn't been sent yet.

Part of this is because Amex have two billing elements, the first is the date that they want you to pay by, the 2nd is when the next bill comes out. If you pay before the later then everything is fine, but they'd prefer you to do it before the former.

This is clearly a historical thing with Amex and it clearly reflects back into their core operational systems which are almost certainly batch oriented. What this also means is that like many companies out there Amex haven't really adapted their systems or processes for the web they've just lobbed the paper processes on line which delivers oddities such as this which aren't possible in a paper only world.

When people put systems on-line they often seem to forget that the interactional model for on-line working is significantly different to off-line working. If you want customers to engage more in your on-line solution and move away from the more manual and higher cost channels then it really isn't good enough to shift crap processes onto the web, you should be looking at how customers will be interacting with your company in real-time and therefore what new processes and opportunities this can bring.

I did feel like phoning up the call-centre and asking "which minimum payment should I make, the one that says minimum payment or the one with the lowest value" but I decided my life was too short to waste time on that.

In praise of vendor shipped AMI's and virtual machines

Roman Stanek is one of those guys who consistently gets things right and his point around AMIs and Virtual Machines not being SaaS is absolutely 100% spot on. These aren't SaaS solutions they are PaaS solutions and they do indeed leave a huge amount of work for the developer to do. Part of the problem is of course that the name Software as a Service is of course wrong its really Service as a Service that you are buying.

So that said I do disagree with him that shipping AMIs is just the same as shipping DVDs for one very very big reason

AMIs and Virtual Machines must run

This for me is a big leap forward from vendors as while it still means that I have to build my application I don't have to spend days or even weeks trying to install software that quite clearly has never been tested from the DVD or downloads that they have on the site. Some wonderful pieces I had in the past include

• A software & hardware manufacturer whose software worked on competitors operating systems but not their own
• A software vendor whose instructions for connecting their two products had never been tested... I know this as it did not work at all
• Spending 2 weeks installing from DVDs and downloads from a vendor and eventually having them on-site trying to do it for themselves, failing and then "getting back to us" a week later with an installer than actually worked
• A single software vendor whose two products that were required to work together required two differently patched versions of the Java runtime
• A vendor whose installation DVD was missing some core jar files, which they denied at firt but the old "Class not found" exception was a bit of a give away

So I'd like to praise AMIs and vendor delivered virtual machines for the basic progress that at least now it is spectacularly hard for a vendor to ship a non-working image. It might be an ugly image with lots of hacky patches required, but at least they've had to do all that dirty work and not you.

So don't con yourself that you are doing SaaS, you aren't you are doing PaaS at best. But do rejoice in that vendors are at last forced to prove that the software runs before shipping to you. Having a vendor delivered AMI or virtual machine that doesn't run really would set a new low bar in the sorts of quality that they expect customers to put up with.

So I say hail this new move away from DVDs and towards images, because personally I'm sick and tired of debugging their installers.

Why do games need an operating system?

Using my iPhone and looking at a PS3 console in Selfridges made me think about what the future could hold for games. While there are companies looking at doing server side games and a VDI solution to the end device I just don't think that matches against Moore's Law. But equally the model of downloading and installing loads of games onto a single device with multiple different anti-piracy tools doesn't seem to make sense either.

Given that "bare-metal" concepts have been around for a while now, BEA had their bare-metal version of the application server, wouldn't it make sense for games to be a full single virtual image? So you don't boot the OS then start the game, you just boot the image which just contains the game and what ever minimal drivers it needs.

Now some people will point out "what about the drivers?" and there is a slight point there. But would it be really so hard to define an image where you select your graphics cards et al and it constructs a fully-optimised image just for you? Upgrade your kit and then you just upgrade the image.

Now others will bleat "but how will I pirate it if its a custom environment?" and to be honest I don't care, your problem.

What are the advantages? Well from a piracy perspective it clearly reduces the potential if the game is running on a custom micro-kernel and is tied against a specific license that is created when you create the image. From a performance perspective it can be tuned to the 9s as there is nothing else running. From an end-user perspective it means using your PC like a console, or indeed your console like a console, and selecting the game to play at boot up.

Once you start thinking like this of course it then becomes a question as to what other applications don't really require an OS and the concept of isolation via VMs. Windows 7 is doing this with its XP mode which really opens up the question as to when you don't need Windows for certain applications.

Virtualisation is a server technology that is going to have a massive impact on the desktop.