Why relative metrics matter in delivering success.

George Bush today gave one of those great lessons in why its really important to think in relative terms when talking about metrics for success in any programme.

cutting wasteful or bloated programmes that will save tax payers over $18bn

Now this sounds pretty good until you realise that this is against a total budget of $3trn. Yes folks $3trn... in other words this "saving" represents 0.6% of the total budget.

This is something I've found with a lot of technology change programmes. Numbers that sound really good just don't stand up to scrutiny when you do the maths. So you see numbers like spend $2m and we will increase productivity by 10%. This sounds great, until you look at the yearly spend and find that they spend $5m a year, which means the ROI on that $2m is FOUR years. The trouble is that the following year they ask for another $2m for "new" technology and the expected 10% still hasn't been delivered.

The other thing that I see are another Bush speciality, fake unmeasurable metrics. So the 10% is a good one in IT, because in most companies there isn't a real measure of actual productivity. People shout "agile" in the mistaken belief that this means you magically get better. Unless you are measuring success in an open, honest and above all accountable way then its just your opinion of the improvement against those whose opinion will be that its just the same or worse.

The point therefore on any change programme, and especially when your company might be facing more troubling economic times is that an SOA programme should have three clear sets of metrics

1. Financial Metrics - Relative to today how much more bang for the buck will you get and how will you prove that.
2. Delivery Metrics - Absolute measures of productivity (concept-to-market cost/time, function point cost/time, etc) and how they will improve
3. Operational Metrics - How will the operation of the system improve (non-functionals, cost to support, fix/resolve timescales)

Everyone of these should be expressible simply and automatically measurable, requiring at most a spreadsheet calculation.

Otherwise you are just doing a George Bush and talking up a tiny piece as if it matters in the grand scheme of things while avoiding all the really big problems and genuine accountability.

Technorati Tags: SOA, Service Architecture