When successful systems go bad - boiling frogs with Technical Debt

One of the challenges I often see in companies is when successful systems go bad. These aren't the systems that were delivered 3 times over time and 5 times the budget, these are the systems that many years ago delivered real benefits for the business and delivered in a reasonable time and budget.

The problem is that all those years ago the team in question was focused absolutely on getting the system live and successful, and like teams often do they cut corners. This started the technological debt of the system and began to act as a drag on future projects from day 1. Thanks however to the talent of that original team and the abject failure of systems elsewhere the success was rightly lauded and the system held as a shining jewel.

Roll forwards a couple of years and the situation as evolved. Those smart developers are now managers and some of them have left the company all together. The team profile has changed and the odds are the talent pool has decreased. Those pieces that the first team had missed out: metrics, unit tests, documentation are starting to be felt, but the current team would struggle to justify the cost to put them in and the rate of progress, while slowing, still delivers in a reasonable time frame. The level of debt is increasing and some of those short cuts in the first build are becoming evident and the newer short cuts to keep things on track are getting ever more desperate. Cut/Paste/Modify is probably becoming a normal strategy at this stage but people are building successful careers, quite rightly, based on their previous success.

Roll forwards five or more years and the situation has evolved again. The managers are becoming directors, more disconnected from the actual technology but still aware of what it represented to them all those years ago. The people working on it now have little connection to the original vision and are in a full on duct-tape mode. The pace of change has slowed to a crawl, the cost of change has gone through the roof and the ability to actually innovate has all bit disappeared. The problem is that this has happened slowly and over an extended period of time so people are still thinking that they have a wonderfully flexible system.

Its pretty much like boiling a frog, no-one has noticed that the pace has slowed and that the major cost of all projects is from the technological debt of the system. Some retrofitting efforts have been made no doubt, and these are lauded as being important, but the fundamental challenge is that the code base now is hugely fragmented from a management perspective while maintaining points of critical instability. Testing and releases take an age because changing one thing breaks five others, all of which are then just patched.

I'm not talking here about back-end transactional systems in which change is an irregular thing, who cares if the COBOL is hard to maintain if last year you only modified 5 lines, I'm talking about dynamic systems that are meant to be flexible and agile and have suffered greatly under 5 or more years of continual development of variable quality.

The challenge is that senior IT managers in these types of companies often are wedded emotionally to the system and can create elaborate arguments which are based around their perception of the system when it was built and their desire to see what was a great system continue to be a focus for the business. Arguments that off the shelf components now offer a better and more flexible approach or that starting again would be cheaper than next years development budget to add 5 features will just fall on deaf ears.

But its something that people in IT must do as a normal part of their business, to step back and realise that 5 or 10 years is a huge period of time in IT and that there will have been significant changes in the business and IT market during that time which will change your previous assumptions. The right answer might be to continue on the current path and to invest to a level that removes the technical debt, it might be to do a full rebuild in a new technology or even in the same technology just based on the learnings from the last 5 years, or it might be that what was differentiating before has now become a commodity and you can replace it with a standardised package (and make the business change from differentiating to standardising with its additional cost benefits).

So have an honest look at those successful systems and ask yourself the question Is the frog boiling? be honest, think about the alternatives, and above all think about what the right economic model would be for that system. This last bit is important, differentiating systems are about top line growth, about Capital investment (CapEx) and looking at the business ROI. Standardised systems are about cost saving in IT and the business and measuring by the Cost to Serve (OpEx).

Here is the check list to see if you have a boiling frog

1. Yearly development spend is higher than the initial development estimate
2. Many people in IT have a significant emotional attachment, but don't code
3. "differentiation" and "flexibility" are the watch words
4. The code quality metrics are in the scary territory (or aren't collected)
   
5. There have been several large "refresh" projects through the code base
6. The competition seems to be getting new features to market quicker
7. The pace of change on other systems is limited by the pace of change of the boiling frog
8. Each generation of buzzwords is applied to the solution, but no major refactoring has been done

I'm sure there are others but another one is simple

If the business gave you the development budget for the next two years and asked you if you could rebuild the site with a couple of friends for that amount would you

1. Say "god no its way to complicated and high quality"
2. Say "No way, it would cost a bit more than that"
3. Say "Ummm I think so"
4. Say "definitely, when do I start?"
5. Bite their hand off at the elbow and give them an order form

If you are scoring 3 or more then you've probably got a boiling frog.

Technorati Tags: SOA, Service Architecture