When successful systems go bad - boiling frogs with Technical Debt

One of the challenges I often see in companies is when successful systems go bad. These aren't the systems that were delivered 3 times over time and 5 times the budget, these are the systems that many years ago delivered real benefits for the business and delivered in a reasonable time and budget.

The problem is that all those years ago the team in question was focused absolutely on getting the system live and successful, and like teams often do they cut corners. This started the technological debt of the system and began to act as a drag on future projects from day 1. Thanks however to the talent of that original team and the abject failure of systems elsewhere the success was rightly lauded and the system held as a shining jewel.

Roll forwards a couple of years and the situation as evolved. Those smart developers are now managers and some of them have left the company all together. The team profile has changed and the odds are the talent pool has decreased. Those pieces that the first team had missed out: metrics, unit tests, documentation are starting to be felt, but the current team would struggle to justify the cost to put them in and the rate of progress, while slowing, still delivers in a reasonable time frame. The level of debt is increasing and some of those short cuts in the first build are becoming evident and the newer short cuts to keep things on track are getting ever more desperate. Cut/Paste/Modify is probably becoming a normal strategy at this stage but people are building successful careers, quite rightly, based on their previous success.

Roll forwards five or more years and the situation has evolved again. The managers are becoming directors, more disconnected from the actual technology but still aware of what it represented to them all those years ago. The people working on it now have little connection to the original vision and are in a full on duct-tape mode. The pace of change has slowed to a crawl, the cost of change has gone through the roof and the ability to actually innovate has all bit disappeared. The problem is that this has happened slowly and over an extended period of time so people are still thinking that they have a wonderfully flexible system.

Its pretty much like boiling a frog, no-one has noticed that the pace has slowed and that the major cost of all projects is from the technological debt of the system. Some retrofitting efforts have been made no doubt, and these are lauded as being important, but the fundamental challenge is that the code base now is hugely fragmented from a management perspective while maintaining points of critical instability. Testing and releases take an age because changing one thing breaks five others, all of which are then just patched.

I'm not talking here about back-end transactional systems in which change is an irregular thing, who cares if the COBOL is hard to maintain if last year you only modified 5 lines, I'm talking about dynamic systems that are meant to be flexible and agile and have suffered greatly under 5 or more years of continual development of variable quality.

The challenge is that senior IT managers in these types of companies often are wedded emotionally to the system and can create elaborate arguments which are based around their perception of the system when it was built and their desire to see what was a great system continue to be a focus for the business. Arguments that off the shelf components now offer a better and more flexible approach or that starting again would be cheaper than next years development budget to add 5 features will just fall on deaf ears.

But its something that people in IT must do as a normal part of their business, to step back and realise that 5 or 10 years is a huge period of time in IT and that there will have been significant changes in the business and IT market during that time which will change your previous assumptions. The right answer might be to continue on the current path and to invest to a level that removes the technical debt, it might be to do a full rebuild in a new technology or even in the same technology just based on the learnings from the last 5 years, or it might be that what was differentiating before has now become a commodity and you can replace it with a standardised package (and make the business change from differentiating to standardising with its additional cost benefits).

So have an honest look at those successful systems and ask yourself the question Is the frog boiling? be honest, think about the alternatives, and above all think about what the right economic model would be for that system. This last bit is important, differentiating systems are about top line growth, about Capital investment (CapEx) and looking at the business ROI. Standardised systems are about cost saving in IT and the business and measuring by the Cost to Serve (OpEx).

Here is the check list to see if you have a boiling frog

1. Yearly development spend is higher than the initial development estimate
2. Many people in IT have a significant emotional attachment, but don't code
3. "differentiation" and "flexibility" are the watch words
4. The code quality metrics are in the scary territory (or aren't collected)
   
5. There have been several large "refresh" projects through the code base
6. The competition seems to be getting new features to market quicker
7. The pace of change on other systems is limited by the pace of change of the boiling frog
8. Each generation of buzzwords is applied to the solution, but no major refactoring has been done

I'm sure there are others but another one is simple

If the business gave you the development budget for the next two years and asked you if you could rebuild the site with a couple of friends for that amount would you

1. Say "god no its way to complicated and high quality"
2. Say "No way, it would cost a bit more than that"
3. Say "Ummm I think so"
4. Say "definitely, when do I start?"
5. Bite their hand off at the elbow and give them an order form

If you are scoring 3 or more then you've probably got a boiling frog.

Technorati Tags: SOA, Service Architecture

Religion as a Service

Okay a few weeks ago I had a brain-wave for a new business. What do you really need for a business to take off in the "as a Service" space?

1. It needs to be 80%+ commodity
2. You need a large customer base
3. You need the end customer to add their own differentiation

Above all I wanted a business that would be a much higher margin one than simply a SaaS business, which meant including the actual business process work as well.

Hence the idea for "Religion as a Service". Focusing initially on the Abrahamic religions (where lets face it around 80% of the rules and processes are the same) this would expand to enable a more active selection of deity (or alien race) and a greater degree of customisation around the rules of your newly created faith.

Religion as a Service goes far beyond simply providing you with a customised version of your "one single book that has all the truth in it" to actually providing you with a modern industrialised solution to your religion's needs. This includes

1. A multi-language call centre to handle donations
2. A set of "white-labelled" preachers who can be branded to your own faith (for cheaper religions this can be mutualised if the demand isn't there yet)
   
3. An "avoid damnation" generic TV show including presenter with amazing hair that can be tailored to your religious target market
4. Believers on Demand - a set of white labelled individuals from your core demographic to create the impression of an already thriving faith
5. iPhone versions of your sacred text
6. Place of Worship in a box - a high value service that converts buildings into a faith centre, in a similar manner to the successful "Irish pub in a box" model
7. PR support to help you claim discrimination and subjugation
   

All of this is backed by a robust IT and BPO solution that handles training, indoctrination, HR, payroll, finance and accounts.

Because "Religion as a Service" isn't limiting itself to a single religious ideal it will be able to offer dramatically cheaper costs than establishing your own religion from scratch. For people looking to establish a new evangelical Christian sect for instance you can be up and running in a matter of hours and proclaiming yourself to be the 2nd coming of Christ without all of the massive overheads that this normally entails.

People with more demanding needs, for instance wanting to create a full church for the flying spaghetti monster, are still able to leverage the underlying resources and staff of RaaS but will need to invest more in the customisation of the central "core text", as this is however simply a reference volume and our staff are supported by the very latest holy knowledge management tools even the most esoteric demands can be met and scaled. This means that your local Cargo cult can rapidly be turned into a world wide phenomenon.

Religion as a Service's ability to leverage knowledge and resources across multiple faith channels means that it can offer new religions a much more efficient manner of scaling its followers and help a sect turn into a profitable business much more effectively.

Now all I need is a VC to fund it and I'm away.

Technorati Tags: SOA, Service Architecture

The Clint Eastwood school of change

Change is hard, change against people who don't want to change is extremely hard bordering on impossible. In any change programme therefore you need to be clear about what you are up against and what success looks like.

This is where Clint Eastwood can help. Sure Chuck Norris can run around the world and punch himself in the back of the head but Clint Eastwood has many more lessons on delivering change in difficult circumstances and presenting different approaches.

This isn't for the collaborative occasions when you need to work with people, Clint isn't very good at that. This is for the occasions where you need to overcome people who are actively working against you and the objectives of change.

So what does Clint teach us? First off he teaches us that there are different types of people who we need to overcome.

There are people in authority who are abusing their position to make sure the change doesn't happen and are instead driving their own competing change agenda(Pale Rider).

There are people within a given group who are working against you (Gran Torino)

There are people who operate in a different management structure and are using that independence to try and prevent change (Unforgiven)

And sometimes there are people who need to be run down and shot because they are never going to be part of the new world (several films)

The point here is how does Clint deal with these challenges? The answer is that there are some obvious similarities and a few key differences.

Firstly the similarities.

Clint never gets angry, at least not against the people who are targeted as the blockers to change. This is a really important lesson. If people are blocking change you really can't just shout and scream at them, it doesn't work and just makes you look impotent.

Clint is focused. Clint doesn't have a huge number of approaches to deal with the situation and he never runs into the situation without significant amounts of planning and thought. This means that when Clint decides to take action he has already determined what the outcome he wants is, and then makes sure that it happens.

Clint is talented. Clint makes sure he is the big guy in any engagement, this doesn't mean shouting his head off it means making sure that he knows exactly what he needs to do and making sure he has the skills and resources to deliver against it. Clint makes sure that what ever the other guy is doing that he has the skills to adapt to the situation and make clear that he is in control ("Do you feel lucky? Well do ya punk"). The point here is that Clint backs his ability against his competition and this is a core way that he ensures the right outcome.

Clint is clear and concise. Why say 100 words when a stare or a grunt will do? Don't waffle, don't prevaricate , just make it absolutely clear what you want and how you will achieve it.

Now the differences

There really aren't that many, its really about how the similarities are applied differently.

The point here is that these key Clint lessons are how anyone should be looking at difficult change where you have a group of people who are adamantly set out against you.

Know you facts, know what they know, know what you want and concentrate on achieving that.

The lesson of Gran Torino is also that losing a battle can also be winning the war. I've had a number of experiences where you pull people into a "Road Runner" moment, i.e. you draw them to a point where they think they have won, then as the dust clears they realise they have stepped off the cliff and are about to fall.

The lessons of the "Dollars" films is that sometimes you just have to face it that people won't come on the journey and they need to be sidelined or exited. Don't try and get them on-board, its pointless, just work out the easiest way to eliminate the,

The lessons of Unforgiven are that people in groups often act in an uncoordinated way if you attack them individually. Don't go for the overall group, work out individual weaknesses and use that to drive the group apart.

The lessons of Pale Rider are that coordinating those that do support change behind you (also known as the Magnificent Seven approach) and using your focus and talent can overcome the larger but over-confident blockers.

Change programmes come in lots of guises but often you'll find yourself come to a moment where you realise that there is a group of people who just won't be coming on the journey with you. Then you've got to ask yourself the question

What would Clint do?

Technorati Tags: SOA, Service Architecture

Why would a cloud appliance be physical?

IBM often lead in technology areas and with the history of LPAR on the mainframe they've got a background in virtualisation that most competitors would envy. So clearly with cloud they are going to go after it. Sometimes they'll do what they did with SOA and tag a (IMO) dog of a product with the new buzzword (MQSI = Advanced ESB - I'm looking at you) and other times they will actually do something right.

Now a product that can handle the deploy and manage instances sounds like a good idea. IBM have created just such a product which basically acts as a dispenser and manager for WebSphere Hypervisor edition images. The WebSphere Cloudburst Appliance will deploy, it will reclaim, it will monitor and it will manage. Very nice for people who have large WebSphere estates.

And this is what the product looks like

Yes I did say look like because IBM have built this cloud manager into a physical box. Now appliances for things that need dedicated hardware acceleration I understand, but why on earth is something that is about managing virtual machines, something that might be doing bugger all for large periods of time not in itself a virtual image?

Given that the manager is unlikely to be a major CPU hog it seems like an ideal thing to be lobbed into the cloud itself (yes I know its not really a cloud, but lets go with the marketing people for now, they've made a bigger mistake here IMO). If it was in the cloud then you could add redundancy much more easily and of course it would require its own dedicated rackspace and power.

Like I said I can understand why you might like a virtual machine to do what the CloudBurst appliance does, but I have no idea why you would want a dedicated physical machine to work on a low CPU task. As IBM expand this technology into DB2 and other WebSphere elements you could end up with 20 "Cloudburst" appliances managing and deploying to a single private cloud. How much better for these to be cloud appliances in the truest sense and to be virtualised within the cloud infrastructure itself.

A physical box to deploy virtual images makes no sense at all.

Technorati Tags: SOA, Service Architecture

Single Canonical Form? Only Suicidal Dinosaurs need apply

Now I said a while ago that Single Canonical form wasn't for SOA well now I've been doing some SaaS projects and I've realised with traditional modesty that not only am I right, but that people who are still pushing it as an approach can be described as suicidal dinosaurs.

If SaaS is anywhere in your future, and it will be unless you are a military secure establishment and even then it might me, then GIVE UP NOW on the idea that you can mandate data standards in applications and create a single great big view that represents everything. It isn't going to happen, you are now back in the wild old world of multiple different systems each with their own specific exchange formats and....

you have to cope with it

Moaning about it or trying to push back on it because it doesn't meet your "corporate data model" isn't going to get you very far, its liable to get you heading out the door in fact.

Single Canonical Form is for people who believe in the great big database in the sky theory of architecture, it didn't work for SOA but some people still tried to force it in. Now the love child of SOA and the Cloud is coming to destroy it completely. The only sensible policy is to look at an "active" MDM strategy and a brokerage approach to communication between systems ideally based around a federated data strategy that leaves information its its source systems but provides references between them.

The brokerage challenge and active MDM challenge will only grow as SaaS becomes more common. There is no ability to inflict, and I choose the word advisedly, your Single Canonical Form on SaaS providers so you have to actually take a sensible solution oriented approach to data consistency and visibility.

Single Canonical Form is a dinosaur like approach but it was one which some people still managed to get away with proposing as a way to create a career. Well SaaS is the meteorite heading towards their earth and the choice is evolve or die.

Technorati Tags: SOA, Service Architecture

The challenge of "build for the web" to SaaS economic models

Build for the web is a strong meme. Most of the time people focus on the technical elements of this and the end user elements. What I haven't seen people talk about much is the impact that this has on the current financial models from SaaS vendors. SaaS can give great benefits for companies by enabling an OpEx rather than CapEx model but the solutions today do assume that a person is using them. Working on the system I'm looking at right now its clear that this model is okay in most cases in 2009 but won't be the case for long.

Think about a world where you are looking to combine a SaaS CRM provider (user based licensing) with a Logistics solution (shipment based) and an analytics solution (user based) and create a new unified sales/marketing and shipping solution.



All fine. But now lets say you get clever and you realise that from the CRM solution what you are really interested is tracking every interaction you have with customers. Your internal people aren't interested in the various options the CRM solution has, they just want to be logging what is going on but using the analytics to provide them with visibility of what they should be upselling and what are the current globally successful products.

Looking forwards as SaaS vendors shift towards different utility models which address their financial demands (e.g. what is a "user" if a remote service or other SaaS solution is accessing it?) the complexity of information sharing is liable to increase, both in terms of licensing ("you shall not cache or store information from XaaS outside of the system except for your own individual use, it shall not be loaded into another application where users other than yourself can access it") and in terms of efficient financial management.

The transactional model, shipments, however remains valid and is very difficult to get around. Sure you can be more efficient on the bundling but you are always going to pay for the actual transaction. This presents both a challenge and opportunity for user based licensing solutions, they can move away from it towards a more transactional based approach, e.g. accounts/prospects/campaigns/etc for CRM, and start providing a business with a more direct measure of the actual value they get from an IT system.

The SaaS revolution has a long way to run and its economic shift is only just starting.

Technorati Tags: SOA, SaaScloud

SaaS and the Cloud a development challenge

A while back I blogged on how to do ERP with a middleware solution. The point was to leave the package untouched while adding your customisations in an environment that was better suited to the challenge. It made upgrades easier and also would help to reduce your development timescales.

Well the world is moving on and now I'm looking at a challenge of SaaS + cloud. A standardised package delivered as SaaS where we need to extend it using a cloud solution to enable it to scale to meet some pretty hefty peaks. So how to meet this challenge? Well first off lets be clear I'm not talking about doing APEX in Salesforce I'm talking about treating the SaaS solution in the same way as I'd treat an ERP, namely as the service engines that provide the transactional functionality. What the cloud part would need to do would be to do the piece on top, the user interaction, pulling in some other web delivered elements in a late integration piece.

Model wise its the same


We've got a bunch of back-end services with interfaces (Web Services mainly) and we need to build on top.

The first new challenge is that of bandwidth and latency. "Build for the web" is all very well but there is nothing quite like a gigabit ethernet connection and six feet of separation. Here we have the information flowing over the web (an unreliable network performance wise) and we need to provide a good responsive service to the end user. So clearly we need to invest a bit around our internet bandwidth. Using something like Amazon clearly helps as they've done a lot of that already but you do need to keep it in mind and it becomes more important to make sure its not "Chatty" as those latency hops really add up.

The next piece of course is that we need to cache some information "Use REST" I hear people cry, the trouble is that even if I used REST (or rather if the SaaS provider did) then I'd still have to over-ride the HTTP Cache headers and build some caching in myself. Why? Well the SaaS solution has a per transaction model in certain sections and a per user in others, this means I need to limit users until the point at which they REALLY have to be known to the SaaS solution and I need to cache transactions in a way that reduces the money going to the SaaS vendor. So here the caching is 20% performance and 80% economic. Its an interesting challenge in either REST or WS-* as you are going against the policy that the service provider would set.

So the objective here is to build proxy services on the cloud side which handle these elements. These proxies are going to be reasonably dumb, maybe a basic rules engine to control pieces but are there to make sure that the "web" doesn't get in the way of performance. These proxies will however have a database that enables searching across sub-queries as well as the matching of exact queries (e.g. "find me all letters between A and Z" should enable the query "find me all letters between M and U" to be done as well without a SaaS hit) not sure yet whether we will go for a full database or do some basic pieces ala Amazon.

"Build for the web" is a mantra that many people are supporting these days, and there are good reasons for making your services available via that channel. But combining solutions and still delivering high performance is a significant challenge particularly when economic contracts can rule approaches such as the basic REST approaches redundant.

So when looking to build for the web think about the structure of your application and in particular the impact of latency and bandwidth on its performance as you look to consume other web applications. If you do have a financial contract in place with a SaaS vendor be very clear what you are paying for.

Technorati Tags: SOA, Service Architecture