Wednesday, October 31, 2007

The Ivory Tower of Implementation Optimism v Long term viability

One of the things I've been banging on about during presentations recently is how to bits of IT thinking are the major contributions to the current mess that IT finds itself in.

First off is the view that new technologies and solutions will clean up the mess. This is wrong for the simple reason that in almost all companies this represents less than 20% of the total IT spend (often its less than 10% or even 5%). This means that 80% of the spend isn't being touched or looked at. A simple 80/20 rule would tell you that this approach is wrong. Unfortunately IT continues to believe that yet more lipstick on the pig will make the pig attractive.

The second one is that projects and in particular the focus on Cost to Live (CTL) over TCO are set up to create bad IT estates. By focusing people's mind on the go live date it encourages people to make short cuts and decisions that while helping the project hit its cost and time figures ensures that the TCO is driven further up. In other words the decisions in the 20% actually make the 80% worse.

One of the things I've worked with companies on, and which is in the book, is that different parts of the business require different delivery approaches and technologies. This means looking at the business and understanding where you should be targetting your best people. I've said it before that if I had a bunch of Dan Creswells to work with then I'd be making sure that they were focused on the highest value areas, the ones that require constant change (i.e pretty much never drop into standard support) and just letting them do it the way that works for them, and I'd do this making sure that they were responsible for the TCO not just CTL.

However that is about 2% (at most) of the heads out there in IT (remember Meyfroidt's Law). The skill in architecture in these other areas therefore is much, much harder. Architecting a solution in a team of talented people that will be supported by talented people is easy, almost trivial. To architect a solution that will be built with people without your skills and talent and which can then be supported by people who don't have the talents of those developers is a real and challenging art.

The way I look at it is this. I went to university in York and in York there is York Minster, one of the most impressive buildings I've ever seen or been inside. When it was built the art of building was a real craft that required master craftsmen all over the place to get it built. The end result is a fantastic achievement that is testament not just to the architect who designed it but to the talents of the builders.

Move on to the 19th Century (and even before) and building had become more of a commodity. That cathedral to scientific knowledge that is the Natural History Museum was built with much lower skilled builders who were working to a defined plan. The end result is almost equally impressive as York Minster and yet the skills profile was dramatically different.

The other lesson to be learned from building is that the long term viability of solutions is more important than the initial build cost. The Tacoma Narrows disaster is a great example of focusing on CTL, with disastrous long term consequences.

The goal therefore of architecture when constructing it for the 98% is to enable the solution to be built and sustained successfully by that 98%. You are not architecting the solution for yourself you are architecting it for others. This requires a much higher degree of skill and architectural competence than simply aiming for the "best" technology for the best developers. This is why I disagree with Dan that there is a risk of Architecture RIP when looking at these areas, its more of a risk in the project and developer optimistic area IMO.

This is also why I have to disagree with Steve Vinoski's fatalistic assumption that this means that there is No hope for IT. In fact I'd say that there is no hope for IT if we take Steve's advice and concentrate on CTL
If you’re in a position of technical leadership or project management and you’re asked to come in ahead of schedule and under budget, my advice is that you’re generally more likely to succeed with REST and dynamic languages than with the alternatives because their inherent constraints allow for better focus.

Now I'd like to see some references to back this up (a nice manufacturing plant example for instance) but even so this just repeats the mantra of "this is best for CTL". It doesn't challenge the status quo and just tries to focus on doing "new" things over improving the old. It also assumes that the quality of the developers is at the same level as Steve, which is more than slightly unrealistic.

Steve is right that there are great developers out there in the enterprise space, but he is wrong in assuming that it is viable to just employ the talented. Simply put these people won't work in support of legacy systems (for starters) and are wasted when implementing low value or commodity systems. The reality of a good enterprise IT community is that it is a distended bell curve of quality v quantity

Now some people might find it depressing, but tough its the reality. If you aren't looking at architecture from a business perspective and constraining it by the reality of the world then you aren't doing good architecture.

The hope for IT and the reason that architecture is required is to start focusing on the 80% and on the long term viability of solutions and to stop focusing on the short termism and Ivory Tower of "lets only use brilliant developers". If we can focus as architects on enabling the talented to link up with the rapidly evolving and enable the rest to deliver and support successfully then we will finally start progressing IT forwards.

Talented developers create architectures that they can build and support. Talented architects create solutions that can be built and supported by others.

There is much more actual talent in the later than there ever will be in the former.

Technorati Tags: ,


PetrolHead said...

I'm not sure we're disagreeing as such - in fact I wonder if we're talking at cross purposes.

The core issue for me is the conflation of technology and architecture and the fact that more often than not we throw out architecture early and jump straight to technology decisions. Such behaviour means that considerations like this one put forward by your good self....

"The goal therefore of architecture when constructing it for the 98% is to enable the solution to be built and sustained successfully by that 98%. You are not architecting the solution for yourself you are architecting it for others."

....are also thrown out.

So, if you end up using a framework that's fine but it should be because you've done sufficient consideration of available resources, design, analysis and so on to be sure that's the correct option rather than just knee-jerking a la:

The answer is J2EE/WebServices/ESB/RDBMS etc, what's the question.


PetrolHead said...

Hmmm and in light of the above I would suggest that we're probably coming down to a rule of thumb like, you can have as many commodity programmers as you like but you'd better make sure you have at least one damn fine architect.


Steve Jones said...

Dan I couldn't agree more on technology != architecture. You are spot on that its a case of REST/ESB/J2EE or whatever is the answer... now lets use that hammer to hit stuff.

I'd say that the more commodity developers you have the more strong architects you need. Its almost a "emperor and plebs" approach.

What I would say is that commodity developers tend to work well with commodity platforms.

PetrolHead said...

"What I would say is that commodity developers tend to work well with commodity platforms."

Uh huh so long as you stay within the confines of what the frameworks and high-level architecture can support.

There is a breakout point in here somewhere such that a talented team will be able to do stuff that can't be done by others. And businesses will need to ensure that what they're asking for is below that breakout point or face high costs, poor software etc. At the end of the day, one can only compensate so much for mediocrity.


Steve Jones said...

Which is what I mean when I say target the good people at the important areas. All too often I see talent wasted as people create technically great things in areas that just aren't important to the business.

That is the point of using a commodity, you have to choose it where a commodity works. You have to have a real business reason to not use a commodity not just a technology one.

Anonymous said...

Ugh, this rings so true to me.

What happens when your CTO walks by someone's cube, looks at the "wow" factor of a technology, and decides to implement it -- without the discussion of "Is this really what we need?"

I'm a dev/architect lead on a project, and I see failure written all over it because of tight deadlines and unproven technologies -- much like Dan was saying. The answer has been given, but no one is asking the question.

Thanks for writing this -- it makes me feel better that someone else is going through the same crud.

Though to answer you 98% question, the CTO decision is to throw contractors at the problem -- after all, the 98% who have to clean it up won't be the CTO's responsibility once he moves on to the next place to cause a new problem. Perhaps that's the bigger problem... let someone else deal with the mess.