The Ivory Tower of Implementation Optimism v Long term viability

One of the things I've been banging on about during presentations recently is how to bits of IT thinking are the major contributions to the current mess that IT finds itself in.

First off is the view that new technologies and solutions will clean up the mess. This is wrong for the simple reason that in almost all companies this represents less than 20% of the total IT spend (often its less than 10% or even 5%). This means that 80% of the spend isn't being touched or looked at. A simple 80/20 rule would tell you that this approach is wrong. Unfortunately IT continues to believe that yet more lipstick on the pig will make the pig attractive.

The second one is that projects and in particular the focus on Cost to Live (CTL) over TCO are set up to create bad IT estates. By focusing people's mind on the go live date it encourages people to make short cuts and decisions that while helping the project hit its cost and time figures ensures that the TCO is driven further up. In other words the decisions in the 20% actually make the 80% worse.

One of the things I've worked with companies on, and which is in the book, is that different parts of the business require different delivery approaches and technologies. This means looking at the business and understanding where you should be targetting your best people. I've said it before that if I had a bunch of Dan Creswells to work with then I'd be making sure that they were focused on the highest value areas, the ones that require constant change (i.e pretty much never drop into standard support) and just letting them do it the way that works for them, and I'd do this making sure that they were responsible for the TCO not just CTL.

However that is about 2% (at most) of the heads out there in IT (remember Meyfroidt's Law). The skill in architecture in these other areas therefore is much, much harder. Architecting a solution in a team of talented people that will be supported by talented people is easy, almost trivial. To architect a solution that will be built with people without your skills and talent and which can then be supported by people who don't have the talents of those developers is a real and challenging art.

The way I look at it is this. I went to university in York and in York there is York Minster, one of the most impressive buildings I've ever seen or been inside. When it was built the art of building was a real craft that required master craftsmen all over the place to get it built. The end result is a fantastic achievement that is testament not just to the architect who designed it but to the talents of the builders.

Move on to the 19th Century (and even before) and building had become more of a commodity. That cathedral to scientific knowledge that is the Natural History Museum was built with much lower skilled builders who were working to a defined plan. The end result is almost equally impressive as York Minster and yet the skills profile was dramatically different.

The other lesson to be learned from building is that the long term viability of solutions is more important than the initial build cost. The Tacoma Narrows disaster is a great example of focusing on CTL, with disastrous long term consequences.

The goal therefore of architecture when constructing it for the 98% is to enable the solution to be built and sustained successfully by that 98%. You are not architecting the solution for yourself you are architecting it for others. This requires a much higher degree of skill and architectural competence than simply aiming for the "best" technology for the best developers. This is why I disagree with Dan that there is a risk of Architecture RIP when looking at these areas, its more of a risk in the project and developer optimistic area IMO.

This is also why I have to disagree with Steve Vinoski's fatalistic assumption that this means that there is No hope for IT. In fact I'd say that there is no hope for IT if we take Steve's advice and concentrate on CTL

If you’re in a position of technical leadership or project management and you’re asked to come in ahead of schedule and under budget, my advice is that you’re generally more likely to succeed with REST and dynamic languages than with the alternatives because their inherent constraints allow for better focus.

Now I'd like to see some references to back this up (a nice manufacturing plant example for instance) but even so this just repeats the mantra of "this is best for CTL". It doesn't challenge the status quo and just tries to focus on doing "new" things over improving the old. It also assumes that the quality of the developers is at the same level as Steve, which is more than slightly unrealistic.

Steve is right that there are great developers out there in the enterprise space, but he is wrong in assuming that it is viable to just employ the talented. Simply put these people won't work in support of legacy systems (for starters) and are wasted when implementing low value or commodity systems. The reality of a good enterprise IT community is that it is a distended bell curve of quality v quantity

Now some people might find it depressing, but tough its the reality. If you aren't looking at architecture from a business perspective and constraining it by the reality of the world then you aren't doing good architecture.

The hope for IT and the reason that architecture is required is to start focusing on the 80% and on the long term viability of solutions and to stop focusing on the short termism and Ivory Tower of "lets only use brilliant developers". If we can focus as architects on enabling the talented to link up with the rapidly evolving and enable the rest to deliver and support successfully then we will finally start progressing IT forwards.

Talented developers create architectures that they can build and support. Talented architects create solutions that can be built and supported by others.

There is much more actual talent in the later than there ever will be in the former.

Technorati Tags: SOA, Service Architecture

Separation of Policy from implementation or why I've turned on the human check

One of the basic rules of SOA is that you should keep the operational polices of a service separate from the logic. In other words the bits that dictate how, when and whom for a service are managed differently from the "what" of the service.

Today I've had to turn on the human check for comments as I've had a series of SPAM links put onto the blog. This is in effect implemented as a policy, changing the policy doesn't alter the implementation (i.e. the process of adding a comment) it just adds in an additional step that fulfills the policy requirement that there should be a person commenting.

This is why things like WS-Policy, WS-Security and the WS-RX group specifications are important. They keep the policy pieces away from the implementation pieces and enable you to manage them independently.

Technorati Tags: SOA, Service Architecture

Why republicans don't like SOA

I often look around for non-technical examples to use of why the people piece is more important than the technology piece and today via the wonder of YouTube and an American friend who sent me a couple of links I have a cracking one.

You know in companies the problem you have within IT or with business folks who just won't accept reality? You know the people who say "yes but if things we different then..." or "I don't care what the facts say I think this is better". You know these people, they are the types who make the same screw ups over and over again just because that is how they screwed up before, almost like a bad track record is proven track record.

The challenge often is that these people act as blockers to any change programme because it challenges, or refutes, their basically held beliefs and opinions. These might be "It must all be SAP" or "We can't change the business" or "Waterfall/Agile/Flying Monkeys/REST/Web Services are the only way to do anything" or some other single view point which doesn't like the idea of change and hates the idea of multiple ways of working and change (as an aside can people who don't like change please get out of IT).

Now to the bit that proves the republican party doesn't like SOA and prefers monolithic software delivery, well some of them anyway. Three republican presidential candidates indicated that they didn't believe Evolution was true. One of these Mike Huckabee even questions the temerity of considering this a reasonable question to ask. This is the standard response when you question people's opinions (or cushy jobs) and look to have change in an organisation. Things are declared as "off limits" for political reasons or marked down as inviolate elements that cannot be challenged.

People are the problem with change and technology isn't enough as even facts and reason often don't work.

Technorati Tags: SOA, Service Architecture

UML and SOA a beginners guide

One of the most common questions I get asked is how Business SOA works with UML. How do you capture Use Cases in an SOA world? How do you do activity diagrams, sequence diagrams etc.

The first point is that you should define the Business Service Architecture first, before you start into the IT project elements. This is the bit that gives the SOA context to everything else.

Now lets look at UML and in particular those pesky Use Cases. The simple rule here is that

• Service => Use Case Package
  
• Capability => Use Case.
  

In other words the requirements for each service need to be gathered within the bounds of that service, not from end to end of the application. The point here is that when doing the Use Case for a service the actor doesn't need to be a person the actor is any external entity which wishes to interact with the service. The interaction with other services can be done by <> statements to the capability definitions from another service package. The good thing about this is that you can use Use Case stubs (i.e. no actual detail) where the service is implemented using another approach (e.g. when using BPMN to implement SOA). This means that your Use Case model is complete for the services that it defines and still allows the flexibility of choosing different approaches for different services.

So that is how I'd say to do UML Use Cases with Business SOA.

Next up there are sequence diagrams and activity diagrams. For these I'd say sticking to the same rules for BPMN is a must, namely have a core thread within one service and only make invocations and requests to other services. This makes it much easier to enforce the separation of concerns and also helps to stop the leaking complexity that often comes into these diagrams.


The business SOA gives the structure, the UML the definition. This is the power as it gives context to the rest and enables different approaches to work together.

Technorati Tags: SOA, Service Architecture

How to create performant Web Services and XML

I was at an event the other day when a chap said that XML/Web Services etc were fine "unless you wanted to do thousands of transactions a minute in which case you needed something much more streamlined".
Now I was sitting at a table at the time and we all pretty much agreed the statement was bollocks. So I thought I'd do a quick guide to create performant Web Services and XML applications.

1. Use XML at the boundaries. If you are writing Java/C/Ada/Python/Ruby etc then once you've made the hit to translate into your native language... keep it there. When you cross another service boundary then that probably needs XML, but don't keep marshalling to/from XML every time you move around inside a service.
2. KISS. If you have 25 XSLTs and indirections in your ESB then it will run like a dog
3. Don't have a central bus that everything goes through, think federation

Oh and of course the most important one

1. BUY HARDWARE

Seriously, hardware is cheap so my basic rule (and it hasn't failed yet) is as follows:

If you are out by a low multiple (under 3) or a few percentage points then scale your XML processing like a Web site (lots of thin servers at the front). If you are out by high multiples then you've screwed up somewhere.

Now the later one can be that the vendor has screwed up, so make them fix it. But in the last 7 years of doing XML and Web Services I can say that in environments running lots and lots of transactions (thousands a second even) that I've yet to find that XML and Web Services didn't scale.

The key if you have a performance problem is to really look at the pipeline and see where the real time is being spent. Several times I've had people claim "Web Service performance issues" then found that 1ms is being spent in the WS bit and 5 seconds in the application, or people blaming a specific element (XSLT/XML parser/etc) and then finding out that it is a bug in an implementation (one in a certain vendors stack looked almost like a wait loop bug). These elements aren't performance issues with Web Services or XML, they are bugs and issues with the applications and implementations.

XML and Web Services are not the most efficient things in the world. But then I'm currently working on a computer where this browser (inefficient editor) is running in side a VM (inefficient) on a box with 3 other VMs running on it (very inefficient)... and it is still working fine. A home computer now comes for a reasonable price with FOUR CPUs inside it (remember when a front end started at 4 x 1 CPU systems?) and Moores Law continues to run away. The issue therefore isn't whether XML/WS (or event XML/REST) is the most efficient way of running something but whether it is good enough to be used. New approaches such as streaming XML take the performance piece on still further and make it even less of an issue than it was before. This is about changing the libraries however and not the principles. XML/Web Service works from a server perspective.

So server processing isn't the issue. So maybe its network bandwidth... again I'd say no. Lob on gzip and you will burn up some more CPUs but again that is cheap in comparison with spending a bunch of time writing, testing and debugging software. This creates a much lighter over the wire message and again just seems to be fine in implementations where the wire size is important.

The only piece I've found that continues to be an issue is something that is nothing to do with Web Services or XML, but which is more of an issue in REST, and that is chatty behaviour over the network. Being chatty over a network gives latency, no matter what your bandwidth is there will still be a degree of latency and you can't get away from that.

So basically the solution is to communicate effectively and accurately and to do so in reasonably coarse grained ways. This is hardly news at all as it has applied to all communication systems in IT. As networks speed up and latency comes down then maybe even this will become less of a problem, but for today its certain a place to avoid excess. Google for instance on a second search (cached) to "Steve Jones SOA blog" responds in 80ms. A coarse grained approach that has 1 interaction per request and 10 requests will have a network induced lag of at least 800ms, a chatty approach that has 5 interactions per request will have 4000ms or 4 seconds. Chatty = non performant.

So basically I've found that Web Services and XML do scale, hardware is cheap, that Stupidity doesn't scale and that networks still introduce latency.

Its not rocket science but some people seem to think that its still 1999.

Technorati Tags: SOA, Service Architecture

Change the business not the package

I've spotted a bit of a worrying trend around SOA and package delivery namely that people seem to be thinking that SOA magically means that you can now customise packages safely. The goal becomes once again "fit the package to the business" this unfortunately is the road to ruin. It doesn't matter what the package is, whether Finance, ERP, niche or even a small off the shelf thing for a point solution the basic rule remains the same. You are going package because it doesn't differentiate you, it isn't an area where you will gain competitive advantage and if everyone in your sector ran the same package, it really wouldn't matter.

This is a critical starting point. If you want to use a package solution you have to accept that this means you have chosen to take a non-differentiated route. If your content is the thing that is your USP then this doesn't mean that you shouldn't use a CMS it just means you should recognise that it is the content that is the USP, not the publishing and managing of the content.

Too often people go through extensive requirements gathering before selecting a package and then try and find the "best fit" to those requirements. The project kicks off and they try and implement the remaining requirements. This is suicide, don't do it.

First off start with a set of objectives and principles that will guide the package selection. One of these principles must be that the goal will be to change the business approach to fit the selected package. Then choose the package. The next state is then to ask "how can I cope with what the package does" this leads to the understanding of what business change can be done.

So how does SOA help in this process? Well first off you can use a Business Service Architecture to understand the drivers and interactions at a high level, this really helps to clarify where the package will be used and to make clear the broad business context in which it fits. So if you are looking at packages across multiple areas you should still be thinking about the different areas based on their different business context and drivers, one size doesn't fit all.

So SOA doesn't mean that you can customise the package anymore today that you could before. There is a rough calculation that I've seen a few times around package implementations (the numbers vary a bit but the thought is important)

So basically the more you customise the more likely you are to fail. Now while it looks okay down the bottom end, don't be fooled.

Now these aren't hard and fast numbers, indeed they could be under playing the impact of customisation.

SOA doesn't help in terms of package customisation. A decent BSA can help you understand how the package will work (if at all) and the boundaries that it will have, but the same rules on customising the box remain.

There is however a new choice when it comes to adding differentiation into a business service which is partly implemented using a package.... but not in this post.

Technorati Tags: SOA, Service Architecture

REST is dead long live SIP

One of the things that always amazes me in technology is how people seem to think that technology revolutions have suddenly stopped because they've found a hammer. Whether it be Web Services, REST, EAI, ERP, Excel or anything else certain classes of IT people seem to become blind to the history of IT and think that because they are happy with what they are using that there will be no other developments.

As an example of this lets look at REST (as implemented by HTTP) and SIP, now REST lays down a bunch of constraints and has its HTTP implementation as the way to do everything. HTTP is of course limited in what it can achieve and certain tasks it just doesn't seem to be the right way to go, VOIP being one example, now SIP provides some similar elements in terms of philosophy of simplification (which is nice) but enables multiple protocol interaction and can (in theory) support application to application communication.

Now I've been looking at it and I'm not sure why you couldn't use SIP as a boot strap before electing to use HTTP, VOIP, Video or something else for communication between applications and from applications to people. After all wouldn't it be nice for the application to send you an SMS when something changes rather than polling a web server? These people talk about using SIP to kick off VoiceXML as a reasonable approach today, and VoiceXML is an HTTP based standard.

This would of course mean that HTTP, and potentially REST, would be relegated to specific application problems with an increased requirement for sophistication around the actual mode of interaction between systems and between systems and users.

The real point here is that no one technology will ever remain for ever as the pinnacle of achievement. It is either replaced (e.g. CORBA to Web Services) or commoditised (e.g. TCP to HTTP) and in both cases it ceases to be the cutting edge of IT delivery.

Will SIP relegate HTTP to "one" of the options? Who knows. But what I do know is that backing HTTP to be the cutting edge of IT delivery for the next 20 years is on a hiding to nothing.

Technorati Tags: SOA, Service Architecture

Why procurement is meant to be hated

There are three hated departments in most companies, they compete to be the most loathed of the bunch but one stands out from the other two as different.

The departments? HR, IT and Procurement.

These are always loathed and cursed and accused of preventing progress or just creating a set of bureaucratic processes that stifle peoples soul and remove the joy from the world. Only one of them has an excuse however.... procurement.

Now the reason that procurement is hated is that in fact this is the job of procurement. Have you ever wondered why procurement systems are almost always a pig to use, and why every "upgrade" seems to work in a completely different and more perverse way?

The reason is simple, the goal of procurement is to stop people buying things. If the CFO wanted to make it easy they would just give everyone credit cards and a cheery wave of "whatever you want is fine". The reality is that the purpose of procurement, a centralised service in the business, is to monitor and control all purchasing decisions, from a new pen through to a new building. The system should be painful to both suppliers and users, ensuring that both sides try and avoid interaction unless completely required. In this way procurement helps to focus spending on what is really required and removes spending on items that are just "wanted" because its not worth the effort.

This is an important thing when looking at building a business service architecture, understand the true drivers of services and don't assume that there are common elements across them all or that the end-users opinion always is the most important. Usability, performance and reliability are all great things when interacting with your customers, but in procurement they are positively to be avoided and a focus on annoying the user and making things slow and painful and prone to failure helps to deliver on the goal of reducing spend.

Don't assume that the goal is always to make things better, or that a system that is seen as "rubbish" isn't actually doing its job. Understand the business value that a service has and the drivers that create that value. This helps you to understand what "good" looks like in different parts of the business and focus your investment and effort accordingly. Don't assume that everything in a business needs to improve and get better, somethings either aren't important enough (general ledger anyone?) or are actually better off being bad (procurement).

So that is why procurement is meant to be hated. The more hated it is, the more successful it is actually becoming.

So that really leaves HR and IT in their daily fight to the bottom of the satisfaction league.

Technorati Tags: SOA, Service Architecture

Should IT deliberately create chaos?

There is an old IT joke

A farmer, an architect, a gardener and an IT guy are talking about which is the oldest profession. Now these chaps are bible literalists so they think the Garden of Eden was the start of everything.

The farmer says "Mine is the oldest profession as Adam had to tend all the animals and get all the fruit from the trees, if he wasn't a farmer he would have starved."

The gardener jumped in saying "yes but someone had to design the garden of Eden and put all the plants in place, so God must be a gardener"

"No, no" said the architect "First God had to create order and structure out of the chaos and build the universe so he must have been an architect"

"Ahhh" said the IT guy "but where did you think the chaos came from?"

Now the reason for saying this is that I saw one of the best pro-REST presentations at the Enterprise SOA conference in Belgium last week (which actually talked about business SOA). One bit I did have issue around was a concept that disorder and fragmentation was the "norm" of enterprise IT and thus the problems were the same as the web (its slide 37)

Internet vs. Enterprise
One is a gigantic, uncontrollable anarchy of heterogeneous systems with varying quality that evolve independently and constantly get connected in new and unexpected ways.

The other is a worldwide, publicly accessible series of interconnected computer networks that transmit data by packet switching using the standard Internet Protocol (IP).

Now in part Stefan has a good point, namely that IT systems currently suck. But where I'd disagree is that the goal of IT should be to create such a chaotic system with so little governance and control. This is one challenge I have when people talk about applying Web principles to the enterprise, it misses out on a fundamental difference between businesses and the internet. Namely that of compulsion.

It is true that IT unfortunately does create IT estates that are a mess and which have in the main part been driven by people who focused on the technologies first. This isn't however true for the businesses that they support, there is the ability in companies to compel change and to enforce conformity. Companies establish procurement departments, HR departments and even IT departments with these aims. Taking the Web view and applying it to business seems to imply that the chaotic nature of the web is the correct approach to take in creating an IT estate that works for the business. This doesn't make sense.

We need to understand in IT that sometimes compulsion is a better approach, if you are putting in SAP its best to fit the business to the package not the other way around, equally if you are looking at links between businesses is it better to take the standardised approach of procurement and enforce conformity of contract or to enable a "dynamic discovery" approach (whether UDDI or via MIME)?

IT needs to learn from the business an understand the power of enforcement and contracts, rather than thinking that its the business that needs to learn from the chaos that IT creates.

(Oh and Stefan, join the revolution. My presentation on SOA Methodology is web delivered :) )

Technorati Tags: SOA, Service Architecture

Great parody on technical IT

Pete Lacey has a great parody on the technical focus of IT folks. Its a wonderful trawl through the way that business people ask for solutions and how IT in return focuses on technology.

In this hilarious romp through how businesses continued to ask the same questions and the IT department continued to offer them yet another technology that would this time solve the problem. There is a great bit in there where he demonstrates his comedic brilliance by pretending that he thinks that VOIP is a business requirement (rather than the requirement of course being to reduce the phone bill), this wonderfully highlights how technologists can't see the difference between a business requirement and a technology solution.

There are great bits in there to highlight, almost too many to mention, but for me the real bit that made me realise how great a parody was this bit

"And… Well, you know the rest. They picked the wrong technology again, despite the fact that the right choice was staring them in the face. Like CORBA before it, SOAP..."

Brilliant, a superb highlighting of the extreme arrogance of technologists in thinking that the next technology really will be the silver bullet. He then drives the point home by brilliantly misunderstanding SOA and equating it to network oriented computing (although I think just saying Distributed Computing would have underlined the parody a little better).

Thanks to Pete for the shout-out at the end to my BSA post. He hilariously pretends to misunderstand the difference between architecture and requirements gathering, a common problem for technical IT.

I don't think I've seen a better post that sums up the problems of technologists thinking that technology is always the solution and that anything more contextual is a bad idea.

Truly well done Pete, a great post on the issues of technologists focusing only on the technology and assuming that this will meet the business needs.

Technorati Tags: SOA, Service Architecture

Business Service Architecture - is it time to admit defeat on SOA?

Over on the Yahoo email list there are a bunch of emails around SOA/REST etc etc and as ever Mr Tilkov summed it up well when he talked about Business level SOA being implementable using REST, but technical level SOA being different to REST.

So what I'm thinking is it time to bring clarity to all of this and admit that the techies and vendors have "won" the battle for the SOA name? Time to give up on the SOA name as something that has been co-opted purely to the technology and which is sometimes more focused on selling products than on delivering solutions?

My proposal is that we should start being clear when we mean business SOA by calling it a Business Service Architecture. The goal is to clearly differentiate business SOA from technical SOA and help move the debate on from the current technology centric debate towards one that is about changing how IT works to make it focused more on business solutions than technical implementations.

Business Service Architecture - Business problems solved, no matter what the technology.

Technorati Tags: SOA, Service Architecture

Exception Management and SOA - a Webinar

Last month I recorded a Webinar with Vitria (one off registration required) on how exception management fits into SOA. The key point I was trying to get across in the presentation is that exception management is a business issue and that SOA should be about the business and therefore thinking about business exceptions is a core requirement for any successful SOA.

Exception management is about when things go wrong, two core points here

1. Sometimes its okay for things to fail and ignore them
2. Exception management is a cost overhead, getting people back onto the primary flow is value creation

SOA as technology wouldn't think about exception management, SOA as business must.

Technorati Tags: SOA, Service Architecture