Five Nines in a Service Oriented World - Part 2(c) - Understanding the minimum operating requirements

Okay so its "nice" to have the preferred form of address for a customer when you put up the order response page. But should you display an "order not processed" just because the service that tells you if its "Mr Jones" or "Steve Jones" isn't available or should you just say "Dear Customer" instead? Well clearly for something like that the answer is go with "Dear Customer".

This isn't limited however to just trivial data formatting elements it can apply to things that appear to be much more critical. The key here is to understand how to aggressively degrade the application. What this means isn't just planning for when a call fails, but actively making things fail and reducing the operation of the system to a minimal subset, and ensuring that the reliability of that subset is maximised.

With the failure lists above it was a case of something failing and then the calling service coping, with this approach its actually a case of deliberately not accessing services and operating as if they never existed. Why would you want to do this? To reduce the risk of knock on effects from failures in terms of system resources, live/dead locks and data errors. If you can shutdown into a "safe" mode of operation you can at least keep the lights on and keep the core business running.

As an example if you have a system that actually runs the core production line at a drug company and there are a number of systems that can change what is being made and which the production line reports. The critical factor is just keeping the line operating, as for every hour its not working means lost revenue for the company, then here you could look at the safe mode as designing the system so it can operate successfully without any links to external systems. This could involve log files being shifted after hours, or even tapes being couriered to another data centre. It means understanding how long this form of operation can continue and giving IT and the business time to put in place contingency plans. The point here is that the manufacturing line is the minimum operating condition, if there is any risk that external factors could force it to operate below peak efficiency then these should be ruthlessly shutdown and the core allowed to continue to operate.

This sort of approach is very important when dealing with 3rd party systems where they could be legal or trust issues that require you to shutdown access in a hurry either because you feel the remote system has been compromised or because they are not meeting their SLAs.

The difference between minimal operation and planning for failure is that the services might not actually have failed, but an operational decision is made to work without them. This is the critical difference. Planning for failure is about coping with failure that happens operationally an implementing a mitigation plan. Deliberately degrading the system is a business and technical decisions where it is decided that the risk of failure or information error outweighs the benefits.

Another example of this would be when a system has to handle dramatically increased loads due to an external event, maybe a surge in demand due to an overly successful marketing campaign or some external problem that has caused a surge in exception conditions. Rather than the system taking the classic "normal" two step IT solution to this problem:

1. Specify the hardware to a level so large it bankrupts the company
2. When the system falls over because the company refused to go into bankruptcy go "na, na, told you so"

There is another choice here which is to enable the business to have the "oh shit" moment and then start turning off pieces that they currently decide are non-critical and just operate the core that is required to handle this unexpected event. This might mean, for an issue, concentrating resources on the support functions and for a marketing campaign it might mean taking the business decision to take the orders and batch up the payments at the end of the day or preventing people from searching for products as 95% of people are looking for the specific element from the marketing campaign.

Design a system so element can be deliberately failed is a big change in the way applications are built today, but in a distributed SOA environment is going to become more and more important.

Old ways won't work. So what does it take to actually do this in a system? First off it means that you need to understand for each service what it calls and have a simple "Shall I call" check that can be toggled at runtime (not exactly difficult these days), if the answer is "don't" then you need to have a mitigation plan in place. Stage 1 is to put the check in place, and have no mitigation. This is the cheap first way of enabling your system to adapt in future as such challenges become your operational reality. The important bit is to really understand what the actual core operation is for your business so you can start planning for that and not creating an SOA environment that is rich and dynamic when everything is fine, and when there are issues its buggered.

Its not a tough question to ask the business... but its one I've rarely seen asked.

Technorati Tags: SOA, Service Architecture

Five Nines in a Service Oriented World - Part 2(b) - Planning for failure

At some stage in a services lifecycle one of the interactions that it makes will fail, therefore start from the assumption that it will fail and start thinking of the impact and cost of that failure. There are four basic scenarios for failure of an invocation

1. Catastrophic - Failure of the invocation renders the consumer invalid, in other words future calls to the consumer (not the service) are invalid as a result of this failure. Its a really bad design position to get into when this happen but there are some cases where its possible (for example hardware failure), mostly its down to the stupidity of the people doing the design.
2. Temporal Failure - Failure of the invocation renders the current processing of the consumer invalid. This is a very normal scenario, for instance when you run out of database connections or when a network request fails. This is the standard for most systems, it just propagates the failure all over the network.
3. Degraded Service - Failure means the consumer can continue but is not operating at optimal capacity. This is a great place to be as it means the service is coping with failure and not propagating it around the network
4. DCWC - Don't care, Won't Care - This is where the invocation was an embellishment or optimisation that can be safely ignored. Example here would be the introduction of a global logging system where the backup is to local file. From a consumers perspective there is no change in QoS or operation, there are some operational management implications but these have well defined work arounds and are not related to the core business operation.

The goal of a high-availability SOA is to get everything into the Degraded or DCWC categories. This means planning the work around right from the start. Part of the question is whether the high-availability is really business justified, if it is then its time to start understanding how to fail.
Primary, Secondary and Beyond
The easiest solution is to provide a fallback is to have redundant primary nodes. This is a very common solution for hardware failure (clusters) and can also be used to solve network connection issues. What might be done here is to have multiple primary services which are hosted in different environments, thus meaning if one environment fails that operation is seemless. The trouble is that redundant primary often is prohibitively expensive as it can require complex back-end synchronisation of data with all of the problems that this brings.

Next up is the idea of having secondary, or greater, services or routes. As a simple example lets take an order service that is trying to submit the order to the actual transactional system.

1. Submit request via WS over internet - Failure on exception
2. Submit request via WS over VPN - Failure on exception
3. Submit request via WS over JMS - Failure on exception to connect
4. Log to file - Failure on exception to connect

Here what we have is a variable QoS, the ISDN connection is slower, but uses the same technology and would be expected to contain the same responses. The JMS and log to file are both long term operational elements and so any expected responses cannot be assumed to happen within the required time period. So for order submission this might mean a lack of confirmation of the order and no ability to commit to a delivery time. That is looking at Primary/Secondary/etc from a network connectivity perspective but this isn't the only way to consider it. Other examples could be from a business perspective in terms of supplier preferences

1. Primary - GeraldCash - Retry once on network failure, raise Critical on 2nd failure
2. Secondary - FredPay - Try once, raise warning on failure
3. Tertiary - Bank of Money - Retry once on network failure, raise Emergency on 2nd failure
4. Final - Route user to call centre

Here we have a great financial deal with GeraldCash, and indeed we expect them to work all the time (hence Criticality is high) and them not being available will reduce the margin on a transaction and they will have to pay penalties. Next up is FredPay, its cheap enough but not the best from a reliability perspective, finally from a technical delivery perspective we have the Bank of Money, very reliable but the most expensive by a mile. If everything fails and we don't want to lose the transaction we could try and route it to the call centre for some offline processing.

So planning for failure means understanding what else could be done and what other calls could be made that would deliver the functionality required but within different bounds to the original request.

Planning for failure means its critical to protect your service from those that it is calling. Using a proxy pattern is an absolute minimum operational requirement, because its in that proxy (potentially via some infrastructure elements) that this failover can be done. You do not want the core business logic worrying about the failover tree, it just needs to be able to cope with the changing capabilities that are available to it. This means that you need to design the system to have a set of failure modes even if you can't see that being needed to day this doesn't mean you build the failure modes at this time but that you put the basics of the framework in place to enable it. And is a proxy really that much of an overhead? Nope I don't think so either.

Planning for failure should be a minimum for an SOA environment. This means planning for both functional and system failures and understanding the risks and mitigations that they present.

Technorati Tags: SOA, Service Architecture

Five Nines in a Service Oriented World - Part 2(a)

I'm going to break this down into a series of posts on the topic as it just gets a bit big in one post. The first post will cover the basic principles and options, then the subsequent posts will detail those options.

So in the first article on five nines I talked about the challenge of SOA reliability and how the goal of reaching five nines really gets difficult when you are talking about distributed applications. I take on board one of the comments about getting the "right" information being the goal of an operating SOA and its that point that needs to be born in mind. The goal of operation is to deliver a system that returns results which are acceptable to its consumers at that point in time given the operational constraints that exist..

So first off lets start with the different options of solving the problems raised by Deutch's Fallacies. So sure some people will say "well duh!" but it worth stressing that there are different options to solving the basic problem of distributed reliability and those different options are applicable based on the differing business and technical drivers on your service. One size doesn't fit all. If you want a magic bullet, go and talk to a vendor they'll be happy to sell you a nickel plated one.

Distributed high availability is hard, the goal here is to help understand what type of "hard" problem you are facing.

1. Plan for failure
2. Understand what the "minimum" operating requirement is
3. Understand the time critically of information
4. Understand the accuracy requirement of information

Taking these one at a time they have some big impacts on how you design, build and support SOA environments. The primary challenge though is to accept a basic truth perfect operation is impossible if you strive for perfection then you will never deliver five nines, the goal of High Availability SOA is to deliver acceptable operation at all times

Technorati Tags: SOA, Service Architecture

SOA Vendor Ratings - Q1 2007

It was back in May last year that I did my first assessment of the "main" SOA vendors, so first up I'm going to revisit that list, then I'm going to add in (and I really will this time) assessments from some of the smaller players and open source. As ever the following is my view, nothing to do with who I work for and can't be assumed to come close to reality etc, etc, etc.

This assessment has the same summary info as the previous one which means

1. IT Vision - What are they going to do in IT, implementation of applications, integration with backends, sort of the technical end of SOA
2. IT Implementation - Great powerpoints guys, but what about the products...
3. Business Vision - What are the doing for the business, what is the content and how will it work for the business
4. Business Implementation - as before, what exists beyond the powerpoints
5. Standards - SOA implementation is massively about standards, how much does this company implement and drive standards
6. Stability - How stable is the current product set and roadmap, will they be shifting strategy and leaving you in the lurch, or going out of business and doing the same

This time however I'm going to go a bit deeper on the actual vendor reviews. So first off here is the summary


Now you will notice that everyone has shifted quite a bit on the business vision side, this is partly because they have but also because I've broadened out the business side to include the operational challenges of managing SOA at the business level. Oh and you'll also notice there are two IBM assessments... which probably makes that a good place to start

N.B. The blue line is my assessment of "now", the redline is my prediction for where they will be in 3 years time.

IBM
So why are there two IBM assessments? Well the first one is based around the roadmap that IBM tell everyone, the one that still includes MQSI, sorry "Advanced ESB". The second one is based on what I think is the real roadmap and this comes from bitter experience of watching clients with MQ Workflow and WebSphere Interchange Server believe that they would continue as well. I don't buy the Advanced ESB line, and I don't buy the "you've got to use a proprietary product that is a bugger to install" rather than a single standards based platform and I really don't buy the "There are things that MQSI^H^H^H^HAdvanced ESB does that just can't be done in J2EE". Hence the reason there is the IBM assessment based on IBM
and my view on their "real" roadmap
Lets be clear here, J2EE is the way forwards for IBM. Having something that has a completely different development, deployment, management and versioning approach makes no sense and what is left that is important can't be done in Process Server et al today? A bit of multi-protocol support and the ability to do COBOL Copybook? The shame is that IBM do have in their J2EE based stack a really good set of products for developing applications. They are still pretty weak at the business and pan-enterprise level but they have added the registry and of course have one of the broader tooling suites out there. Oddly however this tooling support doesn't appear to extend to testing where the async testing support appears to be limited to JUnit, which isn't exactly great as JUnit is poor at async (as I know from testing an MQSI infrastructure using JUnit). With CBM they actually have a business modelling approach, but unfortunately that still looks like its considered "special" so isn't yet in the tool suite so everyone can use it. Good suite, good for applications, good vision (where it isn't subverted) but they are a bit weak across the enterprise they really need to start being honest around their roadmap so people can start planning for the Java based solution that is bound to come.
BEA
Well they've bought a few more companies and the Aqualogic and Weblogic brands are really beginning to take shape. They still don't have anything in terms of methodology at the business service level and this really is going to be an issue in the coming years. They've started talking about the situational applications (Aqualogic area) and this split of backend handling and business focus really does make a lot of sense.
They really need to beef up around the governance and testing side though, it really isn't good enough to have a "preferred" partner, its either "use what ever you want" or its "in the box". Testing especially is an issue, they don't have any async testing which isn't great for projects and future viability. The current messages around Tuxedo as a very expensive Adaptor for mainframes is also a bit odd, hence the knock down there. Great product suite, great stack, good split of business and technology, but they need to focus more around the operationals for SOA in the same way as they have previously done around the application server.
Oracle
You really have to give the folks at Oracle credit, this time last year they had no ESB (except if you believe some of the analyst reports) and to be honest I thought it was going to take them a long time to get something that is properly separated. Sure there is the continued huge focus on "BPEL" as the answer to world hunger but there is certainly something coming together. This year is a big year for them as its the release of version 11 of the stack, with their membership of both the JBI and SCA/SDO camps its going to be very interesting to see the quality of what comes out in that new version.
Weak in the "business side" particularly around the modelling piece (and a great big EA tool is not the answer IMO) the operational side of the tool is okay but where they really shine out is around the testing, they actually have some async testing that can be linked back to a continual build, see it is possible. Integration is okay but a bit basic right now and the designer elements of the tool aren't really up to snuff from an SOA perspective. A good stack, an amazing rate of acceleration but its fair to say that there are still plenty of areas for improvement for the 11 AS release.
SAP
The gap between Oracle and SAP continues to widen in terms of the independent viability of the middleware stack. They've had some good thinking around the futures of all of this and the visioning is strong, the question is whether they can ever separate the Packaged application futures from the demands of the middleware, its a similar problem to the one that Microsoft have, but at least with SAP they are binding it to actual business value and business information.
Basically if you are doing SAP then its worth doing, and indeed its probably the only way, but if its a choice as a broad technology stack across the enterprise then this probably isn't the one you are looking for.
Sun
Will Sun deliver on the vision that was put forward last year, or will an EAI centric view of SOA emerge? There is lies the basic dilemma for Sun at the moment. They have a good EAI centric product in JCAPS (the old SeeBeyond stuff) and a great set of future tools (As demoed by Charles Beckham for me at JavaOne last year) the challenge now is to make that tooling shift while keeping the solidity of the underlying platform. At least with SeeBeyond and JCAPS its all based around J2EE so they don't have the mess that some others have.
The thing that knocks Sun down from an application development and operational perspective is that the current tools are very "me" centric, by that I mean that they assume that everything runs on JCAPS, the other knockdown is the debacle that is JavaSE 6 which really doesn't help the perception of Sun as a company that wants to solve enterprise problems. I really hope Sun bring it all together and start focusing up at the business problems where they currently aren't really involved at all. Great integration stack, really good for doing interfaces onto systems, needs to broaden out (using the tools that they actually have) into being an application stack and from there on towards the business.
Microsoft
Microsoft's progression around SOA since last May? Well they've released an operating system which has a proprietary async process model in it and they have a decent client side development model for web services.... Linking technology so directly to an operating system release is just plain bonkers, its as dumb as putting a Web Service stack into the JavaSE 6 release.

BizTalk remains the "heart" of much of the SOA messaging but its essentially the same product as 2004, which isn't great. Everyone else has moved on and it will be interesting to see if Microsoft come up with something equivalent to SCA, or even adopt it now its going into OASIS. With the Longhorn release due this year its really time for them to step up the focus around the enterprise and particularly improve their lifecycle and design support tooling. Microsoft Motion is a good business focused way of creating views on an enterprise, but unfortunately it appears too often to have been subverted into a "buy product" pitch. Either Microsoft want to play in the enterprise software space or they've decided that its not worth the effort, this year should outline which of those it is.
The ratings and categories explained
Now a quick summary on what the ratings actually mean, first off this is an assessment against what "perfect" would be today, rather than all time perfect (i.e. if someone stays at the state of the "now" then they'll always be a 5. The numbers are as follows

1. Very very basic, not really functional
2. Basic, meets some powerpoint and demo needs, but not much else, might be via a 3rd party to make it actually work.
   
3. Can be used by the skilled
4. Actually a pleasure to use and helps you move forwards
5. Cooking with Gas

So really its an exponential scale rather than linear.

Now for the categories

• BSA - Business Service Architecture, the ability to model the enterprise as services
• BSB - Business Service Bus
• BPM - Proper SOA and business centric Process Management
• Registry - A service registry
• Management - Ability to manage and configure operational services
• Monitoring - SLA and monitoring of services and interactions, independent of the vendor
• Testing - Testing of services at all stages of the lifecycle, especially async testing
• App Design - Ability to develop applications that consist of multiple services
• App Dev - Ability to develop services and applications that co-ordinate them
• App Process - Application level process models (where BPEL sits) and its ability to work in a proper SOA way
• App Model - The overall conceptual model of SOA applications that the vendor pushes
• ISB - The integration service bus, getting things out of older systems
• Adaptors - How easy is it to get things out of old systems
• Int Model - Integration Model, the conceptual model that the vendor pushes for integration
• Standards - How well does the vendor implement and support standards
• SCA/SDO - How well is the vendor progressing down the SCA/SDO path
• JBI - How well is the vendor progressing down the JBI path
• WS-* - How well is the vendor at supporting WS-* (WS-TX excluded)
• J2EE - How well do they support J2EE (standardised operating environment = lower support costs, no matter how much people bleat)
• Roadmap Honesty - How well (IMO) does the published roadmap reflect what will really happen

As ever comments welcomed, particularly in this case in terms of what I should assess next.

UpdateTo be clear this is about the Technology vendors, for those looking to start SOA the this is the secondary thing the most important is knowing what the actual services should be.

Technorati Tags: SOA, Service Architecture

Five Nines in a Service Oriented World - Part 1 the problem

SOA systems are more prone to failure than traditional IT systems, SOA systems which rely on "the Web", HTTP or other network centric approaches are more prone to failure than those which rely on conceptual model and then implement locally. I'd say its staggering how Peter Deutch's 7 fallacies of network computing are being ignored in this latest technology driven approach (and I'm talking here about SOD IT rather than Business SOA) but unfortunately its not staggering, its exactly what is to be expected from an industry that loves to re-invent the wheel and where every 3-5 years it looks like yet another bunch of teenagers who "know best" are pushing the new "cool" approach. The teenagers are of course the vendors and the technology fan-boys.

But isn't the "Web" really reliable? Isn't SOA going to make systems more reliable? Errr no it isn't if we carry on designing shttp://www2.blogger.com/img/gl.link.gifystems in the same bad old ways and just thinking that designing for distribution is the same as designing it in a single box. The Internet (as opposed to the Web) is reliable, but that is because it was designed to be, the Web wasn't.

The question raised at my meeting this week was "how many nines would a service need to have to support five nines at the system level" now the mathematically challenged out there (and I've seen them will do the following...
We need an average of five nines... but because of the "weakest" link principle its probably safest to have everything at five nines
And before anyone says "no-one would be that stupid" I've seen this sort of thinking on many occasions. So what is the real answer? Well lets assume a simple system of 6 services

To make this simple the central service A is the only one that relies on any other services and A's code and hardware is "perfect" therefore the only unreliability in A comes from its calls to the other 5 services. The question therefore is how reliable must they be for A to achieve five nines? There is a single capability that has to be enacted on each service and that capability results in change to the systems state (not just reads). The reliability of A is the combined probability of failure of all of the interactions:

So if we assume that all interactions are equally reliable (makes the maths easier) then to find the reliability of A we have:

Note that here we aren't including performance as a measure but it should be pretty clear that performance of a networked request is slower than a local request and that things such as reliability over a network have a performance cost.

Scenario 1 - WS without RM
In this scenario we have standard WS-I without Reliable messaging so we have two points of unreliability, the network and the service itself. This means there are ten interactions. Feeding that into the formula this means that each interaction (and therefore each service) needs to support six nines of reliability in order to deliver five nines for A.

Scenario 2 - WS with RM
Now using RM (or SOAP over JMS) we can partly eliminate network failure from our overall reliability so now we are thinking about give interactions which means 5 interactions which gives us a 99.9998% availability requirement on the services. Still pretty stunning (and a great example of diminishing returns).

Scenario 3 - REST no GET
In this scenario using REST its assumed that the URIs for the resources are already available and no searching is required to get to the resources, this means that returns from each service contain the link to the resource on the other service. This is exactly the same as the standard WS scenario so again its six nines required.

Scenario 4 - REST with GET
In this approach we are doing a GET first to check what the current valid actions are on the resource (dynamic interface) this means that we have twice the number of calls over the network which means twice the number of interactions. This gives a pretty stunning reliability requirement of 99.99995% for the services.

This of course is for a trivial example where we are talking about a single service which calls 5 services which don't call any other services. If we take some of the more "chatty" suggestions that are out there for the "right" way to implement services then we start on down a road where the numbers just get... well plain silly. If we assume that each of the 5 services itself calls 5 services each time its called then the impact goes up in Scenario 4 (6 x 20 interactions) to needing over seven nines of reliability in each of the services, to put that into perspective that is under 3 seconds of downtime a year. That just isn't sensible however so the real question with SOA is

How do I change my thinking to enable services to fail but systems to succeed?

Technorati Tags: SOA, Service Architecture

Two challenges for SOA and Web 2.0

I was presenting to one of our important clients at work this week doing my "what is SOA" pitch around Business Service architecture with another set of technical vendor assessments (due out soon) and the CTO from the client raise a few key points which I think are worth giving some air

1. SOA and Web 2.0 aren't "simpler" in fact its a whole lots more complex, each generation of IT has created more complexity, and SOA + Web 2.0 is liable to be worse
2. Reliability is a nightmare in a distributed service based world

To be honest its great to speak to someone who isn't drinking from the vendor cool-aid but it really does highlight the problems that are currently being glossed over. I've talked before about the challenge of async that everyone seems to ignore but the challenge is actually bigger than just async, its all about the increase challenges around versioning when you have more things to version, its about the increase challenge of worrying about networks and external services and still delivering a decent service.

I'll deal with point 2 in my next post, but the key here is that to make SOA deliver business agility it means that IT will have to deal with a more complex IT problem, that means there has to be a busienss value for the solution or you are better off putting everything in a mainframe.

Technorati Tags: SOA, Service Architecture