Designing for perfection - a fools errand

Ever heard of 2nd project syndrome? Its the concept that engineers when they have delivered a successful first project they will then tend to over engineer the next solution putting in all the bits that they wished they had done on the first project.

YAGNI is my favourite "agile" principle. Now personally I didn't realise it was agile until I was told, I'd been happily asking "why do we need that?" for years without realising it was agile, I just thought it was standard engineering. I've had some discussions recently, online and offline, where fairly smart people seem to be suffering some sort of collective brain freeze around "flexibility".

The basic argument tends to go with me asking how their solution is simple and them telling me how its flexible and will automatically adapt to change in the future. Now sometimes this requires a big bunch of intelligent coding on the client side (i.e. by someone else) but that's okay because the "service" is flexible. Pointing out that there will be multiple clients for the service so in effect its just creating more work doesn't seem to wash. Then you dig a little deeper and find out that there is a bunch of additional development being planned on the service side as well to deliver all this flexibility... with no requirements today that demand any of it.

The other bit of the argument tends to go around whether there needs to be flexibility in a given area. One conversation recently was around submitting a particular financial report and then accessing the information later. The old batch file approach wasn't cutting it (genuinely wasn't) so they wanted to replace this with a new XML/SOA/WOA/WTFA solution. There were three options.

1. Replace the batch with an XML dump (not great as just wrapped the old problem in new technologies
2. Create a single record submission approach
3. Create a fully flexible invoicing management solution

Now the correct answer was 2. but the team pushed hugely for 3 as they could do lots of cool technology and create infinite flexibility and adaptability.

Then I asked a question

there are 30 fields in the invoice, how long has that been the case

The answer was that the 30 fields, of which only 10 were normally used, had been static for about 20 years.

So what was the point of option 3?

The answer was that the developers wanted to demonstrate being technologically clever rather than business clever.

The Clifton Suspension Bridge was over engineered, but at least Brunel [corrected] still just aimed to build a bridge. Some software developers (I won't use the term engineers) seem to believe that unless the bridge can be used as a rocket ship them it isn't flexible enough. For myself I say it gets from A to B, it does what it needs to and its been there over a hundred years.

Business Effectiveness is the goal, not technological perfection.

Technorati Tags: SOA, Service Architecture

SLAs and reporting - whose truth to believe?

One of the core concepts in SOA is the idea that a service should have a Service Level that it agrees to meet (the SLA), this might be technical in terms of up times, response times, amount of information to be handled. In more sophisticated services it could be more business oriented in terms of cost to serve, order to ship time, conversion rate, stock levels etc. These agreements can apply both ways, as in the service commits to respond within 20ms but the consumer can't send more than 20 messages a second.

SLAs are a guarantee that something will be done and there should be penalties in place when a violation occurs. In theory this should be a simple case of measurement, but all too often this is something that is overlooked. People define the SLAs but forget the old adage on KPIs that if you allow someone to measure their own KPIs they will be always be successful.

I've been looking for a simple demonstration of this problem for a while, most of them are specific to a given business so don't work generically. But thanks to the folks in Redmond I've now got a great example, it may or may not be there fault but its a good example of measurement problems.

I run Windows XP under Parallels on a MacBook Pro. Now I just do basic office work so I've given it a C: drive with 15GB. This should be a decent amount for the basic office files (Outlook files are stored on a dedicated share). The trouble is I've run out of space....
So what files are taking up the space? Well a quick "WinDirStat" on the C drive (after doing the same exercise with Windows properties) came up with the stat that the files on the drive take up around 5.4GB. This leaves me with around 10GB unaccounted for.
Here is an example of a producer/consumer SLA. The producer (Windows XP) has committed to provide me with around 15GB of storage for my office files. It then reports that I have violated the client SLA for the service and it will now perform like a dog. Equally my information is that the performance of the producer has severely degraded and I am unable to add more files despite being significantly below the agreed limit.

In this case it is the producer who measures the KPIs and even though the independent measure suggests it to be incorrect it is not possible to challenge the producers statement.

What this says is that when looking at KPIs and SLAs for services you need to think about independent measurement being part of the basic requirement. This implies that measurement is done at the service boundary by a 3rd party which must track these SLAs over time. Otherwise you'll just end up with Windows saying "disk space full please free up some space" and then telling you that you don't have enough files available to make a dent in the space.

The final piece therefore is around arbitration. It is quite possible in the setup that I'm using that something is going screwy around Windows that isn't the fault of the producer or the consumer its due to a 3rd party (e.g. the VM) once you get into this stage you need to think about arbitration. The challenge here is that the consumer is still due compensation from the producer, but the producer may have a counter claim against the 3rd party. This is the final piece about SLAs in a professional SOA environment. Its important to think "back to back" in your SLAs otherwise they won't actually mean anything. If a service commits to responding in 20ms but none of the things it relies upon will make any such guarantee then its just corporate optimism (at best) or fraud (at worst).

SLA management is a core part of the shift of SOA away from technology and into the business domain. With all the WS-* arguments going around I'm still stunned that there is no WS-Contract or WS-SLA because its that which would really separate WS-* from other technical choices.

SLAs are about

1. Defining the terms
2. Defining the penalities
3. Measuring the operation
4. Arbitrating the violation
5. Spreading the risk down the chain

Its the independent measurement that helps make all the rest of it honest.

(Oh and if anyone has any idea about the 10GB I'd love to know where its gone!)

Technorati Tags: SOA, Service Architecture

Software licensing in a virtual world

Reading an article at El Reg on Oracle's licensing model for Sun and thinking about my comments on SOA pricing models there appears to be a fundamental disconnect between the direction that IT is taking and the pricing to get there.

When you move into the virtualisation world the problem becomes even greater. Lets say that I have a blade server with 1000 x 2 core CPUs and on that I virtualise up to have 10 different grid environments made up of 500 x 4 CPU boxes. Now at peak (e.g. for the analytics piece at Tax time) I want to run one of those environment 100% for 1 day, but the rest of the time it takes 1% of the grid or less. Different loading profiles occur for all the environments, but overall I average about 90% utilisation on the blade server.

A smart, and enviromentally strong, use of the hardware. But here is where the business case breaks down.

Every single software vendor has a scale up, not a scale down model

This means that if you are going to use 2,000 cores at peak then you have to license for 2,000 cores even if that peak is only 60 seconds long. This means that the hardware savings are completely eliminated by the software costs. So you just build some "adequate" solutions that can't scale to the peak in the most efficient way (so the Tax calculation takes two weeks instead of one day) but at least you aren't being burnt.

This is one of the biggest challenges right now in deploying Services Fabrics and in consolidating enterprise infrastructures. The solution is to have a more capacity oriented model, which of course the vendors tend not to like as it will inevitably mean less revenue as there is currently a lot of paid for, but rarely used, capacity.

Open Source software clearly changes the dynamic somewhat but clearly it covers only some of the estate. The question therefore is how will software licensing change in a heavily virtualised world? Because if it doesn't that virtualised world won't happen.

Technorati Tags: SOA, Service Architecture

How you know its SOA

Over on the Yahoo SOA list there has been a discussion on how you know if a solution is service oriented. Now beyond the superb OASIS SOA Reference Model I'd say there is a very simple test.

1. Do you have the SOA "picture"
2. Can you map the picture to the implementation?
3. Can you map the picture to how its operated?
4. Can you map the picture to how you are organised?

What I mean by the picture is something like this


Now this is quite a high level picture but if that implementation was truly service oriented then I'd expect to see these services made available with interfaces as defined by the picture. This would mean its an implemented SOA (SOD IT). If in the operation of the system you can show how each of these services is managed as a specific entity and how it is managed inline with its KPIs and Priorities (e.g. like the Manufacturing Service) then its an operational SOA (SOO?). Finally if you can show how the organisation is set up around those business services then you can really say that you are service oriented in your bones.

Looking at the technology isn't the place to see if you are really doing SOA, you've got to look at the architecture before you know that.

Technorati Tags: SOA, Service Architecture

Autosave isn't always your friend - the world of unintended consequences

I'm sure we've all either suffered from or taken advantage of the old "track changes" or history aspects of Microsoft Word, you know the bit where you are able to read what the writer actually wrote first time, like a price of £100,000 before they found out the budget was £200,000. I've seen some bad things in my time on this with people shipping confidential information to a company accidentally by using an old document from one company as the base for a new company.

Now however I've just seen a cracker that comes from the Autosave feature in a certain online tool. The person in question wrote a pretty good document but obviously had a bad phone call at one stage while writing it and let their fingers do the talking

"If that idiot asks me to do one more stupid f***ing change just because he doesn't understand how to breath without assistance then I'm going to tear is f***ing head off. How on earth did that tw*t ever pass an interview."

This text was then rapidly deleted, but not before the autosave had kicked in and added it to the history. The target of the abuse indeed doesn't have the ability to work out how to use the editor, let alone the history, but I let the writer know anyway and we now have a nice new document that just contains the final text.

This came as a bit of a wake up call to me as I regularly will just let my fingers ramble while I'm thinking about what to write next, and I have on occasion written elements in a document for the person next to me to read while in a very dull meeting pretending to do the minutes.

Autosave is a great feature, but it might save more than you intend.

Technorati Tags: SOA, Service Architecture

Formalism over hacking - contracts, compilers and why engineering matters

At Uni I learnt Ada, in my first job I did Eiffel (and C) and in my second I did Ada (and C) as a result of which a bunch of people sent me a link to this article on how Ada delivered the sort of success that C programmers can only dream about. Udi Dahan then had a post over on InfoQ about how a "traditional" approach failed and required a much more engineering solution and some smart contracts.

Now I'm not going to say that Udi's project would have worked better first time in Ada (it wouldn't) but what it does highlight is something that Ada really helped people to focus on..... what is the range of acceptable results. Udi's project suffered because the expected range of data (tens) and the actual (hundreds of thousands) were so massively different.

The solution that Udi's team came up with appears pretty masterful and a great example of scaling a solution by looking at how the big clumps of data are actually assembled. By looking at the ranges and the manner in which they were created they were able to create a really great engineering solution.

Over on the Ada side the article highlights how Ada really upfronts the cost of development by making people really consider what is possible, what is probable and what shouldn't be allowed. By having these contracts it means that everything is explicit and this produces a greater focus on quality at those early stages of the project.

My experience with Ada was that it not only helped in ensuring this quality but that it really helped in ensuring the quality of the average developers on the team. Bluntly point an average developer in Ada was a productive member of the project, an average developer in C was a complete liability.

With all the talk of dynamic languages, scripting and late validation it is worth considering that when it really has to work it is always best to take an engineering approach and look at the contracts and ranges of information and to restrict the system to the correct handling of ranges rather than assuming that the system will be able to correctly handle everything.

As SOA becomes a standard way of operation and the issues of networked computing increase a focus on quality and contracts will become ever more important.

Technorati Tags: SOA, Service Architecture

Change is where the technology isn't

Over the past bunch of years I've seen a consistent thing about SOA "change" programmes, namely that they actually look at only two things.



The green bits on the slide, the technology and the IT architecture. Sometimes the architecture change is only aspirational and its a focus on the technology alone. The point is that the real change is in the purple and blue bits. This is where change actually matters. For an IT organisation to transform itself it needs to really change the purple element and shift away from a technology focus and towards a business focus. This change will have a greater impact on the success of the IT architecture and the technology than any change that is made lower down.

For a business to really capitalise on SOA it needs them to treat SOA like a successful ERP programme, namely to think of the business change needed to leverage the new solution.

If you are focusing on the green bits then all you are doing is an implementation project not a real change programme. Change means changing the way you work, not just the tools that you work with.

Technorati Tags: SOA, Service Architecture

Idea for proximity wakeup call

This is a simple idea to stop people missing their stop on a train or bus station by using a proximity detection from the phone or other mobile device of a person. Once the device is within 10 miles (configurable) of the destination it will emit a loud noise or vibrate in order to inform its owner that they should be awake and ready.

Yes I missed my stop on the train today.

Technorati Tags: Prior Art

Information Oblivion - data only counts where it works

Reading a few email lists recently I've noticed a worrying trend around SOA that can be compared the the "struct" problem of OO. People are trying to look at information independently from its business context. Often this is the single canonical form mentality but its part of a broader problem where people (often called information architects) push forwards an idea that the information is the important bit and that all systems do is move information around and update the information.

A lot of these efforts try and treat the data as an independent entity and aim to create data sources which will act independently and therefore be able to be "used" across the enterprise.

The problem with this is that it works okay for after the fact data, so having a single Customer data source could work, if its just about the basic customer information. Having a historical record of orders is also okay. The point about these bits of data is that they are about recording what has happened. Where this approach falls down is when you try and apply that approach to what is happening. The problem here is that this temporal information only makes sense in the context of the current execution.

Disconnecting data from the business service that manipulates it just means that you have to put the interpretation logic (the bit in the service that understands what the fields mean at different stages) needs to either shift into the data service (so its not a data service) or into every single service that uses the data (which is bonkers).

The basic philosophy that has served me well is that data only counts where it is used. After the fact reporting is a data centric element and suits data stores, this is because the use is really about the information and just shifting, its structs, but where the data is related to activities then keep the data close to the action, dump it into the store later but don't do that until you've finished doing what you need with it.

Information is power, but only if you act on it correctly.

Technorati Tags: SOA, Service Architecture

Terry Pratchett Architects

Now I'm a huge fan of Terry Pratchett and a thought just occurred to me. There are two groups of Architects who have disdain for technology

1. Those that can't cut it in development and have escaped into Powerpoint
   
2. Those who have gone through technology delivery and out the other side

Let me explain. I've worked with some very smart technology people down the years and in my job have the pleasure of meeting a lot of smart technology people (as an absolute number, as a percentage... I wish) and what I've noticed is that there is a small subset of the very best who don't actually appear to like any technology very much. Sure they can use it and make it successful but its always got limitations and its never as elegant as it should be.

The problem is that sometimes its hard to spot the difference between the prat who doesn't understand and is diminishing the importance of technology and the superlative architect who is diminishing the importance of technology because he knows what needs to be delivered but has to get everyone else bought in.

Now one advantage in rooting out the BAs (Bluffing Architects) is that they will use buzzwords all over the place and mostly use language that is more suited towards sociology than engineering. But sometimes they can be cunning and have read all of chapter 1 of a technology book. So here are my guide to checking whether someone is an Architect who has passed through (TPA) or just a Detritus (BA).

1. Drill down - a TPA will drill into detail to find out how much you understand and so what level they have to talk to you. A BA will just say "that isn't important"
2. Vendor X says don't do that - claim that a vendor (IBM, SAP, Oracle and Microsoft are always good ones) has said that you should do something different (e.g. buy a load of software). BA will claim their approach matches what the vendor wants "at an abstract level". TPA will explain why you are almost as much of an idiot as the vendor
3. Say you've heard there are bugs in the proposed products - BA will say that this is the vendors problem and that the analysts rank them as leaders. TPA will say "its a product, of course its got bugs in it, but its pointless building it ourselves" or something similar

The big killer pointer though is their laptop. Not what brand it is as it could be a corporate one. The important thing is what is on there that shouldn't be... and by this I mean an IDE. Pretty much every TPA I've ever met still has at least one IDE installed.

TPAs are the most valuable architects in the IT business, they have earned the right to abuse technology because they genuinely know better. The trouble is that all to often architecture is populated by those who left development but didn't want to become project managers.

Technorati Tags: SOA, Service Architecture

Why REST, WS-* and technology are the problem, not the solution

I've said it before and I'll say it again... SOAP v REST is more pointless than vi v emacs. Building on the previous post and in reference to an article at InfoQ I'm really beginning to feel that IT, and most especially the software part, has some form of terrorist organisation going whose job it is to ensure that the business always looks on IT folks with disdain.

The number of times I've seen the following in a "business" case for SOA in the last 8 years has been immense

1. We need to adopt Web Services
2. We need an ESB
3. We need a business rules engine
4. We need a BPEL engine
5. We need to do "agile delivery"
6. We need to do REST
7. We need to do Web 2.0

and it then follows up with

By doing this we will increase our agility, flexibility and costs and it will only cost double what you currently pay in a year for two years to get it done.

Let me sum up what any sensible business hears when the business case says that....

There is a load of new technology that we want to play with, just like last time we will promise you lots of improvements but when asked won't actually be able to give you any real detail beyond handwaving, and lets face it the last 10 times you gave us the cash we delivered bugger all actual benefit.

Pitching REST, WS-*, ESBs etc is exactly what SOA should not be doing. Its about time that IT started looking at genuine business cases and signing up to explicit measures. Who cares if you use REST, WS-* or flying monkeys to do something, if you've committed to delivering a 10% increase in sales then the choice is yours.

By continually pitching a technology centric view of the world IT will continual to marginalise itself and prevent any genuine progress being made.

Technorati Tags: SOA, Service Architecture

SOA, REST, Packages, Web Services and Hamlet

In a few discussions recently where people have been looking across an enterprise I've had the same discussion in three different ways.

In the first one the future was all about packages, everything was going to be in a package, there would be no more software development as a package vendor did everything that they wanted. When I pointed out that this actually covered only the back-office functions and a minimal part of the sales function there was an awkward moment as they realised than in fact what they really meant was "we are going to standardise our back-office and sales processes around vendor X" this is a good objective to have but the point remained that one solution didn't fit all.

In the second one the world was going to be REST, EVERYTHING was going to be REST, the entire enterprise should be viewed as resources and the principles of the Web would be applied all over, most especially when dealing externally. I pointed out that not only did several suppliers already interact with them via Web Services but that they had a massive backend estate that had a reasonably successful integration strategy. After a couple of hours contemplating the refactoring of a huge number (think hundreds) of systems they decided that maybe REST would just be used around the Web part and for data when its appropriate, and then when there weren't already established approaches.

In the third one the future was all about Web Services, EVERYTHING should be a Web Service. Talking to a Mainframe? Use a Web Service. Talking within an application between one bit and another... use a Web Service. They were even looking at WS-TX. A similar discussion to the REST one then unfolded and now their B2B approach is looking at Web Services and their inter business domain comms will be Web Services.

This all put me in mind of Hamlet and the famous like

There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy

The point is that there is no single answer to a complex IT environment. I've always argued this around Business SOA. Business SOA gives a context not a complete solution. Technology approaches can only ever give part of the solution. Taking a fundamentalist approach to IT is always a bad idea and it does help progress IT forwards.

There are lots of bad ideas in IT, often they are good ideas which are applied in bad places.

Technorati Tags: SOA, Service Architecture

SOA pricing needs to have a business model

One of the challenges of SOA when its adopted across an enterprise is who pays for what, one of the big gaps in most SOA environments is anything that sensibly measures this. Part of the challenge is that in a distributed environment it is hard to always say who is actually using something. As many "security" policies are just application to application (or at best service to service) all you know is that "EmployeeManagement called Payroll". Does this mean that HR (owners of EmployeeManagement) should pay? Or does it indicate a "normal" request that is covered in the costs of finance? What if we use proper end-to-end user security? Does the user pay?

The point here is that creating a pricing model which fairly attributes the costs is a hard problem, but not an unsolvable one. Today there are, within most businesses, areas which are considered "cost centres" and others which are "Profit & Loss" the costs of the "cost centres" are normally centrally allocated and considered overhead which the business has to spend, but where the focus is on cost reduction. In the P&L
areas they are expected to manage themselves for business advantage, this means that when they have impacts from other areas it impacts their profitability. The end result is that a cost model for an organisation needs two things

1. Utility Pricing
2. Attribution

The first is complex, it means you need to know genuinely how much something costs, how many CPU cycles it will take up, how much storage it takes and of course any implicated costs in the interaction. This has an obvious first impact which is that infrastructure (including hardware and software) needs to be measurable at a single interaction level. If one person does a search for a specific entity then its a cheap thing, if they search for a million things then it might be expensive.

The second element is very hard indeed if you try and do it end to end. The "simple" way of handling this is that cost is incurred by the consumer, if this consumer is another service then that service has a decision on whether to pass the cost on, or to absorb the cost. Effectively this means that instead of Transaction management spanning an enterprise there need to be cost management facilities.

Shifting IT onto a business footing means that IT needs to change the way it works. The goal of IT should be to deliver an IT environment that looks like the business, evolves like the business and is costed like the business. Right now there is nothing that is really helping with the later.

While IT continues to be funded via projects and support it will struggle to really adapt to the business, by changing its funding and costing to an integral part of its operation will help IT really deliver the value that it can.

Technorati Tags: SOA, Service Architecture