Making DevOps Business Driven - a service view

I've been doing a bit recently around DevOps and what I've been seeing is that companies that having been scaling DevOps tend to run into a problem: exactly what is a good boundary for a DevOps team? Now I've talked before about how Microservices are just SOA with a new logo, well there is an interesting piece about DevOps as well, its not actually a brand new thing.  Its an evolution and industrialisation of what was leading practice several years ago.

Back in 2007 I gave a presentation on why SOA was a business challenge (full deck at the end) and in there were two pictures that talked about how you needed to change the way you thought about services:

So on the left we've got a view that says that you need to think about a full lifecycle, and on the right you've got a picture that talks about the needs to have an architect, owner and delivery manager (programme manager)
This is what we were doing around SOA projects back in 2007 as a structure and getting the architects and developers (but ESPECIALLY the architects) to be accountable for the full lifecycle.  Its absolutely fantastic to see this becoming normal practice and there are some great lessons out there and technical approaches.

One thing I've not seen however is an answer to what my DevOps team is and how I manage a large number of DevOps teams.  This is where Business Architecture comes in, the point here is that its not enough to just have lots and lots of DevOps teams, you need to align those to the business owners and align them to the structure that is driving them.  You also need to have that structure so one team doesn't just call the 'Buy from Ferrari' internal service without going through procurement first for approval.

So in a DevOps world we are beginning to realize the full-lifecycle view on Business Services, providing a technical approach to automating and managing services that look like the business, evolve like the business and provide the business a structure where they can focus costs where it delivers the most value.

There is much new in the DevOps world, but there is also much we can learn from the Business Architecture space on how to set up DevOps teams to better align to the business and enable DevOps to scale at traditional complex organisations as well as more simple (from a business model perspective) internet companies.

Lipstick on the iceberg - why the local view matters for IT evolution

There is a massive amount of IT hype that is focused on what people see, its about the agile delivery of interfaces, about reporting, visualisation and interactional models.  If you could weight hype then it is quite clear that 95% of all IT is about this area.  Its why we need development teams working hand-in-hand with the business, its why animations and visualisation are massively important.

But here is the thing.  SAP, IBM and Oracle have huge businesses built around the opposite of that, around large transactional elements, things that sit at the backend and actually do the running of the business.  Is procurement something that needs the fancy UI?  I've written before about why procurement is meant to be hated so no that isn't an area where the hype matters.

What about running a power grid? Controlling an aeroplane?  Traffic management? Sure these things have some level of user interaction and often its important that its slick and effective.  But what percentage of the effort is about the user interface?  Less than 5%.  The statistics out there will show that over 80% of spend is on legacy and even the new spend is mainly on transactional elements.

This is where taking a Business SOA view can help, it starts putting boundaries and value around those legacy areas to help you build new more dynamic solutions.  But here is a bit of the dirty secret.

The business doesn't care that its a mess behind the scenes.... if you make it look pretty

Its a fact that people in IT appear regularly shocked at.  But again this is about the SOA Christmas, the business users care about what they interact with, about their view for their purposes. They don't care if its a mess for IT as long as you can deliver that view.

So in other words the hype has got it right, by putting Lipstick on the Iceberg and by hyping the Lipstick you are able to justify the wrapping and evolution of everything else.  Applying SOA approaches to Data is part of the way to enable that evolution and start delivering the local view.

The business doesn't care about the iceberg... as long as you make it look pretty for them. 

Microservices is SOD all within SOA

Microservices is a Service Oriented Delivery approach, all within a Service Oriented Architecture context.
(Long Title ;)

Ok so a few more updates since the last time I wrote about Microservices and I think its worth just updating as it really is heavily underlining why Microservices is a Service Oriented Delivery approach that absolutely can fit within a Service Oriented Architecture.  Lets be clear it isn't a bad thing to be that as delivery is where the rubber hits the road.  But every section that is written further underlines why the 'its not SOA' argument falls very shallow.

Infrastructure Automation
Not going to disagree here as this is just standard practice.  I actually think this isn't far enough forward looking in terms of infrastructure. Technologies such as CloudFoundry, supported by IBM, SAP, HP, Rackspace, EMC, VMWare, Pivotal, etc, move beyond simply automating the infrastructure side towards actually automating deployment at the application level.

This is a really solid SOD tip, certainly something I'd recommend.  Using technologies such as CloudFoundry can really help here and move beyond infrastructure automation.

Design for Failure 
Again a good tip in a service oriented architecture, back in 2007 I actually wrote about the challenges of five 9's and one was about planning for failure in SOA.  Again this is a really solid piece of advice but I feel it over simplifies the challenge.  Sometimes a service has failed because its impossible for it to succeed, particularly the case if using external services.  So 'restarting' is only an option sometimes and its important to understand the types of scenarios that you'll need to handle.

Evolutionary Design
This one comes from the school of the obvious.  The example used talks of a website, the Guardian, that was built as a monolith and was rearchitected to microservices.  I too have such an example of a large airline website that was built as a monolith and was getting hard to maintain and was broken down into various services and boundaries and separately compilable elements, this was in 2004-2006 however so again this really isn't new.

Yes of course you should look at something that will evolve... again a good tip but not something that marks Microservices down as anything other than a Service Oriented Delivery approach.

The goal of a good service oriented architecture is to "look like the business, evolve like the business and be costed in line with the business value" and Microservices lays down some nice rules for implementing certain parts of an enterprise but those are best served by an honesty that its an implementation choice within a broader Service Oriented Architecture.  That doesn't devalue it in any way, just places it in the right context.

Microservices is SOD, all within the context of SOA - and yes this time I added the comma.

I'd also like to put out a hat-tip to Loraine Lawson who pointed me towards the excellent Fallacy poster with the note that the SOA side-bar is really a 'Fallacy Fallacy'. 

Microservices is SOA, for those who know what SOA is.

Ok so its started a bit of debate on Twitter and now there have been emails, but in the spirit of openness I thought I'd better blog.  Now its good that Martin has now added a side bar on SOA to his article on Microservices but that really makes it worse in many ways.  I'll get to that at the end but first off lets explain why Microservices is just another SOA implementation pattern.  Its SOD for SOA

Lets break down the key precepts of Microservices and compare them against the OASIS SOA Reference Model (published 2006)

Componentization via Services

There follows a long text that can be reduced to

"Service Oriented Architecture (SOA) is a paradigm for organizing and utilizing distributed capabilities that may be under the control of different ownership domains. " 

There is an awful lot of words about deployment and process models in the Microservices piece but this single sentence at the start of the OASIS SOA RM is much more powerful because

1. It means that IT and non-IT services can be represented in a single approach
2. It immediately includes the major challenge - that of politics and ownership

OASIS then goes nicely into WTF actually is a service,

• The capability to perform work for another 
• The specification of the work offered for another 
• The offer to perform work for another

Again this is not a tech requirement which means your architecture can actually start from a business perspective rather than process threads.  In Fowler's microservices textual description (no bullet points here) we have

• services are out-of-process components who communicate
• explicit component interface
• ..... nothing on this bit 

OASIS then adds the kicker

in SOA, services are the mechanism by which needs and capabilities are brought together.

Remember that phrase 'mechanism' its going to be important....

Organized around Business Capabilities

Here is where Martin begins to get it wrong.  The OASIS SOA RM defines a capability as 

A real-world effect that a service provider is able to provide to a service consumer

The point here is the difference between a capability (the bit that does the work) and the service (the organising construct) is really important.  What we found when doing SOA in the wild for over a decade, and all the people on the OASIS SOA RM had lots of experience doing this, was that the organising framework was separate from the actions themselves.  The reason this is crucially important is that people started often making services where service = capability so you ended up with lots and lots of services (ummm if I was being insulting I'd call them microservices) here are some posts from 2006 and 2007 that explain why its really not great to confuse the two concepts and it made it into the SOA Antipatterns as well.  Now the actual text is pretty much ok, but again its lack of reference to the past and making a crucial mistake does not help people learn how things are evolving and what can be improved.

New?  Hell even I wrote a book which talked about how to model, manage and set up teams around this approach. The SOA Manifesto (2009) talks about key principles behind SOA (I still prefer the RM though) from a big group of practitioners.  The point here is that there are two problems, first the confusion of service and capabilities and secondly the lack of recognition of hierarchies importance in governance.

Products not Projects

Here it goes  beyond the OASIS SOA RM which doesn't talk about delivery models... however fortunately it goes beyond absolutely nothing that was said before.  A quick hit on Google just shows how many pieces there are in this space, some better than others and some products better than others but really this is not new.  I used to use the phrase 'Programs not projects' and always talked about assigning architects for the full lifecycle 'to make them accountable'.    Again its not that this statement is in anyway wrong, its just that its in no-way new.  We've known that this has helped for years but it has a significant issue: cost.

Lets say you have a service that requires some amazingly smart analytics, some hard core coding and some wonderful UI design.  You hire the very best, they cost a fortune and your service is awesome.  Do you really want those folks sitting around waiting for requirement changes?  No.  This is why for me the concentration is at the architecture and ownership level that consistency must be maintained, not at the developer level.  For companies where software development is their business you can include development but for most organisations in the real-world economics prevent the 'you built it you run it' mentality.  Again its a lack of learning that undermines the message.

Here I'm not disagreeing, it would be super ideal to have the same team always maintaining the code they write, its just not practical but thinking in terms of discreet pieces with their own heartbeats is a good thing.  Most decent SOA programs I've seen have recognised this and had different delivery schedules for different services.  What becomes important is the integrity of the service and its ability to be independently maintained, within the context of its management hierarchy, relegating this to 'keep the same developers' misses some of the power of SOA.  Again this emphasises that Microservices is just another SOD approach, a potentially good one in some circumstances but not something that will work for everyone and every circumstance...

Smart endpoints and dumb pipes

I really agree with this, but I think its better put in a different way.  In the OASIS SOA RM there is the concept of an 'execution context'.  Which is the bit that lets a service be called and its capability invoked.  Clearly the end-point is 'smart' as its what does the work, hence the phrase 'mechanism' used above.  The 'pipes' may or may not be dumb (the 'pipes' talking to those Rovers on Mars are pretty smart I think) but what they are is without value.  This was a crucial finding in SOA and is well documented in the SOA RM.    The execution context is where all of the internal plumbing is done but its the Service that provides the access to the capability and its the capability which delivers the value.

So its not wrong to say 'smart endpoints and dumb pipes' but its better to say that the end-points have the value and the pipes are just infrastructure, this gives a better guidance on why you shouldn't be focusing on the pipes.

Now it does make a good point about not having smarts in the information fabric, but again this isn't new or unusual in decent SOA implementations.  I collaborated with the folks from Progress Software back in 2007 on the Business Service Bus concept which is just about having mediation (security, basic transformation) in the Bus.  There are good reasons why these things go there, cross-referencing between different ontologies is one.  This also plays into the 'always proxy' pattern that most decent SOA folks did.

So its again not wrong, its just that its not new, and it further underlines the inherent SOA nature of Microservices.

Decentralized Governance

This really is another bit that I really do agree with, I've talked about the People's Democratic Republic of IT for years at conferences and finally got around to blogging on it.  The whole principle of business driven SOA is that the governance model better matches the business.  So again I think its not bad advice its just that SOA gives so much more than Microservices in terms of governance.  SOA as described in the OASIS SOA RM allows these principles to be applied to all IT assets not just those implemented with a specific implementation style and indeed to just to IT assets meaning its an approach that business schools are teaching.

Microservices and SOA

The last thing I'll cover here is how the SOA sidebar is really a red herring, its not a true definition of SOA instead rolling out the old 'big' ESB and WS-* trope that was so loved by the RESTafarians when explaining why their way was better.  The claim is that this article 'crisply' describes the Microservices style and thus is valid in comparison with SOA as 'SOA means so many things'.  This I fundamentally disagree with, firstly because Microservices would be better served as an implementation approach if it could explain how it fits with non-Microservices approaches, something that SOA does a great job of doing, and second that it can't even say what makes a service 'micro' with services ranging from decent sized (12 person) teams down to individuals.

Conclusion

This for me is why Microservices is just a Service Oriented Delivery approach for a well architected SOA solution.  SOA provides the contextual framework, provides most of the rules that Microservices aims to adhere to but more over gives a broader context within which Microservices fit within a complex enterprise.  Calling out WS-* for the one millionth time or 'big' ESB and talking about massively complex projects is simply a shot at a different challenge.

Additionally the fact that one of the references that is used is that of Netflix who actually use the term 'fine-grained SOA' as recognised in the footnotes sort of underlines the fact and the fact that another (Amazon) also says its SOA.

I think its great that SOA is now coming back to the fore in the market as the hype around the plumbing (WS-* v REST) dies down and that the learnings of companies who have been doing this for over 10 years is now being talked about.  But that is the way to talk about it: what is the state of the 'now' in Service Oriented implementation and architecture?

Microservices - Money for old rope or re-badging SOA for the cool kids

Hat tip to John Evedemon for the heads up on this one.  Martin Fowler is peddling a new approach, 'Microservices' which... wait for it is a way of developing applications as a suite of services.  Each one of which has its own process thread and 'communicates via lightweight mechanisms' such as.... over HTTP.

But wait there is more, you'll be stunned to know that these services can be built using different programming languages and even use different data stores.

Now down in the footnotes it makes a reference to Netflix talking about 'fine grained SOA' so its there that we begin to get the sniff of an old idea wrapped up in some shiny new wrapping there are a few things you need to do when saying this.  The first is critical don't learn or reference previous approaches except negatively.

Most languages do not have a good mechanism for defining an explicit Published Interface.

Now I really wish there was an approach that enabled a Service to publish a definition of itself of the Web, a sort of Web Service Description, if only there was such a language I might call it... oh I don't know Web Service Description Language.  The rest of the article talks about things that were common discussions around 2001, making things independently deployable but recognising that interface changes can have knock on impacts.  Hell I could even think of an Interface Definition Language or IDL that might do that as well.

Martin Fowler really should know better than this in not paying any heed to what has gone before and promoting an approach as if its actually new.  The article reads like an extremely basic description of SOA from about 2000 without either the industrialisation of WS-* or the dynamic power of things like Jini.  Above all it doesn't move forwards the question of how to architect such service architectures and how they need to map to the business and how although there might be fine grained services it is critical to understand the management structure and hierarchy of those services to actually enable the degree of autonomy and flexibility required in these sorts of architectures.

Microservices is just another take on SOA and one that doesn't move the game forwards as it yet again focuses on technical aspects over the business architecture that is realised.  Putting forwards as new something that would be recognisable to people working on CORBA in the 90s and certainly Web Services in 2001 is just poor quality advice.  It neither learns from the past, educates the reader nor moves the game forwards, its just selling an old approach with a new name.

Microservices?  What is that SOA 3.0?  Nope its just an old school form of SOA without the learnings that came from doing that.

 

Why in Business driven information its the consumers view that matters

When doing the Business Data Lake pieces it took me back to a view that I had around SOA in that you should take the consumers view when designing a service.  This I think is more critical when looking at analytics and reporting where it really is all about the consumption.

What does this mean though to think about data from the consumers perspective?  We've all had the '3 V's' shoved at us and mostly realised the one that counts in Big Data is actually value.  So taking a Business SOA look at data means that you need to think about the business view and crucially the business value to understand what this actually means.

To this I'd say there are three ways that you can measure whether something is business driven or not

1. The Natural View - is the view on information being presented one that is naturally understandable by a given part of the business
2. Value based cost - does the cost being charged reflect the value being delivered
3. Dynamic Performance - ramp-up, ramp-down as the demand requires

These come back to a mantra I used a lot back when I was talking about the true impact of SOA on an IT estate: Create an IT estate that looks like the business, evolves like the business and is costed based on the business value it delivers.

With SOA in the operational sense this meant having a clear Business Service Architecture and that thinking now applies to Data.  There is no difference between the business views and KPIs between the operational and post-transactional world, indeed the more that analytics becomes the difference in operations the less that difference can be tolerated.

The difference however is in how information is accessed.  In the operational world when you want information from another domain you request it on demand.  In the post transactional world however this is about providing access to the landed (stored) information from other areas in the context that a given domain wants to see it.  Its here that new technologies add value as they enable that distillation to be done into the right business context, indeed enabling the same information to be distilled by different business areas in the right way for their context.

This is the same as you do in the operational space, requesting information from another business service and then converting the result into what makes sense in your local context.

By having a single consistent model between both the operational and post-transactional world you make integrating analytics much easier as you are not requiring your consumers to mentally shift between an operational view and the local view.

So as with a business service architecture which represents the business model so now that business consumer view should be reflected in your analytical applications to create a single model for IT that spans both operations and analytics and now gives the business that local view, the natural view for them, of information.

Now to the other two pieces: Value based Cost and Dynamic Performance.  These two are linked as value is not something that is fixed over time, closing the books is something where high performance is justified at a quarter end or other accounting deadline, but having a high performance solution when not required is wasted cost.

Therefore the new generation of solutions are about Elastic Analytics, that is analytics which can adapt and change based on the business demand.  There needs to be an end to the 'I think I might need in memory so I'll put it all in-memory' or worse the 'Damn I had it all on disk and now I need in-memory'.

The future is about Analytics aligned to the business not aligned to an idealised IT view.

How Business SOA thinking impacts data

Over the years I've written quite a bit about how SOA, when viewed as a tool for Business Architecture, can change some of the cherished beliefs in IT.  One of these was about how the Single Canonical Form was not for SOA and others have talked about how MDM and SOA collaborate to deliver a more flexible approach.  Central to all of these things has been that this has been about information and transactions 'in the now' the active flowing of information within and between businesses and how you make that more agile while ensuring such collaboration can be trusted.

Recently however my challenge has been around data, the post-transactional world, where things go after they've happened so people can do analytics on them.  This world has been the champion of the single canonical form, massive single schemas that aim to encompass everything, with the advantage over the operational world that things have happened so the constraints, while evident, are less of a daily impact.

The challenge of data however is that the gap between the post-transactional and the operational world has disappeared.  We've spent 30 years in IT creating a hard-wall between these ares, creating Data Warehouses which operate much like batch driven mainframes and where the idea of operational systems directly accessing them has been aggressively discouraged.  The problem is that the business doesn't see this separation.  They want to see analytics and its insight delivered back into the operational systems to help make better decisions.

So this got me thinking, why is it that in the SOA world and operational world its perfectly ok for local domains and businesses to have their own view on a piece of data, an invoice, a customer, a product, etc but when it comes to reporting they all need to agree?  Having spent a lot of time on MDM projects recently the answer was pretty simple:

They don't

With MDM the really valuable bit is the cross-reference, its the bit that enables collaboration.  The amount of standardisation required is actually pretty low.  If Sales has 100 attributes for Customer and Finance only 30 and in-fact it only takes 20 to uniquely identify the customer then its that 20 that really matter to drive the cross reference.  If there isn't any value in agreeing on the other attributes then why bother investing in it?  Getting agreement is hard enough without trying to do it in areas where the business fundamentally doesn't care.

This approach to MDM helps to create shorter more targeted programs, and programs that are really suited to enabling business collaboration.  You don't need to pass the full customer record, you just pass the ID.

So what does this combination of MDM and SOA mean for data, particularly as we want analytics to be integrated back into operations?

Data solutions should look more like Business SOA solutions and match the way the business works

In simple terms it means the sort of thinking that led to flexibly integrated SOA solutions should now be applied to Data.  Get rid of that single Schema, concentrate on having data served up in a way that matches the requirements of the business domains and concentrate governance on where its required to give global consistency and drive business collaboration.  That way you can ensure that the insights being created will be able to be managed in the same way as the operational systems.

With SOA the problem of people building applications 'in the bus' led me to propose a new architectural approach where you don't have one ESB that does everything but accept that different parts of the business will want their own local control.  The Business Service Bus concept was built around that and with the folks at IBM, SAP, Microsoft and Oracle all ensuring that pretty much everyone ends up with a couple of ESB type solutions its the sort of architecture I've seen work on multiple occasions.  That sort of approach is exactly what I now think applies to data.

The difference?

Well with data and analytical approaches you probably want to combine data from multiple sources, not just your own local information, fortunately new (Java) technologies such as Hadoop are changing the economics of storing data so instead of having to agree on schemas you can just land all of your corporate data into one environment and let those business users build business aligned analytics which sit within their domain, even if they are using information shared by others.  MDM allows that cross reference to happen in a managed way but a new business aligned approach removes the need for total agreement before anything can get done.

With Business SOA driven operations we had the ability to get all the operational data in real-time and aggregate at the BSB level if required, with Business SOA driven data approaches we can land all the information and then enable the flexibility.  By aligning both the operational and post-transactional models within a single consistent Business aligned approach we start doing what IT should be doing all along

Creating an IT estate that looks like the business, evolves like the business and that is costed in-line with the value it delivers.

Applying Business SOA thinking to data has been really interesting and what led to the Business Data Lake concept, its early days clearly but I really do believe that getting the operational and data worlds aligned to the business in a consistent way is going to be the way forwards.

This isn't a new and radical approach, its just applying what worked in operations to the data space and recognising that if the goal of analytics is to deliver insight back into operations then that insight needs to be aligned to the business operations from the start so it can adapt and change as the operational business requires.

The boundaries from the operational and post-transactional world have gone, the new boundaries are defined by the business and the governance in both areas is about enabling consistent collaboration.

REST isn't undead in the enterprise... its still born

Its always depressing to see fanbois bleating and moaning about their beloved technology or piece of bling not being universally liked. This is normally put down, in a wonderfully immature way, to the failure of "the other side" to see their point of view rather than any innate failings of their beloved approach. SOAP is not Dead - Its Undead is a classic of the genre.

Why hasn't REST succeeded in the enterprise? Not of course because it isn't actually any better than SOAP for enterprise scenarios and is indeed much, much worse in many. Nope.

But first there is the reason why REST is successful

His presentation showed that 73% of the APis on Programmable Web use REST. SOAP is far behind but is still represented in 17% of the APIs.

This is like going to France, doing a language survey and declaring that French is the most popular language in the US. So what would doing the same query on the likes of Oracle, SAP, IBM or Microsoft's enterprise technology stacks deliver? I assumed the number to beat would be huge and got ready for some serious searching.... but the number to beat is 2368... errr seriously? I've worked in single enterprises where they had more SOAP endpoints than that. When you include the libraries of WSDLs from SAP and Oracle and the Behemoth that is Oracle AIA has so many that Oracle don't boast about it as it might make it look complicated. Back in 2005 folks at Oracle boasted about over 3000 web services across their applications. Now before people bleat about this being proof of SOAP complexity... that just makes you a hypocrite if you on one hand use the ProgrammableWeb stats as "proof" of RESTs success but then try and use the massive volume of WSDLs out there as proof of SOAP's "complexity.

I'm also here not even into the global standards that use SOAP every day for B2B, people like SWIFT, Open Airlines... shall I go on and on? 2400 APIs is a success? SOAP isn't anywhere near that? Like I say its like going to France and claiming French is the most spoken language on the planet.

All this just proves what I've said for a long time. REST works for information traversal, but its not set up for the enterprise. So what is the issue with REST not displacing SOAP in the enterprise?

"All the tools, hires, licenses & codebase has been built around SOAP for a decade," Loveless wrote on Twitter. "Hard to turn on a dime."

Wow, the bare facedness of this statement is hard to beat. REST has been kicking on this door for over half of that time and some folks argue that in fact it predates SOAP. So it really is bullshit to claim that its all the fault of tools & codebase. SOAP replaced old EAI approaches in a couple of years in new enterprise projects. We went from a situation with everyone in about 1998 doing proprietary EAI integration, with occasional CORBA for RPC, to everyone by 2002 doing Web Services with some JMS. People in 2005 were telling me that REST was the future and REST would win, and now SIX YEARS LATER people are bleating about 10 years of SOAP adoption...

If an approach is better for integration in the enterprise it will be adopted. REST isn't better, yet, for enterprise integration because it fundamentally remains a developer approach not a professional enterprise approach. SOAP isn't complex, technically it might suck (hell my father said "great we've now got enough processing cycles to burn that ASCII-RPC has finally made it") but conceptually its simple, and when managing complex estates with lots of different people that conceptual simplicity on the head.

Michael Cote of Redmonk hits the nail partially on the head when he says

"As enterprise development teams start including cloud technologies in their applications, incompatible cloud platforms and APIs will be a huge road block," said Michael Cote, analyst at RedMonk. "We're already seeing a clamoring for tools and services that integrate this spaghetti bowl of end-points, and they're only going to become more important to realizing the benefits of cloud development."

In other words the lack of a formal contract and standard interface mechanism remains the real reason why REST isn't being adopted in the enterprise.

What SOAP did was solve a problem that the enterprise had. How do I describe integration interfaces so my systems on different technology stacks can communicate and do so in a way that enables my teams to work independently of each other. REST does not solve this problem in an effective way and bleating about "dynamic interfaces" being "better" misses the whole point of what has made B2B and Machine 2 Machine integration successful down the years, namely a focus on people-centric approaches rather than technical centric ones.

Unfortunately with REST there appears to be an active movement to stop this professionalism creeping into it and defining new standards that will actually make REST better for the enterprise.

REST needs a standard way to publish its API and for a way to notify changes in that API. This is a "solved" problem in IT but for some reason the REST community appears to prefer blaming others for the lack of enterprise success of their technology rather than facing up to the simple reality:

SOAP got some things right and the biggest thing it got right was a shareable and toolable contract (WSDL) which enabled interfaces to be published in a standard way which included by functional and data standards.

SOAP isn't undead, its very much living in the enterprise and indeed being the only real viable approach when integrating package solutions from a number of vendors (a massive piece of enterprise IT). REST however barely registers, less than 2500 APIs after all these years of development? Pathetic.

REST for the enterprise isn't undead... its been still-born for over five years.

Technorati Tags: SOA, Service Architecture

The three magic questions of business SOA (and IT Strategy)

Having a chat with someone today I was reminded of a presentation I gave a few years ago. I went to the client knowing that they'd spent a lot of money in the last few years on an IT "refresh" and the word SOA had been used rather a lot. The programme wasn't seen as being a success and the business were bitching that money had been wasted.

My opening point was that the business was sort of right and so I posed three questions

1. Do you have a clear vision of where you want your IT estate to be in 3 years
2. Do you want the IT estate to reflect that vision
3. Do you want the costs for IT to reflect the different values in that vision

The answers were "No, Yes, Yes" which wasn't a huge surprise but was my real point to both sides of what had turned into a political mudslinging competition.

IT had set off to try and do 2 and 3 without knowing 1 and the business had effectively said "we don't know what we want... but we know its not that". Neither of these was a good idea. So the focus therefore was on creating that vision (a business service driven vision) and doing the heatmap that meant IT could then align itself (number 2 and 3) to that vision.

Now I've said before that IT and Chairman Mao have a lot in common but there is another famous phrase that applies here

"A journey of a thousand miles begins with a single step"

Now that might be true, but its really important to know where you are going otherwise you look like a bit of a pillock in a thousand miles.

So can you answer the three magic questions of Business SOA and IT strategy?

Technorati Tags: SOA, Service Architecture

One year on: zero progress for Enterprise REST as Java races backwards

Just under a year ago I blogged about how REST and Sun had put Enterprise IT back five years so I thought it was about time to update that view and see what has happened in the last 12 months in the enterprise integration and governance space.

So on the REST front we've seen.... ummm.... struggling here.

Lets be clear, I'm talking here about REST as an enterprise integration approach, not as a way of exposing a Web API for content aggregation but as a functional integration approach for enterprises. Something to replace the "fundamentally flawed" WS-* that REST is so much better than. So what is the progress this year? Zip, zero, nada. Yup a few minor tweaks into enterprise stacks that say they can produce REST interfaces, but in reality most of them can't and the key problems of interface publication, versioning and testing remain unsolved.

Am I being harsh on REST? I don't think so. Its had more than enough time, and hype, to address the real problems of enterprise computing and step out of the niche, and in revenue terms it is a niche, of Web content aggregation. REST is great for that niche, but that wasn't the pitch made, the pitch made was that REST was great, WS-* sucked and REST would solve all the problems that existed with WS-*. This hype, and the smart people who followed it, has led to a stagnation in enterprise integration that is really hitting enterprise computing in real revenue terms. Its slowing the adoption of cloud computing and generally meaning that IT departments are less credible and less successful than they should be.

So one year on and REST continues to flatter to deceive.

What about Java? Well here the story is actually worse. I honestly believe that the REST crowd are a smart bunch of bunnies and are trying to solve problems, just not realising that most developers aren't as smart as them and that interface dynamism is actually a bad thing. With Java that is sadly not the case. With Mark Rheinhold now actively subverting the JCP and the debacle in the JCP generally over the last year its hard to think that the situation is doing anything other than getting worse.

What does this mean for Enterprise IT? Well a couple of things, it means that SAP, IBM and Oracle the three headed beasts of most IT estates don't have a clear future integration improvement roadmap, its all still based around WS-* as it was 4 to 5 years ago and they are all adding proprietary "tweaks" which "help" people when using their platforms. It also means that their core platform, Java, is stagnating and not getting some of the fundamental changes it needs to address the cloud and dynamic scalability. In particular the continuing "kitchen sink" approach of the Java dictatorship means that reduced profile VMs that are tuned to specific tasks (like Mobile potentially) just aren't being addressed which is leading directly to a fragmentation in the core operating platform.

What does then mean? Well with REST it means that enterprise IT shops are relying on WS-* for integration but still coming across things like vendors not having WS-Security support and with the decline of WS-I the number of "strange" defects is liable to be on the rise. Efforts around more formal contract approaches are dead. This means that the "spaghetti" of enterprise IT, which looked to improving in the first 6 years of the new millenium is actually now getting worse again. REST aimed at WS-* squarely and surely and has certainly hit it with a wounding shot, unfortunately it turns out that REST is incapable of being the replacement for WS-* it wanted to be. REST is Brutus, WS-* is Caesar.

And with Java? It means we are seeing language fragmentation and platform fragmentation which means that the support costs of IT estates are going to rise and so the on going challenge of reducing operating costs to enable investment in new solutions is swinging back against the business and towards entrenched IT estates.

I really can't see how a large scale enterprise IT programme is better off in 2011 than it was in 2005.

Technorati Tags: SOA, Service Architecture

MDM, SOA and BPM - making processes simple

BPM is once again a hot topic. It seems several years ago where I said "if you want to get enterprise cash, learn BPEL"... in fact it was 5 years ago! However BPM really hasn't delivered on the promises and that is for three reasons

1) The Promises were too big
2) The implementations were too complex

3) BPEL rapidly became a Visual COBOL approach

Now I've said before that BPM screws up SOA and that SOA makes BPM simple, I'll stick with that and I'll now raise another piece in conjunction with SOA and MDM. This is actually a way to make BPM work more effectively, more simply and reduce the issues around elements such as process [de|]hydration. Namely that if you are in an area where BPM is the best implementation approach for a specific service's capability then using MDM helps to make its implementation simpler as you can work on key exchange rather than on content exchange.

Rather than BPM folks worrying about the schemas and exchange of core information that is left to MDM (where it belongs) and the SOA pieces handles the business domains part to make sure processes have clear boundaries and can be well managed.

Technorati Tags: SOA, Service Architecture

Data Services are bogus, Information services are real

One of the questions that used to be asked, or proposed as fact, in old school SOA was the idea of "Data Services" these were effectively CRUD wrappers on Database tables and the idea was that they were reusable across the enterprise. I've said many times this is a dumb idea as the importance is actually about the information, which means context and governance.

Now the other day when I was talking about MDM a bright spark pointed out that I hated data services but wasn't MDM just about data services?

Its a good challenge, mainly because that is the problem about how many people view MDM. MDM is, when done well is about the M and the M not the D, i.e. its more about Mastery and Management than it is simply about Data. What does that mean?

Well lets take everybody's favourite MDM example "Customer" a Data driven approach would give us a Service of

Service Customer

• Capability: Create
• Capability: Update
• Capability: Delete
• Capability: Read

Now this is the "D" approach to MDM and SOA, also known as the Dunce's approach, its about Data Services and viewing the world as a set of data objects.

The smart approach is to view MDM as an information challenge and delivering information services so instead of the data centric approach we get

Service Customer

• Capability: Establish Prospect
• Capability: Establish Customer
• Capability: Modify
• Capability: Change Status
• Capability: Find
• Capability: Archive Customer
• Capability: Validate Customer
• Capability: Merge Customers
• Capability: Split Customer

Here we start exposing the customer service and make clear two things

1. We are talking about the customer in context
2. We reserve the right to say "no" to a request

So this is where customer genuinely can be used from a single service across the enterprise. This service takes on the responsibility to authorise the customer, or at least ensure that authorisation is done, it sets the quality standards and governance standards and doesn't allow people to do what ever they want with it. It includes the MDM processes elements around customer management and provides a standardised way of managing a customer through its lifecycle.

This is fundamentally the difference between a business SOA driven approach which concentrates on the services in their business context and with their business governance and a technical driven approach which looks to expose technical elements as via a technical wrapper.

Technorati Tags: SOA, Service Architecture

The business don't care about technology that works

I had a great conversation with someone today talking about a project where the business really doesn't care about the technology. Why don't they care?

Because they are getting the business outcome they want

That is the key point on why they don't care. Because the technology is providing them with what they want. They couldn't care less whether its a new shiny implementation using REST or some clunky old solution using string, spit and a bloke called Dave with hammers.

All too often IT folks make the mistake of trying to develop and "operationally perfect" solution from a technical perspective but which takes longer to get there from a business perspective, thus meaning that its perceived by the business as a failure and all they are hearing from IT is a list of wonderful technologies which are the reason why they aren't getting their business outcome.

The same goes when the programme goes over budget due to the customisations of the package or the unexpected complexities of the cloud solution. The excuses are almost always technical but all the business cares about is the outcome.

So next time you are looking at the architecture and thinking REST, WS, Dynamically scalable et al just ask yourself this simple question

If I made it simpler and didn't aim for technical perfection would I deliver the business outcome quicker?

Because the business doesn't care how you do it... just that it is done

Technorati Tags: SOA, Service Architecture

I don't care about cloud because I don't care about tin

I'll start with an admission, I've worked with cloud providers for quite a few years now and the reason is not because I'm excited about elastic scalability of compute and storage... its exactly because I don't care about elastic scalability of compute and storage. I've said before that Tin Huggers are a major blocker to cloud adoption... but now I wonder if cloud itself is actually part of a broader problem...

Cloud is just tin, virtual tin it doesn't actually have a point, it doesn't actually do anything...

Great cloud services such as Amazon AWS are great not because they provide a bunch of tin, but because they provide a set of services which enable the virtualisation of tin. You aren't buying tin its the service, but all the service provides is virtual tin.

SaaS however is different because SaaS provides you with some business capabilities, you are buying a set of business services and you are buying them "as is". Cloud is in one sense just a revision of current IT models where you are building your stuff on virtualised infrastructure, sure its a bit more dynamic but at the end of the day you are still building your own stuff the only difference is that rather than it being hosted in a data centre you never visit but on tin you've bought its hosted in a data centre you don't even know where it is on tin you are renting.

So my point is that cloud is in one sense dull, its the same reason I don't care about the telephone infrastructure, sure I make phone calls and I'm glad its all there, but its the phone and the services I care about, the infrastructure can go hang. This doesn't mean cloud or the phone infrastructure aren't important building blocks for lots of things and clearly SaaS builds on cloud, but equally clearly SaaS is the point that (in the words of the OASIS SOA RM) delivers the real world effect while Cloud just forms part of the execution context.

Lots of cloud marketing and words out there is really just old style IT with a minor bit of lip gloss applied by using "cloud", but does that actually deliver a better service to the business? Sure sometimes the dynamism is good, but sometimes you'd be better off just buying a SaaS service and people are using Cloud interchangeably with SaaS to deliberately muddy the waters and pretend that by doing Cloud they are in fact really doing SaaS and being more business centric.

Cloud is IT centric, SaaS is business centric.

And that is why I care about SaaS and don't care about Cloud. I want to know what services the business can run not how "dynamic" or "scalable" the tin is, I've heard those conversations all my career and they've always bored me. Software is scalable, tin just gets bigger (horizontally or vertically). Cloud is a diversion, sometimes its a successful diversion but in 80%+ of cases SaaS is the true revolution and confusing it with virtual tin isn't helping move us forwards.

Clouds are boring, because Clouds do nothing, its what you run on Clouds that counts and most of the time SaaS is better than old style custom build but on a shinier set of tin.

Technorati Tags: Cloud, Service Architecture

Why MDM and SOA are shifting out of IT

Now I've said previously why MDM is required for successful SOA but there is another important piece around MDM and SOA that is happening at the moment and explains both why MDM and SOA haven't historically gone together and, crucially, why the new trends are liable to help bring them together next year.

Why SOA and MDM didn't go together

The first question is why historically MDM and SOA projects tended to not go together. Sure organisations would "do" SOA programmes and would "do" MDM programmes but rarely would the SOA programmes and MDM programmes be tightly joined. There are I think three reasons for this

1 - Different bits of IT

The first challenge is that MDM and SOA were normally done by different bits of IT with different mentalities. MDM was done by the "data" folks who worried about data warehouses and saw data as the most important thing in the world. SOA was done by the enterprise integration and architecture folks who worried about operational processes and integration. The MDM folks tended to have a post-transactional view of the world where things were "true" or "false" while the SOA folks tended to view things as "in processes" or "completed".

These different communities in IT have very different backgrounds and focuses. ETL, Data Warehouses and big databases rule in the MDM/Data side while fast transaction throughput is the key for the SOA folks.

2 - Different bits of vendors

The next challenge is that the vendors have exactly the same split in their view of the world. Look at IBM for instance. SOA is in the WebSphere brand while MDM is in the InfoSphere brand, two very different parts of IBM Software with independent management structures and teams, sure the idea is that they should co-operate and work together but we all know that different divisions in companies like to do things differently. This also means that you often get different sales people for the different pieces of software. Oracle for instance put MDM in the Applications area while SOA is in the Middleware space, if you want to use their MDM products with their middleware products (for instance using the MDM PIP) this means you have to deal with two different sales people to get what you want.

3 - They've been IT projects

The final reason is that MDM and SOA have traditionally been internal IT programmes owned and managed by IT and largely invisible to the business. This means that the two cultural elements above are then hard-baked into the solution which ensures that the SOA and MDM programmes are kept distinct.

Why its changing

So why is it changing? Well the first reason is that people have started finally realising that SOA is a business thing and needs to be viewed from a business architecture perspective. This has been a long time coming but finally it appears people are thinking about the challenges of business services, service value and the business architecture. At the same time MDM is shifting as well, firstly its shifting away from post transactional into an operational space which means it needs to consider operational processes and performance in a way it hasn't before, this means that the MDM/Data folks now have to not only talk to the SOA folks but they have to... shock horror... actually become MDM/SOA folks.

MDM is also shifting in that business people want to do more active information management, both in terms of newer analytical tools and secondly in terms of including that mastered and high-quality information into core operational processes. This means that the business realises that the old "data landfill" approaches which were poor before are massively hindering the analytical models and have a direct impact on the efficiency of the operational processes. By taking control recognising that Information needs Mastering the business is now actively taking over those processes and thus MDM is moving out of the background into being a key part of the corporate information strategy.

So that is why MDM and SOA are shifting out of IT, its because being within IT made them technically centric programmes that often failed to deliver the promised value. By enabling the business to more actively own its IT estate, through Business SOA, and its core information, through MDM, it suddenly ceases to be a question of competing IT cultures but a simple question of how to present a consistent operational and analytical view of the business.

SOA and MDM are made to be together.... that fixes the enterprise culture... but will it fix the vendors?

Technorati Tags: SOA, Service Architecture

MDM isn't a hub - its just simpler to implement it that way

One of the things about MDM that people often get wrong is the idea that MDM provides a central information hub around a given entity and its relationships.

It doesn't.

MDM provides 3 core facilities

1. Cross-referencing of core entities between systems
2. Standardisation around the critical "matching" attributes
3. Synchronisation of attributes modified within multiple systems

Only one of these really requires some form of centralisation, the x-ref, the rest can be handled via governance processes and integration processes without requiring a central system. You can implement match and merge within the integration layer or end application then propagate those changes along.

However in the world of Simple IT and "doing one thing well" its liable to be much more effective to have an MDM solution that manages this integration and which is designed to do that integration rather than building it all yourself.

MDM doesn't require a central solution, you'll just probably find it simpler that way

However there are good reasons

Technorati Tags: SOA, Service Architecture

Simple IT - a proposal

I've been thinking more and more about why Simplicity is hard and I've come up with a few key things that Simple IT needs to be and how you judge Simple IT. Part of this links back to the SOA anti-patterns and it comes down to a few key questions

1. Can your IT estate be described as a series of discreet elements
2. Can each of these elements be easily maintained within their business context
3. Can each of these elements be simply described

This comes down to that old principle of "one thing well", in IT this doesn't mean low level services, it can be very high-level services, for instance putting in an HR system which does everything that the business wants in a vanilla package solution would meet all of these requirements. HR is in fact a single discreet element and its delivered via a single package, this matches how the business wants to manage that area.

So the building blocks of a simple IT strategy are not all of the same size, they are of the size that makes sense within the context of the business architecture

In a simple IT approach the focus is always on the on going evolution of the IT estate in line with business strategy and not based on a single project delivery. The way IT is set up tends to focus on the The Ivory Tower of Implementation Optimism v Long term viability which drives against simplicity and towards complexity.

So the focus of simple IT is to value

Long term evolution over short term expediency

Architectural clarity over coding efficiency

Business Strategy over IT strategy

Simple IT is also to

Drive IT costs in-line with their business value

Drive IT technology selection in-line with the business value

Create clear upgrade boundaries between different business value areas

Manage IT based on the different business value areas

The point here is that simple IT is not actually about making a single project faster, its about making the 2nd project and its support faster and more efficient. This means having control and direction into which the right approaches can be used, Agile where its about value creation and reaction, package where its about standardisation and SaaS where its about utility.

This is about having the business architecture, having the heatmap, and then aligning IT clearly into those areas.

Simple IT is hard, it requires control, it requires vision and it requires focus.

Technorati Tags: SOA, Service Architecture

MDM made easy - why people make Master Data Management hard

Okay so I've spent a while in the last 10+ years on various MDM (Master Data Management) programmes and its come down to a very simple reason why lots of MDM programmes fail..

People screw up MDM programmes by forgetting what MDM actually is.

The first bit is that people look at the various different MDM packages and then really miss the point. Whether its Oracle UCM, Informatica, SAP MDM, IBM MDM, Initiate, TIBCO or anything else people look at it and go...

Right so this is what we do, what else can we fit into it

Now this is a stupid way to behave but its what most people do. The use the MDM piece as the starting point. The reality is that MDM is only about two things

1. The cross references of the core entity between all the systems in the IT estate
2. The data quality and governance around the core entity to uplift its business value

And that really is it. The more attributes that are added beyond the those required for these two elements then the more expensive and more complex and less effective your MDM solution will be.

The other core part of this is that there really are only a very limited number of bits to MDM, its about three things

1. Things - assets, accounts, parts, products, etc
2. Parties - individuals, organisations, customers, suppliers, etc
3. Locations - postal addresses, email address, web addresses, physical address, geo locations, etc

That really is it from an entity perspective then you've got the relationships for those entities (and remember a relationship is just the keys to do the match) which means

1. Parties to Things
2. Parties to Locations
3. Things to Locations
4. (Parties to Things) to Locations (e.g. a persons specific account address)

There really isn't much else to do it really is just

So all you need for MDM to succeed is the above entity model and a list of attributes required to uniquely identify a high quality version of that entity.

So why do people screw up?

Well first off is because they do something like picking a customer mastering package and then trying to master product information or product relationships within it (e.g. Asset to Location).

Secondly because they start extending the number of attributes rather than creating a cheap ODS.

Thirdly because they create their own versions of the relationships because of a belief that business rules that they have change the entity model rather than actually just being data quality rules against the standard entity model.

So the reality is that MDM is simple, its about the data quality and matching, which means its about the business governance of information.

And that is the real reason that most MDM programmes fail to deliver, because in order to get to the business governance of information and make MDM simple you have to do an awful lot of hard work in building up the trust between the business areas and IT that governance can be done in a way that benefits rather than impacts business.

MDM isn't a technology solution its a simple business solution, the problem is that IT people tend to make it a complex IT solution because they can't make it a simple business solution.

Technorati Tags: SOA, Service Architecture

Simplicity is hard

One of the things I get to do a lot of is look at IT estates that are a complete and utter mess. Systems overlap in functionality, are difficult to maintain and the links between them are more complicated than Glenn Beck's issues with reality. When doing a Business Service Architecture it becomes clear that the big issue here is that IT doesn't learn the lesson of Unix Do one thing and do it well. In SOA, particularly business driven SOA this is the whole point of services, they do one thing and they are designed to be integrated.

Having the "services" as clean though is pointless if what you have under the covers is just the same old crud with some REST or WS-* lipstick on top, you actually have to have an implementation that is clean all the way down or you are still screwed.

The BSB Specification was based around that principle of doing one thing well, and the whole point of the DSB/BSB split is to keep it simple.

This then becomes the real issue, its actually really hard to architect and deliver simply. In the MDM space for instance you see MDM solutions that morph into MDM + ODS + Reference Data Management solutions. "clean" ERP installations are destroyed by customisation and the Java solution gets some crufty bolt ons because "it was easier to do it there". The delivery builds the blob with lipstick on it and suddenly we are no better off.

Why does this happen? Well more and more I believe its because the SIMPLE pictures that describe a business architecture are either not drawn at all or are abandoned because of their simplicity. People, architects especially, don't like putting in place the rigour and control that is required to deliver a simple solution, its much easier to deliver a blob and let people cope with it in support. Simplicity isn't a valued commodity because it doesn't allow people to show off their understanding of complexity.

"Je N'ai fait celle-ci plus longue que parceque je n'ai pas eu le loisir de la faire plus courte.
--I have only made this letter rather long because I have not had time to make it shorter."
Pascal. Lettres provinciales, 16, Dec.14,1656. Cassell's Book of Quotations, London,1912. P.718.

Simplicity takes time and effort and the end result is much more satisfying, easier to explain, easier to maintain and easier to use. Most people however take the easy route to complexity.

Technorati Tags: SOA, Service Architecture

MDM is required for successful SOA

One of the things that I've noticed over the years about successful SOA programmes is that they all undertake a formal approach to MDM. By programme here I'm not talking about one website using REST or WS-*, I'm talking about a strategic transformation initiative that aims to move the IT estate forwards to a more business centric approach.

Why is MDM important for SOA? Well there are a couple of reasons.

1. MDM stops you having to do Single Canonical Form
2. MDM helps you start from a point of federation

Now there are a number of MDM approaches that enterprises take and I'll massively generalise them into three groups

1. Digital Landfill MDM
2. Operational MDM
3. Federated MDM

The first is the most common and is generally, in fact pretty much exclusively, a post-transactional approach where lots and lots of systems feed down into the "MDM" solution which is really there to provide the cross referencing required for a data warehouse reporting solution. These tend to be database centric and batch oriented and not overly useful for a real-time or operational environment

The second is where the MDM solution has actually taken on a transactional approach where information, for instance customer details, are actually held within the operational MDM solution and the cross referencing information is stored and available in real-time. This really is the minimum level at which an SOA environment should be operating. The MDM solution is there to ensure the loose coupling between services and making sure that a minimal reference model approach to information sharing can be handled.

The third is where there is no central MDM solution but it is still operating transactionally. This is the very very hard and not really to be attempted yet type approach. Here you can think about semantic web and those sorts of funky technologies and on doing the matching and cross referencing on an on demand basis. I'm pretty sure this type of solution won't work very well, or indeed potentially at all, but its really what lots of people are trying to do today when they have an ESB, lots of services and nothing set up to do the mappings.

So basically the point is simple. A transactional, operational, real-time MDM solution is an absolute requirement for a professional SOA environment. Without it you are going to have nasty tight integrations between areas at the data layer or some amazingly complex mappings that will fail on a regular basis. I'm not even talking here about the benefits of data quality improvements by using a decent MDM solution I'm just talking about something that manages the cross-referencing of similar information between systems and services.

The odds are however that in doing your SOA solution you've left MDM being something that just feeds the data warehouse and looks at reporting.

Technorati Tags: SOA, Service Architecture