The People's Democratic Republic of IT

IT is a communist state in many organisations, one that believes in rigid adherence to inflexible approaches despite clear indications that they inhibit growth and a central approach to planning that Mao and Stalin would have thought is taking things a little too far. This really doesn't make sense in the capitalistic world of business and the counter-revolution is well under way. Its

I don't think the word 'Enterprise' is really worth anything in terms of something being a single standard Enterprise approach.  Whether that is Enterprise Resource Planning, Enterprise Data Warehouse, Enterprise Service Bus or Enterprise Architecture you either end up with multiple solutions or a central solution that isn't used to the level it was envisaged so you get lots of solutions on the side.

Part of this is because in the capitalistic world of business it appears that communist style central planning has been, and remains, the normal approach.  This People's Democratic Republic of IT approach has two key parts to it

1. IT knows best and will give everyone 'each according to their needs' and decide what those needs are.
2. Cultish following of other communist plans, independent of whether the users want them.

The world of integration is a great example of the latter.  Do you know how much the business cares about whether you integrate two systems using REST, SOAP,sockets or flying monkeys?  ZERO.  Hell probably even less than zero in that they have an active disinterest in it.  Yet in IT we don't take this as a guidance of 'its not important, lets commoditise the fuck out of it'.  Nope we continue to 'innovate' where it really doesn't matter and we do so because a whole heap of hype tells us to... business hype?  Of course not, its hype from people who think they've discovered the universal hammer that turns everything into a nail.

On the former its the realm of 'Enterprise Architecture' and EDWs that really underline just how much IT often resembles the politburo.  Here groups of worthy individuals set about on the business equivalent of the Cultural Revolution or Stalin's grand plan for agriculture.  They just know that if everyone would just work in the same way then everything would be so much better.  So off they trot pushing a single solution and historically this was pushed all the way through to production and the business went:

"Well its not what I wanted but its a bit less shit than what I've got"

So IT created grand strategic plans (and I've said before there is no such thing as IT strategy) often in areas that the business really didn't care and off the business went and started using DropBox, salesforce.com and Amazon.

In effect the Shadow IT efforts of the business are analogous to the black market economies that often thrived in communist countries in the 80s.  Getting on doing what they need to do and being a lot more efficient than the state in doing it.  What we are seeing today is that as budget shift more and more towards the business the shadow IT market is getting bigger and bigger and the central planning has suddenly hit an issue.

The business understands technology

 Maybe not in the depth that IT does, but what the business understands is a bit more valuable

They understand how to focus on outcomes that add value, not technology hype.

So now as the Enterprise Architect says "you cannot do that, it is against our policy" the business says "stuff that for a game of soldiers, your policy doesn't work for us.".  The business is having its Berlin Wall moment, and while the IT communist state, the People's Democratic Republic of IT (because communist states love claiming they are democratic) might hold on for a while the reality is that the world is beginning to come crashing down.

Its time for IT to embrace capitalism, embrace value over technology and outcomes over acronyms.

How Business SOA thinking impacts data

Over the years I've written quite a bit about how SOA, when viewed as a tool for Business Architecture, can change some of the cherished beliefs in IT.  One of these was about how the Single Canonical Form was not for SOA and others have talked about how MDM and SOA collaborate to deliver a more flexible approach.  Central to all of these things has been that this has been about information and transactions 'in the now' the active flowing of information within and between businesses and how you make that more agile while ensuring such collaboration can be trusted.

Recently however my challenge has been around data, the post-transactional world, where things go after they've happened so people can do analytics on them.  This world has been the champion of the single canonical form, massive single schemas that aim to encompass everything, with the advantage over the operational world that things have happened so the constraints, while evident, are less of a daily impact.

The challenge of data however is that the gap between the post-transactional and the operational world has disappeared.  We've spent 30 years in IT creating a hard-wall between these ares, creating Data Warehouses which operate much like batch driven mainframes and where the idea of operational systems directly accessing them has been aggressively discouraged.  The problem is that the business doesn't see this separation.  They want to see analytics and its insight delivered back into the operational systems to help make better decisions.

So this got me thinking, why is it that in the SOA world and operational world its perfectly ok for local domains and businesses to have their own view on a piece of data, an invoice, a customer, a product, etc but when it comes to reporting they all need to agree?  Having spent a lot of time on MDM projects recently the answer was pretty simple:

They don't

With MDM the really valuable bit is the cross-reference, its the bit that enables collaboration.  The amount of standardisation required is actually pretty low.  If Sales has 100 attributes for Customer and Finance only 30 and in-fact it only takes 20 to uniquely identify the customer then its that 20 that really matter to drive the cross reference.  If there isn't any value in agreeing on the other attributes then why bother investing in it?  Getting agreement is hard enough without trying to do it in areas where the business fundamentally doesn't care.

This approach to MDM helps to create shorter more targeted programs, and programs that are really suited to enabling business collaboration.  You don't need to pass the full customer record, you just pass the ID.

So what does this combination of MDM and SOA mean for data, particularly as we want analytics to be integrated back into operations?

Data solutions should look more like Business SOA solutions and match the way the business works

In simple terms it means the sort of thinking that led to flexibly integrated SOA solutions should now be applied to Data.  Get rid of that single Schema, concentrate on having data served up in a way that matches the requirements of the business domains and concentrate governance on where its required to give global consistency and drive business collaboration.  That way you can ensure that the insights being created will be able to be managed in the same way as the operational systems.

With SOA the problem of people building applications 'in the bus' led me to propose a new architectural approach where you don't have one ESB that does everything but accept that different parts of the business will want their own local control.  The Business Service Bus concept was built around that and with the folks at IBM, SAP, Microsoft and Oracle all ensuring that pretty much everyone ends up with a couple of ESB type solutions its the sort of architecture I've seen work on multiple occasions.  That sort of approach is exactly what I now think applies to data.

The difference?

Well with data and analytical approaches you probably want to combine data from multiple sources, not just your own local information, fortunately new (Java) technologies such as Hadoop are changing the economics of storing data so instead of having to agree on schemas you can just land all of your corporate data into one environment and let those business users build business aligned analytics which sit within their domain, even if they are using information shared by others.  MDM allows that cross reference to happen in a managed way but a new business aligned approach removes the need for total agreement before anything can get done.

With Business SOA driven operations we had the ability to get all the operational data in real-time and aggregate at the BSB level if required, with Business SOA driven data approaches we can land all the information and then enable the flexibility.  By aligning both the operational and post-transactional models within a single consistent Business aligned approach we start doing what IT should be doing all along

Creating an IT estate that looks like the business, evolves like the business and that is costed in-line with the value it delivers.

Applying Business SOA thinking to data has been really interesting and what led to the Business Data Lake concept, its early days clearly but I really do believe that getting the operational and data worlds aligned to the business in a consistent way is going to be the way forwards.

This isn't a new and radical approach, its just applying what worked in operations to the data space and recognising that if the goal of analytics is to deliver insight back into operations then that insight needs to be aligned to the business operations from the start so it can adapt and change as the operational business requires.

The boundaries from the operational and post-transactional world have gone, the new boundaries are defined by the business and the governance in both areas is about enabling consistent collaboration.

Why MDM and SOA are shifting out of IT

Now I've said previously why MDM is required for successful SOA but there is another important piece around MDM and SOA that is happening at the moment and explains both why MDM and SOA haven't historically gone together and, crucially, why the new trends are liable to help bring them together next year.

Why SOA and MDM didn't go together

The first question is why historically MDM and SOA projects tended to not go together. Sure organisations would "do" SOA programmes and would "do" MDM programmes but rarely would the SOA programmes and MDM programmes be tightly joined. There are I think three reasons for this

1 - Different bits of IT

The first challenge is that MDM and SOA were normally done by different bits of IT with different mentalities. MDM was done by the "data" folks who worried about data warehouses and saw data as the most important thing in the world. SOA was done by the enterprise integration and architecture folks who worried about operational processes and integration. The MDM folks tended to have a post-transactional view of the world where things were "true" or "false" while the SOA folks tended to view things as "in processes" or "completed".

These different communities in IT have very different backgrounds and focuses. ETL, Data Warehouses and big databases rule in the MDM/Data side while fast transaction throughput is the key for the SOA folks.

2 - Different bits of vendors

The next challenge is that the vendors have exactly the same split in their view of the world. Look at IBM for instance. SOA is in the WebSphere brand while MDM is in the InfoSphere brand, two very different parts of IBM Software with independent management structures and teams, sure the idea is that they should co-operate and work together but we all know that different divisions in companies like to do things differently. This also means that you often get different sales people for the different pieces of software. Oracle for instance put MDM in the Applications area while SOA is in the Middleware space, if you want to use their MDM products with their middleware products (for instance using the MDM PIP) this means you have to deal with two different sales people to get what you want.

3 - They've been IT projects

The final reason is that MDM and SOA have traditionally been internal IT programmes owned and managed by IT and largely invisible to the business. This means that the two cultural elements above are then hard-baked into the solution which ensures that the SOA and MDM programmes are kept distinct.

Why its changing

So why is it changing? Well the first reason is that people have started finally realising that SOA is a business thing and needs to be viewed from a business architecture perspective. This has been a long time coming but finally it appears people are thinking about the challenges of business services, service value and the business architecture. At the same time MDM is shifting as well, firstly its shifting away from post transactional into an operational space which means it needs to consider operational processes and performance in a way it hasn't before, this means that the MDM/Data folks now have to not only talk to the SOA folks but they have to... shock horror... actually become MDM/SOA folks.

MDM is also shifting in that business people want to do more active information management, both in terms of newer analytical tools and secondly in terms of including that mastered and high-quality information into core operational processes. This means that the business realises that the old "data landfill" approaches which were poor before are massively hindering the analytical models and have a direct impact on the efficiency of the operational processes. By taking control recognising that Information needs Mastering the business is now actively taking over those processes and thus MDM is moving out of the background into being a key part of the corporate information strategy.

So that is why MDM and SOA are shifting out of IT, its because being within IT made them technically centric programmes that often failed to deliver the promised value. By enabling the business to more actively own its IT estate, through Business SOA, and its core information, through MDM, it suddenly ceases to be a question of competing IT cultures but a simple question of how to present a consistent operational and analytical view of the business.

SOA and MDM are made to be together.... that fixes the enterprise culture... but will it fix the vendors?

Technorati Tags: SOA, Service Architecture