Why your IT strategy failed or why the business hates IT

One of the most depressing things in IT is our inability to learn.  From the 'Oh look our massive waterfall project ran over budget' to the 'I really can't maintain the code we wrote that doesn't have a design or documentation' we do the same things as an industry over and over again.  Most depressing however is the phrase 'The IT Strategy would work if the business would just change'.

To illustrate this I'd like to tell you a tale, the names will not be used to protect the guilty but it sums up nicely the lunacy of many IT strategy efforts.

Many moons ago I was working for the business side of a large and very successful company.  This company was seen as a genuine market leader and I was working on some very complex mathematics around predictive analytics that the business folks wanted to make their supply chain run better.  There were two highlights through the process from the IT department.

The first was when discussing how such a solution would be implemented and integrated into the operational systems.  IT had a strategy you see and by strategy I mean they had picked a couple of vendors, the solution I was working on had some very specific requirements and wasn't available from those vendors.  An Enterprise Architect from the IT department said in a pretty well attended meeting

'It doesn't matter what the business wants, if we say it isn't going in, it isn't going in.'

The project continued on however as the business saw value in it and wanted to understand what could be done.  One of the key pieces was that we'd need some changes in how operational processes worked, not big ones but more we'd change the way people worked within the existing processes by giving them better information.  To this end we had a workshop with the business and certain key IT folks and worked out how we'd have to design the interfaces and processes to work within the current business environment and culture.  It was a good workshop and the business folks were very happy.

Then came IT, IT you see had a big strategic project to replace all of the existing systems with 'best of breed' solutions.  I'd always assumed that given the massive budget for that program the business was fully engaged... then this happened....

One of the IT folks chirped up and said : "We need to have a workshop so we can tell you what your new operational processes are going to be"

Note the 'tell'... to which the most senior business guy there (a board member IIRC) said

"How do you mean tell us?"

IT Guy: "The new systems have new processes and we need to tell you what they are so you can change."

Business Guy:"Have you done an impact analysis against our current processes?"

IT Guy: "No we've just defined the Best Practice To-Be processes, you need to do the impact and change management.  We need the meeting so we can tell you what the To-Be processes are"

Business Guy in a voice so dripping with sarcasm I thought we'd have a flood: "I look forward to our IT department telling the business what Best Practice is for our industry."

IT Guy, completely failing to read the sarcasm: "Great we'll get it organised"

This is one of the most visible examples of my career on why IT strategies fail.  I've said before there is no such thing as IT strategy its the job of IT to help automate and improve the business strategy, that means thinking tactically and taking strategy from the business model.

"Culture eats strategy for breakfast"

This is the reality and an IT approach that seeks to drive over the culture and dictate from a position of technology purity will fail.  You can change the culture, its hard and its not a technology thing, but you always need to be aware of the culture in order to succeed.

IT Strategy, if such a thing exists, is there to make the business better not to make IT better.

Why in Business driven information its the consumers view that matters

When doing the Business Data Lake pieces it took me back to a view that I had around SOA in that you should take the consumers view when designing a service.  This I think is more critical when looking at analytics and reporting where it really is all about the consumption.

What does this mean though to think about data from the consumers perspective?  We've all had the '3 V's' shoved at us and mostly realised the one that counts in Big Data is actually value.  So taking a Business SOA look at data means that you need to think about the business view and crucially the business value to understand what this actually means.

To this I'd say there are three ways that you can measure whether something is business driven or not

1. The Natural View - is the view on information being presented one that is naturally understandable by a given part of the business
2. Value based cost - does the cost being charged reflect the value being delivered
3. Dynamic Performance - ramp-up, ramp-down as the demand requires

These come back to a mantra I used a lot back when I was talking about the true impact of SOA on an IT estate: Create an IT estate that looks like the business, evolves like the business and is costed based on the business value it delivers.

With SOA in the operational sense this meant having a clear Business Service Architecture and that thinking now applies to Data.  There is no difference between the business views and KPIs between the operational and post-transactional world, indeed the more that analytics becomes the difference in operations the less that difference can be tolerated.

The difference however is in how information is accessed.  In the operational world when you want information from another domain you request it on demand.  In the post transactional world however this is about providing access to the landed (stored) information from other areas in the context that a given domain wants to see it.  Its here that new technologies add value as they enable that distillation to be done into the right business context, indeed enabling the same information to be distilled by different business areas in the right way for their context.

This is the same as you do in the operational space, requesting information from another business service and then converting the result into what makes sense in your local context.

By having a single consistent model between both the operational and post-transactional world you make integrating analytics much easier as you are not requiring your consumers to mentally shift between an operational view and the local view.

So as with a business service architecture which represents the business model so now that business consumer view should be reflected in your analytical applications to create a single model for IT that spans both operations and analytics and now gives the business that local view, the natural view for them, of information.

Now to the other two pieces: Value based Cost and Dynamic Performance.  These two are linked as value is not something that is fixed over time, closing the books is something where high performance is justified at a quarter end or other accounting deadline, but having a high performance solution when not required is wasted cost.

Therefore the new generation of solutions are about Elastic Analytics, that is analytics which can adapt and change based on the business demand.  There needs to be an end to the 'I think I might need in memory so I'll put it all in-memory' or worse the 'Damn I had it all on disk and now I need in-memory'.

The future is about Analytics aligned to the business not aligned to an idealised IT view.

How Business SOA thinking impacts data

Over the years I've written quite a bit about how SOA, when viewed as a tool for Business Architecture, can change some of the cherished beliefs in IT.  One of these was about how the Single Canonical Form was not for SOA and others have talked about how MDM and SOA collaborate to deliver a more flexible approach.  Central to all of these things has been that this has been about information and transactions 'in the now' the active flowing of information within and between businesses and how you make that more agile while ensuring such collaboration can be trusted.

Recently however my challenge has been around data, the post-transactional world, where things go after they've happened so people can do analytics on them.  This world has been the champion of the single canonical form, massive single schemas that aim to encompass everything, with the advantage over the operational world that things have happened so the constraints, while evident, are less of a daily impact.

The challenge of data however is that the gap between the post-transactional and the operational world has disappeared.  We've spent 30 years in IT creating a hard-wall between these ares, creating Data Warehouses which operate much like batch driven mainframes and where the idea of operational systems directly accessing them has been aggressively discouraged.  The problem is that the business doesn't see this separation.  They want to see analytics and its insight delivered back into the operational systems to help make better decisions.

So this got me thinking, why is it that in the SOA world and operational world its perfectly ok for local domains and businesses to have their own view on a piece of data, an invoice, a customer, a product, etc but when it comes to reporting they all need to agree?  Having spent a lot of time on MDM projects recently the answer was pretty simple:

They don't

With MDM the really valuable bit is the cross-reference, its the bit that enables collaboration.  The amount of standardisation required is actually pretty low.  If Sales has 100 attributes for Customer and Finance only 30 and in-fact it only takes 20 to uniquely identify the customer then its that 20 that really matter to drive the cross reference.  If there isn't any value in agreeing on the other attributes then why bother investing in it?  Getting agreement is hard enough without trying to do it in areas where the business fundamentally doesn't care.

This approach to MDM helps to create shorter more targeted programs, and programs that are really suited to enabling business collaboration.  You don't need to pass the full customer record, you just pass the ID.

So what does this combination of MDM and SOA mean for data, particularly as we want analytics to be integrated back into operations?

Data solutions should look more like Business SOA solutions and match the way the business works

In simple terms it means the sort of thinking that led to flexibly integrated SOA solutions should now be applied to Data.  Get rid of that single Schema, concentrate on having data served up in a way that matches the requirements of the business domains and concentrate governance on where its required to give global consistency and drive business collaboration.  That way you can ensure that the insights being created will be able to be managed in the same way as the operational systems.

With SOA the problem of people building applications 'in the bus' led me to propose a new architectural approach where you don't have one ESB that does everything but accept that different parts of the business will want their own local control.  The Business Service Bus concept was built around that and with the folks at IBM, SAP, Microsoft and Oracle all ensuring that pretty much everyone ends up with a couple of ESB type solutions its the sort of architecture I've seen work on multiple occasions.  That sort of approach is exactly what I now think applies to data.

The difference?

Well with data and analytical approaches you probably want to combine data from multiple sources, not just your own local information, fortunately new (Java) technologies such as Hadoop are changing the economics of storing data so instead of having to agree on schemas you can just land all of your corporate data into one environment and let those business users build business aligned analytics which sit within their domain, even if they are using information shared by others.  MDM allows that cross reference to happen in a managed way but a new business aligned approach removes the need for total agreement before anything can get done.

With Business SOA driven operations we had the ability to get all the operational data in real-time and aggregate at the BSB level if required, with Business SOA driven data approaches we can land all the information and then enable the flexibility.  By aligning both the operational and post-transactional models within a single consistent Business aligned approach we start doing what IT should be doing all along

Creating an IT estate that looks like the business, evolves like the business and that is costed in-line with the value it delivers.

Applying Business SOA thinking to data has been really interesting and what led to the Business Data Lake concept, its early days clearly but I really do believe that getting the operational and data worlds aligned to the business in a consistent way is going to be the way forwards.

This isn't a new and radical approach, its just applying what worked in operations to the data space and recognising that if the goal of analytics is to deliver insight back into operations then that insight needs to be aligned to the business operations from the start so it can adapt and change as the operational business requires.

The boundaries from the operational and post-transactional world have gone, the new boundaries are defined by the business and the governance in both areas is about enabling consistent collaboration.

The only V that counts in Big Data is Value

So what is Big Data?  Its Variety, Velocity, Volume right?  But what does that really mean?  Should I get loads of data and drop it into Hadoop, pull in anything I can lay my hands on and I'm now 'doing Big Data'?

Should I plug in my packet monitoring software and store in Hadoop and I'm doing Big Data?  Should I get as many different data sources and that means I'm doing Big Data?

The reality is that one thing hasn't changed and its the key driver for Big Data in the way that it should be in any IT program - Value.  What is the point of what you are doing?  Does the business care? If there isn't a point or the business doesn't care then why on earth are you doing it?

Stop focusing on the 3 Vs of Big Data and worry only about the fourth, the one that really matters

Value.