Tuesday, January 28, 2014

EDW in the Library with Single Canonical Form - get a clue about killing the business

The game Cluedo (or just plain Clue in North America) is about discovering which person committed the murder, in what room using what.  What is amazing is that in IT we have the easiest game of Cluedo going and yet over and over again we murder the poor unfortunate business in the same way, then stand back and gasp 'I didn't know that would kill them'.

I talk about the EDW, the IT departments hammer to which every question of 'I don't have the information I need' looks like a nail.  The EDW is the murderer of information agility, the constrainer of local requirements and the heavy weight bully of the data landscape.  But its weapon of choice is more blunt than the lead pipe - the Single Canonical Form.  The creation of which requires compromise, limitation and above all a bloody indifference to the actual local needs of business users.

An EDW is normally actually only trying to answer a question at a high level of corporate consistency, so financial roll-up, a bit of a horizontal view around customer and maybe some views around procurement... although the latter is normally better done on its own.  The point is that it really isn't Enterprise beyond the fact that Enterprise is a lie.  It can be really good at doing that top level view, of creating a corporate data mart but the effort that it requires to do so often stifles the agility in local business units and chokes the throat of local information initiatives.

The good news is that IT didn't do this for completely bloody minded reasons, it did it because IT had constraints, data storage costs first amongst them and IT had a hard wall between the world of the operational transaction and the world of post-transactional analytics.  So the EDW worked in that limited space and with those restrictions.

The challenge now is that the restrictions of gone, storage  costs are now amazingly low when looking at Hadoop and the wall between operations and analytics has gone, with operations being the primary place that analytics is new able to deliver insight at the point of action.  This was the thinking that I put into the Business Data Lake an approach that matches the business environment, leverages the thinking behind Business SOA applied to data.

So lets put down the EDW, lets walk away from the single canonical form and get a clue.  Its going to take time for IT departments, as well as analysts, vendors and consultants, to be weaned off the EDW drug but I firmly feel that in 5 years time we will be looking at a world where the IT department no longer says:

"You need an EDW, lets design the schema, should be ready early next year"

and instead says

"Sure, I'll knock up a solution in the BDL for you, be ready on Tuesday"

The customer is always right, and our customer in IT is telling us that its the local view that counts... so can we stop battering them with a global view that doesn't fit their local problem.

No comments: