Monday, January 06, 2014

Six reasons your Big Data Hadoop project will fail in 2014

Ok so Hadoop is the bomb, Hadoop is the schizzle, Hadoop is here to solve world hunger and all problems.  Now I've talked before about some of the challenges around Hadoop for enterprises but here are six reasons that Information Week is right when it says that Hadoop projects are going to fail more often than not.

1. Hadoop is a Java thing not a BI thing
The first is the most important challenge, I'm a Java guy, I'm a Java guy who thinks that Java has been driven off a cliff by its leadership in the last 8 years but its still one of the best platforms out there.  However the problems that Hadoop is trying to address are analytics problems, BI problems.  Put briefly BI guys don't like Java guys and Java guys don't like BI guys.  For Java guys Hadoop is yet more proof that they can do everything, but BI guys know that custom build isn't an efficient route to deliver all of those BI requirements.

On top of that the business folks know SQL, often they really know SQL, SQL is the official language of business and data.  So a straight 'No-SQL' approach is doomed to fail as you are speaking French to the British.  2014 will be the year when SQL on Hadoop becomes the norm but you are still going to need your Java and BI guys to get along, and you are going to have to recognise that SQL beats No-SQL.

2. You decide to roll-your own
Hadoop is open source, all you have to do is download it, install it and off you go right?  There are so many cases of people not doing that right that there is an actual page explaining why they won't accept those as bugs.  Hadoop is a bugger to install, it requires you to really understand how distributed computing works, and guess what?  You thought you did but it turns out you really didn't.  Distributed computing and multi-threaded computing are hard.

There are three companies you need to talk to Pivotal, Cloudera and Hortonworks and how easy can they make it? Well Pivotal have an easy Pivotal HD Hadoop Virtual Machine to get you started and even claim that that they can get you running a Hadoop cluster in 45 minutes.

3. You are building a technical proof of concept... why?
One reason that your efforts will fail is that you are doing a 'technical proof of concept' at the end of which you will amazingly find that something used in some of the biggest analytics challenges on planet earth at the likes of Yahoo fits your much, much smaller challenge.  Well done, you've spent money proving the obvious.

Now what?  How about solving an actual business problem?  Actually why didn't you start by solving an actual business problem as a way to see how it would work for what the business faces?  Technical proof of concepts are pointless, you need to demonstrate to the business how this new technology solve their problems in a better (cheaper, faster, etc) way.

4. You didn't understand what Hadoop was bad at
Hadoop isn't brilliant at everything analytical... shocking eh?  So that complex analytics you want to do which is effectively a complex 25 table join and then do the analytics... yeah that really isn't going to work too well.  Those bits where you said that you could do that key business use case faster and cheaper and then it took 2 days to run?

Hadoop is good a some things, but its not good at everything.  That is why folks are investing in SQL technologies on top of Hadoop, some of which like Pivotal's HAWQ or Cloudera's Impala, with Pivotal already showing how the bridge between traditional MPP and Hadoop is going to be made.

5. You didn't understand that its part of the puzzle
One of the big reasons that Hadoop pieces fail to really deliver is that they are isolated silos, they might even be doing some good analytics but people can't see that analytics where they care about it.  Sure you've put up some nice web-pages for people but they don't use that in their daily lives.  They want to see the information pushed into the Data Warehouse so they can see it in their reports, they want it pushed to the ERP so they can make better decisions... they might want it in many many places but you've left it in the one place that they don't care about it.

When looking at the future of your information landscape you need to remember that Hadoop and NoSQL are just a new tool, a good new tool and one that has a critical part to play but its just one new tool in your toolbox.

6. You didn't change
The biggest reason that your Hadoop project will fail however is that you've not changed some of your basic assumptions and looked how Hadoop enables you to do things differently.  So you are still doing ETL to transform into some idealised schema which is based on a point in time view of what is required.  You are doing that into a Hadoop cluster which couldn't care less about redundant or unused data and where the costs of that are significantly lower than doing another set of ETL development.

You've carried on thinking about grand enterprise solutions to which everyone will come and be beholden to your technical genius.

What you've not done is sit back and think 'the current way sucks for the business can I change that?' because if you had you'd have realised that using Hadoop as a Data substrate/lake layer makes more sense than ETL and you'd have realised that its actually local solutions that get used the most not corporate ones.

Your Hadoop project will fail because of you
The main reason Hadoop projects will fail is because you approach using a new technology with an old mindset, you'll try and build a traditional BI solution in a traditional BI way and you'll not understand that Java doesn't work like that, you'll not understand how Map Reduce is different to SQL and you'll plough on regardless and blame the technology.

Guess what though?  The technology works at massive scale, much, much bigger than anything you've ever deployed.  Its not the technology, its you.

So what to do?
.... I think I'll leave that for another post

Thursday, December 19, 2013

Why your IT strategy failed or why the business hates IT

One of the most depressing things in IT is our inability to learn.  From the 'Oh look our massive waterfall project ran over budget' to the 'I really can't maintain the code we wrote that doesn't have a design or documentation' we do the same things as an industry over and over again.  Most depressing however is the phrase 'The IT Strategy would work if the business would just change'.

To illustrate this I'd like to tell you a tale, the names will not be used to protect the guilty but it sums up nicely the lunacy of many IT strategy efforts.

Many moons ago I was working for the business side of a large and very successful company.  This company was seen as a genuine market leader and I was working on some very complex mathematics around predictive analytics that the business folks wanted to make their supply chain run better.  There were two highlights through the process from the IT department.

The first was when discussing how such a solution would be implemented and integrated into the operational systems.  IT had a strategy you see and by strategy I mean they had picked a couple of vendors, the solution I was working on had some very specific requirements and wasn't available from those vendors.  An Enterprise Architect from the IT department said in a pretty well attended meeting
'It doesn't matter what the business wants, if we say it isn't going in, it isn't going in.'
The project continued on however as the business saw value in it and wanted to understand what could be done.  One of the key pieces was that we'd need some changes in how operational processes worked, not big ones but more we'd change the way people worked within the existing processes by giving them better information.  To this end we had a workshop with the business and certain key IT folks and worked out how we'd have to design the interfaces and processes to work within the current business environment and culture.  It was a good workshop and the business folks were very happy.

Then came IT, IT you see had a big strategic project to replace all of the existing systems with 'best of breed' solutions.  I'd always assumed that given the massive budget for that program the business was fully engaged... then this happened....

One of the IT folks chirped up and said : "We need to have a workshop so we can tell you what your new operational processes are going to be"

Note the 'tell'... to which the most senior business guy there (a board member IIRC) said

"How do you mean tell us?"

IT Guy: "The new systems have new processes and we need to tell you what they are so you can change."

Business Guy:"Have you done an impact analysis against our current processes?"

IT Guy: "No we've just defined the Best Practice To-Be processes, you need to do the impact and change management.  We need the meeting so we can tell you what the To-Be processes are"

Business Guy in a voice so dripping with sarcasm I thought we'd have a flood: "I look forward to our IT department telling the business what Best Practice is for our industry."

IT Guy, completely failing to read the sarcasm: "Great we'll get it organised"

This is one of the most visible examples of my career on why IT strategies fail.  I've said before there is no such thing as IT strategy its the job of IT to help automate and improve the business strategy, that means thinking tactically and taking strategy from the business model.
"Culture eats strategy for breakfast"
This is the reality and an IT approach that seeks to drive over the culture and dictate from a position of technology purity will fail.  You can change the culture, its hard and its not a technology thing, but you always need to be aware of the culture in order to succeed.

IT Strategy, if such a thing exists, is there to make the business better not to make IT better.

Tuesday, December 17, 2013

Why in Business driven information its the consumers view that matters

When doing the Business Data Lake pieces it took me back to a view that I had around SOA in that you should take the consumers view when designing a service.  This I think is more critical when looking at analytics and reporting where it really is all about the consumption.

What does this mean though to think about data from the consumers perspective?  We've all had the '3 V's' shoved at us and mostly realised the one that counts in Big Data is actually value.  So taking a Business SOA look at data means that you need to think about the business view and crucially the business value to understand what this actually means.

To this I'd say there are three ways that you can measure whether something is business driven or not
  1. The Natural View - is the view on information being presented one that is naturally understandable by a given part of the business
  2. Value based cost - does the cost being charged reflect the value being delivered
  3. Dynamic Performance - ramp-up, ramp-down as the demand requires
These come back to a mantra I used a lot back when I was talking about the true impact of SOA on an IT estate: Create an IT estate that looks like the business, evolves like the business and is costed based on the business value it delivers.

With SOA in the operational sense this meant having a clear Business Service Architecture and that thinking now applies to Data.  There is no difference between the business views and KPIs between the operational and post-transactional world, indeed the more that analytics becomes the difference in operations the less that difference can be tolerated.

The difference however is in how information is accessed.  In the operational world when you want information from another domain you request it on demand.  In the post transactional world however this is about providing access to the landed (stored) information from other areas in the context that a given domain wants to see it.  Its here that new technologies add value as they enable that distillation to be done into the right business context, indeed enabling the same information to be distilled by different business areas in the right way for their context.

This is the same as you do in the operational space, requesting information from another business service and then converting the result into what makes sense in your local context.

By having a single consistent model between both the operational and post-transactional world you make integrating analytics much easier as you are not requiring your consumers to mentally shift between an operational view and the local view.

So as with a business service architecture which represents the business model so now that business consumer view should be reflected in your analytical applications to create a single model for IT that spans both operations and analytics and now gives the business that local view, the natural view for them, of information.

Now to the other two pieces: Value based Cost and Dynamic Performance.  These two are linked as value is not something that is fixed over time, closing the books is something where high performance is justified at a quarter end or other accounting deadline, but having a high performance solution when not required is wasted cost.

Therefore the new generation of solutions are about Elastic Analytics, that is analytics which can adapt and change based on the business demand.  There needs to be an end to the 'I think I might need in memory so I'll put it all in-memory' or worse the 'Damn I had it all on disk and now I need in-memory'.

The future is about Analytics aligned to the business not aligned to an idealised IT view.

Thursday, December 05, 2013

How Business SOA thinking impacts data

Over the years I've written quite a bit about how SOA, when viewed as a tool for Business Architecture, can change some of the cherished beliefs in IT.  One of these was about how the Single Canonical Form was not for SOA and others have talked about how MDM and SOA collaborate to deliver a more flexible approach.  Central to all of these things has been that this has been about information and transactions 'in the now' the active flowing of information within and between businesses and how you make that more agile while ensuring such collaboration can be trusted.

Recently however my challenge has been around data, the post-transactional world, where things go after they've happened so people can do analytics on them.  This world has been the champion of the single canonical form, massive single schemas that aim to encompass everything, with the advantage over the operational world that things have happened so the constraints, while evident, are less of a daily impact.

The challenge of data however is that the gap between the post-transactional and the operational world has disappeared.  We've spent 30 years in IT creating a hard-wall between these ares, creating Data Warehouses which operate much like batch driven mainframes and where the idea of operational systems directly accessing them has been aggressively discouraged.  The problem is that the business doesn't see this separation.  They want to see analytics and its insight delivered back into the operational systems to help make better decisions.

So this got me thinking, why is it that in the SOA world and operational world its perfectly ok for local domains and businesses to have their own view on a piece of data, an invoice, a customer, a product, etc but when it comes to reporting they all need to agree?  Having spent a lot of time on MDM projects recently the answer was pretty simple:
They don't
With MDM the really valuable bit is the cross-reference, its the bit that enables collaboration.  The amount of standardisation required is actually pretty low.  If Sales has 100 attributes for Customer and Finance only 30 and in-fact it only takes 20 to uniquely identify the customer then its that 20 that really matter to drive the cross reference.  If there isn't any value in agreeing on the other attributes then why bother investing in it?  Getting agreement is hard enough without trying to do it in areas where the business fundamentally doesn't care.

This approach to MDM helps to create shorter more targeted programs, and programs that are really suited to enabling business collaboration.  You don't need to pass the full customer record, you just pass the ID.

So what does this combination of MDM and SOA mean for data, particularly as we want analytics to be integrated back into operations?
Data solutions should look more like Business SOA solutions and match the way the business works
In simple terms it means the sort of thinking that led to flexibly integrated SOA solutions should now be applied to Data.  Get rid of that single Schema, concentrate on having data served up in a way that matches the requirements of the business domains and concentrate governance on where its required to give global consistency and drive business collaboration.  That way you can ensure that the insights being created will be able to be managed in the same way as the operational systems.

With SOA the problem of people building applications 'in the bus' led me to propose a new architectural approach where you don't have one ESB that does everything but accept that different parts of the business will want their own local control.  The Business Service Bus concept was built around that and with the folks at IBM, SAP, Microsoft and Oracle all ensuring that pretty much everyone ends up with a couple of ESB type solutions its the sort of architecture I've seen work on multiple occasions.  That sort of approach is exactly what I now think applies to data.

The difference?

Well with data and analytical approaches you probably want to combine data from multiple sources, not just your own local information, fortunately new (Java) technologies such as Hadoop are changing the economics of storing data so instead of having to agree on schemas you can just land all of your corporate data into one environment and let those business users build business aligned analytics which sit within their domain, even if they are using information shared by others.  MDM allows that cross reference to happen in a managed way but a new business aligned approach removes the need for total agreement before anything can get done.

With Business SOA driven operations we had the ability to get all the operational data in real-time and aggregate at the BSB level if required, with Business SOA driven data approaches we can land all the information and then enable the flexibility.  By aligning both the operational and post-transactional models within a single consistent Business aligned approach we start doing what IT should be doing all along
Creating an IT estate that looks like the business, evolves like the business and that is costed in-line with the value it delivers.
Applying Business SOA thinking to data has been really interesting and what led to the Business Data Lake concept, its early days clearly but I really do believe that getting the operational and data worlds aligned to the business in a consistent way is going to be the way forwards.

This isn't a new and radical approach, its just applying what worked in operations to the data space and recognising that if the goal of analytics is to deliver insight back into operations then that insight needs to be aligned to the business operations from the start so it can adapt and change as the operational business requires.

The boundaries from the operational and post-transactional world have gone, the new boundaries are defined by the business and the governance in both areas is about enabling consistent collaboration.

Monday, December 02, 2013

The only V that counts in Big Data is Value

So what is Big Data?  Its Variety, Velocity, Volume right?  But what does that really mean?  Should I get loads of data and drop it into Hadoop, pull in anything I can lay my hands on and I'm now 'doing Big Data'?

Should I plug in my packet monitoring software and store in Hadoop and I'm doing Big Data?  Should I get as many different data sources and that means I'm doing Big Data?

The reality is that one thing hasn't changed and its the key driver for Big Data in the way that it should be in any IT program - Value.  What is the point of what you are doing?  Does the business care? If there isn't a point or the business doesn't care then why on earth are you doing it?

Stop focusing on the 3 Vs of Big Data and worry only about the fourth, the one that really matters

Value.

Monday, November 04, 2013

Zuckerberg and the unreality of valley thinking

Bill Gates laid into Mark Zuckerberg's vision of internet access being the most important thing in solving the world's ills and another article at The Register compared Silicon Valley to 'Sheldonville' which I really agree with.  At the heart of lots of these 'visions' is an insular view of technology within Silicon Valley.

How insular?  Lets say Blue Ridge Mountains style insular.  The normal vision is that valley thinking will change the world if only people would just use that technology.  We saw it in the .com bust with companies that preached a huge vision on how they would replace the normal bricks and mortar and how everything would be changed and replaced....

And guess what?  It takes a hell of a lot longer to change culture than a VC/IPO cycle would like the end result is that the companies fail.  The VC/IPO money does however distort the market, look at Amazon's 'profit' statements against a food retailer - Wholefoods - and a failing retailer - Best Buy.


Revenue Profit
Amazon $21.27bn $97m
Wholefoods $2.2bn $113m
Best Buy $16.7bn $1.9bn

Now I know that Amazon aren't strictly a valley company but its the same sort of mentality, I'll come to some great examples of valley mentality in a second.  So Amazon is a great company doing some really cool and innovative things but thanks to their special technical sauce they are allowed to significantly undercut their competition because they don't need to make as much profit to keep shareholders happy.

That is a big advantage and one that distorts markets.

Now how about some other valley thinking examples?
  • Jonathan Schwartz and the over optimism of 'Open Source will defeat all' and pitching to financial analysts about the number of downloads the company had is a good one.  It ultimately doomed Sun to acquisition and really was a great example of valley thinking.
  • The startup I met whose strategy was 'rip out SAP' in order to use their software.
  • The integration of JAX-WS and Javascript into Java
  • The whole .com 'boom' based on 'visitor' numbers
  • The whole social media 'boom' based on 'user' numbers
  • Hell every single startup whose philosophy is that companies will rip out working technology to replace it with their new shiny and unproven technology
The valley often does come up with good stuff, but does valley thinking overall make the world a better place?  Sure the valley came up with Java.... and then the valley screwed it up.  The valley came up with REST... oh brilliant our lives are so much better.

My point of this rant is that software investment is driven by the thinking in a short stretch of land, some of it can be good, some of it can be bad, but when you look at companies like IBM, SAP, ARM, etc its quite amazing just how positive an impact they've had on technology, and arguably the world, without being driven at any stage by massive hype cycles.  I'd argue its exactly because these folks are mainly out of the valley that the often deliver practical innovations and improvements rather than impractical visions.

Zuckerberg's 'vision' for helping the world is just another example of that and Bill Gates is spot on about what is really important.  The valley is a great place, its a buzzing place, but its as much to do with the majority of technology decisions as Fantasy Football has to do with the NFL, sure they are related but one of them is doing real work, even if lots of money is going into the former, its the latter that tends to focus on profits.