Big Data - are you the house or the played?

Coming back from the EMC World conference in Las Vegas I was looking at the people playing on the slots and making ridiculous bets at Craps and wondering 'don't these people know anything about statistics?'.  Lets be clear I get the idea of it being fun, but when you sit next to someone on a blackjack table who twists when the dealer is showing a six and they've got 15 is just depressing.

There is a business lesson for us here about the power of data and in particular the differentiation that Big Data will have between businesses.  Those that really leverage Big Data and Predictive analytics and integrate it back into business processes will have a significant 'house' advantage over their competition.  The retailer who knows what the next hot trend will be or who can negotiate prices based on a long term demand view will under-cut and out perform their rivals.  In the same way as Vegas knows the returns on the slots and table games and innovates to improve their odds so Big Data and Predictive Analytics will help some companies dominate their sector.

Things like Hadoop, R and the like are just tools in this journey and companies that focus on the technology are like gamblers who claim to have 'a system' for a game like roulette.  Sure they might get lucky a couple of times but over time they are just going to lose to the house.  The companies that win will be the ones like the casinos which run statistics and predictive models to a level that helps to assure their advantage.

So are you the house or the played?

Federated Caching in the world of the 64GB mobile

There is something that is beginning to irritate me, ok something else.  Its mobile applications that don't cache.  I'm fed up of travelling on a train or being on a plane and the end result being that my iPad or iPhone app doesn't work because I'm in an area that doesn't have reasonable network coverage.  I was using the App earlier when it had WiFi and it was all fine but the app requires me to be 'always connected' which I just feel is plain lazy.  My mobile phone has about 16GB of free space, my iPad has slightly less thanks to the videos but its still several GB of space.

I'm particularly talking here about BI tools and enterprise apps which have no concept of a variable network.  If I've approved a travel expense for someone via email then when I connect the email gets sent, if its on a mobile 'always on app' I can't do that.  Why?  I'm actually losing functionality over email.  If its a BI app and I'm looking at sales reports for a given geo then if I can't access it on the plane then I'm losing functionality over Excel.

Federated Caching is often a tough challenge but is one that we've solved over, and over, and over again in IT.  It shouldn't even be 'save to' on the device it should be a case that if you are doing the reports and have a cache size set on your device then the application should automatically pull the information locally.

The next generation of enterprise mobility will be about removing things like email and Excel or it will just be another pretty technology that does some stuff better but I'm still forced back to the old tools when I step outside the mobile view of a perfectly connected world.

Federated caching needs to be your starting point in a mobile world, you need to leverage the power of the device in a smart way not use it as just a browser and you need to deal with the reality that your users will not always be connected no matter what the adverts say.

Software Developers are you ready for the cage fight with the BI guys?

In all of my career to date in IT there really has been three clear worlds in IT, the software development guys who are the bespoke tailors, the package guys who deliver off the shelf and the BI guys.

I'm going to admit a prejudice here.  Until a couple of years ago my impression of BI guys was folks who were one step up from Excel guys.  It was dead-data and done in batches, sure it had to be done but it really wasn't gool.  The BI guys couldn't handle the operational requirements of the software developers and we really were not interested in Cubes, Star-Schemas and all of the stuff they did to generate what was basically an Excel spreadsheet on steroids.

Now having spent some more time on that side of the fence I've come to respect it more, but there is now an interesting evolution.  Contextual BI.  That is where BI guys talk about integrating BI and Analytics information back into operational processes.  In other words its BI guys talking about integrating to SAP and integrating into BPM and integrating into operational applications generally.

An IT turf war is about to break-out.  I'd argue that the software development guys have a bit of an advantage here, what is the current 'cool kids' BI tool?  Its Hadoop.  What do you need to know to really grok Hadoop?  Yup Java, a software development language.  Software Dev guys grok agile in a way that few BI guys do and are used to getting down into operational requirements.

But we suck at analytics and its massively expensive to develop the sorts of visualisations that BI guys get out of the box.  So this means its either a case of one side taking over the other, or we are going to have to co-operate... and I actually think this is the way forwards.  Software Development guys know how to do real-time scaling and operations, its not a trivial skill, and the BI guys know how to get us the right information into our processes and have the right visualisation tools.

So Software Developers out there its time to reach out and embrace your BI cousins and start looking at how you can work together to create an information fabric which spans the analytical models on historical data, supplements it with real-time analytics and then integrates that back into the operational process.  I know this collaboration is going to be hard and we've all got to get over prejudices to make this happen but the business is demanding it so we need to get it done.  Start thinking about information as a seemless infrastructure where delivery to the point of action is the challenge not delivery to a report.  Software Developers and BI guys working together can create this and deliver a real information fabric to the enterprise.

And don't worry Software Developers and BI guys will still have a common enemy, I'm not crazy, its still perfectly ok to not get on with the package guys.

IT is a fashion industry

You know when people laugh at the fashion industry for saying that 'blue is the new black' and because of its ridiculous amount of fawning over models, designers and the like?  Is that really different to IT?  We've got our fashion houses - Google, Facebook, Apple.  We've got the big bulk conglomerates IBM, Oracle, SAP, Microsoft and oh hell the fawning that goes around...

I'd say the comparison goes even deeper however.  EAI, Web Services, REST... what are these?  They are all integration approaches.  EAI was going to save the enterprise and create a well managed estate that the business could use and could be changed easily and enable integration with external companies... Web Services were going to save the enterprise by standardising the interfaces enabling a well managed estate the business could use and could be easily... REST was going to save all of IT by enabling interfaces that could be dynamically changed and enable integration...

The point is that the long term challenge is the same, system to system integration, yet we have fad based approaches to solve that challenge.  Its like the fashion industry and dress lengths, it goes up and down, but its still a dress.  The real difference in IT however is that the fashion industry does this better, sure they change the hem, but it still works as a dress.  In IT we concentrate so much on the hem length that we don't even bother with the fact that system to system integration appears to be as hard in 2013 as it was in 1999.  We even know why, the Silver Bullet tells us that technology won't solve the problem on its own.  But do we listen?  No because we are followers of fashion.

This analogy to fashion applies to the age discrimination in IT, we love the young and shiny, and age discrimination against new entrants is wonderfully not present.  However the flip side of that is there is an over emphasis on the new in IT, so we prefer doing things in 'new' ways rather than in 'working' ways, and unlike in the fashion industry we don't actually learn from sales what is successful.  If we've got a fad (hello REST) that works in some places but not in others we'll keep on pushing that fashion even as it fails to set the world on fire.  Its the emperor's new clothes effect, and in IT we do the equivalent of the beauty industry.  In the beauty industry you'll see adverts for 'age defying creams' advertised by 16 year old models.  In IT you'll see enterprise solutions pushed by using Google as an example.  We love the new, we love the young, and we really rather hate facing up to the fact that IT is quite an old industry now and 90%+ of the stuff out there is a long way from new and shiny.

The analysts and vendors are the Vogue and Fashion Houses in this world, the pushing of the new as the 'must have' technology and dire warnings if you dare to actually make the old stuff work.  The concentration on the outfit (the technology product) and little about how it actually works in the real-world (operations).  You know when you see outfits at London, New York, Milan or Paris fashion weeks been shown on the news under the 'what madness do designers think we will wear next' section.  Is that so different from an analyst or vendor pushing a new technology without explaining at all how it will fit into the operations of your current business?  We will see things like people declaring the end to SQL... and then a few years later those same people championing SQL as the approach, now that they've realised people can operate their technology if they do that.

The final place I'll talk about IT and fashion is in the 'rebadging' that we see.  In the fashion industry you see old ideas rehashed and pushed down the catwalk as being 'retro'.  There is at least some honesty in the fashion industry as they talk about it being inspired by an era, when we all know what they mean is 'I didn't have an original idea, so I copied one that was old enough that people will think its original to copy'.

In IT we don't even have the honesty of the fashion industry, what we do is see a new trend and claim that old technologies are actually part of that new trend.  We'll take an old EAI tool and slap on an SOA logo, we'll take a hub and spoke broker and call it an ESB.  This re-badging of technology goes on and on, sometimes you'll be in a meeting and suddenly realise 'hang on, I used that 12 years ago... how the hell is it now new?'. This would be fine if the focus was on building a robust product, but too often its just about how to get on an RFP and shift a few more units with actual investment in new approaches being few and far between.

IT and the fashion industry are miles apart in many ways, but the faddish nature of our industries make us very similar.  The problem is that fashion is allowed to be faddish, its not expected that a business will rely on something made 20 years ago but with IT this faddish behaviour is a big problem.  We are meant to be constructing systems on which a business can rely, not just today but 5 years from now and still be leveraging in 10, 20 or even 30 years time if its well constructed and does the job well.  There are mainframe systems out there doing exactly that, and why haven't they been replaced?  Because the new stuff didn't do the job.

IT needs to stop being like the fashion industry and more like the aircraft manufacturing industry, sure they have 'fads' like an all composite aircraft, but that is based on sound data as well as strategic vision. Its not just based on it being what the cool kids do.   We can do the cool stuff, we can do the new stuff, but we need to recognise that there is lots of stuff out there that needs to change, we can't just use Google or Facebook as references or examples, that would be like Boeing selling a plane technology based on what people did in movies, its just too far removed from the enterprise reality.  We need to stop blindly following IT fashion and start critically appraising it and shouting 'emperor's new clothes' when its bullshit.  Most of all we need to look at enterprise technologies based on how they improve the 'now' not based on 'if only we could replace everything', evolution is the revolution in IT.

Its time for IT to grow up and take responsibility for the mess we've created.

The Hadoop hump - why enterprises struggle to move from Proof of Concept to Enterprise deployment

At the recent Hadoop Summit in Amsterdam I noticed something that has been bothering me for a while.  Lots of companies have done some great Proof of Concepts with Hadoop but they are rarely turning those into fully blown operational solutions.  Being clear I'm not talking about the shiny, shiny web companies where the business is technology and the people who develop are the people who support, I'm talking about those dull companies that make up the 99.9% of businesses out there where IT is part of the organisation and support is normally done by separate teams.

There are three key reasons for this Hadoop hump

1. Hadoop is addressing problems in the BI space, but is a custom build technology
2. Hadoop has been created for developers not support
3. BI budgets are used to vertically scaled hardware

These reasons are about people not technologies.  Hadoop might save you money on hardware and software licenses but if you are moving from report developers in the BI space to Map Reduce/R people in Hadoop and most critically requiring those same high value people in support its the people costs that prevent Hadoop being scaled.  The last one is a mental leap that I've seen BI folks struggle to make, they are used to going 'big box' and talking about horizontal scalability and HDFS really doesn't fit with their mindset.  

These are the challenges that companies like Cloudera, Pivotal and Hortonworks are going to have to address to make Hadoop really scale in the enterprise.  Its not about technical scale, its about the cost of people.

Big Data, Fast Data, Orange Data, Blue Data - its decisions that count not data

Oh the chanting is out, Big Data, Fast Data, the three 'V's and of course the ubiquitous elephant are roaming across the IT landscape as the next great hype monster.  Its going the same way as pretty much every IT hype exercise.  Yes this links to the hype cycle but the way it happens is sadly predictable in IT.

1. Company has a business problem they think about it in an innovative way
2. Company needs to build a new piece of technology to help with the process, or chooses to just because they can
3. Company announces the business results they have
4. IT industry focuses on the technology
5. IT industry creates a sticker - 'Big Data'
6. IT vendors produce versions of that technology or start claiming their technology does that anyway
7. IT vendors ramp up the marketing spend around the technology
8. Everyone forgets why this all started

Why is it that Google and Yahoo created and use Hadoop?  Well its because they had a completely different scale of data problem and needed to do challenging analytics in a different way.  Whether it be ad serving or search itself the point was that the aim was functional the data was required to deliver on that functional goal.

When people rave about Social Media, Open Data and various Big Data sources and talk about Hadoop in glowing terms that is all fine and good, but the real point here is about decisions and making them better.  Its about what you can improve in your business not about having to store every piece of data out there.  Concentrating on HDFS as being the important thing in Big Data is looking at flooring being the only important piece of a house.

The hype is on with Big Data, the hype is on with Fast Data but the reality is back to why people started looking at these volumes of information and critically about what it actually means when you start considering massive scale information in terms of governance, management and analytics and that all comes back to a simple set of questions.

What decisions do I need to make better?  What information do I need to make those decisions better? How do I get the right analysis of that information delivered to the point where the decision is made?

We need to stop focusing on Hadoop, R, HANA and all of the technical pieces and start looking at the real trend here, and that is the integrating of analytics into operational processes, the old world of transactional data v analytical data has gone and that is a massive change for IT, not simply in technology but more critically in mindset.

The real shift is not 'Big Data' its the end to post-transactional reporting, its about a single IT infrastructure that mixes real-time information with historical analytics to support better operational decision making.

As with SOA the real shift is a mental one not a technical one and its the mental shift that is hardest to achieve.  Can traditional IT departments move away from the separation of BI from Operational systems?  Clearly the business is going to do that so the only question is whether the IT department is along for the ride or is just the kindergarden where people play with toys.