Why your IT strategy failed or why the business hates IT

One of the most depressing things in IT is our inability to learn.  From the 'Oh look our massive waterfall project ran over budget' to the 'I really can't maintain the code we wrote that doesn't have a design or documentation' we do the same things as an industry over and over again.  Most depressing however is the phrase 'The IT Strategy would work if the business would just change'.

To illustrate this I'd like to tell you a tale, the names will not be used to protect the guilty but it sums up nicely the lunacy of many IT strategy efforts.

Many moons ago I was working for the business side of a large and very successful company.  This company was seen as a genuine market leader and I was working on some very complex mathematics around predictive analytics that the business folks wanted to make their supply chain run better.  There were two highlights through the process from the IT department.

The first was when discussing how such a solution would be implemented and integrated into the operational systems.  IT had a strategy you see and by strategy I mean they had picked a couple of vendors, the solution I was working on had some very specific requirements and wasn't available from those vendors.  An Enterprise Architect from the IT department said in a pretty well attended meeting

'It doesn't matter what the business wants, if we say it isn't going in, it isn't going in.'

The project continued on however as the business saw value in it and wanted to understand what could be done.  One of the key pieces was that we'd need some changes in how operational processes worked, not big ones but more we'd change the way people worked within the existing processes by giving them better information.  To this end we had a workshop with the business and certain key IT folks and worked out how we'd have to design the interfaces and processes to work within the current business environment and culture.  It was a good workshop and the business folks were very happy.

Then came IT, IT you see had a big strategic project to replace all of the existing systems with 'best of breed' solutions.  I'd always assumed that given the massive budget for that program the business was fully engaged... then this happened....

One of the IT folks chirped up and said : "We need to have a workshop so we can tell you what your new operational processes are going to be"

Note the 'tell'... to which the most senior business guy there (a board member IIRC) said

"How do you mean tell us?"

IT Guy: "The new systems have new processes and we need to tell you what they are so you can change."

Business Guy:"Have you done an impact analysis against our current processes?"

IT Guy: "No we've just defined the Best Practice To-Be processes, you need to do the impact and change management.  We need the meeting so we can tell you what the To-Be processes are"

Business Guy in a voice so dripping with sarcasm I thought we'd have a flood: "I look forward to our IT department telling the business what Best Practice is for our industry."

IT Guy, completely failing to read the sarcasm: "Great we'll get it organised"

This is one of the most visible examples of my career on why IT strategies fail.  I've said before there is no such thing as IT strategy its the job of IT to help automate and improve the business strategy, that means thinking tactically and taking strategy from the business model.

"Culture eats strategy for breakfast"

This is the reality and an IT approach that seeks to drive over the culture and dictate from a position of technology purity will fail.  You can change the culture, its hard and its not a technology thing, but you always need to be aware of the culture in order to succeed.

IT Strategy, if such a thing exists, is there to make the business better not to make IT better.

Why in Business driven information its the consumers view that matters

When doing the Business Data Lake pieces it took me back to a view that I had around SOA in that you should take the consumers view when designing a service.  This I think is more critical when looking at analytics and reporting where it really is all about the consumption.

What does this mean though to think about data from the consumers perspective?  We've all had the '3 V's' shoved at us and mostly realised the one that counts in Big Data is actually value.  So taking a Business SOA look at data means that you need to think about the business view and crucially the business value to understand what this actually means.

To this I'd say there are three ways that you can measure whether something is business driven or not

1. The Natural View - is the view on information being presented one that is naturally understandable by a given part of the business
2. Value based cost - does the cost being charged reflect the value being delivered
3. Dynamic Performance - ramp-up, ramp-down as the demand requires

These come back to a mantra I used a lot back when I was talking about the true impact of SOA on an IT estate: Create an IT estate that looks like the business, evolves like the business and is costed based on the business value it delivers.

With SOA in the operational sense this meant having a clear Business Service Architecture and that thinking now applies to Data.  There is no difference between the business views and KPIs between the operational and post-transactional world, indeed the more that analytics becomes the difference in operations the less that difference can be tolerated.

The difference however is in how information is accessed.  In the operational world when you want information from another domain you request it on demand.  In the post transactional world however this is about providing access to the landed (stored) information from other areas in the context that a given domain wants to see it.  Its here that new technologies add value as they enable that distillation to be done into the right business context, indeed enabling the same information to be distilled by different business areas in the right way for their context.

This is the same as you do in the operational space, requesting information from another business service and then converting the result into what makes sense in your local context.

By having a single consistent model between both the operational and post-transactional world you make integrating analytics much easier as you are not requiring your consumers to mentally shift between an operational view and the local view.

So as with a business service architecture which represents the business model so now that business consumer view should be reflected in your analytical applications to create a single model for IT that spans both operations and analytics and now gives the business that local view, the natural view for them, of information.

Now to the other two pieces: Value based Cost and Dynamic Performance.  These two are linked as value is not something that is fixed over time, closing the books is something where high performance is justified at a quarter end or other accounting deadline, but having a high performance solution when not required is wasted cost.

Therefore the new generation of solutions are about Elastic Analytics, that is analytics which can adapt and change based on the business demand.  There needs to be an end to the 'I think I might need in memory so I'll put it all in-memory' or worse the 'Damn I had it all on disk and now I need in-memory'.

The future is about Analytics aligned to the business not aligned to an idealised IT view.

How Business SOA thinking impacts data

Over the years I've written quite a bit about how SOA, when viewed as a tool for Business Architecture, can change some of the cherished beliefs in IT.  One of these was about how the Single Canonical Form was not for SOA and others have talked about how MDM and SOA collaborate to deliver a more flexible approach.  Central to all of these things has been that this has been about information and transactions 'in the now' the active flowing of information within and between businesses and how you make that more agile while ensuring such collaboration can be trusted.

Recently however my challenge has been around data, the post-transactional world, where things go after they've happened so people can do analytics on them.  This world has been the champion of the single canonical form, massive single schemas that aim to encompass everything, with the advantage over the operational world that things have happened so the constraints, while evident, are less of a daily impact.

The challenge of data however is that the gap between the post-transactional and the operational world has disappeared.  We've spent 30 years in IT creating a hard-wall between these ares, creating Data Warehouses which operate much like batch driven mainframes and where the idea of operational systems directly accessing them has been aggressively discouraged.  The problem is that the business doesn't see this separation.  They want to see analytics and its insight delivered back into the operational systems to help make better decisions.

So this got me thinking, why is it that in the SOA world and operational world its perfectly ok for local domains and businesses to have their own view on a piece of data, an invoice, a customer, a product, etc but when it comes to reporting they all need to agree?  Having spent a lot of time on MDM projects recently the answer was pretty simple:

They don't

With MDM the really valuable bit is the cross-reference, its the bit that enables collaboration.  The amount of standardisation required is actually pretty low.  If Sales has 100 attributes for Customer and Finance only 30 and in-fact it only takes 20 to uniquely identify the customer then its that 20 that really matter to drive the cross reference.  If there isn't any value in agreeing on the other attributes then why bother investing in it?  Getting agreement is hard enough without trying to do it in areas where the business fundamentally doesn't care.

This approach to MDM helps to create shorter more targeted programs, and programs that are really suited to enabling business collaboration.  You don't need to pass the full customer record, you just pass the ID.

So what does this combination of MDM and SOA mean for data, particularly as we want analytics to be integrated back into operations?

Data solutions should look more like Business SOA solutions and match the way the business works

In simple terms it means the sort of thinking that led to flexibly integrated SOA solutions should now be applied to Data.  Get rid of that single Schema, concentrate on having data served up in a way that matches the requirements of the business domains and concentrate governance on where its required to give global consistency and drive business collaboration.  That way you can ensure that the insights being created will be able to be managed in the same way as the operational systems.

With SOA the problem of people building applications 'in the bus' led me to propose a new architectural approach where you don't have one ESB that does everything but accept that different parts of the business will want their own local control.  The Business Service Bus concept was built around that and with the folks at IBM, SAP, Microsoft and Oracle all ensuring that pretty much everyone ends up with a couple of ESB type solutions its the sort of architecture I've seen work on multiple occasions.  That sort of approach is exactly what I now think applies to data.

The difference?

Well with data and analytical approaches you probably want to combine data from multiple sources, not just your own local information, fortunately new (Java) technologies such as Hadoop are changing the economics of storing data so instead of having to agree on schemas you can just land all of your corporate data into one environment and let those business users build business aligned analytics which sit within their domain, even if they are using information shared by others.  MDM allows that cross reference to happen in a managed way but a new business aligned approach removes the need for total agreement before anything can get done.

With Business SOA driven operations we had the ability to get all the operational data in real-time and aggregate at the BSB level if required, with Business SOA driven data approaches we can land all the information and then enable the flexibility.  By aligning both the operational and post-transactional models within a single consistent Business aligned approach we start doing what IT should be doing all along

Creating an IT estate that looks like the business, evolves like the business and that is costed in-line with the value it delivers.

Applying Business SOA thinking to data has been really interesting and what led to the Business Data Lake concept, its early days clearly but I really do believe that getting the operational and data worlds aligned to the business in a consistent way is going to be the way forwards.

This isn't a new and radical approach, its just applying what worked in operations to the data space and recognising that if the goal of analytics is to deliver insight back into operations then that insight needs to be aligned to the business operations from the start so it can adapt and change as the operational business requires.

The boundaries from the operational and post-transactional world have gone, the new boundaries are defined by the business and the governance in both areas is about enabling consistent collaboration.

The only V that counts in Big Data is Value

So what is Big Data?  Its Variety, Velocity, Volume right?  But what does that really mean?  Should I get loads of data and drop it into Hadoop, pull in anything I can lay my hands on and I'm now 'doing Big Data'?

Should I plug in my packet monitoring software and store in Hadoop and I'm doing Big Data?  Should I get as many different data sources and that means I'm doing Big Data?

The reality is that one thing hasn't changed and its the key driver for Big Data in the way that it should be in any IT program - Value.  What is the point of what you are doing?  Does the business care? If there isn't a point or the business doesn't care then why on earth are you doing it?

Stop focusing on the 3 Vs of Big Data and worry only about the fourth, the one that really matters

Value.

Zuckerberg and the unreality of valley thinking

Bill Gates laid into Mark Zuckerberg's vision of internet access being the most important thing in solving the world's ills and another article at The Register compared Silicon Valley to 'Sheldonville' which I really agree with.  At the heart of lots of these 'visions' is an insular view of technology within Silicon Valley.

How insular?  Lets say Blue Ridge Mountains style insular.  The normal vision is that valley thinking will change the world if only people would just use that technology.  We saw it in the .com bust with companies that preached a huge vision on how they would replace the normal bricks and mortar and how everything would be changed and replaced....

And guess what?  It takes a hell of a lot longer to change culture than a VC/IPO cycle would like the end result is that the companies fail.  The VC/IPO money does however distort the market, look at Amazon's 'profit' statements against a food retailer - Wholefoods - and a failing retailer - Best Buy.

	Revenue	Profit
Amazon	$21.27bn	$97m
Wholefoods	$2.2bn	$113m
Best Buy	$16.7bn	$1.9bn

Now I know that Amazon aren't strictly a valley company but its the same sort of mentality, I'll come to some great examples of valley mentality in a second.  So Amazon is a great company doing some really cool and innovative things but thanks to their special technical sauce they are allowed to significantly undercut their competition because they don't need to make as much profit to keep shareholders happy.

That is a big advantage and one that distorts markets.

Now how about some other valley thinking examples?

• Jonathan Schwartz and the over optimism of 'Open Source will defeat all' and pitching to financial analysts about the number of downloads the company had is a good one.  It ultimately doomed Sun to acquisition and really was a great example of valley thinking.

• The startup I met whose strategy was 'rip out SAP' in order to use their software.
• The integration of JAX-WS and Javascript into Java
• The whole .com 'boom' based on 'visitor' numbers
• The whole social media 'boom' based on 'user' numbers
• Hell every single startup whose philosophy is that companies will rip out working technology to replace it with their new shiny and unproven technology

The valley often does come up with good stuff, but does valley thinking overall make the world a better place?  Sure the valley came up with Java.... and then the valley screwed it up.  The valley came up with REST... oh brilliant our lives are so much better.

My point of this rant is that software investment is driven by the thinking in a short stretch of land, some of it can be good, some of it can be bad, but when you look at companies like IBM, SAP, ARM, etc its quite amazing just how positive an impact they've had on technology, and arguably the world, without being driven at any stage by massive hype cycles.  I'd argue its exactly because these folks are mainly out of the valley that the often deliver practical innovations and improvements rather than impractical visions.

Zuckerberg's 'vision' for helping the world is just another example of that and Bill Gates is spot on about what is really important.  The valley is a great place, its a buzzing place, but its as much to do with the majority of technology decisions as Fantasy Football has to do with the NFL, sure they are related but one of them is doing real work, even if lots of money is going into the former, its the latter that tends to focus on profits.

Quality is a side-effect not the goal in MDM

I put out a tweet the other day with this title and I think its worth elaborating on what I mean.  Lots of MDM efforts I see have the goal of 'improve data quality' and this is a mistake.  I'm not saying that data quality isn't a good thing but that in itself its not actually a goal.

What do I mean?  Well lets take an analogy or three, if you are looking to buy a diamond then do you buy the very, very best and the very very biggest?  If you are looking for diamonds to use in cutting or industrial grinding then the answer is of course 'no', the quality really wouldn't be appropriate in those uses it would be a waste of money.  What if you are looking to put a 1.6 litre engine in a car aimed at the local commuter market, do you look around for the most powerful, most expensive, built to the highest quality standards?  Well that would probably be one of the engines slated to go into a Formula 1 car next season.  Sure it generates huge power and is a quality engine but its not fit for purpose.

Now for the final analogy.  You are looking to provide translation services for the Iranian nuclear discussions.  Should you go and get the cheapest price from someone who promises they can speak 'Iranian' or do you invest from someone who actually is proven as a translator for Persian, Gilaki and Mazandarani and describes their Kurdish as 'passable'?

The point here is that in each case the goal defines the level of quality required, quality in itself is about having an acceptable level of quality to meet your goal which in some occasions might be very little indeed.

So what is the real goal of MDM?  Its about enabling business collaboration and communication the power of MDM is really in the cross-reference, the bit that means you know the customer in one division is the same as another and that the product they are buying is the same in two different countries.  If the quality is awful but the cross-reference works then in many occasions you don't need to invest more in quality unless there is a business reason to do so.  Most of the time that business reason is that you cannot achieve the collaboration without having a decent level of quality.  To match customers across business areas requires you to have an standard definition, so your customer on-boarding needs a certain level of rigour, your product definition needs to work to standards that are agreed across the business.

So in focusing on the collaboration, in focusing on where the business wants to collaborate you focus MDM and you focus where quality needs to be achieves.  Focusing on quality as a goal is a very IT centric thing, focusing on collaboration and through that enabling quality is a business thing.

And MDM is certainly a business thing. 

Single Canonical Fail

There are few things out there in IT more delusional than the Single Canonical Form, the idea that IT can define a super schema, a schema so complete, so pure that all will bow down before it.  Sheer idolatry.  Whether it is for integration or for Data Warehousing the reality is that a Single schema is never going to be ‘canonical’, different people have different perspective and its this very contention between business areas that actually drives the business forwards.  Sales obeys the rules from Finance in certain areas and in others is in open rebellion as the KPIs for Sales compete against the need for regulation in Finance.  To forecast correctly means rigor and repeatability, but anyone who phones up Sales with an open checkbook is going to find their order fulfilled despite the claims of a sales process.

At the heart of a Single Canonical Form is a simple premise ‘as long as everyone can agree’ it’s the sort of premise that is wonderfully naïve in its inception. The reality is sadly that such a simplistic view ignores local perceptions and attempts to force a straight-jacket upon the business by providing a single, almost Stalinistic, view upon them.  By starting with that beguiling premise IT sets out on a journey that can only end on failure.  The Sales, Finance and Operations teams all have local KPIs, different division and regions have different strategies and all may have a different view on how they sell and whom they sell to.  This does not mean the business is dysfunctional or wrong, it simply means that the business is complex and not constrained within a single view of what should be done.

The Single Canonical Form aims to achieve the unobtainable and by doing so creates its own downfall.  Because it doesn’t meet the objectives of everybody then individuals are forced to create their own local solutions as the agility of the single canonical form is relatively, or indeed astronomically, low.  The goal of a single canonical form is to create a single view on one of the most variable things in a business: the view on information.  One part of the business may require only 10 pieces of information about a customer, another 200, neither are right or wrong it is simply their own local information, they critical element is that an individual customer be recognized across the two, not that 210 attributes be agreed.  The same goes for invoices, orders, contacts and everything else: agree when it matters, don’t bother when it doesn’t.

The Single Canonical Form is a straight-jacket on the enterprise, it’s a dumb idea based on an unachievable idea.  Its time for IT to grown up and work differently.

Speaking in Public isn't private - and the internet is a public space

With all the scandal around Edward Snowden I have to say I'm mostly in the camp of 'surely everyone knew that spying agencies spied on people?', but the most surprising is when it comes to the 'scandal' that they might be listening to our communications over the internet.

The internet?  The one created and originally funded by DARPA?  The open internet with the IP protocol that means packets are by default openly routed and unencrypted?

Or do you mean SMTP with its unencrypted openly routed emails?

Or HTTP with its unencrypted data?  Even HTTPS only encrypts the data, you can still openly find out what page someone was looking at in terms of the IP address just not the data being exchanged.

Seriously did anyone actually think that people are not watching this stuff?  When did this become a surprise?  I knew that 30 years ago people were spying on this.  Didn't we all have a .sigs back then that said 'Hello to my friends in Cheltenham and Langley'?

Agencies have spied on people, even allies in fact maybe ESPECIALLY allies for hundreds of years.  Its the whole point of funding spying agencies and the internet just makes that spying easier.  Its a public conversation that you are having on Twitter, Facebook or over email.  You might think its private but its really just shouting out of a window and public speech isn't something you should be surprised if its being tracked.

This isn't a question of 'only the innocent have nothing to fear' but its a question that actually by bringing this more into the open we risk it being used beyond its current scope of spies and into regular police forces and its that which would scare me more.  The whole risk is that spying becomes a mainstream police activity not something from specialist organisations whose primary focus is on genuine national threats, not someone forgetting to put the garbage bin out on a Tuesday night.

Spying is real, it has been for hundreds of years and its sadly got a place in a world of cyber and other terrorist threats.  That people have got upset about the tracking of information over public networks says much more about the lack of understanding of how the Internet works than of a big brother state that is completely new.

Surely REST isn't the travelling salesman does design?

Occasionally I run across things on the web, this time tweeted by Mark Baker (@distobj), that make me read them several times.  The link he tweeted is to this page on a Nokia API and in particular this section...

Biggest advantages noticed is that the API itself becomes the documentation of the API

• Without HATEOAS developers would get data, go to a documentation, and figure out how to build the next request

• With HATEOAS developers learn what are the available next actions

Either of these would get me firing you I think.  I'm a big fan of documentation and I'm a big fan of design.  What I'm not a big fan of is people who treat the design process as a series of abstract steps when the only thing that matters is the next step.

Lets be clear, having clear documentation available is important.  What would be poor would be two things:

1. Having to wait for someone to build an application before I can start writing a client for it
2. Having documentation that is some sort of 'mazy of twisty passages' where you only see one step ahead

This to me is at the heart of the death of design in IT, the lauding of everything as architecture and the massive chasm that appears to be developing between that and writing the code.  I'm repeatedly seeing approaches which are code centric rather than design centric and the entire history of IT tells us that this isn't the best way forwards.  Don't try me on the 'I just refactor' line as if that is an answer, spending 1 day thinking about a problem and planning out the solution (design) is worth 5 days of coding and 20 days of subsequent refactoring.

I'd really like a developer to be able to map out what they want to do, be able to read the documentation in one go and understand how they are going to deliver on the design.  I don't want a developer context switching between API, code and pseudo-design all the time and getting buried in the details.

This is part of my objection to REST, the lack of that up-front work before implementation - if I have separate client and service teams I don't want to run a waterfall project of 'service finishes, start client' and if those are two separate firms I want to be able to verify which one has screwed up rather than a continual cycle of 'its in the call response' and 'it was different yesterday'.  In other words I'd like people to plan for outcomes.  This doesn't mean not using REST for implementation it just means that the design stage is still important and documentation of the interface is an exchangeable artefact.  Now if the answer is that you have a Mock of the API up front and a webcrawler can extract the API documentation into a whole coherent model then all well and good.

Because the alternative is the travelling salesman problem.  If I don't know the route to get things done and am making a decision on the quickest way one node at a time then I'm really not looking at the efficiency of the end-to-end implementation just the easiest way to do the next step.

This current trend of code centric thinking is retarding enterprise IT and impacting the maintainability of REST (and other) solutions.  This isn't a question of there being a magic new way of working that means design isn't important (there isn't) its simply a question of design being continually undermined and discarded as an important part of the process.  Both of the scenarios outlined in the article are bad, neither represents good practice.  Choosing whether your manure sandwich is on a roll or a sub doesn't change the quality of the filling.

Think first, design first, publish first... then code.

Minimum on the wire, everything in the record

I've talked before about why large canonical models are a bad idea and how MDM makes SOA, BPM and a whole lot of things easier.  This philosophy of 'minimum on the wire' helps to create more robust infrastructures that don't suffer from a fragile base class problem and better match the local variations that organisations always see.

One of the things I really like about IT however is how it starts putting new challenges and how simple approaches really make it easier to address those new challenges.  One of those is the whole Big Data challenge which I've been looking at quite a bit in the last 12 months and there is a new philosophy coming out of that which is 'store everything', not 'store everything in a big data warehouse' but 'store everything as raw data in Hadoop'.    There are really three sources of 'everything'

1. Internal Transactional systems
2. External Information & transactional systems
3. Message passing systems

So we now have a really interesting situation where you can minimise what is on the wire but store much more information.  By dropping the full MDM history and cross reference into Hadoop you can use that to say exactly what the information state was at a point in time when a message was passed across the bus.  In other words you can have a full-audit trace of both the request and the impact that the request had on the system.

One of the big advantages of Hadoop is that it doesn't require you to have that big canonical model and again this is where MDM really kicks in with an advantage.  If you are looking for all transactions from a given customer you just take the system x-ref from MDM as your input and then you can do a federated Map Reduce routine to get the system specific information without having to go through a massive data mapping exercise.

Hadoop means there is even less reason to have lots flying about on the wire and even more justification for MDM being at the heart of a decent information approach.

Google and Yahoo have it easy or why Hadoop is only part of the story

We hear lots and lots of hype at the moment around Hadoop, and it is a great technology approach, but there is also lots of talk about how this approach will win because Google and Yahoo are using it to manage their scale and thus this shows that their approach is going to win in traditional enterprises and other big data areas.

Lets be clear, I'm not saying Hadoop isn't a good answer for managing large amounts of information what I'm saying is that Hadoop is only part of the story and its arguably not the most important.  I'm also saying that Google and Yahoo have a really simple problem they are attempting to fix, in comparison with large scale enterprises and the industrial internet they've got it easy.  Sure they've got volume but what is the challenge?

1. Gazillions of URIs and unstructured web pages
2. Performant search
3. Serving ads related to that search

I'm putting aside the gmails and google apps for a moment as those aren't part of this Hadoop piece, but I'd argue are, like Amazon, more appropriate reference points for enterprises looking at large scale.

So why do Google and Yahoo have it easy?

First off while its an unstructured data challenge this means that data quality isn't a challege they have to overcome.  If google serve you up a page when you search for 'Steve Jones' and you see the biology prof, sex pistols guitarist and Welsh model and you are looking for another Steve Jones you don't curse google because its the wrong person, you just start adding new terms to try and find the right one,  if Google slaps the wrong google+ profile on the results you just sigh and move on.  Google don't clear up the content.

Not worrying about data quality is just part of the not having to worry about master data and reference data challenge.  Google and Yahoo don't do any master data work or reference data work, they can't as their data sets are external.  This means they don't have to set up governance boards or operational process changes to take control of data, they don't need to get multiple stakeholders to agree on definitions and no regulator will call them to account if a search result isn't quite right.

So the first reason they have it easy is that they don't need to get people to agree.

The next reason is something that Google and Yahoo do know something about and that is performance, but here I'm not talking about search results I'm talking about transactions, the need to have a confirmed result.  Boring old things like atomic transactions and importantly the need to get back in a fast time.  Now clearly Google and Yahoo can do the speed part, but they have a wonderful advantage of not having to worry about the whole transactions stuff, sure they do email at a great scale and they can custom develop applications to within an inch of their life...  but that isn't the same as getting Siebel, SAP and old Baan system and three different SOA and EAI technologies working together.  Again there is the governance challenge and there is the 'not invented here' challenge that you can't ignore.  If SAP doesn't work the way you want... well you could waste time customising it but you are better off working to what SAP does instead.

The final reason that Google and Yahoo have it easy is talent and support.  Hadoop is great, but as I've said before companies have a Hadoop Hump problem and this is completely different to the talent engines at Google and Yahoo.  Both pride themselves on the talent they hire and that is great, but they also pay top whack and have interesting work to keep people engaged.  Enterprises just don't have that luxury, or more simply they just don't have the value to hire stellar developers and then also have those stellar developers work in support.  When you are continually tuning and improving apps like Google that makes sense, when you tend to deliver into production and hand over to a support team it makes much less sense.

So there are certainly things that enterprises can learn from Google and Yahoo but it isn't true to say that all enterprises will go that way, enterprises have different challenges and some of them are arguably significantly harder than system performance as they impact culture.  So Hadoop is great, its a good tool but just because Google and Yahoo use it doesn't mean enterprises will adopt it in the same way or indeed that the model taken with Google and Yahoo is appropriate.  We've already seen NoSQL become more SQL in the Hadoop world and we'll continue to see more and more shifts away from the 'pure' Hadoop Map Reduce vision as Enterprises leverage the economies of scale but do so to solve a different set of challenges and crucially a different culture and value network.

Google and Yahoo are green field companies, built from the ground up by IT folks.  They have it easy in comparison to the folks trying to marshall 20 business divisions each with their own sales and manufacturing folks and 40 ERPs and 100+ other systems badly connected around the world.

Big Data - are you the house or the played?

Coming back from the EMC World conference in Las Vegas I was looking at the people playing on the slots and making ridiculous bets at Craps and wondering 'don't these people know anything about statistics?'.  Lets be clear I get the idea of it being fun, but when you sit next to someone on a blackjack table who twists when the dealer is showing a six and they've got 15 is just depressing.

There is a business lesson for us here about the power of data and in particular the differentiation that Big Data will have between businesses.  Those that really leverage Big Data and Predictive analytics and integrate it back into business processes will have a significant 'house' advantage over their competition.  The retailer who knows what the next hot trend will be or who can negotiate prices based on a long term demand view will under-cut and out perform their rivals.  In the same way as Vegas knows the returns on the slots and table games and innovates to improve their odds so Big Data and Predictive Analytics will help some companies dominate their sector.

Things like Hadoop, R and the like are just tools in this journey and companies that focus on the technology are like gamblers who claim to have 'a system' for a game like roulette.  Sure they might get lucky a couple of times but over time they are just going to lose to the house.  The companies that win will be the ones like the casinos which run statistics and predictive models to a level that helps to assure their advantage.

So are you the house or the played?

Federated Caching in the world of the 64GB mobile

There is something that is beginning to irritate me, ok something else.  Its mobile applications that don't cache.  I'm fed up of travelling on a train or being on a plane and the end result being that my iPad or iPhone app doesn't work because I'm in an area that doesn't have reasonable network coverage.  I was using the App earlier when it had WiFi and it was all fine but the app requires me to be 'always connected' which I just feel is plain lazy.  My mobile phone has about 16GB of free space, my iPad has slightly less thanks to the videos but its still several GB of space.

I'm particularly talking here about BI tools and enterprise apps which have no concept of a variable network.  If I've approved a travel expense for someone via email then when I connect the email gets sent, if its on a mobile 'always on app' I can't do that.  Why?  I'm actually losing functionality over email.  If its a BI app and I'm looking at sales reports for a given geo then if I can't access it on the plane then I'm losing functionality over Excel.

Federated Caching is often a tough challenge but is one that we've solved over, and over, and over again in IT.  It shouldn't even be 'save to' on the device it should be a case that if you are doing the reports and have a cache size set on your device then the application should automatically pull the information locally.

The next generation of enterprise mobility will be about removing things like email and Excel or it will just be another pretty technology that does some stuff better but I'm still forced back to the old tools when I step outside the mobile view of a perfectly connected world.

Federated caching needs to be your starting point in a mobile world, you need to leverage the power of the device in a smart way not use it as just a browser and you need to deal with the reality that your users will not always be connected no matter what the adverts say.

Software Developers are you ready for the cage fight with the BI guys?

In all of my career to date in IT there really has been three clear worlds in IT, the software development guys who are the bespoke tailors, the package guys who deliver off the shelf and the BI guys.

I'm going to admit a prejudice here.  Until a couple of years ago my impression of BI guys was folks who were one step up from Excel guys.  It was dead-data and done in batches, sure it had to be done but it really wasn't gool.  The BI guys couldn't handle the operational requirements of the software developers and we really were not interested in Cubes, Star-Schemas and all of the stuff they did to generate what was basically an Excel spreadsheet on steroids.

Now having spent some more time on that side of the fence I've come to respect it more, but there is now an interesting evolution.  Contextual BI.  That is where BI guys talk about integrating BI and Analytics information back into operational processes.  In other words its BI guys talking about integrating to SAP and integrating into BPM and integrating into operational applications generally.

An IT turf war is about to break-out.  I'd argue that the software development guys have a bit of an advantage here, what is the current 'cool kids' BI tool?  Its Hadoop.  What do you need to know to really grok Hadoop?  Yup Java, a software development language.  Software Dev guys grok agile in a way that few BI guys do and are used to getting down into operational requirements.

But we suck at analytics and its massively expensive to develop the sorts of visualisations that BI guys get out of the box.  So this means its either a case of one side taking over the other, or we are going to have to co-operate... and I actually think this is the way forwards.  Software Development guys know how to do real-time scaling and operations, its not a trivial skill, and the BI guys know how to get us the right information into our processes and have the right visualisation tools.

So Software Developers out there its time to reach out and embrace your BI cousins and start looking at how you can work together to create an information fabric which spans the analytical models on historical data, supplements it with real-time analytics and then integrates that back into the operational process.  I know this collaboration is going to be hard and we've all got to get over prejudices to make this happen but the business is demanding it so we need to get it done.  Start thinking about information as a seemless infrastructure where delivery to the point of action is the challenge not delivery to a report.  Software Developers and BI guys working together can create this and deliver a real information fabric to the enterprise.

And don't worry Software Developers and BI guys will still have a common enemy, I'm not crazy, its still perfectly ok to not get on with the package guys.

IT is a fashion industry

You know when people laugh at the fashion industry for saying that 'blue is the new black' and because of its ridiculous amount of fawning over models, designers and the like?  Is that really different to IT?  We've got our fashion houses - Google, Facebook, Apple.  We've got the big bulk conglomerates IBM, Oracle, SAP, Microsoft and oh hell the fawning that goes around...

I'd say the comparison goes even deeper however.  EAI, Web Services, REST... what are these?  They are all integration approaches.  EAI was going to save the enterprise and create a well managed estate that the business could use and could be changed easily and enable integration with external companies... Web Services were going to save the enterprise by standardising the interfaces enabling a well managed estate the business could use and could be easily... REST was going to save all of IT by enabling interfaces that could be dynamically changed and enable integration...

The point is that the long term challenge is the same, system to system integration, yet we have fad based approaches to solve that challenge.  Its like the fashion industry and dress lengths, it goes up and down, but its still a dress.  The real difference in IT however is that the fashion industry does this better, sure they change the hem, but it still works as a dress.  In IT we concentrate so much on the hem length that we don't even bother with the fact that system to system integration appears to be as hard in 2013 as it was in 1999.  We even know why, the Silver Bullet tells us that technology won't solve the problem on its own.  But do we listen?  No because we are followers of fashion.

This analogy to fashion applies to the age discrimination in IT, we love the young and shiny, and age discrimination against new entrants is wonderfully not present.  However the flip side of that is there is an over emphasis on the new in IT, so we prefer doing things in 'new' ways rather than in 'working' ways, and unlike in the fashion industry we don't actually learn from sales what is successful.  If we've got a fad (hello REST) that works in some places but not in others we'll keep on pushing that fashion even as it fails to set the world on fire.  Its the emperor's new clothes effect, and in IT we do the equivalent of the beauty industry.  In the beauty industry you'll see adverts for 'age defying creams' advertised by 16 year old models.  In IT you'll see enterprise solutions pushed by using Google as an example.  We love the new, we love the young, and we really rather hate facing up to the fact that IT is quite an old industry now and 90%+ of the stuff out there is a long way from new and shiny.

The analysts and vendors are the Vogue and Fashion Houses in this world, the pushing of the new as the 'must have' technology and dire warnings if you dare to actually make the old stuff work.  The concentration on the outfit (the technology product) and little about how it actually works in the real-world (operations).  You know when you see outfits at London, New York, Milan or Paris fashion weeks been shown on the news under the 'what madness do designers think we will wear next' section.  Is that so different from an analyst or vendor pushing a new technology without explaining at all how it will fit into the operations of your current business?  We will see things like people declaring the end to SQL... and then a few years later those same people championing SQL as the approach, now that they've realised people can operate their technology if they do that.

The final place I'll talk about IT and fashion is in the 'rebadging' that we see.  In the fashion industry you see old ideas rehashed and pushed down the catwalk as being 'retro'.  There is at least some honesty in the fashion industry as they talk about it being inspired by an era, when we all know what they mean is 'I didn't have an original idea, so I copied one that was old enough that people will think its original to copy'.

In IT we don't even have the honesty of the fashion industry, what we do is see a new trend and claim that old technologies are actually part of that new trend.  We'll take an old EAI tool and slap on an SOA logo, we'll take a hub and spoke broker and call it an ESB.  This re-badging of technology goes on and on, sometimes you'll be in a meeting and suddenly realise 'hang on, I used that 12 years ago... how the hell is it now new?'. This would be fine if the focus was on building a robust product, but too often its just about how to get on an RFP and shift a few more units with actual investment in new approaches being few and far between.

IT and the fashion industry are miles apart in many ways, but the faddish nature of our industries make us very similar.  The problem is that fashion is allowed to be faddish, its not expected that a business will rely on something made 20 years ago but with IT this faddish behaviour is a big problem.  We are meant to be constructing systems on which a business can rely, not just today but 5 years from now and still be leveraging in 10, 20 or even 30 years time if its well constructed and does the job well.  There are mainframe systems out there doing exactly that, and why haven't they been replaced?  Because the new stuff didn't do the job.

IT needs to stop being like the fashion industry and more like the aircraft manufacturing industry, sure they have 'fads' like an all composite aircraft, but that is based on sound data as well as strategic vision. Its not just based on it being what the cool kids do.   We can do the cool stuff, we can do the new stuff, but we need to recognise that there is lots of stuff out there that needs to change, we can't just use Google or Facebook as references or examples, that would be like Boeing selling a plane technology based on what people did in movies, its just too far removed from the enterprise reality.  We need to stop blindly following IT fashion and start critically appraising it and shouting 'emperor's new clothes' when its bullshit.  Most of all we need to look at enterprise technologies based on how they improve the 'now' not based on 'if only we could replace everything', evolution is the revolution in IT.

Its time for IT to grow up and take responsibility for the mess we've created.

The Hadoop hump - why enterprises struggle to move from Proof of Concept to Enterprise deployment

At the recent Hadoop Summit in Amsterdam I noticed something that has been bothering me for a while.  Lots of companies have done some great Proof of Concepts with Hadoop but they are rarely turning those into fully blown operational solutions.  Being clear I'm not talking about the shiny, shiny web companies where the business is technology and the people who develop are the people who support, I'm talking about those dull companies that make up the 99.9% of businesses out there where IT is part of the organisation and support is normally done by separate teams.

There are three key reasons for this Hadoop hump

1. Hadoop is addressing problems in the BI space, but is a custom build technology
2. Hadoop has been created for developers not support
3. BI budgets are used to vertically scaled hardware

These reasons are about people not technologies.  Hadoop might save you money on hardware and software licenses but if you are moving from report developers in the BI space to Map Reduce/R people in Hadoop and most critically requiring those same high value people in support its the people costs that prevent Hadoop being scaled.  The last one is a mental leap that I've seen BI folks struggle to make, they are used to going 'big box' and talking about horizontal scalability and HDFS really doesn't fit with their mindset.  

These are the challenges that companies like Cloudera, Pivotal and Hortonworks are going to have to address to make Hadoop really scale in the enterprise.  Its not about technical scale, its about the cost of people.

Big Data, Fast Data, Orange Data, Blue Data - its decisions that count not data

Oh the chanting is out, Big Data, Fast Data, the three 'V's and of course the ubiquitous elephant are roaming across the IT landscape as the next great hype monster.  Its going the same way as pretty much every IT hype exercise.  Yes this links to the hype cycle but the way it happens is sadly predictable in IT.

1. Company has a business problem they think about it in an innovative way
2. Company needs to build a new piece of technology to help with the process, or chooses to just because they can
3. Company announces the business results they have
4. IT industry focuses on the technology
5. IT industry creates a sticker - 'Big Data'
6. IT vendors produce versions of that technology or start claiming their technology does that anyway
7. IT vendors ramp up the marketing spend around the technology
8. Everyone forgets why this all started

Why is it that Google and Yahoo created and use Hadoop?  Well its because they had a completely different scale of data problem and needed to do challenging analytics in a different way.  Whether it be ad serving or search itself the point was that the aim was functional the data was required to deliver on that functional goal.

When people rave about Social Media, Open Data and various Big Data sources and talk about Hadoop in glowing terms that is all fine and good, but the real point here is about decisions and making them better.  Its about what you can improve in your business not about having to store every piece of data out there.  Concentrating on HDFS as being the important thing in Big Data is looking at flooring being the only important piece of a house.

The hype is on with Big Data, the hype is on with Fast Data but the reality is back to why people started looking at these volumes of information and critically about what it actually means when you start considering massive scale information in terms of governance, management and analytics and that all comes back to a simple set of questions.

What decisions do I need to make better?  What information do I need to make those decisions better? How do I get the right analysis of that information delivered to the point where the decision is made?

We need to stop focusing on Hadoop, R, HANA and all of the technical pieces and start looking at the real trend here, and that is the integrating of analytics into operational processes, the old world of transactional data v analytical data has gone and that is a massive change for IT, not simply in technology but more critically in mindset.

The real shift is not 'Big Data' its the end to post-transactional reporting, its about a single IT infrastructure that mixes real-time information with historical analytics to support better operational decision making.

As with SOA the real shift is a mental one not a technical one and its the mental shift that is hardest to achieve.  Can traditional IT departments move away from the separation of BI from Operational systems?  Clearly the business is going to do that so the only question is whether the IT department is along for the ride or is just the kindergarden where people play with toys.

The single eye of enterprise architecture

There is a famous phrase

In the kingdom of the blind, the one eyed man is king

IT has always had a problem communicating with the business, and the business communicating with IT.  To fix this IT created something called Enterprise Architecture which aimed to provide a framework around the internal IT estate in a manner that helped that conversation. We can argue how successful that was but what is clear is that with the rising consumerisation of IT and the massive increase in technical literacy amongst business people that this boundary approach to IT with EA as the gatekeeper isn't really going to work.  Historically the EA function was that one-eyed man, it had enough depth in IT to provide some control and enough engagement with the business to act as the leader through the challenges of IT.

The business isn't blind to IT anymore.

So what has been the reaction?  Unfortunately it appears to be in some cases to take a single-eyed view of the problem, a single eye looking through a single lens.  That single eye recognises the increase in sophistication of the business and sees the rise of Value Networks and Business Architecture and thinks 'Oooooh, we have to do the business architecture stuff as well' and makes that a sub-section in the overall encompassing enterprise architecture.

This is missing an opportunity.  A huge opportunity.  EA was created in an era where IT organisations were internal only and where external integration was extremely rare.  In the world of SaaS, Cloud, Social and OpenData however this is no longer the case, outside-in is the norm.

Sure we need groups to looking at integration standards and help with IT operations, but the communication with the business doesn't need an interpreter in the same way anymore. Business Architecture is a rightly growing area but its not a subset of EA, or even an extension of EA its a discipline in its own right, one that owes more to business schools than to IT departments.  So what does IT need to do?  Well it needs to be able to translate that architecture into solutions much more quickly.

IT needs to look at the business with two-eyes again, not try and extend and old concept through its own view of the world.

Why NoSQL became MORE SQL and why Hadoop will become the Big Data Virtual Machine

A few years ago I wrote an article about "When Big Data is a Big Con" which talked about some of the hype issues around Big Data.  One of the key points I raised was about how many folks were just slapping on Big Data badges to the same old same old, another was that Map Reduce really doesn't work they way traditional IT estates behave which was a significant barrier to entry for Hadoop as a new technology.  Mark Little took this idea and ran with it on InfoQ about Big Data Evolution or Revolution? Well at the Hadoop Summit in Amsterdam this week the message was clear...

SQL is back, SQL is key, SQL is in fact the King of Hadoop

Part of me is disappointed in this.  I've never really liked SQL and quite liked the LISPiness of Map Reduce but the reason behind this is simple.

When it comes to technology adoption its people that are key, and large scale adoption means small scale change

Think about Java.  A C language (70s concept) derivative running on a virtual machine (60s)  using some OO principles (60s) with a kickass set of libraries (90s).  It exploded because it wasn't a big leap and I think we can now see the same sort of thing with Hadoop now that its stopped with purity and gone for the mainstream.  Sure there will be some NoSQL pieces out there and Map Reduce has its uses but its this change towards using SQL that will really cause Hadoop usage to explode.What is good however is that the Hadoop philosophy remains in-tact, this isn't the Java SE 6 debacle where aiming after 'Joe Six-pack' developer resulted in a bag of mess.  This instead is about retaining that philosophy of cheap infrastructure and massive scale processing but adding a more enterprise friendly view (not developer friendly, enterprise friendly) and its that focus which matters.

Hadoop has the opportunity to become the 'JVM of Big Data' but with a philosophy that the language you use on that Big Data Virtual Machine is down to your requirements and most critically down to what people in your enterprise want to use.

Its great to see a good idea grow by taking a practical approach rather than sticking to flawed dogma. Brilliant work from the Hadoop community I salute you!

BTVision iPlayer not working - and the excuses BT use

I've had an experience in the last few months with the folks from BT that really brought home how bad support can be when they have a real issue and how they look to trot out excuses to fob you off when they dont actually have a fix.

So in January iPlayer stopped working at my mother's house.  She isn't a techy so I took over support.  The first call went fine, and a secondary follow up call with 2nd line support was scheduled.

They said the issue was the use of a 3rd party router

Now the challenge is that iPlayer was working with that router previously, also every other iPlayer device in the house is working perfectly well, laptops, iPads, iPhones.  So I check it out with the official router, the problem remains, but I switch back to the 3rd party Asus router as its quite a bit better.  This is BT Infinity so the router is purely the LAN and WiFi.

For the next few calls the story was the same and a 2nd line call was scheduled.  This phoned the wrong number, left a message saying they'd call back... which never happened.

The next time, and we are six weeks in now, the story changed.

This is a well known fault with the BT Vision box it will be fixed in 48 hours

3 days later... yup you guessed it, still no iPlayer so the calls start again.  On each of the previous calls I've got through the menus, let them profile the line and do everything they want to.  I must have seen BBC iPlayer failing to work on 4 different calls.  So this time I just said 'I want this fixed, how can you fix it?'

The YouView box doesn't have this problem

But the person on 1st line support can't do anything about that so their supervisor is called (actually they were meant to call me back, didn't so I called again) this person tries to transfer me to the customer options team... but can't actually make the transfer properly to an agent to explain the situation so just dumps me in the queue.

Customer Options tell me that they can't switch my mother to YouView for .... £300.  I point out that my problem is just to get a box that does what BT advertise BT Vision for, namely having BBC iPlayer.  This is apparently not their problem and in the words of the UK call centre person

I've never heard of a problem with the BT Vision box

So she can't help me, she just wants my mother to pay several hundred quid or sign up to a new more expensive contract.  In other words to get the service that BT advertised her answer is to charge more money... isn't that extortion?  No that involves threats... its fraud I think.  Her coup de grâce however was offering to transfer me back to the fault support people who'd transferred me to her in the first place as they said they couldn't fix the problem.

So then I tweeted to BT Vision who referred me to BT Care who got me to fill in a form which resulted in a call which.... resulted in me talking to the fault department where I was told the following...

1. This has been a fault on all BT Vision boxes since January 11th
2. The issue is with the Content Delivery partner, the internet is ok but for BT Vision the BBC iPlayer/TV signal isn't being broadcast
3. The internet iPlayer is ok but BT Vision doesn't use the internet for iPlayer

I know that the last 2 are really the same excuse but I've put it down twice as I really got her to go over, and over, this topic and she did change the story a bit.  But her claim was that the issue was with the content delivery partner.  I said I just didn't believe this as I'd never heard of BBC iPlayer working over the air and couldn't see how that would really work.  She then settled down to

Its a software problem and it will be fixed on 31st March

So that was a nice clear statement, very like the '48 hours' statement and one that I couldn't help think was just meant to get me off the phone and have me waiting around before calling again in 3 weeks because they really have no clue and no intention of fixing the problem.

What has been amazing in this is just how disconnected the systems are at BT.  The BT Options team had no view on my customer complaints and service history, quite clearly or they'd have seen the fault report.  The fault people clearly had no idea about the Options team and how they worked as when they offered to switch me to them they didn't know that it was simply a question of extorting money.  2nd line support is clearly not interested in fixing problems but just making calls and getting off the phone and if the reality is that this is a broad problem with BT Vision there is a ridiculous amount of fluff and bullshit being thrown around.

If it really does impact all BT Vision customers then why isn't it more visible on the BT Help site?  The only issues appear to be from 2012.  If its really to do with that router (I doubt it) tell me the ports and protocols and I'll test/fix it.  If the only box that BT have that works as advertised is YouView then shouldn't I just get that box so I get the service they advertised?

The shame is that this is the only thing that doesn't work, but its become such an integral part of how people watch TV these days its a major issue.  BBC iPlayer is a key feature of BT Vision and it not working massively devalues the box, worst however is the constant bullshit and fabrication by BT Support.  I took over doing this as my mother isn't technical, I hate to think what they would have told her had she phoned.  So at the end of this BT are saying they have a known fault, its been known since January but....

Its taken 2 1/2 months for them to admit that

Is it really worth wasting my time, call time and support time on a known issue in the hope that you can fob people off to think its their mistake or issue?  That really doesn't seem to be the way to run a support call, the lack of a single view on the customer and the ability for someone to actually case manage across channels and areas is also a massive gap.  Dealing with BT is less like dealing with a company and more like dealing with a series of separate organisations whose only interest is to shift the blame to someone else.

Not good, not good at all

Java needs to start again... from BEFORE JavaSE 6

Back in 2006 I wrote a post on why JavaSE 6 wasn't for the Enterprise with all of the cruft that had been added and one comment I made around the inclusion of a Web Server was its potential impact on security.  Well it appears that thanks to this bloatware approach and lack of focus on core stability Java is now one of the number 1 security threats out there.  Apple have been hacked thanks to this and government agencies are saying turn off Java, something I've done myself.

Can anyone in Java land seriously imagine this happening before JavaSE6?  Java was the secure platform, the rock solid, no security issues, bet your business on it.  We'd laugh at Flash, we'd guffaw at Microsoft and chuckle uncontrollably at the script kiddies.  So did JavaSE 6 even meet its aim of keeping Java relevant with Joe Sixpack developer?  Well no, we've seen a massive increase in language fragmentation since JavaSE 6.  Does the current roadmap for Java address the structural and industry trends of things like Android, Big Data, Enterprise Computing?  Again I'd have to say no.

I'm a Java fan, I spent a large bunch of time investing heavily in the Java community and I got a lot out of it.  Its horrible to see how its now a hack and slash platform riddled with security issues.  So lets face it

1. Java as an in browser solution is dead
2. Java for the desktop will make it the year after Linux does

Therefore can we please concentrate on the core of Java, both for mobility (remember when Java dominated mobile games and apps?) and for the enterprise computing market... you know where the money is.

Java has, since JavaSE 6, been an unmitigated disaster, its turned strong foundations and principles into a complete and utter mess.   I was on JavaSE 6, I saw it happen before my eyes and I couldn't stop it because (lets be blunt here) I'm no James Gosling.

Java needs to recognise the market and restructure around it or its irrelevance and the massive impact of that on enterprise software and support is assured.  I know I speak for lots of Java guys out there when I say we are up for the challenge and we want to help but that starts by recognising the scale of the mistakes made and drawing up a completely new roadmap based on a clear vision of what enterprise and mobile computing needs today.

Big Data and smart maths aren't new, that is the GOOD THING about it

One of the things that annoys me sometimes, and its quite a long list, is when people proclaim something as 'new' when in fact its just a case that its gone mainstream.  The problem I have with this is that it normally means that they've forgotten all of the learnings of previous generations of implementation and are starting from scratch and making the same old mistakes.  We saw this with the rise of the internet which threw 20 years of research into human computer interaction and it took us about 10 years to get to the stage where people started thinking about design in a human rather than media company way (hat tip to Google who seemed to have read their technology history).

Big Data and Predictive Analytics aren't new either.  The UK's Met Office has been a Big Data & Predictive Analytics organisation since its foundation in 1854... 1854 folks.  Now sure you can say 'oh but they didn't have Hadoop' but that doesn't mean that in context it wasn't a preditive analytics organisation from its foundation and what its being doing for the last 30+ years has certainly been 'Big Data' and Predictive Analytics.  Simulations for Nuclear Explosions are pretty Big Data as well, and those clever chaps down at CERN and the LHC are building on decades of work in handling and processing massive data sets.  Remeber SETI@Home?  A project that had a massive data set and farmed out small chunks of it to individual servers for it to be processed then returned... almost Map Reduce like one would say.

What about in-memory data and real-time analytics?  Well have you ever flown on a plane?  Ever wondered just how the Air Traffic Controller sees your flight information, trajectory and warnings if it looks like you might crash into another aircraft?  Strangely this doesn't involve a series of database queries, its a stream of information with some very smart maths which identifies the planes, some other maths which works out their trajectories and then yet more maths which looks for potential collisions.  All of this is done in real real-time, not 'real-time' in a business sense where a second is ok, but if it takes 5 then so what you'll just moan a bit, real-time as in people might die if you don't do all of this in a fixed time.

The point that makes Big Data and Predictive Analytics a REAL trend and one that will have an impact is that we actually know this stuff works.  Its just that now its cheaper for other people to do rather than governments and large scale financial organisations.  We know that stochastic maths and other rather clever approaches can give decent predictions in chaotic systems, we know that you can adjust models in real-time, but only certain types of models and we know that there are limits to what things like Map Reduce can do from an analytics perspective that the in-memory is great but that memory is not infinite (unless you have a Turing Machine).

We do this commoditisation, for that it was it is, of technologies a disservice by claiming its 'new'.  Its power is that it is now a commodity, something available to all, that doesn't make it magic and that doesn't make it bullshit, what it makes it is REAL.

Big Data and Predictive Analytics will work because its built on solid foundations which are proven over decades of use.  It will fail if people don't recognise that and start from scratch because one thing that is true is that Predictive Analytics can be hard, infinitely more so if you don't learn from history.