How integration guys created a data security nightmare

There has been a policy in integration that has stored up a really great challenge of data security, and by great I don't mean 'fantastic' I mean 'aw crap'.  Its a policy that was done for the best of reasons and one that really will in future represent a growing challenge to Big Data and federated information.

The policy can be described as this:

Users authenticate with Apps, Apps authenticate with the database and Apps authenticate with the ESB/EAI/Integration

What this means is that Users don't authenticate against any of the federated data.  This is normally glossed over by saying that 'the source application is responsible for filtering' but the reality is that applications rarely do this terribly well.  Put it this way, most of the time when a front end banking system accessing your account information the only reason its getting your account data is because of the account ID it has, if it for some reason had the wrong account ID stored then it would display something else even though that information isn't yours.

This approach has tended to work for operational systems however because they work in linear ways on data sets, you know what you are showing and you pull out what you want to show.  The trouble is as these federated sets are then shifted into next generation Big Data solutions or accessed in a federated query is that the security model completely breaks down because application to application security doesn't recognise the individual actually making the request.

So now we have a world where the data sources don't do data security and privacy 'by design' they do it 'by functionality'.  So the reason you get your account information is because of that magic ID, but there is nothing stopping a piece of code saying 'return account information from account ID, account ID +1 and account ID +3. Its just practice that stops that rather than a fundamental information security approach.

There are mechanisms that can help with this but historically they've been more pain than its worth.  Its going to be interesting seeing how the next generation of joined up analytics and operational IT estates will retrofit user and role level analytical security into a world of application to application authentication.

Single Canonical Fail

There are few things out there in IT more delusional than the Single Canonical Form, the idea that IT can define a super schema, a schema so complete, so pure that all will bow down before it.  Sheer idolatry.  Whether it is for integration or for Data Warehousing the reality is that a Single schema is never going to be ‘canonical’, different people have different perspective and its this very contention between business areas that actually drives the business forwards.  Sales obeys the rules from Finance in certain areas and in others is in open rebellion as the KPIs for Sales compete against the need for regulation in Finance.  To forecast correctly means rigor and repeatability, but anyone who phones up Sales with an open checkbook is going to find their order fulfilled despite the claims of a sales process.

At the heart of a Single Canonical Form is a simple premise ‘as long as everyone can agree’ it’s the sort of premise that is wonderfully naïve in its inception. The reality is sadly that such a simplistic view ignores local perceptions and attempts to force a straight-jacket upon the business by providing a single, almost Stalinistic, view upon them.  By starting with that beguiling premise IT sets out on a journey that can only end on failure.  The Sales, Finance and Operations teams all have local KPIs, different division and regions have different strategies and all may have a different view on how they sell and whom they sell to.  This does not mean the business is dysfunctional or wrong, it simply means that the business is complex and not constrained within a single view of what should be done.

The Single Canonical Form aims to achieve the unobtainable and by doing so creates its own downfall.  Because it doesn’t meet the objectives of everybody then individuals are forced to create their own local solutions as the agility of the single canonical form is relatively, or indeed astronomically, low.  The goal of a single canonical form is to create a single view on one of the most variable things in a business: the view on information.  One part of the business may require only 10 pieces of information about a customer, another 200, neither are right or wrong it is simply their own local information, they critical element is that an individual customer be recognized across the two, not that 210 attributes be agreed.  The same goes for invoices, orders, contacts and everything else: agree when it matters, don’t bother when it doesn’t.

The Single Canonical Form is a straight-jacket on the enterprise, it’s a dumb idea based on an unachievable idea.  Its time for IT to grown up and work differently.

SaaS integration - making the ERP mistakes on a bigger scale

One of the most frustrating things in IT is the totally amazing ability of people not to learn from past experiences. The following are all the sorts of things I've recently heard at conferences, vendor presentations, business presentations and in company architecture practices.

"We don't need an MDM solution as Salesforce is going to be our only customer repository"
"Integration is simple, its all just REST or Web Services, we don't need to worry about that"
"We are moving to SaaS because it doesn't require integration and dealing with IT"
"The business are on their own if they do SaaS, we just deal with the internal IT"

And a whole litany of others over the last few years. The general theme is that business folks are commissioning SaaS solutions and in collusion with the technically naive are setting up entire new estates beyond the firewall. Meanwhile the internal IT groups are often washing their hands of this deliberately and fighting against the change.

Lets start with the first statement, as its the one I've heard several times.

I don't need MDM, my SaaS CRM is my master


This probably wins it in my book for the least ability to learn from IT and business history.  This is exactly what folks did in the first CRM rush in the 90s and have spent the last 15+ years trying to recover from.  Fragmentation of customer information is a fact, even more so in these days of social media, so starting a strategy with the idea that an externally provided solution in which you have little or no say and which is set up to be good as a SaaS solution not as an enterprise source for customer matching, merging and dissemination is like doing a CRM project in the 90s by lobbing money at consultants and saying "build whatever you like lads".... really not a good idea.

If you are going to look externally for SaaS, and there are good business reasons to do so, then the first question should be how to create the unified information landscape that modern businesses require.  That is an MDM problem, which only gets bigger as you include suppliers, products, materials and all of the other core entities that exist.

Integration is simple, its just REST/Web Services
While the CRM one is a joint IT/biz error then its this one where IT really excels itself in ignoring the past.  Integration is a hard problem in IT, its made much harder if you don't have MDM style solutions, when looking at SaaS the fact that you have published interfaces helps slightly but you still have the challenges of integrating between multiple different solutions, mapping the information, mapping the structure and of course updating all this when it changes.  That however is of course just the technical plumbing.  Then you have to look at your business processes that span across SaaS solutions and the enterprise as well as where to add new services into that environment.

The spaghetti mass of ERP and enterprise in the 90s which EAI aimed, and mainly failed, to solve will be nothing compared to this coming morass of externally competing companies who have a real commercial reason to keep you locked to their platforms and approaches and have the actual ability to make things tough as they can change their platforms as they want without asking for permission.

We are moving to SaaS so we don't have to deal with IT
This is a common refrain that I hear, but its very short sighted, what it really means is that the current IT department is broken and not meeting the demands of the business and not even properly explaining to the business what is going on.  This really is just storing up problems for the future, or just delegating the problem externally to another group who will end up being your IT department in future, but probably harder to shift and change that your current group.

If the business do SaaS that is their problem, we just do internal IT
I like to think of this as the IT redundancy programme, its a wilful attempt to ignore the real world and it really is only going to end badly.

Summary
The point here is that moving to SaaS is actually a bigger challenge for integration and information management than the old ERP challenge, but most companies are entering it with the same wild-eyed wonder that companies entered the ERP/CRM decade of the 90s.  Leaping in and just assuming that historical problems of integration will disappear.  The reality is that companies need to be more structured and controlled when it comes to SaaS and the IT departments must be more proactive in setting up the information and integration infrastructure to enable this switch.



Technorati Tags: SOA, Service Architecture