MDM isn't about data quality its about collaboration

I'm going to state a sacrilegious position for a moment: the quality of data isn't a primary goal in Master Data Management

Now before the perfectly correct 'Garbage In, Garbage Out' statement let me explain.  Data Quality is certainly something that MDM can help with but its not actually the primary aim of MDM.

MDM is about enabling collaboration, collaboration is about the cross-reference

Why do you do an MDM project?  The answer is to join information between multiple systems and multiple parts of the organisation.  Its so the customer in SFDC is the same as the customer in SAP and in Oracle Financials and when that customer hits the website you know who they are.  Its so the sales person can see all the invoices, orders and other elements related to their customer.  Its so you can see how a product goes through the various parts of the R&D and supply chain processes and track it all the way.

If everything was in one big system with a single database then you wouldn't really need MDM you'd just need data quality to make sure the single record was a good one.  You need MDM because you are attempting to join across systems and business units.  So the real value from MDM is that cross reference that tells you who the customer is and where all the information about them lives in the various systems... even if you never clean any of it.

So this is how you sell MDM to the business, not about data quality which is a secondary benefit, but as something that will enable the business to better collaborate and function more effectively.

Sometimes Quality doesn't count

The reality is that total quality isn't always what the business wants, they know some data is dodgy so the question is how dodgy and knowing that when you use it to make decisions.  Lots of social media is amazingly poor quality, but taken in volume trends can be seen.  What makes it more valuable though is when you can enable that cross-reference between the high-quality and the lower quality so you can see the trends of your customers and products not just trends in noise.

Focus on collaboration, focus on the cross reference, quality will follow
So having said that Data Quality isn't a primary focus it is actually how you enable that pesky cross reference, but you do so only on the information that matters, the core information required for the cross reference.  Thus you get a higher quality core identification of the customer and everyone understands why they are doing it, the quality enables the cross reference which enables the collaboration.

If the business don't care about quality why do you?
Now once you have that quality core, a minimum set of attributes required to uniquely identify the customer, then often you want to expand that quality to more attributes but stop and think.  Have the business asked me? That is quite an important point.  You might think its an absolute disaster that a given attribute isn't used in a standard way, but it could be that no-one in the business gives a stuff, so tell them about the issue but let them decide if they want to spend the money making it better.  If they don't document that they don't so if they come back you can say 'great so lets re-prioritise it' which is much better than 'oh so I spent money doing something that doesn't matter'.

The more you federate the more collaboration matters
The reason that MDM matters is that more and more business is about collaboration, both internal and external, this means that the business value of MDM has really shifted from being about the data quality in reports to being an integral part of how a business works.  Data Quality isn't irrelevant in this world but its turned from being the goal of MDM to being a tool that helps enable the primary goal which is collaboration.  As the need to digitally collaborate with partners and customers increases so the business value of that MDM cross reference increases both in operations and as the bit that helps you link up all of those big data sources to create a global view.

MDM is the Rosetta Stone that enables people to collaborate, so focus on collaboration not quality. 

How British Airways failed to use the information they have

Having worked with companies a lot in the past few years on how to create a better customer experience and in-particular through MDM to help effectively identify the customer I know just what is possible even in very challenging environments.  The Airline industry is not one of those areas but today with British Airways I received another example of how a company might have all the information required to deliver a good customer service but chooses not to provide that information to where it counts.

I was flying into Heathrow, on a transatlantic flight I booked recently, I'd originally planned to head into London and from then on to Paris via train but checking the prices it turned out flying was cheaper and as I was already at Heathrow it seemed the smart and good corporate citizen thing to do, so I made another booking.

So lets see what BA knew about me at this stage....

• All of my passport information - the goldest of gold standards
• My frequent flier, with shiny gold card indicating I need to get out less
• All my flights with them in the last 10+ years
• How many times I've missed a flight with them due to my own fault (zero)
• How many times weather or their issues have caused me to miss a flight (was 3, now 4)

So in the text book of Consumer MDM they have the perfect set of source data to identify the individual (the passport) and a unique number the customer wants to give out (frequent flier ID).  It really couldn't be easier in a world where the ticket (which has the ID) and the gate (which scans the passport) to identify the individual.

They also had two bookings.

One that flew out on a Sunday and was due to land at 2pm on Monday, and a return leg for Wednesday (seriously I need to get out less. 

One that flew from Heathrow to Paris at 15:15 on Monday and returned on Wednesday

 Shared across these bookings are my details, indeed when my iPhone downloaded the two boarding passes it automatically paired them together as a single journey.

So what did BA do?  Well the flight into Heathrow was delayed, but I still had 45 minutes to make the connection.  There were a couple of others on the flight racing for the same flight... I reached the connections check-point first to be told:

You've been taken off the flight

The couple behind me (same inbound, same connection) were let through... this irritated me somewhat but it was only when I got to the rebooking desk that irritation turned into disbelief and outright annoyance.

BA: You missed the conformance check. 

Me: I was on the flight from Phoenix, you knew that

BA: We can't see that its on a separate booking

Me: With your airline, made with your website

BA: Yes but its a separate booking

Lets be clear, I like BA, I like the service from the individuals but this is a great example of how a company fails to leverage the information it has to deliver a decent customer service.  Its not giving its people the right information, information it has, to make the right decisions.

My iPhone automagically recognised that a flight taking off from an airport less than 2 hours after I landed from a previous one was part of a single journey, independent of how many booking IDs there were.  BAs systems have this information, I can see it in my Exec club profile, but at the front line they see only the booking.

Now the smart thing would be for BA to have a very simple check on bookings that says 'if bookings land and take off from the same airport within a set period, say 4 hours, we should consider them as one booking so our airport staff can see what is going on'.  At the very least the people at the airports should have access to all my flight bookings so before they kick a customer, and one they've marked out as a priority customer, off a flight the person can check 'wonder if there is an inbound they are on that we know about?'.

This is truly a great example of a company undermining its excellent customer service by not providing its staff with the information to deliver it.  This isn't about a lack of information, its not even about a lack of good quality information, its about the inability to integrate that information into the business processes where it is needed.

Disclaimer: I actually did a bunch of work for BA several years ago around BA.com and other customer facing parts of the business.  

How Business SOA thinking impacts data

Over the years I've written quite a bit about how SOA, when viewed as a tool for Business Architecture, can change some of the cherished beliefs in IT.  One of these was about how the Single Canonical Form was not for SOA and others have talked about how MDM and SOA collaborate to deliver a more flexible approach.  Central to all of these things has been that this has been about information and transactions 'in the now' the active flowing of information within and between businesses and how you make that more agile while ensuring such collaboration can be trusted.

Recently however my challenge has been around data, the post-transactional world, where things go after they've happened so people can do analytics on them.  This world has been the champion of the single canonical form, massive single schemas that aim to encompass everything, with the advantage over the operational world that things have happened so the constraints, while evident, are less of a daily impact.

The challenge of data however is that the gap between the post-transactional and the operational world has disappeared.  We've spent 30 years in IT creating a hard-wall between these ares, creating Data Warehouses which operate much like batch driven mainframes and where the idea of operational systems directly accessing them has been aggressively discouraged.  The problem is that the business doesn't see this separation.  They want to see analytics and its insight delivered back into the operational systems to help make better decisions.

So this got me thinking, why is it that in the SOA world and operational world its perfectly ok for local domains and businesses to have their own view on a piece of data, an invoice, a customer, a product, etc but when it comes to reporting they all need to agree?  Having spent a lot of time on MDM projects recently the answer was pretty simple:

They don't

With MDM the really valuable bit is the cross-reference, its the bit that enables collaboration.  The amount of standardisation required is actually pretty low.  If Sales has 100 attributes for Customer and Finance only 30 and in-fact it only takes 20 to uniquely identify the customer then its that 20 that really matter to drive the cross reference.  If there isn't any value in agreeing on the other attributes then why bother investing in it?  Getting agreement is hard enough without trying to do it in areas where the business fundamentally doesn't care.

This approach to MDM helps to create shorter more targeted programs, and programs that are really suited to enabling business collaboration.  You don't need to pass the full customer record, you just pass the ID.

So what does this combination of MDM and SOA mean for data, particularly as we want analytics to be integrated back into operations?

Data solutions should look more like Business SOA solutions and match the way the business works

In simple terms it means the sort of thinking that led to flexibly integrated SOA solutions should now be applied to Data.  Get rid of that single Schema, concentrate on having data served up in a way that matches the requirements of the business domains and concentrate governance on where its required to give global consistency and drive business collaboration.  That way you can ensure that the insights being created will be able to be managed in the same way as the operational systems.

With SOA the problem of people building applications 'in the bus' led me to propose a new architectural approach where you don't have one ESB that does everything but accept that different parts of the business will want their own local control.  The Business Service Bus concept was built around that and with the folks at IBM, SAP, Microsoft and Oracle all ensuring that pretty much everyone ends up with a couple of ESB type solutions its the sort of architecture I've seen work on multiple occasions.  That sort of approach is exactly what I now think applies to data.

The difference?

Well with data and analytical approaches you probably want to combine data from multiple sources, not just your own local information, fortunately new (Java) technologies such as Hadoop are changing the economics of storing data so instead of having to agree on schemas you can just land all of your corporate data into one environment and let those business users build business aligned analytics which sit within their domain, even if they are using information shared by others.  MDM allows that cross reference to happen in a managed way but a new business aligned approach removes the need for total agreement before anything can get done.

With Business SOA driven operations we had the ability to get all the operational data in real-time and aggregate at the BSB level if required, with Business SOA driven data approaches we can land all the information and then enable the flexibility.  By aligning both the operational and post-transactional models within a single consistent Business aligned approach we start doing what IT should be doing all along

Creating an IT estate that looks like the business, evolves like the business and that is costed in-line with the value it delivers.

Applying Business SOA thinking to data has been really interesting and what led to the Business Data Lake concept, its early days clearly but I really do believe that getting the operational and data worlds aligned to the business in a consistent way is going to be the way forwards.

This isn't a new and radical approach, its just applying what worked in operations to the data space and recognising that if the goal of analytics is to deliver insight back into operations then that insight needs to be aligned to the business operations from the start so it can adapt and change as the operational business requires.

The boundaries from the operational and post-transactional world have gone, the new boundaries are defined by the business and the governance in both areas is about enabling consistent collaboration.

Quality is a side-effect not the goal in MDM

I put out a tweet the other day with this title and I think its worth elaborating on what I mean.  Lots of MDM efforts I see have the goal of 'improve data quality' and this is a mistake.  I'm not saying that data quality isn't a good thing but that in itself its not actually a goal.

What do I mean?  Well lets take an analogy or three, if you are looking to buy a diamond then do you buy the very, very best and the very very biggest?  If you are looking for diamonds to use in cutting or industrial grinding then the answer is of course 'no', the quality really wouldn't be appropriate in those uses it would be a waste of money.  What if you are looking to put a 1.6 litre engine in a car aimed at the local commuter market, do you look around for the most powerful, most expensive, built to the highest quality standards?  Well that would probably be one of the engines slated to go into a Formula 1 car next season.  Sure it generates huge power and is a quality engine but its not fit for purpose.

Now for the final analogy.  You are looking to provide translation services for the Iranian nuclear discussions.  Should you go and get the cheapest price from someone who promises they can speak 'Iranian' or do you invest from someone who actually is proven as a translator for Persian, Gilaki and Mazandarani and describes their Kurdish as 'passable'?

The point here is that in each case the goal defines the level of quality required, quality in itself is about having an acceptable level of quality to meet your goal which in some occasions might be very little indeed.

So what is the real goal of MDM?  Its about enabling business collaboration and communication the power of MDM is really in the cross-reference, the bit that means you know the customer in one division is the same as another and that the product they are buying is the same in two different countries.  If the quality is awful but the cross-reference works then in many occasions you don't need to invest more in quality unless there is a business reason to do so.  Most of the time that business reason is that you cannot achieve the collaboration without having a decent level of quality.  To match customers across business areas requires you to have an standard definition, so your customer on-boarding needs a certain level of rigour, your product definition needs to work to standards that are agreed across the business.

So in focusing on the collaboration, in focusing on where the business wants to collaborate you focus MDM and you focus where quality needs to be achieves.  Focusing on quality as a goal is a very IT centric thing, focusing on collaboration and through that enabling quality is a business thing.

And MDM is certainly a business thing. 

Minimum on the wire, everything in the record

I've talked before about why large canonical models are a bad idea and how MDM makes SOA, BPM and a whole lot of things easier.  This philosophy of 'minimum on the wire' helps to create more robust infrastructures that don't suffer from a fragile base class problem and better match the local variations that organisations always see.

One of the things I really like about IT however is how it starts putting new challenges and how simple approaches really make it easier to address those new challenges.  One of those is the whole Big Data challenge which I've been looking at quite a bit in the last 12 months and there is a new philosophy coming out of that which is 'store everything', not 'store everything in a big data warehouse' but 'store everything as raw data in Hadoop'.    There are really three sources of 'everything'

1. Internal Transactional systems
2. External Information & transactional systems
3. Message passing systems

So we now have a really interesting situation where you can minimise what is on the wire but store much more information.  By dropping the full MDM history and cross reference into Hadoop you can use that to say exactly what the information state was at a point in time when a message was passed across the bus.  In other words you can have a full-audit trace of both the request and the impact that the request had on the system.

One of the big advantages of Hadoop is that it doesn't require you to have that big canonical model and again this is where MDM really kicks in with an advantage.  If you are looking for all transactions from a given customer you just take the system x-ref from MDM as your input and then you can do a federated Map Reduce routine to get the system specific information without having to go through a massive data mapping exercise.

Hadoop means there is even less reason to have lots flying about on the wire and even more justification for MDM being at the heart of a decent information approach.

Bling and ignorance - how cloudy thinking will screw up IT

Today it was annouced that Progress Software are going to 'divest' their BPM and SOA technologies and instead are going to focus on cloud technologies that don't really exist yet. This is indicative of a mentality I see around so first lets start with some credentials: 1) I worked with Google around a SaaS partnership in 2007 around Google Apps 2) I delivered an SFDC solution in 2008 3) I've worked with Amazon, VMWare and lots of other cloud based companies So lets be clear I think cloud and SaaS can be extremely useful things, the problem comes when people begin to think they are magic. Lets be clear SaaS is a software package pre-installed and configured that you can pay for on demand - the change isn't the software its the capacity and the charging model Cloud is an infrastructure play to provide old school tin in a virtualised manner in a way that can be quickly provisioned and paid for on demand. That really is it. The problem I see is that people take the new bling and forget the old lessons. So people say 'I'm am using SFDC and it is my customer MDM I don't need anything else'... in the same way that people said that about Siebel in 2000 and found out in 2002 that it wasn't true. They say 'its on the cloud so I don't need to worry about architecting for performance, I'll just scale the hardware' as people did around 'pizza box' architectures in 2000. I hear 'we don't need to worry about integration, its on the cloud'... like people did around SOA/ESB/WS-* in ... 2000. The problem is that the reality is the opposite. The more federation you have the more process and control you need. The more information is federated the more requirements you have for Master Data Management. Cloud solutions are not silver bullets and they are not always simple solutions to integrate with other systems, great on their own but not good with others. Rapidly people are building federated IT estates with federated data and no practice and process around integration which leads to a massive EXTERNAL spaghetti mess, something that makes the EAI problems of the 90s look like a walk in the park. Or to put in another way... isn't it great how people are making more work for those of us in IT.

How Skype treat me as a transaction not an individual

I blogged at work on how the challenge of Omnichannel engagement means you need to treat the customer as an individual not as a transaction.

Well here is an example of a company doing the exact opposite.

First some background.  At work I tend to use a VPN that takes me out through the Netherlands but regularly I'm in different countries and I tend to use Skype.  Now Skype know a lot of information about me:

1. My credit card number
2. My home address (which includes the country)
3. email address
4. All of the locations from which I've connected before

So what happens when I access, as a logged in user, from http://www.skype.co.uk (notice I'm forcing it to use the UK site at this stage, surely another hint....)

Brilliant eh?  Its all in Dutch.  So Skype take all the information that they know about me and then totally ignore it in favour of an IP lookup to get a country and therefore a language.  This page is truly special as the top of the page has not a single thing to enable me to shift the language into English.  Certainly makes it more 'fun' in adding credit to my account... did it succeed?  Is that an error message?  How on earth would I know its in Dutch.

Now on my mobile there is a screen for profile that includes language... a useful thing... but something that isn't on the profile on the website, which also declares that anything you put on the profile is going to be shared with the world (not what I want).

So Skype is falling down on several cases.  Firstly its not using the internal information that it has to offer me a personalised service, I'm in the UK, I live in the UK I don't want to change my language based on the country I currently sit in.

Secondly its profile configuration is inconsistent between its mobile platform and its web platform.  I have miles more things on the mobile platform but I'm unlikely to set them as they will be shared with the world.  But I decided to set the country and the language as much as I love the Netherlands my Dutch is non-existant.

So I set up the profile (left) and look what I got on the web (right)

So see the issue?  The country hasn't been picked up and the language isn't even an option, the website certainly didn't use the preferences from my mobile device to give me the language I wanted on the web.

Skype is multi-channel not Omnichannel.  Clearly Skype don't have a real-time customer information mastering process that keeps their channels in sync and equally clearly their website doesn't use the customer profile to customise the web experience to the individual instead it uses the IP address to customise it to the network connection.

Omnichannel is about treating the individual like an individual independent of channel.  And its that which requires good real-time operation centric MDM.

42 - Or do you understand your Big Data question?

What is the ultimate Big Data question?  Well it is of course the Biggest Question... the question... of Life, the Universe and Everything.... but that poses a problem: in an analytical world do you really understand the question?

(or in short form if you are in a rush)

The point here is that one of the major challenges with Big Data is that we are moving away from simple SQL driven questions 'who are my top ten customers' or 'how much did I sell last week' into much more analytical and predictive questions such as 'if I reduce the price of goats cheese how much more red wine will I sell'

This presents a new set of challenges because analytics can give you simple answers '15% more' which then lead you to drop the price of goats cheese.  The network effect of that change however means less beer is sold and less hard cheese is sold so now you are over stocked in beer and hard cheese, both of which have a use-by date.  The point is that the question was badly formed but correctly answered.  Greater degrees of abstraction also introduce greater degrees of assumption in those creating the models.  So while the business has asked a small and concise question 'where to put the next store' the model has made certain assumptions that may or may not be the case. How are these assumptions shown to the business and if they are can the business even understand them?

Today there exists a problem of chained spreadsheets, in the future the issue of chained analytical models is going to make the connection between the business 'question' and the 'answer' more complex and harder to understand and put more power into the hands of mathematicians who prove good at converting abstract questions into good models.  This also means that there will be ever more importance placed on getting control of the definition of information into that model (what is a customer, what is a product, how do you identify them... MDM stuff) these are the bits that the business can control.  The core information, the sources and the quality control around them.

Big Data answers - only as good as your understanding of the question.

SaaS integration - making the ERP mistakes on a bigger scale

One of the most frustrating things in IT is the totally amazing ability of people not to learn from past experiences. The following are all the sorts of things I've recently heard at conferences, vendor presentations, business presentations and in company architecture practices.

"We don't need an MDM solution as Salesforce is going to be our only customer repository"
"Integration is simple, its all just REST or Web Services, we don't need to worry about that"
"We are moving to SaaS because it doesn't require integration and dealing with IT"
"The business are on their own if they do SaaS, we just deal with the internal IT"

And a whole litany of others over the last few years. The general theme is that business folks are commissioning SaaS solutions and in collusion with the technically naive are setting up entire new estates beyond the firewall. Meanwhile the internal IT groups are often washing their hands of this deliberately and fighting against the change.

Lets start with the first statement, as its the one I've heard several times.

I don't need MDM, my SaaS CRM is my master


This probably wins it in my book for the least ability to learn from IT and business history.  This is exactly what folks did in the first CRM rush in the 90s and have spent the last 15+ years trying to recover from.  Fragmentation of customer information is a fact, even more so in these days of social media, so starting a strategy with the idea that an externally provided solution in which you have little or no say and which is set up to be good as a SaaS solution not as an enterprise source for customer matching, merging and dissemination is like doing a CRM project in the 90s by lobbing money at consultants and saying "build whatever you like lads".... really not a good idea.

If you are going to look externally for SaaS, and there are good business reasons to do so, then the first question should be how to create the unified information landscape that modern businesses require.  That is an MDM problem, which only gets bigger as you include suppliers, products, materials and all of the other core entities that exist.

Integration is simple, its just REST/Web Services
While the CRM one is a joint IT/biz error then its this one where IT really excels itself in ignoring the past.  Integration is a hard problem in IT, its made much harder if you don't have MDM style solutions, when looking at SaaS the fact that you have published interfaces helps slightly but you still have the challenges of integrating between multiple different solutions, mapping the information, mapping the structure and of course updating all this when it changes.  That however is of course just the technical plumbing.  Then you have to look at your business processes that span across SaaS solutions and the enterprise as well as where to add new services into that environment.

The spaghetti mass of ERP and enterprise in the 90s which EAI aimed, and mainly failed, to solve will be nothing compared to this coming morass of externally competing companies who have a real commercial reason to keep you locked to their platforms and approaches and have the actual ability to make things tough as they can change their platforms as they want without asking for permission.

We are moving to SaaS so we don't have to deal with IT
This is a common refrain that I hear, but its very short sighted, what it really means is that the current IT department is broken and not meeting the demands of the business and not even properly explaining to the business what is going on.  This really is just storing up problems for the future, or just delegating the problem externally to another group who will end up being your IT department in future, but probably harder to shift and change that your current group.

If the business do SaaS that is their problem, we just do internal IT
I like to think of this as the IT redundancy programme, its a wilful attempt to ignore the real world and it really is only going to end badly.

Summary
The point here is that moving to SaaS is actually a bigger challenge for integration and information management than the old ERP challenge, but most companies are entering it with the same wild-eyed wonder that companies entered the ERP/CRM decade of the 90s.  Leaping in and just assuming that historical problems of integration will disappear.  The reality is that companies need to be more structured and controlled when it comes to SaaS and the IT departments must be more proactive in setting up the information and integration infrastructure to enable this switch.



Technorati Tags: SOA, Service Architecture

MDM, SOA and BPM - making processes simple

BPM is once again a hot topic. It seems several years ago where I said "if you want to get enterprise cash, learn BPEL"... in fact it was 5 years ago! However BPM really hasn't delivered on the promises and that is for three reasons

1) The Promises were too big
2) The implementations were too complex

3) BPEL rapidly became a Visual COBOL approach

Now I've said before that BPM screws up SOA and that SOA makes BPM simple, I'll stick with that and I'll now raise another piece in conjunction with SOA and MDM. This is actually a way to make BPM work more effectively, more simply and reduce the issues around elements such as process [de|]hydration. Namely that if you are in an area where BPM is the best implementation approach for a specific service's capability then using MDM helps to make its implementation simpler as you can work on key exchange rather than on content exchange.

Rather than BPM folks worrying about the schemas and exchange of core information that is left to MDM (where it belongs) and the SOA pieces handles the business domains part to make sure processes have clear boundaries and can be well managed.

Technorati Tags: SOA, Service Architecture

Data Services are bogus, Information services are real

One of the questions that used to be asked, or proposed as fact, in old school SOA was the idea of "Data Services" these were effectively CRUD wrappers on Database tables and the idea was that they were reusable across the enterprise. I've said many times this is a dumb idea as the importance is actually about the information, which means context and governance.

Now the other day when I was talking about MDM a bright spark pointed out that I hated data services but wasn't MDM just about data services?

Its a good challenge, mainly because that is the problem about how many people view MDM. MDM is, when done well is about the M and the M not the D, i.e. its more about Mastery and Management than it is simply about Data. What does that mean?

Well lets take everybody's favourite MDM example "Customer" a Data driven approach would give us a Service of

Service Customer

• Capability: Create
• Capability: Update
• Capability: Delete
• Capability: Read

Now this is the "D" approach to MDM and SOA, also known as the Dunce's approach, its about Data Services and viewing the world as a set of data objects.

The smart approach is to view MDM as an information challenge and delivering information services so instead of the data centric approach we get

Service Customer

• Capability: Establish Prospect
• Capability: Establish Customer
• Capability: Modify
• Capability: Change Status
• Capability: Find
• Capability: Archive Customer
• Capability: Validate Customer
• Capability: Merge Customers
• Capability: Split Customer

Here we start exposing the customer service and make clear two things

1. We are talking about the customer in context
2. We reserve the right to say "no" to a request

So this is where customer genuinely can be used from a single service across the enterprise. This service takes on the responsibility to authorise the customer, or at least ensure that authorisation is done, it sets the quality standards and governance standards and doesn't allow people to do what ever they want with it. It includes the MDM processes elements around customer management and provides a standardised way of managing a customer through its lifecycle.

This is fundamentally the difference between a business SOA driven approach which concentrates on the services in their business context and with their business governance and a technical driven approach which looks to expose technical elements as via a technical wrapper.

Technorati Tags: SOA, Service Architecture

Why MDM and SOA are shifting out of IT

Now I've said previously why MDM is required for successful SOA but there is another important piece around MDM and SOA that is happening at the moment and explains both why MDM and SOA haven't historically gone together and, crucially, why the new trends are liable to help bring them together next year.

Why SOA and MDM didn't go together

The first question is why historically MDM and SOA projects tended to not go together. Sure organisations would "do" SOA programmes and would "do" MDM programmes but rarely would the SOA programmes and MDM programmes be tightly joined. There are I think three reasons for this

1 - Different bits of IT

The first challenge is that MDM and SOA were normally done by different bits of IT with different mentalities. MDM was done by the "data" folks who worried about data warehouses and saw data as the most important thing in the world. SOA was done by the enterprise integration and architecture folks who worried about operational processes and integration. The MDM folks tended to have a post-transactional view of the world where things were "true" or "false" while the SOA folks tended to view things as "in processes" or "completed".

These different communities in IT have very different backgrounds and focuses. ETL, Data Warehouses and big databases rule in the MDM/Data side while fast transaction throughput is the key for the SOA folks.

2 - Different bits of vendors

The next challenge is that the vendors have exactly the same split in their view of the world. Look at IBM for instance. SOA is in the WebSphere brand while MDM is in the InfoSphere brand, two very different parts of IBM Software with independent management structures and teams, sure the idea is that they should co-operate and work together but we all know that different divisions in companies like to do things differently. This also means that you often get different sales people for the different pieces of software. Oracle for instance put MDM in the Applications area while SOA is in the Middleware space, if you want to use their MDM products with their middleware products (for instance using the MDM PIP) this means you have to deal with two different sales people to get what you want.

3 - They've been IT projects

The final reason is that MDM and SOA have traditionally been internal IT programmes owned and managed by IT and largely invisible to the business. This means that the two cultural elements above are then hard-baked into the solution which ensures that the SOA and MDM programmes are kept distinct.

Why its changing

So why is it changing? Well the first reason is that people have started finally realising that SOA is a business thing and needs to be viewed from a business architecture perspective. This has been a long time coming but finally it appears people are thinking about the challenges of business services, service value and the business architecture. At the same time MDM is shifting as well, firstly its shifting away from post transactional into an operational space which means it needs to consider operational processes and performance in a way it hasn't before, this means that the MDM/Data folks now have to not only talk to the SOA folks but they have to... shock horror... actually become MDM/SOA folks.

MDM is also shifting in that business people want to do more active information management, both in terms of newer analytical tools and secondly in terms of including that mastered and high-quality information into core operational processes. This means that the business realises that the old "data landfill" approaches which were poor before are massively hindering the analytical models and have a direct impact on the efficiency of the operational processes. By taking control recognising that Information needs Mastering the business is now actively taking over those processes and thus MDM is moving out of the background into being a key part of the corporate information strategy.

So that is why MDM and SOA are shifting out of IT, its because being within IT made them technically centric programmes that often failed to deliver the promised value. By enabling the business to more actively own its IT estate, through Business SOA, and its core information, through MDM, it suddenly ceases to be a question of competing IT cultures but a simple question of how to present a consistent operational and analytical view of the business.

SOA and MDM are made to be together.... that fixes the enterprise culture... but will it fix the vendors?

Technorati Tags: SOA, Service Architecture

MDM isn't a hub - its just simpler to implement it that way

One of the things about MDM that people often get wrong is the idea that MDM provides a central information hub around a given entity and its relationships.

It doesn't.

MDM provides 3 core facilities

1. Cross-referencing of core entities between systems
2. Standardisation around the critical "matching" attributes
3. Synchronisation of attributes modified within multiple systems

Only one of these really requires some form of centralisation, the x-ref, the rest can be handled via governance processes and integration processes without requiring a central system. You can implement match and merge within the integration layer or end application then propagate those changes along.

However in the world of Simple IT and "doing one thing well" its liable to be much more effective to have an MDM solution that manages this integration and which is designed to do that integration rather than building it all yourself.

MDM doesn't require a central solution, you'll just probably find it simpler that way

However there are good reasons

Technorati Tags: SOA, Service Architecture

MDM made easy - why people make Master Data Management hard

Okay so I've spent a while in the last 10+ years on various MDM (Master Data Management) programmes and its come down to a very simple reason why lots of MDM programmes fail..

People screw up MDM programmes by forgetting what MDM actually is.

The first bit is that people look at the various different MDM packages and then really miss the point. Whether its Oracle UCM, Informatica, SAP MDM, IBM MDM, Initiate, TIBCO or anything else people look at it and go...

Right so this is what we do, what else can we fit into it

Now this is a stupid way to behave but its what most people do. The use the MDM piece as the starting point. The reality is that MDM is only about two things

1. The cross references of the core entity between all the systems in the IT estate
2. The data quality and governance around the core entity to uplift its business value

And that really is it. The more attributes that are added beyond the those required for these two elements then the more expensive and more complex and less effective your MDM solution will be.

The other core part of this is that there really are only a very limited number of bits to MDM, its about three things

1. Things - assets, accounts, parts, products, etc
2. Parties - individuals, organisations, customers, suppliers, etc
3. Locations - postal addresses, email address, web addresses, physical address, geo locations, etc

That really is it from an entity perspective then you've got the relationships for those entities (and remember a relationship is just the keys to do the match) which means

1. Parties to Things
2. Parties to Locations
3. Things to Locations
4. (Parties to Things) to Locations (e.g. a persons specific account address)

There really isn't much else to do it really is just

So all you need for MDM to succeed is the above entity model and a list of attributes required to uniquely identify a high quality version of that entity.

So why do people screw up?

Well first off is because they do something like picking a customer mastering package and then trying to master product information or product relationships within it (e.g. Asset to Location).

Secondly because they start extending the number of attributes rather than creating a cheap ODS.

Thirdly because they create their own versions of the relationships because of a belief that business rules that they have change the entity model rather than actually just being data quality rules against the standard entity model.

So the reality is that MDM is simple, its about the data quality and matching, which means its about the business governance of information.

And that is the real reason that most MDM programmes fail to deliver, because in order to get to the business governance of information and make MDM simple you have to do an awful lot of hard work in building up the trust between the business areas and IT that governance can be done in a way that benefits rather than impacts business.

MDM isn't a technology solution its a simple business solution, the problem is that IT people tend to make it a complex IT solution because they can't make it a simple business solution.

Technorati Tags: SOA, Service Architecture

MDM is required for successful SOA

One of the things that I've noticed over the years about successful SOA programmes is that they all undertake a formal approach to MDM. By programme here I'm not talking about one website using REST or WS-*, I'm talking about a strategic transformation initiative that aims to move the IT estate forwards to a more business centric approach.

Why is MDM important for SOA? Well there are a couple of reasons.

1. MDM stops you having to do Single Canonical Form
2. MDM helps you start from a point of federation

Now there are a number of MDM approaches that enterprises take and I'll massively generalise them into three groups

1. Digital Landfill MDM
2. Operational MDM
3. Federated MDM

The first is the most common and is generally, in fact pretty much exclusively, a post-transactional approach where lots and lots of systems feed down into the "MDM" solution which is really there to provide the cross referencing required for a data warehouse reporting solution. These tend to be database centric and batch oriented and not overly useful for a real-time or operational environment

The second is where the MDM solution has actually taken on a transactional approach where information, for instance customer details, are actually held within the operational MDM solution and the cross referencing information is stored and available in real-time. This really is the minimum level at which an SOA environment should be operating. The MDM solution is there to ensure the loose coupling between services and making sure that a minimal reference model approach to information sharing can be handled.

The third is where there is no central MDM solution but it is still operating transactionally. This is the very very hard and not really to be attempted yet type approach. Here you can think about semantic web and those sorts of funky technologies and on doing the matching and cross referencing on an on demand basis. I'm pretty sure this type of solution won't work very well, or indeed potentially at all, but its really what lots of people are trying to do today when they have an ESB, lots of services and nothing set up to do the mappings.

So basically the point is simple. A transactional, operational, real-time MDM solution is an absolute requirement for a professional SOA environment. Without it you are going to have nasty tight integrations between areas at the data layer or some amazingly complex mappings that will fail on a regular basis. I'm not even talking here about the benefits of data quality improvements by using a decent MDM solution I'm just talking about something that manages the cross-referencing of similar information between systems and services.

The odds are however that in doing your SOA solution you've left MDM being something that just feeds the data warehouse and looks at reporting.

Technorati Tags: SOA, Service Architecture