Tuesday, March 25, 2014

Microservices is SOD all within SOA

Microservices is a Service Oriented Delivery approach, all within a Service Oriented Architecture context.
(Long Title ;)

Ok so a few more updates since the last time I wrote about Microservices and I think its worth just updating as it really is heavily underlining why Microservices is a Service Oriented Delivery approach that absolutely can fit within a Service Oriented Architecture.  Lets be clear it isn't a bad thing to be that as delivery is where the rubber hits the road.  But every section that is written further underlines why the 'its not SOA' argument falls very shallow.

Infrastructure Automation
Not going to disagree here as this is just standard practice.  I actually think this isn't far enough forward looking in terms of infrastructure. Technologies such as CloudFoundry, supported by IBM, SAP, HP, Rackspace, EMC, VMWare, Pivotal, etc, move beyond simply automating the infrastructure side towards actually automating deployment at the application level.

This is a really solid SOD tip, certainly something I'd recommend.  Using technologies such as CloudFoundry can really help here and move beyond infrastructure automation.

Design for Failure 
Again a good tip in a service oriented architecture, back in 2007 I actually wrote about the challenges of five 9's and one was about planning for failure in SOA.  Again this is a really solid piece of advice but I feel it over simplifies the challenge.  Sometimes a service has failed because its impossible for it to succeed, particularly the case if using external services.  So 'restarting' is only an option sometimes and its important to understand the types of scenarios that you'll need to handle.

Evolutionary Design
This one comes from the school of the obvious.  The example used talks of a website, the Guardian, that was built as a monolith and was rearchitected to microservices.  I too have such an example of a large airline website that was built as a monolith and was getting hard to maintain and was broken down into various services and boundaries and separately compilable elements, this was in 2004-2006 however so again this really isn't new.

Yes of course you should look at something that will evolve... again a good tip but not something that marks Microservices down as anything other than a Service Oriented Delivery approach.

The goal of a good service oriented architecture is to "look like the business, evolve like the business and be costed in line with the business value" and Microservices lays down some nice rules for implementing certain parts of an enterprise but those are best served by an honesty that its an implementation choice within a broader Service Oriented Architecture.  That doesn't devalue it in any way, just places it in the right context.

Microservices is SOD, all within the context of SOA - and yes this time I added the comma.

I'd also like to put out a hat-tip to Loraine Lawson who pointed me towards the excellent Fallacy poster with the note that the SOA side-bar is really a 'Fallacy Fallacy'. 

Tuesday, March 18, 2014

Microservices is SOA, for those who know what SOA is.

Ok so its started a bit of debate on Twitter and now there have been emails, but in the spirit of openness I thought I'd better blog.  Now its good that Martin has now added a side bar on SOA to his article on Microservices but that really makes it worse in many ways.  I'll get to that at the end but first off lets explain why Microservices is just another SOA implementation pattern.  Its SOD for SOA

Lets break down the key precepts of Microservices and compare them against the OASIS SOA Reference Model (published 2006)

Componentization via Services

There follows a long text that can be reduced to

"Service Oriented Architecture (SOA) is a paradigm for organizing and utilizing distributed capabilities that may be under the control of different ownership domains. " 
There is an awful lot of words about deployment and process models in the Microservices piece but this single sentence at the start of the OASIS SOA RM is much more powerful because
  1. It means that IT and non-IT services can be represented in a single approach
  2. It immediately includes the major challenge - that of politics and ownership
OASIS then goes nicely into WTF actually is a service,
  • The capability to perform work for another 
  • The specification of the work offered for another 
  • The offer to perform work for another
Again this is not a tech requirement which means your architecture can actually start from a business perspective rather than process threads.  In Fowler's microservices textual description (no bullet points here) we have

  • services are out-of-process components who communicate
  • explicit component interface
  • ..... nothing on this bit 

OASIS then adds the kicker
in SOA, services are the mechanism by which needs and capabilities are brought together.
Remember that phrase 'mechanism' its going to be important....

Here is where Martin begins to get it wrong.  The OASIS SOA RM defines a capability as 
A real-world effect that a service provider is able to provide to a service consumer
The point here is the difference between a capability (the bit that does the work) and the service (the organising construct) is really important.  What we found when doing SOA in the wild for over a decade, and all the people on the OASIS SOA RM had lots of experience doing this, was that the organising framework was separate from the actions themselves.  The reason this is crucially important is that people started often making services where service = capability so you ended up with lots and lots of services (ummm if I was being insulting I'd call them microservices) here are some posts from 2006 and 2007 that explain why its really not great to confuse the two concepts and it made it into the SOA Antipatterns as well.  Now the actual text is pretty much ok, but again its lack of reference to the past and making a crucial mistake does not help people learn how things are evolving and what can be improved.

New?  Hell even I wrote a book which talked about how to model, manage and set up teams around this approach. The SOA Manifesto (2009) talks about key principles behind SOA (I still prefer the RM though) from a big group of practitioners.  The point here is that there are two problems, first the confusion of service and capabilities and secondly the lack of recognition of hierarchies importance in governance.

Products not Projects

Here it goes  beyond the OASIS SOA RM which doesn't talk about delivery models... however fortunately it goes beyond absolutely nothing that was said before.  A quick hit on Google just shows how many pieces there are in this space, some better than others and some products better than others but really this is not new.  I used to use the phrase 'Programs not projects' and always talked about assigning architects for the full lifecycle 'to make them accountable'.    Again its not that this statement is in anyway wrong, its just that its in no-way new.  We've known that this has helped for years but it has a significant issue: cost.

Lets say you have a service that requires some amazingly smart analytics, some hard core coding and some wonderful UI design.  You hire the very best, they cost a fortune and your service is awesome.  Do you really want those folks sitting around waiting for requirement changes?  No.  This is why for me the concentration is at the architecture and ownership level that consistency must be maintained, not at the developer level.  For companies where software development is their business you can include development but for most organisations in the real-world economics prevent the 'you built it you run it' mentality.  Again its a lack of learning that undermines the message.

Here I'm not disagreeing, it would be super ideal to have the same team always maintaining the code they write, its just not practical but thinking in terms of discreet pieces with their own heartbeats is a good thing.  Most decent SOA programs I've seen have recognised this and had different delivery schedules for different services.  What becomes important is the integrity of the service and its ability to be independently maintained, within the context of its management hierarchy, relegating this to 'keep the same developers' misses some of the power of SOA.  Again this emphasises that Microservices is just another SOD approach, a potentially good one in some circumstances but not something that will work for everyone and every circumstance...

Smart endpoints and dumb pipes

I really agree with this, but I think its better put in a different way.  In the OASIS SOA RM there is the concept of an 'execution context'.  Which is the bit that lets a service be called and its capability invoked.  Clearly the end-point is 'smart' as its what does the work, hence the phrase 'mechanism' used above.  The 'pipes' may or may not be dumb (the 'pipes' talking to those Rovers on Mars are pretty smart I think) but what they are is without value.  This was a crucial finding in SOA and is well documented in the SOA RM.    The execution context is where all of the internal plumbing is done but its the Service that provides the access to the capability and its the capability which delivers the value.

So its not wrong to say 'smart endpoints and dumb pipes' but its better to say that the end-points have the value and the pipes are just infrastructure, this gives a better guidance on why you shouldn't be focusing on the pipes.

Now it does make a good point about not having smarts in the information fabric, but again this isn't new or unusual in decent SOA implementations.  I collaborated with the folks from Progress Software back in 2007 on the Business Service Bus concept which is just about having mediation (security, basic transformation) in the Bus.  There are good reasons why these things go there, cross-referencing between different ontologies is one.  This also plays into the 'always proxy' pattern that most decent SOA folks did.

So its again not wrong, its just that its not new, and it further underlines the inherent SOA nature of Microservices.
This really is another bit that I really do agree with, I've talked about the People's Democratic Republic of IT for years at conferences and finally got around to blogging on it.  The whole principle of business driven SOA is that the governance model better matches the business.  So again I think its not bad advice its just that SOA gives so much more than Microservices in terms of governance.  SOA as described in the OASIS SOA RM allows these principles to be applied to all IT assets not just those implemented with a specific implementation style and indeed to just to IT assets meaning its an approach that business schools are teaching.
The last thing I'll cover here is how the SOA sidebar is really a red herring, its not a true definition of SOA instead rolling out the old 'big' ESB and WS-* trope that was so loved by the RESTafarians when explaining why their way was better.  The claim is that this article 'crisply' describes the Microservices style and thus is valid in comparison with SOA as 'SOA means so many things'.  This I fundamentally disagree with, firstly because Microservices would be better served as an implementation approach if it could explain how it fits with non-Microservices approaches, something that SOA does a great job of doing, and second that it can't even say what makes a service 'micro' with services ranging from decent sized (12 person) teams down to individuals.

Conclusion

This for me is why Microservices is just a Service Oriented Delivery approach for a well architected SOA solution.  SOA provides the contextual framework, provides most of the rules that Microservices aims to adhere to but more over gives a broader context within which Microservices fit within a complex enterprise.  Calling out WS-* for the one millionth time or 'big' ESB and talking about massively complex projects is simply a shot at a different challenge.

Additionally the fact that one of the references that is used is that of Netflix who actually use the term 'fine-grained SOA' as recognised in the footnotes sort of underlines the fact and the fact that another (Amazon) also says its SOA.

I think its great that SOA is now coming back to the fore in the market as the hype around the plumbing (WS-* v REST) dies down and that the learnings of companies who have been doing this for over 10 years is now being talked about.  But that is the way to talk about it: what is the state of the 'now' in Service Oriented implementation and architecture?

Wednesday, March 12, 2014

What is real-time? Depends on who you ask

"Real-time" its a word that gets thrown about a lot in IT and its worth documenting a few of the different ways it gets used

Hard Real-time
This is what Real-time Java was created to address (along with Soft Real-time) what is this?  Easiest way to say it is that often in Hard Real-time environments the following statement is true
If it doesn't finish in X milliseconds then people might die
So if you miss a deadline its a systems failure.  Deadlines don't have to be super small, they could be '120 seconds' but the point is that you cannot miss them.

Soft Real-time
This was another use case for RTJ, here there are deadlines but missing deadlines isn't fatal but does degrade the value of the result and results in degraded performance.  Again though we are talking about deadlines not performance
If it doesn't finish in X milliseconds the system risks failing and will be performing sub-optimally
Machine real-time
This is when two machines are communicating on a task or processes within a machine, here the answer is 'as fast as is possible' there aren't any deadlines but the times are always measured below the microsecond level.  These are calculations and communications that get done millions and billions of times so a shift from 0.1ms to 1ms over a billion attempts means that the end-to-end work takes just over a day against 11 days.  This is the world of fast and HPC where the communications and processes need to be slimmed for speed.
Every microsecond counts, slim-it, trim-it because we're going to do it a billion times
Transactional real-time
Transactional real-time is about what it says, the time to complete a transaction, something that hits the database and returns.  Here we are in the millisecond to a tenth of a second type of range,  its this number that determines how internally responsive an application is.  This is the end to end time from initiation to response and at that point the state of the system has been changed.
Don't make me wait for you
User transactional real-time
User transaction real-time is what looks fast to a user of a system, this varies from interactional systems where it means sub-second to internet solutions and web-sites where it might mean 5 seconds or so.  Again this is the end-to-end time and includes something actually having happened and being able to check that it has happened.
Be fast enough so the user thinks its magic
BI reporting real-time
Next up are the pieces that are the views on the recent reality the time it takes for a report to be generated from 'landed data'.  Here BI guys tend to think of real-time in the same way as User Transactional real-time, 5 or so seconds is therefore acceptable.  This isn't however an end-to-end time as the data is already landed and all that is happening is the reports being done quickly.  Crucially however this is about reporting, its not about having transactional systems hitting the reporting system.
Let the user do all the reports and variations they want, we can handle it and not annoy them
BI real-time
The next definition is for the end-to-end of BI, so the extracting of data from a source system, the loading and transformation of that data and finally a report being done which includes that new information.  Here for BI the real-time definition can get longer and longer.  I've had clients say that 5 minutes, 15 minutes or even 2 hours are considered acceptable 'real-time' responses.  The point here is that when the current response time is 'next day' then something that is only minutes or even a few hours delayed is a significant increase in performance and is considered responsive.
Tell me shortly what went wrong right now
Chuck Norris Real-Time
In Chuck Norris real-time its happened before you even knew that you wanted it to happen.
Problem solved 

Tuesday, March 11, 2014

Microservices - Money for old rope or re-badging SOA for the cool kids

Hat tip to John Evedemon for the heads up on this one.  Martin Fowler is peddling a new approach, 'Microservices' which... wait for it is a way of developing applications as a suite of services.  Each one of which has its own process thread and 'communicates via lightweight mechanisms' such as.... over HTTP.

But wait there is more, you'll be stunned to know that these services can be built using different programming languages and even use different data stores.

Now down in the footnotes it makes a reference to Netflix talking about 'fine grained SOA' so its there that we begin to get the sniff of an old idea wrapped up in some shiny new wrapping there are a few things you need to do when saying this.  The first is critical don't learn or reference previous approaches except negatively.
Most languages do not have a good mechanism for defining an explicit Published Interface.
Now I really wish there was an approach that enabled a Service to publish a definition of itself of the Web, a sort of Web Service Description, if only there was such a language I might call it... oh I don't know Web Service Description Language.  The rest of the article talks about things that were common discussions around 2001, making things independently deployable but recognising that interface changes can have knock on impacts.  Hell I could even think of an Interface Definition Language or IDL that might do that as well.

Martin Fowler really should know better than this in not paying any heed to what has gone before and promoting an approach as if its actually new.  The article reads like an extremely basic description of SOA from about 2000 without either the industrialisation of WS-* or the dynamic power of things like Jini.  Above all it doesn't move forwards the question of how to architect such service architectures and how they need to map to the business and how although there might be fine grained services it is critical to understand the management structure and hierarchy of those services to actually enable the degree of autonomy and flexibility required in these sorts of architectures.

Microservices is just another take on SOA and one that doesn't move the game forwards as it yet again focuses on technical aspects over the business architecture that is realised.  Putting forwards as new something that would be recognisable to people working on CORBA in the 90s and certainly Web Services in 2001 is just poor quality advice.  It neither learns from the past, educates the reader nor moves the game forwards, its just selling an old approach with a new name.

Microservices?  What is that SOA 3.0?  Nope its just an old school form of SOA without the learnings that came from doing that.
 

What are the types of Data Scientist?

There are various views going around on what a Data Scientist is and what their value is to an organisation and the salaries they command.  To me however asking 'what is a Data Scientist?' is like asking 'What is a Physicist?' sure 'someone who studies Physics' might be a factually accurate but pointless definition.  How does that separate someone who did Physics in High School from Albert Einstein?  How does it separate the folks at CERN from someone using implicit Newtonian mechanics to play pool?   So it is with Data Science, but with an added twist.
Data Science is spectacularly badly defined
So yes you have courses cropping up at universities claiming to teach Data Science, you have consultants who have some mildly fancy Excel spreadsheets claiming they are Data Scientists.  In my career I've had the pleasure of working with some real Data Scientists, quite a lot of the time they didn't call themselves that but its what they were.  They used Data and applied some really fancy maths to deliver insight that just couldn't be attained with out it.  So I'll pick up the challenge laid down by Giga on whether Data Science is real or not and say 'yes.... but' here are my four groups of Data Scientists that I've worked with.

Data Magicians or Professor Data Science
Arthur C. Clarke once said that any sufficiently advanced technology is indistinguishable from magic.  This is how I feel when I work with people in this group.  They normally have mathematical or physics centric PhDs (often several), often focused in specific areas such as fluid dynamics, economics or super specific such as wind-turbines.  Why is what they do Magic?  Because these are the folks who work on 'next' and it is not a big group but these are the CERN folks of Data Science.  The reason its science is because its testable and provable.  They can show that their algorithm would have produced 5% improvement in performance over the past 5 years, and as it moves forward show how their approach has made a difference to the performance of a business.

How to know they are doing real Data Science?  Well the first hint is the mathematics, its the stuff where you remember the symbols but the combination of them all together now looks like gibberish, and yet these folks are arguing over specific parts of the formula as to how it can be improved.  Its like watching developers arguing over the right way to handle machine to machine communication.  You know who is smart by the outcome and the focus on the specific not the general.  Being blunt however very few companies need these folks and if they do they need very few.  Working with external organisations who have a good eco-system or Data Science structure is going to be better than having a lone Data Magician wandering around getting bored.    New algorithm development is not a regular thing.

Data Operators or Resident Data Scientist
The next group is what lots of companies will see value in.  These aren't the Magicians or Professors but they are crucial to making Data Science have value.  These are the people who take predefined algorithms, statistical or machine learning, and then apply them to a specific company scenario and most crucially keep the parameters up to date so the algorithm continues to perform.  These are the operational side of Data Science, the people for whom its a regular day job.  They can't do the design of a new algorithm but they can deliver specific value with an existing one which is what really counts for a business.  These folks are adapt at choosing the right approach and choosing between algorithms to choose the right way to deliver the most value.

These are the people most companies need to be thinking about, people who can take libraries like MADlib, languages like R or tools like SAS and then apply it to your local challenge, deliver the value and provide the on-going support to keep it effective.  Companies need access to these people either internally or as part of an external service, but in a more Outsourcing/regular way than with the project driven Magicians.  These people will have a formal mathematical or physics background, often up to the PhD level, but their abilities are more focused to application than invention.  These guys understand the formulas however and that is how you know they are real.

Data Hackers or Odd Job Data Scientist
The next group are people with a bit of skill, maybe a bit of training, but they aren't at the level of sophistication of an operator and are miles away from being a Magician.  Sadly these people often don't know this.  These are folks who've take the tools, learnt a bit about how they work and most often have fixated on a specific way of solving a specific problem with specific tools.  These folks are out there today and aiming to get work by being 'the one eyed man in the kingdom of the blind'. These folks can cost only $30 an hour apparently.  These folks add limited value but can improve the current situation in the same way as a decent report writer could.  In fact there is much overlap between the report writer and these people.  That doesn't mean they have zero value but that you need to know what they are doing.

Using these folks under the guidance of a Resident Data Scientist can add value, but don't mistake knowing how to apply one Machine Learning technique for actual knowledge.  The folks in this group know how the tool works but don't understand the mathematics behind it.

Data Science Bluffers or MS Office Data Scientists
This is the last group, and I'd say the one most responsible for the idea that Data Science might not be a real thing.  These are people who put a 'predict future stock price movement' box on their diagrams and mutter 'its Data Science'.  They are the people who get a spreadsheet with a bunch of data, apply a very basic statistical function and claim 'Hey its Data Science'.  These are the bluffers of Data Science the people who are trying to play the one eyed man and hoping that no-one notices they have a blindfold on.

This is a big group right now, populated in a large part by consultancies often those that specialise in Excel spreadsheet type of work and now claiming that this is the Data Science insight that the company needs.  These guys understand neither Data Science tools, nor the mathematics behind them but they do know how to create a good deck and Excel sheet.


Still doubt its real?
Still wondering whether this sort of thing is real? Well I'll give you a problem to solve - Medium Term Conflict Analysis in Air Traffic Management that is a 'hard' problem and one that requires really complex Mathematics to understand all of the pieces at play in the sky to achieve effectively.   Here is another one, you have 50 suppliers, 500 stores, 20 distribution centres, 100,000 SKUs and information streaming in about sales, social insight and stock levels... how do you efficiently procure, ship and stock to maximise profits?  Don't forget to include cannibalisation between brands and the costs of wasted stock or stock-room space.

There are lots of other challenges where Data Science adds value but the point is that you need to understand what sort of Data Science you are trying to achieve.  For most people this isn't getting Data Magicians, its about getting Data Operations folks, the Resident Data Scientists.

So its real, but don't forget to challenge people who say they are a Data Scientist and work out which bucket they fall into

  

Friday, March 07, 2014

BI change is coming, time to get over it and get on with the job

One of the things that always stuns me in IT is how people don't appear to like change.  Whether it was the EAI folks pushing back on Web Services in 2000 in favour of their old-school approaches.  The package guys pushing back against SaaS or now the BI guys pushing back against the new wave of BI technologies and approaches the message is always the same:
We are happy doing what we are doing, its great, I've done it for years, I don't want to change
That really isn't a long term career plan in IT.  This is the industry where revolution happens on a regular basis.  That doesn't mean you shouldn't be a skeptic, everyone in IT should be a skeptic and really ask for new approaches to prove themselves.  Its why I still don't think REST is 'the answer' to integration, I just haven't seen it proven to work at scale in the enterprise.

Sometimes however you can see the wave coming.  The Internet was one obvious wave and its impact on the enterprise was huge.  Now however there is another wave that I feel is equally obvious.  The wall between operations and analytics, between transactional systems and BI systems is being smashed down.  Its a wall IT put there as its been cheaper and simpler for us to have it there, but the business doesn't want these worlds separate any more.

This means that change is inevitable.  This is a change in the information space which is analogous to the desktop PC impact over the previous mini and mainframe computer era.  The change is going to be large.

Its going to require

  • Joined up thinking between applications, transactions, processes, analytics and information
  • Its going to require thinking in the local context while enabling global governance
  • Its going to require new governance models that better match the business
  • Its going to require all speeds of analytics, and the ability to change the speed as the business demand evolves.
This is a big wave and you cannot stand Canute like on the beach wishing for it to go away.  Much better to embrace the change and start planning for it as that way you won't be washed away.

IT changes.... its why its useful and its why is an interesting business.  Right now BI is at the centre of one of the most exciting changes in 20 years, we should celebrate that and get on with making it happen because otherwise we will wake up in 5 years and find our company has either struggled to compete or fail, or has succeed despite of us.