Microservices is SOD all within SOA

Microservices is a Service Oriented Delivery approach, all within a Service Oriented Architecture context.
(Long Title ;)

Ok so a few more updates since the last time I wrote about Microservices and I think its worth just updating as it really is heavily underlining why Microservices is a Service Oriented Delivery approach that absolutely can fit within a Service Oriented Architecture.  Lets be clear it isn't a bad thing to be that as delivery is where the rubber hits the road.  But every section that is written further underlines why the 'its not SOA' argument falls very shallow.

Infrastructure Automation
Not going to disagree here as this is just standard practice.  I actually think this isn't far enough forward looking in terms of infrastructure. Technologies such as CloudFoundry, supported by IBM, SAP, HP, Rackspace, EMC, VMWare, Pivotal, etc, move beyond simply automating the infrastructure side towards actually automating deployment at the application level.

This is a really solid SOD tip, certainly something I'd recommend.  Using technologies such as CloudFoundry can really help here and move beyond infrastructure automation.

Design for Failure 
Again a good tip in a service oriented architecture, back in 2007 I actually wrote about the challenges of five 9's and one was about planning for failure in SOA.  Again this is a really solid piece of advice but I feel it over simplifies the challenge.  Sometimes a service has failed because its impossible for it to succeed, particularly the case if using external services.  So 'restarting' is only an option sometimes and its important to understand the types of scenarios that you'll need to handle.

Evolutionary Design
This one comes from the school of the obvious.  The example used talks of a website, the Guardian, that was built as a monolith and was rearchitected to microservices.  I too have such an example of a large airline website that was built as a monolith and was getting hard to maintain and was broken down into various services and boundaries and separately compilable elements, this was in 2004-2006 however so again this really isn't new.

Yes of course you should look at something that will evolve... again a good tip but not something that marks Microservices down as anything other than a Service Oriented Delivery approach.

The goal of a good service oriented architecture is to "look like the business, evolve like the business and be costed in line with the business value" and Microservices lays down some nice rules for implementing certain parts of an enterprise but those are best served by an honesty that its an implementation choice within a broader Service Oriented Architecture.  That doesn't devalue it in any way, just places it in the right context.

Microservices is SOD, all within the context of SOA - and yes this time I added the comma.

I'd also like to put out a hat-tip to Loraine Lawson who pointed me towards the excellent Fallacy poster with the note that the SOA side-bar is really a 'Fallacy Fallacy'. 

Microservices is SOA, for those who know what SOA is.

Ok so its started a bit of debate on Twitter and now there have been emails, but in the spirit of openness I thought I'd better blog.  Now its good that Martin has now added a side bar on SOA to his article on Microservices but that really makes it worse in many ways.  I'll get to that at the end but first off lets explain why Microservices is just another SOA implementation pattern.  Its SOD for SOA

Lets break down the key precepts of Microservices and compare them against the OASIS SOA Reference Model (published 2006)

Componentization via Services

There follows a long text that can be reduced to

"Service Oriented Architecture (SOA) is a paradigm for organizing and utilizing distributed capabilities that may be under the control of different ownership domains. " 

There is an awful lot of words about deployment and process models in the Microservices piece but this single sentence at the start of the OASIS SOA RM is much more powerful because

1. It means that IT and non-IT services can be represented in a single approach
2. It immediately includes the major challenge - that of politics and ownership

OASIS then goes nicely into WTF actually is a service,

• The capability to perform work for another 
• The specification of the work offered for another 
• The offer to perform work for another

Again this is not a tech requirement which means your architecture can actually start from a business perspective rather than process threads.  In Fowler's microservices textual description (no bullet points here) we have

• services are out-of-process components who communicate
• explicit component interface
• ..... nothing on this bit 

OASIS then adds the kicker

in SOA, services are the mechanism by which needs and capabilities are brought together.

Remember that phrase 'mechanism' its going to be important....

Organized around Business Capabilities

Here is where Martin begins to get it wrong.  The OASIS SOA RM defines a capability as 

A real-world effect that a service provider is able to provide to a service consumer

The point here is the difference between a capability (the bit that does the work) and the service (the organising construct) is really important.  What we found when doing SOA in the wild for over a decade, and all the people on the OASIS SOA RM had lots of experience doing this, was that the organising framework was separate from the actions themselves.  The reason this is crucially important is that people started often making services where service = capability so you ended up with lots and lots of services (ummm if I was being insulting I'd call them microservices) here are some posts from 2006 and 2007 that explain why its really not great to confuse the two concepts and it made it into the SOA Antipatterns as well.  Now the actual text is pretty much ok, but again its lack of reference to the past and making a crucial mistake does not help people learn how things are evolving and what can be improved.

New?  Hell even I wrote a book which talked about how to model, manage and set up teams around this approach. The SOA Manifesto (2009) talks about key principles behind SOA (I still prefer the RM though) from a big group of practitioners.  The point here is that there are two problems, first the confusion of service and capabilities and secondly the lack of recognition of hierarchies importance in governance.

Products not Projects

Here it goes  beyond the OASIS SOA RM which doesn't talk about delivery models... however fortunately it goes beyond absolutely nothing that was said before.  A quick hit on Google just shows how many pieces there are in this space, some better than others and some products better than others but really this is not new.  I used to use the phrase 'Programs not projects' and always talked about assigning architects for the full lifecycle 'to make them accountable'.    Again its not that this statement is in anyway wrong, its just that its in no-way new.  We've known that this has helped for years but it has a significant issue: cost.

Lets say you have a service that requires some amazingly smart analytics, some hard core coding and some wonderful UI design.  You hire the very best, they cost a fortune and your service is awesome.  Do you really want those folks sitting around waiting for requirement changes?  No.  This is why for me the concentration is at the architecture and ownership level that consistency must be maintained, not at the developer level.  For companies where software development is their business you can include development but for most organisations in the real-world economics prevent the 'you built it you run it' mentality.  Again its a lack of learning that undermines the message.

Here I'm not disagreeing, it would be super ideal to have the same team always maintaining the code they write, its just not practical but thinking in terms of discreet pieces with their own heartbeats is a good thing.  Most decent SOA programs I've seen have recognised this and had different delivery schedules for different services.  What becomes important is the integrity of the service and its ability to be independently maintained, within the context of its management hierarchy, relegating this to 'keep the same developers' misses some of the power of SOA.  Again this emphasises that Microservices is just another SOD approach, a potentially good one in some circumstances but not something that will work for everyone and every circumstance...

Smart endpoints and dumb pipes

I really agree with this, but I think its better put in a different way.  In the OASIS SOA RM there is the concept of an 'execution context'.  Which is the bit that lets a service be called and its capability invoked.  Clearly the end-point is 'smart' as its what does the work, hence the phrase 'mechanism' used above.  The 'pipes' may or may not be dumb (the 'pipes' talking to those Rovers on Mars are pretty smart I think) but what they are is without value.  This was a crucial finding in SOA and is well documented in the SOA RM.    The execution context is where all of the internal plumbing is done but its the Service that provides the access to the capability and its the capability which delivers the value.

So its not wrong to say 'smart endpoints and dumb pipes' but its better to say that the end-points have the value and the pipes are just infrastructure, this gives a better guidance on why you shouldn't be focusing on the pipes.

Now it does make a good point about not having smarts in the information fabric, but again this isn't new or unusual in decent SOA implementations.  I collaborated with the folks from Progress Software back in 2007 on the Business Service Bus concept which is just about having mediation (security, basic transformation) in the Bus.  There are good reasons why these things go there, cross-referencing between different ontologies is one.  This also plays into the 'always proxy' pattern that most decent SOA folks did.

So its again not wrong, its just that its not new, and it further underlines the inherent SOA nature of Microservices.

Decentralized Governance

This really is another bit that I really do agree with, I've talked about the People's Democratic Republic of IT for years at conferences and finally got around to blogging on it.  The whole principle of business driven SOA is that the governance model better matches the business.  So again I think its not bad advice its just that SOA gives so much more than Microservices in terms of governance.  SOA as described in the OASIS SOA RM allows these principles to be applied to all IT assets not just those implemented with a specific implementation style and indeed to just to IT assets meaning its an approach that business schools are teaching.

Microservices and SOA

The last thing I'll cover here is how the SOA sidebar is really a red herring, its not a true definition of SOA instead rolling out the old 'big' ESB and WS-* trope that was so loved by the RESTafarians when explaining why their way was better.  The claim is that this article 'crisply' describes the Microservices style and thus is valid in comparison with SOA as 'SOA means so many things'.  This I fundamentally disagree with, firstly because Microservices would be better served as an implementation approach if it could explain how it fits with non-Microservices approaches, something that SOA does a great job of doing, and second that it can't even say what makes a service 'micro' with services ranging from decent sized (12 person) teams down to individuals.

Conclusion

This for me is why Microservices is just a Service Oriented Delivery approach for a well architected SOA solution.  SOA provides the contextual framework, provides most of the rules that Microservices aims to adhere to but more over gives a broader context within which Microservices fit within a complex enterprise.  Calling out WS-* for the one millionth time or 'big' ESB and talking about massively complex projects is simply a shot at a different challenge.

Additionally the fact that one of the references that is used is that of Netflix who actually use the term 'fine-grained SOA' as recognised in the footnotes sort of underlines the fact and the fact that another (Amazon) also says its SOA.

I think its great that SOA is now coming back to the fore in the market as the hype around the plumbing (WS-* v REST) dies down and that the learnings of companies who have been doing this for over 10 years is now being talked about.  But that is the way to talk about it: what is the state of the 'now' in Service Oriented implementation and architecture?

What is real-time? Depends on who you ask

"Real-time" its a word that gets thrown about a lot in IT and its worth documenting a few of the different ways it gets used

Hard Real-time
This is what Real-time Java was created to address (along with Soft Real-time) what is this?  Easiest way to say it is that often in Hard Real-time environments the following statement is true

If it doesn't finish in X milliseconds then people might die

So if you miss a deadline its a systems failure.  Deadlines don't have to be super small, they could be '120 seconds' but the point is that you cannot miss them.

Soft Real-time
This was another use case for RTJ, here there are deadlines but missing deadlines isn't fatal but does degrade the value of the result and results in degraded performance.  Again though we are talking about deadlines not performance

If it doesn't finish in X milliseconds the system risks failing and will be performing sub-optimally

Machine real-time
This is when two machines are communicating on a task or processes within a machine, here the answer is 'as fast as is possible' there aren't any deadlines but the times are always measured below the microsecond level.  These are calculations and communications that get done millions and billions of times so a shift from 0.1ms to 1ms over a billion attempts means that the end-to-end work takes just over a day against 11 days.  This is the world of fast and HPC where the communications and processes need to be slimmed for speed.

Every microsecond counts, slim-it, trim-it because we're going to do it a billion times

Transactional real-time
Transactional real-time is about what it says, the time to complete a transaction, something that hits the database and returns.  Here we are in the millisecond to a tenth of a second type of range,  its this number that determines how internally responsive an application is.  This is the end to end time from initiation to response and at that point the state of the system has been changed.

Don't make me wait for you

User transactional real-time
User transaction real-time is what looks fast to a user of a system, this varies from interactional systems where it means sub-second to internet solutions and web-sites where it might mean 5 seconds or so.  Again this is the end-to-end time and includes something actually having happened and being able to check that it has happened.

Be fast enough so the user thinks its magic

BI reporting real-time
Next up are the pieces that are the views on the recent reality the time it takes for a report to be generated from 'landed data'.  Here BI guys tend to think of real-time in the same way as User Transactional real-time, 5 or so seconds is therefore acceptable.  This isn't however an end-to-end time as the data is already landed and all that is happening is the reports being done quickly.  Crucially however this is about reporting, its not about having transactional systems hitting the reporting system.

Let the user do all the reports and variations they want, we can handle it and not annoy them

BI real-time
The next definition is for the end-to-end of BI, so the extracting of data from a source system, the loading and transformation of that data and finally a report being done which includes that new information.  Here for BI the real-time definition can get longer and longer.  I've had clients say that 5 minutes, 15 minutes or even 2 hours are considered acceptable 'real-time' responses.  The point here is that when the current response time is 'next day' then something that is only minutes or even a few hours delayed is a significant increase in performance and is considered responsive.

Tell me shortly what went wrong right now

Chuck Norris Real-Time
In Chuck Norris real-time its happened before you even knew that you wanted it to happen.

Problem solved 

Microservices - Money for old rope or re-badging SOA for the cool kids

Hat tip to John Evedemon for the heads up on this one.  Martin Fowler is peddling a new approach, 'Microservices' which... wait for it is a way of developing applications as a suite of services.  Each one of which has its own process thread and 'communicates via lightweight mechanisms' such as.... over HTTP.

But wait there is more, you'll be stunned to know that these services can be built using different programming languages and even use different data stores.

Now down in the footnotes it makes a reference to Netflix talking about 'fine grained SOA' so its there that we begin to get the sniff of an old idea wrapped up in some shiny new wrapping there are a few things you need to do when saying this.  The first is critical don't learn or reference previous approaches except negatively.

Most languages do not have a good mechanism for defining an explicit Published Interface.

Now I really wish there was an approach that enabled a Service to publish a definition of itself of the Web, a sort of Web Service Description, if only there was such a language I might call it... oh I don't know Web Service Description Language.  The rest of the article talks about things that were common discussions around 2001, making things independently deployable but recognising that interface changes can have knock on impacts.  Hell I could even think of an Interface Definition Language or IDL that might do that as well.

Martin Fowler really should know better than this in not paying any heed to what has gone before and promoting an approach as if its actually new.  The article reads like an extremely basic description of SOA from about 2000 without either the industrialisation of WS-* or the dynamic power of things like Jini.  Above all it doesn't move forwards the question of how to architect such service architectures and how they need to map to the business and how although there might be fine grained services it is critical to understand the management structure and hierarchy of those services to actually enable the degree of autonomy and flexibility required in these sorts of architectures.

Microservices is just another take on SOA and one that doesn't move the game forwards as it yet again focuses on technical aspects over the business architecture that is realised.  Putting forwards as new something that would be recognisable to people working on CORBA in the 90s and certainly Web Services in 2001 is just poor quality advice.  It neither learns from the past, educates the reader nor moves the game forwards, its just selling an old approach with a new name.

Microservices?  What is that SOA 3.0?  Nope its just an old school form of SOA without the learnings that came from doing that.

 

What are the types of Data Scientist?

There are various views going around on what a Data Scientist is and what their value is to an organisation and the salaries they command.  To me however asking 'what is a Data Scientist?' is like asking 'What is a Physicist?' sure 'someone who studies Physics' might be a factually accurate but pointless definition.  How does that separate someone who did Physics in High School from Albert Einstein?  How does it separate the folks at CERN from someone using implicit Newtonian mechanics to play pool?   So it is with Data Science, but with an added twist.

Data Science is spectacularly badly defined

So yes you have courses cropping up at universities claiming to teach Data Science, you have consultants who have some mildly fancy Excel spreadsheets claiming they are Data Scientists.  In my career I've had the pleasure of working with some real Data Scientists, quite a lot of the time they didn't call themselves that but its what they were.  They used Data and applied some really fancy maths to deliver insight that just couldn't be attained with out it.  So I'll pick up the challenge laid down by Giga on whether Data Science is real or not and say 'yes.... but' here are my four groups of Data Scientists that I've worked with.

Data Magicians or Professor Data Science
Arthur C. Clarke once said that any sufficiently advanced technology is indistinguishable from magic.  This is how I feel when I work with people in this group.  They normally have mathematical or physics centric PhDs (often several), often focused in specific areas such as fluid dynamics, economics or super specific such as wind-turbines.  Why is what they do Magic?  Because these are the folks who work on 'next' and it is not a big group but these are the CERN folks of Data Science.  The reason its science is because its testable and provable.  They can show that their algorithm would have produced 5% improvement in performance over the past 5 years, and as it moves forward show how their approach has made a difference to the performance of a business.

How to know they are doing real Data Science?  Well the first hint is the mathematics, its the stuff where you remember the symbols but the combination of them all together now looks like gibberish, and yet these folks are arguing over specific parts of the formula as to how it can be improved.  Its like watching developers arguing over the right way to handle machine to machine communication.  You know who is smart by the outcome and the focus on the specific not the general.  Being blunt however very few companies need these folks and if they do they need very few.  Working with external organisations who have a good eco-system or Data Science structure is going to be better than having a lone Data Magician wandering around getting bored.    New algorithm development is not a regular thing.

Data Operators or Resident Data Scientist
The next group is what lots of companies will see value in.  These aren't the Magicians or Professors but they are crucial to making Data Science have value.  These are the people who take predefined algorithms, statistical or machine learning, and then apply them to a specific company scenario and most crucially keep the parameters up to date so the algorithm continues to perform.  These are the operational side of Data Science, the people for whom its a regular day job.  They can't do the design of a new algorithm but they can deliver specific value with an existing one which is what really counts for a business.  These folks are adapt at choosing the right approach and choosing between algorithms to choose the right way to deliver the most value.

These are the people most companies need to be thinking about, people who can take libraries like MADlib, languages like R or tools like SAS and then apply it to your local challenge, deliver the value and provide the on-going support to keep it effective.  Companies need access to these people either internally or as part of an external service, but in a more Outsourcing/regular way than with the project driven Magicians.  These people will have a formal mathematical or physics background, often up to the PhD level, but their abilities are more focused to application than invention.  These guys understand the formulas however and that is how you know they are real.

Data Hackers or Odd Job Data Scientist
The next group are people with a bit of skill, maybe a bit of training, but they aren't at the level of sophistication of an operator and are miles away from being a Magician.  Sadly these people often don't know this.  These are folks who've take the tools, learnt a bit about how they work and most often have fixated on a specific way of solving a specific problem with specific tools.  These folks are out there today and aiming to get work by being 'the one eyed man in the kingdom of the blind'. These folks can cost only $30 an hour apparently.  These folks add limited value but can improve the current situation in the same way as a decent report writer could.  In fact there is much overlap between the report writer and these people.  That doesn't mean they have zero value but that you need to know what they are doing.

Using these folks under the guidance of a Resident Data Scientist can add value, but don't mistake knowing how to apply one Machine Learning technique for actual knowledge.  The folks in this group know how the tool works but don't understand the mathematics behind it.

Data Science Bluffers or MS Office Data Scientists
This is the last group, and I'd say the one most responsible for the idea that Data Science might not be a real thing.  These are people who put a 'predict future stock price movement' box on their diagrams and mutter 'its Data Science'.  They are the people who get a spreadsheet with a bunch of data, apply a very basic statistical function and claim 'Hey its Data Science'.  These are the bluffers of Data Science the people who are trying to play the one eyed man and hoping that no-one notices they have a blindfold on.

This is a big group right now, populated in a large part by consultancies often those that specialise in Excel spreadsheet type of work and now claiming that this is the Data Science insight that the company needs.  These guys understand neither Data Science tools, nor the mathematics behind them but they do know how to create a good deck and Excel sheet.


Still doubt its real?
Still wondering whether this sort of thing is real? Well I'll give you a problem to solve - Medium Term Conflict Analysis in Air Traffic Management that is a 'hard' problem and one that requires really complex Mathematics to understand all of the pieces at play in the sky to achieve effectively.   Here is another one, you have 50 suppliers, 500 stores, 20 distribution centres, 100,000 SKUs and information streaming in about sales, social insight and stock levels... how do you efficiently procure, ship and stock to maximise profits?  Don't forget to include cannibalisation between brands and the costs of wasted stock or stock-room space.

There are lots of other challenges where Data Science adds value but the point is that you need to understand what sort of Data Science you are trying to achieve.  For most people this isn't getting Data Magicians, its about getting Data Operations folks, the Resident Data Scientists.

So its real, but don't forget to challenge people who say they are a Data Scientist and work out which bucket they fall into

  

BI change is coming, time to get over it and get on with the job

One of the things that always stuns me in IT is how people don't appear to like change.  Whether it was the EAI folks pushing back on Web Services in 2000 in favour of their old-school approaches.  The package guys pushing back against SaaS or now the BI guys pushing back against the new wave of BI technologies and approaches the message is always the same:

We are happy doing what we are doing, its great, I've done it for years, I don't want to change

That really isn't a long term career plan in IT.  This is the industry where revolution happens on a regular basis.  That doesn't mean you shouldn't be a skeptic, everyone in IT should be a skeptic and really ask for new approaches to prove themselves.  Its why I still don't think REST is 'the answer' to integration, I just haven't seen it proven to work at scale in the enterprise.

Sometimes however you can see the wave coming.  The Internet was one obvious wave and its impact on the enterprise was huge.  Now however there is another wave that I feel is equally obvious.  The wall between operations and analytics, between transactional systems and BI systems is being smashed down.  Its a wall IT put there as its been cheaper and simpler for us to have it there, but the business doesn't want these worlds separate any more.

This means that change is inevitable.  This is a change in the information space which is analogous to the desktop PC impact over the previous mini and mainframe computer era.  The change is going to be large.

Its going to require

• Joined up thinking between applications, transactions, processes, analytics and information
• Its going to require thinking in the local context while enabling global governance
• Its going to require new governance models that better match the business
• Its going to require all speeds of analytics, and the ability to change the speed as the business demand evolves.

This is a big wave and you cannot stand Canute like on the beach wishing for it to go away.  Much better to embrace the change and start planning for it as that way you won't be washed away.

IT changes.... its why its useful and its why is an interesting business.  Right now BI is at the centre of one of the most exciting changes in 20 years, we should celebrate that and get on with making it happen because otherwise we will wake up in 5 years and find our company has either struggled to compete or fail, or has succeed despite of us.

The next big wave of IT is Software Development

I can smell a change coming, the last few years have seen cloud and SaaS on the rise and seen a fragmentation in application development (thanks in a large part to the appalling stewardship of Java) and a real focus of budgets around BI and 'vanilla' package approaches.  Now this is a good thing, both because I jumped out of the Java boat onto the BI boat a few years ago but also because its really helped shift the investment focus away from 'Big iT' towards 'Big It' - by which I mean the focus has shifted very firmly towards Information over technology.

Now however companies are waking up to the fact that as good as a Salesforce.com or SAP is it really doesn't matter if everyone in your industry is using it, these are not your differentiators.  Your differentiators are what matter in terms of accelerating growth and outperforming your competitors.

This is where software development comes back and I predict the following four waves

• Wave 1 - the personal developer
• Wave 2 - the team developer
• Wave 3 - the enterprise developer
• Wave 4 - back to the package

Big Data will drive the new Software Development explosion

Big Data is the hype today and it will be the foundation of this new era as information is the key, fast data will however rapidly become just, if not more, important than 'Big' which means that the ability to integrate into transactions and deliver differentiation will be critical.  This is why we'll see a resurgence in software development based on new approaches and new platforms but we'll see the same sort of consolidation phases that we saw around Java.  Maybe this wave will last 10 years, maybe 5, maybe 20 but what appears certain is that the wave is coming.

This isn't the older wave though, it never is in IT, its the same sort of thing but now applied to the next generation of problems.  This is about information and collaboration and digitization of organisations, its about taking all these vast amounts of internal and external information and driving better outcomes and crucially about collaborating between organisations.

Lets be clear: this is going to be hard.

Back with Java we had a Database, we had SQL, we had (eventually) an App Server... that my friends was a walk in the park compared with what is coming.  I'll write a new post shortly on what the new world could look like but suffice to say you need the following

1) An inherent understanding of what makes information work - Master Data, Big Data, Analytics

2) An understanding of how to drive impact - how to engage users

3) An understanding of how to work with people outside your organisation

You thought doing a Java project with offshore delivery and a bunch of legacy integration points was hard?  Hang on to your hats... 

Software Development Wave 4: back to the package

The end of the next Software Development wave will be when Software development against 'eats itself' as it did with with technologies like Hadoop showing a new value in information, with platforms like SFDC showing new pre-build services, where people like GoodData have turned BI into SaaS.  So we will see the same evolution again and a new generation of commoditisation which drives consolidation and cost saving to replace things that would differentiate today but will be a commodity in 5-10 years time.

This is what always happens.  SAP was created because people had written custom software for factories for years and they saw a market for a platform.  Siebel was born because people had built software to manage client contacts for years and Salesforce.com because it wasn't really that important that the software be unique to your business.

Software Development always leads to packages in the enterprise space because there are lots of companies doing similar things.  Master Data Management is a great example of an old problem in the information space which has been heavily commoditised by vendors in the last 10 years.  Hadoop has turned the challenge of handling massive data volumes into a file system.

Software development waves lead to a new wave of package solutions, the "buy v build" question is always asked and software development wins when the answer is 'there is nothing we can buy' and package wins when the answer is 'this does 90%+ of what we want for a fraction of the risk'. In between there are grey areas, which is why the transition is never clean, but the reality is that after this next wave of software development we should expect to see the next generation of package solutions.  I think however there is a good chance that these next ones will be markedly different to the current generation.

The next generation will be based around information and insight and less around repeatable processes.  The old packages will exist, like a geological epoch, but there will be less and less reason to change what is established in those areas, the new value will be built on their foundations, first by software development and then by the commoditisation and packaging of the new generation of solutions.

This is part of a series of posts on Why Software Development is the next big Wave and is preceded by Wave 1: The Personal Developer, Wave 2: The team Developer and Wave 3: the Enterprise Developer

Software Development Wave 3: the enterprise developer

This is the stage at which software development begins to commoditise itself, its no surprise that underneath all that Salesforce.com scripting lurked rather a lot of Java code.  This wave sees the rise of the libraries, the utilities and above all the commoditisation of software in a way that enables the majority of developers to be useful in the enterprise.  This was the goal of Spring, JEE and numerous other frameworks but also the aim of BPEL and other visual process approaches which leveraged the Java platform.  Now some may argue this wave hasn't crashed, but I'm sorry it has, the innovation in the core enterprise space in the last few years has been practically zero, the brains have been elsewhere even if the money hasn't.

The new wave is going to do much of the same as the old wave, including mistaking SOA for being about end-points rather than architecture.  This time however the orchestrations and work is going to be much more about straddling organisations and collaboration than just delivering internal solutions.  Having multiple companies working together on something is going to need some pretty fancy tooling and its my money that this space sees the next generation of enterprise software solutions that kick off the next wave of packaged software/SaaS focused investment in IT.

This is part of a series of posts on Why Software Development is the next big Wave and is preceded by Wave 1: The Personal Developer and Wave 2: The team Developer and followed by Wave 4: Back to the package

Software Development Wave 2 - the team developer

The problem with Wave 1 was that it didn't scale, I mean sure lots of the personal developers claimed it did scale, often laughing at large scale developments and going 'Me and four mates could do that in a couple of weeks' often they attempted to do that and suddenly realised that when you get a few people together it gets a bit more complicated and when that few gets over 20 it begins to get really, really complicated.  Back in the Java days this era saw the rise in the importance of design, but design focused on development rather than pictures, like with TogetherJ against the older school Rational products.

At the start of the 21st century this battle between the personal and the team developer was fought out on the floor of the JavaOne conference, '.com' champions of the personal style battled it out with a new breed of Java developers who were working for dull and boring corporations.  The noise and the hype in 2000, and indeed into 2001 given that not everyone had popped, was strongly on the side of the personal developers but the actual cash, rather than paper money, was on the side of the enterprise folks.

When the bubble had finally popped a new wave was about, sure Agile was kicking off but even in its most extreme of XP debates the talk was about quality, about testing, about making it maintainable.  Suddenly the battle wasn't between the personal and the team it was between what type of team dynamic was best.

Some organisations are beginning to hit this wave again, but I'd argue that it won't be the mainstream mentality for another couple of years.  What we'll see this time is a focus back on design but also a focus towards eliminating many of the things that made team dynamics so poor, things like deployment times, infrastructure management etc.  Cloud and now PaaS offers like CloudFoundry are changing the game here already but its still a jump for companies to move towards a next generation team development infrastructure.

This next generation will also see teams work between organisations, giving a whole new team dynamic challenge for us to overcome.  This means that the new generation is going to be less about processes and transactions (as the Java wave was) and more about information collaboration and digitization.   So while traditional design approaches such as UML might make a comeback I'd suggest that new approaches that concentrate around information-centric collaboration are going to be created and become more popular.

The challenge of the team developer era however was, and is, that its still for a minority.  The Java community between 2001 and 2003 was absolutely exploding but arguably the tools and techniques were still a bit above the majority.  The big question was how to roll out software development to everyone...

This is part of a series of posts on Why Software Development is the next big Wave and is preceded by Wave 1: The Personal Developer and followed by Wave 3: the Enterprise Developer and Wave 4: Back to the package

Software Development Wave 1: The Personal Developer

This is the wave we are in at the moment and its the wave that we last saw in the late 90s, this is where technologies enabled single people to build small specific things really quickly.  Java and its applets really were the peak of this first wave back then but now we are seeing people use technologies such as R, Python and others to create small solutions that offer really good point value.

Right now I see lots of this bubbling on the edges, just as I did in the late 90s.  The core of enterprises back then was about the SAP and other ERP projects, my work was in 'true' customer development (there isn't a package for Air Traffic Control) but as I moved into the enterprise space I saw a real shift away from rigour in software development towards more personal solutions where maintenance was considered less of an issue.  The Perl script that did something useful but only Dave could change, the C program that worked... but no-one had the source anymore.  The stored procedure that did everything you could ever want... as long as you wanted what was required 2 years ago.

The challenge with the personal developer was maintenance and evolution, people coded for themselves to support and for them to create on their own or in very small teams.  What we've seen already is that people are moving up the stack and using technologies like Hive and HAWQ to give older school interfaces on this data as that means more developers can work together... which brings its own challenges.

This is part of a series of posts on Why Software Development is the next big Wave and is followed by Wave 2: The team Developer, Wave 3: the Enterprise Developer and Wave 4: Back to the package

Musketeers Day - All Four One and One Four All

Okay in the spirit of brotherly love, helping people out and of course International Talk Like a Pirate Day I think we should declare April the 1st 2014 as Musketeers Day.  Why? Well for the vast Gregorian Calendar (non-US) part of the world the date will be 4/1/14 or All "four one and one four" All.

In honour of that day, and as it falls on April Fool's day to boot I declare the following:

1. All people should declare fealty to the French King
2. Each office should have an appointed D'Artagnan who must do most of the work while others take the credit
3. Wearing of swords is mandatory
4. As are floppy hats

Musketeers Day will not happen for another 100 years!