Making DevOps Business Driven - a service view

I've been doing a bit recently around DevOps and what I've been seeing is that companies that having been scaling DevOps tend to run into a problem: exactly what is a good boundary for a DevOps team? Now I've talked before about how Microservices are just SOA with a new logo, well there is an interesting piece about DevOps as well, its not actually a brand new thing.  Its an evolution and industrialisation of what was leading practice several years ago.

Back in 2007 I gave a presentation on why SOA was a business challenge (full deck at the end) and in there were two pictures that talked about how you needed to change the way you thought about services:

So on the left we've got a view that says that you need to think about a full lifecycle, and on the right you've got a picture that talks about the needs to have an architect, owner and delivery manager (programme manager)
This is what we were doing around SOA projects back in 2007 as a structure and getting the architects and developers (but ESPECIALLY the architects) to be accountable for the full lifecycle.  Its absolutely fantastic to see this becoming normal practice and there are some great lessons out there and technical approaches.

One thing I've not seen however is an answer to what my DevOps team is and how I manage a large number of DevOps teams.  This is where Business Architecture comes in, the point here is that its not enough to just have lots and lots of DevOps teams, you need to align those to the business owners and align them to the structure that is driving them.  You also need to have that structure so one team doesn't just call the 'Buy from Ferrari' internal service without going through procurement first for approval.

So in a DevOps world we are beginning to realize the full-lifecycle view on Business Services, providing a technical approach to automating and managing services that look like the business, evolve like the business and provide the business a structure where they can focus costs where it delivers the most value.

There is much new in the DevOps world, but there is also much we can learn from the Business Architecture space on how to set up DevOps teams to better align to the business and enable DevOps to scale at traditional complex organisations as well as more simple (from a business model perspective) internet companies.

The next big wave of IT is Software Development

I can smell a change coming, the last few years have seen cloud and SaaS on the rise and seen a fragmentation in application development (thanks in a large part to the appalling stewardship of Java) and a real focus of budgets around BI and 'vanilla' package approaches.  Now this is a good thing, both because I jumped out of the Java boat onto the BI boat a few years ago but also because its really helped shift the investment focus away from 'Big iT' towards 'Big It' - by which I mean the focus has shifted very firmly towards Information over technology.

Now however companies are waking up to the fact that as good as a Salesforce.com or SAP is it really doesn't matter if everyone in your industry is using it, these are not your differentiators.  Your differentiators are what matter in terms of accelerating growth and outperforming your competitors.

This is where software development comes back and I predict the following four waves

• Wave 1 - the personal developer
• Wave 2 - the team developer
• Wave 3 - the enterprise developer
• Wave 4 - back to the package

Big Data will drive the new Software Development explosion

Big Data is the hype today and it will be the foundation of this new era as information is the key, fast data will however rapidly become just, if not more, important than 'Big' which means that the ability to integrate into transactions and deliver differentiation will be critical.  This is why we'll see a resurgence in software development based on new approaches and new platforms but we'll see the same sort of consolidation phases that we saw around Java.  Maybe this wave will last 10 years, maybe 5, maybe 20 but what appears certain is that the wave is coming.

This isn't the older wave though, it never is in IT, its the same sort of thing but now applied to the next generation of problems.  This is about information and collaboration and digitization of organisations, its about taking all these vast amounts of internal and external information and driving better outcomes and crucially about collaborating between organisations.

Lets be clear: this is going to be hard.

Back with Java we had a Database, we had SQL, we had (eventually) an App Server... that my friends was a walk in the park compared with what is coming.  I'll write a new post shortly on what the new world could look like but suffice to say you need the following

1) An inherent understanding of what makes information work - Master Data, Big Data, Analytics

2) An understanding of how to drive impact - how to engage users

3) An understanding of how to work with people outside your organisation

You thought doing a Java project with offshore delivery and a bunch of legacy integration points was hard?  Hang on to your hats... 

Software Development Wave 4: back to the package

The end of the next Software Development wave will be when Software development against 'eats itself' as it did with with technologies like Hadoop showing a new value in information, with platforms like SFDC showing new pre-build services, where people like GoodData have turned BI into SaaS.  So we will see the same evolution again and a new generation of commoditisation which drives consolidation and cost saving to replace things that would differentiate today but will be a commodity in 5-10 years time.

This is what always happens.  SAP was created because people had written custom software for factories for years and they saw a market for a platform.  Siebel was born because people had built software to manage client contacts for years and Salesforce.com because it wasn't really that important that the software be unique to your business.

Software Development always leads to packages in the enterprise space because there are lots of companies doing similar things.  Master Data Management is a great example of an old problem in the information space which has been heavily commoditised by vendors in the last 10 years.  Hadoop has turned the challenge of handling massive data volumes into a file system.

Software development waves lead to a new wave of package solutions, the "buy v build" question is always asked and software development wins when the answer is 'there is nothing we can buy' and package wins when the answer is 'this does 90%+ of what we want for a fraction of the risk'. In between there are grey areas, which is why the transition is never clean, but the reality is that after this next wave of software development we should expect to see the next generation of package solutions.  I think however there is a good chance that these next ones will be markedly different to the current generation.

The next generation will be based around information and insight and less around repeatable processes.  The old packages will exist, like a geological epoch, but there will be less and less reason to change what is established in those areas, the new value will be built on their foundations, first by software development and then by the commoditisation and packaging of the new generation of solutions.

This is part of a series of posts on Why Software Development is the next big Wave and is preceded by Wave 1: The Personal Developer, Wave 2: The team Developer and Wave 3: the Enterprise Developer

Software Development Wave 3: the enterprise developer

This is the stage at which software development begins to commoditise itself, its no surprise that underneath all that Salesforce.com scripting lurked rather a lot of Java code.  This wave sees the rise of the libraries, the utilities and above all the commoditisation of software in a way that enables the majority of developers to be useful in the enterprise.  This was the goal of Spring, JEE and numerous other frameworks but also the aim of BPEL and other visual process approaches which leveraged the Java platform.  Now some may argue this wave hasn't crashed, but I'm sorry it has, the innovation in the core enterprise space in the last few years has been practically zero, the brains have been elsewhere even if the money hasn't.

The new wave is going to do much of the same as the old wave, including mistaking SOA for being about end-points rather than architecture.  This time however the orchestrations and work is going to be much more about straddling organisations and collaboration than just delivering internal solutions.  Having multiple companies working together on something is going to need some pretty fancy tooling and its my money that this space sees the next generation of enterprise software solutions that kick off the next wave of packaged software/SaaS focused investment in IT.

This is part of a series of posts on Why Software Development is the next big Wave and is preceded by Wave 1: The Personal Developer and Wave 2: The team Developer and followed by Wave 4: Back to the package

Software Development Wave 2 - the team developer

The problem with Wave 1 was that it didn't scale, I mean sure lots of the personal developers claimed it did scale, often laughing at large scale developments and going 'Me and four mates could do that in a couple of weeks' often they attempted to do that and suddenly realised that when you get a few people together it gets a bit more complicated and when that few gets over 20 it begins to get really, really complicated.  Back in the Java days this era saw the rise in the importance of design, but design focused on development rather than pictures, like with TogetherJ against the older school Rational products.

At the start of the 21st century this battle between the personal and the team developer was fought out on the floor of the JavaOne conference, '.com' champions of the personal style battled it out with a new breed of Java developers who were working for dull and boring corporations.  The noise and the hype in 2000, and indeed into 2001 given that not everyone had popped, was strongly on the side of the personal developers but the actual cash, rather than paper money, was on the side of the enterprise folks.

When the bubble had finally popped a new wave was about, sure Agile was kicking off but even in its most extreme of XP debates the talk was about quality, about testing, about making it maintainable.  Suddenly the battle wasn't between the personal and the team it was between what type of team dynamic was best.

Some organisations are beginning to hit this wave again, but I'd argue that it won't be the mainstream mentality for another couple of years.  What we'll see this time is a focus back on design but also a focus towards eliminating many of the things that made team dynamics so poor, things like deployment times, infrastructure management etc.  Cloud and now PaaS offers like CloudFoundry are changing the game here already but its still a jump for companies to move towards a next generation team development infrastructure.

This next generation will also see teams work between organisations, giving a whole new team dynamic challenge for us to overcome.  This means that the new generation is going to be less about processes and transactions (as the Java wave was) and more about information collaboration and digitization.   So while traditional design approaches such as UML might make a comeback I'd suggest that new approaches that concentrate around information-centric collaboration are going to be created and become more popular.

The challenge of the team developer era however was, and is, that its still for a minority.  The Java community between 2001 and 2003 was absolutely exploding but arguably the tools and techniques were still a bit above the majority.  The big question was how to roll out software development to everyone...

This is part of a series of posts on Why Software Development is the next big Wave and is preceded by Wave 1: The Personal Developer and followed by Wave 3: the Enterprise Developer and Wave 4: Back to the package

Software Development Wave 1: The Personal Developer

This is the wave we are in at the moment and its the wave that we last saw in the late 90s, this is where technologies enabled single people to build small specific things really quickly.  Java and its applets really were the peak of this first wave back then but now we are seeing people use technologies such as R, Python and others to create small solutions that offer really good point value.

Right now I see lots of this bubbling on the edges, just as I did in the late 90s.  The core of enterprises back then was about the SAP and other ERP projects, my work was in 'true' customer development (there isn't a package for Air Traffic Control) but as I moved into the enterprise space I saw a real shift away from rigour in software development towards more personal solutions where maintenance was considered less of an issue.  The Perl script that did something useful but only Dave could change, the C program that worked... but no-one had the source anymore.  The stored procedure that did everything you could ever want... as long as you wanted what was required 2 years ago.

The challenge with the personal developer was maintenance and evolution, people coded for themselves to support and for them to create on their own or in very small teams.  What we've seen already is that people are moving up the stack and using technologies like Hive and HAWQ to give older school interfaces on this data as that means more developers can work together... which brings its own challenges.

This is part of a series of posts on Why Software Development is the next big Wave and is followed by Wave 2: The team Developer, Wave 3: the Enterprise Developer and Wave 4: Back to the package

Software Developers are you ready for the cage fight with the BI guys?

In all of my career to date in IT there really has been three clear worlds in IT, the software development guys who are the bespoke tailors, the package guys who deliver off the shelf and the BI guys.

I'm going to admit a prejudice here.  Until a couple of years ago my impression of BI guys was folks who were one step up from Excel guys.  It was dead-data and done in batches, sure it had to be done but it really wasn't gool.  The BI guys couldn't handle the operational requirements of the software developers and we really were not interested in Cubes, Star-Schemas and all of the stuff they did to generate what was basically an Excel spreadsheet on steroids.

Now having spent some more time on that side of the fence I've come to respect it more, but there is now an interesting evolution.  Contextual BI.  That is where BI guys talk about integrating BI and Analytics information back into operational processes.  In other words its BI guys talking about integrating to SAP and integrating into BPM and integrating into operational applications generally.

An IT turf war is about to break-out.  I'd argue that the software development guys have a bit of an advantage here, what is the current 'cool kids' BI tool?  Its Hadoop.  What do you need to know to really grok Hadoop?  Yup Java, a software development language.  Software Dev guys grok agile in a way that few BI guys do and are used to getting down into operational requirements.

But we suck at analytics and its massively expensive to develop the sorts of visualisations that BI guys get out of the box.  So this means its either a case of one side taking over the other, or we are going to have to co-operate... and I actually think this is the way forwards.  Software Development guys know how to do real-time scaling and operations, its not a trivial skill, and the BI guys know how to get us the right information into our processes and have the right visualisation tools.

So Software Developers out there its time to reach out and embrace your BI cousins and start looking at how you can work together to create an information fabric which spans the analytical models on historical data, supplements it with real-time analytics and then integrates that back into the operational process.  I know this collaboration is going to be hard and we've all got to get over prejudices to make this happen but the business is demanding it so we need to get it done.  Start thinking about information as a seemless infrastructure where delivery to the point of action is the challenge not delivery to a report.  Software Developers and BI guys working together can create this and deliver a real information fabric to the enterprise.

And don't worry Software Developers and BI guys will still have a common enemy, I'm not crazy, its still perfectly ok to not get on with the package guys.