Thursday, May 10, 2007

Engineering v Art the challenge of the masses v the talent

There has been more discussion recently around the contracts v late validation argument and Stefan Tilkov has a brief position around how the REST v WS links in and that thought that maybe these really were two different worlds that would never agree. I'm with Stefan, who is on the late validation side, that this is a big divide and is indeed something that has gone on in IT since I can remember.

Where I do disagree though is whether this is a good or a bad thing to have these camps. Now I'm clearly biased as I'm on the contract side but I thought I'd put the case as to why contract based and enforcement, and static languages, represent the engineering approach to IT delivery while dynamic languages and late validation is the approach taken by those who consider IT to be an art. This doesn't make the later inherently wrong, but it does mean that it is hugely predicated on the talent of the person doing the job.

This for me is the problem. When I used to do a lot of "proper" User Interface design there was often talk about the challenges of WILI v KISS. Everyone knows KISS, everyone recites it as an empty mantra, but most people end up doing WILI. WILI is "Well I like it" and is used as the rationalisation of lots of bad decisions, with UI design this was added to by the "well Microsoft do that" school of justification (which is fine if you are doing a word extension, but not if you are doing radar displays).

The same WILI and "appeal to authority" approach is part of the great Technology Delusion that runs straight through IT, part of the problem is simply one of talent. If I could have a team in support of Stefan and Mark Baker I'd have no trouble whatsoever doing REST, if I could have Larry Wall in support I'd be reasonably happy to have PERL code there and so the list of the truly talented goes on. Give me Dan Creswell in support and that Jini idea looks fantastic, hell I'd even say go for a massive project using Spring 1.0 if Rod Johnson had to maintain the XML files.

The appeal of dynamic languages to people who genuinely understand how computer systems should work is clearly appealing, as is pushing the bounds of technology and creating new and exciting ways that are just a little bit better than what came before. These are the people to whom IT is an art form, something that requires imagination and craft and to whom that improvement is important as they are trying to push the boundaries.

I've used dynamic languages, I mess around with them on my personal projects, but I never tend to recommend them on projects and indeed actively ban them when I am the architect.

So why do I choose to have strict contracts, static languages, early validation of everything and extremely rigorous standards applied to code, build and test? The answer was nicely highlighted by Bill Roth on his post around JavaOne Day 1, there are SIX MILLION Java developers out there. This is a clearly staggering number and means a few things.
  1. There are lots of jobs out there in Java
  2. Lots of those people are going to be rubbish or at least below average
But I think this Java groups is a good cross section of IT, it ranges from the truly talented at the stratosphere of IT down to the muppets who can't even read the APIs so write their own functions to do something like "isEmpty" on a string.

The quality of dynamic language people out there takes a similar profile (as a quick trawl of the groups will show) and here in lies the problem with IT as art. If you have Da Vinci, Monet or even someone half-way decent who trained at the Royal College of Art then the art is pretty damned fine. But would you put a paintbrush in the hand of someone who you have to tell which way round it goes and still expect the same results?

IT as art works for the talented, which means it works as long as the talented are involved. As soon as the average developer comes in it quickly degrades and turns into a hype cycle with no progress and a huge amount of bugs. The top talented people are extremely rare in application support, that is where the average people live and if you are lucky a couple of the "alright" ones.

This is why the engineering and "I don't trust you" approach to IT works well when you consider the full lifecycle of a solution or service. I want enforced contracts because I expect people to do something dumb maybe not on my project but certainly when it goes into support. I want static languages with extra-enforcement because I want to catch the stupidity as quickly as possible.

But most of all I restrict choices because the number of people talented enough to make them is vanishingly small I count myself lucky in my career because I've worked with some pretty spectacular people, but even in the best of cases it was around 10% of the project and in none of the cases did those people more into support. People will no doubt bleat that dynamic interfaces give some sort of increased flexibility, my experience however is that it just leads to a right pain in the arse which is a bitch to debug.

Unfortunately in IT the level of self-perception is not strong so far too many people think they are at the top table when in reality they should be left in the sandpit. This leads to people taking on dynamic languages, late validation, multi-threading, async and the like with a single minded belief that either
  1. they will do this because Expert X said it was the right thing to do and they worship at the altar of Expert X and even when it sucks they will say "but X says its right so you are wrong"
  2. They will do something because it looks easy and not understand the consequences
This is the reason I am against dynamic languages and all of the fan-boy parts of IT. IT is no-longer a hackers paradise populated only by people with a background in Computer Science, it is a discipline for the masses and as such the IT technologies and standards that we adopt should recognise that stopping the majority doing something stupid is the goal, because the smart guys can always cope.

IT as art can be beautiful, but IT as engineering will last.

Technorati Tags: ,


PetrolHead said...

Hmmm not sure it's engineering versus art. I certainly consider myself an engineer, I'm fact driven, I like data points to justify my choices rather than be subjective.

Maybe what we're really talking about is more akin to parenting and trust. You don't trust a 2 year old with power tools but you'd let a skilled tradesman use whatever tools and materials they prefer so long as you either:

(a) Won't need to touch it again for the rest of it's useful life.

(b) Could guarentee to have someone of equal skill around to maintain it over it's useful life.

Alas in IT we have a lot of 2 year olds, you can't trust them, you can't let them have powertools, they have no common sense and worst of all no awareness of any of their limitations. Oh and they have no concept of engineering or art they just "do stuff" on a whim and only stop (well sometimes) when Daddy shouts "No!".


Dustin Ted Whitney said...

I've been thinking about this a lot lately too, and I agree with your points, and I'd add that the IDEs for staticly typed languages are a huge benefit as well -- something dynamicly typed languages miss out on.

I've been writing a lot of Groovy code lately, and I've found myself giving all of my variables static types so Eclipse (w/groovy plug-in) can do code completion. Then I ask myself, "Why am I even writing Groovy?"

jarober said...

Sadly no - Code completion works pretty well in Smalltalk. You guys need to take your blinders off and try something different: Actually expect good work from people. If you don't expect much, you won't get much.

Steve Jones said...

I expect a good level of work from someone based on the level of their abilities. It would be pointless expecting me to paint the Mona Lisa as I'm crap at Art, but I can be trusted to paint a room.

Expecting good quality for a given ability is what you should aim for, not expect everyone to be able to produce the same work to the same quality.

The Gassy One said...

Dude you are right on! Not only that, you can back yourself up with the Normal Distribution. The sad fact is the majority of a population clusters around an average, no matter what value you want to measure. It's not Art versus Engineering, it's Mean versus Standard Deviation. As you point out, the very-skilled are few; no coincidence, they are many SDs from the mean, in the thin tail. It's the opposite of SQC; you want to be far from the mean, not close to it.

Patrick Logan said...

Groovy is, really, not much more than a toy that's been around for a few years. Let's go look at mature dynamic languages: Lisp and Smalltalk. Both of these languages have had good native code generation, IDEs, code completion, and refactoring tools for many years, even decades.

Steve Jones said...

Having used (and liked) Lisp its hard to argue that it is a language for the average developer.... Powerful? yes.... simple? No.

Anonymous said...

"But I think this Java groups is a good cross section of IT, it ranges from the truly talented at the stratosphere of IT down to the muppets who can't even read the APIs so write their own functions to do something like "isEmpty" on a string."

What are you saying here? How do you check for empty/null strings? Are the Commons guys muppets for writing StringUtils.isEmpty()?

Anonymous said...

My opinion exactly, with different words.

I've said that bondage-and-discipline languages like Java make sense for big I.T. where you can never be sure who's going to (mis)use your code,

and dynamic languages like Python are better for small teams of "the best of the best," or one.

Anonymous said...

I find your idea comparable to the strive to libertarianism (freedom, no restrictions) vs. statism (or protectionism, paternalism).

The former is better for a society of competent, independent, lawful individuals, and the latter fits the weaker and sometimes more corrupt individuals.

Since there are all kinds of populations in society, neither is the "right" choice or path to be won over. Each kind of people need to have their own way.

Anonymous said...

This static vs. dynamic typing thing is a classic case of confusing a technique with a desired result. Static typing is merely one way of quickly catching errors. There are other ways to do it, such a an automated test system combined with test-driven development, and using that doesn't require much more discipline than not declaring everything as "Object".

At least some of us using Ruby over Java are big fans of static typing; I know that I am. It's just that Java's implementation of it is so awful that we'll take the pain of debugging the problems in Ruby before the pain of dealing with Java's type system. (Well, there's also Java's inability to express a DSL, which is probably related.)

Anonymous said...

If you can't trust someone to figure out your code, what are you doing letting them touch it to begin with?

Those muppets you speak of should be kept away from the code. As far, far away as possible.

Anonymous said...

I don't rightly understand how someone can argue that Programming should be oriented towards engineering and then, in the same breath also argue that it should be capable of being achieved by people with little capacity for skilled work.

Do you think there are many Chemical or Structural Engineers that are "muppets" as you put it? If there are, I don't think they last very long.

Anonymous said...

Thanks for the interesting article, but I think the reason things look so bleak is exactly because you would currently compare a computer scientist to an artist. Programming shouldn't be an art form, and yet it is.

Further, there's nothing theoretically stopping software from being simpler to construct. We're constructing bridges with toothpicks and they are brittle, require tremendous planning and are prone to be abandoned half-finished. And it has nothing to do with that the people are not smart enough.

One indicator that I'm right is that there are no Monets; there are no Da Vinci's in software. Even the most talented people make brittle software. Think of the best software companies in the world, who hire only the best, and I will show you their published software which is brittle, required tremendous planning and was prone to be abandoned half-finished.

Steve Jones said...

David, I agree with what you say, but I would say that the quality of people at the software companies isn't always as high as you might think, mainly (IMO) because they have a pretty limited pool of talent (those willing to live on the west coast).

Anon. Structural Engineers are a good example of what I mean. The create engineering plans of elements which are then built by people who hit stuff with hammers. The talent difference between the engineer and the implementer is extremely wide, the same as it often is in IT.

I could make a joke about Chem Eng and "users" but I won't :)

Anonymous said...

I enjoyed the comment about civil and chemical engineers being "muppets" or not. It's interesting to consider: these folks are licensed for a reason: your lives are in their hands. How many programmers do you know where you'd trust your life to the correct operation of a system they'd written?

But more interesting is the misconception introduced by Steve Jones, that software development should be split into designers and implementers, much the way civil engineering is.

I don't buy this at all, because there's a very basic and key difference between software designs and civil engineering designs. The latter has a physical component: someone has to follow the specification, take the steel and put the final result together. But software designs and the final product are the same thing: a specification. One might be at a higher level than another, and all too frequently the specifications aren't actually the same, but in the end they're both specifications. The equivalent of a construction worker in the software world is the computer running your program. We have the ability to make the computer execute these higher level specifications; we just don't do that as often as we should.

control valves said...

I believe construction of such projects requires knowledge of engineering and management principles and business procedures, economics, and human behavior.

Anonymous said...

Oh boy this will never end.

People who are not able to program should NOT do it. End of the story.

Giving dummies Java instead of Ruby Python you name it is the same as giving a retarded a club instead of a gun: they're still going to hurt someone, it just takes more time and more pain for the kill.

Steve Jones said...

It isn't a question of "can programme" its a question of effectiveness. In comparison with Bill Joy then everyone's coding sucks, should everyone therefore stop? Nope, its about making people effective for their level.

Claiming that anyone worse than you shouldn't code isn't a starting point for a debate.