Wednesday, December 13, 2006

Can we please be engineers not optimists?

Via Stefan Tilkov I came across this Mark Baker article and well I'm going to set myself up on the opposite side of the fence here, in fact I'm going to go over that fence, across the field and sit in the pub watching him sink slowly into a quagmire.

One of the worst things I see in IT over and over again is the rage against formalism and verification, its just bizarre how often it happens. Now Mark might have a great new idea for time dependent evolvable validation... but for me he is wrong in principle.

First of all lets start with a the basic tenant that everyone should hold dear

KISS - Keep it Simple Stupid

And another is one of the few new Agile terms that I liked which formalised an age old saying

YAGNI - You aren't going to need it.

And before I get into it also have a look at The second System effect from the best book in IT.

When dealing with 3rd parties a basic rule of thumb is trust no-one and CYA (Cover your ass) this means verifying both what you send out as well as verifying what you receive. There are very good engineering reasons for this. Lets say that you have said that there is a limit of two items per order, as that is the business decision. If someone sends 2344 you want to reject that straight away, and schema verification is a great way to do that.

Engineering means designing a system that works for the task it is required to do, and if one of the things it needs to do is allow change then it needs to allow that change in a controlled and measured way.

The concept that Mark puts forwards of validating only at the last possible moment is somewhat beguiling as it has the idea that everything will turn out right, but if the poor bugger on the order system is expecting massive numbers then the odds are he won't have coded it expect them. This isn't their fault, if you say "its 1 or 2" then its pointless him writing loads of code to handle a million items.

Validating at the edges however does require you accept that when change happens it may break things. But the reality is that most of the time this is what you want. if you are dealing with someone who suddenly sends in a new document with a field that says "by responding to this message you are agreeing to give us all your money and your first born child" then you'd like to know.

Validate as early as possible, make sure that things are correct and hold people to account as early as possible. I've seen companies lose literally millions of pounds (billions of dollars :) because they didn't enforce interface contracts effectively, this led to problems down the line and systems that were very hard to untangle.

There are an awful lot of stupid people out there and it is your job to protect your systems and your business from their stupidity, this means not trusting what they sent you and checking it yourself for validity. Optimism is not a good engineering trait, pessismism is. That is why you validate what you send to make sure it doesn't come back on you, and validate what you receive to make sure the other person isn't an idiot.

To use a physical engineering analogy, you check that the wing is put on properly as soon as it is attached, not by looking out of the window at 30,000ft to see if it is still there.

Technorati Tags: ,


Mark said...

Ok. I don't doubt that there are cases where early validation is the best approach, I just don't think that's a general approach to be used on the Web.

I'd be interested to hear how you'd support independent evolvability. Or are you happy to just sacrifice that in general?

Steve Jones said...

I'm not sure why you think that a firm and enforced contract stops independent evolution. For me it clearly bounds that evolution and details what is valid and what will be considered invalid. Having a fixed contract when combined with elements such as mediation and industry standard practices of version & release enable systems to regularly evolve independently today.

Put it this way you wouldn't recommend on the web that people just accept any packets into their browser that are not compliant with HTTP on the basis that this allows the server to "evolve" its own communication protocol. Having rigid standards(which is part of the power of REST) that have specific meanings can in fact add significant flexibility to systems as it enables effective evolution.

Bill de hOra said...

"I'd be interested to hear how you'd support independent evolvability."

This is the nub of the problem: expected outcome is this people will talk past each for the next month.

If two parties sign up for a contract, by definition they don't want independent evolvability on that axis. They've signed a waiver that results in (albeit point-to-point) uniformity.

Now take HTTP - independent evolvability is an important backing principle of the protocol, but not of methods - that axis is closed, and rightly so.

Hence I think I'm disagreeing with both of you..

Steve Jones said...

I don't think I am disagreeing with you Bill, what I'm saying is that tight specifications can actually add flexibility to systems, specifically because they restrict certain choices (or axis). Engineers are very good at adapting to tight specifications and using them for new purposes.

I personally don't see how having clear constraints prohibits independent evolution. Its the a major idea of engineering to define boundaries (and their flexibility) that enables different entities to exist and evolve independently.

Mark said...

Hmm, I don't think that's the case, Bill.

Independent evolvability is a concern anytime there's independent parties ... even in the situation you describe because either the contracts change, or how they're used does (and the data exchanged as part of the contract), and coordinating software upgrades is a pain the ass any way you slice it.

It's true that the fewer the number of parties to the contract the easier it will be to absorb, but it's never desirable to be coupled in this way even if it's bareable in some cases.

Steve - I don't believe all "firm and enforced" contracts stop independent evolution. The devil's in the details of the "enforced". Please have a look at the scenario in my blog post and describe an alternate solution to that problem.

Steve Jones said...


Your scenario is that a range value is increased between A and B, well given that A can't handle the new range... you don't actually say what will happen to A if it accepts that value (it might format the hard-drive for all I know!). If you are saying that client A is set up to accept a broader range of values but B can only dispatch a minimal set in the first iteration... well I'd suggest there that you specify the range to be what the B/A interaction should handle. This is just the same as designing a Radar system to cope with Mach 6 planes even when there are currently no Mach 6 planes in operation but they are expected to exist in the near lifetime of the system.

This is of course when the consumer can accept more than the producer can create. In the reverse situation then the consumer must perform filtering to accept that information which it can handle, again schema validation can be a good way to do this.

Anonymous said...

Its a very nice blog for...
architects in bangalore , architects in bangalore , interior designers in Bangalore , interior designers in Bangalore , architects in bangalore , architects in bangalore , interior designers in bangalore