What with the hype of Web 2.0 I thought I'd make a bold, easy to prove wrong prediction about what will be talked about as Web Services 2.0, namely semantic Web Services.
Semantic Web Services, for instance the WSDL-S submission to W3C aim to describe the web in terms of a series of ontologies ideally in a standardised ontology language these can then be mapped, potentially automatically from one form to another to create a richly described environment that is not limited, or based on, a basic WSDL description. The ontology could include all manner of details, restrictions and rich information about itself that would help in the discovery and implementation processes.
This of course is the theory. The problem is that the tools are currently pretty basic, and are based around an assumption that I'm not sure applies that often in the enterprise, namely that I don't really know what I'm calling. While having a single enterprise data model is always a non-starter, its not right to then assume that all information can be defined in any old way and a rich ontology is required to help drive the mapping process. Most integration projects have got by fine with what they have at the moment and the amount of definition will really have to justify itself to get used.
Semantic Web Services might be the killer app for Web Services 2.0 and create a rich environment by which invocations are automatically mapped using sexy mathematical and semantic technologies.
Or of course you might still want to make sure that your PO is mapped to the right field on the invoice and do that via a simple phone call. And this is the challenge for me on how Semantic Web Services will fly in the main. Some of the technology will definitely be useful (they talk about pre-conditions and effects, but only in terms of the data not functional verification) but given that mappings are irregular elements, are are liable to stay so while companies want to know who they are dealing with, will semantics really give a real benefit for the definite increase in complexity that they bring.
If Web Services 2.0 really is about the Semantic Web then its going to get a whole lot more scary out there. The joy of Web Services 1.0 was that SOAP meant SIMPLE, with WS-* its got more complex, and with Semantics its going to get very complex indeed.
Sunday, February 26, 2006
Contracts: Easy in Java, hard in WebServices
With RC1 of Obligate its worth quickly explaining why something that was done in pretty short time in Java is proving really hard to do in Web Services.
There are as I see it three main problems
There are as I see it three main problems
- WebServices are an interface specification only, contracts are often about logic
- There are lots of choices about how to do it, none of which do everything
- Many of the specification people don't work on end-user problems
The first is a major challenge. When we wanted to do obligate we just downloaded GroovyRules and used JSR 94, a little bit of Java SE 5 annotations and it was done.
With Web Services its much more complex, you could apply an XSLT to do increased data restrictions, use WSDL-S (Semantics) and OWL files to do some of the richer comparisons between data like comparing fields and the like. And then you could tie in BPEL to do the functional preconditions that need to be done. Its not quite so easy really is it? And to me this is part of the challenge raised by point 3. When people are using Web Services and talking about governance they want to do the sort of simple business things like "not between 2am and 5am" or "Must have valid PO" which require a call to another functional service or system facility.
What we need is a simple way to define these things, via tools, where the massive complexity on what needs to be done is nicely hidden from business. It also needs to be applied in a consistent manner, such as WS-Policy, and definitely not in the WSDL file itself.
Obligate RC1
Well its time to go to RC1 with Obligate which is a Java annotations based solution to the Design by Contract problem. Its a bit invasive in this first edition, but can be used either directly as a service locator approach, or with a bit of code or generation, ontop of EJBs. As soon as we get a decent EJB 3.0 environment we will be adding EJB 3.0 support, and when we have an app server that does Java EE on Java SE 5.0 with decent EJB generation then we'll add in support for that.
Performance isn't currently obligate's strength, although on even a basic laptop it can do 200,000 calls with full contracts in under 400 seconds, which means 2 milliseconds per call. The aim is that this should be used in pre-production and test environments, paticularly if attempting to do continual live environments.
So if you are feeling brave have a look, there is currently a debate going on in the Java SE 6 group on whether parameter name reflection should be added to the reflection APIs, obligate has a decent example of where it would help.
Performance isn't currently obligate's strength, although on even a basic laptop it can do 200,000 calls with full contracts in under 400 seconds, which means 2 milliseconds per call. The aim is that this should be used in pre-production and test environments, paticularly if attempting to do continual live environments.
So if you are feeling brave have a look, there is currently a debate going on in the Java SE 6 group on whether parameter name reflection should be added to the reflection APIs, obligate has a decent example of where it would help.
Tuesday, February 21, 2006
Federated Data, backup and recovery, or how to recover from a BSOD
One impact of more powerful desktops and things like Windows Workflow Foundation (WWF) or even the ability to run "lightweight" application servers on desktops is that information is going to become more and more federated with challenges around data-synchronisation increasing ever further. Service state (if using WWF for example this would be business process state) will become distributed around the organisation, not always stored on "nice" central databases that can be backed up centrally. This means that backup and recovery, and potentially encryption, of this information is going to become more important and of course more challenging.
Taking an example from myself today and using email as the semi-critical information. I take regular (once a week) backups of my files, but today the following happened: I copied all my emails off the Exchange server into local folders (business state local) and then installed a new piece of software (a firewall) that promptly fried the machine resulting in a BSOD, and a failure of Windows Safe mode to start. This meant that I'd just lost the "state" of the last 2 days, which is a bit of a pity as that includes where I'm meant to be tomorrow and some emails that I could do with still having. Now my backup could take me back to Saturday, but this would mean (in our Service analogy) having some pretty complex compensation logic in the center to cope with "lost" service and process state. The requirement therefore was to access the disk itself to get the raw (but encrypted) information for re-entry into the last saved state.
If you are looking at future challenges for your enterprise look no further than having services and business process living across the enterprise across multiple application environments, many of them under direct user control. Architecturally this is fine in a decent service architecture, but for implementation it is not the sort of area to get into lightly. Distributing services across the network and to users makes a lot of sense, but the challenge of federated data and federated context as well as the pure technical challenges of invoking user centric and specific services are one area that organisations need to plan heavily for before making the leap.
As an aside the solution was to get a Linux Live CD (Slax) and copy the elements onto a USB key, this was after Windows Recovery also failed to help.
Taking an example from myself today and using email as the semi-critical information. I take regular (once a week) backups of my files, but today the following happened: I copied all my emails off the Exchange server into local folders (business state local) and then installed a new piece of software (a firewall) that promptly fried the machine resulting in a BSOD, and a failure of Windows Safe mode to start. This meant that I'd just lost the "state" of the last 2 days, which is a bit of a pity as that includes where I'm meant to be tomorrow and some emails that I could do with still having. Now my backup could take me back to Saturday, but this would mean (in our Service analogy) having some pretty complex compensation logic in the center to cope with "lost" service and process state. The requirement therefore was to access the disk itself to get the raw (but encrypted) information for re-entry into the last saved state.
If you are looking at future challenges for your enterprise look no further than having services and business process living across the enterprise across multiple application environments, many of them under direct user control. Architecturally this is fine in a decent service architecture, but for implementation it is not the sort of area to get into lightly. Distributing services across the network and to users makes a lot of sense, but the challenge of federated data and federated context as well as the pure technical challenges of invoking user centric and specific services are one area that organisations need to plan heavily for before making the leap.
As an aside the solution was to get a Linux Live CD (Slax) and copy the elements onto a USB key, this was after Windows Recovery also failed to help.
Monday, February 20, 2006
Multi-core problems for all
One element of service architectures, paticularly when dealing with asychronous processing and multiple requests, that has been a common challenge has been coping with the issues of concurrency. Now with the rise of multi-core this wonder is open to all, and with it the challenges. For anyone doing Java there is of course the Doug Lea book Concurrent Programming in Java, which is a must for anyone trying out multi-threaded code. The problem is that multi-threaded code is hard, not just difficult its actually hard and many of the problems only occur when you get an increased level of concurrency, and sometimes only when you get truly concurrent execution. Multi-core enables this problem to be found by all.
First of all lets think of the challenge, right now dual core has become normal. Next year it will be quad core, so assuming your application has a 7 year life expectancy (at least) this means 2013, by which time the "normal" will probably be 32 core. This means that process engines that don't handle asychronous processing well are going to have scalability issues in the future as doing multiple tasks at the same time and then collating the results is going to be a much more effective approach than simple straight through processing.
Having 32 cores (although with scheduling the OS will probably see 4 times as many) means that when planning the desktop applications in your enterprise architecture you'll have to think about multi-threading in applications that have historically been single threaded, or at least assumed there was only one processor.
As a final attempt to put into perspective how much of a challenge this is for companies. Microsoft's current top line operating system is Windows Server DataCenter edition 64bit this has a maximum of 64 processor support. So in our 2013 world this won't run on a laptop which with 32 cores each reporting 4 threads needs 128 way support in the OS. This flagship OS has a fairly frightening shelf-life in a multi-core world (assuming 4 threads per core (ala Sun) becomes standard by 2009):
First of all lets think of the challenge, right now dual core has become normal. Next year it will be quad core, so assuming your application has a 7 year life expectancy (at least) this means 2013, by which time the "normal" will probably be 32 core. This means that process engines that don't handle asychronous processing well are going to have scalability issues in the future as doing multiple tasks at the same time and then collating the results is going to be a much more effective approach than simple straight through processing.
Having 32 cores (although with scheduling the OS will probably see 4 times as many) means that when planning the desktop applications in your enterprise architecture you'll have to think about multi-threading in applications that have historically been single threaded, or at least assumed there was only one processor.
As a final attempt to put into perspective how much of a challenge this is for companies. Microsoft's current top line operating system is Windows Server DataCenter edition 64bit this has a maximum of 64 processor support. So in our 2013 world this won't run on a laptop which with 32 cores each reporting 4 threads needs 128 way support in the OS. This flagship OS has a fairly frightening shelf-life in a multi-core world (assuming 4 threads per core (ala Sun) becomes standard by 2009):
- 2013 won't run on a 1-way box
- 2011 maxed on a 1-way box, won't run on a 2-way
- 2009 maxed on a 2-way box, won't run on a 4-way
- 2007 (2 threads) maxed on an 8-way
I'm guessing that Vista will be addressing this, and I also hope they'll be upgrading their concurrency support in the .NET languages.
But simply put, having 128 way laptops is something that is liable to happen in the next ten years, and if all of your code whether in a J2EE container or on a desktop application isn't designed to take advantage of it you'll effectively be running single core code and wasting 127 processors.
Saturday, February 11, 2006
Is everyone on the BPEL bus?
Now that BPEL 2.0 is racing towards standardisation and IBM and SAP have led the way on human workflow with BPEL4People with Oracle apparently closely following the specification and helping to display some of the myths out there and clearly looking to put BPEL at the heart of its product suites. If you look at the BPEL 2.0 group all of the vendors are clearly there, how enaged they are is clearly another question (628 posts involving sun, 15,000 with IBM, 14,000 with BEA, 271 for SAP, 14,100 for Microsoft and 810 for Oracle) but part of the question is how commited to BPEL 2.0 these vendors will be and when we can expect to see BPEL 2.0, and potentially BPEL4People within their product suites.
In a hugely unscientific poll of the various vendors websites we get 22 hits for IBM, a single Blog entry at Sun saying that BPEL 2.0 is the first "open" BPEL, and for the rest BEA has quite a few articles out there, SAP don't have much (two articles) but then that is pretty normal for SAP on technology elements, they've been working with IBM on BPEL4People though, Oracle are much the same with just three hits, but a decent forum thread or two. And then there is Microsoft, with an impressive ZERO hits which is pretty staggering considering they are the co-chair of the group (I tried again on MSN to no avail).
So from the outside it looks like Microsoft are lagging behind on BPEL while pushing XAML, their own interal proprietary schema approach in the upcoming "Windows Workflow Foundation", something that was originally aimed at dynamically creating GUIs.
Clearly this can all change in the next few months as MS begin to announce what is in BizTalk next, and the J2EE based vendors announce their roadmaps. Certainly as a recommendation I'd say companies should be looking to standardise on BPEL 2.0, and whatever becomes the human workflow standard extension, over the next couple of years, it would be a real shame if some vendors left themselves at the bus stop with a single platform solution.
In a hugely unscientific poll of the various vendors websites we get 22 hits for IBM, a single Blog entry at Sun saying that BPEL 2.0 is the first "open" BPEL, and for the rest BEA has quite a few articles out there, SAP don't have much (two articles) but then that is pretty normal for SAP on technology elements, they've been working with IBM on BPEL4People though, Oracle are much the same with just three hits, but a decent forum thread or two. And then there is Microsoft, with an impressive ZERO hits which is pretty staggering considering they are the co-chair of the group (I tried again on MSN to no avail).
So from the outside it looks like Microsoft are lagging behind on BPEL while pushing XAML, their own interal proprietary schema approach in the upcoming "Windows Workflow Foundation", something that was originally aimed at dynamically creating GUIs.
Clearly this can all change in the next few months as MS begin to announce what is in BizTalk next, and the J2EE based vendors announce their roadmaps. Certainly as a recommendation I'd say companies should be looking to standardise on BPEL 2.0, and whatever becomes the human workflow standard extension, over the next couple of years, it would be a real shame if some vendors left themselves at the bus stop with a single platform solution.
Sunday, February 05, 2006
Snowboarding SOA
I've just spent rather a nice weekend in Whistler after meeting with Duane, chair of the SOA-RM group, and clearly I'm thinking too much about Services because to me a resort is great example of non-IT SOA in action.
Firstly of course you have the whole domain, the overall service that is being provided. Within this there are two dependent services that are offered, namely lifts and runs. Runs come in two flavours namely "on" and "off-piste". Lifts come in a few flavours but all contain a number of identical sub-services, for example the chairs. People can also bring their own capabilities which include the ability to hike between (or most often out of) runs. The lifts link with different runs, and runs link to different lifts, creating dependencies between the services (for instance there are a whole heap of runs that rely on the Harmony Chair being open).
The consumers of the lifts and pistes have a certain expectation of service (does the Emerald Chair always have those problems?) and have an SLA which for the lifts is the amount of people it can shift up the mountain. You pay a subscription to the resort in order to use the various services, at some places this may limit the number of service instances (lifts) you can access. In general the resort, its lifts and the runs are exactly the services that boarders come to enjoy.
Whistler... its the SOA boarding venue of choice.
Firstly of course you have the whole domain, the overall service that is being provided. Within this there are two dependent services that are offered, namely lifts and runs. Runs come in two flavours namely "on" and "off-piste". Lifts come in a few flavours but all contain a number of identical sub-services, for example the chairs. People can also bring their own capabilities which include the ability to hike between (or most often out of) runs. The lifts link with different runs, and runs link to different lifts, creating dependencies between the services (for instance there are a whole heap of runs that rely on the Harmony Chair being open).
The consumers of the lifts and pistes have a certain expectation of service (does the Emerald Chair always have those problems?) and have an SLA which for the lifts is the amount of people it can shift up the mountain. You pay a subscription to the resort in order to use the various services, at some places this may limit the number of service instances (lifts) you can access. In general the resort, its lifts and the runs are exactly the services that boarders come to enjoy.
Whistler... its the SOA boarding venue of choice.
Thursday, February 02, 2006
Continual Live - moving continuous integration on a step
Another, more technical, element that SOA can enable is that of "continual live" which builds on the concepts of continuous integration but moves more towards a continual simulation and system test approach rather than focusing on unit testing. The basic principles of Continual Live are pretty simple
The contract element is the trickiest, its something we've had a crack at with Obligate but that is just for Java and its early days, as there aren't any real approaches for that so 2 is currently an optional step for most projects. You do still get value from doing continual live.
This means that once you have defined all your services you are able to deploy the whole application to a server and run some system tests. Depending on how you set up the mock implementations these tests could all pass or all fail, most likely (if you've written decent tests) some will pass and others will fail. Assuming that you write decent tests this means that over time as services are implemented the number of failed cases is reduced. Because services are actually calling other services this means that the tests are more system test in nature rather than being unit test, this approach also means that as you start putting screens on the system you don't have to stub out calls that "haven't been implemented yet" you can write the code as required and demonstrate a running environment much earlier in a project.
Tools like cruise control and ant can be easily extended to support a continual live approach, the key change is that instead of just writing unit tests (which should continue, especially when you don't have contracts) you can also write system tests. If you are using a contract based environment then unit tests become less required and its actually the system tests that matter more, as the contracts will enforce the validity of operation of the service.
So basically, Continual Live is about getting a "running" system up as early as possible to help identify inter-service problems based on actual invocation and to track the evolution of the system. If combined with contracts it provides a pretty cheap way to build very robust systems and identify clearly who is causing the problems.
- Define Service Interfaces first
- Define a formal contract for the service interface
- Implement a Mock implementation with explicit known behaviour
The contract element is the trickiest, its something we've had a crack at with Obligate but that is just for Java and its early days, as there aren't any real approaches for that so 2 is currently an optional step for most projects. You do still get value from doing continual live.
This means that once you have defined all your services you are able to deploy the whole application to a server and run some system tests. Depending on how you set up the mock implementations these tests could all pass or all fail, most likely (if you've written decent tests) some will pass and others will fail. Assuming that you write decent tests this means that over time as services are implemented the number of failed cases is reduced. Because services are actually calling other services this means that the tests are more system test in nature rather than being unit test, this approach also means that as you start putting screens on the system you don't have to stub out calls that "haven't been implemented yet" you can write the code as required and demonstrate a running environment much earlier in a project.
Tools like cruise control and ant can be easily extended to support a continual live approach, the key change is that instead of just writing unit tests (which should continue, especially when you don't have contracts) you can also write system tests. If you are using a contract based environment then unit tests become less required and its actually the system tests that matter more, as the contracts will enforce the validity of operation of the service.
So basically, Continual Live is about getting a "running" system up as early as possible to help identify inter-service problems based on actual invocation and to track the evolution of the system. If combined with contracts it provides a pretty cheap way to build very robust systems and identify clearly who is causing the problems.
Service Oriented Realisation - how SOA impacts projects
One area that is often overlooked, or (IMO) done badly, is how project management and project structures are able to change in an SOA environment. SOA can be done in a "traditional" way, even a waterfall way, but this doesn't take advantage of the power of SOA.
The key element that SOA enables for projects is that it can turn them into programmes. This means that instead of considering a project as a bunch of people delivering on requirements, its actually a bunch of service delivery teams working within a programme. On large scale projects this sort of thinking is required, but on a larger scale. The advantage of SOA is that it can help to reduce the amount of communication require in a programme, it defines the boundaries outside of which people really shouldn't care, and helps to identify (but as Dan Creswell points out not really define) the dependencies between services.
A key benefit of this is the ability to target your crack teams at the hard work and target the majority of the developers at the simpler elements. It also means that as a project manager (now programme manager) you are able to better identify issues that are going to have wider impacts and gives a structure in which these elements can be spotted earlier on.
As an example, if a project has 100 use cases you start developing and you can say that UC 20 and 40 are going to be late but you tend to split the work up by use case (a functional thread), if those 100 use cases are implemented by 10 services then you can say that Service 2 has two elements that are going to be late. Because you've defined the interfaces to the services upfront you've also mitigated the impact of that delay onto other areas of the programme.
SOA enables Service Oriented Realisation (SOR, because lets face it Service Oriented Delivery of IT is never going to catch on) which is about changing away from delivery of individual requirements to the delivery of multiple projects which each aim to deliver sets of requirements. This is a massive shift for most project managers and its going to be interesting to see how many SOA projects continue to be delivered using waterfall and standard requirements driven iterative approaches.
The key element that SOA enables for projects is that it can turn them into programmes. This means that instead of considering a project as a bunch of people delivering on requirements, its actually a bunch of service delivery teams working within a programme. On large scale projects this sort of thinking is required, but on a larger scale. The advantage of SOA is that it can help to reduce the amount of communication require in a programme, it defines the boundaries outside of which people really shouldn't care, and helps to identify (but as Dan Creswell points out not really define) the dependencies between services.
A key benefit of this is the ability to target your crack teams at the hard work and target the majority of the developers at the simpler elements. It also means that as a project manager (now programme manager) you are able to better identify issues that are going to have wider impacts and gives a structure in which these elements can be spotted earlier on.
As an example, if a project has 100 use cases you start developing and you can say that UC 20 and 40 are going to be late but you tend to split the work up by use case (a functional thread), if those 100 use cases are implemented by 10 services then you can say that Service 2 has two elements that are going to be late. Because you've defined the interfaces to the services upfront you've also mitigated the impact of that delay onto other areas of the programme.
SOA enables Service Oriented Realisation (SOR, because lets face it Service Oriented Delivery of IT is never going to catch on) which is about changing away from delivery of individual requirements to the delivery of multiple projects which each aim to deliver sets of requirements. This is a massive shift for most project managers and its going to be interesting to see how many SOA projects continue to be delivered using waterfall and standard requirements driven iterative approaches.
Subscribe to:
Posts (Atom)