What is striking about the WS-Empire striking back (borrowing from Mark’s post) is not the attempt to "standardize" certain new features for SOAP messages, but the apparent lack (or break-down) of a cohesive guiding principle for architecting SOAP-based web services. I’m referring to WS-ResourceTransfer here.

Let me get to the basics. According to SOAP 1.2,

SOAP Version 1.2 is a lightweight protocol intended for exchanging structured information in a decentralized, distributed environment. … using XML technologies, an extensible messaging framework containing a message construct that can be exchanged over a variety of underlying protocols.

If I buy into this model, I would take it that the message describes everything – the operation/verb, the data, and any additional instructions to process the data.

But if we look at WS-ResourceTransfer, it says that

it intends to form an essential core component of a unified resource access protocol for the Web services space…

and then goes on to define a standardized technique for accessing resources using semantics familiar to those in the system management domain: get, put, create and delete.

What? One of the strongest arguments in the REST vs WS-* debate so far has been that a uniform interface (as in REST) is not adequate to describe the varieties of operations that enterprises need to do (with WS-*). But then why attempt to define a uniform interface familiar to those in the system management domain when the WS-* side concluded that uniform interfaces don’t meet the needs? Looks like a resource access spin.

Yesterday, I was tipped by one of my ex-colleagues at BEA that both JSR-286 and WSRP 2.0 have been finalized and submitted for voting by JCP and OASIS respectively. I was an active contributor to both these specifications till mid 2007. Congratulations to both the working groups!

To quote HighScalability on Scaling Twitter,

In other words, expect your users to be as creative as you are (if note more), and let them compete to build interesting applications around your software.

Recently, there have been a number of blog posts and articles on this topic, and this time around, the REST side is winning, and I find two reasons for this shift. Firstly, HTTP is more ubiquitous than ever. Secondly, WS-* has reached a point where the complexity of specifications and related abstractions overtook the basics. This recent shift reminds me of the history of CORBA, and how it had its glory and then quickly lost it.

Looking at WSDL, SOAP, and related base WS specifications, the basic problem they try to solve is to keep the interfaces and messages independent of how messages are transported from one point to another. This kind of transport agnosticism is set in stone in some of the base specs. Most implementors of web services followed suite, and architected the core runtime such that “transport” is a last minute detail that can be changed at will without changing the semantics of application. Given that the goal is to keep the application independent of the transport, some of the basic ideas in SOAP and WSDL are not unnatural, although that does not excuse the proponents from the complexity and unparsable language introduced in some of the specs.

The trouble is that, the folks who designed WS standards and implementations solely for transport agnostic scenarios ended up selling the technology as a means to build loosely coupled distributed applications. With that, folks came up with explanations for why SOAP has headers, why SOAP needs additional security specifications, why you need special purpose web service proxies, and so on. This silliness went to the extent that some folks started circulating FUD that you can’t get the right “ilities” with REST. This kind of pitch works as long as the majority of applications are talking to each other over diverse transports, but that is not the reality today.

The reality is that, HTTP is more predominant today than ever. When the transport is HTTP, it makes more sense to design the interface to take advantage of the transport to the extent possible, and not try to hide the transport. With this approach, there is no need to bend over backwards and invent crazy specifications for things that could be solved very simply with HTTP. I find that this a bigger reason for the recent shift towards REST over WS. This shift may have less to do with the core principles of REST.

This change is somewhat similar to what happened to CORBA. When CORBA was designed, it was meant to solve a similar problem – i.e. to keep applications written in heterogeneous languages talk to each other. But then, it was sold as a general purpose distributed object technology, and ended up becoming uber-complex, but complexity can not be sold for ever. That is precisely what is happening to WS today.

The discussion at Dare Obasanjo’s blog, and the references to some other recent posts motivate me to look at some of the broader arguments being made. It is clear that processing JSON data using JavaScript is trivial, and that is what makes JSON a popular alternative over XML currently. All other advantages and disadvantages over XML are incidental. If we take out JavaScript from the picture, JSON is not a clear winner. Both JSON and XML take some effort to parse and process outside JavaScript.

Unfortunately, some in the industry are missing this point, and concluding that JSON is a clear winner over XML for other reasons such as being a simpler format, faster and so on. For instance, in his post JSON and XML, Tim Bray seems to be favoring JSON because “JSON’s single prpose design is to put struts on the wire” That’s not why web developers favor JSON over XML. It is the complexity of processing on the browser, and JSON makes it trivial.

I tend to agree with Don Box, who commented that “JSON has a huge hill to climb to displace XML for new applications that don’t already have a JavaScript/browser component”. We need some more serious efforts to simplify JSON creating/parsing in languages other than JavaScript to make JSON more general-purpose than it is today. Although there is no information that can be represented in an XML document that cannot be represented in a JSON document, it does not mean that all new applications should switch from XML to JSON. For now, if a browser is the consumer of the data, consider JSON over XML.

Does validation mean that interface and implementation are tightly coupled, making the service hard to maintain? Mark Baker posted some more questions on validation yesterday. Validation does have merits in some cases, and the bigger challenge is designing extensible interfaces.

His first comment is that

For an outsider/client, it does not matter whether a service failed due to validation error or some other failure. All that the client wants to know is whether the message was processed, and if not, why not. What difference does it make whether a scheme validator failed because “ABC123” is not a valid 5-digit number or some code block failed because it needs a valid ZIP code to proceed? In that sense, what is the difference between the a structural/value constraint in a schema or some error generated in code.

Another way to look at this is in terms of preconditions and post-conditions. As part of the interface design, it is sensible to express what the preconditions are and what the post-conditions are. Preconditions tell the client what needs to happen for the software to be able to process a request. Agreed that schema languages do not have the smartness to represent all possible preconditions, but they can simplify the effort required to implement certain preconditions in code. Let’s say, the software need to process a purchase order, and certain pieces of data are required to make a purchase order meaningful – for example, it may need certain product descriptions, some contact info etc. To verify these preconditions, the software needs to check that these elements are supplied, and that they make sense (i.e. the software understands those product descriptions and contact info). In most cases, it is possible to express these preconditions in a schema language, and let a validator assert some of the preconditions. This saves time and effort. As I argued in my previous post, upon a validation error, whether to fail immediately or continue is a use-case specific.

There is one particular case where I would consider validation dangerous. It some cases the same software may be described by multiple interfaces. In those case, it is better to keep the software itself decoupled from the interface description, i.e. from the schema language used for describing the interface, and verify the preconditions within code and not rely on a validator.

Preconditions are important to make an interface description useful. The key challenge here is specifying the preconditions such that they are extensible and not set in stone. The current version of the software may only be able to process locations expressed as <address><street></street><city></city></address>, but the next version may also be able to process a “street, city, zip” format, or even geo-coordinates. As the software gets smarter, it needs to be able to process more kinds of messages without significantly changing the interface itself. That is where the schema languages are lacking. It is true that schemas can be used to address extensibility – but in reality – the solutions are painful to implement – like versioning a schema, introducing extension points and so on. The must-ignore rule is easy enough to implement, but the rest of the schema extensibility solutions are not.

Sanjeeva Weerawarana of Axis-fame gave a candid and somewhat introspective talk on Web Services Middleware a few weeks ago at Google. The talk was mostly about the history behind SOAP and WSDL, about how Axis1 was designed, how Axis2 is radically different, and the grand scheme of things around Axis. A video of this talk is available at http://video.google.com. This is a talk worth listening to. Sanjeeva does a good job at summarizing things around SOAP, WSDL and Axs. Based on his overview, and my own limited experience, I think Axis2 is based on a good architecture, but it may have come late to the field, and with lots of bugs and compatibility problems.

It is amusing to watch one of the SOAP-proponents admit that “letting a Java class be turned into a web service” was a big mistake. I must congratulate him for being so candid about it. He also points out that every web services stack worth a name now follows the same paradigm – write a Java class, click a few buttons, and out comes a web service – to become part of a grand SOA soup. Of course, this approach demos well – marketing folks love this kind of stuff. JSR 181 takes it one step further by letting you add Java annotations to Java classes to turn them into a web service. I bet this class-first approach will turn out into a nightmare once the service is put into serious use. Well – it is too late.

Coming back to Axis, IMO, Axis 1.x was good enough for most applications. It promotes the class first approach, has a built in Java-XML binding framework, and works reasonably well. It was widely used, and most bugs got fixed over time. May be having Java-XML binding at the core was a mistake – but who cares – most developers write code using the generated beans anyway. Apparently this was not good enough for some, and Axis had to be rewritten.

I’m not a big fan of rewriting software for architectural reasons. Just because the original version was successful, it is naive to assume that the rewrite will be equally, if not more, successful. Moreover, rewrites usually break feature, API and bug compatibility – and Axis2 did that completely.

Here is my personal experiece with Axis. Our team at work uses Axis to drive some web services for testing our product. Our experience with Axis versions upto 1.3 was satisfactory. We did generate POJOs from a WSDL and schemas, and wrote test clients using the generated code. Then came a new version of the WSDL, and Axis 1.3 could not handle that. We ran into some serialization bugs, and so wanted to try 1.4. No luck with that either. Then we tried the all new Axis2. What a disappointment? It is completely incompatible with Axis 1.x at the source level itself. Moreover, it had new kinds of feature incompatibilities and bugs forcing us to hack it to make it work. Another consequence of this rewrite exercise is that bugs filed against Axis 1.x now go into the use new version please void.

To give a more concrete example, Axis 1 had a nifty little feature to let the stub instance replay cookie headers between requests. You just call setMaintainSession(true) on the stub instance. Internally, this class makes the stub capture all Set-Cookie headers from responses and send Cookie headers on future requests. You might ask why would I want to care about transport headers while writing web services. Well, web services is a leaky abstraction, and protocol agnosticism does not work in reality.

Axis2 supposedly has an equivalent of setMaintainSession(true. The equivalent is to do something like

Options options = new Options();
options.setManageSession(true);
ServiceClient sender = new ServiceClient();
sender.setOptions(options);

But this turned out be more complex than I originally thought. This was not a replacement at all. Instead of capture any cookie, Axis2 looks for some internal cookie named axis_session. I suppose that this is the name of the cookie that Axis2 may set on the server side. I was more horrified to see some WS-Addressing related code for session management. I have not spent enough time to make it work, but I came across some code using WS-Addressing for session management in Axis2. This is also confirmed by this article Axis2 Session Management.

There are few more stories about feature incompatibility. See [axis2] a little frustrated for example. (Thanks to my colleague Nate Lipke for forwarding this link.)

One last comment about Axis2. Right now, it does not implement JAX-WS and related specs, making it even harder for JEE fans to consider using Axis2. I wonder why Axis user community (including some corporations who built their products on top of Axis) did not speak up against the rewrite idea. Oh well – repeating a success story is such a bad idea.

P.S.: Just to spice your reading experience up a bit, also check out The S stands for Simple.

Carlos Perez at manageability.org posted an interesting discussion on OO-styled interactions, and I can’t agree more.

It is very common to find most people listing statelessness as one of the key principles of service API design. Statelessness is an overloaded term. To my experience, when someone says that a service is stateful, he/she usually means that the service maintains some internal state. We can argue all day long on whether maintaing state is good or bad – the answer usually depends on what the service is doing. In most cases, whether a service needs to maintain state or not is a service implementation choice – service users do not need to know or care about it.

From the client’s point of view, statefulness of a service usually implies that interactions with the service need to happen in some sequence to be meaningful, i.e. the client maintains some context with the service it is interacting with. Context is what a client should care about – not about whether the service is stateful or stateless.

Shared context makes service invocation interesting and meaningful. The service needs to define the context which may then be updated by the service or the client, and may even include references to any internal state maintained by the service itself. Services that don’t require any client-managed context are usually general-purpose utility services. Most usuful services need to depend on some share context to be useful. Context is what makes it possible to break a task into sub-tasks and let the client invoke sub-tasks in some defined sequence.

Some designers may prefer combining shared context with the inputs and outputs of service invocations. IMO, keeping the context separate from the data being processed and returned makes the service interface definition lot more cleaner, easier to understand, and extend. One problem with most programming models used for implementing services is that they don’t provide for shared context. For instance, take stateful EJBs. They are becoming extinct (rightly so) because they emphasize on the stateful nature and not on the shared context. With stateful EJBs, sharing the context meant that the client held onto the bean till the work is done. That is not the same as the shared context. Unfortunately, later layers like Java web services extended the same notion, and don’t provide for shared context within the supported programming models. You can easily bind a web service implementation to a class or a EJB, but you need to invent your own mechanisms for supporting shared context. Then there is the Web Servces Composite Application Framework, but I don’t think that is what service developers really want to worry about.

This also reminds me of The Need for Shared Context by Anne Thomas Manes, which argued for the need for defining a shared context.

Another harmful aspect OO-styled interactions that Carlos did not mention is idempotency. One of the problems with object-oriented development is that it is difficult to tell whether a given service invocation is idempotent or not.

This post is in response to Sameer Tyagi’s article on using JAX-WS for RESTful web services posted at the Sun Developer Network. I find a few points stated in this article troubling if not inaccurate – (a) it presents REST in a limited scope thereby showing SOAP in better light, and (b) it misses some key differences between REST and SOAP based web services. Beyond this, the example cited in this article just shows why you SHOULD NOT use JAX-WS for REST style web services. In this post, I would like to discuss the whys behind the two statements I just made.

Let me first list each point mentioned about when to use REST and when to use SOAP, and comment on each. The statements in italics below are from this article.

1. Use REST when the web services are completely statelesss.

   This article contents that REST is suited for stateless services. To the contrary, most people prefer keeping SOAP style web services stateless. I also find it easier to manage state over REST based services than SOAP based services. With REST based services, you can either manage state with cookies or with the state encoded in the URI. With SOAP based services, the only option is to encode the state within the envelope.

2. Use REST when bandwidth is particularly important and needs to be limited.

   REST based web services may be less-bloated. That may or may not turn into better performance – it depends on the network and the latency.

3. Use REST for web service delivery or aggregation into existing web sites … using technologies such as Asynchronous JavaScript with XML (AJAX).

   This point is a stretch for two reasons. It is equally possible to invoke SOAP style services via XMLHttpRequest. There are pros and cons to each approach. Secondly, I would not use JAX-WS for driving Ajax responses as using JSON over XML has several advantages and JAX-WS is not meant for anything other than XML.

4. Use REST when the service producer and service consumer have a mutual understanding of the context and content being passed along and use SOAP when a format contract mus be established to describe the interface that the web service offers.

   Both the statements are misleading. I think these statements are based on the assumption that WSDL and schemas provide such a contract for SOAP based web services. This is only half-true. WSDL and schemas can describe the structure of messages (data, headers, faults etc.), but you still need to describe the semantics of operations and messages outside the WSDL. Without a description of those semantics, it is not difficult for different people to come up with different interpretations. WSDL and schemas don’t describe the semantic meaning.

5. Use SOAP when The architecture must address complex nonfunctional requirements such as Transactions, Security,
   Addressing, Trust, Coordination, and so on that go beyond simple CRUD operations possible with REST.

   This article contends that real enterprise applications need more than the CRUD operations that are possible with HTTP. Actually, this article starts by saying how HTTP is related to SQL for CRUD capabilities. This is nothing but a mis-interpreation of HTTP. HTTP defines some standard verbs with defined semantics. There are two ways you can use HTTP for REST style services – by encoding the operations in the URI, and/or by defining new verbs. This gives a wide range of possibilities for surfacing services – definitely not limited to CRUD. In any case, how is this better than encoding the operations in the header/body of a SOAP message and send all messages via POST?

   Regarding the non-functional requirements such as Transactions, Security, Addressing, Trust, and Coordination, except for security, we are yet to see any one using the others seriously today. Who said security cannot be addressed with RESTful web services?

What this article forgets to mention is that RESTful services don’t hide the fact that they are operating over HTTP, where as SOAP style services try to hide this and encourage protocol agnosticism. The result of this protocol agnosticism is API bloat with too many indirections. I have been developing web services for over three years, and I don’t have problems with the lower level APIs (SAAJ, DOM etc.) – what frustrates me is the bloat added by the higher-level APIs (dispatchers, handlers, XML binding etc.) trying to hide the protocol.

JAX-WS for RESTful Services?

JAX-WS (like its dead-cousin JAX-RPC) is designed with protocol abstraction in mind – so they carry the bloat I just mentioned. The API goes on to great trouble in hiding the protocol with layer upon layer – may be the thought was if there are too many layers on top of the protocol, nobody would notice the protocol?

Yes, you can use JAX-WS for REST style services, but as I show below, using JAX-WS makes your code inherit some of that bloat. It adds more complexity without necessarily offering any advantages. Note that the example in this article uses JAXB in combination with JAX-WS, and my comments are limited to using JAX-WS.

First, let me briefly describe the example provided in this article.

1. A purchase order service to get, create, update, or delete purchase orders. This service is implemented using the
   javax.xml.ws.Provider interface
   and some annotations of JAX-WS. The invoke() of this class method processes the request by first checking the request
   verb and the path of the URI used to make the request.
2. A client that uses the javax.xml.ws.Service
   and javax.xml.ws.Dispatch
   interfaces of JAX-WS to talk to the purchase order service.

This example is simple, yet demonstrates the bloat very well. Let’s see the bloat.

1. The servlet API is best suited for providing an entry point for REST-style services. The API has the right abstractions for serving services over HTTP. But since JAX-WS wants to stay protocol agnostic, there is no way you can bind JAX-WS provider classes to servlets. But still JAX-WS leaks some protocol details via the javax.xml.ws.handler.MessageContext
   (which IMO, makes JAX-WS more usable than JAXRPC). So, this example ends up using the leaks to get details of the request API.

   public Source invoke(Source source) {
   try{
   MessageContext mc = wsContext.getMessageContext();
   String path = (String)mc.get(MessageContext.PATH_INFO);
   String method = (String)mc.get(MessageContext.HTTP_REQUEST_METHOD);
   System.out.println("Got HTTP "+method+" request for "+path);
   if (method.equals("GET"))
   return get(mc);
   if (method.equals("POST"))
   return post(source, mc);
   if (method.equals("PUT"))
   return put(source, mc);
   if (method.equals("DELETE"))
   return delete(source, mc);
   throw new WebServiceException("Unsupported method:" +method);
   } catch(JAXBException je) {
   throw new WebServiceException(je);
   }
   }
   

   The above code from this article makes a good case why you should servlet API and not JAX-WS. Since this service is bound to HTTP, this code had to obtain the request info directly via the leaked abstractions. Here is how you could rewrite this using the servlet API.

   
// Nothing


   Except for message parsing, there is no need to write code to detect the verb. The good old javax.servlet.http.HttpServlet does this job, and would delegate to methods like doGet(), doPost() etc.

2. The purchase order service writes back fault messages and response error codes. Here again, we see the SOAP concepts leaking into the REST-style purchase order service.

3. The client side code is similarly more complex like the service implementation. The client first creates a Service, adds a Port, then creates a Dispatcher, creates a RequestContext, and then invokes the service.

   private  void retrieveOrder() {
   service = Service.create(qname);
   service.addPort(qname, HTTPBinding.HTTP_BINDING, url + "ABC-123");
   Dispatch dispatcher = service.createDispatch(qname, Source.class, Service.Mode.MESSAGE);
   Map requestContext = dispatcher.getRequestContext();
   requestContext.put(MessageContext.HTTP_REQUEST_METHOD, "GET");
   Source result = dispatcher.invoke(null);
   printSource(result);
   }
   

   There are too many abstractions in this code snippet. It uses APIs designed for SOAP and WSDL to implement REST-style invocations. But why not use HttpURLConnection for the same task?

   // pseudo code
   URL url = new URL(...);
   HttpURLConnection connection = (HttpURLConnection) url.openConnection();
   connection.setDoInput(true);
   connection.setRequestMethod("GET");
   // ...
   InputStream is = connection.getInputStream();
   // Read and bind to a JAXB object
   

   This code is not trivial either, but you are atleast dealing with HTTP directly – no SOAPy abstractions in the way of doing REST.

Due to the fact that JAX-WS hides HTTP, you will also notice that neither the client nor the service can easily deal with things like headers and cookies. This article makes a good point that REST style services can take advatange of HTTP caching, but unfortunately, JAX-WS won’t let you do it easily.

My Conclusion

In summary, this articles makes an attempt to justify that JAX-WS can beused for programming REST-style web services. It should also have
taken the time to justify why you must use JAX-WS as opposed to, let’s say, servlets and a HTTP client?

If the JAX-WS API is really intent on supporting REST-style programming, here are some the things that it needs to do

1. Provide an easier way to process XML and JSON messages on the server side without hiding HTTP, preferrably by extending the servlet API.
2. Provide an wasier way to write HTTP client by creating some light-weight abstractions on the top of HttpURLConnection so that writing XML to, and reading XML from HTTP connections is simplified.

As of today, I would not recommend using JAX-WS for REST-style web services.

Update (09/03/2006): I just came across Mark Baker’s interview at http://www.infoq.com/articles/mark-baker-REST. In response to a question on combining REST with more conventional web services approaches, he says it’s technically possible, but from what I’ve seen of Web services toolkits (even those that claim to “support REST”), there’s a very good chance that it’ll be more trouble than it’s worth. That says it all.

Will the goal to keep web services protocol agnostic cause web services to
implode? May be so, given that protocol abstraction is making web services
design and deployment more complex than necessary.

A number of people have already blogged about this issue, see, e.g. David Orchard’s

Underlying Protocol is a completely leaky abstraction, or Stefan Tilkov’s
Mapping WS-Addressing to HTTP. Both the posts are quite old, but the facts remain – that most SOAP based web services are designed for protocol independence, with no practical justifiable reasons.  Web services APIs (e.g. those in the JEE umbrella like JAX-WS) are still designed with this abstraction in mind. As a result, implementations of these APIs as well as applications using these APIs rarely take advantage of the protocol. Consider this recent SOAy article titled SOA framework with Apache Geronimo and POJOs which states that designing for location and protocol independence is somehow a good thing. This idea seems quite popular still.

At the message design level, the downside of protocol abstraction is message complexity. A lot of details end up being specified in the message payload (e.g. as SOAP headers) when the same data could more efficiently be carried at the protocol level (e.g. as URIs or headers in HTTP). Examples include WS-Addressing, and some aspects of WS-Security token profiles. Protocol abstraction does not come cheap. It forces some extra processing cycles to create/process messages.

An example closer to my experience is transporting resources over a web service (in WSRP 2.0). Resources are usually arbitrary documents or media that are consumed by user agents. By this very nature, it makes a lot of sense to be able to transport HTTP headers for caching, ranges etc. These headers make it efficient to serve resources to user agents. But since WSRP is a web services protocol, the temptation is to abstract these headers into the message itself. Then the question arises – what would you call them – httpHeaders, or transportHeaders or something more abstract (and vague)? It seems silly to have to abstract out these headers into something vague (to keep it independent of HTTP) – just so that this data
can be carried over diverse protocols like HTTP and SMTP, or even a protocol involving pigeons? Ultimately such attempts at keeping web services protocol agnostic will underutilize the power and flexibility available at the protocol level. This is sad.

Another side effect of this protocol abstraction is the existence of a number of SOAy products for routing, monitoring, SLAs etc. These products may be offering some value, but, IMO, are solving problems at the wrong level, i.e. message level instead of the transport level (e.g. HTTP). Take routing for example. HTTP proxies have been doing routing requests/responses for a number of years, quite efficiently so. I can’t say the same thing about SOAP proxies that look at the XML, run some rules, and then route each message to an end point – basically adding a number of processing cycles to already complex web services.

Good for PowerPoint, not for deployment.