06:29 PM, Thursday, February 21, 2008

Idempotency Explained

In response to David Peterson's The SimpleDB Epiphany: I Finally GET It... Why RFC 2616 Is To Blame I posted a quick note here, and a few comments on his blog. He asked a few questions, but as I got caught up with my day job, I could not completely elaborate my comment on why idempotency for pipelined requests is a protection and not a limitation. Here is the rest.

Here is why his assertion is inaccurate. Let's say, a client pipelines four requests to a server:

  • Get (GET) something
  • Update (PUT) something
  • Update (PUT) something else
  • Delete (DELETE) something

Now, let's imagine that the server started processing these pipeline of requests and responded with the the first response. At this time, let's say the client lost the connection to the server. But the client can not tell whether (a) the server processed all or some of the remaining requests but the response did not reach the client, or (b) whether the server did not process any of the remaining. The client can retry these requests, but doing so should not have side effects, e.g., creation of duplicate records, or deleting data that is not supposed to be deleted. But since all the requests in the above pipeline are idempotent, the client can safely retry the entire pipeline. That's the role of idempotency. Executing idempotent requests more than once yields the same result as would result by executing it once, barring bad server implementations.

In my version of SimpeDB, the following requests could safely be pipelined.

  • Delete (DELETE) a domain
  • List (GET) domains
  • Update (PUT) attributes
  • Delete (DELETE) attributes
  • Get (DELETE) attributes
  • Query (GET) items

All these are idempotent requests. The only non-idempotent requests in this API are:

  • Create (POST) a domain
  • Create (POST) attributes

These two kinds of requests can not safely be retried as doing so would have negative side-effects, e.g. creation of duplicate domains or attributes, and hence should not be pipelined irrespective of what RFC 2616 says. Fixing 2616 to allow non-idempotent requests in a pipeline is not the solution.

Overloading GET for all types of operations does not necessarily give the ability for clients to pipeline requests and thereby, for the server to provide the parallelization needed for a service like SimpleDB. The same rules still apply.

Comments

Hi Subbu,

Thanks for taking the time to extend your post! My only question is what you mean exactly by,

In my version of SimpeDB, the following requests could safely be pipelined.

Delete (DELETE) a domain
List (GET) domains
Update (PUT) attributes
Delete (DELETE) attributes
Get (DELETE) attributes
Query (GET) items

Given that a majority of pre-existing HTTP(S) libraries will not pipeline PUT requests, how then could they be pipelined?

I have been looking for this formation. Do you know which libraries do not support pipelining for PUT?

On a related note, I noticed the most often-cited page on pipelining at http://www.mozilla.org/projects/netlib/http/pipelining-faq.html incorrectly implies that PUT is not idempotent. I just wrote to the purported author for clarification.

>> I have been looking for this formation. Do you know which libraries do not support pipelining for PUT?

To be honest, now that I actually look it up, I can't find any references in the .NET documentation (the library I most often use) that specify that GET and HEAD are the only supported verbs for HTTP Pipelines. For some reason, however, that "fact" has always been stuck in my head when I consider using HTTP Pipelines in my code. I could very easily be completely off, so really need to dig deeper to verify.

>> On a related note, I noticed the most often-cited page on pipelining at http://www.mozilla.org/projects/netlib/http/pipelining-faq.html incorrectly implies that PUT is not idempotent. I just wrote to the purported author for clarification.

Nice catch! Nothing worse than "official" docs that are incorrect and or misleading! It seems that there is a lot of misinformation out there. If nothing else, regardless of what RFC 2616 does or does not specify, by the time we have this researched out we'll end up with a much better understanding of the reality of what is/is not supported and why.

Tim Olsen said:

>All these are idempotent requests. The only >non-idempotent requests in this API are:
>
> * Create (POST) a domain
> * Create (POST) attributes

I'm not sure about your version of the API, but Amazon's docs are pretty clear that creating a domain is idempotent:

http://docs.amazonwebservices.com/AmazonSimpleDB/2007-11-07/DeveloperGuide/SDB_API_CreateDomain.html

"Note

CreateDomain is an idempotent operation; running it multiple times using the same domain name will not result in an error response."

Unfortunately, because the API hides the resource behind query arguments (which are mixed with the de-facto method!), you are forced to use a POST here which clients can only assume is non-idempotent.

A much better API for creating domain names would have been:

PUT /domains/mydomainname

The client would then clearly know that the request is idempotent

Thanks for pointing out that the CreateDomain is an idempotent operation. I overlooked that.

You are right, such an operation should be modeled as a PUT.

Hi Subbu,

Have you put your original RESTful design to code as a wrapper around SDB? If no, please let me know. I would be *very* interested in writing/helping to write the interface (I have access to SDB, so this would be done using the live SDB API) if not. It seems to me that regardless of the backend infrastructure, creating a RESTful "ODBC"-layer that could easily hook into any HTTP(S)-accessible DB would be a *tremendous* benefit to ensure that regardless of the backend implementation, the frontend can maintain a proper RESTful architecture.

Please let me know if this sounds at all interesting to you. Thanks!

Assaf Arkin wrote a ruby wrapper called DeHorrible. Please check this at

http://blog.labnotes.org/2007/12/17/dehorrible-restifying-simpledb/

Leave a comment