POST is non-idempotent by design, and clients are not supposed to resubmit their POST requests. But if network or software fails before the client could read the POST response, it would not know whether the server processed the request or not. It it resubmits the request, it could cause duplicate resources when POST is used for resource creation, or other negative side effects when POST is used for other purposes. This is a well-known problem with POST for a long time. For the user-facing web, we can play some tricks such as using hidden form parameters or disabling form submit buttons. But how to deal with this problem for non-HTML RESTful apps?
There were some discussions in the past about this topic. One was a long thread rest-discuss group about a null POST idea, and the second was the POST Once Exactly (POE) I-D by Mark Nottingham. The null POST idea can work, but requires the client to treat POST differently from the way specified in HTTP, i.e., the client needs to do some extra work outside the protocol. The POE idea is comprehensive, but that I-D has since expired.
Here is one possible way to revive the POE idea, but (a) without requiring any extensions to the HTTP protocol, (b) without asking the client to learn new tricks, and (c) without breaking URI opacity. Borrowing the POE name from Mark's I-D, let me call this the POE Pattern.
Step 1. Design POE keys. A POE key is an opaque server-generated key, unique for each POST request.
Step 2: Link to POE URIs. Provide links for POST through other resources, with each URI encoding a POE key.
<muppets>
<link href="http://example.org/muppets" rel="self"/>
<link href="http://example.org/muppets;poe=123gh3j3" rel="edit"/>
<muppet>
...
</muppet>
<muppet>
...
</muppet>
<muppet>
...
</muppet>
...
</muppets>
The second link in the above has a POE URI that the client can use for just one POST.
There are a couple of problems with this style of linking. Firstly, it breaks caching, since we need to keep each POE URI unique for each client, and shared representations through caching will lead to shared POE URIs. Secondly, if the server decides to expire POE keys after a set amount time, the linked POE URI would become invalid. Even when the server is enable to overcome these, the clients still needs to do a GET on a related collection before it can POST.
The second step may not always be efficient and practicable, but worth considering. That brings me to step 3.
Step 3: Redirect to a valid POE URI. If a client uses a pre-published non-POE URI or a POE URI with an expired key, redirect the client to a freshly minted POE URI.
POST /example.org/muppets ... 307 Temporary Redirect Location: http://example.org/muppets;poe=124gh3j3
The client will then use the new location for POST. I am using 307 here since HTTP client libraries are not supposed to automatically follow such a redirect, and secondly we don't want clients to turn the POST into a GET while automatically following a redirect (which can happen for a 302 with misinterpreted client implementations).
Step 4: Process POE. When a client submits POST to a fresh POE URI, let the server process the request, and invalidate the POE URI. If the client resubmits the POST request, it can do one two things. If the server is capable of remembering the outcome of the previous successful POST and if it can determine that the current request would result in the same/similar outcome, it can redirect the client to that location.
POST /muppets;poe=124gh3j3 ... 303 See Other Location: http://example.org/muppet/456
If not, it can simply fail the request.
POST /muppets;poe=124gh3j3 ... 405 Method Not Allowed ... explain why in the body ...
One downside with this pattern is that the server ends up doing some extra work to manage POE keys. On the plus side, it can free the client from dealing with non-idempotency of POST and from learning non-standard protocol tricks.
[Credits for this pattern go to the two references cited above.]
{ 6 comments… read them below or add one }
First off great post, keep up the good work, this is one of my favorite blogs.
I just wanted to throw out a idea, why not take a hash of the POST request data (or a subset of it, whatever determines uniqueness) and store it on the server to match with the success of the POST. This would be the “transaction id” of sorts. If a client then submits a duplicate POST, the hash would match and you’d throw a 409? error.
By using a hash of the POST variables, you could even choose the variables that make the POST unique. If what we are trying to prevent is duplicate POST transactions, what easier way would there be other than comparing the data sent in the POST request?
What am I missing?
Thanks
This POE pattern does not require the server to determine if two given POST requests contained the same data. So it can fail fast.
In some cases, the fact that the request contained the same data may not necessarily mean that they are duplicate requests. Say, e.g. two requests to Twitter with the same data by the same user.
But if the server can determine that they are the same, and it also remembers the resource that was created the first time, instead of returning a 409, it can return a 303 (See Other) with the Location of that resource. That is, there is no need to fail the request in this case. If not, it can return a 405 (which I think is more appropriate than 409).
Thanks for this interesting idea. Might be fine for custom clients but will probably not work to well with standard browsers. RFC 2616 states:
If the 307 status code is received in response to a request other
than GET or HEAD, the user agent MUST NOT automatically redirect the
request unless it can be confirmed by the user, since this might
change the conditions under which the request was issued.
And at least the current Firefox (3.0.4) asks for user confirmation on POST redirect (also when specifying a local/relative redirect location).
Since I am looking for a solution that works for my custom written client and for ordinary web browsers, I’ll stay with the original POE draft and use the special POE headers defined in there. That will enable POE only for my custom client (since it requires custom request handling), but at least it won’t break standard web browsers (since my server will still accept ordinary POST requests without POE header).
Any better suggestions?
The use of 307 is intentional here. We don’t want the client to automatically follow such a redirect.
I agree when talking about an HTTP library as the client.
But I was talking about the user client. I don’t want the user to be asked if a POE redirect should be followed. This problem occurs when the actual http library is supplied by the browser (like in ajax or xul apps; e.g. XmlHttpRequest). At least Firefox/XulRunner handle the 307 automatically and ask the user for confirmation. I couldn’t find a way to override that behaviour.
That’s why I’m using a modified version of the original POE draft instead. Only modified it to exchange all poe related information in special, non-standard headers, instead of encoding it in the URI (because this was easier to implement in my case).
I see. If you are writing the app for browsers, then the most common solution is anyway to include hidden form parameters, and do a 303 to the same page. When that page is regenerated, the server can include a different hidden parameter in the form. But it looks like you may be writing a specialized app.