Slides from my workshop on ql.io at the Node Summit.

Here is an example of how idempotency can save end users a precious second or so.

Several weeks ago, one of the teams using ql.io reported that they were seeing random socket hang up errors in server logs.

error: { stack: [Getter/Setter],
 arguments: undefined,
 type: undefined,
 message: 'socket hang up',
 uri:
  { value: 'http://clientalerts.ebay.com/...'

The source of this log was the error handler of the http client

request = http.request(...);
request.on('error', function(err) {
    // log the error
});

This error was so random that we could not find a reproducer on our dev or testing environments for a while. We then wrote a ql.io script that emulates the production use case, ran that script using siege to emulate the production usage, and captured the traffic using tcpdump. While we were to able to capture a few instances of socket hang up errors with this setup, the tcpdump did not give definite answers as there were too much data to sift through. The client (siege) was making requests in parallel to a cluster of node instances with each instance maintaining its own socket pool. But there were two clues:

• The origin was closing the connection after some usage by sending a FIN packet, but none of the accompanying HTTP responses included the Connection: close header. This is quite common as connection handling is often done at the reverse proxy tier which may not insert this header in the last response before closing the connection.
• The number of new connections opened during the test was a few less than the number of connections closed. This indicates that some requests were being made on closed connections causing the socket hang up errors.

This lead us to Issue 1135 – a timing related issue in node.js where inflight requests made by the client after the FIN packet was sent caused the socket hang up error. The resolution was to retry the request if possible.

HTTP GET, PUT and DELETE requests are retriable in case of such network errors, and the server implementations of these methods are supposed to support such retries.

But here is the rub. There is a lot of bad deployed HTTP server code out in the wild that might choke when a client retries in case of such network errors. Hence, node.js can not retry automatically without some indication from the client application. In the end, we decided to retry the request when an idempotent HTTP request caused by a select statement encounters network failures such as the above. This helped us save an extra roundtrip from the client to ql.io.

This is a reminder of why idempotency is a necessary promise to keep in HTTP server code.

I don’t like writing client apps that use APIs. Writing HTTP client code to talk to APIs is verbose, repetitive, chatty and slow. This is in addition to addressing latency and bandwidth constraints and core functionality of the client app – such as building a snappy UI or supporting some other business use case.

First – the interface mismatch. APIs are almost always designed from the producer’s point of view. If I’m building an API, I would look at my data model, my performance/scalabilty requirements, and design the API to meet the use cases that I think are the most common across all consumers my of my API. That is, my goal would be to maximize the use and reuse of my API. In this process, I am bound to tradeoff special requirements of my consumers and try to stick to a common denominator. This creates a mismatch between what the consumer needs and what I’m offering. Here is an example.

A client app needs to search a list of products to get their IDs, some details and some user generated content like reviews for each product. There are three APIs that the client needs to interact with to get this data – one to search products, one to get details for each product, and another to get reviews for each product.

This is not a made up or hypothetical example. What the client needs is this.

A single API, that takes a keyword, and returns certain fields for each product found – no more or no less.

What the client got are three APIs that give some data that includes what the client needs plus some more. The API design makes sense for the producers of these APIs – search folks are focused on indexing and serving low latecny results, the details API wants to serve all the gazillion details that are relevant to products, and the reviews API wants to focus on low latency writes and potentially stale reads. But, for the consumer, an ideal interface is one that gives just the fields it needs with one single request.

Second – writing client code is slow, repetitive and verbose. For the above example, here is what the client needs to do:

• Call the search API to find products, and extract product IDs from the response.
• Call the details API n times – once per product ID – to get the details, and then extract the fields needed from the response
• Call the reviews API n times – once per product ID – to get reviews, and the extract the fields needed from the response

Unless you got canned SDKs for these APIs, writing code to make 2*n HTTP requests and process the responses can take a while. In one of the Java implementations I looked at, the code was over 300 lines long – that was after excluding the request making and response parsing code which was already factored into into supporting libraries. If the client needs to implement ten such use cases, you got 3000 lines of code just doing data retrieval.

Third – Getting enhancements you need from producer teams slows you down. For my example above, having a bulk API for details and reviews simplifies the client to some extent. But for both good and bad reasons, API changes that the clients need don’t happen immediately. They take time, usually some sprints. Sometimes never. The reality is that teams have their own priorities. Getting a single API for the above won’t likely happen!

Fourth – requests for use cases like this are chatty. Three hundred lines of code may not be a big deal, but making so many HTTP requests is. Here is a common evolution of an implementation. To simplify the discussion, let’s assume that we have bulk APIs for details and reviews.

Take 1: Use blocking IO to make 3 HTTP requests.

Code complexity of this implementation may be low, but it takes sum(t) where t is the time for each API request.

Take 2: If latency is a concern, parallelize the requests. After the search API returns product IDs, you can make 2 requests in parallel and join the responses.

Code complexity now increases, but it only takes t1 + max(t) for the whole retrieval, where t1 is the time taken to search.

Now imagine that the client needs to call yet another API based on review IDs.

Take 3: Orchestrate the retrieval based on dependencies.

In another example I’ve seen recently, a native app makes 17 HTTP requests with some dependencies before painting the UI. The code is over 3000 lines long! Of a team of 3 developers, one developer is dedicated to writing and maintaining this code.

Now move the client farther from the API servers. In addition to the code complexity, the client will have to deal with cost of the network. You may want to move all the request dance to some middle tier that is closer to the API servers than the client.

Fifth – consistent RESTful APIs don’t matter as much as we think. I would love to see every API to be RESTful, consistent, hypertext driven, and more importantly, interoperable. The reality is that getting consistent APIs is hard – particularly those that are done by distributed teams. For the record – I vehemently hate SOAP APIs and the SOAP mindset. I dislike RPC-style operation names tunneled over HTTP. I frown and cringe whenever I see unnecessary custom headers and complicated formats. I wish POST+XML goes away. I wish every API gets rewritten to the modern understanding of HTTP and REST, serving JSON.

But I would rather spend my time enabling interoperability than preaching for consistency.

Hypermedia does not help either. Hypermedia can help navigation among resources, but navigation is not a concern in the above use case. The client app’s challenges are aggregation, parallelization, and orchestration of forks and joins.

So, what can we do about it?

Why would I write a long blog post if I have nothing to offer?

I’m excited to reveal that, at eBay, my team has been working on a new platform to simplify use cases like the above:

• Bring down number of lines of code to call APIs
• Automatically orchestrate API calls as needed
• Create new consumer-centric APIs that wrap existing producer-centric APIs
• Cut down bandwitdh usage and latency

Best of all, this platform will soon be on github. If you are interested in taking a sneak peek and want to provide feedback, please email me (subbu/at/subbu/dot/org) with your github ID. Please see ql.io for more info.

For a soon-to-open sourced nodejs based platform that I have been working on since June this year, I needed a way to send some optional telemetry data back to the browser without mixing it with the real data that the client code is interested in. The first thing I looked at was socket.io. Socket.io’s claim to fame is that it aims to “make real-time apps possible in every browser and mobile device”. But it comes at certain cost that may not be apparent. Since then I gave up on socket.io and switched to the plain WebSocket API on the client side and WebSocket-Node on the server. Here is why.

Cross browser WebSocket fallback is a fallacy. Such fallbacks exist because WebSocket support in browsers continues to limp and the spec itself is not done yet. Interoperability among browsers is not yet possible due to lack of protocol parity. For instance, about a month ago, I had to forego supporting Safari in favor of better experience in latest versions of Chrome and Firefox. But what’s wrong with fallbacks?

WebSocket interactions are bound to a connection – once the client opens a TCP connection, it can send and receive several messages using the WebSocket API. But with fallback approaches that rely on JSONP, XHR and iframes, clients and servers use HTTP for serializing messages.

Therein lies a key difference. With the WebSocket protocol, you can scope any state for the duration of the connection and remain state free across connections. For instance if you want to send some events as they occur to the browser, you can associate the subscription state with the socket. You don’t need to replicate that subscription state across all the server nodes that are serving WebSocket traffic. This makes horizontal scalability possible.

Not so with HTTP based fallbacks where every message could potentially come in over a different TCP connection. To account for this you either need to pin the client to a given node so that the same node handles all connections from a client, or replicate state across all the nodes serving the HTTP traffic. socket.io does the latter by using Redis. Consequently you end up trading off reliability and/or scalability.

This leads to my next point – if scalability is your concern, graceful degradation may just be sufficient in a lot of use cases. You can use feature detection in browsers to decide whether to use WebSocket protocol or not. Here is a simple approach I now use on the client side.

This helps me offer a degraded user experience when WebSocket support is not available, and I don’t have to introduce shared state on the server.

These are the slides from my talk yesterday at restfest.

A while ago I showed how chatty some well-known apps are on my iPhone. But this issue is neither new nor unique to apps on phones and similar devices. Efficient data retrieval from distributed/decentralized servers is a well-recognized problem in distributed computing. For instance, in the abstract of his November 1994 paper A Note on Distributed Computing, Jim Waldo notes the following (emphasis mine).

We argue that objects that interact in a distributed system need to be dealt with in ways that are intrinsically different from objects that interact in a single address space. These differences are required because distributed systems require that the programmer be aware of latency, have a different model of memory access, and take into account issues of concurrency and partial failure.

Most front-end developers by now know and follow the best practices that Yahoo!’s Exceptional Performance team documented a few years ago. However, the REST community may have missed the bus and has some catching up to do. Performance of RESTful Apps is not one of the most frequently talked about topics online or in print. From talking to various teams, it often seems that a great deal of time is spent on URI/representation design, schemas, use of the uniform interface for CRUD, using the hypertext constraint etc. No doubt – these topics are all very important, but understanding and accounting for the performance characteristics in the design and implementation of server and client apps is no less crucial.

Here are some techniques to help build high-performance RESTful apps.

Composites for Performance

The best of all web performance techniques is to minimize the number of HTTP requests. However, RESTful Apps rarely follow this practice. This difference stems from how each side sees resources.

On one side, front-end folks optimize their servers to serve bulk representations for CSS, JavaScript or image sprites, or even data URIs for images to reduce the number of HTTP requests and thereby latency. On the front-end, most resources are in fact composites.

On the other side, API/service developers prefer clean looking resources and URIs (shown on the right side above). Though this can lead to chatty network usage, there is one specific advantage in offering a set of resources that are independent and less coupled with other resources – it leaves room for clients to innovate. It lets them combine data from multiple resources in numerous ways that the resource developers could not possibly think of.

Loose coupling is another benefit of this approach, as clients can evolve rapidly on their own.

The expense is of course latency, particularly when those client apps are not very close to the servers. Each client may need to submit several requests to the server in order get its job done. So, how do we go about fixing this without sacrificing the flexibility of less-coupled resources? One answer is to use composite resources. See my RESTful Web Services Cookbook for details.

With a composite, in stead of sending n-number of HTTP requests over 1-n connections, the client can open just one TCP connection to send an HTTP request to retrieve the data it needs – just like a browser getting CSS or Javascript bundles in the front-end. A composite changes a pattern like

GET /something HTTP/1.1
Host: www.example.org

GET /something-else?params HTTP/1.1
Host: www.example.org

GET /some-other-thing-related-to-something?params HTTP/1.1
Host: www.example.org

to

GET /get-all-things-i-need-about-something?params HTTP/1.1
Host: www.example.org

Each composite can generate a projection of state required for one or more clients. These composites can be more specialized than the resources they aggregate – as each composite can cater to particular client needs.

(P.S.: My usage of the term "resource" is not precise here as a "composite" is also a resource.)

This approach also shifts issues related to concurrency (such as ordering of requests based on success or failure), CPU (for generating projections, correlating related data from across representations etc.) and I/O (to fetch back-end resource representations in a serial/parrallel fashion depending on dependencies) workloads from the client to the server.

On the server side, the server hosting composites can also optimize its connection handling to resource servers to reduce TCP handshake and slowstart overhead. For instance, it can maintain pools of persistent/long-lasting connections (e.g., with keep-alive) between servers hosting composites and the resources (shown by bold arrows above).

In this post, I’m not going to discuss software choices to serve composites, but you may need to account for several features:

• multi-tenancy or isolation of code execution, configuration and deployment so that different teams can build composites
• data or control flow for fetching representations in parallel or sequentially based on inter-dependencies
• query languages (such as YQL) to normalize data formats and to easily create projections
• non-blocking or asynchronous I/O to better tackle I/O workloads

I’m personally excited about nodejs and async I/O support in Java 7 as both would let us build small and nimble broker apps to serve composites.

Of course – the idea of a composite is to add an extra layer of indirection on the server side to offset network overhead when performance is at stake. It is not meant to replace loosely coupled resources that can be manipulated using HTTP and linked using hypertext controls like links in representations.

Better Connection Reuse

Long-lasting TCP connections help reduce connection-establishment overhead as well as help the TCP stack settle to appropriate congestion window size. Reusing connections is usually trivial, and pooling is often part of client libraries. But there are a few precautions to take at the application level.

Avoid Explicit Connection Closing

# Don't do this
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 1234
Connection: close

{ ... body ... }

The first precaution to take is not to add Connection: close to requests or responses by default. Carelessly adding this header to requests or responses will prevent connection reuse. There better be a good reason to add this header such as a server that can’t handle too many open connections, or to prevent abuse.

Avoid Close Delimited Messages

# Avoid this
HTTP/1.1 200 OK
Content-Type: application/json

{ ... body ... }

Make sure to include Content-Length or use Transfer-Encoding: chunked so that the recipient of a HTTP message can know when one HTTP request/response message ends and when the next one starts. If a response has neither of these two, then the recipient will need to read the stream till the connection is closed, which means that the connection cannot be reused. Close-delimited HTTP request/response messages are bad for performance.

Read Messages Completely

Incomplete reads will also prevent reuse of the connection. Incomplete reads usually happen when a client receives an error or a redirect response. These kinds of responses can have a body but clients can determine what to do by just looking at the response line and headers. For instance, a 409 response may have a body that explains why.

HTTP/1.1 409 Conflict
Content-Type: text/html
Content-Length: 1234

<html> ... </html>

But not reading the body from the connection my prevent reuse in some frameworks – Java is known for this. Also be wary of libraries that translate 4xx and 5xx response code into exceptions – in this case, in addition to catching the exception, the client will need to read the body.

Tune Idle Connection Timeouts

Servers and clients usually close idle connections upon a timeout to conserve resources. Settings for idle connection timeout may be hard to find or even not exposed in client/server frameworks. When tuning is possible, ensure that the defaults are reasonable.

Try Proxies for Long-Haul Traffic

In some cases a configuration like the following can help:

• Client apps that are short-living (like an app on a mobile/tablet or even on desktops), and hence connections can’t be persistent. Instead, the proxy can keep connections persistent which limits connection establishment cost to the first leg from client app to the nearest proxy.
• Client apps that can’t maintain too many persistent connections – which is still the case for browsers today – though it is slowly changing.

Of course, this approach also lets the server distribute responses to caches on those proxies to further reduce network cost. Many variations of this approach are possible depending on how your servers are distributed and how far are client apps from servers. If you’re new to the idea of REST and are still wondering why HTTP’s uniform interface is such a big deal, here is why – once you implement HTTP reasonably correctly, you can reconfigure servers, proxies and caches as necessary without code changes.

Progressive Serving of Representations

Sometimes it is not the network, but generating a response is the bottleneck. This is particularly true for composite resources or resources that rely on a number of data sources to generate a response to the client. The typical flow in such cases is as follows:

• read the request data such as the path and query string
• decide what to fetch
• fetch data from each dependent source in sequence or concurrently to the extent possible (which depends on dependencies)
• prepare data for the response
• write the data to the response

Of these steps, when I/O for dependent sources is done sequentially, the server takes at least n*t time to generate a response. If all the I/O can be done in parallel, it takes max(t) amount of time, i.e, it performs at least as slow as the slowest source.

On the front-end side, when it takes time to generate a page, a common practice is to turn to XMLHttpRequest or iframes to split the page into fragments and defer loading of slower parts of the page. Both these techniques potentially use additional connections. In a multi-tiered setup, this causes a flood of new requests from the browser to front-end servers, and from there to backend servers and so on. This also introduces new state management and security problems as the server may need to push state first to the browser only to get it back via XMLHttpRequest immediately.

An alternative is to progressively render the page over a single connection. In this case, the flow would be

• read the request data such as the path and query string
• decide what to fetch
• fetch data from fast sources
• initiate requests for slow sources
• serve partial page based on response from fast sources
• as and when a slow source responds, prepare a partial response and write to the client
• after all the sources respond (or after some timeout), write additional chunks and finally end the page

Here, by “chunk” I mean “part of a message” and not an HTTP chunk.

The goal of this technique is to reduce user-perceived latency without using more network connections from the browser. In this flow, browser makes an HTTP request to the front-end server which writes snippets of markup and script over a period of time before ending the response. Since the server does not know the Content-Length of the page, it would use chunked transfer encoding where end of response is triggered by a zero-sized chunk.

This is called “progressive rendering”. This technique is well-known in front-end circles and Facebook calls this technique BigPipe. Progressive rendering depends on two things:

• Server being able to write chunks over lasting connections – asynchronous I/O based servers like nodejs are very attractive for this (see my nodejs example or Bruno‘s example using continuations).
• Clients being able to process response as it arrives – in Javascript capable browsers, this capability is already present.

We can apply this technique for non-front-end resources as well provided (a) it is possible to retrieve data from fast sources before slow resources, and (b) data from fast sources is meaningful to clients. For instance, think of a personalized product resource that includes data about a product plus IDs, links, and brief summaries of related products. In this case, product data can be looked up from storage under a near-constant time (say, about 20 milliseconds) while finding related products may involve performing some computations on the user profile, past purchase history and other derived data which can be time consuming – say, taking up to 500 milliseconds. Here is an example of a progressive representation of such a product resource.

HTTP/1.1 200 OK
Content-Type: multipart/mixed; boundary=abcdef
Transfer-Encoding: chunked

--abcdef
Content-Type: application/json

{ ... product data here ... }

--abcdef
Content-Type: application/json

{ ... related products here ... }
--abcdef--

In this example I used a multipart media type as it provides a visible boundary between different portions of the representations, and the client can read the representation part by part.

If the client is a front-end app that generates an HTML product page for browsers, it can progressively render the product page as soon as it receives the first part, and then render markup for list of related products when the second part arrives.

HTTP/1.1 200 OK
Content-Type: text/html
Transfer-Encoding: chunked

<html>
  <head>...</head>
  <body>
    ... HTML for the product data ...

    <div id="related"></div>

    <!-- script for related products (some milliseconds later) -->
    <script> // update related div ...</script>
  </body>
</html>

This shows how progressive generation of arbitrary representations can be combined with progressive serving of the front-end to reduce perceived latency.

Epilogue

One of the patterns to notice from this post is that design considerations between HTML serving front-end apps and JSON/XML/whatever-speaking RESTful apps are not entirely different. Both rely on the same set of core architectural principles such as the uniform interface, visibility, hypertext, and so on. Whatever lessons we learn on the front end are certainly applicable for the so-called API servers.

Finally, it goes without saying that premature optimization is evil. My goal of this post is to point out the techniques you may already have in your toolkit. Apply them based on the need and experimentation.

If you find this post useful, try my book: RESTful Web Services Cookbook.

I just caught up with the recent reactions to the “hashbang” episode. Of all the reactions, I’m most intrigued by Ben Cherry‘s post. Ben’s point is that the Web has been slowly transforming from “traditional web sites” into “desktop-class applications” served via HTTP, and that “the hashbang is in the unfortunate position of being the messenger” of this transformation. He then goes on to make this case:

We’ve made a tradeoff, however, in making twitter.com into an application using the hash, which is that it now cannot be both an app and a site at the same time.

This argument is not uncommon but is troubling. We hear similar arguments from web developers who find it hard to maintain proper browser back-button behavior, or to make case for “single page apps” (thank goodness – Web 2.0 days are behind us), or when some site resets the state of its pages when some in-memory sessions time out. These are all bad for the user. In all these cases, interoperability and usability of the Web suffers. Web works by agreeing on a number of interoperable protocols and formats so that we can code agents, servers and intermediaries to interpret those protocols and provide a consistent behavior to users. Branding a site as an “app” should not automatically lead to trading off usability and more importantly, interoperability.

Are web apps different from web sites? Yes if you are the developer of the app, but no if you are a user. If you’re a developer trying to provide rich functionality packed in a single page, the differences between serving a static and typically linear HTML pages, and a complex user interface with code tied to UI/UA events are significant. But hey, we should not make usability arguments wearing a developer hat. Try a user hat and think from the user’s point of view. For a user, there is just one Web. Users “logon to the web” to get stuff done. They don’t care about the tech choices we need to make to provide rich interactivity.

What’s the real tradeoff in the hashbang case? The tradeoff here is between usability of two classes of clients – one that does not support Javascript and the other that supports. That’s a reasonable tradeoff! Twitter chose to not render http://twitter.com for non-Javascript browsers. Perhaps they made that by choice by looking at the proportion of users that use non-Javascript browsers.

In the case of Twitter, as Ben notes, HTML5 History API can help – but only partly. Servers like Gawker have also chosen to encode page identity into the fragment portion of URIs – This is a shame. For instance, two GET requests from my command line

curl -v http://gawker.com/
curl -v http://gawker.com/\#\!5769800 # escaped hashbang

come back with the same response from the server (except for some timestamps and headers). If I have a forward-proxy cache in front of my clients, it unnecessarily fills up the cache with duplicate responses. A big deal? May be or may be not. The point of Web is interoperability. Lack of interoperability forbids serendipitous reuse. Branding a site as an app is an unnecessary distraction.

HTTP pipelining is often suggested as a way to dramatically improve page load times, or to solve multi-GET use cases for RESTful applications. Whether pipelining can achieve the intended effect or not truly depends on what gets pipelined and how the server implements pipelining.

When using pipelining, a HTTP client sends idempotent HTTP requests (such as GET) without waiting for response of previous requests, and expects responses to arrive in the same order from the server. HTTP 1.1 says nothing about order of processing of requests on the server side – servers can process each request in sequence or in parallel. All that matters is the order of responses. However, in the real-world, pipelining is not often used due to a number of interoperability issues. Mark Nottingham recently captured some of these issues in an internet draft:

Anecdotal evidence suggests there are a number of reasons why clients don’t use HTTP pipelining by default. Briefly, they are:

1. Server implementations may stall pipelined requests, or close their connection. This is one of the most commonly cited problems.
2. Server implementations may pipeline responses in the wrong order. Some implementations mix up the order of pipelined responses; e.g., when they hit an error state but don’t “fill” the response pipeline with a corresponding representation.
3. A few server implementations may corrupt pipelined responses. It’s been said that a very small number of implementations actually interleave pipelined responses so that part of response A appears in response B, which is both a security and interoperability problem.
4. Clients don’t have enough information about what is useful to pipeline. A given response may take an inordinate amount of time to generate, and/or be large enough to block subsequent responses. Clients who pipeline may face worse performance if they stack requests behind such an expensive request.

Even if we fix all the interoperability issues (such as 1, 2, and 3 above), pipelining will not necessarily improve anything. Unlike non-pipelined requests, clients need to know a bit about the server’s implementation before deciding to pipeline requests. Here is why.

The key constraint in pipelining is that the server must send responses in order. This leads to the so-called head-of-line blocking problem.

Assume that the client opens a connection and sends three GET requests, g1, g2, and g3. Of these, let’s say that g1 takes longer to process than g2 and g3. But the server is still required to return responses in the sequence of g1, g2, and g3. Here is one possible implementation in a multi-threaded server.

• Server receives a connection, and it gives the associated channel/stream to a thread t0
• Server starts parsing the data in t0
• Server finds g1, and hands it off to an application handler h1 in thread t1
• Server finds g2, and hands it off to an application handler h2in thread t2
• Server finds g3, and hands it off to an application handler h3 in thread t3
• h2 finishes first and wants to write response – server blocks it since h1 has not finished yet
• h3 finishes next and wants to write response – server blocks it too since h1 has not finished yet
• h1 wants to write response – since g1 is the first request, the server lets it
• Server unblocks h2, and it writes response
• Server unblocks h3, and it writes it to response

In this model, the server explicitly blocks application handlers from writing response until it is their turn. Alternative implementations are possible:

1. The server can wait to read the next request (i.e., the request line, headers and any body) until the previous request is completely processed.

2. The server can buffer responses of application handlers (at least of those that finish earlier than previous requests) and write them in order to the client.

Over some limited tests during the weekend, I found that both Netty and Tomcat follow the first approach while Nodejs follows the second approach. Both approaches have their limitations, in particular when one of the requests early in the pipeline takes time to complete. In such cases, the client is better off sending g1 over one connection, and pipeline g2 and g3 on a second connection. This will reduce the serialization window on the server. However, in order to make such a choice, the client needs to have some prior idea about workloads involved in processing each request. When such information is difficult to assert (e.g., for a browser sending requests to an arbitrary servers), connection reuse via keep-alive is safer bet than pipelining. In any case, it is better to test before enabling pipelining in clients.

Gabriel Wienberg:

For those of you who have no idea what I’m talking about: when you click on a link on the Internet, where you clicked from gets automatically sent to the site you clicked on (most of the time).

For example, if you’re on yahoo.com and you click to a story at the New York Times, your browser will send to newyorktimes.com some information that you came from yahoo.com — namely, the Web address of the page you were just on. This info is called the Referrer.

At issue here is that sometimes the Referrer contains personal information. In particular, when you use most search engines, your search terms are included in the Referrer. That is, when you search on Google/Bing/etc., and you click on a link, your search terms are sent to the site you clicked on. This search leakage doesn’t happen at DuckDuckGo.

I don’t see why this should not be called FUD. This post does not actually explain how any personally identifiable information (PII) leaks to third parties when you search for something using a search engine. Search terms don’t constitute PII. Duck Duck Go uses a clever trick to get the browser send a Referer header with no search terms, but there are other tricks possible.

At the core, the problem is not with the Referer header. By choice or by poor design, some sites include PII in URIs of pages that link to third-parties. That is the real problem. This is not a problem for Google or Bing to fix. Every site that cares about privacy of its users should just ensure not to leak PII in URIs.

O’Reilly will be offering the ebook version of the RESTful Web Services Cookbook on sale tomorrow (Jan 19, 2011). Here is the scoop I just received from O’Reilly.

Save 60% – Top 25 of 2010

Best of Ebook Deal of the Day – http://post.oreilly.com/rd/9z1ztpijtp7meklse9bq1u34nnvf1fdmc1n0battp2g

For one day only, you can save 60% on our best of “Ebook Deal of the Day” titles – the top 25 of 2010. Ebooks from oreilly.com are DRM-free. You get free lifetime access, multiple file formats, and free updates. One day only. Use discount code DDT25 in the shopping cart.