Slides from my workshop on ql.io at the Node Summit.

Here is an example of how idempotency can save end users a precious second or so.

Several weeks ago, one of the teams using ql.io reported that they were seeing random socket hang up errors in server logs.

error: { stack: [Getter/Setter],
 arguments: undefined,
 type: undefined,
 message: 'socket hang up',
 uri:
  { value: 'http://clientalerts.ebay.com/...'

The source of this log was the error handler of the http client

request = http.request(...);
request.on('error', function(err) {
    // log the error
});

This error was so random that we could not find a reproducer on our dev or testing environments for a while. We then wrote a ql.io script that emulates the production use case, ran that script using siege to emulate the production usage, and captured the traffic using tcpdump. While we were to able to capture a few instances of socket hang up errors with this setup, the tcpdump did not give definite answers as there were too much data to sift through. The client (siege) was making requests in parallel to a cluster of node instances with each instance maintaining its own socket pool. But there were two clues:

• The origin was closing the connection after some usage by sending a FIN packet, but none of the accompanying HTTP responses included the Connection: close header. This is quite common as connection handling is often done at the reverse proxy tier which may not insert this header in the last response before closing the connection.
• The number of new connections opened during the test was a few less than the number of connections closed. This indicates that some requests were being made on closed connections causing the socket hang up errors.

This lead us to Issue 1135 – a timing related issue in node.js where inflight requests made by the client after the FIN packet was sent caused the socket hang up error. The resolution was to retry the request if possible.

HTTP GET, PUT and DELETE requests are retriable in case of such network errors, and the server implementations of these methods are supposed to support such retries.

But here is the rub. There is a lot of bad deployed HTTP server code out in the wild that might choke when a client retries in case of such network errors. Hence, node.js can not retry automatically without some indication from the client application. In the end, we decided to retry the request when an idempotent HTTP request caused by a select statement encounters network failures such as the above. This helped us save an extra roundtrip from the client to ql.io.

This is a reminder of why idempotency is a necessary promise to keep in HTTP server code.

We all like to do code reviews, but in practice code reviews don't happen as meticulously as intended. I know of two reasons:

1. Code reviews create extra work. Print outs, folks sitting in meeting rooms in front of laptops, or submitting diffs to a tool are all extra work for the team.
2. Code reviews are out of band. Though some tools are integrated with source control systems, they are still not part of the commit process. Consequently, they get ignored over time.

But since we moved our project to github, we have made code reviews a part of the flow without adding extra out-of-band work to anyone in the team.

Before we started, code reviews were not our top priority. We had the following goals in stead:

1. Keep the head always stable, tested, and ready for deployment.
2. Keep the other members in the team abreast of code changes happening. This is particularly needed by newer team members.
3. Ensure code quality.
4. KISS and DRY.

In order to make these happen, we adopted the following model.

1. Create repos under an org.

First create an org in github, and then create a team with with push and pull rights, and add members to the team.

2. Use forks for development.

Every member forks the repo, clones the fork, and works against the fork. All code pushes happen in the fork.

3. Avoid cloning the main repo except for merging purposes.

Since forks are used for development, there is no need to fork the main repo. To avoid any confusion, if cloning the main repo is required, clone in a temp directory and nuke it after the work is done.

4. If you make an accidental commit to the master, apologize to the team, and back off your changes.

This happened early on a couple of times, and we decided that the best course is to back off those changes as soon as possible.

5. Submit pull requests when ready.

Once the code is pushed to the fork, submit a pull request.

6. Let someone else review and merge pull requests

This is when code reviews happen. Github sends out notifications to the team members when a pull request is made, and someone else in the team reviews and merges the code. Any discussions and comments happen on the pull request page which leave a trace of all discussions for later reference.

Code reviews are an integral part of this flow. We take some additional steps to make code reviews easy and effective.

1. Keep commits and pull requests small. In particular, avoid submitting pull requests that do fix/change more than one thing.
2. Include tests in each pull request.
3. In most cases, reject a pull request if there is no test included in it.

Large commits usually get less scrutiny due to the mental comprehension load, and the above steps help us pay enough attention to each change.

Thanks to Github!

Here is a quick example to show how to manage an app in a cluster. Node’s (v0.6.x) cluster API is simple, but it takes a few extra steps to make the cluster management scriptable. The example below shows how.

You can then do

node cluster-sample.js

to start app in a cluster, and then

node  stop

to cleanly stop the cluster or

node cluster-sample.js shutdown

to gracefully shutdown.

I am super-excited for to announce the public release of ql.io today.

ql.io is a declarative, evented, data-retrieval and aggregation gateway for HTTP APIs. Simply put, this is our answer to the pain points that I recently described in my APIs are a Pain post.

Here are some key points that I want to highlight about ql.io.

Built on node.js

We chose node.js for its evented programming model and a dynamic language. This choice has kept us productive in the last 5-6 months as we built the core from ground up. I would love to share more details about our experience once I have some concrete numbers to show.

A SQL and JSON based DSL

ql.io uses a SQL and JSON based DSL to write scripts to make HTTP requests. This idea has been tried before, most notably by YQL, and in a slightly different context by unQL. But ql.io pushes the envelope further by cutting down the coding costs of writing fork-join code. See the Language Overview to see how.

Procedural style

ql.io scripts are procedural looking, but the underlying engine executes them in the best order possible based on a dependency analysis. Consider the script below.

a = select a from A;
b = select b from B;
c = select c from C where cparam = "{a.someparam}";
d = select d from D where dparam1 = "{c.someparam}" and dparam2 = "{b.param}";

Each statement is scheduled for execution when all its dependencies have been met. This leads to significant reduction in number of lines of code. One of the first apps I built condensed a 300-line Java app into a 10 lines of script.

Interoperability

My interest in interoperability started with my work at Yahoo! with Bruno. The challenge was how to ensure that HTTP APIs built by decentralized teams interoperate. The first attempt at this problem was a stick that says "you shall conform to such and such rules, or else …" But this is not a viable idea. The second attempt involved placing some minimal constraints such as links and identifiers on each API to make them interoperable. While it sounded like a good idea at the time, it has the same problem as the stick.

The challenge with interoperability is how to make the API Web interoperable without introducing new agreements (or protocols) between them. It turns out that we already agreed on some basic protocols such as TCP, HTTP, URIs, XML, JSON and so on on the Web. What if we turn the problem of interoperability from new protocols to software? That is what we tried in ql.io. We made ql.io as a platform that enables interoperability between decentralized HTTP APIs.

ql.io is capable of interacting with any HTTP speaking entity. You got SOAP APIs? Fine! ql.io can speak to it, and mix the response with responses from other APIs that are more conformant to HTTP. Try this query to find for yourself.

See Announcing ql.io for the complete announcement.

I don’t like writing client apps that use APIs. Writing HTTP client code to talk to APIs is verbose, repetitive, chatty and slow. This is in addition to addressing latency and bandwidth constraints and core functionality of the client app – such as building a snappy UI or supporting some other business use case.

First – the interface mismatch. APIs are almost always designed from the producer’s point of view. If I’m building an API, I would look at my data model, my performance/scalabilty requirements, and design the API to meet the use cases that I think are the most common across all consumers my of my API. That is, my goal would be to maximize the use and reuse of my API. In this process, I am bound to tradeoff special requirements of my consumers and try to stick to a common denominator. This creates a mismatch between what the consumer needs and what I’m offering. Here is an example.

A client app needs to search a list of products to get their IDs, some details and some user generated content like reviews for each product. There are three APIs that the client needs to interact with to get this data – one to search products, one to get details for each product, and another to get reviews for each product.

This is not a made up or hypothetical example. What the client needs is this.

A single API, that takes a keyword, and returns certain fields for each product found – no more or no less.

What the client got are three APIs that give some data that includes what the client needs plus some more. The API design makes sense for the producers of these APIs – search folks are focused on indexing and serving low latecny results, the details API wants to serve all the gazillion details that are relevant to products, and the reviews API wants to focus on low latency writes and potentially stale reads. But, for the consumer, an ideal interface is one that gives just the fields it needs with one single request.

Second – writing client code is slow, repetitive and verbose. For the above example, here is what the client needs to do:

• Call the search API to find products, and extract product IDs from the response.
• Call the details API n times – once per product ID – to get the details, and then extract the fields needed from the response
• Call the reviews API n times – once per product ID – to get reviews, and the extract the fields needed from the response

Unless you got canned SDKs for these APIs, writing code to make 2*n HTTP requests and process the responses can take a while. In one of the Java implementations I looked at, the code was over 300 lines long – that was after excluding the request making and response parsing code which was already factored into into supporting libraries. If the client needs to implement ten such use cases, you got 3000 lines of code just doing data retrieval.

Third – Getting enhancements you need from producer teams slows you down. For my example above, having a bulk API for details and reviews simplifies the client to some extent. But for both good and bad reasons, API changes that the clients need don’t happen immediately. They take time, usually some sprints. Sometimes never. The reality is that teams have their own priorities. Getting a single API for the above won’t likely happen!

Fourth – requests for use cases like this are chatty. Three hundred lines of code may not be a big deal, but making so many HTTP requests is. Here is a common evolution of an implementation. To simplify the discussion, let’s assume that we have bulk APIs for details and reviews.

Take 1: Use blocking IO to make 3 HTTP requests.

Code complexity of this implementation may be low, but it takes sum(t) where t is the time for each API request.

Take 2: If latency is a concern, parallelize the requests. After the search API returns product IDs, you can make 2 requests in parallel and join the responses.

Code complexity now increases, but it only takes t1 + max(t) for the whole retrieval, where t1 is the time taken to search.

Now imagine that the client needs to call yet another API based on review IDs.

Take 3: Orchestrate the retrieval based on dependencies.

In another example I’ve seen recently, a native app makes 17 HTTP requests with some dependencies before painting the UI. The code is over 3000 lines long! Of a team of 3 developers, one developer is dedicated to writing and maintaining this code.

Now move the client farther from the API servers. In addition to the code complexity, the client will have to deal with cost of the network. You may want to move all the request dance to some middle tier that is closer to the API servers than the client.

Fifth – consistent RESTful APIs don’t matter as much as we think. I would love to see every API to be RESTful, consistent, hypertext driven, and more importantly, interoperable. The reality is that getting consistent APIs is hard – particularly those that are done by distributed teams. For the record – I vehemently hate SOAP APIs and the SOAP mindset. I dislike RPC-style operation names tunneled over HTTP. I frown and cringe whenever I see unnecessary custom headers and complicated formats. I wish POST+XML goes away. I wish every API gets rewritten to the modern understanding of HTTP and REST, serving JSON.

But I would rather spend my time enabling interoperability than preaching for consistency.

Hypermedia does not help either. Hypermedia can help navigation among resources, but navigation is not a concern in the above use case. The client app’s challenges are aggregation, parallelization, and orchestration of forks and joins.

So, what can we do about it?

Why would I write a long blog post if I have nothing to offer?

I’m excited to reveal that, at eBay, my team has been working on a new platform to simplify use cases like the above:

• Bring down number of lines of code to call APIs
• Automatically orchestrate API calls as needed
• Create new consumer-centric APIs that wrap existing producer-centric APIs
• Cut down bandwitdh usage and latency

Best of all, this platform will soon be on github. If you are interested in taking a sneak peek and want to provide feedback, please email me (subbu/at/subbu/dot/org) with your github ID. Please see ql.io for more info.

For a soon-to-open sourced nodejs based platform that I have been working on since June this year, I needed a way to send some optional telemetry data back to the browser without mixing it with the real data that the client code is interested in. The first thing I looked at was socket.io. Socket.io’s claim to fame is that it aims to “make real-time apps possible in every browser and mobile device”. But it comes at certain cost that may not be apparent. Since then I gave up on socket.io and switched to the plain WebSocket API on the client side and WebSocket-Node on the server. Here is why.

Cross browser WebSocket fallback is a fallacy. Such fallbacks exist because WebSocket support in browsers continues to limp and the spec itself is not done yet. Interoperability among browsers is not yet possible due to lack of protocol parity. For instance, about a month ago, I had to forego supporting Safari in favor of better experience in latest versions of Chrome and Firefox. But what’s wrong with fallbacks?

WebSocket interactions are bound to a connection – once the client opens a TCP connection, it can send and receive several messages using the WebSocket API. But with fallback approaches that rely on JSONP, XHR and iframes, clients and servers use HTTP for serializing messages.

Therein lies a key difference. With the WebSocket protocol, you can scope any state for the duration of the connection and remain state free across connections. For instance if you want to send some events as they occur to the browser, you can associate the subscription state with the socket. You don’t need to replicate that subscription state across all the server nodes that are serving WebSocket traffic. This makes horizontal scalability possible.

Not so with HTTP based fallbacks where every message could potentially come in over a different TCP connection. To account for this you either need to pin the client to a given node so that the same node handles all connections from a client, or replicate state across all the nodes serving the HTTP traffic. socket.io does the latter by using Redis. Consequently you end up trading off reliability and/or scalability.

This leads to my next point – if scalability is your concern, graceful degradation may just be sufficient in a lot of use cases. You can use feature detection in browsers to decide whether to use WebSocket protocol or not. Here is a simple approach I now use on the client side.

This helps me offer a degraded user experience when WebSocket support is not available, and I don’t have to introduce shared state on the server.

These are the slides from my talk yesterday at restfest.

Every other new OSS project is now awesome, fast, robust or even battle-hardened. If you really want your OSS project to outlive a swarm of retweets over a day, here is some advice:

• Think of some concrete scenarios to help us understand what is awesome/fast/robust/whatever about your stuff.
• Write tests for those scenarios.
• Publish your results and provide as many details as you can about your tests.
• Finally, publish the source code for your tests.

I’m sure that this is hard-work – but you want us to clone your project, or even fork it – right?

I came across MessagePack from multiple sources in recent weeks. MessagePack claims to be a "binary-based efficient object serialization library". The bar chart on MessagePack’s site claims that MessagePack is four timers faster than JSON serialization and deserialization.

Really? If true, it is able to optimize for both speed and space at the same time. I needed to find out. I did a quick test using @pgriess's node-msgpack implementation for Node (if you’re using recent versions of Node, use the fork https://github.com/jmars/node-msgpack in stead). Here are the numbers for 18k of JSON for a total of 10000 test runs on Node v0.4.8 on Ubuntu.

msgpack pack:   12705 ms
msgpack unpack: 1668 ms
json    pack:   1575 ms
json    unpack: 1271 ms

For this size, MessagePack’s serialization is roughly 8 times slower than v8′s JSON serialization. Here are the results of a very small piece of data (48 bytes – same as the one used by Peter’s bench.js) for 10000 runs.

msgpack pack:   108 ms
msgpack unpack: 69 ms
json    pack:   31 ms
json    unpack: 19 ms

In other words, I’m unable to reproduce Peter’s numbers. The differences are likely due to changes in Node and v8 since the time Peter did his tests. You can try it out yourself by doing the following:

git clone https://github.com/jmars/node-msgpack
make
# before the next step, edit bench.js to require lib/msgpack
node bench.js

This is no good. There are other features that concern me as well. It is not clear why the developers of MessagePack chose to bundle together RPC (an application issue), pipelining (a protocol level issue), connection pooling (again an application issue) etc with a format design.