BigPipe Done in Node.js

I’ve implemented a proof of concept of BigPipe in Java (should run as-is in every servlet container):

See his blog post for the Java servlet class.

Here is the same (or more?) written in Node.js.

var http = require('http');
var sys = require('sys');
var url = require("url");

http.createServer(function(request, response) {
    // Write the document
    response.writeHead(200, {"Content-Type" : "text/html"});
    response.write("<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\"" +
            "   \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">");
    response.write("<script type=\"text/javascript\">function arrived(id,text) { var b=document.getElementById(id); b.innerHTML = text; }</script>");
    response.write("</head><body><div>Progressive Loading");
    for(var i = 0; i < 6; i++) {
        response.write("<div id='" + i + "'>&;lt;/div>");
    }
    response.write("</div>");

    // Now the snippets
    var down = 6;
    for (i = 0; i < 6; i++) {
        var proxy = http.createClient(2000, "localhost");
        var proxyRequest = proxy.request("GET", "/?id=" + i,
                                         {"host" : "localhost"});
        proxyRequest.addListener('response', function (proxyResponse) {
            --down;
            proxyResponse.addListener('data', function(chunk) {
                response.write(chunk, 'binary');
            });
            proxyResponse.addListener('end', function() {
                if(down == 0) {
                    response.end();
                }
            });
        });
        proxyRequest.close();
    }
    response.write("</body></html>");

}).listen(8080);

http.createServer(function(request, response) {
    // Some delay upto upto 2 seconds
    var delay = Math.round(Math.random() * 2000);

    setTimeout(function() {
        var params = url.parse(request.url, true);
        var id = params.query.id;
        response.writeHead(200, {"Content-Type" : "text/html"});
        var content = "<span>Content of Module " + id + "</span>";
        response.write("<script>" +
            "arrived('" + id + "', '" + content + "');" +
             "</script>");
        response.close();
	}, delay);
}).listen(2000);

I've no comments on the technique itself. The basic idea may have been implemented a number of times in different languages. The mileage varies based on the language/environment used.

Tagged as: node.js

{ 5 comments… read them below or add one }

derjan (@derjan) (@derjan) March 13, 2011 at 9:46 pm

@stilkov for the next time you should add one about making #REST faster with Facebook’s BigPipe http://goo.gl/Lg7mO

A random link (@randlink) August 17, 2011 at 12:54 am

BigPipe Done in Node.js http://j.mp/ndzI0l #link

Rares Mirica September 2, 2011 at 10:19 am

Does the browser begin executing and rendering the fragments before closing the connection at line 52: response.close()?

I have experimented a little myself trying to replicate bigpipe in nodejs without success because the browser wouldn’t start executing the code before the response was fully sent, I believe this has something to do with content-length header or the lack thereof.

Rares Mirica September 2, 2011 at 11:05 pm

Later edit: i meant response.end at line 30

Subbu Allamaraju September 2, 2011 at 11:13 pm: Have you verified that the JS is written before the end body tag?