Joey Nelson Blog Embarrassment

We’ve been using Node.js at work to run a broadcast-only comet server. Node is awesome but it’s still a fairly young project. There is some good documentation, but it assumes that you kind of know what you’re doing. There are some okay tutorials out there but none of them really seem to go very far beyond basic hello world examples or minimal chat servers.

If you’re used to working in event-oriented Javascript and can wrap your head around using it on the server side, that’s pretty much half the battle. But there are a still a few things that can trip you up.

Our comet server is pretty simple. Every once in a while, the web server will POST a JSON object containing a message for a particular channel. The comet server will receive the message, figure out which clients are subscribed to that channel and send them the message. This is how we had the /publish path set up:

'/publish': function(req, res){
	req.addListener("data", function(data) {
		//parse data and do stuff with it
	});
}

This worked perfectly in development, but once we started load testing we started getting a bunch of error messages about invalid JSON. This is because Node receives the POST data in chunks and fires the "data" event for each chunk. With only a little bit of traffic, that first chunk would always contain everything we needed. But once the traffic got crazy, we were only getting one piece of the request.

Node provides an "end" event, so all we have to do is keep writing those chunks to a string as we get them. So this is how we create the server:

var server = http.createServer(function(req, res) {
	req.setEncoding("utf8");
	req.content = '';
 
	paths[req.url.pathname].apply(this, [req, res]);
}).listen(80);

And this is the /publish entry in paths:

'/publish': function(req, res){
	req.addListener("data", function(chunk) {
		req.content += chunk;
	});
 
	req.addListener("end", function() {
		//parse req.content and do stuff with it
	});
}

So obviously this is something that’s not too hard to figure out, but it’s very easy to do it the wrong way and not realize it until things start falling apart in production.