Daemonic Dispatches

More details about the NetSweeper DoS attack.

After some exchanging a few emails with a developer at NetSweeper, it has become clear that what appeared to be a denial of service attack was indeed, as my NetSweeper corollary suggested, simply the behaviour of a very broken robot. The long series of requests made (3224 of them, over a 40 minute period) was a partial replay of legitimate traffic from a system running portsnap -- with the critical difference that the system running portsnap waited until one connection had closed before opening another.

Aside from the protocol violations I listed before (the rapid-fire requests from a system which is neither an interactive user-agent nor a proxy, the fraudulent User-Agent header, and the inobservance of robots.txt), a tcpdump of packets between my server and NetSweeper uncovered two serious bugs in their "categorization engine":

1. NetSweeper only ever sends one request per HTTP connection, and does not send a "Connection: close" header; yet it describes itself as an HTTP/1.1 client. To quote RFC 2616,

   HTTP/1.1 applications that do not support persistent connections MUST include the "close" connection option in every message.

   To fix this, NetSweeper should either start including that header line in its requests or label itself as an HTTP/1.0 client. Both fixes are trivial and would have the effect of allowing Apache to close the connection (and free resources) immediately after sending its response.
2. NetSweeper does not shutdown sockets properly. After Apache gave up waiting for further requests and sent a FIN packet to close the connection, NetSweeper acknowledged the FIN (placing Apache's end of the connection into the FIN_WAIT_2 state, and NetSweeper's end of the connection into the CLOSE_WAIT state)... and then did nothing for 15 minutes. Finally, 917 seconds after Apache closed its end of the connection, NetSweeper closed its end by sending a FIN packet (to which it received a RST response, since FreeBSD drops idle half-closed connections after a minute).

   The fix for this is also very simple: There's a missing shutdown(2) call somewhere in NetSweeper's source code.

I've never been very favourably inclined towards web filtering engines, on general principle; but based on the level of coding and standards non-compliance I've seen here, I'm surprised that at least one of them functions at all.

blog comments powered by Disqus