Daemonic Dispatches

The busy freebsd-update server

Based on what I saw when FreeBSD 6.3-RELEASE was announced, I didn't expect any problems -- there was a visible increase in traffic, but it didn't come anywhere close to tying up the server. I hadn't accounted for two important factors:

• Upgrading from FreeBSD 6.x to FreeBSD 7.0 involves more and larger updates than upgrading to FreeBSD 6.3.
• FreeBSD 7.0 is far more popular than FreeBSD 6.3.

On Thursday, February 28th, 885 systems used FreeBSD Update to upgrade to FreeBSD 7.0-RELEASE; of these, 282 were running FreeBSD 7.0 betas or release candidates, 404 were running FreeBSD 6.3, 174 were running FreeBSD 6.2, 13 were running FreeBSD 6.1, 11 were running FreeBSD 6.0, and 1 was running FreeBSD 5.5.

In total, update1.freebsd.org handled 50.1 million HTTP requests -- an average of 58 requests per second -- serving up 130939 distinct files and patches totalling 39.9 GB -- an average data rate of 3.7 Mbps (not counting HTTP/TCP/IP overhead). The effect over this traffic on the server is perhaps best illustrated by the following two MRTG graphs; the first graph shows total and active Apache processes, while the second shows incoming and outgoing bandwidth:

A few notes are in order concerning the above graphs:

1. This server has an uplink capped at 10 Mbps; on several occasions it came very close to that limit.
2. The primary reason it didn't hit the 10 Mbps limit more is that for five hours Apache was at the maximum number of processes I had configured (100) and all of them were busy handling requests. When I woke up on Thursday morning (around 1800 UTC -- 10AM in my time zone) I logged in and increased Apache's process limit.
3. When I wrote the code for converting MRTG bandwidth statistics into 95th percentile and GB/month values, I didn't bother handling leap years.

In short, the FreeBSD Update server was handling about as much traffic as it is capable of handling (at least unless its uplink is upgraded and I switch from Apache to a faster web server), and there were most likely some people who tried to use FreeBSD Update between 1200 UTC and 1800 UTC and found that the server was either very slow or completely unresponsive. If you had problems upgrading, please try again later -- perhaps a random day next week, since as I write this I already see the load increasing as Friday afternoon (UTC) approaches. For myself, I've learned an important lesson: Next time there's a FreeBSD release, I'm going to make sure there are several FreeBSD Update mirrors ready to share the load.

One final addendum: While my bsdiff binary patching tools is usually highly efficient -- for security updates, it routinely provides a greater than fifty-fold reduction in download size -- it performed quite poorly overall at producing patches for upgrading from FreeBSD 6.x to FreeBSD 7.0, providing only a five-fold reduction in download size. Why? Because FreeBSD 6.x uses gcc 3.4, while FreeBSD 7.0 uses gcc 4.2. Such a major change in compiler means that even binaries compiled from identical source code differ throughout, dramatically reducing the potential for bsdiff (or any other binary patch tool) to identify similarities. Let this be a lesson to anyone who uses binary patches to update devices: Think twice before changing compilers!