Secure FreeBSD ports tree updating

Portsnap is a system for securely downloading and updating a compressed snapshot of the FreeBSD ports tree, and using this compressed snapshot to extract or update a (uncompressed) copy of the ports tree.

Historically, most people have used CVSup to keep their ports tree up to date, but CVSup has a number of limitations:

CVSup is insecure. The protocol uses no encryption or signing, and any attacker who can intercept the connection can insert arbitrary data into the tree you are updating.
CVSup isn't end-to-end. Related to the previous point, this means that anyone who can compromise a CVSup mirror can feed arbitrary data to the people who are using that mirror.
CVSup isn't designed for frequent small updates. While CVSup is very good at distributing CVS trees, and is very efficient for updating a tree which has been significantly changed (eg, by a month or more of commits), it transmits a list of all the files in the tree, which makes it quite inefficient if only a few files have changed.
CVSup uses a custom protocol. This can cause problems for people behind firewalls -- outgoing connections on port 5999 need to be permitted -- and it needs a heavyweight server (cvsupd).

Portsnap avoids these problems by operating over HTTP (using FreeBSD's fetch(1) utility and a new experimental pipelined HTTP client), signing the snapshots using OpenSSL, and using more sophisticated delta compression to distribute the snapshots.

In order to compare the bandwidth usage of cvsup and portsnap for frequent updates of the FreeBSD ports tree, I used each method to update across a 58-hour period from 12:15 AM on January 13th, 2005 to 10:15 AM on January 15th, 2005. While cvsup used 6388kB of bandwidth (2418kB up, 3969kB down), portsnap only used 370kB (108kB up, 262kB down) -- an improvement of over a factor of 17. In addition, cvsup used 212 seconds of cpu time compared to portsnap's 32 seconds of cpu time -- a difference of over a factor of six -- and 534 seconds of wall-clock time compared to portsnap's 75 seconds of wall-clock time (but since the wall-clock time depends upon network congestion and server loads, it isn't particularly significant).

Portsnap stores a compressed snapshot of the ports tree (around 40 MB) on disk, by default in /usr/local/portsnap. This compressed snapshot can then be extracted as needed (eg, into /usr/ports).

Portsnap is in the FreeBSD base system for all versions from 5.5 upwards (including 6.0, which was released before 5.5); users of earlier FreeBSD releases can install portsnap from the ports tree (sysutils/portsnap).

I have a nice portsnap usage graph showing the number of systems running each version of portsnap (as approximated by the updates they fetch).

Usage

Install sysutils/portsnap from the FreeBSD ports tree.
To fetch a compressed snapshot, or update your current compressed snapshot, run `portsnap fetch`.
To extract the ports tree, run `portsnap extract`.
After extracting a ports tree, to update it to reflect changes in the compressed snapshot, run `portsnap update` -- this is much faster than `portsnap extract` because it avoids extracting directories which haven't changed.

Please note that when extracting a copy of the FreeBSD ports tree, portsnap will remove existing files. If you have any local changes (eg, extra patches which you've added into a port's files/ directory) then you'll have to put those back after updating the ports tree.

Please send any comments about this to me at "cperciva" at this domain.