Daemonic Dispatches

Tarsnap archive checkpointing

To explain why this is an important feature, I need to first provide some background on how tarsnap works. Tarsnap operates with a snapshotted backup model, so each time a new archive is created, it is up to the tarsnap client to figure out which bits need to be uploaded and which bits don't. For reasons of efficiency and security, this means that it's necessary for the tarsnap client to keep a local database telling it which bits have been previously stored; this is kept in the tarsnap "cache" directory and is typically about 0.1% of the size of the data stored. If the tarsnap client dies (usually due to losing its internet connectivity or because the system it's running on crashes) in the middle of creating an archive, this local database would not reflect the state of information stored on the tarsnap server; so to ensure that tarsnap never has corrupted archives or "orphaned" data, the tarsnap client-server protocol is transactional: If the tarsnap client dies in the middle of creating an archive, the uploaded data will not be "committed", but will instead be deleted the next time tarsnap runs -- making the server state match the client state. That is, if you don't have any checkpoints set.

With the new --checkpoint-bytes <N> option to tarsnap, the tarsnap client will create a checkpoint after each <N> bytes of uploaded data. Creating a checkpoint involves operations at both the client and the server side: On the client side, tarsnap writes out to disk a new database indicating all of the bits which exist on the server including all the bits contained in the current archive up to that checkpoint; on the server side, the checkpoint is recorded for future reference. If tarsnap is run after a crash and it discovers that a checkpoint had been marked, it asks the server to commit that -- at which point the server knows to commit all of the data stored prior to the checkpoint, but to discard anything uploaded after that point (since any data which was uploaded after the most recent checkpoint will not be included in the client's database of uploaded bits). When an archive checkpoint is committed, it is given a ".part" name, just like archives truncated due to hitting a bandwidth limit or SIGQUIT -- so there is no danger of an archive checkpoint being confused with a successfully completed archive.

In the past month there have been a few instances where people have left tarsnap uploading data for several days only to end up losing all of that progress; obviously they weren't very happy about this, and neither was I. With archive checkpointing, this is no longer something to worry about (as long as people use it -- but I've added this to the provided sample configuration file, so hopefully this option will not be ignored).

Now for the bad news: Because older versions of tarsnap don't support checkpoints, they don't know what to do if a checkpoint is left over from a previous run of tarsnap. As a result, if you use tarsnap 1.0.20 or higher to create an archive, you should never use an earlier version of tarsnap to create or delete archives on that machine again. Archives created with tarsnap 1.0.20 can still be read using pre-1.0.20 versions of the tarsnap client; it's only creating and deleting archives (i.e., operations which use the tarsnap cache directory and need to know which bits are on the server) which shouldn't be performed with old versions of the tarsnap client.

As always, the new version of tarsnap is available from the tarsnap beta testing download page.

Next up on the list of frequently requested features: Passphrased key files.