Daemonic Dispatches

On the use of a life

Okay, so, what do we think about TarSnap? Dude was obviously a genius, and spent his time on backups instead of solving millennium problems. I say that with the greatest respect. Is this entrepreneurship thing a trap?

First, to dispense with the philosophical argument: Yes, this is my life, and yes, I'm free to use — or waste — it however I please; but I don't think there's anything wrong with asking if this is how my time could be best spent. That applies doubly if the question is not merely about the choices I made but is rather a broader question: Is our society structured in a way which encourages people to make less than the greatest contribution they could?

That said, I do object somewhat to the premise of the question — specifically the statement that I "spent [my] time on backups".

On one hand, it is true: Tarsnap has been my full time job since 2006. I do occasional consulting — more in the early years than I have recently — but financially speaking that's a rounding error; it's Tarsnap which pays the bills (including allowing me to buy the house which I'll be moving into next week). On the other hand: My work on Tarsnap has extended considerably into adjacent fields.

In 2009, having had many users ask for passphrase-protected Tarsnap key files, and having determined that the current state of the art of password based key derivation was sorely lacking, I invented scrypt — and in the process, opened up a whole new field of cryptography. Sure, I was doing this because it was something I could do to make Tarsnap more secure; but it would be a stretch to place this under the umbrella of "spending my time working on backups".

In 2011, wanting to connect daemons on disparate hosts together securely, and not being happy with the existing TLS-based options, I wrote spiped. While I think it remains largely underappreciated by the world at large, I nonetheless consider it a significant contribution to computer security — and, like scrypt, while I created it to serve Tarsnap's needs, it would be a stretch to place such a general-purpose open-source tool under the narrow umbrella of "working on backups".

Around the same time, I started working on kivaloo, my high performance key-value data store. This may be the least used of all of the software I've written — I'm not aware of anyone else using it at present (although being open source software that doesn't necessarily preclude the possibility) — but I consider it to be some of my best code and I think it may become used in more niches than merely Tarsnap in the future.

Starting in 2006, and accelerating significantly after Amazon launched the "M3" family of HVM-enabled EC2 instances in 2012, I've been creating and maintaining the FreeBSD/EC2 platform. While I don't have any precise statistics on its usage, a survey last year found that 44% of people running FreeBSD in the cloud use Amazon EC2; so — despite the fact that only 22 people currently provide sponsorship for my efforts — it's clear that my work here has been productive. Again, while I was working on this because I wanted to run FreeBSD in EC2 for Tarsnap, I don't think it can be placed entirely into the category of "working on backups".

Of course, the question at hand isn't whether I've done anything useful, but rather whether this was the most useful way I could have spent these years. Judging by the reference to the Millennium Problems, I imagine that the specific alternative they had in mind was a research career; indeed, between my Undergraduate studies in number theory under the late Peter Borwein and my Doctoral studies in Oxford I might have considered seriously working on the Birch and Swinnerton-Dyer conjecture had my life taken a different path. (A very different BSD from the one with which I am currently involved!)

So why am I not an academic? There are many factors, and starting Tarsnap is certainly one; but most of them can be summarized as "academia is a lousy place to do novel research". In 2005, I made the first publication of the use of shared caches in multi-threaded CPUs as a cryptographic side channel, and in 2006 I hoped to continue that work. Having recently received my doctorate from Oxford University and returned home to Canada, I was eligible for a post-doctoral fellowship from Canada's National Sciences and Engineering Research Council, so I applied, and... I didn't get it. My supervisor cautioned me of the risks of doing work which was overly novel as a young academic: Committees don't know what to make of you, and they don't have any reputational prior to fall back upon. Indeed, I ran into this issue with my side channel attack: Reviewers at the Journal of Cryptology didn't understand why they were being asked to read a paper about CPU design, while reviewers at a computer hardware journal didn't understand why they were being asked to read about cryptography. It became clear, both from my own experiences and from advice I received, that if I wanted to succeed in academia I would need to churn out incremental research papers every year — at very least until I had tenure.

In many ways, starting my own company has given me the sort of freedom which academics aspire to. Sure, I have customers to assist, servers to manage (not that they need much management), and business accounting to do; but professors equally have classes to teach, students to supervise, and committees to attend. When it comes to research, I can follow my interests without regard to the whims of granting agencies and tenure and promotion committees: I can do work like scrypt, which is now widely known but languished in obscurity for several years after I published it; and equally I can do work like kivaloo, which has been essentially ignored for close to a decade, with no sign of that ever changing.

Is there a hypothetical world where I would be an academic working on the Birch and Swinnerton-Dyer conjecture right now? Sure. It's probably a world where high-flying students are given, upon graduation, some sort of "mini-Genius Grant". If I had been awarded a five-year $62,500/year grant with the sole condition of "do research", I would almost certainly have persevered in academia and — despite working on the more interesting but longer-term questions — have had enough publications after those five years to obtain a continuing academic position. But that's not how granting agencies work; they give out one or two year awards, with the understanding that those who are successful will apply for more funding later.

In short, academic institutions systemically promote exactly the sort of short-term optimization of which, ironically, the private sector is often accused. Is entrepreneurship a trap? No; right now, it's one of the only ways to avoid being trapped.