FreeBSD on EC2 via defenestration
On December 13th I announced the availability of FreeBSD on EC2 t1.micro instances, and on March 22nd I announced the availability of FreeBSD on EC2 cc1.4xlarge "cluster compute" instances; but for most people, these options were either too small or much too large. Today I am happy to announce the availability of FreeBSD on EC2 m1.large and m1.xlarge "standard" instances; m2.xlarge, m2.2xlarge, and m2.4xlarge "high-memory" instances; and c1.xlarge "high-CPU" instances.The primary factor which has been limiting FreeBSD availability on EC2 until now is FreeBSD's poor paravirtualized Xen support. FreeBSD's 32-bit paravirtualized Xen support is far from perfect; and FreeBSD doesn't have 64-bit paravirtualized Xen support at all. Aside from the recently launched "cluster compute" instances, EC2 uses paravirtualization for all UNIX instances; and at the present time, t1.micro instances are the only ones compatible with FreeBSD's paravirtualized Xen support.
EC2 Windows instances, on the other hand, are launched in HVM (Hardware Virtual Machine) mode — which FreeBSD supports very well. If we can somehow trick EC2 into launching FreeBSD the same way as it launches Windows, we can run FreeBSD on a wide range of instance types. How can we trick EC2? Take advantage of the fact that Elastic Block Store disks can be detached from EC2 instances and reattached to different instances, and replace the boot disk of a "Windows" instance with a disk containing FreeBSD. In other words, defenestrate the EC2 instance. (Note to pedants: While "defenestrate" usually means "to throw out of a window", etymologically it could equally mean "to throw windows out" — and the Oxford English Dictionary does show a recorded use in this sense dating from 1927.)
I actually tried this over a year ago without success; I'm not sure why it didn't work at that point, but a few weeks ago a little bird suggested that I should try it again, and this time it worked... modulo a slight incompatibility in the Xen block device parameter negotiation protocol which I needed to patch around. Once I had a "Windows" instance running FreeBSD, turning it into an AMI was easy: EC2 provides a "Create Image" operation which takes a running instance and bundles it — including its attached disks and metadata such as the fact that it is a "Windows" instance — into a new AMI.
There is one downside to this approach, however: Because, as far as EC2 is concerned, instances launched from the defenestrated AMI are still running Windows, they are billed at the hourly rates for Windows instances — anywhere from 24% to 71% higher than the pricing for non-Windows instances. We can hope that some day Amazon will make HVM available without the need for defenestration; but until then, FreeBSD users will be paying the Windows tax on the cloud just like we do on laptops.
Still, even with the Windows tax, I imagine "filling in the gap" between t1.micro instances and cluster compute instances will prove useful to many FreeBSD users. In the US-East region FreeBSD 8.2b-RELEASE (8.2 plus my patches) is now available for 64-bit instances as ami-c69862af; go spawn some daemons!
The spiped secure pipe daemon
Three weeks ago, Bump Technologies released their new Scalable TLS Unwrapping Daemon. As someone who has written in the past about using stunnel to unwrap HTTPS connections while keeping OpenSSL away from other sensitive code, I found this quite exciting: STUD is just a few hundred lines of code, compared to almost ten thousand lines for stunnel. Since complexity is highly correlated with insecurity, I trust a simple daemon which only unwraps SSL/TLS far more than I trust a multifunctional monolith.A few days later, I was looking at my kivaloo data store, contemplating the next steps I should take with it, and I decided that it was time to add some security. After all, I'm going to be using it in Tarsnap, and the Tarsnap web server needs to access user accounting data so that people can see their account balances and recent usage; since I want the web front-end kept separate from the core backup service code, traffic will need to go over the internet. My first thought was to build this into kivaloo directly, but then I reconsidered: Why not write a generic secure pipe daemon, and keep the complexity of encryption and authentication out of kivaloo? Enter spiped.
Spiped is a tool for creating secure tunnels using a pre-shared key. You connect to a socket on one machine, and the "client-side" spiped you're running on that system opens a connection to the "server-side" spiped running on another machine, which then opens a connection to the predefined target. The two spiped daemons use the pre-shared key to securely negotiate encryption and authentication keys; and then they shuttle data back and forth. Thanks to select(2) and non-blocking network I/O, spiped can easily handle hundreds of simultaneous connections within a memory footprint of a few MB.
In effect, spiped is a replacement for 'ssh -L'; but it has a few advantages:
- You don't need to be running sshd. Given the relative complexities of the two daemons, it's vastly more likely that your server will be compromised via a vulnerability in sshd than via a vulnerability in spiped. (In fact, if you need to run SSH for the benefit of interactive logins, you're probably better off having sshd only listening on 127.0.0.1 and connecting to it via spiped.)
- You don't need a persistent connection. 'ssh -L' works by holding a single TCP connection open and multiplexing incoming connections over it; as a result, if you lose your internet connection you'll need to re-establish the tunnel. (On my laptop, I've been sending email via an SSH tunnel for years — and I have a cron job which runs every minute to re-establish the tunnel if it has died.)
- No more checking SSH host keys. If you can connect via spiped, you've got the right keys — if the "shared" keys used by the spiped client and server don't match, the connection-creation handshaking will fail and connections will be immediately closed.
I sent the spiped code around to a few people on Friday, and I haven't heard any major complaints; so I've released spiped version 1.0.0 on the spiped website today; there is also an SVN repository for anyone interested in keeping track of future development (not that I expect there will be very much) and a mailing list for discussions about spiped.
Enjoy!