How to build FreeBSD/EC2 images
I have been building FreeBSD/EC2 images for the past three years, and based on the email I have been receiving, most people have been either using these images directly or modifying them to create images which suit their needs. However, there are some people who want to build their own images ab initio — most often, companies which have products built on "customized" versions of FreeBSD — and while I have helped a few people do this, it's better if my help is not needed. To this end, earlier today I published my code for building FreeBSD AMIs. At its core, this process has two steps: First, building a disk image; and second, turning it into an AMI.The process of building a disk image starts out the same as for any other platform: Slice and partition a disk, create a filesystem, then extract the FreeBSD bits from a release ISO. Prior to FreeBSD 10.0 this was a more complex task, since a custom kernel needed to be built; but FreeBSD now has Xen HVM support in the GENERIC kernel, which ships on release ISOs.
Having installed a "clean" FreeBSD system into a disk image, I then add extra code to support using the image in Amazon EC2. This consists of four ports in the FreeBSD ports tree:
- sysutils/ec2-scripts: This is the only truly essential port, as it contains code for configuring the system to allow logins using the SSH public key provided via EC2; logging the SSH host keys to the EC2 console; and handling configinit data to allow an EC2 instance to be provided with configuration parameters when it is launched.
- sysutils/firstboot-freebsd-update: This port downloads and installs updates to the FreeBSD base system when the EC2 instance first boots. This is important since there may be many important security and errata updates between when the AMI is created and when an instance is launched months or years later.
- sysutils/firstboot-pkgs: This port downloads and installs FreeBSD packages (code compiled from the ports tree) when the EC2 instance first boots. This makes it possible for an instance to be launched with newer code than was available when the AMI was created; and (together with the functionality provided by configinit) makes it possible to launch a new FreeBSD instance and have it available moments later with a set of software installed and ready to use.
- sysutils/panicmail: This port is not particularly specific to EC2, although I have had an earlier version of the code in my EC2 images for a long time. The panicmail system provides an automated mechanism for reporting of FreeBSD kernel panics to a centralized location, and we're hoping it will prove useful in identifying and fixing stability issues — both those related to the FreeBSD/EC2 platform, and those occuring in other FreeBSD environments.
Having created a disk image — a task which my FreeBSD/EC2 build code performs in EC2, writing directly to an Elastic Block Store volume, for convenience — the second step is to turn this into an EC2 AMI. This is made more complex by a bit of history: When Amazon EC2 was first created using Xen, there was only one virtualization method available, namely "paravirtualization" ("PV" for short); so that's what EC2 used. Later, as virtualization gained popularity, Intel and AMD added support into their CPUs for "hardware virtualization", and Xen's HVM mode was born; but EC2 stuck with what was working for them, namely PV mode.
Unfortunately, FreeBSD needs Xen HVM mode (or some later hybrid modes). In 2008, Amazon announced support for running Microsoft Windows inside EC2, which also needed Xen HVM mode but had higher prices due to the Windows licenses involved; but the operating system world — not just FreeBSD, but also Linux vendors — continued to move towards taking advantage of hardware virtualization, and finally in July 2010 Amazon announced their first "Cluster Compute" instances with HVM support; for the past three years, all of the new EC2 instance types have fully supported HVM. For this reason, from a single disk image I build two FreeBSD/EC2 AMIs: One which pretends to be Windows (and has the higher "Windows" hourly pricing) but can run on all instance types; and one which can only run on "modern" EC2 instances types but has the standard "UNIX" hourly pricing.
In an ideal world, creating a "Windows" AMI from a disk image would just require a single API call to add the "Windows" label; alas, EC2 is not ideal. Instead, I use a trick I call defenestration: I launch a Windows EC2 instance; "throw the windows out" by detaching and deleting its boot disk; attach the FreeBSD disk image as the root disk; and then tell EC2 to create an AMI from the "Windows" instance. Since EC2 knows that the instance was launched from a Windows AMI, it affixes the "Windows" label automatically (if it didn't, rebundling Windows instances wouldn't work), providing the HVM environment FreeBSD needs — as well as the "Windows tax" we don't.
Creating the "UNIX HVM" image, on the other hand, is just a single API call — but an API call which until a few months ago most people couldn't make. For some reason (I'm sure there was one, but I can't imagine what it was) when Amazon first made HVM available on Cluster Compute instances, they decided to keep the process for creating HVM images secret. Fortunately, the right people within Amazon knew about my work on FreeBSD/EC2, and — under a Non-Disclosure Agreement — provided me with special EC2 "API tools" code and set an "allowed to create HVM images" flag on my AWS account. (This is far from the only thing Amazon has quietly done to help me, but it's something they've confirmed I can now talk about publicly.) Fortunately, Amazon changed their stance on HVM image creation a few months ago, and the VirtualizationType parameter to the RegisterImage API call is now publicly available and documented — which is why I'm able to share my EC2 image building code publicly rather than limiting it to people who have signed Amazon NDAs.
While I'm currently building both "Windows" and "UNIX" images, I imagine that at some point I will stop building the "Windows" images; as I mentioned above, all the new EC2 instance types since 2011 support running FreeBSD without the "Windows" label, and now that Amazon has released the m3.medium EC2 image type at a price of $0.11/hour there aren't many situations left where it would make sense to run one of the older instance types.
I think most FreeBSD users will still find it easiest to continue using the FreeBSD/EC2 images which I am publishing, but for those people who want to build their own images: the code is out there — have fun!