Daemonic Dispatches

FreeBSD/EC2 history

A couple years ago Jeff Barr published a blog post with a timeline of EC2 instances. I thought at the time that I should write up a timeline of the FreeBSD/EC2 platform, but I didn't get around to it; but last week, as I prepared to ask for sponsorship for my work I decided that it was time to sit down and collect together the long history of how the platform has evolved and improved over the years.

Normally I don't edit blog posts after publishing them (with the exception of occasional typographical corrections), but I do plan on keeping this post up to date with future developments.

• August 25, 2006: Amazon EC2 launches. It supports a single version of Fedora Core 4; FreeBSD is not available.
• December 13, 2010: I manage to get FreeBSD running on EC2 t1.micro instances.
• March 22, 2011: I manage to get FreeBSD running on EC2 "cluster compute" instances.
• July 8, 2011: I get FreeBSD 8.2 running on all 64-bit EC2 instance types, by marking it as "Windows" in order to get access to Xen/HVM virtualization. (Unfortunately this meant that users had to pay the higher "Windows" hourly pricing.)
• January 16, 2012: I get FreeBSD 9.0 running on 32-bit EC2 instances via the same "defenestration" trick. (Again, paying the "Windows" prices.)
• August 16, 2012: I move the FreeBSD rc.d scripts which handle "EC2" functionality (e.g., logging SSH host keys to the console) into the FreeBSD ports tree.
• October 7, 2012: I rework the build process for FreeBSD 9.1-RC1 and later to use "world" bits extracted from the release ISOs; only the kernel is custom-built. Also, the default SSH user changes from "root" to "ec2-user".
• October 31, 2012: Amazon launches the "M3" family of instances, which support Xen/HVM without FreeBSD needing to pay the "Windows" tax.
• November 21, 2012: I get FreeBSD added to the AWS Marketplace.
• October 2, 2013: I finish merging kernel patches into the FreeBSD base system, and rework the AMI build (again) so that FreeBSD 10.0-ALPHA4 and later use bits extracted from the release ISOs for the entire system (world + kernel). FreeBSD Update can now be used for updating everything (because now FreeBSD/EC2 uses a GENERIC kernel).
• October 27, 2013: I add code to EC2 images so that FreeBSD 10.0-BETA2 and later AMIs will run FreeBSD Update when they first boot in order to download and install any critical updates.
• December 1, 2013: I add code to EC2 images so that FreeBSD 10.0-BETA4 and later AMIs bootstrap the pkg tool and install packages at boot time (by default, the "awscli" package).
• December 9, 2013: I add configinit to FreeBSD 10.0-RC1 and later to allow systems to be easily configured via EC2 user-data.
• July 1, 2014: Amazon launches the "T2" family of instances; now the most modern family for every type of EC2 instance (regular, high-memory, high-CPU, high-I/O, burstable) supports HVM and there should no longer be any need for FreeBSD users to pay the "Windows tax".
• November 24, 2014: I add code to FreeBSD 10.2 and later to automatically resize their root filesystems when they first boot; this means that a larger root disk can be specified at instance launch time and everything will work as expected.
• April 1, 2015: I integrate the FreeBSD/EC2 build process into the FreeBSD release building process; FreeBSD 10.2-BETA1 and later AMIs are built by the FreeBSD release engineering team.
• January 12, 2016: I enable Intel 82599-based "first generation EC2 Enhanced Networking" in FreeBSD 11.0 and later.
• June 9, 2016: I enable the new EC2 VGA console functionality in FreeBSD 11.0 and later. (The old serial console also continues to work.)
• June 24, 2016: Intel 82599-based Enhanced Networking works reliably in FreeBSD 11.0 and later thanks to discovering and working around a Xen bug.
• June 29, 2016: I improve throughput on Xen blkfront devices (/dev/xbd*) by enabling indirect segment I/Os in FreeBSD 10.4 and later. (I wrote this functionality in July 2015, but left it disabled by default a first because a bug in EC2 caused it to hurt performance on some instances.)
• July 7, 2016: I fix a bug in FreeBSD's virtual memory initialization in order to allow it to support boot with 128 CPUs; aka. FreeBSD 11.0 and later support the EC2 x1.32xlarge instance type.
• January 26, 2017: I change the default configuration in FreeBSD 11.1 and later to support EC2's IPv6 networking setup out of the box (once you flip all of the necessary switches to enable IPv6 in EC2 itself).
• May 20, 2017: In collaboration with Rick Macklem, I make FreeBSD 11.1 and later compatible with the Amazon "Elastic File System" (aka. NFSv4-as-a-service) via the newly added "oneopenown" mount option (and lots of bug fixes).
• May 25, 2017: I enable support for the Amazon "Elastic Network Adapter" in FreeBSD 11.1 and later. (The vast majority of the work — porting the driver code — was done by Semihalf with sponsorship from Amazon.)
• December 5, 2017: I change the default configuration in FreeBSD 11.2 and later to make use of the Amazon Time Sync Service (aka. NTP-as-a-service).
• July 17, 2018: I teach the FreeBSD AMI-building code to send Amazon SNS notifications when new AMIs are published; users can subscribe to the topic arn:aws:sns:us-east-1:782442783595:FreeBSDAMI.
• August 26, 2018: I disable the atkbd0 and atkbdc0 devices in EC2 AMIs; this has the effect of skipping the probing and attaching of the PS/2 mouse (not present) and keyboard (emulated, but not usable) and shaves 2.5 seconds off the kernel boot time.
• February 16, 2019: I make unofficial ZFS AMIs available.
• April 4, 2019: The first arm64 FreeBSD AMIs are available.
• October 2, 2019: I switch FreeBSD AMIs from using a "dual-dhclient" shell script (which spawns dhclient processes for IPv4 and IPv6) to using a daemon; this makes it possible to stop and restart the "dhclient" process and have it behave as expected.
• October 15, 2019: Support for hot plug/unplug of devices on amr64 Nitro instances is added by John Baldwin. This functionality was already available on Xen-based EC2 instances, but was lost with the shift from Xen devices to NVMe disks passed through on a (partially virtualized) PCI bus. (This code was not merged into the stable/12 branch until too late to be included in FreeBSD 12.1-RELEASE, but will be present in 12.2-RELEASE.)
• November 4, 2019: FreeBSD 12.1-RELEASE ships; this is the first release to have support for EC2 arm64 instance types.
• November 22, 2019: FreeBSD/ARM 12 is added to the AWS Marketplace.
• January 27, 2020: I release imds-filterd, which makes it possible to use IAM Roles for EC2 without exposing credentials to unauthorized processes.
• April 8, 2020: SNS notifications about new FreeBSD AMIs now contain an "Architecture" field, in order to more easily distinguish between amd64 and arm64 AMIs.
• April 27, 2020: I switch FreeBSD 13.0 over to use_nvd=0, enabling the use of the more advanced nda disk driver.
• May 4, 2020: I release a FreeBSD port of the ebsnvme-id tool.
• May 17, 2020: I add ebsnvme-id to AMIs, enable a new /dev/aws/disk/ tree which allows disks to be identified based on their EBS volume IDs, "Linux device names", or ephemeral disk serial numbers, and add support for simplified automounting of EFS disks.
• June 9, 2020: After months of debugging efforts, a long-standing stability issue in the ENA driver is fixed in an Errata Notice.
• June 30, 2020: EC2 AMIs now use Portsnap and FreeBSD Update mirrors hosted in AWS.
• September 13, 2020: EC2 AMIs now spawn the DHCPv6 client via the "router solicitation daemon" rtsold.
• June 21, 2021: The -n flag is now set in /boot.config, instructing gptboot to not wait for a keypress; this speeds up the boot process by 3 seconds.

The current status

The latest FreeBSD release (13.0) supports: IPv6, Enhanced Networking (both generations), Amazon Elastic File System, Amazon Time Sync Service, both consoles (Serial + VGA), and every EC2 instance type except arm64 "bare metal" (although I'm not sure if FreeBSD has drivers to make use of the FPGA or GPU hardware on those instances).

On arm64, device hot-plug does not currently work; nor does "power management" via the EC2 API — "reboot" has no effect while "shutdown" does nothing until EC2 times out and forcibly halts the instance.

There is a known issue affecting timekeeping on T3 instances.

When a FreeBSD/EC2 instance first launches, it uses configinit to perform any desired configuration based on user-data scripts, and then (unless configinit is used to change this) resizes its root filesystem to fit the provided root disk, downloads and installs critical updates, sets up the ec2-user user for SSH access, and prints SSH host keys to the consoles.

If there's something else you think FreeBSD should support or a change you'd like to see to the default configuration, please let me know.