Daemonic Dispatches

FreeBSD ZFS AMIs Now Available

Hi EC2 users,

FreeBSD 12.0-RELEASE is now available as AMIs with ZFS root disks in all 16
publicly available EC2 regions:

[List elided; see the GPG-signed email for AMI IDs]

The ZFS configuration (zpool named "zroot", mount point on /, /tmp,
/usr/{home,ports,src}, /var/{audit,crash,log,mail,tmp}) should match what
you get by installing FreeBSD 12.0-RELEASE from the published install media
and selecting the defaults for a ZFS install.  Other system configuration
matches the FreeBSD 12.0-RELEASE AMIs published by the release engineering
team.

I had to make one substantive change to 12.0-RELEASE, namely merging r343918
(which teaches /etc/rc.d/growfs how to grow ZFS disks, matching the behaviour
of the UFS AMIs in supporting larger-than-default root disks); I've MFCed
this to stable/12 so it will be present in 12.1 and later releases.

If you find these AMIs useful, please let me know, and consider donating to
support my work on FreeBSD/EC2 (https://www.patreon.com/cperciva).  If
there's enough interest I'll work with the release engineering team to add
ZFS AMIs to what they publish.

In the interests of both transparency and following my own advice of "we need to write more how-to-do-foo walkthroughs", here's the process I used to create those images.

1. Launch a FreeBSD 12.0-RELEASE AMI Builder instance; in the us-east-1 region, this is ami-09baac3ede1d33201. Due to issues related to disk device names, the AMI Builder doesn't work on the latest "nitro" instances; and because of what we're going to be doing, we need an instance with lots of RAM — so I used a t2.2xlarge instance for this.
2. Wait about 10 minutes (the AMI Builder boots into RAM but then needs to spend some time extracting FreeBSD before you can connect); then SSH in as ec2-user and su to root (no password needed).
3. The AMI Builder starts us off with FreeBSD installed onto a UFS filesystem, so we need to move those bits safely out of the way:

   mdconfig -a -t swap -s 3G -u 2
   newfs /dev/md2
   mkdir /mdisk
   mount /dev/md2 /mdisk
   tar -czf /mdisk/base.tgz --exclude .snap -C /mnt .
   umount /mnt
   

4. Next, wipe the old UFS bits out of the way and repartition the disk:

   gpart destroy -F ada0
   dd if=/dev/zero bs=128k of=/dev/ada0
   gpart create -s gpt ada0
   gpart add -a 4k -s 512K -t freebsd-boot ada0
   gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
   gpart add -a 1m -t freebsd-zfs -l disk0 ada0
   

5. Create all of the standard FreeBSD/ZFS datasets:

   zpool create -o altroot=/mnt -O compress=lz4 -O atime=off -m none -f zroot ada0p2
   zfs create -o mountpoint=none zroot/ROOT
   zfs create -o mountpoint=/ -o canmount=noauto zroot/ROOT/default
   mount -t zfs zroot/ROOT/default /mnt
   zfs create -o mountpoint=/tmp -o exec=on -o setuid=off zroot/tmp
   zfs create -o mountpoint=/usr -o canmount=off zroot/usr
   zfs create zroot/usr/home
   zfs create -o setuid=off zroot/usr/ports
   zfs create zroot/usr/src
   zfs create -o mountpoint=/var -o canmount=off zroot/var
   zfs create -o exec=off -o setuid=off zroot/var/audit
   zfs create -o exec=off -o setuid=off zroot/var/crash
   zfs create -o exec=off -o setuid=off zroot/var/log
   zfs create -o atime=on zroot/var/mail
   zfs create -o setuid=off zroot/var/tmp
   zpool set bootfs=zroot/ROOT/default zroot
   

6. And now we can extract FreeBSD back onto the newly-ZFS disk:

   tar -xf /mdisk/base.tgz -C /mnt
   

7. We no longer have a UFS filesystem, so we can get rid of the current contents of fstab(5); but we do need some configuration settings to support ZFS instead:

   : > /mnt/etc/fstab
   echo 'zfs_load="YES"' >> /mnt/boot/loader.conf
   echo 'kern.geom.label.disk_ident.enable="0"' >> /mnt/boot/loader.conf
   echo 'kern.geom.label.gptid.enable="0"' >> /mnt/boot/loader.conf
   echo 'vfs.zfs.min_auto_ashift=12' >> /mnt/etc/sysctl.conf
   echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
   

8. And in order to make ZFS expand when we first boot, we need an updated script for /etc/rc.d/growfs:

   svnlite export https://svn.freebsd.org/base/head/libexec/rc/rc.d/growfs
   cp growfs /mnt/etc/rc.d/growfs
   

9. Finally we can "poweroff" the instance

   shutdown -p now
   

   and then use the EC2 Management Console to create an image from the instance (or run aws ec2 create-image).

Thanks to the freebsd-cloud and freebsd-fs mailing lists, and Ben Woods in particular, for testing and feedback on my earlier attempts.