Daemonic Dispatches

Some new FreeBSD/EC2 features: EFS automount and ebsnvme-id

As my regular readers will be aware, I've been working on and gradually improving FreeBSD/EC2 for many years. Recently I've added two new features, which are available in the weekly HEAD and 12-STABLE snapshots and will appear in releases starting from 12.2-RELEASE.

The first of these features is automounting Amazon Elastic File System filesystems. For those of you not familiar with it, Amazon EFS is essentially NFS-as-a-service; it provides a POSIX filesystem with scalable performance, replicated across Availability Zones. To use EFS automounting, specify it in the automounter configuration and enable the automounter

# echo '/efs -efs' > /etc/auto_master 
# sysrc autofs_enable="YES"

then either reboot or start the required services

# service automountd start
# service autounmountd start
# service automount start

Having done this, any access to the path /efs/FSID (e.g., /efs/fs-01234567) will automatically and transparently mount that filesystem.

This feature is long overdue — I had intended to add this three years ago after Rick Macklem and I got FreeBSD's NFS code working with Amazon EFS — and really just comes down to a single script (/etc/autofs/special_efs) which implements the -efs automount map. That said — overdue or not, trivial or not — I think this will be a useful feature.

The second feature is the addition of ebsnvme-id and the new /dev/aws/disk tree. Users of Amazon Linux may be familiar with ebsnvme-id; this is a utility which prints information about the disks attached to EC2 "Nitro" instances (where they are exposed as NVMe devices). In particular, the ebsnvme-id tool allows you to see the EBS volume ID of EBS volumes and the "Linux device name" which was specified when the volume was attached — which is not particularly meaningful on FreeBSD, but theoretically knowing that /dev/nda1 was attached via the EC2 API as "/dev/sdh" might be useful?

The ebsnvme-id which I wrote for FreeBSD is compatible with the version in Amazon Linux but has some additional functionality: You can also ask it about "Instance Store" (aka. "Ephemeral") disks, for which it will identify the disk serial number.

Using the tool and some devd magic, FreeBSD now maintains a tree under /dev/aws/disk containing the symlinks of the forms

• /dev/aws/disk/ebs/vol-0123456789abcdef
• /dev/aws/disk/linuxname/sdh
• /dev/aws/disk/ephemeral/SERIALNO

which point at the appropriate device nodes (typically /dev/ndaX for some value X).

I hope these new features prove useful to FreeBSD users! If you'd like to support my continuing work on the FreeBSD/EC2 platform, please consider contributing to my FreeBSD/EC2 Patreon; it's much easier to justify taking time away from my day job when it feels like people appreciate my work!

My new FreeBSD Laptop: Dell Latitude 7390

As a FreeBSD developer, I make a point of using FreeBSD whenever I can — including on the desktop. I've been running FreeBSD on laptops since 2004; this hasn't always been easy, but over the years I've found that the situation has generally been improving. One of the things we still lack is adequate documentation, however — so I'm writing this to provide an example for users and also Google bait in case anyone runs into some of the problems I had to address.

A few months ago, after my System76 Galago Pro had its second experience with a dead/swelling batery, I decided that it was time to replace it. On February 15th, I ordered a Dell Latitude 7390.

This is an older model of laptop — it originally launched in 2018 — but I've always found that Dell Latitudes are well built, and this one came with a very attractive price tag: Rather than the original price of $3599 CAD ($2600 USD), Dell Canada was selling it for $1049 CAD ($750 USD). I can only assume that it was the last of their stock and they wanted to clear out that production line.

The laptop came with:

• An Intel i7-8650U CPU (Quad Core, 8 MB L3 cache)
• 8 GB RAM
• A 256 GB SATA M.2 SSD with Windows 10 preinstalled
• A 13.3" 1920x1080 display (non-touch)
• An Intel 8265 WiFi chipset, and
• A 60 W-hour battery.

The laptop arrived on March 6th, and I made a couple upgrades:

• I replaced the 8GB SODIMM with a 16 GB SODIMM ($89 CAD), and
• I replaced the 256 GB SSD with an Intel 660p 512 GB NVMe disk ($94 CAD).

In addition to giving me double the disk space, upgrading the disk allowed me to keep the Windows installation intact in case I ever want to put this laptop back into use running Windows.

After these upgrades, I didn't quite have my ideal laptop — I would have preferred 32 GB of RAM (Dell's specifications state that this laptop supports a maximum of 16 GB), a second disk (theoretically the "WWAN" PCIe slot can hold an M.2 2242 NVMe disk, but comments online suggest that the Dell BIOS doesn't allow SSDs in that slot) and a TrackStick — but for what I paid I really can't complain.

Installing FreeBSD

I downloaded the "memstick" image from the latest FreeBSD 12.1-STABLE weekly snapshot using my old laptop and wrote it to a USB Disk. Normally I would use a RELEASE image (which would make some later steps easier too) but a few recent changes I had made and some of the code needed to support the I2C touchpad weren't in the last release. (If you're reading this a few months into the future, use 12.2 or a later release instead of the 12-STABLE branch.)

To boot from a USB stick on the Dell Latitude 7390, I needed to press F12 when I turned the system on to enter the boot disk selection menu.

Installing FreeBSD was easy for me, as I've done it many times in the past; but I did change several defaults:

• When selecting which parts of FreeBSD to install, I included the "src" distribution. Normally I wouldn't do this; but I needed the FreeBSD source code for building some kernel modules later.
• When configuring ZFS, I turned on full disk encryption (as I have done on all of my laptops recently).
• I increased the swap space from 2 GB (the default) to 16 GB. I hope that this system never needs to use swap space — but if its 16 GB of RAM isn't enough, it's unlikely that 2 GB of swap space would be enough to help either.
• I turned off sshd — I don't intend to connect to this laptop remotely — and turned on ntpd and powerd — I want the clock kept accurate, and I want to save power by running the CPU slower when it's not under load. (These can be enabled or disabled via /etc/rc.conf later, but the installer offers them so I figured I might as well do it here.)
• From the "system hardening" screen, I enabled the clear_tmp, disable_syslogd, and disable_sendmail options — the first two are just on principle, while Sendmail is something I don't use and don't want (I use qmail instead). I don't enable any of the other "hardening" options since they don't have any significant benefit but remove functionality which I find useful.
• When creating my "cperciva" user account, I added the account to the wheel and video groups. The former is necessary so that I can su to root; the latter is necessary for applications to access accelerated video functionality. (If you forget these, you can log in as root and add them later, but it's a nuisance — and I mention them here because I have on numerous occasions needed to do exactly that.)

After the installation completes, I asked the installer to reboot the system and removed the USB disk.

Initial FreeBSD configuration

The first thing I did was to turn off the annoyingly loud console beep:

# echo kern.vt.enable_bell=0 >> /etc/sysctl.conf

and then I told FreeBSD to use "link aggregation" networking, failing over automatically from the wired ethernet (if connected) to the WiFi network

# sysrc wlans_iwm0="wlan0"
# sysrc ifconfig_wlan0="WPA"
# sysrc ifconfig_em0="ether MY:WI:FI:MAC:ADD:RESS"
# sysrc cloned_interfaces="lagg0"
# sysrc ifconfig_lagg0="laggproto failover laggport em0 laggport wlan0 DHCP"

provided my WiFi network credentials via /etc/wpa_supplicant.conf

network={
	ssid="MY NETWORK SSID"
	psk="MY NETWORK PASSWORD"
}

and started the network (service netif restart). I want a firewall, so I enabled FreeBSD's pf with a very simple configuration:

# echo "block in all" > /etc/pf.conf
# echo "set skip on lo0" >> /etc/pf.conf
# echo "pass out all keep state" >> /etc/pf.conf
# sysrc pf_enable="YES"
# service pf start

Now for the most annoying part: Because I'm running 12.1-STABLE rather than 12.1-RELEASE (due to the aforementioned patches and recently added support fo the I2C touchpad) I couldn't rely completely on packages built by the FreeBSD Project. Instead, I needed to build two packages of kernel modules — drm-kmod and iichid — via the ports tree.

This meant:

1. Downloading and extracting a ports tree:

   # portsnap fetch extract
   

2. Building the ports:

   # make -C /usr/ports/graphics/drm-kmod install clean BATCH=YES
   # make -C /usr/ports/sysutils/iichid install clean BATCH=YES
   

3. And locking those packages to ensure that they won't be "upgraded" automatically to versions compiled on FreeBSD 12.1-RELEASE:

   # pkg lock -y -g '*kmod*' iichid
   

If I had been able to install 12.2-RELEASE (which does not exist yet at the time I am writing this), those steps would simply have been pkg install iichid drm-kmod.

Now that I had those packages in place, I could download the other ones I needed to get a basic GUI running:

# pkg install xorg kde5 sddm

And then do some basic configuration to turn on the GUI:

• Telling FreeBSD to load the Intel video driver:

  # sysrc kld_list="/boot/modules/i915kms.ko"
  

• Telling FreeBSD to load the I2C HID driver (needed for the touchpad to work):

  # sysrc kld_list+="/boot/modules/iichid.ko"
  

• Telling Xorg to use devices exposed via libinput (i.e., the touchpad):

  # cp /usr/local/share/X11/xorg.conf.d/40-libinput.conf \
      /usr/local/etc/X11/xorg.conf.d/
  

• Telling the display manager to start at boot:

  # sysrc sddm_enable="YES"
  

At this point I could reboot and watch my laptop boot into a GUI where I logged in as cperciva, opened a console (Ctrl-Alt-T), and used su to become root so I could do more configuration.

More FreeBSD configuration

The BIOS on the Dell Latitude 7390 configures the speakers and headphone jack as being independent audio outputs. This isn't what I want — I want to have sound go to the speakers by default but switch to the headphone jack and mute the speakers automatically if I plug in headphones. To do this, I told FreeBSD that the headphones (which are nid33 on this laptop) are part of the same audio set (as=1) as the speakers but should use mute-and-switch-to-headphones behaviour (seq=15):

# echo 'hint.hdaa.0.nid33.config="as=1 seq=15"' >> /boot/loader.conf

Speaking of audio, there's a bug in the HDMI codec which results in annoying warnings ("Unexpected unsolicited response with tag 63" and "Command timeout on address 2") being logged to the kernel console. Ed Maste has been working on fixing the underlying issues, but he provided a workaround which let me silence the warnings:

# echo 'compat.linuxkpi.i915_disable_power_well="0"' >> /boot/loader.conf

My preferred (KDE) environment doesn't need D-Bus for much, but one place it is needed is to allow the GUI tools to see the battery state and interact with power management:

# sysrc dbus_enable="YES"

CPU bugs often get fixed via microcode patches, so I want to have those:

# pkg install devcpu-data
# echo 'cpu_microcode_load="YES"' >> /boot/loader.conf
# echo 'cpu_microcode_name="/boot/firmware/intel-ucode.bin"' >> /boot/loader.conf

I want my laptop to boot faster — which means not having the boot loader wait 10 seconds in case I want to change how I boot, and not having the FreeBSD kernel wait for USB devices before mounting the root filesystem (I know that my root filesystem is on the NVMe disk):

# echo 'autoboot_delay="0"' >> /boot/loader.conf
# echo 'hw.usb.no_boot_wait="1"' >> /boot/loader.conf

As mentioned above, I like to use qmail rather than sendmail; I also use spiped to tunnel email to and from my mail server:

# pkg install netqmail ucspi-tcp spiped
# echo `hostname` > /var/qmail/control/me
# echo ":127.0.0.1:8025" > /var/qmail/control/smtproutes
# rm /var/qmail/alias/.qmail-*
# echo cperciva > /var/qmail/alias/.qmail-default
# cp /var/qmail/doc/mailer.conf.sample /etc/mail/mailer.conf
# sysrc qmailsend_enable="YES"
# sysrc qmailsmtpd_enable="YES"
# sysrc qmailsmtpd_host="127.0.0.1"
# echo "127.0.0.1:allow" > /etc/tcp.smtp
# sysrc spiped_enable="YES"
# sysrc spiped_pipes="SMTP POP3"
# sysrc spiped_pipe_SMTP_mode="client"
# sysrc spiped_pipe_SMTP_source="[127.0.0.1]:8025"
# sysrc spiped_pipe_SMTP_target="mail.tarsnap.com:8025"
# sysrc spiped_pipe_SMTP_key="/etc/spiped/smtp.key"
# sysrc spiped_pipe_POP3_mode="client"
# sysrc spiped_pipe_POP3_source="[127.0.0.1]:110"
# sysrc spiped_pipe_POP3_target="mail.tarsnap.com:8110"
# sysrc spiped_pipe_POP3_key="/etc/spiped/pop3.key"

And finally while the webcam daemon is installed already (the kde5 package pulls it in as a dependency) I want it to be enabled and I want the cperciva user to be able to access it:

# echo 'cuse_load="YES"' >> /boot/loader.conf
# sysrc webcamd_enable="YES"
# pw groupmod webcamd -m cperciva

Desktop configuration

At this point I had a fully functional desktop environment, but it wasn't set up exactly how I wanted. I don't like the default KDE wallpaper, so I installed new wallpaper

# pkg install wallpapers-freebsd-kde

and told KDE to use it (right click on desktop; "Configure Desktop"; scroll down and select the right wallpaper).

I want to be able to print from within KDE, so I enabled CUPS (which was already installed as a KDE dependency) and started it (necessary since I wanted to use it without rebooting):

# sysrc cupsd_enable="YES"
# service cupsd start

and then told KDE about my printer ("System Settings" -> "Printers" -> "Add Printer"; manual URI of http://192.168.1.60:631/ipp and the Generic PostScript Printer driver).

I want to be able to control the brightness of the screen (aka. the backlight connected to the Intel video chipset):

# pkg install intel-backlight

and in particular to be able to control it via the function keys (Fn+Up, Fn+Down) which Dell exposes via ACPI:

# echo 'acpi_video_load="YES"' >> /boot/loader.conf
# cp /usr/local/share/examples/intel-backlight/acpi-video-intel-backlight.conf \
    /usr/local/etc/devd/

There's a long list of software I want to be able to use (or in some cases, would prefer to avoid, but need nonetheless):

# pkg install firefox libreoffice thunderbird chromium
# pkg install nano konversation
# pkg install xournal ImageMagick7 pdftk
# pkg install texlive-full
# pkg install git subversion diffstat portlint sloccount
# pkg install autoconf autoconf-archive automake libtool
# pkg install base64 wkhtmltopdf
# pkg install tarsnap

And finally I want to perform hourly backups to Tarsnap — but in the background, and not spinning up the CPU to perform them (line wrapping added in the first line to avoid breaking web browsers):

# echo '47 * * * * root nice /usr/local/bin/tarsnap -c --keyfile /root/tarsnap.key
    -f `date +\%Y-\%m-\%d:\%H` /root /etc /usr/local/etc /usr/home' > /etc/cron.d/tarsnap
# sysrc powerd_flags="-N"

KDE configuration

Like most desktop environments, KDE has a myriad settings which can be adjusted, and I change many to match my personal taste, ranging from the Application Launcher (I prefer the "menu" version) to the keyboard shortcuts for changing desktops (I use Meta+Tab for this, since I don't make use of KDE "Activities").

The one particularly noteworthy option I set is to tell KDE to "Sleep" when the laptop lid is closed; in combination with the (default) setting to lock the screen when sleeping, this gives me a convenient way to "put the laptop away" when I don't want to shut it down completely. (Note however that it is unable to "Hibernate" since FreeBSD does not have the necessary support for Suspend-to-Disk.)

Current status

So where are we at now? Here's a short summary:

• Graphics: Working (requires drm-kmod).
• Backlight (brightness) control: Working (requires intel-backlight and acpi-video-intel-backlight.conf).
• Keyboard backlight: Working (controlled via hotkeys).
• Touchpad: Working (requires 12.1-STABLE or later, iichid port, and 40-libinput.conf).
• Volume up/down hotkeys: Working (via KMix).
• Volume mute hotkey: Not working.
• Ethernet: Working (em driver).
• Wi-Fi: Working (iwm driver).
• Bluetooth: Not tested.
• Fan: Comes on as needed (not often).
• SD Card reader: Not tested.
• Suspend/resume: Working.
• Webcam: Working (requires cuse and webcamd_enable).
• HDMI: Working.
• USB: Working.
• Audio output: Working (autoswitch from speakers to headphones requires loader.conf setting).
• Battery: 8-12 hours under typical use depending on backlight brightness; 2 hours of buidworld -j8.

After the above installation and configuration — which took 75 minutes, mostly spent downloading 4 GB worth of packages — there's just one things left to do: Copy all my data across from my old laptop.

What didn't work the first time

While the above installation and configuration process all worked in the end, it wasn't nearly as smooth the first time — or the second time, or the third time. Some of the issues I ran into:

• The first time I installed FreeBSD, I installed FreeBSD 12.1-RELEASE. This caused problems for the touchpad — it was very unresponsive and the pointer often "lagged" up to half a second behind — because the release didn't have some I2C code which was later added in 12.1-STABLE.
• After upgrading the system to 12.1-STABLE, I found that the video driver didn't work. While FreeBSD aims to maintain compatibility in "stable" branches, occasionally issues slip in — and one such issue arose in the 12-STABLE branch making the drm-kmod package built on 12.1-RELEASE incompatible with the 12.1-STABLE kernel.
• Once I determined that I needed 12.1-STABLE (and confirmed that with appropriately compiled packages, both video and touchpad worked) I decided to reinstall from a 12.1-STABLE ISO. I downloaded the "memstick" ISO, and... it was corrupted. Or rather, truncated; I guess something broke with the mirroring of that week's snapshots.
• No matter, I thought; I can use the "mini-memstick" ISO instead; this downloads the parts of FreeBSD you want at install time rather than including them all on the ISO. This one wasn't corrupted — but when the installer got to downloading the bits I wanted to install, it couldn't recognize my WiFi network. Fortunately this was only a minor annoyance: I plugged my laptop into my wired network and everything was fine.
• One of the times (I can't remember which) I tried to install the chromium package, it turned out that there wasn't a package available: That week's build in the FreeBSD package-building cluster failed because the ever-growing build had just crossed the 24 hour limit and the package builder decided that it must have gotten stuck. I reported this to the FreeBSD ports manager team and they increased chromium's time limit so that it could build successfully — but in the mean time I had to build the port myself (which took about six hours on my laptop).
• Before I started working on this laptop, the backlight control keys would adjust acpi_video's notion of what the brightness should be, and the intel-backlight tool was able to adjust the actual brightness, but there was no connection between the two; I had to add code to the FreeBSD kernel to expose brightness changes via devd and wrote the necessary configuration file (acpi-video-intel-backlight.conf) to invoke intel-backlight to make the necessary changes.
• When I first set up the lagg interface to have FreeBSD switch automatically between wired and wireless networks, I found that I needed to load the WiFi driver (and firmware) kernel modules via loader.conf rather than having them loaded automatically by the recently-added "devmatch" tool: If I relied on devmatch, then FreeBSD's startup scripts would try to create the link aggregation interface before the wifi driver was loaded, and (unsurprisingly) fail to aggregate a network interface which didn't exist. A one-line fix to the devmatch rc.d script moved it earlier in the boot process and made two loader.conf lines unnecessary.
• Finally — and most confusingly — I found at one point that plugging or unplugging AC power would result in the laptop going to sleep (S3 Suspend). This turned out to be due to an interaction between ACPI, upowerd, and KDE's power management options: ACPI was not sending a notification if the laptop lid was opened while the system was asleep, resulting in upowerd thinking that the lid was still closed... and when KDE got a message about the AC power status changing, it checked all of the statuses related to power management, found that the lid was "closed", and decided that it should go to sleep. In the end, all I needed was a one-line fix to FreeBSD's ACPI code, but it took three days to find that one line!

Conclusion

Is FreeBSD ready for the desktop? Yes and no. Yes, in that I have a very nice FreeBSD laptop where everything works the way I want. But no, in that it took me two months worth of fiddling with this in my spare time to fix some of the "glitches" which arose; while there wasn't anything particularly challenging, I expect that most people would give up long before they fixed all of the issues I ran into.

On the other hand, can FreeBSD be ready for the desktop? Absolutely. I've fixed the issues I ran into — and once we have FreeBSD 12.2-RELEASE with packages built for that release the process of bringing up a GUI will be much easier, as well. The biggest thing FreeBSD needs is to have developers acquiring laptops and carefully working their way through the issues which arise; the FreeBSD Foundation has already started doing this, and I hope in the months to come they — and other FreeBSD users — will publish reports telling us which laptops work and what configuration they need.