Daemonic Dispatches

FreeBSD on EdgeRouter Lite - no serial port required

I recently bought an EdgeRouter Lite to use as a network gateway; I had been using a cheap consumer wifi/NAT device, but I wanted the extra control I could get by running FreeBSD rather than whatever mangled version of Linux the device came with. Someone wrote instructions on installing FreeBSD onto the EdgeRouter Lite two years ago, but they rely on using the serial port to reconfigure the boot loader — perfectly straightforward if you have a serial cable and know what you're doing, but I decided to take the opportunity to provide a more streamlined process.

Required:

• An EdgeRouter Lite.
• A small Phillips ("plus") screwdriver.
• Another system running FreeBSD.

Steps:

1. Fetch a FreeBSD HEAD source tree. (FreeBSD 10-STABLE is not supported yet. I think this might change between now and 10.3-RELEASE.)
2. Download the image building script.
3. Run ./buildimg.sh /path/to/src/tree disk.img.
4. Remove three small screws from the back of the EdgeRouter Lite. Open the case and remove the USB drive. (Mine was held very firmly in place. I found that wiggling it towards and away from the board allowed me to gradually ease it free.)
5. Plug the USB disk into the system where you built the FreeBSD image.
6. Run dd if=/dev/USBDISK of=ERL.img where USBDISK is the name of the USB disk device (probably da0), to make a backup of the EdgeRouter Lite software in case something breaks and you need to restore it later.
7. Run dd if=disk.img of=/dev/USBDISK (where USBDISK is as before) to write the FreeBSD disk image onto the EdgeRouter Lite USB disk.
8. Plug the USB disk back into the EdgeRouter Lite, close the box, and replace the three screws.

There are three gigabit ethernet ports on the EdgeRouter Lite, marked on the case as "eth0", "eth1", and "eth2"; in FreeBSD, they show up as "octe0", "octe1", and "octe2" in the same order. With the configuration on my image:

• "eth0" should be connected to your upstream internet connection; the system runs DHCP on this connection to get an IP address and sends internet traffic out this port.
• "eth1" should be connected to your LAN; the system provides IP addresses within the 192.168.1.0/24 block via DHCP, and runs a recursive DNS server (also advertised via DHCP) and an NTP server on 192.168.1.1.
• "eth2" is like "eth1", except that it gets IP addresses from the 192.168.2.0/24 block and traffic from this port (and only this port) can connect via SSH using the username and password "ubnt". (I recommend leaving this port unused except when you need to SSH in to reconfigure something.)

That's pretty much all you need to know to install and use FreeBSD on the EdgeRouter Lite; but there are some interesting tricks involved in the script which builds the disk image, so for the rest of this blog post I will provide a brief "walkthrough" of the script.

---

#!/bin/sh -e

Shell scripts are run by /bin/sh. (No matter what some misguided Linux users may think, /bin/bash is not a standard shell.) The -e option tells the shell interpreter to exit if any of the commands fail — if something goes wrong, we should stop and let the user see what happened rather than continuing and producing a broken disk image later.

if [ -z "$1" ] || [ -z "$2" ]; then
	echo "buildimg.sh srcdir disk.img"
	exit 1
fi
SRCDIR=$1
IMGFILE=$2

This script takes two options: The location of the FreeBSD source tree, and the name of the file to use for the disk image.

# Set environment variables so make can use them.
export TARGET=mips
export TARGET_ARCH=mips64
export KERNCONF=ERL

The EdgeRouter Lite is a 64-bit MIPS system; FreeBSD HEAD has a kernel configuration already defined for it.

export ALL_MODULES=YES
export WITHOUT_MODULES="cxgbe mwlfw netfpga10g otusfw ralfw usb rtwnfw"

Unfortunately that kernel configuration disables building kernel modules; we want those so that we can use pf later. Some modules fail to build — I didn't investigate exactly why, but it was something related to firmware blobs — so we turn those off explicitly.

# Create working space
WORKDIR=`env TMPDIR=\`pwd\` mktemp -d -t ERLBUILD`

We create some temporary working space under the current directory. On many systems /tmp isn't large enough to hold a complete installation of FreeBSD, so I overrode the default there.

# Build MIPS64 world and ERL kernel
JN=`sysctl -n hw.ncpu`
( cd $SRCDIR && make buildworld -j${JN} )
( cd $SRCDIR && make buildkernel -j${JN} )

Build the MIPS64 world and kernel. The -j flag tells make to run several commands in parallel; we consult sysctl to find out how many CPUs we have available for the build.

# Install into a temporary tree
mkdir ${WORKDIR}/tree
( cd $SRCDIR && make installworld distribution installkernel DESTDIR=${WORKDIR}/tree )

We create a tree and install FreeBSD into it. The installworld and installkernel targets install the userspace and kernel binaries respectively; the distribution target installs standard configuration files.

# Download packages
cp /etc/resolv.conf ${WORKDIR}/tree/etc/
pkg -c ${WORKDIR}/tree install -Fy pkg djbdns isc-dhcp43-server
rm ${WORKDIR}/tree/etc/resolv.conf

The FreeBSD project provides precompiled binary packages for the 64-bit MIPS architecture; this allows us to put packages into the image we're building while avoiding the headaches of cross-building them. However, we cannot cross-install packages either, since packages can run scripts when they are installed — scripts which (since we're not building this disk image on a MIPS64 system) we won't be able to run. Instead, we simply download the packages into the image; they will be installed when the system first boots.

# FreeBSD configuration
cat > ${WORKDIR}/tree/etc/rc.conf <<EOF

The /etc/rc.conf file is the "master configuration file" on FreeBSD; most enabling/disabling of services is done here, as well as some more specific configuration.

hostname="ERL"

Every host needs a name. We'll call this "ERL", lacking any better inspiration.

growfs_enable="YES"

We're building a disk image which we'll write onto the provided USB disk, but the image is smaller than the disk; when the system first boots, this tells it to expand the root partition to fill the available space.

tmpfs="YES"
tmpsize="50M"

This is probably unnecessary, but I like to have a memory disk mounted on /tmp; if for some reason temporary files get created here, this will avoid burning up the flash storage.

ifconfig_octe0="DHCP"
ifconfig_octe1="192.168.1.1 netmask 255.255.255.0"
ifconfig_octe2="192.168.2.1 netmask 255.255.255.0"

We run DHCP on the "upstream" connection, but provide static network parameters for the "LAN" connections.

pf_enable="YES"
gateway_enable="YES"

We're going to use the PF firewall; and we're going to be forwarding packets (both via the network address translation and between the two LAN ports) so we need that option too.

sendmail_enable="NONE"
sshd_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
svscan_enable="YES"
dhcpd_enable="YES"
dhcpd_ifaces="octe1 octe2"
EOF

We don't want to run sendmail; we do want to run sshd (we'll use PF to restrict access, however); we do want to run ntpd, and we want it to set its clock when it starts, no matter how far off it is (the EdgeRouter Lite doesn't have a battery-powered clock, so it boots with a wildly wrong time set); we want to run svscan so that it can launch dnscache for us; and we want to run a dhcp daemon for the two LAN interfaces.

cat > ${WORKDIR}/tree/etc/pf.conf <<EOF
# Allow anything on loopback
set skip on lo0

# Scrub all incoming traffic
scrub in

# NAT outgoing traffic
nat on octe0 inet from { octe1:network, octe2:network } to any -> (octe0:0)

# Reject anything with spoofed addresses
antispoof quick for { octe1, octe2, lo0 } inet

# Default to blocking incoming traffic but allowing outgoing traffic
block all
pass out all

# Allow LAN to access the rest of the world
pass in on { octe1, octe2 } from any to any
block in on { octe1, octe2 } from any to self

# Allow LAN to ping us                                       
pass in on { octe1, octe2 } inet proto icmp to self icmp-type echoreq

# Allow LAN to access DNS, DHCP, and NTP
pass in on { octe1, octe2 } proto udp to self port { 53, 67, 123 }
pass in on { octe1, octe2 } proto tcp to self port 53

# Allow octe2 to access SSH
pass in on octe2 proto tcp to self port 22
EOF

Fairly straightforward PF configuration: NAT outgoing traffic onto the "upstream" connection; allow the local network to access DNS, DHCP, and NTP; and allow octe2 to access SSH. I opted to only allow ICMP echo request packets from the LAN side — some people prefer to respond to pings from anywhere, but I decided that for a general purpose image it was better to err in the direction of being silent. Similarly I decided to simply drop bad packets rather than sending TCP RST or ICMP unreachable responses.

mkdir -p ${WORKDIR}/tree/usr/local/etc
cat > ${WORKDIR}/tree/usr/local/etc/dhcpd.conf <<EOF
option domain-name "localdomain";
subnet 192.168.1.0 netmask 255.255.255.0 {
        range 192.168.1.2 192.168.1.254;
        option routers 192.168.1.1;
        option domain-name-servers 192.168.1.1;
}
subnet 192.168.2.0 netmask 255.255.255.0 {
        range 192.168.2.2 192.168.2.254;
        option routers 192.168.2.1;
        option domain-name-servers 192.168.2.1;
}
EOF

This provides a basic configuration for ISC DHCPD. I have a feeling that this could be simplified to have a single configuration block covering both LAN ports.

# Script to complete setup once we're running on the right hardware
mkdir -p ${WORKDIR}/tree/usr/local/etc/rc.d
cat > ${WORKDIR}/tree/usr/local/etc/rc.d/ERL <<'EOF'
#!/bin/sh

I mentioned earlier that we couldn't cross-install packages; we take care of that now, with a script which runs the first time FreeBSD boots. The quotes around EOF in the here-document syntax instruct the shell that variables should not be expanded — important since we're creating a shell script which uses several shell variables.

# KEYWORD: firstboot

The "firstboot" keyword tells /etc/rc that this script should only be run the first time that the system boots.

# PROVIDE: ERL
# REQUIRE: NETWORKING
# BEFORE: LOGIN

# This script completes the configuration of EdgeRouter Lite systems.  It
# is only included in those images, and so is enabled by default.

. /etc/rc.subr

: ${ERL_enable:="YES"}

name="ERL"
rcvar="ERL_enable"
start_cmd="ERL_run"
stop_cmd=":"

This is fairly standard rc.d script boilerplate.

ERL_run()
{

	# Packages
	env SIGNATURE_TYPE=NONE pkg add -f /var/cache/pkg/pkg-*.txz
	pkg install -Uy djbdns isc-dhcp43-server

We want to install the two packages we downloaded into the image earlier.

	# DNS setup
	pw user add dnscache -u 184 -d /nonexistent -s /usr/sbin/nologin
	pw user add dnslog -u 186 -d /nonexistent -s /usr/sbin/nologin
	mkdir /var/service
	/usr/local/bin/dnscache-conf dnscache dnslog /var/service/dnscache 0.0.0.0
	touch /var/service/dnscache/root/ip/192.168

We configure dnscache to be launched by svscan and respond to DNS requests from the LAN.

	# Create ubnt user
	echo ubnt | pw user add ubnt -m -G wheel -h 0

We could have created this user while creating the disk image, but since we needed to have a firstboot script anyway it was easier to do it here. The -h 0 option means "read the password from standard input", which is why we're echoing it in from there.

	# We need to reboot so that services will be started
	touch ${firstboot_sentinel}-reboot

Part of the rc.d "firstboot" mechanism is to allow scripts to ask for the system to be rebooted after the first boot (and all the associated system initialization) is complete. In this case, we need to reboot in order to have svscan and isc-dhcpd running (since they weren't installed yet when the boot process started).

}

load_rc_config $name
run_rc_command "$1"
EOF
chmod 755 ${WORKDIR}/tree/usr/local/etc/rc.d/ERL

More boilerplate. The rc.d script must be executable.

# We want to run firstboot scripts
touch ${WORKDIR}/tree/firstboot

The sentinel file /firstboot tells FreeBSD that the system is booting for the first time and "firstboot" scripts should be run. At the end of the first boot, /etc/rc deletes this file.

# Create FAT32 filesystem to hold the kernel
newfs_msdos -C 33M -F 32 -c 1 -S 512 ${WORKDIR}/FAT32.img
mddev=`mdconfig -f ${WORKDIR}/FAT32.img`
mkdir ${WORKDIR}/FAT32
mount -t msdosfs /dev/${mddev}  ${WORKDIR}/FAT32
cp ${WORKDIR}/tree/boot/kernel/kernel ${WORKDIR}/FAT32/vmlinux.64
umount /dev/${mddev}
rmdir ${WORKDIR}/FAT32
mdconfig -d -u ${mddev}

The EdgeRouter Lite boot loader expects to launch a Linux kernel which is found at /vmlinux.64 within a FAT32 filesystem. Fortunately, it doesn't check that the kernel it's launching is Linux... so we create a FAT32 filesystem and drop a FreeBSD kernel in, named "/vmlinux.64" so that the EdgeRouter Lite boot loader launches it for us. (Our kernel is only about 10 MB, but the minimum size for a FAT32 filesystem is 33 MB.)

# Create UFS filesystem
echo "/dev/da0s2a / ufs rw 1 1" > ${WORKDIR}/tree/etc/fstab
makefs -f 16384 -B big -s 920m ${WORKDIR}/UFS.img ${WORKDIR}/tree

We use the makefs tool to create a UFS filesystem from the installed FreeBSD tree. The MIPS64 hardware is big-endian, and UFS is not endian-agnostic, so we need to tell makefs to create a big-endian filesystem; this also means that (assuming we're using a little-endian system to build this disk image) we can't mount the filesystem on the system we're using to create it.

# Create complete disk image
mkimg -s mbr		\
    -p fat32:=${WORKDIR}/FAT32.img \
    -p "freebsd:-mkimg -s bsd -p freebsd-ufs:=${WORKDIR}/UFS.img" \
    -o ${IMGFILE}

The EdgeRouter Lite boot loader expects the kernel to be found on the first MBR slice; and the FreeBSD ERL kernel configuration expects the root filesystem to be found at da0s2a so we'd better put it there.

# Clean up
chflags -R noschg ${WORKDIR}
rm -r ${WORKDIR}

Once we finish building the disk image, we don't need our staging tree or the separate filesystem images any more.