Daemonic Dispatches

FreeBSD major version upgrades

I posted yesterday about using FreeBSD Update to perform FreeBSD minor version upgrades, e.g., upgrading from FreeBSD 6.2 to FreeBSD 6.3 or from FreeBSD 7.0-BETA1.5 to FreeBSD 7.0-BETA2. Today I'm going to write about the more complicated process needed for major version upgrades, e.g., upgrading from FreeBSD 6.x to FreeBSD 7.x.

NOTE: Before proceeding, if you have FreeBSD Update scheduled to run from a cron job, disable the cron job! Bad Things will happen if FreeBSD Update tries to download security updates in the middle of this upgrade process.

The first step necessary is to make sure that you can rebuild all of your installed third-party software (for most people, this means everything installed from the ports tree). Later in the upgrade process it will be necessary to rebuilt all of the ports so that they link to libraries from the new FreeBSD release -- so before starting to upgrade FreeBSD itself, we need to make sure that rebuilding the ports won't fail.

# portsnap fetch update
...
# portupgrade -a
...

This makes sure that what you have installed matches what's currently in the FreeBSD ports tree and thus that nobody has updated a port and somehow made it not build any more.

As I write this, "upgrade" support does not yet exist in the version of FreeBSD Update in the FreeBSD base system, so the next step is to download the script. If you are running FreeBSD 7.0-RC1 or FreeBSD 6.3-RC1 or later, you should already have this new version of FreeBSD Update installed, so you can skip this step.

# fetch http://www.daemonology.net/freebsd-update/freebsd-update-upgrade.tgz
freebsd-update-upgrade.tgz 100% of 21 kB 74 kBps
# fetch http://www.daemonology.net/freebsd-update/freebsd-update-upgrade.tgz.asc
freebsd-update-upgrade.tgz.asc 100% of 187 B 214 kBps
# gpg --verify freebsd-update-upgrade.tgz.asc freebsd-update-upgrade.tgz
gpg: WARNING: using insecure memory!
gpg: please see http://www.gnupg.org/faq.html for more information
gpg: Signature made Sat Nov 10 04:38:54 2007 PST using DSA key ID CA6CDFB2
gpg: Good signature from "FreeBSD Security Officer <security-officer@FreeBSD.org>"
# tar -xf freebsd-update-upgrade.tgz

I've signed the tarball containing this script; as Security Officer, I strongly recommend verifying the signature using gpg before proceeding.

When I run FreeBSD Update below, I invoke it as sh freebsd-update.sh -f freebsd-update.conf in order to use this new version of FreeBSD Update and its associated configuration file; if you skiped the above step because you're already running 7.0-RC1 or 6.3-RC1 or later, you can just invoke it as freebsd-update.

This system is presently running FreeBSD 6.3-BETA1, and I'm upgrading it to FreeBSD 7.0-BETA2; obviously, if you're reading this at some point in the future and want to upgrade to something other than FreeBSD 7.0-BETA2, you should change "7.0-BETA2" below and the output from the script will be correspondingly different.

# sh freebsd-update.sh -f freebsd-update.conf -r 7.0-BETA2 upgrade
Looking up update.FreeBSD.org mirrors... 1 mirrors found.
Fetching metadata signature for 6.3-BETA1 from update1.FreeBSD.org... done.
Fetching metadata index... done.
Inspecting system... done.

The following components of FreeBSD seem to be installed:
kernel/smp src/base src/bin src/contrib src/crypto src/etc src/games
src/gnu src/include src/krb5 src/lib src/libexec src/release src/rescue
src/sbin src/secure src/share src/sys src/tools src/ubin src/usbin
world/base world/info world/lib32 world/manpages

The following components of FreeBSD do not seem to be installed:
kernel/generic world/catpages world/dict world/doc world/games
world/proflibs

Does this look reasonable (y/n)? y

Here FreeBSD Update is looking at the system and deciding which parts of FreeBSD you have installed -- obviously if you don't have the docs (documentation) from FreeBSD 6.3-BETA1 installed you aren't likely to want to have the docs from FreeBSD 7.0-BETA2 installed after upgrading.

Fetching metadata signature for 7.0-BETA2 from update1.FreeBSD.org... done.
Fetching metadata index... done.
Fetching 1 metadata patches. done.
Applying metadata patches... done.
Fetching 1 metadata files... done.

WARNING: This system is running a "smp" kernel, which is not a
kernel configuration distributed as part of FreeBSD 7.0-BETA2.
As part of upgrading to FreeBSD 7.0-BETA2, this kernel will be
replaced with a "generic" kernel.
Does this look reasonable (y/n)? y

In FreeBSD 6.x, there are two kernels distributed with the releases: a GENERIC (single-processor) kernel, and an SMP (multi-processor) kernel. This is because in FreeBSD 6.x running with SMP support adds a significant amount of overhead (typically around 5-25%), so on systems with only one CPU it's silly to run with SMP support. In FreeBSD 7.x, this overhead is much smaller, and these days far fewer systems have only a single CPU (since multi-core processors count as several CPUs), so the release engineering team decided to ship only a single kernel -- which is named "GENERIC", but is really the equivalent of the old "SMP" kernel.

In short, there's nothing to worry about here -- whether you were running a GENERIC kernel or an SMP kernel in FreeBSD 6.x, you'll want a GENERIC kernel in FreeBSD 7.x and that's what FreeBSD Update will give you -- but FreeBSD Update isn't smart enough to know this, so it's issuing a warning message just in case.

Inspecting system... done.
Preparing to download files... done.
Fetching 18290 patches.....10....20 ... 18280....18290 done.
Applying patches... done.
Fetching 5039 files... done.

There are a lot of changes between FreeBSD 6.3 and FreeBSD 7.0! Most of these patches and new files being downloaded are because I have the src tree installed -- if you don't have that you'll find that there are far fewer files to download.

Attempting to automatically merge changes in files... done.

The following changes, which occurred between FreeBSD 6.3-BETA1 and
FreeBSD 7.0-BETA2 have been merged into /etc/group:
--- current version
+++ new version
@@ -1,6 +1,6 @@
-# $FreeBSD: src/etc/group,v 1.32.2.1 2006/03/06 22:23:10 rwatson Exp $
+# $FreeBSD: src/etc/group,v 1.35 2007/06/11 18:36:39 ceri Exp $
 #
 wheel:*:0:root,<CENSORED>
 daemon:*:1:
 kmem:*:2:
 sys:*:3:
@@ -9,10 +9,11 @@
 mail:*:6:
 bin:*:7:
 news:*:8:
 man:*:9:
 games:*:13:
+ftp:*:14:
 staff:*:20:
 sshd:*:22:
 smmsp:*:25:
 mailnull:*:26:
 guest:*:31:
Does this look reasonable (y/n)? y

The following changes, which occurred between FreeBSD 6.3-BETA1 and
FreeBSD 7.0-BETA2 have been merged into /etc/nsswitch.conf:
--- current version
+++ new version
@@ -1,7 +1,15 @@
+#
+# nsswitch.conf(5) - name service switch configuration file
+# $FreeBSD: src/etc/nsswitch.conf,v 1.1 2006/05/03 15:14:47 ume Exp $
+#
 group: compat
 group_compat: nis
 hosts: files dns
 networks: files
 passwd: compat
 passwd_compat: nis
 shells: files
+services: compat
+services_compat: nis
+protocols: files
+rpc: files
Does this look reasonable (y/n)? y

The following changes, which occurred between FreeBSD 6.3-BETA1 and
FreeBSD 7.0-BETA2 have been merged into /etc/ttys:
--- current version
+++ new version
@@ -1,7 +1,7 @@
 #
-# $FreeBSD: src/etc/etc.amd64/ttys,v 1.10 2003/10/24 15:44:07 simokawa Exp $
+# $FreeBSD: src/etc/etc.amd64/ttys,v 1.11 2007/05/29 06:37:57 dougb Exp $
 # @(#)ttys 5.1 (Berkeley) 4/17/89
 #
 # This file specifies various information about terminals on the system.
 # It is used by several different programs. Common entries for the
 # various columns include:
@@ -39,11 +39,11 @@
 ttyv3 "/usr/libexec/getty Pc" cons25 on secure
 ttyv4 "/usr/libexec/getty Pc" cons25 on secure
 ttyv5 "/usr/libexec/getty Pc" cons25 on secure
 ttyv6 "/usr/libexec/getty Pc" cons25 on secure
 ttyv7 "/usr/libexec/getty Pc" cons25 on secure
-ttyv8 "/usr/X11R6/bin/xdm -nodaemon" xterm off secure
+ttyv8 "/usr/local/bin/xdm -nodaemon" xterm off secure
 # Serial terminals
 # The 'dialup' keyword identifies dialin lines to login, fingerd etc.
 ttyd0 "/usr/libexec/getty std.9600" vt100 on secure
 ttyd1 "/usr/libexec/getty std.9600" dialup off secure
 ttyd2 "/usr/libexec/getty std.9600" dialup off secure
Does this look reasonable (y/n)? y

FreeBSD Update attempts to automatically merge configuration files as part of the upgrade process (the set of configuration files which it needs to merge on your system may be different from what is shown above, depending on which configuration files you have modified locally); if it can't figure out how to merge changes, it will open up an editor and ask you to manually merge the changes. After you and FreeBSD Update have merged any changes in the configuration files, it displays all the changes (between what you currently have installed and what it's planning on installing) so that you can make sure everything is as it should be.

The following files will be removed as part of updating to 7.0-BETA2-p0:
/boot/kernel/bridge.ko
/boot/kernel/g_md.ko
...
/usr/src/usr.sbin/wicontrol/wicontrol.8
/usr/src/usr.sbin/wicontrol/wicontrol.c

The following files will be added as part of updating to 7.0-BETA2-p0:
/bin/uuidgen
/boot/firmware
...
/usr/src/usr.sbin/wlandebug/wlandebug.8
/usr/src/usr.sbin/wlandebug/wlandebug.c

The following files will be updated as part of updating to 7.0-BETA2-p0:
/.cshrc
/.profile
...
/var/named/etc/namedb/named.conf
/var/yp/Makefile.dist

FreeBSD Update has downloaded all of the files needed to upgrade to the new release. You probably don't want to read through all the thousands of files which are being removed, added, or updated, but the lists are printed (and piped through more(1)) just in case.

Assuming you're satisfied that FreeBSD Update has got things right, you can tell it to go ahead and install:

# sh freebsd-update.sh -f freebsd-update.conf install
Installing updates...
Kernel updates have been installed. Please reboot and run
"freebsd-update.sh install" again to finish installing updates.

When upgrading to a new FreeBSD release, it's important to make sure that you are running the new kernel before you install the new non-kernel components -- otherwise there's a chance of things breaking horribly when programs try to use features which the old kernel doesn't understand. When FreeBSD Update asks you to reboot before continuing, do what it says!

# shutdown -r now

After the system reboots, you can log in again and tell FreeBSD Update to continug installing upgrades:

# sh freebsd-update.sh -f freebsd-update.conf install
Installing updates... done.
Completing this upgrade requires removing old shared object files.
Please rebuild all installed 3rd party software (e.g., programs
installed from the ports tree) and then run "freebsd-update.sh install"
again to finish installing updates.

Again, doing what FreeBSD Update tells you to do is a good idea. Using portupgrade to rebuild everything is a bit tricky since it can get a bit confused when upgrading the programs it uses (ruby and ruby18-bdb), but the following procedure should work:

# portsnap -I update
# portupgrade -f ruby
...
# rm /var/db/pkg/pkgdb.db
# portupgrade -f ruby18-bdb
...
# rm /var/db/pkg/pkgdb.db /usr/ports/INDEX-*.db
# portupgrade -af

At the end of upgrading ruby and ruby18-bdb, portupgrade is likely to print error messages about pkgdb.db being corrupt. Ignore these -- this is why we're deleting files before telling portupgrade to do some more work.

Finally, after all installed ports have been rebuilt, we can finish the process of upgrading the FreeBSD base system and then reboot again so that we get the new versions of daemons (e.g., sshd(8)) running:

# sh freebsd-update.sh -f freebsd-update.conf install
Installing updates... done.
# shutdown -r now

Finally, if you disabled a FreeBSD Update cron job before starting this process, don't forget to turn it back on.

Thanks to Edwin Groothuis and Simon Nielsen for helping debug this upgrade process.

FreeBSD minor version upgrades

I've posted in the past about using a modified version of FreeBSD Update to upgrade from FreeBSD 6.1 to FreeBSD 6.2. I've spent some time working on this code and I think it's now generic enough that it should work for future release upgrades, so I'll soon be committing it to the FreeBSD base system. Consequently, I'm making this post both for the present (in the middle of the FreeBSD 7.0 release cycle) and for future readers who will hopefully be reading this when they want to upgrade to FreeBSD 7.1-BETA1 or later releases.

These instructions apply only to minor version upgrades, e.g., from FreeBSD 6.something to FreeBSD 6.somethingelse, or from FreeBSD 7.something to FreeBSD 7.somethingelse. Upgrading from FreeBSD 6.x to FreeBSD 7.x is a more complicated process, and I will make another post soon to provide instructions on performing major version upgrades.

NOTE: Before proceeding, if you have FreeBSD Update scheduled to run from a cron job, disable the cron job! Bad Things will happen if FreeBSD Update tries to download security updates in the middle of this upgrade process.

As I write this, "upgrade" support does not yet exist in the version of FreeBSD Update in the FreeBSD base system, so the first step is to download the script. If you are running FreeBSD 7.0-RC1 or FreeBSD 6.3-RC1 or later, you should already have this new version of FreeBSD Update installed, so you can skip this step.

# fetch http://www.daemonology.net/freebsd-update/freebsd-update-upgrade.tgz
freebsd-update-upgrade.tgz 100% of 21 kB 74 kBps
# fetch http://www.daemonology.net/freebsd-update/freebsd-update-upgrade.tgz.asc
freebsd-update-upgrade.tgz.asc 100% of 187 B 214 kBps
# gpg --verify freebsd-update-upgrade.tgz.asc freebsd-update-upgrade.tgz
gpg: WARNING: using insecure memory!
gpg: please see http://www.gnupg.org/faq.html for more information
gpg: Signature made Sat Nov 10 04:38:54 2007 PST using DSA key ID CA6CDFB2
gpg: Good signature from "FreeBSD Security Officer <security-officer@FreeBSD.org>"
# tar -xf freebsd-update-upgrade.tgz

I've signed the tarball containing this script; as Security Officer, I strongly recommend verifying the signature using gpg before proceeding.

When I run FreeBSD Update below, I invoke it as sh freebsd-update.sh -f freebsd-update.conf in order to use this new version of FreeBSD Update and its associated configuration file; if you skiped the above step because you're already running 7.0-RC1 or 6.3-RC1 or later, you can just invoke it as freebsd-update.

This system is presently running FreeBSD 6.2-RELEASE-p8, and I'm upgrading it to FreeBSD 6.3-BETA1; obviously, if you're reading this at some point in the future and want to upgrade to something other than FreeBSD 6.3-BETA1, you should change "6.3-BETA1" below and the output from the script will be correspondingly different.

# sh freebsd-update.sh -f freebsd-update.conf -r 6.3-BETA1 upgrade
Looking up update.FreeBSD.org mirrors... 1 mirrors found.
Fetching metadata signature for 6.2-RELEASE from update1.FreeBSD.org... done.
Fetching metadata index... done.
Inspecting system... done.

The following components of FreeBSD seem to be installed:
kernel/smp src/base src/bin src/contrib src/crypto src/etc src/games
src/gnu src/include src/krb5 src/lib src/libexec src/release src/rescue
src/sbin src/secure src/share src/sys src/tools src/ubin src/usbin
world/base world/info world/lib32 world/manpages

The following components of FreeBSD do not seem to be installed:
kernel/generic world/catpages world/dict world/doc world/games
world/proflibs

Does this look reasonable (y/n)? y

Here FreeBSD Update is looking at the system and deciding which parts of FreeBSD you have installed -- obviously if you don't have the docs (documentation) from FreeBSD 6.2-RELEASE installed you aren't likely to want to have the docs from FreeBSD 6.3-BETA1 installed after upgrading.

Fetching metadata signature for 6.3-BETA1 from update1.FreeBSD.org... done.
Fetching metadata index... done.
Fetching 1 metadata patches. done.
Applying metadata patches... done.
Fetching 1 metadata files... done.
Inspecting system... done.
Preparing to download files... done.
Fetching 5088 patches.....10....20 ... 5070....5080.... done.
Applying patches... done.
Fetching 550 files... done.

The following files will be removed as part of updating to 6.3-BETA1-p0:
/usr/share/examples/netgraph/bluetooth/rc.bluetooth
/usr/share/info/bzip2.info.gz
...
/var/named/etc/namedb/PROTO.localhost.rev
/var/named/etc/namedb/make-localhost

The following files will be added as part of updating to 6.3-BETA1-p0:
/boot/kernel/coretemp.ko
/boot/kernel/if_cxgb.ko
...
/var/named/etc/namedb/master/localhost-forward.db
/var/named/etc/namedb/master/localhost-reverse.db

The following files will be updated as part of updating to 6.3-BETA1-p0:
/COPYRIGHT
/bin/[
...
/usr/src/usr.sbin/wpa/wpa_supplicant/wpa_supplicant.conf.5
/var/named/etc/namedb/named.conf

FreeBSD Update has downloaded all of the files needed to upgrade to the new release. You probably don't want to read through all the thousands of files which are being removed, added, or updated, but the lists are printed (and piped through more(1)) just in case.

Assuming you're satisfied that FreeBSD Update has got things right, you can tell it to go ahead and install:

# sh freebsd-update.sh -f freebsd-update.conf install
Installing updates...
Kernel updates have been installed. Please reboot and run
"freebsd-update.sh install" again to finish installing updates.

When upgrading to a new FreeBSD release, it's important to make sure that you are running the new kernel before you install the new non-kernel components -- otherwise there's a chance of things breaking horribly when programs try to use features which the old kernel doesn't understand. When FreeBSD Update asks you to reboot before continuing, do what it says!

# shutdown -r now

After the system reboots, you can log in again and tell FreeBSD Update to finish the upgrade; and then finally reboot a second time so that you'll have the new versions of system daemons (e.g., sshd(8)) running:

# sh freebsd-update.sh -f freebsd-update.conf install
Installing updates... done.
# shutdown -r now

Finally, if you disabled a FreeBSD Update cron job before starting this process, don't forget to turn it back on.

All told, this upgrade took me about 15 minutes -- it's fast enough that there's really no excuse for not upgrading every time a new BETA or RC (release candidate) is released.

Coming up soon: Upgrading from FreeBSD 6.x to FreeBSD 7.x.