Daemonic Dispatches

Automated FreeBSD panic reporting

It is now very common for software to have built-in mechanisms for reporting crashes. Windows, OS X, Ubuntu, Android, KDE, Mozilla... there are few large codebases which don't have any such functionality. Until a few days ago, FreeBSD was an exception: The instructions on Kernel Debugging are hidden away in the "Developer's Handbook", and for users who are not in a position to diagnose the cause of a kernel panic themselves, all that could be done is to submit a bug report via the "send-pr" utility — at which point it would join the other 500+ panic reports sitting in FreeBSD's mostly-ignored GNATS repository. A couple of weeks ago, I decided it was time to do something about this.

On Monday I added new "panicmail" code to the FreeBSD ports tree. This code, if installed, will gather basic information about a FreeBSD kernel panic — not much more than the backtrace — and submit it to a central repository (currently known as my inbox). The submitted data is encrypted to ensure that nobody can snoop on any sensitive information (a backtrace which goes through ZFS would reveal that you use ZFS, for example) and in case you're not sure that you trust me with the contents of your kernel panics, the default behaviour is to email them to root along with instructions for how to submit them once you've verified they don't contain anything confidential.

Once I have a significant number of panic submissions, I'll start processing them to provide FreeBSD developers with aggregate data — information which is not tied to any particular machines, but can still be useful by telling developers where to look for the most common kernel panics.

I'm expecting this to be a very useful tool for the FreeBSD project; I know this type of analysis of automatic crash reports has been invaluable for other operating systems, and two FreeBSD-based projects — my work on FreeBSD/EC2, and FreeNAS — have been using automated panic reporting already, with good results. However, this depends on FreeBSD users install and enable the code... and given that the FreeBSD kernel doesn't panic very often, getting a useful number of panic reports requires that as many FreeBSD users as possible contribute.

To install and enable panic reporting on your FreeBSD system:

1. Install the sysutils/panicmail port.
2. Add the lines

   dumpdev="AUTO"
   panicmail_enable="YES"

   to your /etc/rc.conf file.
3. Make sure that email sent to root goes somewhere (if you're receiving nightly "daily run output" and "security run output" emails, you're good).

Remember, the larger the number of systems configured to submit panics, the more useful this will be!