The HTTP 500 Solution
My online backup service offers bug bounties for mistakes in the client software; while I explicitly exclude bugs in the Tarsnap website (with the exception of cosmetic errors) in an attempt to discourage people who blindly run vulnerability scanners against websites, I still get a lot of bogus "bug reports". One of the more common reports is "I managed to trigger an HTTP 500 Internal Server Error" response; what people don't realize is that this is in fact entirely deliberate, as part of what I call the "HTTP 500 Solution".There are several ways to trigger such responses from my CGI scripts. For example, sending an application/x-www-form-urlencoded input which is syntactically invalid; sending a form or query string which has invalid percent-encoding; or sending a form or query string which encodes for a NUL byte in a name or value. All of these have one thing in common: They should never happen. Upon encountering such an invalid request, my CGI scripts respond very simply: They immediately exit — at which time Apache, having not seen any valid HTTP response from them, sends back a canned "Internal Error" response.
The crux of the matter is this: A CGI script offers a contract. Give it some appropriate input, and it will give you some output. But if you don't give it appropriate input, it doesn't owe you anything. Not a redirect to some other page on the website; not a helpful message to tell you precisely how your input was invalid; and certainly not an attempt to "fix" your input by stripping out the invalid parts. If you don't want to play by the rules, it's perfectly entitled to take its ball and go home.
More importantly, attempting to respond to invalid inputs is dangerous, for two reasons. First, in such a scenario we've already identified that someone is trying to be evil; it's entirely possible that there is more evil going on than we detected. And second, error-handling paths never receive as much testing, because they are not frequently encountered "in the field"; no matter how comprehensive your testing is (and most isn't, since there's almost always a bias towards testing things which should suceed rather than testing things which should fail) it will never cover as many combinations of strange inputs and obscure environments as your code is likely to encounter in production. Poorly tested code has more bugs per line; and when you're faced with the likelihood of a large number of bugs per line of code — whether it's due to poor testing, poor developers, or merely because the code is difficult for reviewers to understand — it's important to minimize the amount of bug-prone code. The shortest possible error-handling path is "exit(1);".
Postel's law states that one should be conservative in what one does, but liberal in what one accepts from others; this is great advice for life, but terrible advice for writing software. When it comes to software security, it's better to be conservative in what one does and paranoid in what one accepts from others — and prone to freeze when surprised.
A FreeBSD AMI Builder AMI
I've been working on the FreeBSD/EC2 platform for a long time; five years ago I finally had it running, and for the past few years it has provided the behaviour FreeBSD users expect — stability and high performance — across all EC2 instance types. Making the platform work was just the first step though; next comes making it usable.Some people are happy with simply having a virtual machine which runs the base FreeBSD system; for them, the published FreeBSD/EC2 images (which, as of FreeBSD 10.2-RELEASE, are built by the FreeBSD Release Engineer) will be sufficient. For users who want to use "stock" FreeBSD but would like to have some extra setup performed when the instance launches — say, to install some packages, edit some configuration files, and enable some services — I wrote the configinit tool. And for users who need to make changes to FreeBSD itself, I added code for building AMIs into the FreeBSD source tree, so you can take a modified FreeBSD tree and run make ec2ami to generate a reusable image.
There was one group for whom I didn't have a good solution yet, however: Users who want to create FreeBSD AMIs with minor changes, without wanting to go to the effort of performing a complete FreeBSD release build. Ironically, I am exactly such a user: All of the EC2 instances I use for my online backup service make use of spiped to protect sshd and provide encrypted and authenticated tunnels to my mailserver and package server; and so having spiped preinstalled with the appropriate keys would significantly streamline my deployment process. While it's possible to launch a FreeBSD EC2 instance, make some changes, and then ask EC2 to create a new AMI out of it, this rarely produces a "clean" AMI: A lot of code runs when an EC2 instance first launches — creating the ec2-user user, installing the appropriate SSH public key, creating SSH host keys, growing the root filesystem if launched with a larger root disk, downloading and installing updates to FreeBSD, downloading and installing packages... — and much of this needs to be manually reverted before a reusable AMI can be created; not to mention command histories and log files written during the configuration process, which the more fastidious among us may wish to avoid publishing. To solve this problem, I present the FreeBSD AMI Builder, now available as ami-28682f42 in the EC2 US-East-1 region.
Much like my depenguinator, the FreeBSD AMI Builder works by booting a full FreeBSD system into a memory disk; having done that, it no longer needs to access anything on disk, and so it can safely unmount and overwrite the existing on-disk filesystem. A fresh FreeBSD/EC2 image is written onto the disk and then mounted at /mnt, at which point you can SSH into the instance — which, since it is running in a memory disk, can allow you to SSH in without touching the image you're creating — and then make whatever changes you'd like. (The one minor disadvantage to running in a memory disk is that t2.micro instances don't have enough RAM; all the larger instance types work fine, though, and this limitation only applies to the process of building AMIs, not to instances you launch later.)
This becomes even better with a clever feature Amazon added to EC2 a few years ago: IAM Roles. In short, IAM Roles allow you to provide ephemeral AWS keys to an EC2 instance. To make use of this, create an IAM Policy which grants access to the EC2 "CreateImage" API call, and an IAM Role which has that policy attached; then launch the FreeBSD AMI Builder with that Role attached. Now you can run
# mkami "My AMI Name" "My AMI Description"and through the magic of IAM Roles the EC2 instance will be able to issue the necessary API call to create an AMI out of itself.
For all that the FreeBSD AMI Builder is a useful tool, it also demonstrates what can be done using configinit: I created the AMI by launching a FreeBSD/EC2 instance with this script provided as EC2 user-data. When the instance booted, it fetched the EC2 user-data; identified it as being a shell script and executed it; and that script assembled the disk image required for running the AMI Builder without my ever having to SSH into the instance. In short, that shell script is a FreeBSD AMI Builder AMI Builder.
But wait, there's more! The FreeBSD AMI Builder is itself a FreeBSD/EC2 AMI, so it too has access to configinit. If you set up the aforementioned IAM Role and launch the AMI Builder with this EC2 user-data
it will install Apache 2.4, enable it, create a new AMI, and then shut down the AMI Builder instance. (The separate pkg commands for downloading and installing packages are necessary to work around a bug in pkg relating to hard links and alternate roots.) A short time later, and you have an AMI with Apache 2.4 preinstalled and enabled, ready to be launched. Of course in this case you could simply use the "stock" FreeBSD AMI and have Apache installed at boot time using the appropriate EC2 user-data; but a clean and repeatable customized AMI build in less than ten minutes is nonetheless impressive!#!/bin/sh export ASSUME_ALWAYS_YES=YES pkg -r /mnt fetch -d apache24 pkg -c /mnt install apache24 echo apache24_enable=YES >> /mnt/etc/rc.conf mkami "FreeBSD 10.2 w/ Apache 2.4" "FreeBSD with Apache pre-installed" shutdown -p now
One of the difficulties I've encountered in getting people to use FreeBSD on EC2 is that there are very few "customized" FreeBSD/EC2 images available. I know new EC2 users often find preconfigured images very helpful, and once they start out on Linux they tend to stay there. While I see lots of exciting things happening in the FreeBSD community, we're not nearly good enough at showing it off to the rest of the world. This may make that final step a little bit easier.
Finally, if you find this useful, please consider donating to the FreeBSD Foundation. As I write this they're at $481,324 out of a target of $1,250,000 of donations for the year, with less than six weeks left; they really do wonderful work supporting the FreeBSD project and deserve all the support we can offer them.