Protecting sshd using spiped
Slightly over a year ago, I announced the release of the spiped secure pipe daemon. The purpose of spiped was to provide a more robust and vastly simpler alternative to 'ssh -L' in cases where a pre-shared secret key could be securely established between hosts. SSH's host key fingerprints are great when one person is setting up a system and wants to broadcast a public value to a large number of users; but a far more common scenario for me is that I am setting up multiple systems and want to allow daemons running on them to talk to each other.In addition to spiped's ability to replace 'ssh -L', there's another way in which spiped can supplement SSH: To restrict access to the SSH daemon. Like many system administrators, I used to restrict access to port tcp/22 on most of my servers based on source IP address; this provided some protection from "zero-day" exploits against OpenSSH, as well as eliminating the annoying "log spam" caused by brute force attacks. This worked fine as long as I always connected from the same location, but heading off to conferences meant that I needed to either tunnel SSH connections over other SSH connections or make temporary changes to my firewall rules.
By running spiped on a server, configured to accept connections on port 8022 and connect to port 22 on the same host, it becomes possible to firewall off port tcp/22 completely and yet still be able to connect over SSH from anywhere in the world — as long as you have the spiped secret key, of course. This approach has one disadvantage however: Since spiped involves a pair of daemons — one running on the server, receiving encrypted connections and establishing unencrypted connections to the target service, and one running on the client, receiving unencrypted connections and establishing encrypted connections to the first spiped dameon — if you want to be able to SSH to N servers via spiped, you need to be running N copies of spiped on your local system. These days, I find myself running a sufficiently large number of systems (a quick count comes to 17, but I've probably forgotten some) that this approach is a bit unwieldy.
Enter spipe, a "client" for the spiped protocol, which I committed to the spiped SVN repository yesterday and released as part of spiped 1.2.0 today. Rather than listening on a socket and encrypting or decrypting many connections, as spiped does, spipe encrypts a single connection — and as the unencrypted end of that connection, uses the standard input and output. In other words, spipe acts like 'netcat' or 'openssl s_client' — or an OpenSSH ProxyCommand.
Setting up spiped to guard access to SSH is now easy: On your server, run
# dd if=/dev/urandom bs=32 count=1 of=/etc/ssh/spiped.key
# spiped -d -s '[0.0.0.0]:8022' -t '[127.0.0.1]:22' -k /etc/ssh/spiped.key
spiped_enable="YES" # if not already present
spiped_pipes="SSH" # and any other pipes you have already configured
spiped_pipe_SSH_mode="decrypt"
spiped_pipe_SSH_source="[0.0.0.0]:8022"
spiped_pipe_SSH_target="[127.0.0.1]:22"
spiped_pipe_SSH_key="/etc/ssh/spiped.key"
Now copy the /etc/ssh/spiped.key file from the server to ~/.ssh/spiped_HOSTNAME_key on your local system, and add the lines
Host HOSTNAME
ProxyCommand spipe -t %h:8022 -k ~/.ssh/spiped_%h_key
It is often said that the UNIX model of software involves writing simple components which each solve one small problem but are versatile enough that they can can be easily assembled to solve more larger and more complex problems. At 4000 lines, spiped is certainly simple, and it's hard to imagine any way that spiped could be useful by itself; but for its ability to be integrated with other daemons to add additional encryption and authentication to them, I think spiped might well qualify as the best UNIX software I've ever written.
Portifying FreeBSD/EC2 startup scripts
Ever since I first announced the availability of FreeBSD/EC2, I've been trying to get rid of all the "secret sauce" involved in my builds. I'm now one step closer to that goal: The FreeBSD/EC2 startup ("rc.d") scripts are now in the FreeBSD ports tree.There was one complication which had me stuck for a while: Installing the port into a new disk image. The usual solution to this problem is to chroot into the image; but since this I'm doing this as part of a release-building process, the binaries in the image under construction are normally from a newer version of FreeBSD than the kernel running from the "host" FreeBSD — which means they might rely on new system calls and fail when those aren't available. As a result, it's necessary to install the sysutils/ec2-scripts port without using anything from inside the image under construction.
I spent several hours struggling with this and asking FreeBSD developers for ideas, and was about to give up on finding a good solution and instead start writing code to perform a "fake" package install — i.e., install the requisite bits and packaging metadata "by hand" rather than using the mechanisms in the FreeBSD ports tree — when Scott Long came up with the solution: Use unionfs to mount parts of the guest on top of the host, then install the port on the host — at which point they end up in the right places in the guest. Or in code:
# mount_unionfs /mnt/image/usr/local /usr/local # mount_unionfs /mnt/image/var/db/pkg /var/db/pkg # cd /usr/ports/sysutils/ec2-scripts && make install clean # umount /var/db/pkg # umount /usr/local
Of course, this trick isn't always going to work for creating disk images with preinstalled packages — if you're building an embedded system running a different architecture, this would install binaries built for the host's architecture instead; the FreeBSD ports tree unfortunately does not have any mechanism for cross-building packages yet. In the case of the FreeBSD/EC2 startup scripts, however, it works perfectly — there aren't any binaries to build at all, never mind cross-architecture problems.
What's left for FreeBSD/EC2? Two kernel patches (one to work around an EC2 bug; one to work around a FreeBSD/Xen/Linux networking incompatibility) and either adding the XENHVM kernel to the FreeBSD release builds or merging the XENHVM bits into the GENERIC kernel; then FreeBSD/EC2 will simply be "release bits plus sysutils/ec2-scripts package plus configuration files". It isn't going to happen for FreeBSD 9.1, but I think the odds are quite good that once FreeBSD 9.2 comes around, FreeBSD/EC2 will be just as much "official FreeBSD" as what you get from any other FreeBSD server hosting service.
Tarsnap now takes credit cards (without Paypal)
One of the most frequent complaints I've heard about Tarsnap over the past four years has been that the only way to pay for Tarsnap usage has been via Paypal. While Paypal works — most of the time — it occasionally rejects payments or flags them as "suspicious", and has a reputation for "freezing" accounts — a bad thing to have happen if you want to access your money in the next 6 months, and a very bad thing to have happen if you're running a company and don't have any other way to take payments. Thanks to Stripe, Tarsnap is no longer wholly dependent upon Paypal: You can now make payments by entering your credit card details directly into the Tarsnap website.I've been excited about Stripe ever since one of the founders, Patrick Collison, emailed me in 2009 to say that he was "trying to build a credit-card processor that doesn't suck". My immediate reaction: "I think that might be the most ambitious plan I've ever heard from someone associated with YC". When Patrick went on to describe how it would work — abstracting away all the paperwork of credit card "merchant accounts", having clean RESTy APIs, avoiding the need for PCI compliance by never having credit card numbers touch my server, etc. — I knew it was going to be a service I would want to use. It took a while before this became possible — when Stripe launched last year, it was limited to the US, and Tarsnap is a Canadian company — but last week they emailed to invite me to beta test their soon-to-be-publicly-launched support for Canadian merchants.
One of the neat tricks used by Stripe is how they allow people to accept credit cards without handling any credit card details. When you fill out the credit card form on the Tarsnap website — or most other Stripe-using sites — and hit the submit button, Stripe's javascript leaps into action. Rather than having the credit card details submitted along with the rest of the form, the Stripe javascript sends them directly to the Stripe servers, which send back a single-use "token". This token then gets submitted with the rest of the form; and when it comes time to charge the credit card, instead of sending an API request to the Stripe servers saying "please charge $X to this credit card", an API request goes out saying "please charge $X to that card I haven't seen but was given this token in place of". Obviously if an attacker manages to compromise the web server which is serving up the credit card entry form, they can redirect the card details — that's unavoidable — but since no credit card numbers ever reach the web server there's no risk of them being improperly stored — a fact of critical importance to the credit card networks, since it ensures that an attacker could only get a few hours or days of cards (assuming the attack is discovered reasonably promptly) rather than being able to retrieve years of previously-used cards.
As useful as this trick is, it posed a problem for Tarsnap. Users make Tarsnap payments after logging in, so any javascript loaded into Tarsnap's payment page is executed with the credentials of a logged-in user. Now, I'm confident in the Stripe team's attention to security, but I don't call Tarsnap "online backups for the truly paranoid" for nothing: There's no way I'm going to trust someone else's javascript with that power. (For the same reason, I use Google Analytics on the Tarsnap website but only when pages are loaded via HTTP, not on pages loaded via HTTPS.)
To get around this problem, I took advantage of a feature of Javascript's "Same Origin" policy: If a page on domain X contains an <iframe> tag which loads content from domain Y, any javascript on the iframed page is executed with the privileges of domain Y — meaning that it can't do anything naughty to content from domain X. Hopefully at some point Stripe will provide such an iframe solution themselves; but until then, since I figured it might be useful to more people than just me, I decided to make this available as a free service to the world. Paymentiframe.com hosts a CGI script which generates a Stripe "payment iframe" given the necessary parameters (form submission URL and Stripe publishable API key, plus other optional parameters to change the submission button text or add hidden form fields) — now anyone who wants to accept credit cards via Stripe can drop a single <iframe> tag into their website and avoid any javascript "contamination".
With that done, the remaining Stripe integration was easy: When a form arrives at the Tarsnap web server with a Stripe token, the Tarsnap account management code goes through the usual steps to ensure that there is a logged-in user, then sends an HTTPS request to the Stripe API endpoint. It gets back a success or failure response — the body of which it doesn't even parse, since Stripe also signals success or failure via the HTTP status code — then performs Tarsnap's standard payment processing steps of adjusting the user's Tarsnap account balance and sending out an email.
I did cheat in one way however: If the Tarsnap server crashes after sending the Stripe API request but before recording the payment, there is no way for it to automatically recover; I would have to manually process the payment in that case. To do things entirely "correctly" I would have to use Stripe's webhooks — like Paypal's IPNs, these send notifications to a CGI script when an event has occurred, and (most importantly) keep resending the notifications until they are successfully processed. I spent a long time agonizing over this — my background in mathematics makes me a firm believer in doing things right and handling even the most obscure edge conditions — but ultimately I decided that this was a sufficiently unlikely case that I could afford to handle it by hand if it arose.
But there you have it: An iframe and an API call, and Tarsnap users no longer need to go anywhere near Paypal. A fact which I'm sure will make many Tarsnap users very happy, and one in particular happier than most: Stripe has been using Tarsnap since late 2010.