Daemonic Dispatches

Broken PMTUD on Amazon EC2

While at Amazon re:invent I had the opportunity to complain to some Amazonians again about an EC2 bug which has been annoying me for a long time: The default firewall rulset is broken. I discovered this three years ago while debugging odd problems experienced by a Tarsnap user — sending a small amount of traffic worked fine, but as soon as large amounts of traffic started moving around, the TCP connection got stuck — and I've been complaining from time to time ever since; but somehow face-to-face communications tend to produce better results than mere emails.

As most standards-aware network administrators know, blocking ICMP is evil. Sure, there are certain ICMP packets which should be blocked; but to quote RFC 2979:

A packet-filtering router acting as a firewall which permits outgoing IP packets with the Don't Fragment (DF) bit set MUST NOT block incoming ICMP Destination Unreachable / Fragmentation Needed errors sent in response to the outbound packets from reaching hosts inside the firewall, as this would break the standards-compliant usage of Path MTU discovery by hosts generating legitimate traffic.

and yet this is exactly what the default EC2 firewall does: It blocks all incoming ICMP packets, including ICMP "fragmentation needed" packets. Worst of all, this doesn't cause networking to completely fail — in which case the problem would be immediately apparent — but instead it causes problems only for hosts with smaller path MTUs... including the aforementioned Tarsnap user, who had recently started using a new ADSL modem with Point-to-point protocol over Ethernet.

The fix for EC2 users is easy, and I recommend applying it universally: Run

ec2-authorize default -P icmp -t 3:4

from the command line, or go into the EC2 management console and add a firewall rule to allow ICMP "destination unreachable" (type 3) "fragmentation needed" (code 4) packets to pass. If you have multiple EC2 security groups or are using multiple AWS regions, you'll need to apply this fix for each of them.

The fix for Amazon is also easy: Change the default firewall ruleset for new users. While I appreciate the argument that Amazon shouldn't make unexpected changes to a user's firewall rules "behind their back", this doesn't apply to new EC2 users; and I see no justification for giving new users a default configuration which both violates internet standards and causes real-world breakage.

Upcoming travel

In the next few weeks, I'm going to be travelling around a bit: two conferences, followed by a bit of touristing in Europe before I return home. Aside from the hours when I'm on planes, I'll still be connected to the world — I plan on buying local prepaid SIM cards for my phone and tethering my laptop — but I may be slower than usual replying to emails. On the other hand, my travels may take me in closer proximity to people; and I'd be very happy to meet people along the way.

My first stop is in Las Vegas for AWS re:invent, on November 27th-30th (the event is the 27th through the 29th, but I'm sticking around for another day before flying out). When I first saw this announced, I was skeptical: It seems far too much "manager-oriented" and nowhere near "developer-ish" enough to be useful (of course, my bar for comparison is BSDCan, which is a phenomenal conference for us technical types). I'm still somewhat skeptical, given that a lot of the talks seem — based on the abstracts on the website — to fall into the category of "stuff you would know if you read the AWS documentation"; but the list of speakers makes me optimistic that there will be interesting things to learn... if not in the official talks, at least in the "hallway track". Another reason for attending is my work on getting FreeBSD into EC2: I've love to meet people who are using, or are considering using, FreeBSD on EC2, and I figure a conference with 5000 AWS users can't be a bad place to look. If you're going to be at re:invent and you're interested in FreeBSD, let me know!

My second stop is in Oslo for Passwords^12 on December 3rd-5th. There I'll be speaking about the scrypt key derivation function; the conference talk in 2009 when I released scrypt was aimed at a fairly general developer audience, so I'm looking forward to presenting it to a more cryptographically-inclined group. I imagine I'll have an opportunity to meet everybody at Passwords^12 during the conference — unlike re:invent, I don't expect Passwords^12 to have 5000 attendees! — but I'd be happy to meet up with anyone else in Oslo to discuss FreeBSD or Tarsnap or scrypt or anything else.

From there I'm travelling on to Vienna for a weekend and some opera — I was concertmaster in a student production of Bohème at university, and there's no way I would pass up an opportunity to see it performed at the Wiener Staatsoper — and then a short stop in Stuttgart before I land in Vancouver at 1AM on December 12th.

In total, this trip involves 2 conferences, 4 destinations, 4 airlines, 9 cities, 10 flights, and 36 hours and 14600 miles of flying. Six months ago I told myself that I would never again schedule myself for two conferences in consecutive weeks; at this point, I'm just hoping my luggage avoids getting lost along the way.