Daemonic Dispatches

Broken PMTUD on Amazon EC2

While at Amazon re:invent I had the opportunity to complain to some Amazonians again about an EC2 bug which has been annoying me for a long time: The default firewall rulset is broken. I discovered this three years ago while debugging odd problems experienced by a Tarsnap user — sending a small amount of traffic worked fine, but as soon as large amounts of traffic started moving around, the TCP connection got stuck — and I've been complaining from time to time ever since; but somehow face-to-face communications tend to produce better results than mere emails.

As most standards-aware network administrators know, blocking ICMP is evil. Sure, there are certain ICMP packets which should be blocked; but to quote RFC 2979:

A packet-filtering router acting as a firewall which permits outgoing IP packets with the Don't Fragment (DF) bit set MUST NOT block incoming ICMP Destination Unreachable / Fragmentation Needed errors sent in response to the outbound packets from reaching hosts inside the firewall, as this would break the standards-compliant usage of Path MTU discovery by hosts generating legitimate traffic.

and yet this is exactly what the default EC2 firewall does: It blocks all incoming ICMP packets, including ICMP "fragmentation needed" packets. Worst of all, this doesn't cause networking to completely fail — in which case the problem would be immediately apparent — but instead it causes problems only for hosts with smaller path MTUs... including the aforementioned Tarsnap user, who had recently started using a new ADSL modem with Point-to-point protocol over Ethernet.

The fix for EC2 users is easy, and I recommend applying it universally: Run

ec2-authorize default -P icmp -t 3:4

from the command line, or go into the EC2 management console and add a firewall rule to allow ICMP "destination unreachable" (type 3) "fragmentation needed" (code 4) packets to pass. If you have multiple EC2 security groups or are using multiple AWS regions, you'll need to apply this fix for each of them.

The fix for Amazon is also easy: Change the default firewall ruleset for new users. While I appreciate the argument that Amazon shouldn't make unexpected changes to a user's firewall rules "behind their back", this doesn't apply to new EC2 users; and I see no justification for giving new users a default configuration which both violates internet standards and causes real-world breakage.

blog comments powered by Disqus