Locking down the Instance Metadata Service: Announcing imds-filterd
I have written in the past about the interaction between the EC2 Instance Metadata Service and IAM Roles; my comment at the time was that this was "EC2's most dangerous feature". Since then we've seen a number of companies fall prey to this trap — probably most famously, Capital One's data breach affecting 100 million credit card customers took advantage of this EC2 feature.As the founder of a startup which cares passionately about security — and as a FreeBSD Security Officer Emeritus and the maintainer of the FreeBSD/EC2 platform — I decided a few months ago that I could no longer sit by idly while this situation continued. Today, I'm happy to announce the availability of imds-filterd.
The imds-filterd tool allows administrators of EC2 instances to lock down which data from the Instance Metadata Service can be accessed by specified system users and groups, thereby making the EC2 Instance Metadata Service compatible with traditional UNIX privilege separation.
UNIX systems have long made extensive use of user and group level privileges for security purposes. Not only do accounts belonging to different human users have different privileges, but daemons often operate with restricted privileges, in order to reduce the system security impact in the event that a vulnerability or misconfiguration is found; some software even operates with different parts running as different "users" at the same time, in order to protect one part of the software from bugs in other parts. Unfortunately, in the EC2 environment every user — including those which are intended to have almost no capabilities whatsoever — can access the Instance Metadata Service, including any Amazon Web Services credentials which it exposes. As a result, the Instance Metadata Service acts as an effective "vulnerability mitigation mitigation", sabotaging attempts by system administrators and software authors to use privilege separation to protect systems from vulnerabilities.
The imds-filterd tool intercepts requests to the Instance Metadata Service and redirects them into a virtualized environment where they are validated based on a configured ruleset and the user and group IDs of the process making the request. This makes it possible to allow processes to access "harmless" paths in the Instance Metadata Service (e.g. to look up the EC2 instance type or availability zone) while allowing only appropriately privileged daemons to access sensitive information such as IAM Role credentials.
FreeBSD users can install imds-filterd via the FreeBSD ports tree as security/imds-filterd, and it will soon be available via the pkg system ("pkg install imds-filterd"). After installing, set imds_filterd_enable="YES" in /etc/rc.conf to enable imds-filterd and (optionally) configure the access control rules via /usr/local/etc/imds.conf. I anticipate porting imds-filterd to other operating systems in the future (patches are welcome).
The imds-filterd code is now available for beta testing and code review via the github repository.
FAQ
Q. How does this compare to Instance Metadata Service v2?A. The changes Amazon made in Instance Metadata Service v2 (aka "IMDSv2") address misconfigurations, but not security vulnerabilities. Requiring that an HTTP PUT be performed in order to obtain a token protects against many open HTTP proxies which permit GETs to pass but do not permit PUTs; and setting an IP TTL of 1 protects most systems which allow attackers to route packets through them (e.g. VPNs). However, nothing in IMDSv2 protects against the situation for which privilege separation is intended — where a process running as an unprivileged user has been compromised. Using imds-filterd inherently protects against a wider range of attacks than relying on IMDSv2.
Q. Why not just use a firewall to limit access to the IMDS?
A. A firewall can completely block access to the IMDS, but traditional
packet filters cannot grant partial access based on the specific path being
requested. There may be processes which have a legitimate need to access
some information from the IMDS while not needing to access other parts. This
will become a more serious consideration if EC2 gains the ability to attach
multiple IAM Roles, since it would be useful to grant processes access to some
roles but not all of them. Firewalls also have a performance cost (see
below).
Q. Why not use a firewall to redirect traffic to the proxy process
rather than using tunnels and routing rules?
A. Packet filtering firewalls inherently carry a per-packet performance cost;
while this may be insignificant for many systems, FreeBSD is routinely used
for high-bandwidth applications (Netflix has delivered 190 Gbps of HTTPS
traffic from a single system) where consulting even a simple firewall ruleset
could pose an unreasonable performance burden. Routing rules, in contrast,
are consulted only when a TCP connection is established.
There is also a manageability benefit, in that many systems already make use of firewalls, and depending on the configuration it might not be obvious where to add new rules for redirecting IMDS traffic. Adding a new rule to a routing table, on the other hand, is highly unlikely to interfere with any existing behaviour.
Q. Why does imds-filterd have two separate processes running?
A. There are two reasons for this. First, system administrators may
need to adjust the filtering ruleset used; having the HTTP proxy code run as a
separate process allows it to be stopped and restarted without halting the
process which intercepts outgoing requests.
Second, the HTTP proxy code runs inside a virtualized network environment in order to avoid interfering with any processes on the host system (e.g., due to binding to TCP port 80).
Q. Why didn't Amazon write this tool (or pay for someone to write it)?
A. I can only speculate that they didn't see the need. Amazon leans
very heavily on the principle that EC2 instances should be used as privilege
boundaries; if you're worried about attackers who have local kernel arbitrary
code execution vulnerabilities in their back pocket, this is the only reasonable
position to take. On the other hand, many attackers can't trivially
escalate from executing code as one user to gaining complete control of a
system, and defense in depth never hurts; so I personally feel that they were
misguided to ignore this possibility.
Q. So who did pay for this? A. This work was sponsored by Tarsnap, my secure online backup startup; and by my FreeBSD/EC2 fundraiser on Patreon. Of course, if Amazon (or anyone else) wants to support this, I certainly wouldn't refuse additional funding.