Daemonic Dispatches

Target reached!

Thanks to 148 generous donations, I have now reached my funding target for my summer of FreeBSD security development. Particularly worthy of note are donations of 1000 USD and 6500 USD, sent by pil.dk and Pair Networks respectively; but I am grateful for all the donations I received, whether five dollars or sixty-five hundred dollars.

For the first two weeks of May I will be finishing my BSDCan paper and presentation, doing some minor cleanup work on the portsnap client, and then attending the conference itself, where I will be talking about some of the difficult not-entirely-technical questions involved in FreeBSD security and gathering feedback about what everybody would like to see in the next version of FreeBSD Update.

After I return from the conference, I'll most likely head straight into rewriting the FreeBSD Update server code, but I don't want to overly commit myself to any particular schedule yet -- for one thing, a security issue could arise which I might need to spend a week investigating -- but I will be trying to keep this site updated with my progress as much as possible.

As they say, watch this space!

My Very Important Response

In a recent article entitled "Colin's Very Important Response", Thomas Ptacek responded to my last post here; while I'm glad that he has admitted to getting some of his facts wrong, there are still some significant errors.

First, Ptacek now claims that I posted my paper 'months after Osvik and Tromer published what is now "Cache Attacks and Countermeasures: the Case of AES"'. His chronology here is completely wrong: while Shamir famously warned of unspecified dangers inherent in Hyper-Threading in the Cryptographers' Track of RSA 2005 -- some four months after I first discovered this problem -- the Osvik-Shamir-Tromer paper was not written until much later: In fact, a few days after I released my paper (at which point it had been circulating for almost three months with only minor changes) I received an email from Tromer describing their paper as not yet being finished. Of course I didn't cite the work of Osvik and Tromer -- not only had I not yet seen their work, they hadn't even finished writing it! (On the other hand, in the version of my paper which I submitted to the Journal of Cryptology four months later -- by which point the Osvik-Shamir-Tromer paper was published on the web -- I do cite their work and point out the similarities.)

Ptacek goes on to list the reasons he accused me of self-promotion. To respond briefly:

• I will not back down from my argument that servers should be secure by default, with an option available to make them insecure and possibly gain some performance.
• I never attempted to "monopolize attention", and I have always agreed that Bernstein's attack also needed to be addressed.
• I never accused anyone of secretly working for Intel; the closest I ever came to this was a tongue-in-cheek question on a FreeBSD mailing list when someone made (inaccurate) comments which mirrored what Intel's PR people had been saying. However, it is true that Intel attempted to stop me from releasing my paper.
• I never picked a fight with Linus; while I did refer to him as a "dictator", I did so in exactly the same manner as everybody else writing in the area of comparative open source project management.
• If I had been wearing only a cryptographer's hat, I would have gone ahead and submitted my paper to J. Cryptology in March 2006. Instead, I wore two hats: first that of a cryptographer, and second that of a member of the FreeBSD Security Team. The policy of the FreeBSD Security Team is that local privilege escalation and information leakage problems are resolved via security advisories; this policy existed long before I became involved in FreeBSD.

Next Ptacek points out that I have not provided evidence that I did not undertake gainful employment during the period when I was working on this issue. I'm not quite sure what evidence he wants -- my income tax return, perhaps? Does he want a list of the companies which asked me to interview for jobs, and the professors who invited me to apply for post-doctoral research positions? Of course, if I were lying about this I could easily forge such documents, so even if I were prepared to make them publicly available -- which I'm not, for obvious reasons -- it would serve little purpose.

After some floundering concerning side channel attacks -- cryptography which, by his own admission, he doesn't understand -- Ptacek concludes by stating that "any localhost kernel of privilege escalation finding Colin published would be more impactful". In the very narrow sense of FreeBSD security, Ptacek is quite correct here. However, unlike most local kernel privilege escalation attacks, which except in very rare cases only affect a single operating system, the Hyper-Threading side channel I demonstrated affected all SMP i386 operating systems. This wide range of affected systems makes an otherwise less significant issue worthy of more widespread attention; but even without that, the fact that my paper was the first publicly available work which demonstrated the exploitability of the shared L1 cache on Hyper-Threaded processors makes it worth noticing.

While the Blogosphere seems to have taken over from Usenet as the home of "Everybody is entitled to their opinion, even if they're completely wrong", I wish people would make more effort to check their facts before criticizing other people: Incorrect facts make the person posting them look ignorant, while incorrect criticisms tend to make both sides look bad.

My Very Important Knob

In a recent article entitled "Colin's Very Important Knob", Thomas Ptacek made a series of untrue statements about my work. In true Internet fashion, I'm taking this opportunity to climb onto my own soapbox to defend myself.

The first inaccuracy in his article was to state that I "cribbed" my Hyper-Threading side channel work "directly from Dan Bernstein". In fact, I did nothing of the sort. I first became aware of Bernstein's work in the area of cache-related side channels when he posted to sci.crypt with a URL to his work on cache timing attacks against AES -- work which, at that time, made no mention of Hyper-Threading. In contrast, I had at that point already been investigating the question of Hyper-Threading side channels for weeks. Furthermore, the attack against RSA implementations on Hyper-Threading is very different from Bernstein's attack against AES implementations: Where Bernstein considered the inevitable effects of a cache upon the performance of code, I considered the effects of code upon the cache and how those could be measured.

The second accusation made by Ptacek is that I "went on a blitz of self-promotion". My collecting of vendor responses -- something Ptacek cites as evidence of my self-promotion --- is nothing unusual when coordinating a security advisory affecting multiple operating systems: When CERT coordinates advisories, they always do this. Given the number of people who emailed me to ask if their Windows systems were affected, I wish Microsoft had also provided me with a response.

Ptacek goes on to claim that "there are lots of easy timing side channels if you can run code on the same host as your target". While he correctly points out that a local attacker can often retrieve information about a passphrase being typed by measuring inter-keystroke timings, he makes the entirely unfounded jump from here to concluding that any information can be stolen in the same manner. If Ptacek is really aware of "lots of easy side channels" which can steal RSA keys, I hope he will come forward with details -- for one thing, I'm sure the OpenSSL developers would like to know about them!

Next, Ptacek implies that I, as a single voice, somehow tricked FreeBSD into disabling Hyper-Threading. Perhaps Ptacek hasn't read the papers which he himself is citing: In an updated version of his AES cache timing paper, Dan Bernstein argues that Hyper-Threading should be disabled, while Osvik, Tromer, and Shamir -- who constructed their Hyper-Threading attack against AES completely independently of my work -- came to the same conclusion. On the operating system side, SCO and several Linux distributions followed FreeBSD's lead; from private discussions I know that some other operating systems were planning on doing likewise, but I do not have confirmation that they did so.

At this point, Ptacek points to a recent FreeBSD commit, wherein I changed the default behaviour to allow shared L2 caches -- but still not shared L1 caches -- in order to avoid halving the performance of Intel Core Duo processors. He claims that my paper 'says L2 cache attacks are "potentially interesting"', which is simply untrue. My paper does point out that the L2 cache provides an interesting covert channel, but at no point do I argue that the L2 cache provides a useful side channel, or that any cryptographic information can be stolen using it. I stand by the comment I made in the commit message he quoted: Nobody has publicly demonstrated a cryptographic side channel which exploits the L2 cache.

Finally, Ptacek makes one accurate comment: We need to pursue cryptographic implementations with data-independent timing. (Ok, he's not quite accurate -- what we really need is data-oblivious implementations, not merely implementations with data-independent timing -- but after everything else he has gotten wrong, let's be generous on this point.) Indeed, I have started working on such a cryptographic library; but it takes a long time to build such a library from the ground up, so as I pointed out a year ago, the only approach which could be implemented quickly was to disable Hyper-Threading.

Still Fundraising for FreeBSD security development

On April 4th, I thought that I had reached my donations target for funding my summer of FreeBSD security development, and asked people to stop sending further donations. Sadly, it seems that this assessment was premature, as it relied upon two large pledges, and it now appears that one of them will not be arriving. Fortunately, Pair Networks -- the other large donor -- has sent $6500 US, which now brings me within $2000 of my target.

If you were intending to donate when I updated my web page on April 4th to say that I had reached my target, please do so now. I know there were several people in this position, so I'm hoping I can reach my target in the next week.

As before, details about the work I plan on doing, how to donate, and a list of the donations I have received, are at my page at FreeBSD.org.

Side channel attacks: Intel vs. AMD

About 18 months ago, I discovered a side channel in the cache implementations on Intel processors with Hyper-Threading. After three months of investigation and another three months of informing and educating vendors about this problem, FreeBSD issued Security Advisory FreeBSD-SA-06:14.htt, and several other operating systems (SCO and Ubuntu quite promptly, but most Linux vendors eventually) have also taken action.

Three weeks ago, Jan Beulich discovered that the FXSAVE and FXRSTOR instructions on AMD processors behave differently from the same instructions on non-AMD processors: On AMD processors, the FOP, FIP, and FDP debugging registers are normally not saved and restored. Much like the Hyper-Threading side channel, this allows programs to spy upon each other and determine the code path and location of data being accessed, even though the value of the data handled is not revealed. This issue has now been fixed in FreeBSD, NetBSD, OpenBSD, Xen, and the Linux kernel.

While these two issues are on first glance rather similar -- they both concern information leakage between processes, can only be exploited by local users, utilize low-level processor mechanics, and are specific to a particular vendor's products -- there are some important differences. When Intel implemented Hyper-Threading in the Pentium 4, simultaneous multi-threading was not a very well understood area, and the problems of information leakage were little known; while it happens that Intel processors were vulnerable, I don't think that Intel did anything wrong in producing such processors -- they were just overtaken by developing research. In contrast, AMD took instructions which had been added by Intel as part of the SSE instruction set, claimed (via the CPU feature flags) to support SSE, and then did not fully implement said instructions in order to make their processors faster. Imagine the outcry if Intel's famous FDIV bug had been implemented deliberately because division was faster if you didn't insist upon getting the right answers!

However, a far more significant difference between these two problems is how the vendors responded. When I first reported the Hyper-Threading problem to Intel, I had trouble finding anybody willing to talk to me at all. Eventually, and thanks in large part to intervention from one of Intel's customers, I was contacted by Ernie Brickell, a respected cryptographer who now works for Intel. Sadly, this communication was still almost exclusively one-way: Intel wanted to know what FreeBSD was planning on doing about the problem, and wanted to know who the other security people were with whom I was discussing the problem, but whenever I asked what Intel was planning on doing, Ernie told me that he 'would need to get permission from a room full of people' before he could tell me anything. In fact, the only thing I learned from talking to Intel was that they didn't consider such information leakage to be a problem at all.

In contrast, AMD's response was superb. In discussions on the vendor-sec mailing list prior to public disclosure of the issue, they first promptly acknowledged the issue and stated when they would make an official response available; next, when we asked to see their response before the issue was publicly disclosed, they agreed to this; and finally, in their response they not only accepted that the problem existed, but they provided considerable background information and two different ways that they problem could be fixed -- this was particularly helpful to FreeBSD, since I wrote our patch based almost entirely upon the recommendations provided by AMD.

I have no intention of getting involved in debates about which company makes the better CPUs; as far as I'm concerned, they both make great processors. As someone working in the field of computer security, however, I can say this: There's at least one area where Intel can definitely learn some lessons from AMD.

Portupgrade errors after upgrading FreeBSD

When I upgraded some systems from FreeBSD 5.4 to FreeBSD 6.0 towards the end of 2005, one of the odd problems I encountered was portupgrade repeatedly rebuilding its database while complaining about an "Inappropriate file type or format".

Recently I've become aware that other people are encountering the same problem, so for the benefit of Google (and other search engines): For some reason there was a database format change between FreeBSD 5.x and FreeBSD 6.x. After upgrading the base system, run portupgrade -fR portupgrade; this will rebuild portupgrade using the new database format, and after this is finished, portupgrade will not keep on rebuilding its database.

I hope this post helps some people; this is a very annoying problem until you figure out how to fix it.