When security goes right
I've written a lot over the years about ways that companies have gotten security wrong; as a pedagogical technique, I find that it is very effective, since people tend to remember those stories better. Today, I'd like to tell a different story: A story about how a problem was fixed.A few weeks ago, I heard that my slow and unreliable ISP was raising their prices, and I decided to look around for a better ISP. I found one in the form of Novus, which has fibre runs to towers around the Vancouver area. I signed up; they came to the building and placed the necessary patch cable in the basement telecommunications room; and I had 50/10 Mbps internet access costing less than my previous 25/2.5 Mbps.
Then I went onto Novus' account management website and found my bandwidth utilization graph. "Cool", I thought, "they're using RRDtool. I wonder if they're generating the standard day/week/month/year graphs, or just the monthly graph." Right click, open in new tab, look at URL. "Huh, that's interesting, it's just a path to a .gif file, not a CGI invocation. Well, maybe they're rewriting usage_graphs to the directory with the graphs for whichever user is logged in. What do I get if I issue a request for the directory itself?"
Oops. Rather than a list of traffic graphs for my account, I received an Apache directory listing with thousands upon thousands of traffic graphs — all the traffic graphs ever created for their customers, in fact — and a few random clicks confirmed that I could indeed download those graphs. Being familiar with RRDtool, I knew exactly how this had happened: One of the common configurations generates traffic graphs on-demand but writes them into a directory to be fetched separately. This can save CPU time compared to always generating graphs, since RRDtool can point at an existing graph rather than regenerating it; but as in this case it isn't always a safe configuration to use.
Time to do something about this; but how? I've learned from experience that front-line technical support should only be contacted as a last resort; alas, the Shibboleet backdoor is not widely implemented. Not only do front-line technical support personnel tend to waste time because they don't understand the issues involved, they often mangle problem reports while forwarding them — more than once I've cornered an engineer and said "so, you never fixed this issue I reported a few months ago..." only to get an amazed "so that's what that ticket was about" reply.
Since I didn't want to phone the technical support line, I looked for another contact. No "security" contact on their website, unfortunately... I could have contacted them on Twitter, but I didn't really want to explain the issue in public (and wouldn't necessarily have reached anyone any more clueful)... Aha! They have valid whois data for their domain. Even better, they have separate "Administrative" and "Technical" contacts — and when companies do this, it's a good indication that the "Technical" contact will go to someone who is actually technical. (For ISPs, whois data for IP address blocks is also often useful; that would have taken me to the same person.)
He wasn't answering his phone when I first called, but ten minutes later I got through:
"Hi, I got your phone number from Novus' whois data. The RRDtool graphs for all your customers are visible. If I go to $URL I can see a list of the graphs and download any of them."After I identified myself (and explained that I was a new customer) he promised to get the problem fixed right away, and within ten minutes Apache was sending 403 (Forbidden) responses for the directory; this solved most of the problem, but the file names were predictably based on the customer account number, so knowing someone's account number could still allow an attacker to fetch their traffic graphs. A short time later they started checking Referer headers; but as I pointed out to them, sending a fake Referer header is easy, so that only helps against the most careless attackers. I suggested several options for solving this, and they chose the simplest: Use a cron job to delete all the generated graphs every minute. While this does leave a very slight exposure, it's narrow enough to be inconsequential: In order to illicitly fetch a customer traffic graph, an attacker would need to be a logged-in Novus customer (that filtering was in place from the start), know the account number of their victim, and issue the request within 60 seconds of the victim viewing their traffic graph. (A completely robust approach would be to have code which checks the graph file name against the account number of the logged-in user, but writing new code always risks introducing new bugs, so I really can't fault them for taking the 99.99% solution.)"That's not good... uhh, who are you?"
After fixing the issue, they sent me an email thanking me, and — much to my surprise — telling me that as a sign of their appreciation they were crediting a month of free internet access to my account. While I award bounties to people who report bugs (security or otherwise) in my company's product, I don't expect other companies to do likewise... especially if they don't publish any such bounties.
Obviously, Novus made a mistake in their original configuration, but in the spirit of Christmas, this post isn't about mistakes: It's about what went right. So what went right here?
First, they were using widely used open source code; if I hadn't been familiar with it, I wouldn't have noticed the problem. Bad guys will always dig deeper than unpaid good guys; so if you're going to benefit from having many eyeballs looking at what you're doing, it's much better if your bugs are shallow.
Second, I saw something and I said something. This is just good citizenship; I could have saved myself half an hour by ignoring the issue, but having seen it I felt that I had a moral duty to report it.
Third, I was able to find a way to report the problem to someone who would understand it and be able to fix it. If you're running a company and you're serious about security, put a page on your website with a GPG key and a security contact email address — this will guarantee that people who find issues will know where to send them, as well as announcing that you will take the issues seriously. If you don't want to go that far, at a minimum you should make sure that you have a valid technical whois contact for your domain: People who think to look up domain whois records are the people you want to hear from.
Fourth, I didn't ask for a "consulting fee" or any other remuneration for disclosing the vulnerability. This is something people new to the field often do, and it's a bad idea. Seriously kids, don't do this. You're not likely to get any money out of it, and you are likely to cross the line into extortion and end up in jail.
Fifth, while I had to "exploit" the vulnerability — loading other customer's traffic graphs — in order to confirm that it existed, I did so only to the minimum extent necessary. Unlike Andrew "weev" Auernheimer, upon finding that information had been accidentally made publicly available I did not proceed to download all of it; I knew that I was not intended to have that information, so I had no moral right to access it regardless of the questionable legality of doing so.
And finally, Novus immediately took the issue seriously and remained in contact with me to ensure that the issue was resolved. Many companies would file this into a bug tracker and have a sysadmin look at it months later; Novus gave it the attention it deserved, and when I wrote back to point out that their Referer-checking fix was inadequate they went back to apply a further fix.
Understanding the technical aspects of security is great, but soft skills matter too. Goodwill to all isn't just a Christmas message; it's also good security.