This appeared as a cover story in Full Circle Magazine Issue #86, June 2014
In the last few weeks (as I write this in late April 2014) two events have combined to deliver a powerful lesson on the security of Open Source software. But it is important to know exactly what the right lesson is. I have seen reports that Heartbleed was a proof of something fundamentally wrong with the Open Source model, as denying the accuracy of Eric Raymond’s famous saying “With many eyeballs all bugs are shallow.” The Heartbleed bug was in a significant number of systems (actually about one-sixth of Internet sites, as far as I can tell from an analysis of how many sites use OpenSSL, and what percent of those use the versions of the software that are affected). There was a bit of hyperbole in how bad it was, but it is no doubt still pretty bad. But how did that happen?
I will refer everyone to an excellent article that has all of the details. It is called How Did the Heartbleed OpenSSL Bug Happen?, and I recommend looking at it. It is short and to the point. Basically, there was a request to have an extension to OpenSSL to provide something called a TLS Heartbeat extension. This is a perfectly reasonable thing to do, and is covered in RFC 6520, Transport Layer Security (TLS) and Datagram Transport Layer Security (DTLS) Heartbeat Extension. As the RFC makes clear, the purpose is to provide a “keep alive” functionality without requiring a renegotiation. OpenSSL was just trying to be compliant in adding a capability that the Internet Engineering Task Force had decided should be provided. But how does the OpenSSL project handle this?
The first thing we notice is that OpenSSL has a core team of just 11 people, most of them volunteers, and only one full-time person devoted to the project. Generally, they get about $2000 a year in donations, and make some money from support contracts. In other words, they are stretched tight. A volunteer in Germany, Dr. Robin Seggelmann, wrote the code to implement RFC and submitted it for review. Dr. Seggelmann is a respected academic and computer science researcher, and there is no possible way to suggest either malice or stupidity here. He did not actually have commit rights to OpenSSL, so he submitted the code to the project members who do have those rights, and they reviewed it. Seeing nothing wrong with the code, and verifying that it did what it said it would do (i.e. implement a Heartbeat) the code was put into production in early 2012.
The problem was discovered by Google researchers and by a Finnish company, Codenomicon, at about the same time, and it was made public in April 2014. There is some suggestion that there was talk from one of the Google people that may have pointed the Codenomicon in the right direction, but perhaps it was simply independent discovery. These things happen. But as Steve Marquess of OpenSSL Foundation said ““The mystery is not that a few overworked volunteers missed the bug; the mystery is why it hasn’t happened more often.”
The other event I want to talk about is the TrueCrypt audit, which released preliminary results recently. As you may recall, in the wake of the Edward Snowden revelations, there was general anxiety about the security of encryption, and people wanted to know if their encryption had been weakened or a backdoor inserted by the NSA, GCHQ, or other government agencies. In the case of TrueCrypt, you again have an open-source project, with the wrinkle that the developers were deliberately anonymous (and based in Eastern Europe). Pre-Snowden that might not have aroused too much speculation, but post-Snowden people wanted answers. The TrueCrypt Foundation did the right thing. They raised money (I contributed to the crowd-funded campaign) and enlisted Dr. Matthew Green, a highly-respected cryptography expert who teaches at Johns Hopkins University, to put together a team to perform an audit of the code. This is a lengthy and difficult task, but the first phase has been completed, and while there are criticisms of certain sloppiness errors, but no sign of any deliberate errors. You can read a good report on this at novainfosec.com, and that article has a link to actual report if you want to read it.. This first phase looked the bootloader and the Windows kernel driver implementations. There is a second phase planned, to go into the cryptography itself, which will use a completely different team of researchers.
So what were the results? Well, TrueCrypt is not perfect, but to expect that would be unrealistic in any case. The audit team did find a certain amount of sloppiness, which probably derives from the fact that the project was done by volunteers and grew organically. But the audit team found no evidence in Phase 1 that there were any deliberate problems or “back doors” in the code. This is good news since this is one of the major open source programs to offer serious encryption. If you want to encrypt a directory, a drive, or an entire computer, this will do the job for you, and so far there is no evidence that the encryption is compromised (though there are things they can do to tighten up the code). And of course we should wait for the Phase 2 audit before giving them a clean bill of health.
Lessons Learned
- These programs are important to the Internet, so where was the support? This gets at a fundamental problem of companies treating Open Source like it is a free lunch. It is not, for as you should know There Ain’t No Such Thing As A Free Lunch (TANSTAAFL). Open source is really just a different model for developing and supporting software, one that relies on participation by all of the interested parties. If all of these companies were relying on OpenSSL, for instance, where was their participation? After the fact it looks like many of them woke up. The Linux Foundation has put together a consortium of major companies, To quote from an Ars Technica article on the subject “Amazon Web Services, Cisco, Dell, Facebook, Fujitsu, Google, IBM, Intel, Microsoft, NetApp, Qualcomm, Rackspace, and VMware have all pledged to commit at least $100,000 a year for at least three years to the “Core Infrastructure Initiative,” Linux Foundation Executive Director Jim Zemlin told Ars.” This initiative will be aimed at more than just OpenSSL, but that is good. It means that these companies are taking seriously their responsibility to support the code they rely on. This is in great contrast to the somewhat ridiculous move by Theo de Raadt to create a fork called LibreSSL. This sounds more like ego than a constructive move. I would stick with OpenSSL and give LibreSSL a pass until such time as they can show a long track record of success. A good general rule in security is that new code is more dangerous than code that has been around for a long time.
- Security is hard, and is a different skill set than most development. Dr. Seggelmann is a smart guy who was trying to implement a requirement in an RFC. His code did in fact do that. It was reviewed by someone else on the OpenSSL team, and they did not see any problems with it and put it into production. It sat there for two years before someone noticed a potential problem. The reason a number of smart people missed this is that it takes a different skill set to do security. In hindsight it is easy to say they should have brought in a specialist, and I think the Core Infrastructure Initiative will help address this.
- Bugs are not shallow if the eyeballs are not there. Both TreuCrypt and OpenSSL had small groups of developers with limited resources. Everyone else just assumed that the code was fine, and never tried to look at it. And given that Security requires a specialized skill set, just adding eyeballs is not enough. they need to be the kind of eyeballs. A question this raises in my mind is about the governance of critical open source projects. Perhaps we need a little more structure to the process to avoid these kinds of problems.
- Fixing this requires money, among other things. One of the key take-aways regarding the OpenSSL project is that they were on what i called a “shoestring” budget, where on average they received $2,000 per year in donations. Contrast this with the cost of the TrueCrypt audit, where they appear to have raised about $60,000 so far, and I doubt that that is any too much. They put together a team of professionals who understand the work, and that can go through $60,000 in no time. I always tell people they need to support Free Software, and that includes financial support. If you are only interested in what you can get for free, you will get these kinds of results because the resources will not be there.
- The advantage of Open Source software is not that it is bug-free. No software of any kind is bug-free. We make a grave mistake to think so. And it probably is not even correct to think that Open Source has fewer bugs. As we have seen, the weakness of the “many eyeballs – shallow bugs” theory is that for many Open Source projects, even critical ones, there simply are not that many eyeballs, and often the ones that are there may not be the ones we need to detect subtle problems such as security issues. That does not imply the opposite, however. The idea that Open Source has issues does not mean that Proprietary Software does any better, as the recent IE bug illustrates (as I write this, people are being advised to stop using IE altogether because of a fundamental security issue. Look up “Operation Clandestine Fox” if you want more details.) The superiority of Open Source is principally that issues generally are addressed quickly. Patches for the Heartbleed bug started to roll out within hours of the disclosure. Patches for the IE bug will at best show up in the next round of Microsoft patches, which could mean waiting a month. Furthermore, with Open Source the whole code is on display, so the quality of our information is much better. With proprietary software the code is never available, the information about the bug tends to be sketchy at best, and in some cases companies will try to keep any information from going out because it could have an adverse affect on their bottom line.
- In the case of OpenSSL, Simon Phipps has offered a very interesting article, based on work of David Wheeler, that points to the license as a source of problems. OpenSSL used a license all of their own which was copy-left, but incompatible with the GPL. And this creates a disincentive for anyone to get involved. He quote Eben Moglen as saying that the open source license acts as the “constitution of the community” which governs how everyone participates. By having a license that no one else uses, they had the effect of putting in ground rules for participation that no one else understood. The lesson here is that you should not try to re-invent the wheel. There are plenty of good, well-understood open source licenses out there, and you should use one of them so that the largest number of contributors will be involved. This is one of the reasons that Phipps, Executive Director of OSI, strongly discourages any new license applications. It just isn’t a good idea, and people need to stop this needless proliferation.
Save as PDF