Customers of the free mail provider Firemail were surprised by an error message some days ago: The certificate for the web page firemail.de was invalid. Comodo had revoked it after we found out that the corresponding private key could be downloaded from the page.
Firemail wasn't the only one with this problem. We found a number of web pages where the private key was no longer private and was accidentally published on the web server. We were able to obtain private keys for 90 valid web page certificates. We obtained them by simply trying out common filenames for keys. In the case of Firemail the filename was "server.key".
File names from popular web instruction guides
Other filenames for private keys were privatekey.key, myserver.key and key.pem. But the most successful search was for filenames of the schema [domainname].key. All these filenames are used in instruction guides that can easily be found with a Google search.
Every web page certificate has a corresponding private key. As the name indicates this key needs to remain private and should never fall into the hands of unauthorized people. Otherwise the HTTPS protected connection is no longer secure. An attacker can use the private key to perform a man-in-the-middle attack and therefore read and manipulate data.
When a certificate authority learns about a compromised certificate then it should revoke it. The OCSP protocol allows checking whether a certificate has been revoked.
OCSP has some problems. In the past browsers have implemented OCSP in a soft fail mode. If a connection to an OCSP server failed then the browser still accepted the certificate as valid. As this hardly provides any security the developers of Chrome have decided to disable OCSP altogether. Better revocation checks could be implemented with a combination of OCSP stapling and the must-staple extension for certificates, but the support for OCSP stapling in common web servers is only rudimentary.
While OCSP only provides limited utility it is still the common mechanism with which certificate authorities can signal that a certificate is no longer valid.
24 hour deadline for compromised certificates
According to the Baseline Reqirements, a set of rules that certificate authorities and browser vendors have agreed upon, certificate authorities shall revoke certificates within 24 hours if a key is compromised.
We informed the corresponding certificate authorities about all the certificates and private keys we found. In most cases this simply works by e-mail. Symantec and its subsidiaries and the company Gandi use a web form for those contacts. In the case of Symantec it is necessary to solve a captcha in order to send the form.
Notable is the way Let's Encrypt handles revocations. The free certificate authoritiy automates certificate issuances with the ACME protocol. The revocation of certificates is also automated. Whoever owns the private key to a certificate can revoke it. The tool Certbot can be used for this.
Mozilla recently has investigated how reporting mechanisms for compromised certificates work. In their CA Communications that is sent regularly to certificate authorities they asked them how to best report such problems. A list of the answers is available online.
Many certificate authorities don't seem to take the 24 hour deadline very seriously. Almost all affected certificate authorities - Comodo, DigiCert, Gandi, Globalsign, Go Daddy and Symantec - took longer than 24 hours for some revocations. To be fair: One of our reports was sent on a Saturday. But the Baseline Requirements rule doesn't have an exception for weekends. Apart from Let's Encrypt only StartCom always reacted within 24 hours. But currently StartCom certificates aren't accepted by popular browsers.
Gandi and Globalsign took particularly long to revoke certificates. Globalsign needed four days to revoke a certificate reported to them. Gandi doesn't operate its certificate authority, it is operated by Comodo. After Comodo learned about this incident they immediately revoked the certificates.
New certificate with old key
A particularly odd case happened with a certificate issued by Comodo. They revoked the certificate shortly after it had been reported, but then issued shortly after that a new certificate with the same private key. After we reported this to Comodo we learned that the same problem occured with a second certificate. In both cases Comodo revoked these new certificates immediately.
This incident shows that data on web servers is an underestimated security risk. Recently we had reported that the German postal service ("Deutsche Post") and many other companies had made private customer data available as SQL dumps. Also Git repositories in web directories often are the reason for private data like passwords being made public.
Private keys for valid certificates were recently also found in other places. The software developer Koen Rouwhorst found a private key embedded in an application from Cisco. Similarly a private key was also included in a software from Spotify. Rouwhorst also found various private keys in Github repositories.