(1 votes, average: 5.00 out of 5)

Loading...

• Facebook
• Twitter
• LinkedIn
• Mail

March 13, 2019 3

Mass Revocation: Millions of certificates revoked by Apple, Google & GoDaddy

in Industry Lowdown

The DarkMatter debate is already having industry-wide ramifications

Millions of SSL/TLS certificates – among other digital certificates – are being revoked right now as a result of an operational error that caused the generation of non-compliant serial numbers.

That, in and of itself, is not terribly interesting. While we’ll discuss the generation of serial numbers in a minute, it’s not a topic that most people care, much less think about. No, what’s interesting is how we got here: how we found out that Apple, Google, GoDaddy – and likely a few other CAs – mis-issued millions of certificates.

Last week we went in-depth covering the ongoing debate over DarkMatter CA’s root application. During that article we discussed the larger ramifications of the decision and how it’s made. Specifically, that much care and consideration needed to be given to the standards and precedents being set.

A week later, the ramifications are already being felt. So, today, we’re going to talk about what happened and how the DarkMatter debate caused three of the biggest companies in the tech industry to realize they had been mis-issuing certificate themselves.

Let’s hash it out.

How do you mis-issue millions of certificates?

Let’s start with the technical side of what happened. If you don’t really want to hear about serial numbers and entropy, go ahead and skip to the next section. Otherwise, we’ll give this a cursory look to help provide some context.

Let’s start with EJBCA, which is an open source certificate authority software package. It can be used on its own to build a complete PKI infrastructure, but many publicly-trusted CAs use it as a Cryptographically Secure Pseudorandom Number Generator (CSPRNG) to generate serial numbers.

Serial numbers are covered in the CA/B Forum Baseline Requirements, section 7.1:

CAs SHALL generate non-sequential Certificate serial numbers greater than zero (0) containing at least 64 bits of output from a CSPRNG

Here’s where things get a bit dicey, and we’ll try to keep this high-level. These serial numbers must be unique, positive integers with 64 bits of entropy. That requires one of the 64 bits to be a fixed value in order to ensure that the serial number is positive. Unfortunately, the default settings for EJBCA didn’t account for that detail and was instead spitting out 63-bit serial numbers.

Why is that a big deal? Adam Caudill explained it perfectly on his blog:

When we are talking about numbers this large, it’s easy to think that 1 bit wouldn’t make much difference, but the difference between 2^64 and 2^63 is substantial – to be specific, 2^63 is off by over 9 quintillion or more specifically 9,223,372,036,854,775,808.

That represents an “unacceptable risk” to the entire ecosystem. Thus, every certificate with a 63-bit serial number that was generated using the EJBCA defaults must now be revoked and replaced with a compliant certificate.

That, by itself, is a big deal. It’s a major business disruption. But that’s not really the whole story, because how we got to this point is equally, if not more important.

Be careful where you point the finger

Let’s go back to the DarkMatter debate that played out last week on the Mozilla root forum. As we discussed last week, while the answer to the question of whether or not Dark Matter Group should have its root included in the various root programs – which would give it the ability to issue trusted digital certificates – seems obvious on its face, the decision will have a great many ramifications.

That’s already proving true.

In an effort to avoid making a decision that looked purely political, there were several parties that went looking for a technical reason to deny DarkMatter’s root application. One such reason presented itself in the form of 235 certificates that had been issued with non-compliant serial numbers.

As Sophos’ Corey Bonnell wrote:

I would like to bolster my previous assertion that the serial number generation scheme used in the DarkMatter certificate hierarchy likely does not meet the requirements set forth in the Baseline Requirements, section 7.1…

This analysis has revealed that all 235 unique certificates have a serial number of 8 octets (64 bits) and a big-endian most significant bit set to 0. Given that the probability of all 64 bits being output from a CSPRNG with a most significant bit value of 0 for all 235 such certificates is 1 in 2^235, it is extremely likely that these certificates do not contain the minimum number of bits (64) output from a CSPRNG and are therefore mis-issued under the Baseline Requirements.

In summation, he’s accusing DarkMatter of mis-issuing certificates with 63-bit serial numbers and providing this as grounds for denying the root application.

In response, DarkMatter’s Scott Rea outlined its method for generating serial numbers:

DarkMatter uses an EJBCA platform with the requisite setting for 64-bit random serial numbers and our source of entropy is a FIPS140 certified HSM, so I too was surprised by the findings you reported. However, during our investigation of this potential issue, we have thus far discovered that the platform appears to be compliant with the requisite standard, and the anomaly you are highlighting is potentially due just to the integer representation you are using in your calculations.

RFC5280 (section 4.1.2.2) defines serialNumber to be a positive INTEGER, and X.690 defines INTEGER to consist of one or more octets and (specifically section 8.3.3) says the octets shall be a two’s complement binary number equal to the integer value. Using the two’s complementary representation means that the output of the octet conversion is a signed integer, and it could be positive or negative – the range of integers from 64-bit numbers being from –(2^63) to [+ (2^63)-1]. But since the RFC requires only positive integers, the 64-bits of output from the CSPRNG function must eventuate only in positive numbers, and negative numbers cannot be used. In two’s complement representation, the leading bit determines whether the number is positive or negative – for positive numbers, the leading bit will always be zero (if it’s a 1, then that represents a negative number which RFC5280 prohibits).

So our findings are that the platform is indeed using 64-bit output from an appropriate CSPRNG for generating serialNumbers, and that the leading zero is exactly what is required to indicate that it is a positive number in two’s complement representation of the INTEGER, which is the requirement under RFC5280. Therefore our findings indicate that the serial number generation scheme used by DarkMatter in it’s certificate hierarchy does meet the requirements set forth in the Baseline Requirements, section 7.1.

This harkens back to what we said earlier about one of the bits needing to be a fixed value to ensure that the entire serial number was a positive integer. This is also the moment that everyone else realized that the problem wasn’t DarkMatter’s it was EJBCA’s default configuration. One that many other CAs were also using.

This is what we call an “oh $#!%” moment.

So far Apple, Google and GoDaddy have admitted to mis-issuing certificates. GoDaddy initially said it was 1.8 million certificates, though they’ve since revised that number down. Apple took credit for 878,000, though about 300,000 of those had already expired or been revoked as of last week. Google, meanwhile, estimated that it had issued over 100,000, though only about 7,100 of them were still valid.

There’s also a possibility this could affect other CAs, too. Though as of right now none have disclosed any mis-issuances stemming from the EJBCA misconfiguration.

We can, at least in part, thank DarkMatter for pointing this out. What we probably shouldn’t do – as we discussed in last week’s article on this subject – is hold on to this as a reason to deny its root application.

Not to say our article was particularly prescient, but:

And what if, by applying that standard to other established CAs, it forces us to re-evaluate their participation, too? Or are we simply condoning applying different standards to different organizations based on their location and other partnerships?

What risk does this pose?

A business risk, sure. A security risk? No. The 64 bits of entropy required for BR compliant serial numbers was a requirement that was added in 2016 in response to an SSL spoofing proof-of-concept that was able to produce collisions (two matching serial numbers) using the MD5 hashing algorithm that was, at the time, generating certificates. Nowadays certificates are generated using SHA-2 (SHA256 or higher), so MD5’s vulnerabilities are no longer an issue.

So, the need for 64 bits of entropy is really more of a safeguard against attacks that could hypothetically be conceived of in the future.

“This is a big deal for CAs and their customers,” Caudill told Ars Technica. “The impact of replacing large numbers of certificates is substantial. From a threat perspective though, this isn’t exploitable. It would require a major breakthrough in cryptography, and even then, 63 bits of entropy provides a huge safety margin. This is a problem because of impact to people and companies; hackers aren’t going to start forging certificates because of this.”

As Caudill alluded to, the bigger issue is the business disruption.

This will be the second major disruption the digital certificate industry has faced in the past year and a half – the first being the Symantec distrust.

That’s kind of a black eye for the whole industry. Most businesses and consumers don’t give a lot of thought to digital certificates until they expire or break. And as we’ve discussed lately, for many businesses certificates and certificate management are more a matter of compliance than security. And businesses are loathe to risk non-compliance.

Suffice it to say there are going to be plenty of very angry organizations when they find out the certificates they use to help ensure compliance are being revoked for being non-compliant.

That sounds like a really fun conversation.

Distrust Google?

This header is facetious. But, one of the other offshoots of this whole situation are some of the similarities it bears to the aforementioned Symantec distrust. Only this time it’s Google that’s mis-issued a bunch of certificates that pose no real-world threat. If you’ll recall, Google practically forced Symantec out of the industry after discovering it had mis-issued 33 test certificates in 2016. Symantec claimed that it was a non-issue because they posed no real threat. Google claimed it represented an unacceptable breach of trust.

Obviously there are some differences, but on some level a mis-issuance is a mis-issuance. And the fact that Google didn’t discover this on its own doesn’t do it any favors.

The point, obviously, isn’t to vilify Google – just to, once again, point out the subjectivity of a lot of these decisions. CAs are going to mis-issue, it happens to literally every. Single. One. The more important part is the response.

GoDaddy is giving itself 30 days to work through its revocations while Apple and Google are operating on much shorter timelines.

That makes sense, too. Apple and Google issue their certificates internally, within their own organizations. They’re not worrying about a bunch of pissed off customers. GoDaddy is. But, giving itself 30 days also may get GoDaddy into even more trouble from a compliance standpoint. The Baseline Requirements mandate the timely revocation of all non-compliant certificates, per section 4.9.1.1:

The CA SHOULD revoke a certificate within 24 hours and MUST revoke a Certificate within 5 days if one or more of the following occurs: …

7. The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement; …

I’d say that obviously the root programs aren’t going to distrust GoDaddy, but three years ago I’d have probably said the same thing about Symantec so I’m just going to leave it at that.

In the meantime, many websites and organizations are going to be scrambling to replace their SSL/TLS certificates – in addition to many code & email signing certificates – with compliant ones.

Thanks, DarkMatter.

As always, leave any comments or questions below…

• #Apple
• #DarkMatter
• #GoDaddy
• #Google
• #Mis-Issuance