Sectigo root CA expiring, may not be handled well by slightly older linux versions

July 6, 2021 . 2 MIN READ

It looks like some older distributions of linux (we’ve seen issues with Deb 9 and earlier and Ubuntu 16.04 and earlier) will not properly expire/ignore this root cert unless it is actually removed. Ubuntu 18 and up, as well as Deb 10 are unaffected. To verify this, create a host based on one of these distributions and roll the time forward Jun 1st or so. Running curl against certain domains will fail with a certificate expired error. For example: curl https://crt.sh.

This happens even if ca-certificates has been updated. This obviously does not affect all domains, but we have a number of critical internal and external endpoints that use comodo/sectigo certs that had the issue.

EDIT: As someone else has pointed out, this is almost certainly an openssl 1.0.2 bug. Unfortunately system upgrades don’t help the situation (unless you upgrade the actual distribution). We have noticed that programs that don’t depend on openssl (for example, compiled go programs, python) won’t exhibit the problem. Ruby, on the other hand, will have the same issues as curl.

EDIT: I’ll go ahead and post the fixes for debian-based systems. Note that reissuing your own certs can fix the issues with those certs, as your provider should no longer include the expired root in the chain. But for domains you don’t control, you will still need to be sure you have the later intermediate certs and have removed the expired cert. For Deb 9/Ubuntu 16:

Edit /etc/ca-certificates.conf — remove AddTrust_External_Root.crt. You can automate this with sed, if need be.
apt update && apt install ca-certificates # Be sure you have the latest bundle
For good measure update-ca-certificates -f -v.

That should remove all the links to AddTrust_ExternalRoot in /etc/ssl/certs. Test with curl against an affected domain

For older distributions (deb 8, ubuntu 14), you may want to do a full systems upgrade first, THEN actually rm /usr/share/ca-certificates/mozilla/AddTrust_External_Root.crt. You should at least be sure you have the latest openssl you can get and that you have the latest ca-certificates installed.

FINAL EDIT: Our only lingering issue, which is easy to work around is that git has issues with the reissued cert for our private repo. On those hosts we have to tell git to not verify the cert. This was only on ubuntu 14