Table of Contents
Introduction
Fixing certificate problems can be really hard, but not if you have the right tools. Let’s do a quick exercise for HTTPS:
Port 443
openssl s_client -connect mail.example.com:443 -servername mail.example.com | openssl x509 -noout -dates Warning: Reading certificate from stdin since no -in or -new option is given depth=2 C=US, O=Internet Security Research Group, CN=ISRG Root X1 verify return:1 depth=1 C=US, O=Let's Encrypt, CN=R3 verify return:1 depth=0 CN=mail.wrongdomain.com verify return:1 notBefore=Feb 1 13:16:14 2024 GMT notAfter=May 1 13:16:13 2024 GMT
Above is problematic:
- CN is wrong domain
- notAfter is today. It should have renewed already!
443 – Digger Deeper
Let’s see what’s going on with Let’s Encrypt using certbot certificates:
# certbot certificates Saving debug log to /var/log/letsencrypt/letsencrypt.log - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Found the following certs: Certificate Name: server-anchor.example.com Domains: wrongdomain1.com anchor mail.rightdomain.com wrongdomain2.com webmail.rightdomain.com Expiry Date: 2024-05-01 13:16:13+00:00 (VALID: 10 hour(s)) Certificate Path: /etc/letsencrypt/live/server.example.com/fullchain.pem Private Key Path: /etc/letsencrypt/live/server.example.com/privkey.pem - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
At least certbot is showing us similar information.
How can we remove these wrong domains and try to renew? Here was one command:
certbot delete --cert-name example.com
Let’s do that now:
certbot delete --cert-name mail.wrongdomain Saving debug log to /var/log/letsencrypt/letsencrypt.log No certificate found with name mail.wrongdomain (expected /etc/letsencrypt/renewal/mail.wrongdomain.conf).
Well that didn’t work. Mmmm.
It turns out more complex syntax is needed when removing domains from certbot. You can’t, you have to re-issue the certificate with all the names:
certbot certonly --cert-name anchordomain -d anchordomain -d mail.example.com -d webmail.example.com
I could choose the Apache options because the server is using it.
443 Checking CRONs for renewal of Let’s Encrypt
Often when Letsencrypt dates are wrong, it’s because there is still a lingering domain attached to the service that shouldn’t be here. By manually running the command below, one can see:
certbot renew
Once I ran that, I noticed an old DNS record was never updated. I updated the DNS, re-ran the command, and the restart Apache. This solved the problem.
443 AH00526 Syntax error on line 1 of …
You may experience this problem when manually renewing certificates to avoid the old ones:
Command:
certbot certonly --cert-name host.last-domain-name-on-server -d host.last-domain-name-on-server
Output
AH00526: Syntax error on line 1 of /etc/apache2/vhosts.d/le_http_01_challenge_pre.conf: Invalid command 'RewriteEngine', perhaps misspelled or defined by a module not included in the server configuration
Let’s Encrypt is great open source software but ridden with bugs. Also the combination of Apache and it’s 1000 places to include and place stuff makes troubleshooting an elitist job. Basically Let’s Encrypt is making up that error because the file le_http_01_challenge_pre.conf doesn’t even exist. You can try the advice below or just start crying. Let’s Encrypt isn’t easy to fix and as of this writing has 179 open issues on GitHub. Sjame.
If the above doesn’t work or is complex to implement, good luck! Let us know how frustrated you are in the comments.
Port 993
openssl s_client -CApath /etc/ssl/certs/ -connect mail.example.com:993 -brief
Port 993 – Digging Deeper
If you run the above command, you should get an IMAP prompt. If you don’t get the IMAP prompt and some kind of hang or delay, it may mean something is wrong.
If you have more than one certificate / tenant on your server, you might find certain email clients (e.g. Google’s email client), starts moaning about the certificate. However, other email clients like Outlook still work. On the Google email client side, you’ll find something like the errors below.
On Kopano this is complicated to troubleshoot.
Check these directories and what they refer to:
/etc/kopano/gateway/
That directory has symbolic links.
Then check this directory:
/etc/letsencrypt/live/servername
You may well find the servername directory is linking to older archived Let’s Encrypt files.
The errors referred to is this transcript using OpenSSL is this:
# openssl s_client -CApath /etc/ssl/certs/ -connect mail.the-domain-i-want.com:993 -brief depth=0 CN = the-domain-i-am-getting-instead.com verify error:num=20:unable to get local issuer certificate depth=0 CN = the-domain-i-am-getting-instead.com verify error:num=21:unable to verify the first certificate CONNECTION ESTABLISHED Protocol version: TLSv1.3 Ciphersuite: TLS_AES_256_GCM_SHA384 Peer certificate: CN = the-domain-i-am-getting-instead.com Hash used: SHA256 Signature type: RSA-PSS Verification error: unable to verify the first certificate Server Temp Key: X25519, 253 bits * OK [CAPABILITY IMAP4rev1 LITERAL+ AUTH=PLAIN] IMAP gateway ready * BYE Connection closed because of timeout
Port 465
openssl s_client -connect mail.example.co.za:465 -servername mail.example.co.za -showcerts | openssl x509 -noout -text | egrep -i "depth|subject:|alternative name|DNS:"
Port 465 – Digging Deeper
Port 465 checking spits out a lot of information so it’s important to focus your eyes on the correct bits. Here is an example:
openssl s_client -connect mail.example-to-test.com:465 -servername mail.example-to-test.com -showcerts | openssl x509 -noout -text | egrep -i "depth|subject:|alternative name|DNS:" depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1 verify return:1 depth=1 C = US, O = Let's Encrypt, CN = R3 verify return:1 depth=0 CN = mail.other-primary-domain.com verify return:1 Subject: CN = mail.other-primary-domain.com X509v3 Subject Alternative Name: DNS:host.the-computer-name.com, DNS:mail.example.com, DNS:mail.dns-points-wrong.com, DNS:mail.example2.com, DNS:mail.example-to-test.com
In the example above, the discrepancy is that although the example want to test is there, there is also an alternative name where the DNS wasn’t correct. For a start, that has to be corrected. Refer to other parts of this documentation if that domain doesn’t exist anymore and has to be removed. &TL;DR: You can’t remove stuff, you have to reissue the certificate without the removed domain.