TLS & Certificates Ops - Street-Level Ops

What experienced operators know from years of certificate fires, chain debugging, and 3 AM expiry alerts. These are the commands you run when TLS is broken and the pager is screaming.

Checking Certificate Expiry

The most common TLS emergency is an expired certificate. Check it before it becomes an outage.

# Check expiry of a remote server's cert
echo | openssl s_client -connect app.example.com:443 -servername app.example.com 2>/dev/null | \
  openssl x509 -noout -dates
# notBefore=Jan  1 00:00:00 2026 GMT
# notAfter=Apr  1 00:00:00 2026 GMT

# One-liner: days until expiry
echo | openssl s_client -connect app.example.com:443 -servername app.example.com 2>/dev/null | \
  openssl x509 -noout -enddate | \
  awk -F= '{print $2}' | \
  xargs -I {} sh -c 'echo $(( ( $(date -d "{}" +%s) - $(date +%s) ) / 86400 )) days remaining'

# Check with curl (verbose shows cert info)
curl -vI https://app.example.com 2>&1 | grep -E "expire|subject|issuer|SSL"

# Check with nmap
nmap --script ssl-cert -p 443 app.example.com

# Batch check multiple hosts
for host in app.example.com api.example.com www.example.com; do
  expiry=$(echo | openssl s_client -connect $host:443 -servername $host 2>/dev/null | \
    openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
  echo "$host: $expiry"
done

# Check a local certificate file
openssl x509 -in /etc/ssl/certs/app.crt -noout -dates -subject -issuer

# Check cert in a Kubernetes Secret
kubectl get secret app-tls-secret -n production -o jsonpath='{.data.tls\.crt}' | \
  base64 -d | openssl x509 -noout -dates -subject -ext subjectAltName

Debugging TLS Handshake Failures

When connections fail with TLS errors, openssl s_client is your primary tool.

# Full handshake debug
openssl s_client -connect app.example.com:443 -servername app.example.com

# Key things to look for in the output:
# 1. "Verify return code: 0 (ok)" — chain validation passed
# 2. "Certificate chain" — shows leaf + intermediates sent by server
# 3. "Protocol" and "Cipher" — negotiated TLS version and cipher suite
# 4. "Server certificate" — the leaf cert details

# Test with a specific TLS version
openssl s_client -connect app.example.com:443 -tls1_2
openssl s_client -connect app.example.com:443 -tls1_3

# Show the full certificate chain
openssl s_client -connect app.example.com:443 -servername app.example.com -showcerts </dev/null 2>/dev/null

# Test with a specific CA bundle (useful for internal CAs)
openssl s_client -connect app.example.com:443 -servername app.example.com \
  -CAfile /path/to/ca-bundle.pem

# Test client certificate (mTLS)
openssl s_client -connect api.example.com:443 \
  -cert client.crt -key client.key -CAfile ca.pem

# Check which ciphers the server supports
nmap --script ssl-enum-ciphers -p 443 app.example.com

Common Handshake Failure Causes

Error	Meaning	Action
verify error:num=20	Unable to get local issuer cert	Server is not sending intermediate cert
verify error:num=10	Certificate has expired	Renew the certificate
verify error:num=21	Unable to verify first cert	CA not in trust store
verify error:num=62	Hostname mismatch	Cert SAN doesn't match the hostname
no peer certificate available	Server didn't send a cert	TLS not configured on server
handshake failure	Protocol/cipher mismatch	Check TLS version and cipher support
self signed certificate	Self-signed cert in chain	Replace with CA-signed cert or add to trust store

cert-manager Troubleshooting in Kubernetes

# Check Certificate status
kubectl get certificate -A
# NAMESPACE    NAME       READY   SECRET         AGE
# production   app-tls    False   app-tls-secret  10m   <-- NOT READY

# Get details on why it's not ready
kubectl describe certificate app-tls -n production
# Look at Events and Conditions sections

# Check the CertificateRequest
kubectl get certificaterequest -n production
kubectl describe certificaterequest app-tls-xyz -n production

# For ACME certificates, check challenges
kubectl get challenges -A
kubectl describe challenge app-tls-xyz-123 -n production
# Look for: State, Reason, Presented

# For ACME orders
kubectl get orders -A
kubectl describe order app-tls-xyz-456 -n production

# cert-manager controller logs
kubectl logs -n cert-manager deploy/cert-manager --tail=100 | grep -i error
kubectl logs -n cert-manager deploy/cert-manager --tail=200 | grep app-tls

# Check the Issuer/ClusterIssuer
kubectl describe clusterissuer letsencrypt-prod
# Look for: Ready condition, ACME account registration status

# Force renewal of a certificate
kubectl cert-manager renew app-tls -n production
# Requires cert-manager kubectl plugin: kubectl krew install cert-manager

# Nuclear option: delete and recreate
kubectl delete certificate app-tls -n production
kubectl apply -f certificate.yaml
# cert-manager will re-request from scratch

Common cert-manager Failures

Problem: HTTP-01 challenge pending forever
Cause: Port 80 not reachable from the internet (firewall, LB config, ingress class mismatch)
Fix: Check ingress controller serves /.well-known/acme-challenge/ on port 80
     kubectl get ingress -A | grep acme

Problem: DNS-01 challenge failing
Cause: IAM permissions wrong, hosted zone ID wrong, DNS propagation timeout
Fix: Check cert-manager logs for Route53/CloudDNS errors
     Verify IAM role: aws sts get-caller-identity
     Check if TXT record was created: dig _acme-challenge.app.example.com TXT

Problem: Certificate issued but Secret not created
Cause: RBAC preventing cert-manager from creating Secrets in target namespace
Fix: Check cert-manager RBAC: kubectl auth can-i create secrets -n production --as system:serviceaccount:cert-manager:cert-manager

Let's Encrypt Rate Limits and Debugging

# Check current rate limit status (no direct API — use CT logs)
curl -s "https://crt.sh/?q=%.example.com&output=json" | \
  jq '[.[] | select(.not_before > "2026-03-12")] | length'
# Count certs issued in the past week

# Common rate limit errors in cert-manager logs:
# "too many certificates already issued for exact set of domains"
# "too many new orders recently"
# "too many failed authorizations recently"

# Solutions:
# 1. Test with staging first (different rate limits)
# 2. Use fewer, broader certificates (wildcard instead of per-subdomain)
# 3. Wait for the rate limit window to reset (usually 1 week)
# 4. Check if a previous cert is still valid and reusable

Certificate Chain Verification

Debug clue: If openssl verify succeeds locally but clients still reject the chain, the server is probably not sending the intermediate certificate. Browsers may cache intermediates from previous visits (masking the problem), but API clients and curl never cache them. Always test with curl --cacert /dev/null https://... to simulate a cold client.

# Verify a complete chain
openssl verify -CAfile root-ca.pem -untrusted intermediate.pem leaf.pem
# leaf.pem: OK

# Verify chain from a server
openssl s_client -connect app.example.com:443 -servername app.example.com \
  -showcerts </dev/null 2>/dev/null | \
  awk '/BEGIN CERT/,/END CERT/' > /tmp/chain.pem
# Then inspect each cert in the chain:
csplit -z /tmp/chain.pem '/-----BEGIN CERTIFICATE-----/' '{*}'
for f in xx*; do
  echo "=== $f ==="
  openssl x509 -in $f -noout -subject -issuer -dates
done

# Check if the server sends the intermediate
openssl s_client -connect app.example.com:443 -servername app.example.com </dev/null 2>/dev/null | \
  grep -c "BEGIN CERTIFICATE"
# 1 = leaf only (MISSING INTERMEDIATE)
# 2 = leaf + intermediate (correct)
# 3 = leaf + intermediate + root (root is unnecessary but not harmful)

Converting Between Formats

# PEM to PKCS#12 (for importing into Windows/Java)
openssl pkcs12 -export -in cert.pem -inkey key.pem -certfile chain.pem \
  -out bundle.p12 -name "app.example.com" -passout pass:changeit

# PKCS#12 to PEM (extracting from Windows/Java export)
openssl pkcs12 -in bundle.p12 -out all.pem -nodes -passin pass:changeit
# Separate cert and key:
openssl pkcs12 -in bundle.p12 -nokeys -out cert.pem -passin pass:changeit
openssl pkcs12 -in bundle.p12 -nocerts -nodes -out key.pem -passin pass:changeit

# DER to PEM
openssl x509 -in cert.der -inform DER -out cert.pem -outform PEM

# PEM to DER
openssl x509 -in cert.pem -outform DER -out cert.der

# Create a full chain PEM (leaf + intermediate)
cat leaf.pem intermediate.pem > fullchain.pem

# JKS operations (Java KeyStore)
# Import PEM cert into JKS
keytool -import -trustcacerts -alias app -file cert.pem -keystore keystore.jks
# List JKS contents
keytool -list -v -keystore keystore.jks
# Convert JKS to PKCS12
keytool -importkeystore -srckeystore keystore.jks -destkeystore store.p12 -deststoretype PKCS12

Generating Self-Signed Certs for Testing

# Quick self-signed cert with SAN (modern clients require SAN)
openssl req -x509 -newkey rsa:2048 -nodes \
  -keyout test.key -out test.crt -days 365 \
  -subj "/CN=test.local" \
  -addext "subjectAltName=DNS:test.local,DNS:localhost,IP:127.0.0.1"

# Self-signed CA + signed leaf (for testing cert chains)
# 1. Create CA
openssl req -x509 -newkey rsa:2048 -nodes \
  -keyout ca.key -out ca.crt -days 3650 \
  -subj "/CN=Test CA"

# 2. Create CSR for leaf
openssl req -new -newkey rsa:2048 -nodes \
  -keyout leaf.key -out leaf.csr \
  -subj "/CN=app.test.local" \
  -addext "subjectAltName=DNS:app.test.local,DNS:localhost"

# 3. Sign with CA
openssl x509 -req -in leaf.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out leaf.crt -days 365 \
  -copy_extensions copyall

# 4. Verify
openssl verify -CAfile ca.crt leaf.crt

Automating Certificate Renewal

certbot (standalone)

# Install certbot
apt install certbot  # Debian/Ubuntu
# or
snap install --classic certbot

# Get a cert (HTTP-01, standalone mode)
certbot certonly --standalone -d app.example.com --agree-tos --email ops@example.com

# Get a cert (DNS-01, Route 53)
certbot certonly --dns-route53 -d "*.example.com" -d example.com

# Renew all certs (run via cron or systemd timer)
certbot renew

# Certs are stored at:
# /etc/letsencrypt/live/app.example.com/fullchain.pem
# /etc/letsencrypt/live/app.example.com/privkey.pem

# Test renewal without actually renewing
certbot renew --dry-run

# Post-renewal hook (reload nginx after renewal)
certbot renew --deploy-hook "systemctl reload nginx"

# Systemd timer (usually auto-installed)
systemctl list-timers | grep certbot

Debugging mTLS Issues

# Test if server requires client cert
openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null 2>&1 | \
  grep "Acceptable client certificate CA names"
# If this shows CA names, the server is requesting mTLS

# Test with client cert
openssl s_client -connect api.example.com:443 \
  -cert client.crt -key client.key -CAfile ca.pem

# curl with mTLS
curl -v --cert client.crt --key client.key --cacert ca.pem \
  https://api.example.com/resource
# Look for "SSL certificate verify ok" in verbose output

# Common mTLS failures:
# "certificate required" — client didn't present a cert
# "bad certificate" — client cert not signed by expected CA
# "certificate unknown" — server doesn't trust client's CA
# "handshake failure" — key/cert mismatch or wrong key usage

# Check client cert has correct key usage
openssl x509 -in client.crt -noout -text | grep -A 2 "Key Usage"
# Must include "TLS Web Client Authentication" in Extended Key Usage

Checking Certificate Transparency Logs

# Query crt.sh for all certs issued for your domain
curl -s "https://crt.sh/?q=%.example.com&output=json" | \
  jq '.[] | {id, common_name: .common_name, not_before, not_after, issuer_name}' | head -50

# Find unexpected certificates (potential compromise or misuse)
curl -s "https://crt.sh/?q=%.example.com&output=json" | \
  jq '.[] | select(.issuer_name | test("Let.s Encrypt") | not) | {common_name, issuer_name, not_before}'

# Monitor with certspotter (CLI tool)
# https://github.com/SSLMate/certspotter
certspotter -watchlist example.com

SNI Debugging

# Many servers on one IP rely on SNI to pick the right cert
# Test what cert you get with SNI:
openssl s_client -connect 203.0.113.50:443 -servername app.example.com </dev/null 2>/dev/null | \
  openssl x509 -noout -subject
# subject=CN = app.example.com

# Test what cert you get WITHOUT SNI:
openssl s_client -connect 203.0.113.50:443 </dev/null 2>/dev/null | \
  openssl x509 -noout -subject
# subject=CN = default.example.com  (or may fail entirely)

# If these differ, the server relies on SNI
# If a client doesn't send SNI, it gets the wrong cert

# Check if a specific client sends SNI:
# Capture TLS handshake
tcpdump -i eth0 -w handshake.pcap port 443 -c 20
# Open in Wireshark, filter: ssl.handshake.extensions_server_name

Emergency: Certificate Expired Right Now

# 1. Confirm the expiry
echo | openssl s_client -connect app.example.com:443 -servername app.example.com 2>/dev/null | \
  openssl x509 -noout -dates

# 2. If using cert-manager, force renewal
kubectl cert-manager renew app-tls -n production
# Wait 30-60 seconds, then check:
kubectl get certificate app-tls -n production

# 3. If cert-manager renewal fails, issue a new cert manually
certbot certonly --standalone -d app.example.com --force-renewal

# 4. Update the Kubernetes Secret manually if needed
kubectl create secret tls app-tls-secret \
  --cert=/etc/letsencrypt/live/app.example.com/fullchain.pem \
  --key=/etc/letsencrypt/live/app.example.com/privkey.pem \
  -n production --dry-run=client -o yaml | kubectl apply -f -

# 5. Restart workloads to pick up the new cert
kubectl rollout restart deployment app -n production

# 6. Verify the new cert is being served
curl -vI https://app.example.com 2>&1 | grep -E "expire|subject"

Quick Reference

• Cheatsheet: TLS & PKI

Pages that link here

• Anti-Primer: TLS Certificates Ops
• Production Readiness Review: Study Plans
• TLS & Certificates Ops
• TLS & PKI Cheat Sheet
• Thinking Out Loud: TLS Certificates Ops