Skip to content

Portal | Level: L1: Foundations | Topics: cert-manager | Domain: Kubernetes

cert-manager — Primer

Why This Matters

Manual TLS certificate management does not scale. With three services you can track renewal dates in a spreadsheet. With 30 services across 4 clusters, you cannot. Missed renewals cause outages. Manually copying certs into Kubernetes Secrets is error-prone and leaks private keys through shell history and CI logs.

Who made it: cert-manager was created by Jetstack (founded 2014), acquired by Venafi in 2020. It became a CNCF incubating project in 2022 — the first project focused entirely on certificate lifecycle management to join the foundation.

cert-manager automates the full certificate lifecycle in Kubernetes:

  • Issuance: requests certificates from Let's Encrypt, an internal CA, Vault PKI, or self-signed
  • Renewal: renews automatically at 2/3 of validity (default), before expiry
  • Storage: writes certs into Kubernetes Secrets that pods and Ingresses consume
  • Ingress integration: annotate an Ingress and cert-manager provisions the cert automatically

Without cert-manager, an operator must remember renewal dates, run certbot, copy PEM files into Secrets, and restart Ingress controllers — for every certificate, in every cluster, forever.

Core Concepts

1. Architecture

cert-manager runs as a set of controllers in the cert-manager namespace:

cert-manager-controller          — core reconciliation loop
cert-manager-cainjector           — injects CA bundles into webhooks
cert-manager-webhook              — validates CRD resources

The controllers watch cert-manager CRDs and drive the certificate lifecycle:

Certificate (desired state)
  └─► CertificateRequest (one-shot signing request)
        └─► Order (ACME order for domain validation)
              └─► Challenge (HTTP-01 or DNS-01 validation)

2. CRDs

CRD Purpose
Certificate Declares a desired cert: domain, issuer, Secret name, duration
Issuer Namespace-scoped issuer configuration (Let's Encrypt, CA, Vault, etc.)
ClusterIssuer Cluster-scoped issuer — accessible from any namespace
CertificateRequest Auto-created per-signing-request (usually don't create manually)
Order ACME order lifecycle object (auto-created)
Challenge ACME challenge (HTTP-01 or DNS-01) — auto-created

3. Installation

# Install via Helm (recommended for production)
helm repo add jetstack https://charts.jetstack.io
helm repo update

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set installCRDs=true \
  --version v1.14.4

# Verify
kubectl get pods -n cert-manager
# NAME                                      READY   STATUS    RESTARTS   AGE
# cert-manager-5c47f46f57-p8wq7             1/1     Running   0          60s
# cert-manager-cainjector-6659d6844-xq8gj   1/1     Running   0          60s
# cert-manager-webhook-547c9d6d-v5lbt       1/1     Running   0          60s

# Install kubectl plugin
kubectl apply -f https://github.com/cert-manager/cmctl/releases/latest/download/cmctl.yaml
# Or via krew:
kubectl krew install cert-manager

4. Issuers

Self-signed Issuer (useful for testing or bootstrapping a CA): 

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}

CA Issuer (sign with your own CA certificate): 

# First, create a Secret holding the CA cert + key
kubectl create secret tls my-ca-secret \
  --cert=ca.crt \
  --key=ca.key \
  -n cert-manager

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ca-issuer
spec:
  ca:
    secretName: my-ca-secret

Let's Encrypt (ACME) — HTTP-01: 

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - http01:
          ingress:
            class: nginx

Let's Encrypt (ACME) — DNS-01 with Route53: 

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dns01
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@example.com
    privateKeySecretRef:
      name: letsencrypt-dns01-account-key
    solvers:
      - dns01:
          route53:
            region: us-east-1
            hostedZoneID: Z1XXXXXXXXXXXXX   # optional but faster
            accessKeyIDSecretRef:
              name: route53-credentials
              key: access-key-id
            secretAccessKeySecretRef:
              name: route53-credentials
              key: secret-access-key

Let's Encrypt — DNS-01 with Cloudflare: 

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-cloudflare
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@example.com
    privateKeySecretRef:
      name: letsencrypt-cloudflare-account-key
    solvers:
      - dns01:
          cloudflare:
            email: ops@example.com
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token

Let's Encrypt — DNS-01 with GCP Cloud DNS: 

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-gcpdns
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@example.com
    privateKeySecretRef:
      name: letsencrypt-gcpdns-key
    solvers:
      - dns01:
          cloudDNS:
            project: my-gcp-project
            serviceAccountSecretRef:
              name: clouddns-sa-key
              key: key.json

Vault PKI Issuer: 

# First enable and configure Vault PKI:
# vault secrets enable pki
# vault write pki/root/generate/internal common_name=example.com ttl=87600h
# vault write pki/roles/example-com allowed_domains=example.com allow_subdomains=true max_ttl=72h

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: vault-issuer
spec:
  vault:
    server: https://vault.example.com
    path: pki/sign/example-com
    auth:
      kubernetes:
        mountPath: /v1/auth/kubernetes
        role: cert-manager
        secretRef:
          name: cert-manager-vault-token
          key: token

5. Certificate Resources

Basic Certificate: 

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example-com-tls
  namespace: default
spec:
  secretName: example-com-tls         # Secret that will hold the cert
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName: example.com
  dnsNames:
    - example.com
    - www.example.com
  duration: 2160h    # 90 days (Let's Encrypt default)
  renewBefore: 720h  # Renew 30 days before expiry (1/3 of 90 days)

Gotcha: A wildcard *.example.com does NOT cover the apex domain example.com itself. Always include both *.example.com and example.com in dnsNames or you will get certificate errors when users visit the bare domain.

Wildcard Certificate (requires DNS-01): 

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-example-com
  namespace: default
spec:
  secretName: wildcard-example-com-tls
  issuerRef:
    name: letsencrypt-dns01
    kind: ClusterIssuer
  dnsNames:
    - "*.example.com"
    - example.com        # include apex too — wildcard doesn't cover it
  duration: 2160h
  renewBefore: 720h

Certificate with IP SAN: 

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: internal-service-tls
  namespace: production
spec:
  secretName: internal-service-tls
  issuerRef:
    name: ca-issuer
    kind: ClusterIssuer
  dnsNames:
    - internal-svc.production.svc.cluster.local
  ipAddresses:
    - 10.0.1.50
  duration: 8760h    # 1 year for internal CA certs
  renewBefore: 720h

6. Ingress Annotation Auto-Provisioning

The simplest way to use cert-manager: annotate your Ingress and cert-manager creates the Certificate automatically.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
  namespace: default
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    # cert-manager.io/issuer: "my-namespace-issuer"   # for namespace-scoped Issuer
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - myapp.example.com
      secretName: myapp-example-com-tls   # cert-manager creates this Secret
  rules:
    - host: myapp.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: myapp
                port:
                  number: 80

7. Certificate Lifecycle and Renewal

cert-manager renews certificates at renewBefore time before expiry. Default behavior:

Certificate issued with 90-day validity
  → cert-manager stores in Secret
  → at 60 days remaining (after 30 days), renews automatically
  → new cert stored in same Secret
  → pods/Ingress pick up new cert (may need restart if mounting as file)

Check certificate status: 

# Via kubectl
kubectl get certificate -A
# NAME               READY   SECRET                     AGE
# example-com-tls    True    example-com-tls            15d

kubectl describe certificate example-com-tls
# Status:
#   Conditions:
#     Type: Ready — Status: True
#     Message: Certificate is up to date and has not expired

# Via cert-manager kubectl plugin
kubectl cert-manager status certificate example-com-tls

# Check expiry directly from the Secret
kubectl get secret example-com-tls -o jsonpath='{.data.tls\.crt}' | \
  base64 -d | openssl x509 -noout -dates
# notBefore=Jan  1 00:00:00 2024 GMT
# notAfter=Apr  1 00:00:00 2024 GMT

8. HTTP-01 vs DNS-01 Challenge Comparison

Factor HTTP-01 DNS-01
Mechanism cert-manager creates a pod/Ingress route at /.well-known/acme-challenge/ cert-manager creates a _acme-challenge TXT record
Wildcard certs Not supported Supported
Private clusters Requires public ingress Works on air-gapped clusters
DNS provider API Not needed Required
Speed Faster (seconds to verify) Slower (DNS propagation, 30–120s)
Use case Single-name certs on public clusters Wildcards, private clusters

Remember: Mnemonic for challenge types: "HTTP for Hosts, DNS for Domains (wildcards)." If you need *.example.com, DNS-01 is the only option — HTTP-01 cannot validate wildcard certificates per the ACME specification (RFC 8555).

9. Troubleshooting Failed Certificate Issuance

When a Certificate is stuck in False/Pending:

# Step 1: Check Certificate status
kubectl describe certificate myapp-tls -n default
# Look for: Status.Conditions and Events

# Step 2: Check the CertificateRequest
kubectl get certificaterequest -n default
kubectl describe certificaterequest myapp-tls-xxxxx -n default

# Step 3: Check the Order (ACME)
kubectl get order -n default
kubectl describe order myapp-tls-xxxxx -n default
# Look for: Status, Reason, Message

# Step 4: Check the Challenge
kubectl get challenge -n default
kubectl describe challenge myapp-tls-xxxxx -n default
# Common issue: "Waiting for DNS record to propagate"
# Common issue: "Error presenting challenge: Route53 access denied"
# Common issue: "HTTP challenge solver pod not running"

# Step 5: Check cert-manager controller logs
kubectl logs -n cert-manager deployment/cert-manager -f --tail=100

# Step 6: Check if HTTP-01 solver is reachable
curl -v http://myapp.example.com/.well-known/acme-challenge/test-probe

# Step 7: Force renewal (delete the Secret to trigger re-issuance)
kubectl delete secret myapp-tls -n default
# cert-manager detects missing Secret and re-issues

# Step 8: Use cmctl for quick checks
kubectl cert-manager check api   # verify cert-manager API is healthy

10. Monitoring Certificate Expiry with Prometheus

cert-manager exposes Prometheus metrics:

# Scrape config for Prometheus
scrape_configs:
  - job_name: cert-manager
    static_configs:
      - targets: ['cert-manager.cert-manager.svc:9402']

Key metrics: 

# Certificates expiring within 7 days
certmanager_certificate_expiration_timestamp_seconds - time() < 604800

# Certificates in Ready=False state
certmanager_certificate_ready_status{condition="False"} == 1

# Alert rule
groups:
  - name: cert-manager
    rules:
      - alert: CertificateExpiringSoon
        expr: |
          (certmanager_certificate_expiration_timestamp_seconds - time()) / 86400 < 14
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Certificate {{ $labels.name }} expires in {{ $value | humanizeDuration }}"

      - alert: CertificateNotReady
        expr: certmanager_certificate_ready_status{condition="False"} == 1
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Certificate {{ $labels.name }} in namespace {{ $labels.namespace }} is not ready"

Quick Reference

# Install cert-manager
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --set installCRDs=true --version v1.14.4

# List all certificates and their status
kubectl get certificate -A

# Check certificate expiry
kubectl get secret <secret-name> -o jsonpath='{.data.tls\.crt}' | \
  base64 -d | openssl x509 -noout -dates

# Force renewal
kubectl cert-manager renew <certificate-name>

# Check ACME challenge status
kubectl get challenge -A

# Tail controller logs
kubectl logs -n cert-manager deployment/cert-manager -f

# Delete Secret to trigger re-issuance
kubectl delete secret <tls-secret-name>

Wiki Navigation

Prerequisites