Remediation: User Auth Failing, OIDC Cert Expired, Fix Is Cloud KMS Rotation¶

Immediate Fix (Cloud — Domain C)¶

The fix requires regenerating the X.509 wrapping certificate for the old signing key using AWS KMS.

Step 1: Regenerate the wrapping certificate via KMS¶

# Generate a new self-signed certificate with 2-year validity for the existing key
$ aws kms get-public-key --key-id alias/keycloak-signing-v1 \
    --output text --query 'PublicKey' | base64 -d > /tmp/v1-pubkey.der

$ openssl req -new -x509 -key /tmp/v1-privkey.pem \
    -out /tmp/v1-cert-renewed.pem \
    -days 730 \
    -subj "/CN=keycloak-signing-ca" \
    -sha256

# Or use KMS to sign the certificate
$ aws kms sign --key-id alias/keycloak-signing-v1 \
    --signing-algorithm RSASSA_PKCS1_V1_5_SHA_256 \
    --message fileb:///tmp/v1-cert-tbs.der \
    --message-type RAW

Step 2: Update the Keycloak signing key certificate¶

# Update the Kubernetes secret with the renewed certificate
$ kubectl create secret generic keycloak-signing-keys -n auth \
    --from-file=realm-key-v1.pem=/tmp/v1-cert-renewed.pem \
    --from-file=realm-key-v2.pem=/tmp/v2-cert.pem \
    --dry-run=client -o yaml | kubectl apply -f -

# Restart Keycloak to pick up the new certificate
$ kubectl rollout restart statefulset/keycloak -n auth
$ kubectl rollout status statefulset/keycloak -n auth

Step 3: Force the auth-gateway to refresh its JWKS cache¶

$ kubectl rollout restart deployment/auth-gateway -n prod
$ kubectl rollout status deployment/auth-gateway -n prod

Step 4: Set up automatic certificate renewal in KMS¶

# Create a Lambda function to renew wrapping certificates 30 days before expiry
$ aws lambda create-function \
    --function-name keycloak-cert-renewer \
    --runtime python3.11 \
    --handler renew.handler \
    --role arn:aws:iam::123456789:role/keycloak-cert-renewer-role \
    --zip-file fileb://renew_function.zip

# Schedule it monthly
$ aws events put-rule \
    --name keycloak-cert-check-monthly \
    --schedule-expression "rate(30 days)"

$ aws events put-targets \
    --rule keycloak-cert-check-monthly \
    --targets "Id"="1","Arn"="arn:aws:lambda:us-east-1:123456789:function:keycloak-cert-renewer"

Verification¶

Domain A (Security) — Auth working¶

# Test login
$ TOKEN=$(curl -s -X POST "https://auth.example.com/realms/prod/protocol/openid-connect/token" \
    -d "client_id=test-client&grant_type=password&username=test&password=test123" | jq -r '.access_token')

$ curl -s -H "Authorization: Bearer $TOKEN" https://api.example.com/v1/user/me | jq .status
"authenticated"

# Check auth failure rate
# rate(auth_failures_total[5m]) / rate(auth_attempts_total[5m])
# Result: 0.001 (0.1% — back to baseline)

Domain B (Kubernetes) — Keycloak serving renewed certificates¶

$ curl -s https://auth.example.com/realms/prod/protocol/openid-connect/certs | \
    jq '.keys[] | select(.kid=="realm-key-v1") | .x5c[0]' -r | \
    base64 -d | openssl x509 -noout -dates
notBefore=Mar 19 08:30:00 2026 GMT
notAfter=Mar 19 08:30:00 2028 GMT

Domain C (Cloud) — KMS rotation automated¶

$ aws events list-rules --name-prefix keycloak-cert
{
    "Rules": [{
        "Name": "keycloak-cert-check-monthly",
        "State": "ENABLED",
        "ScheduleExpression": "rate(30 days)"
    }]
}

Prevention¶

Monitoring: Add certificate expiry monitoring for all OIDC signing key certificates. Alert 30 days before expiry.

- alert: OIDCSigningCertExpiringSoon
  expr: keycloak_signing_cert_expiry_seconds < 2592000  # 30 days
  labels:
    severity: warning
  annotations:
    summary: "OIDC signing key certificate expires in {{ $value | humanizeDuration }}"

Runbook: OIDC key rotation must include certificate renewal for the wrapping X.509 certificates. The key's cryptographic validity and the certificate's validity are independent — both must be managed.
Architecture: Use JWK-based validation (verify the signature against the public key directly) instead of X.509 certificate chain validation. This eliminates the dependency on certificate expiry for key validation. If X.509 is required, set certificate validity to 10 years or use auto-renewal.