Skip to content

Portal | Level: L2: Operations | Topics: Policy Engines | Domain: Kubernetes

Runbook: Kyverno / Policy Engine Blocking Workloads

Symptoms

  • Pod creation rejected: admission webhook "validate.kyverno.svc" denied the request
  • All deployments to a namespace fail
  • Rollbacks also fail (new pods can't be created)
  • Started after a policy change or Kyverno upgrade

Fast Triage (under 2 minutes)

# Get the exact error
kubectl apply -f <manifest> 2>&1

# Check which policy is blocking
kubectl get clusterpolicy -o json | jq '.items[] | select(.spec.validationFailureAction == "Enforce") | .metadata.name'

# Check policy reports for violations
kubectl get policyreport -A

Causes and Fixes

1. Quick Unblock (Switch to Audit Mode)

[!WARNING] Switching a policy to Audit mode unblocks deployments but also masks real violations. Non-compliant workloads will deploy silently with only a policy report entry. Do not leave policies in Audit mode long-term without monitoring reports — you lose enforcement with no visible signal.

If you need to unblock deployments immediately:

# Switch the blocking policy to Audit
kubectl patch clusterpolicy <policy-name> --type=merge \
  -p '{"spec":{"validationFailureAction":"Audit"}}'

This allows pods through while logging violations instead of blocking.

2. Policy Matching Too Broadly

Common issue: policy checks initContainers but init containers injected by webhooks (Istio, Vault) don't comply.

Fix: Update policy to exclude init containers or add exceptions. 

# Use =(initContainers) — only validate if present in the manifest
validate:
  pattern:
    spec:
      containers:
      - resources:
          limits:
            cpu: "?*"
      =(initContainers):
      - resources:
          limits:
            cpu: "?*"

3. Policy Conflicts with System Namespaces

# Check if system pods are affected
kubectl get pods -n kube-system --field-selector=status.phase=Pending

Fix: Add namespace exclusions. 

spec:
  rules:
  - name: check-resources
    exclude:
      any:
      - resources:
          namespaces: ["kube-system", "kube-public", "cert-manager", "istio-system"]

4. Kyverno Webhook Down

kubectl get pods -n kyverno
kubectl get validatingwebhookconfigurations | grep kyverno

If Kyverno pods are down and failurePolicy: Fail, nothing can be deployed.

Fix: 

# Change webhook failure policy to Ignore (temporary)
kubectl get validatingwebhookconfigurations kyverno-resource-validating-webhook-cfg -o yaml | \
  sed 's/failurePolicy: Fail/failurePolicy: Ignore/' | kubectl apply -f -

# Then fix Kyverno
kubectl rollout restart deployment -n kyverno

Verification

# Test deployment
kubectl apply -f <manifest>

# Check policy reports
kubectl get policyreport -n <ns>

# Verify Kyverno is healthy
kubectl get pods -n kyverno

Prevention

  • Always deploy policies in Audit mode first (1-2 weeks)
  • Test policies with all webhook combinations (Istio, Vault, etc.)
  • Set failurePolicy: Ignore on webhooks (prevent total lockout)
  • Exclude system namespaces from all policies
  • Keep a "break glass" procedure to disable policies in emergencies

Wiki Navigation

Related Content