Skip to content

Portal | Level: L2: Operations | Topics: Policy Engines | Domain: Kubernetes

Scenario: Policy Engine Blocking All Deployments

The Prompt

"Nobody can deploy anything to the production namespace. All pod creations are being rejected with 'admission webhook denied the request.' This started after we enabled Kyverno policies. How do you fix it without just disabling the policy engine?"

Initial Report

Slack: "Every deploy to production is failing. Error: 'resource violated policy require-resource-limits: all containers must have CPU and memory limits defined.' But our pods DO have limits set."

Constraints

  • Time pressure: No deployments can go through. Rollbacks also fail.
  • Don't just disable Kyverno: That would leave the cluster unprotected.
  • Must understand why the policy is triggering on compliant resources.

Observable Evidence

  • kubectl apply returns webhook denial
  • The Deployment YAML clearly has resource limits defined
  • Kyverno ClusterPolicy require-resource-limits is in Enforce mode
  • Policy was recently updated

Expected Investigation Path

# 1. Check the exact error
kubectl apply -f deployment.yaml 2>&1
# "validation error: all containers must have CPU and memory limits defined.
#  rule check-limits failed at path /spec/template/spec/initContainers/0/resources/limits/"

# 2. AH HA — it's checking initContainers too, not just containers
# Check the policy
kubectl get clusterpolicy require-resource-limits -o yaml

# 3. Check if init containers have limits
kubectl get deploy my-app -o yaml | grep -A10 initContainers

# 4. Fix option 1: Add limits to init containers
# Fix option 2: Update policy to exclude initContainers
# Fix option 3: Temporarily switch to Audit mode

# 5. Quick unblock (switch to Audit)
kubectl patch clusterpolicy require-resource-limits --type=merge \
  -p '{"spec":{"validationFailureAction":"Audit"}}'

# 6. Fix the policy to handle initContainers properly
# 7. Switch back to Enforce after fixing

Root Cause

The Kyverno policy was checking spec.containers AND spec.initContainers, but the init containers (like Istio's istio-init) injected by webhooks don't have resource limits. The policy matched broadly and blocked all pods that would receive init container injection.

What a Strong Answer Includes

  • Start by reading the exact error message (it tells you which path failed)
  • Understanding that mutating webhooks (Istio injection) add containers AFTER policy validation order matters
  • Quick unblock: switch to Audit mode (don't disable entirely)
  • Fix the policy: use =(initContainers) pattern in Kyverno (validate only if present)
  • Proper fix: use exclude for system-injected containers, or preconditions
  • Prevention: always test policies in Audit mode first, test with all webhook combinations

Wiki Navigation

Related Content