Portal | Level: L2: Operations | Topics: Policy Engines | Domain: Kubernetes

Policy Engine Drills

Remember: Policy engines enforce guardrails at admission time — before a resource is persisted to etcd. Two main tools: Kyverno (YAML-native, lower learning curve) and OPA/Gatekeeper (Rego language, more powerful but harder). Start with Kyverno for label/image/resource policies; consider OPA when you need complex cross-resource logic.

Gotcha: Always deploy new policies in Audit mode first (validationFailureAction: Audit). Switching directly to Enforce can block deployments cluster-wide — including your own monitoring and system pods. Check the policy report (kubectl get policyreport) for violations before switching to Enforce.

Debug clue: If a pod creation is blocked by an admission policy, the error message appears in kubectl describe replicaset events, NOT in kubectl describe pod (because the pod was never created). Check the ReplicaSet or the parent Deployment events for the rejection reason.

Drill 1: Require Labels with Kyverno

Difficulty: Easy

Q: Write a Kyverno policy that requires all Deployments to have team and env labels. Start in Audit mode.

Answer

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-labels
spec:
  validationFailureAction: Audit
  rules:
  - name: require-team-and-env
    match:
      any:
      - resources:
          kinds: ["Deployment"]
    validate:
      message: "Labels 'team' and 'env' are required on Deployments."
      pattern:
        metadata:
          labels:
            team: "?*"
            env: "?*"

`Audit` mode logs violations without blocking. Switch to `Enforce` after fixing existing violations.

---

Drill 2: Block Latest Tag

Difficulty: Easy

Q: Write a policy that blocks any pod using the latest tag or no tag on container images.

Answer

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
spec:
  validationFailureAction: Enforce
  rules:
  - name: require-image-tag
    match:
      any:
      - resources:
          kinds: ["Pod"]
    validate:
      message: "Images must have an explicit tag that is not 'latest'."
      pattern:
        spec:
          containers:
          - image: "*:*"
  - name: disallow-latest
    match:
      any:
      - resources:
          kinds: ["Pod"]
    validate:
      message: "The 'latest' tag is not allowed."
      deny:
        conditions:
          any:
          - key: "{{ request.object.spec.containers[].image }}"
            operator: AnyIn
            value: ["*:latest"]

---

Drill 3: Mutate — Add Default Resources

Difficulty: Medium

Q: Write a Kyverno policy that automatically adds default resource requests (100m CPU, 128Mi memory) to any container that doesn't specify them.

Answer

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-default-resources
spec:
  rules:
  - name: add-default-requests
    match:
      any:
      - resources:
          kinds: ["Pod"]
    mutate:
      patchStrategicMerge:
        spec:
          containers:
          - (name): "*"
            resources:
              requests:
                +(cpu): "100m"
                +(memory): "128Mi"

The `+()` operator means "add if not present" — it won't override existing values.

---

Drill 4: Generate NetworkPolicy

Difficulty: Medium

Q: Write a Kyverno policy that auto-generates a default-deny NetworkPolicy whenever a new namespace is created.

Answer

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: generate-default-deny
spec:
  rules:
  - name: default-deny-ingress
    match:
      any:
      - resources:
          kinds: ["Namespace"]
    exclude:
      any:
      - resources:
          namespaces: ["kube-system", "kube-public", "argocd"]
    generate:
      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      name: default-deny-all
      namespace: "{{request.object.metadata.name}}"
      data:
        spec:
          podSelector: {}
          policyTypes:
          - Ingress
          - Egress

Every new namespace (except excluded ones) gets a default-deny NetworkPolicy automatically.

---

Drill 5: OPA Gatekeeper — Required Labels

Difficulty: Medium

Q: Implement the same "require labels" policy using OPA Gatekeeper (ConstraintTemplate + Constraint).

Answer

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
      validation:
        openAPIV3Schema:
          type: object
          properties:
            labels:
              type: array
              items:
                type: string
  targets:
  - target: admission.k8s.gatekeeper.sh
    rego: |
      package k8srequiredlabels

      violation[{"msg": msg}] {
        provided := {label | input.review.object.metadata.labels[label]}
        required := {label | label := input.parameters.labels[_]}
        missing := required - provided
        count(missing) > 0
        msg := sprintf("Missing required labels: %v", [missing])
      }
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: require-team-env
spec:
  enforcementAction: deny
  match:
    kinds:
    - apiGroups: ["apps"]
      kinds: ["Deployment"]
  parameters:
    labels: ["team", "env"]

---

Drill 6: Pod Security Standards

Difficulty: Easy

Q: Apply the restricted Pod Security Standard to the production namespace in enforce mode and baseline in warn mode.

Answer

kubectl label namespace production \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=latest \
  pod-security.kubernetes.io/warn=baseline \
  pod-security.kubernetes.io/warn-version=latest \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/audit-version=latest

Levels: - `privileged` — unrestricted (default) - `baseline` — minimally restrictive (blocks known escalations) - `restricted` — heavily restricted (best practice for hardened workloads) `restricted` requires: - `runAsNonRoot: true` - Drop ALL capabilities, add only needed ones - `seccompProfile: RuntimeDefault` or `Localhost` - No hostPath, hostNetwork, hostPID, etc.

---

Drill 7: Check Policy Violations

Difficulty: Easy

Q: How do you see which resources are violating policies in Kyverno and Gatekeeper?

Answer

# Kyverno — Policy Reports (per namespace)
kubectl get policyreport -A
kubectl describe policyreport -n production

# Kyverno — Cluster Policy Reports (cluster-scoped)
kubectl get clusterpolicyreport

# Detailed violations
kubectl get policyreport -n production -o yaml | \
  yq '.results[] | select(.result == "fail")'

# Gatekeeper — violations on the constraint
kubectl get k8srequiredlabels require-team-env -o yaml
# Look at .status.violations[]

# Gatekeeper — audit all constraints
kubectl get constraints -o json | \
  jq '.items[] | {name: .metadata.name, violations: .status.totalViolations}'

---

Drill 8: Image Registry Restriction

Difficulty: Medium

Q: Write a Kyverno policy that only allows images from registry.example.com/ and gcr.io/my-project/.

Answer

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  rules:
  - name: validate-registries
    match:
      any:
      - resources:
          kinds: ["Pod"]
    validate:
      message: "Images must come from approved registries."
      pattern:
        spec:
          containers:
          - image: "registry.example.com/* | gcr.io/my-project/*"
          =(initContainers):
          - image: "registry.example.com/* | gcr.io/my-project/*"

The `|` acts as OR. The `=()` on initContainers means "if present, validate."

---

Drill 9: Rollout Strategy — Audit to Enforce

Difficulty: Medium

Q: Describe the process for safely rolling out a new policy from zero to full enforcement.

Answer

Phase 1: Audit (1-2 weeks)
  - Deploy policy in Audit mode
  - Monitor PolicyReports for violations
  - Communicate to teams what will be enforced

Phase 2: Warn (1 week)
  - Switch to Warn mode (Gatekeeper) or keep Audit + add to CI
  - Users see warnings but aren't blocked
  - Fix remaining violations

Phase 3: Enforce with exceptions
  - Switch to Enforce mode
  - Add exclusions for known exceptions:

    exclude:
      any:
      - resources:
          namespaces: ["legacy-app"]
      - resources:
          annotations:
            policies.example.com/exempt: "TICKET-123"

Phase 4: Full enforcement
  - Remove exceptions as teams fix their resources
  - Add policy to CI pipeline (shift left)

# Kyverno: switch from audit to enforce
kubectl patch clusterpolicy require-labels --type=merge \
  -p '{"spec":{"validationFailureAction":"Enforce"}}'

# Gatekeeper: switch from dryrun to deny
kubectl patch k8srequiredlabels require-team-env --type=merge \
  -p '{"spec":{"enforcementAction":"deny"}}'

---

Drill 10: Cosign Image Verification

Difficulty: Hard

Q: Write a Kyverno policy that verifies all images in the production namespace are signed with cosign using a specific public key.

Answer

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  rules:
  - name: verify-cosign-signature
    match:
      any:
      - resources:
          kinds: ["Pod"]
          namespaces: ["production"]
    verifyImages:
    - imageReferences:
      - "registry.example.com/*"
      attestors:
      - count: 1
        entries:
        - keys:
            publicKeys: |-
              -----BEGIN PUBLIC KEY-----
              MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
              -----END PUBLIC KEY-----
      mutateDigest: true      # Replace tag with digest
      verifyDigest: true      # Ensure digest hasn't changed

This ensures: 1. All images are signed with the specified key 2. Tags are replaced with digests (immutable reference) 3. Unsigned images are rejected Sign images with: `cosign sign --key cosign.key registry.example.com/my-app:v1.0`

---

Wiki Navigation

Prerequisites

• Policy Engines (OPA / Kyverno) (Topic Pack, L2)

Related Content

• Interview: Kyverno Blocking Deploys (Scenario, L2) — Policy Engines
• Multi-Tenancy Patterns (Topic Pack, L2) — Policy Engines
• Policy Engines (OPA / Kyverno) (Topic Pack, L2) — Policy Engines
• Policy Engines Flashcards (CLI) (flashcard_deck, L1) — Policy Engines
• Runbook: Kyverno Blocking Workloads (Runbook, L2) — Policy Engines
• Skillcheck: Policy Engines (Assessment, L2) — Policy Engines

Pages that link here

• Drills
• Level 6: Advanced Platform Engineering
• Master Curriculum: 40 Weeks
• Multi-Tenancy Patterns
• Policy Engines (OPA / Kyverno)
• Policy Engines (OPA/Kyverno) - Primer
• Policy Engines - Skill Check
• Runbook: Kyverno / Policy Engine Blocking Workloads
• Scenario: Policy Engine Blocking All Deployments
• Track: Advanced Platform Engineering