Skip to content

Portal | Level: L2: Operations | Topics: Kubernetes Networking | Domain: Kubernetes

Runbook: NetworkPolicy Blocking Traffic

Symptoms

  • Application can't reach other services (timeouts)
  • External requests to pods fail
  • DNS resolution may also fail if egress is blocked

Fast Triage

# List all NetworkPolicies in namespace
kubectl get networkpolicy -n grokdevops
kubectl describe networkpolicy -n grokdevops

# Test connectivity from app pod
kubectl exec -n grokdevops deploy/grokdevops -- wget -qO- --timeout=3 http://grokdevops/health 2>&1 || echo "BLOCKED"

# Test DNS
kubectl exec -n grokdevops deploy/grokdevops -- nslookup kubernetes.default 2>&1 || echo "DNS BLOCKED"

# Check pod labels match policy selectors
kubectl get pods -n grokdevops --show-labels

Likely Causes (ranked)

  1. Deny-all policy applied — blocks all ingress/egress
  2. Missing egress rule for DNS — pods can't resolve names (need port 53 to kube-system)
  3. Label selector too broad — policy matches more pods than intended
  4. Chaos script left a policy — training/interactive/chaos/scripts/toggle_networkpolicy.sh applied but not removed

Evidence Interpretation

What bad looks like: 

$ kubectl exec -n grokdevops deploy/grokdevops -- wget -qO- --timeout=3 http://grokdevops/health
wget: download timed out

$ kubectl exec -n grokdevops deploy/grokdevops -- nslookup kubernetes.default
;; connection timed out; no servers could be reached
- Connection timeouts (not refused) strongly suggest a NetworkPolicy is dropping packets silently. - If DNS also fails (nslookup times out), an egress policy is blocking UDP port 53 to kube-system. - kubectl get networkpolicy -n grokdevops will list active policies. Check spec.podSelector to see which pods are affected.

Fix Steps

  1. Remove the offending policy: 
    kubectl delete networkpolicy <policy-name> -n grokdevops
  2. If it was a chaos script: 
    kubectl delete networkpolicy chaos-deny-all -n grokdevops
  3. If you need the policy but with DNS allowed, add egress for DNS: 
    egress:
- to:
  - namespaceSelector: {}
  ports:
  - protocol: UDP
    port: 53

Verification

kubectl exec -n grokdevops deploy/grokdevops -- wget -qO- --timeout=3 http://grokdevops/health

Cleanup

kubectl get networkpolicy -n grokdevops -l chaos=true --no-headers | awk '{print $1}' | xargs -r kubectl delete networkpolicy -n grokdevops

Unknown Unknowns

  • Without any NetworkPolicy in a namespace, all ingress and egress traffic is allowed. The moment you add ONE policy selecting a pod, that direction (ingress or egress) becomes default-deny for that pod.
  • Adding a single egress policy that allows traffic to your database but forgets DNS (port 53/UDP) will break all name resolution for matched pods.
  • NetworkPolicies are additive (union). Multiple policies selecting the same pod combine their allow rules — there is no way to write a "deny this specific traffic" policy, only "allow this."
  • Policies are enforced by the CNI plugin. If the CNI doesn't support NetworkPolicy (e.g., default Flannel), policies are silently ignored.

[!WARNING] Any egress NetworkPolicy that does not explicitly allow UDP port 53 to kube-system will silently break DNS resolution for all matched pods. This is the most common NetworkPolicy mistake and the symptoms (connection timeouts) do not obviously point to DNS.

Pitfalls

  • Forgetting DNS egress rule — the most common mistake. Any egress policy must include UDP port 53 or DNS breaks.
  • Not checking if a chaos script left a policy — the chaos script toggle_networkpolicy.sh applies chaos-deny-all. Check for policies with label chaos=true.
  • Applying deny-all without exception rules — a deny-all ingress + egress policy with no additional allow policies will isolate pods completely, including from DNS and health checks.

See Also

  • training/interactive/exercises/levels/level-25/k8s-networkpolicy/
  • training/interactive/chaos/scripts/toggle_networkpolicy.sh
  • training/interactive/incidents/scenarios/networkpolicy-block-ingress.sh

Wiki Navigation

Related Content