Kubernetes Networking Footguns¶
Mistakes that cause silent connectivity failures, DNS outages, and traffic blackholes in production clusters.
1. Egress NetworkPolicy without DNS exception¶
You apply an egress NetworkPolicy to restrict outbound traffic. Pods can no longer resolve DNS because UDP/TCP port 53 to CoreDNS is blocked. Every service lookup fails silently — connections time out instead of getting "name not found."
What happens: All service discovery breaks. Pods cannot reach any service by name.
Why: NetworkPolicies are additive deny. If you restrict egress without explicitly allowing DNS (port 53), CoreDNS queries are blocked.
How to avoid: Every egress policy must include a rule allowing UDP and TCP port 53. Make this a template in your policy boilerplate.
Gotcha: DNS uses UDP by default but falls back to TCP for responses over 512 bytes (or 4096 with EDNS0). If you only allow UDP/53, DNSSEC-signed responses and large TXT records (common with SPF) will be truncated and the TCP fallback will be blocked. Always allow both protocols.
2. Service selector does not match pod labels¶
You create a Service with selector: {app: backend} but your pods are labeled app: backend-api. The Service has zero endpoints. Traffic to the Service ClusterIP goes nowhere.
What happens: Requests to the service hang or get connection refused. No error in the Service itself.
Why: Services find backends by label selector match. A mismatch means no endpoints are registered.
How to avoid: After creating a Service, always run kubectl get endpoints <svc-name>. If the list is empty, compare kubectl get svc <name> -o jsonpath='{.spec.selector}' with kubectl get pods --show-labels.
3. Hardcoding ClusterIPs instead of using DNS¶
You hardcode 10.96.45.123 in your app config because it was the ClusterIP at deploy time. After a Service recreation or cluster rebuild, the ClusterIP changes. Your app connects to nothing.
What happens: Silent connectivity failure after any Service recreation.
Why: ClusterIPs are assigned dynamically (unless explicitly set). They are not stable across Service deletions.
How to avoid: Always use DNS names: backend-api.production.svc.cluster.local. DNS is the service discovery mechanism.
4. ndots:5 causing slow external DNS resolution¶
Your app makes many calls to external APIs (e.g., api.stripe.com). Each lookup generates 5 DNS queries (appending search domains) before the final query succeeds. DNS latency adds 50-100ms per external call.
What happens: Slow external API calls. DNS lookup takes 5x longer than necessary.
Why: Default ndots:5 means any domain with fewer than 5 dots gets the search domains appended first.
How to avoid: Add dnsConfig to your pod spec: ndots: "2". Or append a trailing dot to external domains in your app config: api.stripe.com. (FQDN).
5. Assuming RWO volume means single-pod¶
You run two pods on the same node, both mounting the same RWO PersistentVolume. Both succeed because RWO means single-node, not single-pod. The database file gets corrupted by concurrent writes.
What happens: Data corruption from concurrent access.
Why: ReadWriteOnce restricts to a single node, not a single pod. Multiple pods on the same node can mount it.
How to avoid: Use ReadWriteOncePod (K8s 1.27+) for true single-pod access. Or ensure your application handles concurrent access safely.
6. kube-proxy iptables mode with thousands of services¶
Your cluster grows to 5,000+ Services. iptables rule sync takes 30+ seconds. During sync, new connections may be dropped or delayed. CPU usage on nodes spikes during rule updates.
What happens: Slow rule updates, increased latency, and occasional connection failures during sync.
Why: iptables mode updates are O(n) — every rule must be rewritten. At scale, this becomes a bottleneck.
How to avoid: Switch to IPVS mode (mode: "ipvs" in kube-proxy ConfigMap). IPVS uses kernel hash tables for O(1) lookups.
7. NetworkPolicy applied to wrong namespace¶
You create a NetworkPolicy in the default namespace thinking it applies cluster-wide. Pods in production are unaffected. Traffic you intended to block still flows.
What happens: Security policy has no effect. Traffic is not restricted as expected.
Why: NetworkPolicies are namespace-scoped. They only affect pods in the namespace where they are created.
How to avoid: Apply NetworkPolicies in every namespace that needs them. Use a policy engine like Kyverno or OPA Gatekeeper to enforce baseline policies across all namespaces.
8. Headless service returning stale pod IPs¶
You use a headless Service (clusterIP: None) for a StatefulSet. A pod gets rescheduled to a new node with a new IP, but DNS still returns the old IP for a few seconds due to TTL.
What happens: Clients connect to a non-existent IP and get timeouts until the DNS cache expires.
Why: DNS records have a TTL (default 30s in CoreDNS). Stale entries persist until TTL expires.
How to avoid: Use client-side retry logic. Lower CoreDNS TTL for headless services if needed. Ensure clients handle connection failures gracefully.
9. Ingress controller not installed¶
You apply an Ingress resource and expect traffic to route. Nothing happens. No error message — the Ingress resource is created successfully, but no controller is watching for it.
What happens: Ingress rules exist as Kubernetes objects but have no effect. External traffic cannot reach services.
Why: An Ingress resource is just a configuration object. A controller (NGINX, Traefik, etc.) must be installed to implement the routing.
How to avoid: Verify the ingress controller is running before creating Ingress resources. Check kubectl get pods -n ingress-nginx. Set ingressClassName to match your controller.
10. Forgetting externalTrafficPolicy on LoadBalancer services¶
Your LoadBalancer Service uses the default externalTrafficPolicy: Cluster. Traffic hits a node, gets SNAT'd to a backend pod on another node. The backend sees the node's IP, not the client's IP. Your access logs and rate limiting are broken.
What happens: Loss of client source IP. All requests appear to come from cluster-internal IPs.
Why: Cluster policy load-balances across all nodes, adding a hop and masking the source IP.
How to avoid: Set externalTrafficPolicy: Local to preserve source IP. Trade-off: traffic only goes to nodes running backend pods, so you need pods well-distributed across nodes.
Default trap: With
externalTrafficPolicy: Local, nodes without backend pods return ICMP unreachable for health checks. If your load balancer doesn't handle this gracefully (or health check intervals are too long), you get intermittent 502s during rolling deployments when a pod is briefly absent from a node. Pair withPodAntiAffinityto spread pods across nodes.