Skip to content

Anti-Primer: Kubernetes Services And Ingress

Everything that can go wrong, will — and in this story, it does.

The Setup

A team is setting up ingress for 20 microservices behind a single Ingress controller. They are migrating from a manual Nginx setup and assume Kubernetes Ingress works identically. The cutover is tonight.

The Timeline

Hour 0: Missing Backend Service Port

Ingress rule specifies port 80 but the Service targets port 8080 on the pod. The deadline was looming, and this seemed like the fastest path forward. But the result is all requests return 502; the Ingress controller cannot reach the backend.

Footgun #1: Missing Backend Service Port — ingress rule specifies port 80 but the Service targets port 8080 on the pod, leading to all requests return 502; the Ingress controller cannot reach the backend.

Nobody notices yet. The engineer moves on to the next task.

Hour 1: No Default Backend

Does not configure a default backend for unmatched routes. Under time pressure, the team chose speed over caution. But the result is requests to unknown paths return the Ingress controller's default 404 page, leaking infrastructure details.

Footgun #2: No Default Backend — does not configure a default backend for unmatched routes, leading to requests to unknown paths return the Ingress controller's default 404 page, leaking infrastructure details.

The first mistake is still invisible, making the next shortcut feel justified.

Hour 2: TLS Termination Misconfigured

Configures TLS on the Ingress but the backend expects HTTPS; Ingress sends HTTP to the backend. Nobody pushed back because the shortcut looked harmless in the moment. But the result is backend rejects plaintext connections; service appears down despite correct Ingress configuration.

Footgun #3: TLS Termination Misconfigured — configures TLS on the Ingress but the backend expects HTTPS; Ingress sends HTTP to the backend, leading to backend rejects plaintext connections; service appears down despite correct Ingress configuration.

Pressure is mounting. The team is behind schedule and cutting more corners.

Hour 3: Session Affinity Ignored

Stateful app requires sticky sessions but Ingress is configured with round-robin. The team had gotten away with similar shortcuts before, so nobody raised a flag. But the result is users lose session state on every other request; login loops and cart resets.

Footgun #4: Session Affinity Ignored — stateful app requires sticky sessions but Ingress is configured with round-robin, leading to users lose session state on every other request; login loops and cart resets.

By hour 3, the compounding failures have reached critical mass. Pages fire. The war room fills up. The team scrambles to understand what went wrong while the system burns.

The Postmortem

Root Cause Chain

# Mistake Consequence Could Have Been Prevented By
1 Missing Backend Service Port All requests return 502; the Ingress controller cannot reach the backend Primer: Ingress backend port must match the Service port, not the container port
2 No Default Backend Requests to unknown paths return the Ingress controller's default 404 page, leaking infrastructure details Primer: Configure a custom default backend that returns a safe error page
3 TLS Termination Misconfigured Backend rejects plaintext connections; service appears down despite correct Ingress configuration Primer: Match TLS termination point with backend protocol expectations
4 Session Affinity Ignored Users lose session state on every other request; login loops and cart resets Primer: Configure session affinity annotations on the Ingress for stateful backends

Damage Report

  • Downtime: 2-4 hours of pod-level or cluster-wide disruption
  • Data loss: Risk of volume data loss if StatefulSets were affected
  • Customer impact: Intermittent 5xx errors, dropped connections, or full service outage
  • Engineering time to remediate: 10-20 engineer-hours for incident response, rollback, and postmortem
  • Reputation cost: On-call fatigue; delayed feature work; possible SLA breach notification

What the Primer Teaches

  • Footgun #1: If the engineer had read the primer, section on missing backend service port, they would have learned: Ingress backend port must match the Service port, not the container port.
  • Footgun #2: If the engineer had read the primer, section on no default backend, they would have learned: Configure a custom default backend that returns a safe error page.
  • Footgun #3: If the engineer had read the primer, section on tls termination misconfigured, they would have learned: Match TLS termination point with backend protocol expectations.
  • Footgun #4: If the engineer had read the primer, section on session affinity ignored, they would have learned: Configure session affinity annotations on the Ingress for stateful backends.

Cross-References