Skip to content

Answer Key: The Gateway That Returns 502

The System

An API gateway application serving api.megacorp.io, deployed behind an nginx ingress controller in Kubernetes. External traffic flows through the ingress controller to the api-gateway-svc Service, which routes to pods running the API gateway application.

[Internet] --> [nginx-ingress (10.0.50.12)]
                    |
              Ingress rule: api.megacorp.io -> api-gateway-svc:8080
                    |
              [api-gateway-svc (ClusterIP 10.96.117.44:8080)]
                    |
              targetPort: 8080 (BUG — should be 3000)
                    |
              [api-gateway pods] listening on port 3000
                    ^ Connection refused on 8080

What's Broken

Root cause: Port mismatch between the Kubernetes Service targetPort and the application's listening port. The application logs confirm it listens on port 3000. The Helm values define targetPort: 3000, but the deployed Service has targetPort: 8080 (visible in the endpoints output showing port 8080). This means either: - The Helm template has a bug (using .Values.apiGateway.service.port instead of .Values.apiGateway.service.targetPort) - Or the values were not applied in the last deployment

The nginx ingress controller connects to the Service on port 8080, the Service routes to port 8080 on the pod, but nothing is listening there. The connection is refused (errno 111), and the ingress returns 502 to the client.

Key clue: The nginx-ingress log shows "Connection refused" connecting to 10.244.3.18:8080. The api-gateway application log shows "port":3000. The mismatch is the diagnosis.

The Fix

Immediate

Patch the Service to use the correct targetPort: 

kubectl patch svc api-gateway-svc -n api-platform \
  --type='json' -p='[{"op":"replace","path":"/spec/ports/0/targetPort","value":3000}]'

Permanent (fix the Helm template)

Check the Service template for the port mapping: 

helm template api-gateway helm/api-gateway -f helm/values-prod.yaml | grep -A10 "kind: Service"

If the template uses .Values.apiGateway.service.port for targetPort, fix it: 

# In the Service template:
ports:
  - port: {{ .Values.apiGateway.service.port }}
    targetPort: {{ .Values.apiGateway.service.targetPort }}
    protocol: TCP

Then redeploy: 

helm upgrade api-gateway helm/api-gateway -f helm/values-prod.yaml -n api-platform

Verification

# Check endpoints now show port 3000
kubectl get endpoints api-gateway-svc -n api-platform

# Test the service directly
kubectl run -n api-platform curl --rm -i --image=curlimages/curl -- \
  curl -s http://api-gateway-svc:8080/

# Test through ingress
curl -H "Host: api.megacorp.io" https://10.0.50.12/

# Check ingress controller logs for 200s
kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --tail=5

Artifact Decoder

Artifact What It Revealed What Was Misleading
CLI Output Endpoints show port 8080 — but app listens on 3000; pods are Running/Ready (readiness probe may not check the right port) All pods show 1/1 Running which makes it look like the backend is healthy
Metrics 100% 502s with zero 200s and zero response duration = backend never responds Active connections (84) are from the ingress controller retrying, not successful traffic
IaC Snippet Values say targetPort: 3000 but deployed reality is 8080 — values vs deployed state mismatch The values file looks correct, which is the trap: the template may not be using targetPort
Log Lines Application log says port 3000, nginx log says connecting to 8080 — the mismatch The first nginx log line shows the 502 but does not clearly state the port mismatch without reading the second line

Skills Demonstrated

  • Tracing the full request path through Kubernetes (ingress -> service -> endpoints -> pod)
  • Identifying port mapping mismatches between Service and container
  • Understanding the difference between Helm values (desired state) and deployed resources (actual state)
  • Reading nginx ingress controller logs to diagnose upstream failures
  • Recognizing "Connection refused" as a port/process mismatch, not a network issue

Prerequisite Topic Packs