Skip to content

Portal | Level: L1: Foundations | Topics: Kubernetes Networking | Domain: Kubernetes

Runbook: Ingress Returns 404

Symptoms

  • Accessing the application via ingress URL returns 404
  • Service works when port-forwarded directly
  • Ingress resource exists but routing fails

Fast Triage

# Check ingress resource
kubectl get ingress -n grokdevops
kubectl describe ingress -n grokdevops

# Check ingress controller pods
kubectl get pods -n kube-system -l app.kubernetes.io/name=traefik  # k3s default

# Check service and endpoints
kubectl get svc grokdevops -n grokdevops
kubectl get endpoints grokdevops -n grokdevops

# Test service directly
kubectl port-forward svc/grokdevops -n grokdevops 8000:80
curl http://localhost:8000/health

Likely Causes (ranked)

  1. Wrong path or host in ingress spec — path doesn't match app routes
  2. Service name mismatch — ingress backend points to wrong service
  3. No endpoints — service has no ready pods (readiness probe failing)
  4. Ingress controller not running — traefik/nginx not deployed
  5. Path type wrong — Prefix vs Exact vs ImplementationSpecific

Evidence Interpretation

What bad looks like: 

$ curl -H "Host: grokdevops.local" http://192.168.1.100/
404 page not found

$ kubectl port-forward svc/grokdevops -n grokdevops 8000:80
$ curl http://localhost:8000/health
{"status": "ok"}    # works via port-forward
- A 404 via ingress but 200 via port-forward means the issue is in the ingress routing layer, not the app. - Check kubectl describe ingress — look at the Rules section for the host, path, and backend. A mismatch in any of these causes 404. - Empty endpoints (kubectl get endpoints) means the ingress has nowhere to route traffic, which also produces 404.

Fix Steps

  1. Compare ingress path with app routes: 
    kubectl get ingress -n grokdevops -o yaml
# Check .spec.rules[].http.paths[].path matches app routes
  2. Verify backend service exists and has endpoints: 
    kubectl get endpoints -n grokdevops
  3. Fix ingress path if wrong: 
    kubectl edit ingress <name> -n grokdevops

Verification

curl -H "Host: <ingress-host>" http://<node-ip>/health
# or via ingress directly if DNS is configured

Cleanup

None needed beyond fixing the ingress resource.

Unknown Unknowns

[!TIP] pathType: Exact only matches the literal path — / with Exact will 404 on /health, /api, or any subpath. If your app has multiple routes, use pathType: Prefix or add separate Ingress rules for each path.

  • pathType matters: Prefix with / matches all subpaths, Exact with / only matches the root — a request to /health would 404 with Exact.
  • Ingress controllers may rewrite the request path before forwarding. Traefik and nginx handle path stripping differently; check annotations.
  • Multiple Ingress resources with overlapping hosts/paths can conflict silently; the controller picks one and the other returns 404.

Pitfalls

  • Not checking endpoints — if the backend service has zero endpoints (readiness probe failing), the ingress has nothing to route to. Fix readiness first.
  • Testing with the wrong Host header — ingress rules are host-based; curl http://IP/path without -H "Host: ..." may hit the wrong rule or the default backend.
  • Forgetting the ingress class annotation — if multiple ingress controllers exist, each ingress resource must specify which controller handles it via ingressClassName or annotation.

See Also

  • training/interactive/exercises/levels/level-24/k8s-ingress/
  • training/interview-scenarios/10-ingress-404.md
  • training/interactive/incidents/scenarios/service-selector-mismatch.sh
  • training/library/runbooks/kubernetes/readiness_probe_failed.md (if 404 is caused by empty endpoints)

Wiki Navigation

Related Content