Skip to content

Portal | Level: L3: Advanced | Topics: Service Mesh | Domain: Kubernetes

Runbook: Istio 503 Errors After Mesh Enablement

Symptoms

  • All requests return 503: upstream connect error or disconnect/reset before headers
  • Pods show 2/2 Ready (sidecar injected)
  • Application container logs show no incoming requests
  • Started immediately after enabling Istio on the namespace

Fast Triage (under 2 minutes)

# Check istio-proxy logs
kubectl logs deploy/<app> -n <ns> -c istio-proxy --tail=20

# Run Istio analysis
istioctl analyze -n <ns>

# Check service port naming
kubectl get svc -n <ns> -o jsonpath='{range .items[*]}{.metadata.name}{": "}{range .spec.ports[*]}{.name}{" "}{end}{"\n"}{end}'

Causes and Fixes

1. Service Port Not Named with Protocol Prefix

[!WARNING] This is the most common cause of 503s after mesh enablement and the easiest to miss. Istio uses the Service port name prefix (e.g., http-, grpc-, tcp-) to determine the protocol. A port named web or api is treated as raw TCP, causing Envoy to misroute HTTP traffic. Audit all Service port names before enabling Istio on any namespace.

Most common cause. Istio requires port names to start with the protocol.

kubectl get svc <app> -n <ns> -o yaml | grep -A5 ports:
# Bad:  name: web
# Good: name: http-web

Fix: 

kubectl patch svc <app> -n <ns> --type=json \
  -p='[{"op":"replace","path":"/spec/ports/0/name","value":"http-web"}]'

2. Strict mTLS with Non-Mesh Clients

kubectl get peerauthentication -A

Fix: Use PERMISSIVE mode during migration. 

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: <ns>
spec:
  mtls:
    mode: PERMISSIVE

3. Init Container Ordering

If app init containers need network access, Istio's init container (iptables setup) must run first.

Fix: Add to pod spec: 

spec:
  annotations:
    proxy.istio.io/config: '{"holdApplicationUntilProxyStarts": true}'

4. Quick Rollback (If Can't Fix in 5 Minutes)

# Disable injection and restart
kubectl label namespace <ns> istio-injection-
kubectl rollout restart deployment -n <ns>

Verification

# Check requests reach the app
kubectl logs deploy/<app> -n <ns> -c <app-container> --tail=10

# Test with curl
kubectl exec deploy/<app> -c istio-proxy -- curl -s localhost:<app-port>/health

# Verify from outside
curl -s https://<host>/health

Prevention

  • Always test Istio enablement in staging first
  • Audit all Service port names for protocol prefixes before mesh rollout
  • Use canary approach: enable injection per-deployment, not per-namespace
  • Run istioctl analyze before and after mesh enablement

Wiki Navigation

Related Content