--- tags: - k8s - l1 - flashcard-deck - envoy --- [Portal](../../../../library/portal/index.md) | **Level:** [L1: Foundations](../../../../library/portal/levels.md) | **Topics:** [Envoy Proxy](../../../../library/portal/topics.md) | **Domain:** Kubernetes id category difficulty tags question answer source_path envoy/3a1f8c2d4e5b envoy easy envoy, architecture, listeners What is an Envoy listener? A listener defines the address and port where Envoy accepts incoming connections. Each listener has one or more filter chains that process the connection.\n\nExample: listener on 0.0.0.0:8080 with HTTP connection manager routes to clusters based on host/path rules. training/library/topics/envoy/primer.md envoy/7b2e9d1c6f0a envoy easy envoy, architecture, clusters What is an Envoy cluster? A cluster is a logical group of upstream hosts (endpoints). It holds load-balancing policy, circuit breaker thresholds, health check configuration, and TLS settings for reaching a set of upstream services.\n\nExample: cluster "backend-v2" with round-robin LB, 5s timeout, outlier detection ejecting after 5 consecutive 5xx. training/library/topics/envoy/primer.md envoy/4c3d0e8f1a2b envoy easy envoy, architecture, routes What does "first match wins" mean in Envoy routing? Envoy evaluates route rules in order and selects the first rule that matches the request. More-specific routes (exact match, longer prefix) must be placed before broader routes, or they will never be reached.\n\nGotcha: a catch-all prefix: '/' route placed first will match everything — more specific routes below it are dead code. Order matters.\n\nAnalogy: like a switch-case without fall-through — the first match wins, and everything after it is ignored for that request. training/library/topics/envoy/primer.md envoy/9e4f5a7b8c1d envoy easy envoy, xds, apis What does xDS stand for in the context of Envoy? xDS stands for "x Discovery Service" — a family of APIs (LDS, RDS, CDS, EDS, SDS, ADS) that a management plane uses to dynamically push configuration to Envoy without restarts.\n\nRemember: LRCESA — Listeners, Routes, Clusters, Endpoints, Secrets, Aggregated. Six xDS APIs for dynamic config. training/library/topics/envoy/primer.md envoy/2f6b0c9d3e4a envoy easy envoy, admin, interface What is the Envoy admin interface and what port does it use by default? The admin interface is a local HTTP endpoint (default port 9901, or 15000 in Istio sidecars) that exposes config_dump, stats, logging controls, and cluster health. It must never be exposed externally.\n\nGotcha: admin port exposes config_dump (secrets visible!), log controls, drain commands. Never expose beyond localhost. training/library/topics/envoy/street_ops.md envoy/8a5c1d2e7f3b envoy easy envoy, response-flags, 503 What does the Envoy response flag UF mean? UF means "upstream connection failure" — Envoy could not establish or maintain a TCP connection to the upstream host. Common causes: upstream pod crashed, network policy blocking traffic, or wrong port.\n\nRemember: UF = Upstream Failure. Check: pod running? Port correct? Network policy blocking? training/library/topics/envoy/street_ops.md envoy/1b7d3f4a9e6c envoy easy envoy, response-flags, circuit-breaker What does the Envoy response flag UO mean? UO means "upstream overflow" — the circuit breaker threshold was exceeded (max_connections, max_pending_requests, or max_requests) and Envoy returned 503 rather than queue another request.\n\nRemember: UO = Upstream Overflow. Circuit breaker tripped. Check upstream_rq_pending_overflow counter. training/library/topics/envoy/street_ops.md envoy/5d2a8b0c4e9f envoy easy envoy, response-flags, routing What does the Envoy response flag NR mean? NR means "no route" — Envoy received a request but found no matching route in the route table. Common causes: missing route configuration, wrong Host header, or VirtualService misconfiguration in Istio.\n\nRemember: NR = No Route. Check Host header, VirtualService hosts, route prefixes. training/library/topics/envoy/street_ops.md envoy/6e1f9c7b2d0a envoy easy envoy, load-balancing What load-balancing policy should you use when you want session stickiness based on a request header? Ring hash (or Maglev) load balancing. Both use consistent hashing on a specified header (e.g., a session cookie or user ID) to route the same caller consistently to the same upstream host.\n\nExample: hash on x-user-id header to route same user to same backend. Good for caching and websockets. training/library/topics/envoy/primer.md envoy/0c4b6d8f1e3a envoy easy envoy, hot-restart What is Envoy hot restart? Hot restart allows a new Envoy process to take over the listening sockets from the old process without dropping active connections. The old process drains in-flight requests while the new process handles new connections.\n\nUnder the hood: the old Envoy passes file descriptors to the new process via Unix domain sockets using SCM_RIGHTS. The kernel allows two processes to share the same listening socket.\n\nRemember: hot restart enables config changes and binary upgrades without connection drops — essential for service mesh proxies handling thousands of active connections. training/library/topics/envoy/primer.md envoy/3d9e5a2f7b1c envoy medium envoy, xds, ads Why is ADS (Aggregated Discovery Service) safer than using separate xDS streams for CDS, EDS, and LDS? With separate streams, CDS, EDS, and LDS updates can arrive out of order: a new cluster might appear before its endpoints, causing brief 503s. ADS delivers all resource types on a single ordered stream, so Envoy applies updates atomically and consistently. training/library/topics/envoy/primer.md envoy/7f0b4c8e2a6d envoy medium envoy, circuit-breaker, tuning Which Envoy stat should you alert on to detect circuit breaker trips in production? upstream_rq_pending_overflow — it increments every time a request is rejected because max_pending_requests was exceeded. Any non-zero value in production indicates the circuit breaker is actively shedding load.\n\nGotcha: A zero value is normal at startup. Any non-zero in production means you are actively shedding user traffic.\n\nDebug clue: Correlate with upstream_rq_active to see if backends are saturated or if thresholds are too low. training/library/topics/envoy/street_ops.md envoy/2a1d6f3b9c5e envoy medium envoy, circuit-breaker, defaults What are Envoy's default circuit breaker thresholds and why are they often wrong? Defaults are max_connections: 1024, max_pending_requests: 1024, max_requests: 1024. They are often wrong because they are per-sidecar — in a mesh with many callers fanning out to the same service, the aggregate load quickly exceeds these limits, causing spurious 503 UO errors. training/library/topics/envoy/footguns.md envoy/8c5e7d0f4b2a envoy medium envoy, retries, timeouts What is the danger of setting num_retries without also setting per_try_timeout? Without per_try_timeout, each retry attempt inherits the full route timeout. Under a slow upstream, multiple retries each run to the full timeout, multiplying the load on the upstream by the retry count and causing retry storms. training/library/topics/envoy/footguns.md envoy/4b3a9c1e8d6f envoy medium envoy, outlier-detection What does Envoy outlier detection do and how does it differ from active health checks? Outlier detection passively monitors upstream hosts for failure patterns (consecutive 5xx, gateway errors, high latency) and ejects misbehaving hosts from the load-balancing pool. Active health checks probe endpoints on a schedule. Outlier detection reacts to real traffic; health checks detect failures even with no traffic. training/library/topics/envoy/primer.md envoy/6f2c0d5a7e1b envoy medium envoy, wasm, extensibility What is the Proxy-WASM ABI and why does it matter? Proxy-WASM is a vendor-neutral WebAssembly ABI specification for proxy plugins. A WASM filter written to the Proxy-WASM ABI can run on Envoy, NGINX (via ngx_wasm_module), and other compatible proxies — enabling portable filter code across implementations. training/library/topics/envoy/primer.md envoy/1e8b4f6d2c0a envoy medium envoy, grpc, http2 How does Envoy handle gRPC-Web requests from browsers that cannot use HTTP/2 trailers? Envoy's gRPC-Web filter receives an HTTP/1.1 (or trailerless HTTP/2) gRPC-Web request from the browser, translates it to a standard gRPC request to the upstream, and translates the response (including trailers encoding the gRPC status) back to gRPC-Web format. training/library/topics/envoy/primer.md envoy/9d7f3b5c1e2a envoy medium envoy, sds, tls Why is SDS (Secret Discovery Service) preferred over file-based TLS certificates in Envoy? SDS allows a management plane to push new certificates to Envoy over an existing gRPC stream. Envoy uses the new cert for new TLS sessions while completing existing sessions with the old cert — zero-downtime rotation. File-based rotation requires a config reload or restart, creating a window where TLS handshakes can fail. training/library/topics/envoy/footguns.md envoy/5a0c9e4f8b7d envoy medium envoy, access-logs, response-flags What does the Envoy response flag URX mean? URX means "upstream retry exhausted" — Envoy attempted retries up to the configured num_retries limit and all attempts failed. The final response returned to the client is the last upstream failure.\n\nRemember: URX = Upstream Retry eXhausted. All retries failed. Check num_retries and per_try_timeout. training/library/topics/envoy/street_ops.md envoy/0e6d1b8a3f4c envoy medium envoy, dns, cluster What happens if Envoy's DNS refresh rate is longer than your upstream service's DNS TTL? Envoy caches stale DNS entries and routes requests to old IP addresses after deployments. Connections to the stale IPs fail (UF response flag). The failure lasts until the DNS cache expires at the configured refresh interval. training/library/topics/envoy/footguns.md envoy/3b5f7e9a1c2d envoy medium envoy, zone-aware, load-balancing What does zone-aware load balancing do in Envoy? Zone-aware routing biases traffic toward upstream endpoints in the same availability zone as the Envoy instance, reducing cross-AZ latency and data transfer costs. It falls back to cross-zone routing when local zone capacity is insufficient to serve the load. training/library/topics/envoy/primer.md envoy/7c1d4b2f6e0a envoy medium envoy, traffic-shifting, canary How does Envoy implement traffic splitting (e.g., 90% to v1, 10% to v2)? Routes support weighted cluster assignments. The route config maps a single route match to multiple clusters each with a weight. Envoy distributes traffic proportionally — no DNS change or additional load balancer required. training/library/topics/envoy/primer.md envoy/2e9a0d3c5f8b envoy medium envoy, drain, kubernetes Why do Envoy containers in Kubernetes need a preStop lifecycle hook? Without a preStop hook, Kubernetes sends SIGTERM and kills the container immediately. In-flight requests receive TCP RSTs instead of clean HTTP responses. A preStop sleep gives kube-proxy time to drain iptables rules and allows Envoy to stop accepting new connections gracefully before shutdown. training/library/topics/envoy/footguns.md envoy/4d8f2b6a9e1c envoy medium envoy, stats, observability What Envoy stat tracks how many upstream requests are currently in flight to a cluster? cluster..upstream_rq_active — a gauge showing the current number of active (in-flight) requests to that cluster. Correlate with max_requests circuit breaker threshold to detect saturation.\n\nDebug clue: If upstream_rq_active approaches max_requests, circuit breaker trips are imminent. Alert at 80% of threshold.\n\nExample: `curl localhost:15000/stats | grep upstream_rq_active` on a sidecar to check live values. training/library/topics/envoy/street_ops.md envoy/6a3c5d7f0b2e envoy hard envoy, xds, ordering What ordering problem does ADS solve that per-resource xDS streams cannot? With separate CDS, EDS, and LDS streams, a race condition exists: LDS may deliver a listener referencing a new cluster before CDS delivers that cluster's definition, or CDS may deliver a cluster before EDS delivers its endpoints. Envoy processes updates as they arrive; without ordering guarantees, intermediate states can produce NR or UF errors. ADS delivers all resource types on one ordered stream and Envoy applies them as a batch, eliminating the race. training/library/topics/envoy/primer.md envoy/9f1e4c8a2b7d envoy hard envoy, outlier-detection, pool-depletion How can outlier detection accidentally eject an entire upstream cluster and what prevents this? If max_ejection_percent defaults to 100%, every host in a cluster can be ejected. During a rolling restart when multiple hosts briefly return 5xx, consecutive_5xx ejections cascade: ejected hosts remove load from the pool, remaining hosts saturate and also return 5xx, and are ejected in turn. The entire pool empties. Prevention: set max_ejection_percent to 50 (never eject more than half), raise consecutive_5xx to at least 10, and set enforcing_consecutive_5xx to 0 during initial rollout. training/library/topics/envoy/footguns.md envoy/5b0d9f3e7a1c envoy hard envoy, filter-chains, wasm What happens to traffic when a WASM filter VM crashes inside Envoy, and how does fail_open vs fail_close affect this? When a WASM VM crashes, the filter is aborted for that request. With fail_open: true, Envoy passes the request through without filtering (degraded but functional). With fail_close: false (the default), Envoy returns a 500 to the caller. The Envoy process itself does not crash — WASM sandboxing isolates the VM fault. Monitor wasm.runtime_errors to detect crash loops. training/library/topics/envoy/footguns.md envoy/8e2a6c4f1d9b envoy hard envoy, hot-restart, socket-passing What is the mechanism by which Envoy hot restart passes listening sockets from the old to the new process? The old Envoy process acts as the "hot restart parent." The new process connects to the parent via a Unix domain socket and sends a request for each listening socket's file descriptor. The kernel passes the open file descriptors via SCM_RIGHTS ancillary data in a sendmsg call. The new process then binds to those fds and begins accepting connections, while the parent enters draining mode and closes its accept loop. training/library/topics/envoy/primer.md envoy/1c7b3e5d8f2a envoy hard envoy, config-dump, large-mesh What is a practical strategy for extracting a specific cluster's configuration from a large Envoy config_dump without loading the entire blob into memory? Stream the config_dump through a pipeline: curl -s localhost:15000/config_dump | python3 -c "import sys,json; [print(json.dumps(c,indent=2)) for c in json.load(sys.stdin)['configs'] if 'ClustersConfigDump' in c.get('@type','')]" | grep -A20 '"name": "my-cluster"'. Alternatively use the /clusters?format=json endpoint which returns only cluster state and is much smaller than the full config_dump. training/library/topics/envoy/street_ops.md envoy/4f9d1a6b0e3c envoy hard envoy, retry-storms, rate-limiting What combination of Envoy features prevents retry storms while still providing retry protection? Three controls together: (1) per_try_timeout set to route_timeout / (num_retries + 1) to bound retry duration; (2) retry_priority or retry_host_predicate: previous_hosts to avoid retrying the same failed host; (3) max_retries circuit breaker threshold to cap total concurrent retries across all requests to a cluster. Without (3), the global retry concurrency is unbounded — thousands of requests each retrying 3x triples upstream load. training/library/topics/envoy/footguns.md --- ## Wiki Navigation ### Related Content - [Envoy Proxy](../../../../library/topics/envoy/index.md) (Topic Pack, L2) — Envoy Proxy