--- tags: - k8s - l1 - flashcard-deck - consul --- [Portal](../../../../library/portal/index.md) | **Level:** [L1: Foundations](../../../../library/portal/levels.md) | **Topics:** [HashiCorp Consul](../../../../library/portal/topics.md) | **Domain:** Kubernetes id category difficulty tags question answer source_path consul/c3a2631ef49d consul easy consul, service-discovery What two primary problems does Consul solve that most dedicated tools handle separately? Service discovery (finding services) and service mesh (securing service-to-service communication with mTLS). Consul also bundles a KV store, health checking, and ACLs in a single binary.\n\nRemember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together. training/library/topics/consul/primer.md consul/2661045fe48c consul easy consul, architecture How many server agents should you run in a Consul cluster, and why must this number be odd? Run 3 or 5 servers. The number must be odd because Raft requires a quorum of (n/2)+1 to elect a leader. An odd number guarantees exactly one side of a network partition can achieve quorum; an even number can result in split-brain where neither side can elect a leader.\n\nRemember: Consul agents run in server mode (participate in consensus, store data) or client mode (forward requests to servers, run health checks locally). training/library/topics/consul/primer.md consul/0f01c50c4293 consul easy consul, architecture What is the difference between a Consul server agent and a client agent? Server agents store all cluster state (service catalog, KV, ACL tokens) and participate in Raft consensus. Client agents run on every application node, register local services, run health checks, and forward queries to servers — they do not participate in Raft.\n\nRemember: Consul agents run in server mode (participate in consensus, store data) or client mode (forward requests to servers, run health checks locally). training/library/topics/consul/primer.md consul/5118fbe1f443 consul easy consul, service-discovery What DNS name format does Consul use for service discovery, and on which port does Consul serve DNS by default? Format: .service..consul — for example web.service.dc1.consul. Consul serves DNS on port 8600 by default.\n\nRemember: services register with Consul, then other services query Consul DNS (service.consul) or HTTP API to find them. No hardcoded IPs.\n\nExample: dig @127.0.0.1 -p 8600 web.service.consul returns healthy instances of the 'web' service. training/library/topics/consul/primer.md consul/c0be11d6ba66 consul easy consul, health-checks Name three health check types supported by Consul. HTTP (GET to endpoint, passes on 2xx), TCP (TCP connect), Script (shell command, passes on exit 0), TTL (service heartbeats Consul), gRPC, and Alias (mirrors another check). Any three of these is correct.\n\nRemember: Consul supports HTTP, TCP, script, TTL, Docker, and gRPC health checks. Unhealthy services are removed from DNS responses automatically. training/library/topics/consul/primer.md consul/d8107d6face7 consul easy consul, kv What command reads all keys under a prefix in the Consul KV store? consul kv get -recurse — for example: consul kv get -recurse config/web/\n\nRemember: Consul KV = distributed key-value store. Use for config, feature flags, leader election. consul kv put/get. Supports watches for change notification. training/library/topics/consul/street_ops.md consul/34ee7498ab19 consul easy consul, service-mesh What is a Consul intention? An intention is an access control rule that defines whether a source service is allowed or denied to communicate with a destination service. Intentions are enforced by the Envoy sidecar proxies in Consul Connect without requiring application code changes.\n\nRemember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together. training/library/topics/consul/primer.md consul/200ac54534c8 consul easy consul, snapshots What does consul snapshot save do, and what data does it include? It creates a binary point-in-time backup of all Consul cluster state: KV store, ACL tokens, service catalog, sessions, prepared queries, and intentions. It should be taken before every upgrade.\n\nRemember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together. training/library/topics/consul/street_ops.md consul/d5fe1da59866 consul easy consul, gossip What is Serf, and how does it relate to Consul? Serf is a standalone cluster membership and gossip library built by HashiCorp. Consul embeds Serf for its gossip layer — used to detect node failures and propagate events across the cluster. Consul uses two Serf pools: LAN (within a datacenter) and WAN (between datacenters).\n\nRemember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together. training/library/topics/consul/trivia.md consul/944f9b00e966 consul medium consul, raft What is the quorum requirement for a 5-server Consul cluster, and how many server failures can it tolerate? Quorum requires 3 out of 5 servers ((5/2)+1 = 3). It can tolerate 2 server failures while still electing and maintaining a leader.\n\nRemember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together. training/library/topics/consul/primer.md consul/5be8613476c4 consul medium consul, raft What does consul operator raft list-peers show, and what metric indicates replication lag? It shows each server's address, node ID, suffrage (Voter/Nonvoter), and index counters. Replication lag is indicated by a large difference between the leader's CommitIndex and a follower's LastApplied index.\n\nRemember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together. training/library/topics/consul/street_ops.md consul/c797b35eb5ea consul medium consul, service-discovery What is the deregister_critical_service_after health check field, and why is omitting it a problem? It specifies how long a service can remain in critical state before Consul automatically deregisters it. Omitting it means crashed services remain in the catalog indefinitely as ghost entries, slowing catalog queries and confusing operators about what is actually running.\n\nRemember: Consul supports HTTP, TCP, script, TTL, Docker, and gRPC health checks. Unhealthy services are removed from DNS responses automatically. training/library/topics/consul/footguns.md consul/a648016007d9 consul medium consul, acls What is the ACL bootstrapping process, and what token does it produce? Run consul acl bootstrap on a fresh cluster. It produces the bootstrap token — the initial global management token with unrestricted access. This token must be stored securely (e.g., in Vault) immediately because it cannot be recovered if lost.\n\nRemember: Consul ACLs use tokens with policies. Default deny is recommended. bootstrap the ACL system with consul acl bootstrap to get the initial management token. training/library/topics/consul/primer.md consul/6208196c4fc3 consul medium consul, service-mesh How does Consul Connect's mTLS work, and what identity format does it use for certificates? Consul's built-in CA issues short-lived (72-hour) leaf certificates to each service sidecar proxy. Certificates encode SPIFFE-compatible service identity in the format: spiffe:///ns//dc//svc/. Services authenticate each other via certificate, making network path irrelevant.\n\nRemember: Consul Connect = built-in service mesh with mTLS and intention-based access control. Services get sidecar proxies (Envoy) for encrypted communication. training/library/topics/consul/primer.md consul/74f258c6e9dc consul medium consul, consistency What are Consul's three consistency modes for reads, and when should you use consistent vs stale? Default (leader reads with stale fallback, fast), consistent (linearizable via leader, never stale, most expensive), and stale (any server responds, up to ~50ms stale, fastest). Use consistent for lock coordination (KV sessions); use stale for service discovery where small staleness is acceptable.\n\nRemember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together. training/library/topics/consul/primer.md consul/552de6f2cce6 consul medium consul, anti-entropy What is anti-entropy in Consul, and what do anti-entropy warning logs indicate? Anti-entropy is the periodic process (every ~60s) where each client agent reconciles its local service registrations with the server catalog, re-registering any missing services. Warning logs about slow anti-entropy indicate the servers are overloaded — high Raft commit latency or too many services per agent.\n\nRemember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together. training/library/topics/consul/primer.md consul/710ba1093510 consul medium consul, multi-dc What is the difference between WAN federation and mesh gateways in a multi-datacenter Consul setup? WAN federation joins server agents from all datacenters into a shared WAN gossip pool and requires direct reachability between servers on ports 8302 and 8300. Mesh gateways are edge proxies that forward Connect traffic between DCs without requiring direct server-to-server connectivity — preferred for multi-cloud or NAT-separated environments.\n\nRemember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together. training/library/topics/consul/primer.md consul/5a71b51778e2 consul medium consul, kubernetes What does the Consul Helm chart's connect-inject component do? It installs a Kubernetes admission webhook that intercepts Pod creation. When a Pod has the annotation consul.hashicorp.com/connect-inject: "true", the webhook adds an Envoy sidecar container and an init container to configure it for Consul Connect service mesh.\n\nRemember: Consul Connect = built-in service mesh with mTLS and intention-based access control. Services get sidecar proxies (Envoy) for encrypted communication. training/library/topics/consul/primer.md consul/acd83351b6e5 consul medium consul, kv What are Consul sessions used for, and what happens to a KV lock when its session is invalidated? Sessions are the building block for distributed locking. They are associated with health checks; if those checks go critical, the session is invalidated. When a session holding a KV lock is invalidated, the lock is automatically released (or deleted, depending on the lock's behavior setting), preventing orphaned locks after a crash.\n\nRemember: Consul KV = distributed key-value store. Use for config, feature flags, leader election. consul kv put/get. Supports watches for change notification. training/library/topics/consul/primer.md consul/5c6f0640292d consul medium consul, service-mesh What command checks whether a Consul intention would permit a connection from one service to another? consul intention check — for example: consul intention check web api. Returns "Allowed" or "Denied" based on the current intentions.\n\nRemember: Consul Connect = built-in service mesh with mTLS and intention-based access control. Services get sidecar proxies (Envoy) for encrypted communication. training/library/topics/consul/street_ops.md consul/841aadcebacc consul medium consul, gossip What is the SWIM protocol, and why does it scale better than simple heartbeat-based failure detection? SWIM (Scalable Weakly-consistent Infection-style Membership) uses random probing and gossip dissemination to detect failures. Simple heartbeats require O(n²) messages as the cluster grows. SWIM achieves O(log n) message complexity by spreading information through random neighbors, like a biological infection propagating through a population.\n\nRemember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together. training/library/topics/consul/trivia.md consul/520bc23ed36c consul medium consul, prepared-queries What is a Consul prepared query, and what use case makes them valuable? A prepared query is a saved, parameterized service discovery query stored in Consul. They support near-affinity routing (prefer local DC) and automatic failover to other datacenters. They are valuable for geo-aware failover without changing application code — the application queries the same DNS name and Consul handles the routing.\n\nRemember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together. training/library/topics/consul/street_ops.md consul/7476e6fdba00 consul medium consul, history Why was Consul Template created, and what problem does it solve for legacy applications? Consul Template was created to allow legacy applications (which read configuration from files) to benefit from Consul without code changes. It watches Consul KV and service catalog, regenerates text files (nginx configs, HAProxy upstreams, property files) using Go templates, and optionally triggers a reload command when content changes.\n\nRemember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together. training/library/topics/consul/trivia.md consul/bb669def8549 consul medium consul, acls What is the danger of leaving Consul's default intention behavior unchanged in production Connect deployments? Without any intentions configured, Consul Connect's default is to allow all traffic between services — even though mTLS is in use. This means any service can reach any other service, which is not zero-trust. Create a global deny-all intention (consul intention create -deny '*' '*') first, then explicitly allow required connections.\n\nRemember: Consul Connect = built-in service mesh with mTLS and intention-based access control. Services get sidecar proxies (Envoy) for encrypted communication. training/library/topics/consul/footguns.md consul/0f5081a33880 consul hard consul, raft Describe the split-brain recovery procedure when a Consul cluster loses quorum. 1) Count alive servers — if below quorum, do not restart all at once. 2) Check Raft state on surviving servers with consul operator raft list-peers. 3) Remove dead peers with consul operator raft remove-peer -id=. 4) Last resort: stop all servers, wipe data/ directories, restart, then restore from snapshot with consul snapshot restore. 5) Post-mortem: ensure odd server count, check disk I/O and spot-interruption risks.\n\nRemember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together. training/library/topics/consul/street_ops.md consul/984c48eed25e consul hard consul, gossip Walk through the correct 4-step gossip encryption key rotation procedure in Consul. Step 1: Generate a new key with consul keygen. Step 2: Install the new key on the cluster with consul keyring -install (cluster now accepts both keys). Step 3: Verify all agents have the new key via consul keyring -list. Step 4: Promote the new key as primary with consul keyring -use . Step 5: Only after confirming all agents have the new key, remove the old key with consul keyring -remove . Removing the old key before promotion causes a cluster partition. training/library/topics/consul/footguns.md consul/b80970feeb10 consul hard consul, kv What session TTL constraints exist in Consul, and how should a lock-holder keep a session alive across its TTL? Session TTL must be between 10 seconds and 86400 seconds (1 day). A lock-holder should periodically call PUT /v1/session/renew/ before the TTL expires to renew the session. If the holder crashes without renewing, the TTL expires automatically and the lock is released — preventing orphaned locks.\n\nRemember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together. training/library/topics/consul/footguns.md consul/5d07310768ff consul hard consul, architecture At what scale has Consul been tested, and what architectural decision enables it to scale to 100,000+ nodes without growing the server cluster? Consul has been benchmarked at 100,000+ nodes. The enabling architectural decision is the separation of server agents (Raft cluster, typically 3–5 nodes, does not grow with clients) from client agents (gossip only, scale horizontally). The Raft cluster's write volume — not its node count — is the bottleneck; health check batching and anti_entropy_interval tuning allow operators to trade convergence speed for write throughput.\n\nRemember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together. training/library/topics/consul/trivia.md consul/319c7f5cb89f consul hard consul, acls Describe the Consul ACL bootstrap reset procedure when the bootstrap token has been lost. 1) Stop all Consul server agents. 2) Identify the reset index from the error message produced by a failed consul acl bootstrap attempt (the error includes the current reset index). 3) Run consul acl bootstrap -reset-index= to re-enable bootstrapping. 4) Restart server agents. 5) Run consul acl bootstrap to generate a new bootstrap token. 6) Immediately store the new token in Vault and create scoped agent/service tokens.\n\nRemember: Consul ACLs use tokens with policies. Default deny is recommended. bootstrap the ACL system with consul acl bootstrap to get the initial management token. training/library/topics/consul/street_ops.md consul/0a394f9e4e87 consul hard consul, service-mesh Explain the 4-step process for debugging a Connect-enabled service that cannot reach its upstream. 1) Check intentions: consul intention check — an implicit deny or explicit deny intention blocks traffic. \n2) Verify both sidecar proxies are running (kubectl get pods or ps aux | grep envoy). \n3) Inspect Envoy metrics on the source sidecar's admin interface (curl localhost:19000/stats | grep cx_none) for upstream connection failures. \n4) Confirm the destination service is registered and passing health checks (consul health service -passing). Certificate issues show up as TLS handshake errors in Envoy stats. training/library/topics/consul/street_ops.md --- ## Wiki Navigation ### Related Content - [HashiCorp Consul](../../../../library/topics/consul/index.md) (Topic Pack, L2) — HashiCorp Consul