Skip to content

Consul

ā† Back to all decks

36 cards ā€” šŸŸ¢ 9 easy | šŸŸ” 15 medium | šŸ”“ 6 hard

šŸŸ¢ Easy (9)

1. What two primary problems does Consul solve that most dedicated tools handle separately?

Show answer Service discovery (finding services) and service mesh (securing service-to-service communication with mTLS). Consul also bundles a KV store, health checking, and ACLs in a single binary.

Remember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together.

2. How many server agents should you run in a Consul cluster, and why must this number be odd?

Show answer Run 3 or 5 servers. The number must be odd because Raft requires a quorum of (n/2)+1 to elect a leader. An odd number guarantees exactly one side of a network partition can achieve quorum; an even number can result in split-brain where neither side can elect a leader.

Remember: Consul agents run in server mode (participate in consensus, store data) or client mode (forward requests to servers, run health checks locally).

3. What is the difference between a Consul server agent and a client agent?

Show answer Server agents store all cluster state (service catalog, KV, ACL tokens) and participate in Raft consensus. Client agents run on every application node, register local services, run health checks, and forward queries to servers ā€” they do not participate in Raft.

Remember: Consul agents run in server mode (participate in consensus, store data) or client mode (forward requests to servers, run health checks locally).

4. What DNS name format does Consul use for service discovery, and on which port does Consul serve DNS by default?

Show answer Format: .service..consul ā€” for example web.service.dc1.consul. Consul serves DNS on port 8600 by default.

Remember: services register with Consul, then other services query Consul DNS (service.consul) or HTTP API to find them. No hardcoded IPs.

Example: dig @127.0.0.1 -p 8600 web.service.consul returns healthy instances of the 'web' service.

5. Name three health check types supported by Consul.

Show answer HTTP (GET to endpoint, passes on 2xx), TCP (TCP connect), Script (shell command, passes on exit 0), TTL (service heartbeats Consul), gRPC, and Alias (mirrors another check). Any three of these is correct.

Remember: Consul supports HTTP, TCP, script, TTL, Docker, and gRPC health checks. Unhealthy services are removed from DNS responses automatically.

6. What command reads all keys under a prefix in the Consul KV store?

Show answer consul kv get -recurse ā€” for example: consul kv get -recurse config/web/

Remember: Consul KV = distributed key-value store. Use for config, feature flags, leader election. consul kv put/get. Supports watches for change notification.

7. What is a Consul intention?

Show answer An intention is an access control rule that defines whether a source service is allowed or denied to communicate with a destination service. Intentions are enforced by the Envoy sidecar proxies in Consul Connect without requiring application code changes.

Remember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together.

8. What does consul snapshot save do, and what data does it include?

Show answer It creates a binary point-in-time backup of all Consul cluster state: KV store, ACL tokens, service catalog, sessions, prepared queries, and intentions. It should be taken before every upgrade.

Remember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together.

9. What is Serf, and how does it relate to Consul?

Show answer Serf is a standalone cluster membership and gossip library built by HashiCorp. Consul embeds Serf for its gossip layer ā€” used to detect node failures and propagate events across the cluster. Consul uses two Serf pools: LAN (within a datacenter) and WAN (between datacenters).

Remember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together.

šŸŸ” Medium (15)

1. What is the quorum requirement for a 5-server Consul cluster, and how many server failures can it tolerate?

Show answer Quorum requires 3 out of 5 servers ((5/2)+1 = 3). It can tolerate 2 server failures while still electing and maintaining a leader.

Remember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together.

2. What does consul operator raft list-peers show, and what metric indicates replication lag?

Show answer It shows each server's address, node ID, suffrage (Voter/Nonvoter), and index counters. Replication lag is indicated by a large difference between the leader's CommitIndex and a follower's LastApplied index.

Remember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together.

3. What is the deregister_critical_service_after health check field, and why is omitting it a problem?

Show answer It specifies how long a service can remain in critical state before Consul automatically deregisters it. Omitting it means crashed services remain in the catalog indefinitely as ghost entries, slowing catalog queries and confusing operators about what is actually running.

Remember: Consul supports HTTP, TCP, script, TTL, Docker, and gRPC health checks. Unhealthy services are removed from DNS responses automatically.

4. What is the ACL bootstrapping process, and what token does it produce?

Show answer Run consul acl bootstrap on a fresh cluster. It produces the bootstrap token ā€” the initial global management token with unrestricted access. This token must be stored securely (e.g., in Vault) immediately because it cannot be recovered if lost.

Remember: Consul ACLs use tokens with policies. Default deny is recommended. bootstrap the ACL system with consul acl bootstrap to get the initial management token.

5. How does Consul Connect's mTLS work, and what identity format does it use for certificates?

Show answer Consul's built-in CA issues short-lived (72-hour) leaf certificates to each service sidecar proxy. Certificates encode SPIFFE-compatible service identity in the format: spiffe:///ns//dc//svc/. Services authenticate each other via certificate, making network path irrelevant.

Remember: Consul Connect = built-in service mesh with mTLS and intention-based access control. Services get sidecar proxies (Envoy) for encrypted communication.

6. What are Consul's three consistency modes for reads, and when should you use consistent vs stale?

Show answer Default (leader reads with stale fallback, fast), consistent (linearizable via leader, never stale, most expensive), and stale (any server responds, up to ~50ms stale, fastest). Use consistent for lock coordination (KV sessions); use stale for service discovery where small staleness is acceptable.

Remember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together.

7. What is anti-entropy in Consul, and what do anti-entropy warning logs indicate?

Show answer Anti-entropy is the periodic process (every ~60s) where each client agent reconciles its local service registrations with the server catalog, re-registering any missing services. Warning logs about slow anti-entropy indicate the servers are overloaded ā€” high Raft commit latency or too many services per agent.

Remember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together.

8. What is the difference between WAN federation and mesh gateways in a multi-datacenter Consul setup?

Show answer WAN federation joins server agents from all datacenters into a shared WAN gossip pool and requires direct reachability between servers on ports 8302 and 8300. Mesh gateways are edge proxies that forward Connect traffic between DCs without requiring direct server-to-server connectivity ā€” preferred for multi-cloud or NAT-separated environments.

Remember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together.

9. What does the Consul Helm chart's connect-inject component do?

Show answer It installs a Kubernetes admission webhook that intercepts Pod creation. When a Pod has the annotation consul.hashicorp.com/connect-inject: "true", the webhook adds an Envoy sidecar container and an init container to configure it for Consul Connect service mesh.

Remember: Consul Connect = built-in service mesh with mTLS and intention-based access control. Services get sidecar proxies (Envoy) for encrypted communication.

10. What are Consul sessions used for, and what happens to a KV lock when its session is invalidated?

Show answer Sessions are the building block for distributed locking. They are associated with health checks; if those checks go critical, the session is invalidated. When a session holding a KV lock is invalidated, the lock is automatically released (or deleted, depending on the lock's behavior setting), preventing orphaned locks after a crash.

Remember: Consul KV = distributed key-value store. Use for config, feature flags, leader election. consul kv put/get. Supports watches for change notification.

11. What command checks whether a Consul intention would permit a connection from one service to another?

Show answer consul intention check ā€” for example: consul intention check web api. Returns "Allowed" or "Denied" based on the current intentions.

Remember: Consul Connect = built-in service mesh with mTLS and intention-based access control. Services get sidecar proxies (Envoy) for encrypted communication.

12. What is the SWIM protocol, and why does it scale better than simple heartbeat-based failure detection?

Show answer SWIM (Scalable Weakly-consistent Infection-style Membership) uses random probing and gossip dissemination to detect failures. Simple heartbeats require O(nĀ²) messages as the cluster grows. SWIM achieves O(log n) message complexity by spreading information through random neighbors, like a biological infection propagating through a population.

Remember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together.

13. What is a Consul prepared query, and what use case makes them valuable?

Show answer A prepared query is a saved, parameterized service discovery query stored in Consul. They support near-affinity routing (prefer local DC) and automatic failover to other datacenters. They are valuable for geo-aware failover without changing application code ā€” the application queries the same DNS name and Consul handles the routing.

Remember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together.

14. Why was Consul Template created, and what problem does it solve for legacy applications?

Show answer Consul Template was created to allow legacy applications (which read configuration from files) to benefit from Consul without code changes. It watches Consul KV and service catalog, regenerates text files (nginx configs, HAProxy upstreams, property files) using Go templates, and optionally triggers a reload command when content changes.

Remember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together.

15. What is the danger of leaving Consul's default intention behavior unchanged in production Connect deployments?

Show answer Without any intentions configured, Consul Connect's default is to allow all traffic between services ā€” even though mTLS is in use. This means any service can reach any other service, which is not zero-trust. Create a global deny-all intention (consul intention create -deny '*' '*') first, then explicitly allow required connections.

Remember: Consul Connect = built-in service mesh with mTLS and intention-based access control. Services get sidecar proxies (Envoy) for encrypted communication.

šŸ”“ Hard (6)

1. Describe the split-brain recovery procedure when a Consul cluster loses quorum.

Show answer 1) Count alive servers ā€” if below quorum, do not restart all at once. 2) Check Raft state on surviving servers with consul operator raft list-peers. 3) Remove dead peers with consul operator raft remove-peer -id=. 4) Last resort: stop all servers, wipe data/ directories, restart, then restore from snapshot with consul snapshot restore. 5) Post-mortem: ensure odd server count, check disk I/O and spot-interruption risks.

Remember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together.

2. Walk through the correct 4-step gossip encryption key rotation procedure in Consul.

Show answer Step 1: Generate a new key with consul keygen. Step 2: Install the new key on the cluster with consul keyring -install (cluster now accepts both keys). Step 3: Verify all agents have the new key via consul keyring -list. Step 4: Promote the new key as primary with consul keyring -use . Step 5: Only after confirming all agents have the new key, remove the old key with consul keyring -remove . Removing the old key before promotion causes a cluster partition.

3. What session TTL constraints exist in Consul, and how should a lock-holder keep a session alive across its TTL?

Show answer Session TTL must be between 10 seconds and 86400 seconds (1 day). A lock-holder should periodically call PUT /v1/session/renew/ before the TTL expires to renew the session. If the holder crashes without renewing, the TTL expires automatically and the lock is released ā€” preventing orphaned locks.

Remember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together.

4. At what scale has Consul been tested, and what architectural decision enables it to scale to 100,000+ nodes without growing the server cluster?

Show answer Consul has been benchmarked at 100,000+ nodes. The enabling architectural decision is the separation of server agents (Raft cluster, typically 3ā€“5 nodes, does not grow with clients) from client agents (gossip only, scale horizontally). The Raft cluster's write volume ā€” not its node count ā€” is the bottleneck; health check batching and anti_entropy_interval tuning allow operators to trade convergence speed for write throughput.

Remember: Consul integrates with Kubernetes (consul-k8s), Vault (for secrets), Nomad (for scheduling), and Terraform (for provisioning). The HashiCorp stack is designed to work together.

5. Describe the Consul ACL bootstrap reset procedure when the bootstrap token has been lost.

Show answer 1) Stop all Consul server agents. 2) Identify the reset index from the error message produced by a failed consul acl bootstrap attempt (the error includes the current reset index). 3) Run consul acl bootstrap -reset-index= to re-enable bootstrapping. 4) Restart server agents. 5) Run consul acl bootstrap to generate a new bootstrap token. 6) Immediately store the new token in Vault and create scoped agent/service tokens.

Remember: Consul ACLs use tokens with policies. Default deny is recommended. bootstrap the ACL system with consul acl bootstrap to get the initial management token.

6. Explain the 4-step process for debugging a Connect-enabled service that cannot reach its upstream.

Show answer 1) Check intentions: consul intention check ā€” an implicit deny or explicit deny intention blocks traffic.
2) Verify both sidecar proxies are running (kubectl get pods or ps aux | grep envoy).
3) Inspect Envoy metrics on the source sidecar's admin interface (curl localhost:19000/stats | grep cx_none) for upstream connection failures.
4) Confirm the destination service is registered and passing health checks (consul health service -passing). Certificate issues show up as TLS handshake errors in Envoy stats.