Skip to content

🎓 Mission Debrief: Lost Connection - Labels & Selectors

What Happened

Your Service had a selector for app: frontend, but your Pod had the label app: backend. Since the labels didn't match, the Service couldn't find the Pod and had no endpoints. Without endpoints, traffic sent to the Service had nowhere to go.

How Kubernetes Behaved

Services don't directly manage Pods. Instead, they use label selectors to dynamically discover which Pods should receive traffic.

The process: 1. Service created with selector: app: frontend, tier: api 2. Service controller watches all Pods 3. Finds Pods matching ALL selector labels 4. Creates Endpoints object with matching Pod IPs 5. Routes traffic to those endpoints

Your Service was looking for app=frontend, but your Pod was labeled app=backend, so step 3 failed—no matches found.

The Correct Mental Model

Labels are key-value pairs attached to Kubernetes objects:

metadata:
  labels:
    app: backend
    tier: api
    environment: prod
    version: v2

Selectors are queries that filter objects by labels:

selector:
  app: backend    # Matches pods with app=backend
  tier: api       # AND tier=api (both must match)

How Services Use Selectors:

Service selector: {app: backend, tier: api}
                        ↓
Looks for pods with matching labels
                        ↓
Pod 1: {app: backend, tier: api} ✅ Match!
Pod 2: {app: frontend, tier: api} ❌ app doesn't match
Pod 3: {app: backend, tier: db} ❌ tier doesn't match
Pod 4: {app: backend, tier: api, env: prod} ✅ Match! (extra labels OK)
                        ↓
Creates Endpoints with IPs of Pod 1 and Pod 4

Important: All labels in the selector must match, but pods can have extra labels.

Real-World Incident Example

Company: Fintech startup processing $10M daily Impact: 3-hour payment processing outage Cost: $125K in SLA violations + 400 customer support tickets

What happened: During a routine deployment, a DevOps engineer updated the payment service deployment and accidentally changed the pod label from app: payment-processor to app: payment-service.

The Service still had selector app: payment-processor. Result: instant disconnect. All payment pods became unreachable. The load balancer kept the Service alive, but with 0 endpoints.

Why it took 3 hours: - Service remained "up" (no alerts fired) - Health checks targeted the Service, which existed (but had no backends) - Logs showed "connection refused" but team thought it was network issues - Finally discovered via: kubectl get endpoints payment-service showed empty - Checked: kubectl get pods --selector=app=payment-processor showed 0 pods

The fix: Changed the Service selector to match the new label. Endpoints appeared instantly.

Lesson learned: 1. Never change pod labels without updating corresponding services 2. Monitor endpoint counts (alert if endpoints = 0) 3. Use Horizontal Pod Autoscaler labels for consistency 4. Consider using consistent naming conventions

Commands You Mastered

# Check service and its selector
kubectl get service <name> -n <namespace>
kubectl describe service <name> -n <namespace>
kubectl get service <name> -n <namespace> -o yaml | grep -A 5 selector

# Check endpoints (the IPs service routes to)
kubectl get endpoints <name> -n <namespace>
kubectl describe endpoints <name> -n <namespace>

# View pod labels
kubectl get pods --show-labels -n <namespace>
kubectl get pod <name> -n <namespace> --show-labels

# Find pods matching a selector
kubectl get pods --selector=app=backend -n <namespace>
kubectl get pods -l app=backend,tier=api -n <namespace>

# Add/modify labels on running pods
kubectl label pod <name> app=frontend -n <namespace>
kubectl label pod <name> app=backend --overwrite -n <namespace>

# Delete resources
kubectl delete -f <file>.yaml

Label Best Practices

  1. Use recommended labels (from Kubernetes docs): 

    app.kubernetes.io/name: myapp
app.kubernetes.io/instance: myapp-prod
app.kubernetes.io/version: 1.2.3
app.kubernetes.io/component: backend
app.kubernetes.io/part-of: payment-system

  2. Keep selectors simple: 1-3 labels max

  3. Don't change labels on running pods if services depend on them
  4. Use consistent values: backend not Backend or back-end
  5. Document your labels: Maintain a label glossary for your org

Understanding Endpoints

Endpoints are the bridge between Services and Pods:

kubectl get endpoints <service-name>

NAME              ENDPOINTS
backend-service   10.244.0.5:5678,10.244.0.6:5678

                  ↑ These are Pod IPs matching the selector

If ENDPOINTS is empty = service has no backends = traffic fails.

Common reasons for empty endpoints: - ❌ Label mismatch (what you just fixed!) - ❌ No pods exist - ❌ Pods exist but not Ready - ❌ Port mismatch in Service - ❌ Pods in different namespace

What's Next?

Labels are used everywhere in Kubernetes: - Services selecting Pods (you just learned this!) - Deployments managing ReplicaSets - NetworkPolicies filtering traffic - Node selectors for scheduling - Volume claims selecting storage

Next challenge: You'll debug a container port mismatch!