Level 27: Cross-namespace Service Communication - Mission Debrief

Mission Overview

Objective: Fix a frontend application that couldn't communicate with a backend service in a different namespace by using the proper DNS FQDN format.

XP Awarded: 250 XP Difficulty: Intermediate Concepts: Cross-namespace Communication, Kubernetes DNS, FQDN, Service Discovery

---

What You Encountered

You deployed a frontend application in the k8squest namespace that needed to call a backend API in the backend-ns namespace. Both services were running perfectly, endpoints existed, and network connectivity was fine. Yet every API call from the frontend failed with DNS resolution errors.

The culprit? The frontend was using a short service name (api-service) that only resolves within the same namespace.

The Broken Configuration:

# Frontend pod in k8squest namespace
command: ['sh', '-c', 'wget -q -O- http://api-service']

The Problem: - Frontend tries to resolve: api-service - Kubernetes DNS searches: api-service.k8squest.svc.cluster.local - But service is actually in: backend-ns namespace - Result: DNS resolution failed - "could not resolve host"

---

The Root Cause: DNS Namespace Scoping

Understanding Kubernetes DNS

Kubernetes runs an internal DNS server (CoreDNS) that provides service discovery. Every Service gets a DNS record, but the DNS name format depends on whether you're accessing a service in the same namespace or a different namespace.

DNS Architecture:

┌─────────────────────────────────────────────────┐
│  CoreDNS (kube-system namespace)                │
│  DNS Server: 10.96.0.10                         │
│                                                  │
│  DNS Records:                                    │
│  • api-service.backend-ns.svc.cluster.local     │
│    → 10.96.15.23                                │
│  • frontend-service.k8squest.svc.cluster.local  │
│    → 10.96.20.45                                │
└─────────────────────────────────────────────────┘
                      ▲
                      │ DNS Query
                      │
┌─────────────────────┴─────────────────────┐
│  Pod (in k8squest namespace)              │
│  /etc/resolv.conf:                        │
│    nameserver 10.96.0.10                  │
│    search k8squest.svc.cluster.local      │
│           svc.cluster.local               │
│           cluster.local                   │
└───────────────────────────────────────────┘

DNS Search Domains

When a pod tries to resolve a short name like api-service, Kubernetes uses DNS search domains to build FQDN candidates:

Pod's /etc/resolv.conf:

nameserver 10.96.0.10
search k8squest.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

DNS Resolution Process for "api-service":

1. First attempt: api-service.k8squest.svc.cluster.local
2. Looks for service in k8squest namespace

3. Service doesn't exist there → NXDOMAIN

4. Second attempt: api-service.svc.cluster.local

5. Ambiguous (which namespace?)

6. Not a valid query → NXDOMAIN

7. Third attempt: api-service.cluster.local

8. Not a service record

9. → NXDOMAIN

10. Final attempt: api-service (as-is)

11. Not a valid DNS name
12. → Resolution FAILED

Result: "wget: bad address 'api-service'"

Why Cross-namespace Needs FQDN

Same Namespace (works with short name):

Frontend in k8squest → api-service → Resolves to: api-service.k8squest.svc.cluster.local ✅

Different Namespace (needs FQDN):

Frontend in k8squest → api-service → Resolves to: api-service.k8squest.svc.cluster.local ❌
                                      (Service is actually in backend-ns!)

Frontend in k8squest → api-service.backend-ns → Resolves correctly ✅

---

The Fix Explained

What You Changed:

# BEFORE (broken)
command: ['sh', '-c', 'wget -q -O- http://api-service']

# AFTER (solution)
command: ['sh', '-c', 'wget -q -O- http://api-service.backend-ns.svc.cluster.local']

DNS FQDN Format:

<service-name>.<namespace>.svc.<cluster-domain>
     |              |        |         |
     |              |        |         +-- Cluster DNS suffix (default: cluster.local)
     |              |        +------------ Service subdomain
     |              +--------------------- Namespace where service exists
     +------------------------------------ Service name

Examples:

api-service.backend-ns.svc.cluster.local    (Full FQDN - recommended)
api-service.backend-ns.svc                  (Short FQDN - works)
api-service.backend-ns                      (Minimal - works)
api-service                                 (Short name - only same namespace!)

Why the Fix Works:

1. Frontend requests: http://api-service.backend-ns.svc.cluster.local
2. DNS query goes to CoreDNS
3. CoreDNS looks up: Service "api-service" in namespace "backend-ns"
4. Returns ClusterIP: 10.96.15.23
5. Frontend connects to ClusterIP
6. kube-proxy routes to backend pod
7. Success! ✅

---

Real-World Incident: The Microservices Migration Disaster

Company: SaaS platform (project management software) Date: March 2023 Impact: 8 hours of downtime, 15,000 users affected, $890K in SLA penalties

What Happened

The company was migrating from a monolithic architecture to microservices. They decided to use namespace isolation for different service tiers:

• frontend namespace: Web UI, mobile API
• backend namespace: Business logic services
• data namespace: Database services, caching

The migration plan was to move services one-by-one from the monolith to microservices.

Phase 1: Backend Migration (Week 1)

The team deployed the first microservice: user-service in the backend namespace.

apiVersion: v1
kind: Service
metadata:
  name: user-service
  namespace: backend    # New microservice in backend namespace
spec:
  selector:
    app: user-service
  ports:
  - port: 8080

Phase 2: Frontend Update (Week 2)

The frontend team updated their code to call the new microservice:

// Frontend code (running in 'frontend' namespace)
const API_URL = 'http://user-service:8080/api/users';

fetch(API_URL)
  .then(response => response.json())
  .then(users => displayUsers(users))
  .catch(err => console.error('Failed to load users:', err));

Deployment:

apiVersion: v1
kind: ConfigMap
metadata:
  name: frontend-config
  namespace: frontend
data:
  API_URL: "http://user-service:8080"    # ❌ SHORT NAME!

The Problem

The frontend and backend teams tested independently:

Backend Team Testing:

# Created test pod in 'backend' namespace
kubectl run test -n backend --image=curlimages/curl -- curl http://user-service:8080/health

# Worked perfectly! ✅

Frontend Team Testing:

# Tested frontend service endpoints in 'frontend' namespace
kubectl port-forward -n frontend service/web-frontend 3000:80

# Frontend loaded fine ✅
# But they didn't test actual API calls!

Production Deployment (Saturday 2 AM):

• 2:00 AM: Deployed updated frontend with user-service integration
• 2:05 AM: All user-related pages started showing errors
• 2:10 AM: Login page broken (calls user-service)
• 2:15 AM: Profile pages failing
• 2:20 AM: User management admin panel down

User Experience:

User tries to login → Frontend calls http://user-service:8080/auth
                   → DNS: "could not resolve host: user-service"
                   → Error: "Service temporarily unavailable"

Timeline

2:00 AM: Frontend v2.0 deployed to production 2:05 AM: User login failures start (5 failed logins/minute) 2:10 AM: Failure rate spikes (500+ failed logins/minute) 2:15 AM: On-call engineer paged (automated alert: 95% login failure rate) 2:30 AM: Team checking authentication service (appears healthy) 3:00 AM: Network team checks connectivity (all green) 3:30 AM: Someone checks frontend pod logs:

Error: getaddrinfo ENOTFOUND user-service
Error: getaddrinfo ENOTFOUND user-service
Error: getaddrinfo ENOTFOUND user-service

3:45 AM: DNS issue suspected 4:00 AM: Checked DNS resolution from frontend pod:

kubectl exec web-frontend-xxx -n frontend -- nslookup user-service
# Returns: NXDOMAIN (not found)

4:15 AM: Realized user-service is in different namespace! 4:30 AM: Hotfix deployed with FQDN 5:00 AM: Services restored 10:00 AM: Full post-mortem completed

The Damage

• 8 hours of downtime (2 AM - 10 AM full recovery)
• 15,000 users couldn't access their accounts
• $890K in SLA penalties (99.9% uptime guarantee violated)
• Brand damage (negative social media, support tickets)
• Lost productivity (entire engineering team worked emergency overnight)

The Hotfix

Option 1: Update ConfigMap (chosen)

apiVersion: v1
kind: ConfigMap
metadata:
  name: frontend-config
  namespace: frontend
data:
  API_URL: "http://user-service.backend.svc.cluster.local:8080"  # ✅ FQDN

Then restart frontend pods to pick up new config.

Option 2: Move service to same namespace Not chosen because: - Would break namespace isolation strategy - Defeats purpose of service tier separation - Harder to implement under pressure

Option 3: Use ExternalName Service Create a "proxy" service in frontend namespace:

apiVersion: v1
kind: Service
metadata:
  name: user-service
  namespace: frontend
spec:
  type: ExternalName
  externalName: user-service.backend.svc.cluster.local

Not chosen because: - Adds complexity - Harder to understand service topology - Could cause naming conflicts

The Long-Term Fix

The team implemented several changes:

1. Service Naming Convention

# Document all cross-namespace dependencies
apiVersion: v1
kind: ConfigMap
metadata:
  name: service-directory
  namespace: kube-public
data:
  # Format: SERVICE_NAME: FQDN
  user-service: "user-service.backend.svc.cluster.local"
  auth-service: "auth-service.backend.svc.cluster.local"
  database: "postgres-service.data.svc.cluster.local"

2. Environment Variable Standard

# Always use FQDN in environment variables
env:
- name: USER_SERVICE_URL
  value: "http://user-service.backend.svc.cluster.local:8080"
- name: AUTH_SERVICE_URL
  value: "http://auth-service.backend.svc.cluster.local:8080"

3. Integration Testing

# Test cross-namespace connectivity BEFORE production
kubectl run test-frontend -n frontend --image=curlimages/curl -- \
  curl http://user-service.backend.svc.cluster.local:8080/health

4. DNS Validation in CI/CD

# Helm chart validation
{{- if not (contains "." .Values.userServiceUrl) }}
  {{- fail "userServiceUrl must be FQDN (include namespace)" }}
{{- end }}

5. Documentation

# Cross-Namespace Communication Guide

## Namespace Structure
- `frontend`: Web UI, mobile API gateway
- `backend`: Business logic microservices
- `data`: Databases, caches, message queues

## Service Discovery Rules
1. Same namespace: Use short name (`service-name`)
2. Different namespace: Use FQDN (`service-name.namespace.svc.cluster.local`)

## Examples
✅ Frontend → Backend:
   http://user-service.backend.svc.cluster.local:8080

✅ Backend → Data:
   postgres://postgres-service.data.svc.cluster.local:5432/mydb

❌ Frontend → Backend (WRONG):
   http://user-service:8080

Lessons Learned

1. Test Cross-Namespace Communication:
2. Don't just test within each namespace
3. Test actual communication paths

4. Include integration tests in CI/CD

5. Always Use FQDN for Different Namespaces:

6. Short names only work within same namespace
7. FQDN is explicit and prevents errors

8. Document service locations

9. Validate Configuration:

10. Check for FQDN in config files
11. Validate DNS resolution before deployment

12. Use tools to detect short names in cross-namespace calls

13. Monitor DNS Errors:

14. Alert on NXDOMAIN responses
15. Track DNS resolution failures

16. Log service discovery issues

17. Document Namespace Topology:

18. Which services are in which namespaces
19. Which services need to communicate
20. Service dependency map

---

Kubernetes DNS Deep Dive

DNS Record Types

Kubernetes creates several types of DNS records:

1. Service DNS Records (A Records)

# Service: api-service in namespace backend-ns
# Gets A record:
api-service.backend-ns.svc.cluster.local → 10.96.15.23 (ClusterIP)

2. Pod DNS Records (A Records)

# Pod with IP 10.244.1.5 in namespace backend-ns
# Gets A record:
10-244-1-5.backend-ns.pod.cluster.local → 10.244.1.5

3. Headless Service Records (A Records per Pod)

# Headless service (clusterIP: None)
api-service.backend-ns.svc.cluster.local → 10.244.1.5
                                         → 10.244.1.6
                                         → 10.244.1.7

4. SRV Records (for named ports)

# Service with named port "http"
_http._tcp.api-service.backend-ns.svc.cluster.local
  → 10 100 8080 api-service.backend-ns.svc.cluster.local

FQDN Components

api-service.backend-ns.svc.cluster.local
    |          |        |       |
    |          |        |       +-- Cluster domain (configurable)
    |          |        +---------- Service/Pod subdomain
    |          +------------------- Namespace
    +------------------------------- Resource name (service/pod)

Cluster Domain: - Default: cluster.local - Configurable in kubelet: --cluster-domain=example.com - Must match across all nodes

Service Subdomain: - svc for Services - pod for Pods

DNS Resolution Examples

Within Same Namespace (k8squest):

# All of these work:
curl http://api-service
curl http://api-service.k8squest
curl http://api-service.k8squest.svc
curl http://api-service.k8squest.svc.cluster.local

# DNS resolution:
api-service → api-service.k8squest.svc.cluster.local (search domain appended)

Cross-Namespace (k8squest → backend-ns):

# These work:
curl http://api-service.backend-ns
curl http://api-service.backend-ns.svc
curl http://api-service.backend-ns.svc.cluster.local

# This FAILS:
curl http://api-service
# DNS tries: api-service.k8squest.svc.cluster.local (wrong namespace!)

To External Service:

# External DNS (outside Kubernetes)
curl http://api.example.com    # Uses external DNS, not CoreDNS

Pod DNS Configuration

Every pod gets DNS configured via /etc/resolv.conf:

kubectl exec frontend-app -n k8squest -- cat /etc/resolv.conf

Output:

nameserver 10.96.0.10
search k8squest.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Explanation: - nameserver: CoreDNS service IP - search: DNS search domains (appended to short names) - ndots:5: If name has < 5 dots, try search domains first

ndots Behavior:

# "api-service" has 0 dots (< 5)
# Try search domains:
#   1. api-service.k8squest.svc.cluster.local
#   2. api-service.svc.cluster.local
#   3. api-service.cluster.local
# Then try as-is: api-service

# "api-service.backend-ns.svc.cluster.local" has 5 dots (>= 5)
# Try as-is first: api-service.backend-ns.svc.cluster.local ✅

---

Cross-Namespace Communication Patterns

Pattern 1: Frontend → Backend Architecture

┌─────────────────────┐
│  frontend namespace │
│                     │
│  ┌───────────────┐  │
│  │  Web UI       │  │
│  └───────┬───────┘  │
└──────────┼──────────┘
           │
           │ http://api-service.backend.svc.cluster.local
           │
┌──────────▼──────────┐
│  backend namespace  │
│                     │
│  ┌───────────────┐  │
│  │  API Service  │  │
│  └───────┬───────┘  │
└──────────┼──────────┘
           │
           │ postgres://db-service.data.svc.cluster.local
           │
┌──────────▼──────────┐
│  data namespace     │
│                     │
│  ┌───────────────┐  │
│  │  Database     │  │
│  └───────────────┘  │
└─────────────────────┘

Configuration:

# Frontend deployment
env:
- name: API_URL
  value: "http://api-service.backend.svc.cluster.local:8080"

# Backend deployment
env:
- name: DATABASE_URL
  value: "postgres://db-service.data.svc.cluster.local:5432/mydb"

Pattern 2: Multi-Tenant Isolation

┌───────────────────┐    ┌───────────────────┐
│  tenant-a         │    │  tenant-b         │
│  ┌─────────────┐  │    │  ┌─────────────┐  │
│  │ App Instance│  │    │  │ App Instance│  │
│  └──────┬──────┘  │    │  └──────┬──────┘  │
└─────────┼─────────┘    └─────────┼─────────┘
          │                        │
          │ Both access shared services
          │                        │
          └────────────┬───────────┘
                       │
          ┌────────────▼──────────────┐
          │  shared-services          │
          │  ┌──────────────────────┐ │
          │  │ Authentication      │ │
          │  │ Logging             │ │
          │  │ Monitoring          │ │
          │  └──────────────────────┘ │
          └───────────────────────────┘

Configuration:

# Tenant A app
env:
- name: AUTH_SERVICE
  value: "http://auth-service.shared-services.svc.cluster.local"

# Tenant B app (same)
env:
- name: AUTH_SERVICE
  value: "http://auth-service.shared-services.svc.cluster.local"

Pattern 3: Environment Isolation

┌──────────────┐  ┌──────────────┐  ┌──────────────┐
│  dev         │  │  staging     │  │  production  │
│              │  │              │  │              │
│  ┌────────┐  │  │  ┌────────┐  │  │  ┌────────┐  │
│  │  App   │  │  │  │  App   │  │  │  │  App   │  │
│  └────┬───┘  │  │  └────┬───┘  │  │  └────┬───┘  │
└───────┼──────┘  └───────┼──────┘  └───────┼──────┘
        │                 │                 │
        └─────────────────┴─────────────────┘
                          │
                ┌─────────▼──────────┐
                │  databases         │
                │  ┌──────────────┐  │
                │  │ dev-db       │  │
                │  │ staging-db   │  │
                │  │ prod-db      │  │
                │  └──────────────┘  │
                └────────────────────┘

Configuration:

# Dev app
env:
- name: DATABASE_URL
  value: "postgres://dev-db.databases.svc.cluster.local:5432"

# Staging app
env:
- name: DATABASE_URL
  value: "postgres://staging-db.databases.svc.cluster.local:5432"

# Prod app
env:
- name: DATABASE_URL
  value: "postgres://prod-db.databases.svc.cluster.local:5432"

---

Security Considerations

NetworkPolicy for Cross-Namespace Access

By default, pods can access services in any namespace. Use NetworkPolicy to restrict:

# Allow only frontend namespace to access backend services
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-access-policy
  namespace: backend
spec:
  podSelector:
    matchLabels:
      tier: api
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: frontend    # Only frontend namespace
    ports:
    - protocol: TCP
      port: 8080

Namespace Labels:

apiVersion: v1
kind: Namespace
metadata:
  name: frontend
  labels:
    name: frontend

RBAC for Namespace Access

Control which service accounts can access resources in other namespaces:

# Role in backend namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: service-reader
  namespace: backend
rules:
- apiGroups: [""]
  resources: ["services", "endpoints"]
  verbs: ["get", "list"]
---
# RoleBinding allowing frontend service account
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: frontend-can-read-backend-services
  namespace: backend
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: service-reader
subjects:
- kind: ServiceAccount
  name: frontend-app
  namespace: frontend

---

Debugging Cross-Namespace DNS Issues

1. Test DNS Resolution

# From frontend pod
kubectl exec frontend-app -n k8squest -- nslookup api-service
# Should fail: NXDOMAIN

kubectl exec frontend-app -n k8squest -- nslookup api-service.backend-ns.svc.cluster.local
# Should succeed: Returns ClusterIP

2. Check DNS Configuration

# View pod's DNS config
kubectl exec frontend-app -n k8squest -- cat /etc/resolv.conf

# Check CoreDNS is running
kubectl get pods -n kube-system -l k8s-app=kube-dns

3. Verify Service Exists

# Check service in target namespace
kubectl get service api-service -n backend-ns

# Check service has endpoints
kubectl get endpoints api-service -n backend-ns

4. Test Connectivity

# Get service ClusterIP
SERVICE_IP=$(kubectl get service api-service -n backend-ns -o jsonpath='{.spec.clusterIP}')

# Test direct IP connection
kubectl exec frontend-app -n k8squest -- wget -q -O- http://$SERVICE_IP

5. Check CoreDNS Logs

# View CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns

# Look for NXDOMAIN responses
kubectl logs -n kube-system -l k8s-app=kube-dns | grep NXDOMAIN

6. Query CoreDNS Directly

# Get CoreDNS service IP
kubectl get service kube-dns -n kube-system

# Query from pod
kubectl exec frontend-app -n k8squest -- nslookup api-service.backend-ns.svc.cluster.local 10.96.0.10

---

Best Practices

1. Always Use FQDN for Cross-Namespace

# ❌ BAD: Relies on same namespace
env:
- name: API_URL
  value: "http://api-service:8080"

# ✅ GOOD: Explicit namespace
env:
- name: API_URL
  value: "http://api-service.backend.svc.cluster.local:8080"

2. Document Service Locations

# ConfigMap documenting service topology
apiVersion: v1
kind: ConfigMap
metadata:
  name: service-directory
  namespace: kube-public
data:
  services.yaml: |
    frontend:
      - web-ui
      - mobile-api
    backend:
      - user-service
      - auth-service
      - notification-service
    data:
      - postgres-service
      - redis-service

3. Use Environment Variables

# Don't hardcode URLs in application code
# Use environment variables instead
env:
- name: USER_SERVICE_URL
  value: "http://user-service.backend.svc.cluster.local:8080"
- name: AUTH_SERVICE_URL
  value: "http://auth-service.backend.svc.cluster.local:8080"

4. Validate in CI/CD

#!/bin/bash
# Validate all service URLs use FQDN for cross-namespace

# Extract all service URLs from config
SERVICE_URLS=$(grep -r "http://" k8s/ | grep -o "http://[^:]*")

for url in $SERVICE_URLS; do
  service=$(echo $url | sed 's|http://||')

  # Check if it contains a dot (namespace qualifier)
  if [[ ! "$service" =~ \. ]]; then
    echo "ERROR: $service uses short name (add namespace!)"
    exit 1
  fi
done

echo "✅ All service URLs use FQDN"

5. Use Service Mesh for Advanced Routing

For complex multi-namespace architectures, consider a service mesh:

# Istio VirtualService for cross-namespace routing
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: user-service
  namespace: frontend
spec:
  hosts:
  - user-service.backend.svc.cluster.local
  http:
  - route:
    - destination:
        host: user-service.backend.svc.cluster.local
        port:
          number: 8080

6. Monitor DNS Performance

# Prometheus metrics for DNS
sum(rate(coredns_dns_request_duration_seconds_count[5m])) by (type)

# Alert on high DNS failure rate
sum(rate(coredns_dns_response_rcode_count_total{rcode="NXDOMAIN"}[5m])) > 10

---

Key Takeaways

1. DNS Naming Rules:
2. Same namespace: service-name (short name works)
3. Different namespace: service-name.namespace.svc.cluster.local (FQDN required)

4. Full format: <service>.<namespace>.svc.<cluster-domain>

5. DNS Search Domains:

6. Pods get search domains: namespace.svc.cluster.local, svc.cluster.local, cluster.local
7. Short names are expanded using search domains

8. Only searches current namespace by default

9. Why FQDN Matters:

10. Explicit namespace prevents resolution errors
11. Works regardless of pod's namespace
12. Clear and self-documenting

13. Prevents ambiguity

14. Common Mistakes:

15. Using short names for cross-namespace (DNS fails)
16. Wrong namespace in FQDN (service not found)
17. Typos in service names (DNS fails)

18. Testing in same namespace, deploying cross-namespace

19. Real-World Lessons:

20. Always test cross-namespace communication before production
21. Use FQDN in all configuration files
22. Document namespace topology
23. Validate DNS resolution in CI/CD
24. Monitor DNS errors

---

What's Next?

You've mastered cross-namespace service communication! You now understand: - ✅ Kubernetes DNS naming conventions - ✅ When to use short names vs FQDN - ✅ How DNS search domains work - ✅ Debugging DNS resolution issues across namespaces

In the next levels, you'll explore service endpoint management with readiness probes, LoadBalancer vs NodePort services, and headless services for StatefulSets.

Continue your Quest Ladder journey to unlock the next challenge! 🚀

---

Additional Resources

• Kubernetes DNS Documentation
• CoreDNS
• DNS Debugging Guide
• Service Discovery

---

Mission Complete! 🎉 You've earned 250 XP and mastered cross-namespace service discovery!