Multi-Tenancy Patterns - Street-Level Ops

Quick Diagnosis Commands

# List all namespaces with resource quota usage
kubectl get resourcequota --all-namespaces -o custom-columns=\
'NAMESPACE:.metadata.namespace,NAME:.metadata.name,CPU_REQ:.status.used.requests\.cpu,CPU_LIM:.status.used.limits\.cpu,MEM_REQ:.status.used.requests\.memory,MEM_LIM:.status.used.limits\.memory'

# Check quota vs usage for a specific tenant namespace
kubectl describe resourcequota -n tenant-acme

# Find namespaces without resource quotas (danger zones)
kubectl get namespaces -o name | while read ns; do
  ns_name=${ns#namespace/}
  count=$(kubectl get resourcequota -n "$ns_name" --no-headers 2>/dev/null | wc -l)
  [ "$count" -eq 0 ] && echo "NO QUOTA: $ns_name"
done

# Find namespaces without network policies
kubectl get namespaces -o name | while read ns; do
  ns_name=${ns#namespace/}
  count=$(kubectl get networkpolicy -n "$ns_name" --no-headers 2>/dev/null | wc -l)
  [ "$count" -eq 0 ] && echo "NO NETPOL: $ns_name"
done

# Check who has ClusterRoleBindings (potential RBAC overreach)
kubectl get clusterrolebindings -o custom-columns=\
'NAME:.metadata.name,ROLE:.roleRef.name,SUBJECTS:.subjects[*].name'

# View all pods without resource requests (quota violators-in-waiting)
kubectl get pods --all-namespaces -o json | \
  jq -r '.items[] | select(.spec.containers[].resources.requests == null) |
  "\(.metadata.namespace)/\(.metadata.name)"'

# Network policy audit — which namespaces have default-deny?
kubectl get networkpolicy --all-namespaces -o json | \
  jq -r '.items[] | select(.spec.podSelector == {}) |
  "\(.metadata.namespace): \(.metadata.name)"'

# Pod priority class distribution
kubectl get pods --all-namespaces -o json | \
  jq -r '.items[] | "\(.spec.priorityClassName // "none") \(.metadata.namespace)"' | \
  sort | uniq -c | sort -rn

---

Gotcha: ResourceQuota Exists but Pods Deploy Without Requests

You created the quota, but pods still deploy with no resource requests. That means:

1. The quota only has object count limits (pods, services), not compute limits
2. OR you have a LimitRange with defaults that auto-inject requests

Check:

# Is the quota enforcing compute?
kubectl get resourcequota -n tenant-acme -o yaml | grep -A 5 'hard:'

# Is there a LimitRange providing defaults?
kubectl get limitrange -n tenant-acme -o yaml

If the quota has requests.cpu but there is no LimitRange default, pods without resource specs will be rejected by the admission controller. This is actually the correct behavior — the error message is:

Error: pods "myapp" is forbidden: failed quota: tenant-quota:
must specify requests.cpu, requests.memory

But if teams complain, add a LimitRange with sensible defaults so their pods get auto-injected values.

---

Pattern: Namespace Provisioning Automation

Use a Helm chart or Kustomize to stamp out tenant namespaces consistently:

# tenant-values.yaml (per tenant)
tenant:
  name: acme
  team: platform-acme
  environment: production
  quotas:
    cpu_requests: "20"
    cpu_limits: "40"
    memory_requests: 40Gi
    memory_limits: 80Gi
    pods: "100"
    pvcs: "30"
    storage: 500Gi
  network:
    allow_ingress_from:
      - ingress-nginx
      - monitoring
    allow_egress_to_cidrs:
      - 10.0.0.0/8

# Provision a new tenant
helm install tenant-acme ./tenant-chart -f tenants/acme-values.yaml

# Audit all tenants
helm list | grep tenant-

# Update quotas
helm upgrade tenant-acme ./tenant-chart -f tenants/acme-values.yaml

Every tenant gets the same structure: namespace, quota, limitrange, network policies, RBAC. No snowflakes.

---

Gotcha: Cross-Namespace Service Access Silently Works

Default trap: Kubernetes has no network isolation by default. Without NetworkPolicies, every pod can reach every other pod in the cluster. Installing a CNI that supports NetworkPolicies (Calico, Cilium) is necessary but not sufficient -- you must also create the policies.

You set up network policies, but Tenant A can still call Tenant B's service:

# From tenant-acme namespace
kubectl exec -n tenant-acme debug-pod -- \
  curl http://api-service.tenant-beta.svc.cluster.local:8080
# Returns: 200 OK  <-- this should not work

Common causes:

1. No default-deny egress policy — you only denied ingress
2. CNI does not enforce network policies — check your CNI
3. DNS resolution still works — even with egress deny, you might have allowed DNS (port 53) and the CNI only filters at L3/L4

# Verify your CNI supports network policies
kubectl get pods -n kube-system | grep -E '(calico|cilium|weave)'

# If using Cilium, check enforcement
kubectl -n kube-system exec cilium-agent-xxx -- cilium status | grep Policy

# Test actual network enforcement (not just policy existence)
kubectl exec -n tenant-acme debug-pod -- \
  timeout 3 curl -s http://api-service.tenant-beta.svc.cluster.local:8080
# Should timeout/fail with proper deny policies

---

Pattern: Quota Monitoring and Alerting

Quotas prevent overconsumption but tenants hit them unexpectedly during scaling events. Monitor usage proactively:

# Prometheus alert for quota nearing limit
groups:
  - name: multi-tenancy
    rules:
      - alert: NamespaceQuotaNearLimit
        expr: |
          kube_resourcequota{type="used"}
          /
          kube_resourcequota{type="hard"}
          > 0.85
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Namespace {{ $labels.namespace }} is at {{ $value | humanizePercentage }} of {{ $labels.resource }} quota"

      - alert: NamespaceNoQuota
        expr: |
          count by (namespace) (kube_pod_info)
          unless
          count by (namespace) (kube_resourcequota)
        for: 1h
        labels:
          severity: critical
        annotations:
          summary: "Namespace {{ $labels.namespace }} has pods but no ResourceQuota"

# Quick check — percentage used per quota per namespace
kubectl get resourcequota --all-namespaces -o json | \
  jq -r '.items[] | .metadata.namespace as $ns |
  .status | to_entries[] |
  select(.key | startswith("used.")) |
  {ns: $ns, resource: (.key | ltrimstr("used.")),
   used: .value, hard: (input.status.hard[.key | ltrimstr("used.")] // "n/a")} |
  "\($ns)\t\(.resource)\t\(.used)/\(.hard)"'

---

Gotcha: PriorityClass Preemption Surprises

War story: A platform team set a "monitoring" PriorityClass to value 1000000 so monitoring would never be evicted. During a node scale-down, the scheduler preempted tenant production pods to keep monitoring running. Tenants saw their pods killed with no warning. The fix: set preemptionPolicy: Never on monitoring workloads -- they stay when resources are available but do not evict others.

You add priority classes but forget to think about preemption. A high-priority deployment scales up during an incident. The scheduler evicts tenant workloads to make room. Those tenants see pods killed with Preempted status and no obvious explanation.

# Find preempted pods
kubectl get events --all-namespaces --field-selector reason=Preempted

# Check which priority classes allow preemption
kubectl get priorityclasses -o custom-columns=\
'NAME:.metadata.name,VALUE:.value,PREEMPTION:.preemptionPolicy'

For burst/batch workloads, use preemptionPolicy: Never:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: batch-burst
value: 1000
preemptionPolicy: Never   # Can be preempted, but cannot preempt others

---

Pattern: Tenant Onboarding Checklist

Run this checklist for every new tenant namespace:

NAMESPACE="tenant-newcorp"

echo "=== Tenant Audit: $NAMESPACE ==="

# 1. Namespace exists
kubectl get namespace $NAMESPACE && echo "[OK] Namespace exists" || echo "[FAIL] Namespace missing"

# 2. ResourceQuota present
kubectl get resourcequota -n $NAMESPACE --no-headers | wc -l | \
  xargs -I{} bash -c '[ {} -gt 0 ] && echo "[OK] ResourceQuota present" || echo "[FAIL] No ResourceQuota"'

# 3. LimitRange present
kubectl get limitrange -n $NAMESPACE --no-headers | wc -l | \
  xargs -I{} bash -c '[ {} -gt 0 ] && echo "[OK] LimitRange present" || echo "[FAIL] No LimitRange"'

# 4. Default-deny network policy
kubectl get networkpolicy -n $NAMESPACE -o json | \
  jq -e '.items[] | select(.spec.podSelector == {} and .spec.policyTypes == ["Ingress","Egress"])' \
  > /dev/null 2>&1 && echo "[OK] Default-deny policy" || echo "[FAIL] No default-deny"

# 5. RBAC binding exists
kubectl get rolebinding -n $NAMESPACE --no-headers | wc -l | \
  xargs -I{} bash -c '[ {} -gt 0 ] && echo "[OK] RoleBinding present" || echo "[FAIL] No RoleBinding"'

# 6. No ClusterRoleBinding for tenant
kubectl get clusterrolebinding -o json | \
  jq -e ".items[] | select(.subjects[]?.name | test(\"$NAMESPACE\"))" \
  > /dev/null 2>&1 && echo "[WARN] Has ClusterRoleBinding" || echo "[OK] No ClusterRoleBinding"

echo "=== Audit Complete ==="

---

Gotcha: LimitRange Defaults Silently Override Explicit Values

A LimitRange with a max of 4 CPU will reject pods requesting 8 CPU. But a LimitRange with a default of 500m will inject 500m into any container that does not specify limits — even if the developer intentionally omitted them to get burstable QoS.

# Check what LimitRange is injecting
kubectl describe limitrange -n tenant-acme

# Compare pod spec before and after admission
kubectl get pod myapp -n tenant-acme -o yaml | grep -A 5 resources
# You might see limits you never asked for

Document your LimitRange defaults and make them visible to tenants. Surprises here lead to OOMKills when the injected memory limit is too low for the actual workload.

Debug clue: If pods are OOMKilled but developers swear they never set memory limits, check kubectl describe limitrange -n <namespace>. The LimitRange default is silently injecting limits into every container that omits them.

---

Pattern: Cross-Namespace Traffic Debugging

When a legitimate cross-namespace connection fails:

# Step 1: Verify DNS resolution works
kubectl exec -n tenant-acme debug-pod -- nslookup api.tenant-shared.svc.cluster.local

# Step 2: Check network policies on BOTH sides
kubectl get networkpolicy -n tenant-acme -o yaml   # Egress rules
kubectl get networkpolicy -n tenant-shared -o yaml  # Ingress rules

# Step 3: Test connectivity at network level
kubectl exec -n tenant-acme debug-pod -- nc -zv api.tenant-shared.svc.cluster.local 8080

# Step 4: If using Cilium, check policy verdict
kubectl -n kube-system exec cilium-agent-xxx -- cilium monitor --type policy-verdict

# Step 5: Verify namespace labels (used in namespaceSelector)
kubectl get namespace tenant-shared --show-labels
# NetworkPolicy namespaceSelector matches on labels, not names

Common fix: the target namespace is missing the label that the NetworkPolicy's namespaceSelector expects.

Pages that link here

• Anti-Primer: Multi Tenancy
• Multi-Tenancy Patterns