Progressive Hints

Hint 1 (after 5 min)

The HPA wants 6 replicas but only 3 exist. The events show "exceeded quota" — but look at kubectl top nodes: nodes are at 22-34% CPU utilization. The cluster has plenty of capacity. The bottleneck is not node resources — it is the namespace ResourceQuota.

Hint 2 (after 10 min)

The ResourceQuota allows requests.cpu: 2 for the entire marketplace namespace. Each catalog-service pod requests 500m CPU. With 3 pods running, that is 1500m (1.5 CPU) used out of 2 CPU allowed. Adding a 4th pod would require 2000m — exactly at the limit. But the HPA wants 6, which would need 3000m — 50% over quota. The pod quota (20) is not the issue. The CPU request quota (2) is.

Hint 3 (after 15 min)

This is a marketplace catalog service with an HPA configured to scale from 3 to 10 replicas based on CPU utilization. CPU is at 87% (above the 70% target), so the HPA correctly decides to scale to 6 replicas. But the namespace ResourceQuota only allows 2 CPU cores of requests total. Three pods use 1.5 cores, leaving only 500m — enough for one more pod, not three. The quota was set when the service ran 2-3 replicas and was never updated for autoscaling. The cluster autoscaler sees "no unschedulable pods" because the pods are not pending — they are never created (rejected at the API server level by the quota admission controller). This makes the cluster autoscaler blind to the problem.