Hints: prometheus-target-down¶
Hint 1¶
Prometheus can't scrape the target. Check targets page:
- Port-forward Prometheus and check targets
- kubectl get servicemonitor -n grokdevops
Hint 2¶
The issue is likely a ServiceMonitor label mismatch. Check:
- kubectl get svc -n grokdevops --show-labels
- kubectl describe servicemonitor -n grokdevops
- Do the ServiceMonitor selector labels match the Service labels?
Hint 3¶
Check the metrics endpoint is actually serving:
- kubectl exec -n grokdevops <pod> -- wget -qO- http://localhost:8000/metrics | head -3
- Check the port name in ServiceMonitor matches the Service port name
- Check that Prometheus operator's serviceMonitorSelector includes this namespace
Hint 4¶
Fix: align the ServiceMonitor's selector.matchLabels with the actual Service labels, or fix the port name/number. See training/library/runbooks/prometheus_target_down.md for the full checklist.