Observability Drill Answers¶

Drill 1¶

kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090 &
# Then open: http://localhost:9090/targets
# Or via CLI:
curl -s http://localhost:9090/api/v1/targets | python3 -c "import sys,json; data=json.load(sys.stdin); [print(t['labels'].get('job','?'), t['health']) for t in data['data']['activeTargets']]"

Drill 2¶

kubectl get pods -n kube-system -l k8s-app=metrics-server
kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes

If the API call returns data, metrics-server is working.

Drill 3¶

kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80

Access at http://localhost:3000. Default credentials for kube-prometheus-stack: admin/prom-operator (change in production via grafana.adminPassword Helm value).

Drill 4¶

kubectl get pods -n monitoring -l app.kubernetes.io/name=promtail
kubectl get daemonset -n monitoring -l app.kubernetes.io/name=promtail

DESIRED should equal READY. If Promtail is missing from nodes, check nodeSelector/tolerations.

Drill 5¶

kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090 &
curl -s 'http://localhost:9090/api/v1/query?query=up' | python3 -m json.tool

up == 1 means healthy, up == 0 means target down.

Drill 6¶

# Get ServiceMonitor selector
kubectl get servicemonitor -n monitoring -o jsonpath='{range .items[*]}{.metadata.name}: {.spec.selector.matchLabels}{"\n"}{end}'

# Get service labels
kubectl get svc grokdevops -n grokdevops --show-labels

The ServiceMonitor matchLabels must appear in the service's labels.

Drill 7¶

kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80 &
curl -s -u admin:prom-operator http://localhost:3000/api/datasources | python3 -c "import sys,json; [print(d['name'],d['type']) for d in json.load(sys.stdin)]"

Drill 8¶

kubectl get configmap -n monitoring -l app.kubernetes.io/name=promtail -o yaml | grep -A 20 "scrape_configs"

Drill 9¶

kubectl get pods -n monitoring -l app.kubernetes.io/name=tempo
kubectl get svc -n monitoring -l app.kubernetes.io/name=tempo

Drill 10¶

# 1. Get the ServiceMonitor's selector
kubectl get servicemonitor grokdevops -n monitoring -o jsonpath='{.spec.selector.matchLabels}'

# 2. Get the service's labels
kubectl get svc grokdevops -n grokdevops --show-labels

# 3. Compare: do the matchLabels appear in the service labels?

Drill 11¶

kubectl get prometheusrule -n monitoring

Drill 12¶

kubectl get secret -n monitoring prometheus-kube-prometheus-stack-prometheus -o jsonpath='{.data.prometheus\.yaml\.gz}' | base64 -d | gunzip | head -50

Or check the Prometheus ConfigMap/Secret depending on the operator configuration.

Drill 13¶

kubectl exec -n grokdevops deploy/grokdevops -- wget -qO- http://localhost:8000/metrics | head -20

Should return Prometheus-format metrics (lines starting with # or metric names).

Drill 14¶

kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80 &
curl -s -u admin:prom-operator http://localhost:3000/api/search | python3 -c "import sys,json; [print(d.get('title','?')) for d in json.load(sys.stdin)]"

Drill 15¶

echo "=== Prometheus ===" && kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus
echo "=== Loki ===" && kubectl get pods -n monitoring -l app.kubernetes.io/name=loki
echo "=== Promtail ===" && kubectl get pods -n monitoring -l app.kubernetes.io/name=promtail
echo "=== Tempo ===" && kubectl get pods -n monitoring -l app.kubernetes.io/name=tempo
echo "=== Grafana ===" && kubectl get pods -n monitoring -l app.kubernetes.io/name=grafana

All should show Running with READY 1/1 or more.