Solution: Lab Runtime 04 -- Loki No Logs¶
SPOILER WARNING: Try to solve it yourself first.
Hint Ladder¶
Hint 1: Logs flow from containers -> Promtail (DaemonSet) -> Loki -> Grafana. Check each link in the chain.
Hint 2: Are Promtail pods running? Check kubectl get pods -n monitoring -l app.kubernetes.io/name=promtail.
Hint 3: Promtail pods are gone because the DaemonSet has an impossible nodeSelector. No node matches, so no pods are scheduled.
Hint 4: Remove the broken nodeSelector from the Promtail DaemonSet, or run ./fix.sh.
Minimal Solution¶
kubectl patch daemonset -n monitoring promtail --type=json \
-p='[{"op":"remove","path":"/spec/template/spec/nodeSelector/nonexistent-label"}]'
Explain¶
Symptom: New logs stop appearing in Grafana/Loki. Historical logs still visible.
Evidence: kubectl get pods -n monitoring -l app.kubernetes.io/name=promtail returns no pods. The DaemonSet shows DESIRED=0 because no nodes match the nodeSelector.
Root cause: DaemonSets schedule one pod per node that matches the nodeSelector and tolerations. Adding an impossible nodeSelector (nonexistent-label: "true") means no node qualifies, so all Promtail pods are terminated. Without Promtail, container logs aren't scraped and sent to Loki.
Key insight: Loki itself is fine; the pipeline breaks at the collection layer. Always check the log shipper (Promtail/Fluentd/Vector) before suspecting Loki.
Prevent¶
- Don't modify DaemonSet nodeSelectors without understanding impact
- Monitor Promtail pod count in dashboards (alert if zero)
- Use Helm values for nodeSelector to make changes tracked in Git