Skip to content

Portal | Level: L2: Operations | Topics: Loki | Domain: Observability

Runbook: Loki Not Receiving Logs

Symptoms

  • Grafana Loki shows no logs for application namespace
  • {namespace="grokdevops"} returns empty
  • Application is running and producing stdout

Fast Triage

# Check Promtail pods (log collector)
kubectl get pods -n monitoring -l app.kubernetes.io/name=promtail

# Check Promtail logs
kubectl logs -n monitoring -l app.kubernetes.io/name=promtail --tail=50

# Check Loki health
kubectl port-forward -n monitoring svc/loki 3100:3100
curl http://localhost:3100/ready

# Verify application is producing logs
kubectl logs -n grokdevops deploy/grokdevops --tail=5

Likely Causes (ranked)

  1. Promtail not running — DaemonSet crashed or misconfigured
  2. Promtail can't reach Loki — wrong endpoint URL or network issue
  3. Label mismatch — Promtail pipeline not selecting the right labels
  4. Loki not ready — Loki pod crashed or storage full
  5. Node selector / toleration — Promtail not scheduled on the node running grokdevops

Evidence Interpretation

What bad looks like: 

$ kubectl get pods -n monitoring -l app.kubernetes.io/name=promtail
No resources found in monitoring namespace.

$ kubectl get daemonset -n monitoring promtail
NAME       DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE
promtail   0         0         0       0             0
- Promtail pods missing entirely — if DESIRED=0 on the DaemonSet, no log collection is happening on any node. - If Promtail pods exist but show errors, check kubectl logs for connection refused or 429 (Loki rate limit) messages. - Loki itself may be healthy (/ready returns 200) while the pipeline is broken at the Promtail stage.

Fix Steps

[!TIP] Historical logs already stored in Loki persist even if the pipeline breaks — only new logs stop appearing. Before assuming total data loss, widen the time range in Grafana. If you see old logs but not recent ones, the problem is in the collection pipeline (Promtail), not Loki itself.

  1. Restart Promtail if it's crashed: 
    kubectl rollout restart daemonset promtail -n monitoring
  2. Verify Promtail config points to Loki: 
    kubectl get configmap promtail -n monitoring -o yaml | grep -A5 clients
  3. Check Loki is ready: 
    kubectl get pods -n monitoring -l app.kubernetes.io/name=loki
  4. Restore Promtail via Helm if needed: 
    helm upgrade promtail grafana/promtail -n monitoring -f devops/observability/values/values-promtail.yaml

Verification

# In Grafana: Explore → Loki → {namespace="grokdevops"}
kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80
# Open http://localhost:3000 (admin/prom-operator)

Cleanup

None beyond restoring Promtail.

Unknown Unknowns

  • The log pipeline is: container stdout -> Promtail (collects from node) -> Loki (stores) -> Grafana (queries). A break at any stage causes "no logs."
  • Historical logs already stored in Loki persist even if the pipeline breaks — only new logs stop appearing. Check the time range in Grafana.
  • DaemonSet nodeSelector or tolerations affect which nodes get Promtail pods. If your app runs on a node that Promtail skips, those logs are lost.
  • Promtail tracks its read position in /var/log; restarting it does not re-send old logs unless the position file is deleted.

Pitfalls

  • Blaming Loki when Promtail is the problem — always check Promtail first. Loki only stores what it receives.
  • Not checking the DaemonSet spec — a DaemonSet with DESIRED=0 means a nodeSelector or toleration is filtering out all nodes. Fix the scheduling constraint.
  • Restarting Loki instead of Promtail — if Loki is healthy, restarting it achieves nothing and causes a brief query outage.

See Also

  • training/library/guides/troubleshooting.md (Loki section)
  • training/library/guides/observability.md
  • training/interactive/runtime-labs/lab-runtime-04-loki-no-logs/
  • training/interview-scenarios/04-loki-logs-disappeared.md
  • training/interactive/incidents/scenarios/loki-no-logs.sh

Wiki Navigation

Related Content