Skip to content

Runbook: Grafana Dashboard Blank / No Data

Field Value
Domain Observability
Alert User report of blank panels, or Grafana panels showing "No data" for metrics known to exist
Severity P2
Est. Resolution Time 15-30 minutes
Escalation Timeout 30 minutes — page if not resolved
Last Tested 2026-03-19
Prerequisites Grafana admin access, kubectl access, Prometheus UI access (port-forward if needed)

Quick Assessment (30 seconds)

# Run this first — it tells you the scope of the problem
kubectl get pods -n monitoring
If output shows: Grafana or Prometheus pods in CrashLoopBackOff or Error → Start at Step 6 (check logs) before anything else If output shows: All pods Running/Ready → The infrastructure is up; the problem is in configuration or data — continue to Step 1

Step 1: Check Whether Prometheus Itself Has the Data

Why: If Prometheus has no data, the problem is upstream of Grafana. If Prometheus has data but Grafana does not show it, the problem is in the Grafana-to-Prometheus connection or the dashboard query. These require completely different fixes. 

# Port-forward to Prometheus and run a simple query
kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090 &
curl -s 'http://localhost:9090/api/v1/query?query=up' | python3 -m json.tool | grep '"value"' | head -5
Expected output: 
"value": [1710000000, "1"]
If up returns results, Prometheus is working. If it returns empty, see prometheus-target-down.md. If this fails: Port-forward may have failed. Check that the prometheus-operated service name is correct for your deployment: kubectl get svc -n monitoring | grep prometheus

Step 2: Check Grafana Data Source Configuration

Why: A misconfigured data source URL, wrong port, or changed authentication token will cause all dashboards using that data source to show "No data" simultaneously. This is a single point of failure for the whole observability stack. 

# Port-forward to Grafana
kubectl port-forward -n monitoring svc/grafana 3000:3000 &
# Then open http://localhost:3000 and navigate to:
# Configuration (gear icon) → Data Sources → select your Prometheus data source → "Save & Test"

# Also check the data source health via the API (replace <GRAFANA_API_KEY> with a service account token)
curl -s -H "Authorization: Bearer <GRAFANA_API_KEY>" \
  http://localhost:3000/api/datasources | python3 -m json.tool | grep -E '"name"|"url"|"type"'
Expected output: 
"name": "Prometheus",
"type": "prometheus",
"url": "http://prometheus-operated:9090"
Common problems: http vs https, wrong port (9090 vs 9091), stale service name after a Prometheus rename. If this fails: If you cannot log in to Grafana, check for admin credentials in a Kubernetes secret: kubectl get secret -n monitoring grafana-admin-credentials -o yaml

Step 3: Check the Dashboard Time Range and Panel Queries

Why: The most common cause of "No data" is that the user is looking at a time range that contains no data — either too far in the past (data retention expired) or in the future. The second most common is a dashboard variable that is unset, causing the PromQL query to match nothing. 

# Check Prometheus retention settings
curl -s http://localhost:9090/api/v1/status/tsdb | python3 -m json.tool | grep -E '"minTime"|"maxTime"'
Expected output: 
"minTime": 1709913600000,
"maxTime": 1710086400000
Convert these epoch timestamps to verify the data exists within the dashboard's time range.

In the Grafana UI: 1. Open the panel in question → click the panel title → "Edit" 2. Check the Query tab — is the PromQL correct? 3. Check the Time range in the top-right corner — does it overlap with Prometheus' data range? 4. Check if any dashboard variables (dropdowns at the top) are set to "All" when they should be specific, or to a value that no longer exists

If this fails: If you cannot identify the specific panel query, export the dashboard JSON: kubectl get configmap -n monitoring <DASHBOARD_CONFIGMAP> -o yaml

Step 4: Check for Variable/Template Syntax Errors in Dashboard Queries

Why: Grafana dashboard variables are interpolated into PromQL at query time. A variable that resolves to an empty string, or uses the wrong syntax, produces a syntactically valid but data-less query that silently returns empty. 

# Get the raw dashboard JSON to inspect queries
curl -s -H "Authorization: Bearer <GRAFANA_API_KEY>" \
  "http://localhost:3000/api/dashboards/uid/<DASHBOARD_UID>" | python3 -m json.tool | grep '"expr"'
Expected output: 
"expr": "rate(http_requests_total{namespace=\"$namespace\", job=\"$job\"}[5m])"
In the Grafana UI, open a blank panel and switch to the "Query" tab. Run the PromQL manually with the variable values substituted (replace $namespace with the actual namespace value). If the manual query returns data but the dashboard does not, the variable substitution is broken. If this fails: If you do not have the dashboard UID, list all dashboards: curl -s -H "Authorization: Bearer <GRAFANA_API_KEY>" http://localhost:3000/api/search | python3 -m json.tool | grep '"uid"'

Step 5: Check Whether the Specific Metric Exists in Prometheus

Why: If a metric was renamed, removed, or never started being scraped, the dashboard will show "No data" for exactly that panel. This happens during application upgrades that change metric names. 

# Search for metrics matching a partial name
curl -s 'http://localhost:9090/api/v1/label/__name__/values' | python3 -m json.tool | grep '<METRIC_NAME_PREFIX>'

# Query the specific metric from the dashboard to confirm it exists
curl -s 'http://localhost:9090/api/v1/query?query=<METRIC_NAME_FROM_DASHBOARD_QUERY>{job="<JOB_NAME>"}' | python3 -m json.tool
Expected output: 
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [...]
  }
}
If result is empty, the metric does not exist in Prometheus. Check the application's /metrics endpoint directly to confirm the metric name. If this fails: If Prometheus is not responding to API calls, check the Prometheus pod directly: kubectl logs -n monitoring -l app=prometheus --tail=50

Step 6: Check Grafana Logs for Errors

Why: When all else fails, Grafana logs reveal data source connection failures, plugin errors, and query parsing issues that do not surface in the UI. 

# Check recent Grafana logs
kubectl logs -n monitoring deploy/grafana --tail=100

# Filter for errors specifically
kubectl logs -n monitoring deploy/grafana --tail=200 | grep -E 'error|Error|ERRO|failed|Failed'

# Check for data source query errors
kubectl logs -n monitoring deploy/grafana --tail=200 | grep -i 'datasource\|data source\|query'
Expected output (healthy): Logs show normal startup and request handling with no error lines. If this fails: If the Grafana pod is not found by deploy/grafana, find the correct deployment name: kubectl get deploy -n monitoring | grep grafana

Verification

# Confirm data is flowing by running a direct Prometheus query and then checking the panel
curl -s 'http://localhost:9090/api/v1/query?query=<METRIC_FROM_BLANK_PANEL>' | python3 -m json.tool | grep '"status"'
Success looks like: "status": "success" from Prometheus API, and the Grafana panel now shows data after a browser refresh. If still broken: Escalate — see below.

Escalation

Condition Who to Page What to Say
Not resolved in 30 min Observability team "Grafana showing no data for ; Prometheus is healthy; data source test fails with: "
Data loss suspected Observability lead "Possible metrics gap: Grafana has been blank since ; unsure if Prometheus retained data during the outage"
Scope expanding Platform team "All Grafana dashboards blank simultaneously; Prometheus data source returning errors; may be network or auth issue"

Post-Incident

  • Update monitoring if alert was noisy or missing
  • File postmortem if P1/P2
  • Update this runbook if steps were wrong or incomplete
  • Add a synthetic alert rule in Prometheus (absent()) for critical metrics so blank dashboards trigger an alert
  • Store Grafana data source configuration in a ConfigMap or Helm values file (not manual UI changes) to make it auditable
  • Document dashboard UIDs and data source names for each critical dashboard

Common Mistakes

  1. Wrong time range selected: Data exists in Prometheus but the dashboard time range is set to a window with no data (e.g., "Last 15 minutes" when the incident happened 2 hours ago, or "Last 30 days" with retention set to 15 days). Always check the time range first — it takes 5 seconds.
  2. Data source URL wrong after a service rename: If Prometheus was redeployed with a different service name (e.g., prometheus → prometheus-operated), the Grafana data source URL must be updated. Check the exact service name with kubectl get svc -n monitoring.
  3. Dashboard using different label names than what Prometheus exports: Application upgrades often rename labels (e.g., pod_name → pod). A query filtering on the old label name returns empty silently. Always verify the exact label names in Prometheus before assuming the dashboard is correct.
  4. Ignoring the "Test" button result in the data source settings: Grafana's data source test button gives a definitive pass/fail for the connection. If it fails, you have a connection problem. If it passes, the problem is in the query or the data itself.

Cross-References

  • Topic Pack: Grafana and Prometheus Integration (deep background on data source config, PromQL, and dashboard variables)
  • Related Runbook: prometheus-target-down.md — if Prometheus has no data, investigate target scraping first
  • Related Runbook: alert-storm.md — blank dashboards during an incident may coincide with alert storms from the same root cause

Wiki Navigation

Related Content