Skip to content

Log Pipelines: From printf to Dashboard

  • lesson
  • application-logging
  • structured-vs-unstructured-logs
  • syslog
  • journald
  • log-shippers-(fluentd
  • fluent-bit
  • vector
  • logstash)
  • kubernetes-logging-architecture
  • centralized-logging-(elk/efk/loki)
  • log-parsing-and-enrichment
  • backpressure
  • cardinality
  • log-retention
  • cost-management ---# Log Pipelines — From printf to Dashboard

Topics: application logging, structured vs unstructured logs, syslog, journald, log shippers (Fluentd, Fluent Bit, Vector, Logstash), Kubernetes logging architecture, centralized logging (ELK/EFK/Loki), log parsing and enrichment, backpressure, cardinality, log retention, cost management Level: L1-L2 (Foundations to Operations) Time: 75-90 minutes Prerequisites: None (everything explained from scratch)

The Mission

Your team just got access to the new Grafana Loki dashboard. Someone types a query:

{namespace="production", app="checkout-service"} |= "payment failed"

Results: zero lines. But customers are complaining about failed payments right now. The checkout service is definitely logging errors — you can SSH into a pod and see them scrolling past in kubectl logs. The logs exist. They're just not making it to the dashboard.

Some logs from other services show up fine. The ingress controller logs are there. The auth service logs are there. But checkout, inventory, and the notification service? Gone.

Your job: figure out where the logs are disappearing, understand every hop between fmt.Println("payment failed") in the application code and the Grafana query result, and fix the pipeline. Along the way, you'll learn how the entire log pipeline works — from the printf in your code to the pixel on your screen.

Part 1: The Life of a Log Line

Before we debug anything, let's trace what should happen to a single log line. This is the path every log takes from birth to dashboard:

Application code
    │
    ▼
stdout / stderr  (or syslog, or direct file write)
    │
    ▼
Container runtime (containerd / CRI-O)
    │
    ▼
Node filesystem: /var/log/containers/<pod>_<namespace>_<container>-<id>.log
    │
    ▼
Log shipper (Fluent Bit / Vector / Promtail DaemonSet)
    │
    ▼
[Optional] Aggregator (Fluentd / Vector)
    │
    ▼
Parse → Enrich → Filter → Route
    │
    ▼
Storage backend (Elasticsearch / Loki / S3)
    │
    ▼
Query engine → Dashboard (Kibana / Grafana)

That's 7-8 hops. A failure at any hop silently kills your logs. Let's walk through each one.

The printf

It starts in application code. Here's a Go service logging a payment failure:

log.Printf("payment failed: user=%s amount=%.2f err=%v", userID, amount, err)

That writes a free-text string to stdout. Simple. But already, a choice has been made — this is unstructured logging. The alternative:

slog.Error("payment failed",
    "user_id", userID,
    "amount", amount,
    "error", err.Error(),
    "trace_id", span.SpanContext().TraceID().String(),
)
// Output: {"time":"2026-03-23T14:22:01Z","level":"ERROR","msg":"payment failed","user_id":"u-8842","amount":29.99,"error":"card declined","trace_id":"abc123def456"}

That's structured logging — JSON with fields already extracted. The difference matters enormously downstream.

Mental Model: Think of unstructured logs as handwritten letters and structured logs as spreadsheet rows. Both contain the same information, but one requires a human to interpret and the other is immediately machine-readable. Every regex parser in your pipeline exists because someone chose to write a letter instead of filling out the form.

Approach Parse cost Maintainability Query power
Unstructured (log.Printf) High (regex) Fragile — format changes break parsers grep only
Structured (JSON) Near zero Resilient — new fields are additive Field-level queries

Remember: "Structure at the source, parse at the edge." The cheapest place to add structure is in application code. The most expensive is in the pipeline. Every regex parser is technical debt.

Part 2: Where stdout Actually Goes in Kubernetes

Here's the part most people skip. Your application writes to stdout. Then what?

Step 1: Container runtime captures stdout

When a container writes to stdout, the container runtime (containerd on most modern clusters) captures the output through the container's logging driver. containerd wraps each line in a CRI log format:

2026-03-23T14:22:01.456789012Z stdout F {"level":"ERROR","msg":"payment failed","user_id":"u-8842"}

That prefix — timestamp, stream name (stdout or stderr), and a flag (F for full line, P for partial) — is added by the runtime, not your application.

Step 2: Written to the node filesystem

containerd writes these wrapped lines to a file on the node, not inside the container:

# On the Kubernetes node:
ls /var/log/containers/
# checkout-service-7b8f9c4d6e-x2k4n_production_checkout-abc123def456.log

# The file is actually a symlink:
ls -la /var/log/containers/checkout-service-7b8f9c4d6e-x2k4n_production_checkout-abc123def456.log
# → /var/log/pods/production_checkout-service-7b8f9c4d6e-x2k4n_uid123/checkout/0.log

# What's inside:
head -1 /var/log/containers/checkout-service-7b8f9c4d6e-x2k4n_production_checkout-abc123def456.log
# 2026-03-23T14:22:01.456789012Z stdout F {"level":"ERROR","msg":"payment failed","user_id":"u-8842"}

Under the Hood: The symlink chain is: /var/log/containers/<pod>_<ns>_<container>-<id>.log -> /var/log/pods/<ns>_<pod>_<uid>/<container>/0.log. The 0.log is the current log file; when containerd rotates it, 0.log becomes 0.log.20260323-142201 and a new 0.log is created. The container runtime handles this rotation, not logrotate. containerd's default is 10MB per file, 5 files. That's 50MB per container before old logs are discarded.

Step 3: Log shipper reads from the node

A DaemonSet (one pod per node) tails these files and ships the lines to the central storage backend. This is where Fluent Bit, Vector, or Promtail enters the picture.

Node 1:  [Fluent Bit pod] → reads /var/log/containers/*.log → ships to Loki
Node 2:  [Fluent Bit pod] → reads /var/log/containers/*.log → ships to Loki
Node 3:  [Fluent Bit pod] → reads /var/log/containers/*.log → ships to Loki

The shipper needs to: 1. Discover new log files as pods start 2. Track its read position (so it doesn't re-read after restart) 3. Parse the CRI format to extract the actual log message 4. Enrich with Kubernetes metadata (pod name, namespace, labels) 5. Buffer logs when the destination is slow 6. Ship to the backend

That's a lot of moving parts. Any one of them can fail silently.

Flashcard Check: Kubernetes Logging

Question Answer (cover this column)
Where does kubectl logs actually read from? The container runtime's log files on the node (/var/log/pods/), or the kubelet API which reads those same files
Why is /var/log/containers/ full of symlinks? The symlinks provide a consistent, flat naming convention (<pod>_<ns>_<container>-<id>.log) pointing to the actual files organized by pod under /var/log/pods/
What happens to container logs when a pod is deleted? The log files on the node are deleted by the kubelet's garbage collection. If the shipper hasn't read them yet, those logs are lost forever
What's the default log rotation for containerd? 10MB per file, 5 files per container (50MB total). Configurable in containerd's config

Part 3: The Shipper Showdown

Four tools dominate the log shipping space. Each exists because the others had a shortcoming someone couldn't live with.

The Comparison

Fluent Bit Fluentd Vector Logstash
Language C Ruby + C Rust Java (JVM)
Memory footprint ~2-5 MB ~40-100 MB ~10-50 MB ~500 MB+
Throughput (JSON) ~80K events/s ~20K events/s ~100K+ events/s ~30K events/s
Plugin ecosystem ~100 ~900+ ~50 sources/sinks ~200+
Config format INI-style XML-like TOML/YAML Custom DSL
Primary role Edge agent Aggregator Both Aggregator
CNCF status Graduated Graduated N/A (Datadog) N/A (Elastic)
Transform language Limited Lua Ruby plugins VRL (powerful) Ruby-like filters

Name Origin: Fluentd was created by Sadayuki Furuhashi at Treasure Data in 2011 because he was dealing with 13 different log formats. The name evokes "fluent" data flow. Fluent Bit was created by Eduardo Silva at Treasure Data in 2015 as a lightweight C rewrite for resource-constrained environments — the "Bit" signals its smaller size. Vector was created by Timber Technologies (later acquired by Datadog) in 2019, built in Rust because existing agents were too resource-hungry. Logstash was a personal project by Jordan Sissel in 2009, predating the others.

When to Pick What

Kubernetes DaemonSet on every node?        → Fluent Bit or Vector
Central aggregation with complex routing?  → Fluentd or Vector
Already running the Elastic stack?         → Logstash (or switch to Fluent Bit)
Greenfield, maximum performance?           → Vector
Smallest possible memory footprint?        → Fluent Bit
Need 900 plugins for exotic destinations?  → Fluentd

The most common pattern in production Kubernetes: Fluent Bit on every node (DaemonSet), forwarding to Fluentd or Vector as a central aggregator. The edge agent is lightweight, the aggregator handles the heavy transformation and routing.

Trivia: Logstash on the JVM consumes 500 MB+ of RAM just to forward logs. Vector achieves comparable throughput with 10-50 MB. This disparity is why the Elastic Stack community increasingly replaces Logstash with Fluent Bit for log collection, keeping Elasticsearch and Kibana but swapping out the "L" in "ELK."

A Real Fluent Bit Config for Kubernetes

This is what runs on every node in a typical cluster:

# /etc/fluent-bit/fluent-bit.conf

[SERVICE]
    Flush         5
    Log_Level     info
    Daemon        off
    Parsers_File  parsers.conf
    HTTP_Server   On
    HTTP_Listen   0.0.0.0
    HTTP_Port     2020

[INPUT]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    Parser            cri
    DB                /var/log/flb_kube.db
    Mem_Buf_Limit     15MB
    Skip_Long_Lines   On
    Refresh_Interval  10

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
    Merge_Log           On
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On

[OUTPUT]
    Name            loki
    Match           kube.*
    Host            loki-gateway.monitoring.svc.cluster.local
    Port            80
    Labels          job=fluent-bit
    Label_Keys      $kubernetes['namespace_name'],$kubernetes['pod_name'],$kubernetes['container_name']
    Retry_Limit     5

Let's break down the critical settings:

Setting What It Does Why It Matters
DB /var/log/flb_kube.db Tracks file read positions in an SQLite database Without this, Fluent Bit starts from the end of files after restart — losing every log written while it was down
Mem_Buf_Limit 15MB Caps memory usage per input Without this, a traffic spike OOM-kills the Fluent Bit pod
Merge_Log On Parses the JSON inside the CRI log wrapper Without this, your entire structured log is trapped inside a log field as an escaped string
Retry_Limit 5 Retries failed sends 5 times Retry_Limit False means retry forever, which protects against data loss but risks filling the buffer

Gotcha: Mem_Buf_Limit in Fluent Bit defaults to unlimited. Fluentd's total_limit_size also defaults to unlimited. Both will happily consume all available RAM during a downstream outage. Set explicit limits on day one.

Part 4: The Disappearing Logs — Debugging the Pipeline

Back to our mission. Checkout service logs aren't reaching Loki. Let's investigate, thinking through each hop.

OK, let's start from the application and work forward. Is the checkout service actually writing logs?

kubectl logs checkout-service-7b8f9c4d6e-x2k4n -n production --tail=5
# 2026-03-23T14:22:01Z ERROR payment failed user_id=u-8842 amount=29.99
# 2026-03-23T14:22:01Z ERROR payment failed user_id=u-9103 amount=14.50
# 2026-03-23T14:22:02Z INFO  health check passed

Logs are there. So the application is writing to stdout and the kubelet can read them. The break is somewhere between the node filesystem and Loki.

Next hop: is Fluent Bit running on that node?

# Which node is the pod on?
kubectl get pod checkout-service-7b8f9c4d6e-x2k4n -n production -o wide
# NODE: worker-3

# Is the Fluent Bit DaemonSet pod running on worker-3?
kubectl get pods -n logging -l app=fluent-bit -o wide | grep worker-3
# fluent-bit-7kd4f   1/1  Running  0  3d  worker-3

Fluent Bit is running. Let's check its logs for errors.

kubectl logs fluent-bit-7kd4f -n logging --tail=20
# [2026/03/23 14:20:15] [warn] [output:loki:loki.0] retry_limit: 5 reached, dropping 128 records
# [2026/03/23 14:20:20] [warn] [output:loki:loki.0] retry_limit: 5 reached, dropping 256 records
# [2026/03/23 14:21:01] [error] [output:loki:loki.0] HTTP status=429 Too Many Requests

There it is. Loki is returning HTTP 429 — rate limiting. Fluent Bit retries 5 times, then drops the records. Let's check why Loki is rate limiting.

kubectl logs -l app=loki -n monitoring --tail=10
# level=warn msg="per-tenant streams limit exceeded" user=fake tenant=production limit=5000 actual=8247

Per-tenant stream limit exceeded. 8,247 streams when the limit is 5,000. That's a cardinality problem.

Part 5: The Cardinality Bomb

This is one of the most expensive mistakes in log infrastructure, and it deserves its own section.

What Cardinality Means for Logs

In Loki, a stream is a unique combination of labels. Every distinct label set creates a new stream:

{namespace="production", app="checkout", pod="checkout-7b8f9c4d6e-x2k4n"}   ← Stream 1
{namespace="production", app="checkout", pod="checkout-7b8f9c4d6e-m3j8p"}   ← Stream 2
{namespace="production", app="checkout", pod="checkout-7b8f9c4d6e-q9w2r"}   ← Stream 3

Three pods, three streams. Manageable. But if someone adds request_id as a label:

{namespace="production", app="checkout", request_id="req-00001"}   ← Stream 1
{namespace="production", app="checkout", request_id="req-00002"}   ← Stream 2
{namespace="production", app="checkout", request_id="req-00003"}   ← Stream 3
...
{namespace="production", app="checkout", request_id="req-99999"}   ← Stream 99,999

Every unique request creates a new stream. At 100 requests per second, you create 100 new streams per second. Loki's index explodes. Queries time out. The ingester runs out of memory.

The same applies to Elasticsearch, just with a different mechanism — high-cardinality fields in ES create massive inverted indices that consume heap and slow queries.

Mental Model: Labels are for categorization, not identification. A label's value should have at most tens to low hundreds of unique values. Namespace, app, environment, log level — those are labels. Request ID, user ID, trace ID — those belong inside the log line, queried with filter expressions like |= "req-00001" in LogQL or full-text search in ES.

The Fix

Check what labels are being sent:

# What labels is Fluent Bit attaching?
kubectl get configmap fluent-bit-config -n logging -o yaml | grep -A5 Label_Keys
# Label_Keys $kubernetes['namespace_name'],$kubernetes['pod_name'],$kubernetes['container_name'],$kubernetes['labels']['request_id']

Someone added request_id from the pod's Kubernetes labels to the Loki label set. That's the cardinality bomb.

Remove the high-cardinality label from the Fluent Bit config, keep only bounded labels:

[OUTPUT]
    Name            loki
    Match           kube.*
    Host            loki-gateway.monitoring.svc.cluster.local
    Port            80
    Labels          job=fluent-bit
    Label_Keys      $kubernetes['namespace_name'],$kubernetes['container_name']
    # Removed: $kubernetes['labels']['request_id']   ← THIS WAS THE BOMB
    # Also removed pod_name — with 50 replicas, that's 50x streams per service

Gotcha: Pod name as a Loki label seems harmless — every pod has a name. But if your deployment has 50 replicas and does rolling updates daily, you create 50+ new streams per day per service. With 30 services, that's 1,500 new streams daily that Loki must track. Use container_name (bounded) instead of pod_name (unbounded across time).

The Same Problem in Elasticsearch

In ES, the equivalent is mapping explosion. Every unique field in a JSON document creates a mapping entry. If your logs contain:

{"metadata": {"pod-abc123": {"cpu": "100m"}}}

Each pod name becomes a field name in the mapping. ES has a default limit of 1,000 fields per index. Once you hit it, new documents with new pod names are rejected.

# Check field count in an index
curl -s localhost:9200/app-logs-2026.03.23/_mapping | python3 -c "
import json, sys
mapping = json.load(sys.stdin)
fields = list(mapping.values())[0]['mappings']['properties']
print(f'Field count: {len(fields)}')"

Trivia: Organizations routinely report Datadog, Splunk, or CloudWatch logging bills of $100,000-500,000 per month. In extreme cases, debug-level logging with high-cardinality labels has generated bills exceeding the cost of the application's compute and storage combined. Log volume management is now a dedicated engineering discipline.

Flashcard Check: Cardinality

Question Answer (cover this column)
What is a "stream" in Loki? A unique combination of label key-value pairs. Each distinct label set is one stream
Why is request_id a dangerous Loki label? It creates a new stream per request — millions of streams — bloating the index and degrading all queries
What's the rule of thumb for Loki label values? Each label should have at most 10-15 unique values (namespace, environment, app name). Unbounded values belong in the log content
How does high cardinality hurt Elasticsearch differently? In ES, unbounded field names cause mapping explosion (default limit: 1,000 fields per index). Unbounded field values cause massive inverted indices that consume JVM heap

Part 6: Backpressure — When the Pipeline Fights Back

War Story: A team at a mid-size SaaS company deployed a new microservice that logged every incoming request payload at DEBUG level. At 2,000 requests/second, each averaging 4KB, that's 8 MB/second of raw log data. The Fluent Bit DaemonSet on those nodes hit its Mem_Buf_Limit, started dropping logs from all containers on the node (not just the noisy one), and the team lost visibility into 15 other services during a payment processing outage the same afternoon. The noisy service's logs were never even needed — nobody queried them. But the silence from the other 15 services cost them 45 minutes of blind debugging.

This is backpressure. When the destination (Loki, Elasticsearch) is slower than the source (your applications), pressure builds in the pipeline.

Normal:     App (100 events/s) → Buffer → Destination (handles 100/s)     ✓

Trouble:    App (100 events/s) → Buffer → Destination (handles 50/s)      ✗
                                    ↑
                              Buffer filling at 50 events/s
                              Full in: buffer_size / 50

When the buffer fills, one of three things happens:

Strategy What Happens Tradeoff
Block Source is paused — logs queue in the application Safest for data, but if the app blocks on stdout writes, request latency increases
Drop oldest Oldest buffered logs are discarded Lose historical context but keep recent data
Drop newest New incoming logs are discarded Keep the historical data but lose the most recent context (usually worse)

The Two-Tier Buffer

Production pipelines use a two-tier approach:

Tier 1: Memory buffer (fast, limited)
    Size: 10-64 MB
    Purpose: Handle normal traffic bursts
    Speed: Nanosecond access

    ↓ overflow ↓

Tier 2: File buffer (slower, larger)
    Size: 1-4 GB on disk
    Purpose: Survive extended destination outages
    Benefit: Survives agent restarts (memory doesn't)

    ↓ both full ↓

Decision: Block the source or drop logs

# Vector: two-tier buffering
[sinks.loki]
type = "loki"
inputs = ["kubernetes_logs"]
endpoint = "http://loki:3100"

[sinks.loki.buffer]
type = "disk"                  # file-backed buffer
max_size = 2_147_483_648       # 2 GB
when_full = "block"            # block source when full (vs "drop_newest")

Under the Hood: When Fluent Bit's Mem_Buf_Limit is reached on a particular input, it pauses reading from that input's file. The kernel continues buffering the application's stdout writes in a pipe buffer (default 64KB on Linux). If that fills too, the application's write() syscall blocks — meaning your HTTP handler stalls mid-request waiting for its log line to be written. This is how a slow log destination causes application latency. The chain: slow destination -> full pipeline buffer -> paused file reader -> full pipe buffer -> blocked write() -> slow HTTP response.

Part 7: What Happens Before Kubernetes — journald and syslog

Not everything runs in Kubernetes. System daemons, SSH, cron jobs, and kernel events all log through the traditional Linux logging stack. Even on Kubernetes nodes, understanding this stack matters because the kubelet itself logs through journald.

The Log Flow on a Linux System

Application calls syslog()     Systemd service writes to stdout
         │                              │
         ▼                              ▼
    /dev/log socket                  journald
         │                       (binary, structured)
         ▼                              │
      rsyslog ◀─────────────────────────┘
    (text files)                  ForwardToSyslog=yes
         │
         ▼
    /var/log/syslog
    /var/log/auth.log
    /var/log/kern.log
         │
         ▼
      logrotate
    (compress, delete old)

Name Origin: The word "log" comes from maritime navigation. Ships measured speed by throwing a wooden log overboard and timing how long the rope took to pay out. The record of measurements was kept in the "log book." The nautical term dates back to at least the 16th century. Computer logging inherited both the term and the concept — a sequential record of events, ordered by time, for later analysis.

Syslog: 40+ Years Old, Still Everywhere

The syslog protocol was created by Eric Allman in the early 1980s as part of Sendmail. It was not formally standardized until RFC 3164 (2001) and RFC 5424 (2009). Despite no authentication, no encryption, and lossy UDP transport, syslog remains the most widely used log transport protocol in enterprise infrastructure.

Every syslog message has two properties:

Facility (what generated it):
  kern (0), user (1), mail (2), daemon (3), auth (4), local0-7 (16-23)

Severity (how bad is it):
  0=EMERG  1=ALERT  2=CRIT  3=ERR  4=WARNING  5=NOTICE  6=INFO  7=DEBUG

# Send a test syslog message
logger -p local0.err -t myapp "Something went wrong"

# Check it arrived
journalctl -t myapp --since "1 minute ago"

# Or in the traditional file
tail -1 /var/log/syslog
# Mar 23 14:30:01 webserver01 myapp: Something went wrong

journald: Structured Binary Logs

systemd's journal captures everything from systemd-managed services with rich metadata. The binary format was one of systemd's most controversial changes — critics argued binary logs are harder to inspect when the system is broken (exactly when you need logs most).

# Follow a service's logs in real time
journalctl -u kubelet.service -f

# Show JSON output — perfect for piping to jq
journalctl -u kubelet.service -o json --since "5 min ago" | jq '{
    timestamp: .__REALTIME_TIMESTAMP,
    priority: .PRIORITY,
    message: .MESSAGE
}'

# View all metadata for one log entry
journalctl -u kubelet.service -o verbose -n 1
# Shows: _PID, _UID, _EXE, _HOSTNAME, PRIORITY, _TRANSPORT, MESSAGE...

Gotcha: journald has built-in rate limiting: by default 10,000 messages per 30 seconds per service. If a service logs faster than that, journald silently drops messages and writes one "Suppressed N messages" entry. Check with: journalctl -u myservice | grep -i suppress. Disable for critical services in /etc/systemd/journald.conf with RateLimitBurst=0.

Part 8: Centralized Logging — ELK, EFK, and Loki

Three stacks dominate centralized logging. Each represents a different philosophy.

ELK: Elasticsearch + Logstash + Kibana

The original. Indexes the full text of every log line. Powerful full-text search but expensive to operate.

App → Logstash (parse/enrich) → Elasticsearch (index/store) → Kibana (search/visualize)

When to use: You need to search inside log messages with complex queries, faceted search, or you already have Elasticsearch for other purposes (application search, APM).

EFK: Elasticsearch + Fluentd/Fluent Bit + Kibana

Same backend, lighter shipper. Replaced the "L" (Logstash) with Fluentd or Fluent Bit because Logstash's JVM overhead was painful on every node.

Loki + Promtail/Fluent Bit + Grafana

The newcomer. Indexes only labels, not log content. Stores log chunks in cheap object storage (S3, GCS).

App → Fluent Bit (ship) → Loki (label index + object store) → Grafana (LogQL queries)

Name Origin: Loki is named after the Norse trickster god. The name signals that Loki is "lighter" than Elasticsearch, just as the trickster Loki is lighter than Thor. Grafana Labs deliberately chose the lighter-weight approach: index less, store cheaper, query differently.

The Cost Equation

Elasticsearch Loki
Indexes Full text of every log line Only labels (namespace, app, level)
Storage SSD/NVMe for hot data Object storage (S3/GCS) for everything
Query model Inverted index, instant full-text search Grep-like scan of chunks, filtered by labels first
Cost per GB ingested $$$$ $
Query speed (specific string) Fast — indexed Slower — must scan chunks within label range
Query speed (label filter) Fast Fast
Operational complexity High (JVM tuning, shard management, capacity planning) Lower (stateless queriers, object storage)

Trivia: The Elastic Stack (ELK) became the default log pipeline almost by accident. Elasticsearch (2010), Logstash (2009), and Kibana (2013) were three independent open-source projects that happened to work well together. Jordan Sissel created Logstash as a personal project, and the community combined it with Elasticsearch for storage and Kibana for visualization. The organic combination became so popular that Elastic NV was built around the stack.

grep vs Structured Queries

On a single server, grep is king:

grep "payment failed" /var/log/app.log | grep "user_id=u-8842"

Fast, no infrastructure needed. But grep doesn't scale to 500 servers writing 10GB/day each. That's 5TB/day. You can't grep 5TB in real time.

Centralized logging trades infrastructure cost for query power:

# Loki (LogQL):
{app="checkout", namespace="production"} |= "payment failed" | json | user_id = "u-8842"

# Elasticsearch (KQL in Kibana):
service: "checkout" AND message: "payment failed" AND user_id: "u-8842"

Both return results in seconds across terabytes of data. The difference: you're paying for the infrastructure to make that possible.

Part 9: Log Retention, Rotation, and the Money Problem

The Retention Matrix

Not all logs deserve the same treatment:

Error + Fatal logs     → Hot storage (ES/Loki), 30-day retention
Warning + Info logs    → Warm storage, 14-day retention
Debug logs             → Cold storage (S3) or /dev/null in production
Security / Audit logs  → Immutable archive, 1-7 years (regulatory)
Access logs            → Hot 7 days, archive 1 year

Routing by Level in the Pipeline

This is where log routing saves money:

# Vector: route by severity
[transforms.route_by_level]
type = "route"
inputs = ["kubernetes_logs"]
route.errors = '.level == "error" || .level == "fatal"'
route.info = '.level == "info" || .level == "warn"'
route.debug = '.level == "debug"'

[sinks.errors_to_loki]
type = "loki"
inputs = ["route_by_level.errors"]
endpoint = "http://loki:3100"
labels.level = "error"

[sinks.info_to_s3]
type = "aws_s3"
inputs = ["route_by_level.info"]
bucket = "log-archive-production"
key_prefix = "info/%Y/%m/%d/"
compression = "gzip"

[sinks.debug_to_null]
type = "blackhole"
inputs = ["route_by_level.debug"]
# Debug logs in production: gone. Save your money.

Trivia: Organizations that implement intelligent log sampling — keeping 100% of error logs, 10% of warning logs, and 1% of info/debug logs — typically reduce storage costs by 80-90% while retaining the ability to detect and debug most issues. This challenges the "store everything" assumption that drove log pipeline costs sky-high.

Log Rotation on the Node

Even with a centralized pipeline, node-level log rotation matters. If the shipper falls behind and rotation deletes the file before it's read, those logs are gone.

# Check containerd's log rotation settings
cat /etc/containerd/config.toml | grep -A5 max_container_log

# Docker's equivalent
cat /etc/docker/daemon.json
# {
#   "log-driver": "json-file",
#   "log-opts": {
#     "max-size": "50m",
#     "max-file": "5"
#   }
# }

Gotcha: If you set container log rotation too aggressively (e.g., 5MB, 2 files) and your shipper has a 5-minute outage, high-volume containers can rotate through all their log files before the shipper catches up. When the shipper comes back, those logs are gone. Size your rotation to survive your expected worst-case shipper downtime.

Part 10: Sampling High-Volume Logs

At scale, you can't keep everything. A service handling 50,000 requests/second generates around 4GB of logs per hour at one line per request. That's 96GB/day from a single service.

Head-based Sampling

Simple: keep every Nth log line.

# Vector: sample 10% of info logs
[transforms.sample_info]
type = "sample"
inputs = ["route_by_level.info"]
rate = 10    # Keep 1 in 10

Tail-based (Smart) Sampling

Keep all logs from error/slow requests, sample the rest:

# Vector: keep all errors, sample successes
[transforms.smart_sample]
type = "filter"
inputs = ["parsed_logs"]
condition = '''
  .level == "error" ||
  .level == "fatal" ||
  .duration_ms > 5000 ||
  to_int(.status_code) ?? 0 >= 500 ||
  to_float(to_string(get!(., ["_metadata", "sample_key"]))) ?? 1.0 < 0.1
'''
# Keeps: all errors, all slow requests (>5s), all 5xx, plus 10% of everything else

Mental Model: Sampling is insurance math. You don't need every data point to detect a pattern — you need enough data points. If 2% of requests fail, a 10% sample still gives you 0.2% failure rate with enough volume to alert on. What you lose is the ability to find one specific request's logs. For that, use distributed tracing with trace IDs, not log sampling.

Part 11: Putting It All Together — The Fix

Back to our mission. Let's resolve the checkout service log disappearance.

Root cause chain:

  1. Someone added request_id as a Loki label in Fluent Bit's config
  2. This created thousands of streams per minute
  3. Loki's per-tenant stream limit (5,000) was exceeded
  4. Loki returned HTTP 429 to Fluent Bit
  5. Fluent Bit retried 5 times, then dropped the records
  6. All logs from high-traffic services (checkout, inventory, notification) disappeared
  7. Low-traffic services (ingress, auth) stayed under the stream limit and kept working

The fix (in order):

# 1. Remove the high-cardinality label from Fluent Bit config
kubectl edit configmap fluent-bit-config -n logging
# Remove $kubernetes['labels']['request_id'] from Label_Keys

# 2. Restart Fluent Bit to pick up the new config
kubectl rollout restart daemonset fluent-bit -n logging

# 3. Verify streams dropped back to normal
kubectl port-forward svc/loki-gateway 3100:80 -n monitoring &
curl -s http://localhost:3100/loki/api/v1/label | jq '.data | length'
# Should show a manageable number of label names

# 4. Verify logs are flowing again
# In Grafana, run:
# {namespace="production", container="checkout"} |= "payment"
# Results should appear within 30 seconds of the rollout completing

Exercises

Exercise 1: Trace the Path (5 minutes)

On any Kubernetes cluster, trace a log line from application to node filesystem.

# Deploy a test pod that logs
kubectl run log-test --image=busybox -- sh -c 'while true; do echo "{\"level\":\"info\",\"msg\":\"hello from log-test\",\"ts\":\"$(date -u +%FT%TZ)\"}"; sleep 5; done'

# Find which node it's on
kubectl get pod log-test -o wide

# SSH to that node (or use kubectl debug) and find the log file
# Look in /var/log/containers/ for log-test*
# Read the raw file — notice the CRI wrapper around your JSON
What to look for The file at `/var/log/containers/log-test_default_log-test-.log` should contain lines like: 
2026-03-23T15:00:00.123456789Z stdout F {"level":"info","msg":"hello from log-test","ts":"2026-03-23T15:00:00Z"}
The CRI prefix (`timestamp stdout F`) is added by containerd. Your original JSON is after the `F` flag.

Exercise 2: Build a Two-Stage Pipeline (30 minutes)

Using Docker Compose, build a pipeline:

  1. An app container that writes JSON logs to stdout
  2. Fluent Bit reading from Docker's log files
  3. Fluent Bit shipping to a local Loki instance
  4. Grafana querying Loki
Hint: docker-compose.yml skeleton 
services:
  app:
    image: busybox
    command: sh -c 'while true; do echo "{\"level\":\"info\",\"msg\":\"test\",\"counter\":$$((i=i+1))}"; sleep 1; done'

  loki:
    image: grafana/loki:2.9.0
    ports: ["3100:3100"]

  fluent-bit:
    image: fluent/fluent-bit:2.2
    volumes:
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
    depends_on: [loki]

  grafana:
    image: grafana/grafana:10.0.0
    ports: ["3000:3000"]
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
    depends_on: [loki]
You'll need to write the `fluent-bit.conf` that reads Docker container logs and ships to Loki.

Exercise 3: Cause and Fix a Cardinality Bomb (20 minutes)

Using the pipeline from Exercise 2:

  1. Modify the app to include a unique request_id in each log line
  2. Configure Fluent Bit to add request_id as a Loki label
  3. Watch Loki's stream count climb (check /loki/api/v1/label)
  4. Fix the config to keep request_id in the log content but not as a label
  5. Query for a specific request_id using LogQL: {app="test"} |= "req-00042"

Cheat Sheet

Pipeline Debugging Ladder

1. Is the app logging?        → kubectl logs <pod>
2. Is the shipper running?    → kubectl get pods -n logging -l app=fluent-bit
3. Is the shipper erroring?   → kubectl logs <fluent-bit-pod> -n logging
4. Is the destination up?     → curl http://loki:3100/ready  (or ES /_cluster/health)
5. Is the buffer full?        → curl http://fluent-bit:2020/api/v1/metrics/prometheus | grep -E 'dropped|retry'
6. Is cardinality exploding?  → curl http://loki:3100/loki/api/v1/label | jq '.data | length'

Shipper Quick Reference

Task Fluent Bit Vector
Validate config fluent-bit -c config.conf --dry-run vector validate config.toml
Test with stdout Add [OUTPUT] Name stdout Match * Add [sinks.debug] type = "console"
Check metrics curl localhost:2020/api/v1/metrics curl localhost:8686/health
Position tracking DB /path/to/positions.db data_dir = "/var/lib/vector"
Memory limit Mem_Buf_Limit 15MB (per input) buffer.max_size = 268435456 (per sink)

Log Level Routing Rules of Thumb

ERROR/FATAL  → Always ship, always retain (30+ days hot)
WARN         → Ship, retain 14 days hot
INFO         → Ship or sample, retain 7 days (warm/cold)
DEBUG        → Drop in production, or sample 1-10%

Key Metrics for Your Pipeline

fluentbit_input_records_total          — how many logs the shipper is reading
fluentbit_output_retries_total         — destination failures
fluentbit_output_dropped_records_total — logs lost (this is the scary one)
fluentbit_filter_emit_records_total    — logs surviving filters

Takeaways

  1. A log line crosses 7+ boundaries between your code and the dashboard. A silent failure at any hop means lost logs. Monitor the pipeline itself, not just your applications.

  2. Structure at the source, parse at the edge. JSON from the application eliminates an entire category of pipeline failures. Every regex parser you add is a maintenance burden and a throughput bottleneck.

  3. Labels are for categorization, not identification. In Loki, high-cardinality labels (request IDs, user IDs) create a stream explosion that rate-limits your entire pipeline. In Elasticsearch, unbounded field names cause mapping explosion. Put high-cardinality data inside the log line, not in the metadata.

  4. Backpressure is not hypothetical. When the destination is slow, the pipeline fills, and either your logs are dropped or your application blocks on stdout writes. Set explicit buffer limits and overflow policies on day one.

  5. Not all logs deserve the same storage. Route errors to hot storage, info to warm, debug to /dev/null. The difference between a $5K/month and $50K/month logging bill is almost always debug logs in expensive storage.

  6. The pipeline that drops logs silently is worse than one that blocks your app. At least when the app slows down, you notice. Silent log loss is discovered during the incident investigation, when it's too late.