Log Pipelines¶
17 cards — 🟢 3 easy | 🟡 4 medium | 🔴 3 hard
🟢 Easy (3)¶
1. Why is structured logging (JSON) preferred over unstructured (free-text) logs in a pipeline?
Show answer
Structured logs have fields already extracted as key-value pairs, so no parsing is needed. Unstructured logs require regex parsing, which is slow, fragile, and breaks when the log format changes.Remember: Structured logs (JSON) are machine-parseable. Always prefer for new apps.
2. What are the four stages of a log pipeline?
Show answer
Sources (app logs, system logs, container stdout), Collection (Fluentbit, Vector, Filebeat), Processing (parse, filter, enrich, route, buffer), and Destinations (Elasticsearch, S3, Loki, Datadog).Remember: Pipeline: Collect→Parse→Enrich→Route→Store→Query. "CPERSQ."
3. What is log routing, and why is it important?
Show answer
Log routing sends different log types to different destinations based on tags or labels. For example, security/auth logs go to a SIEM with long retention, application errors go to Elasticsearch, and debug logs may go to /dev/null in production. It controls cost and ensures the right logs are in the right place.Remember: Good log pipeline: collect at source, parse centrally, store efficiently, query fast.
Example: Security logs go to SIEM (7-year retention). App errors go to Elasticsearch (30 days). Debug logs go to /dev/null in prod. Route by severity to control cost.
Analogy: Log routing is like mail sorting — different types go to different destinations based on labels.
🟡 Medium (4)¶
1. When would you choose Fluentbit over Fluentd, and vice versa?
Show answer
Fluentbit is written in C with a tiny memory footprint (~2MB), ideal as a lightweight edge agent on every node (Kubernetes DaemonSet). Fluentd is written in Ruby+C with a rich plugin ecosystem (800+), better suited as a central aggregation layer. A common pattern is Fluentbit on nodes forwarding to Fluentd for central processing.Remember: Fluent Bit=lightweight(C). Fluentd=full(Ruby). Bit on nodes, d as aggregator.
2. What is backpressure in a log pipeline, and what are the three options when the buffer fills up?
Show answer
Backpressure occurs when the destination is slow or down and the buffer fills up. The three options are: Block (source slows down — safe but app may stall), Drop (oldest or newest logs discarded — data loss), or Overflow (write to disk when memory buffer is full — uses more I/O but preserves data).Remember: Pipeline: Collect→Parse→Enrich→Route→Store→Query. "CPERSQ."
3. What distinguishes Vector from Fluentbit and Fluentd?
Show answer
Vector is written in Rust for high performance, is a single binary that handles both collection and aggregation, uses TOML/YAML config, has strong typing with its own transform language (VRL), and is ideal for greenfield deployments. It has fewer plugins than Fluentd but offers better throughput.Remember: Fluent Bit=lightweight(C). Fluentd=full(Ruby). Bit on nodes, d as aggregator.
4. Why is multiline log handling necessary, and what happens without it?
Show answer
Stack traces and multi-line log entries (like Java exceptions) span multiple lines but represent a single logical event. Without multiline handling, each line becomes a separate log entry, making debugging impossible because the stack trace is fragmented across unrelated entries.Remember: Good log pipeline: collect at source, parse centrally, store efficiently, query fast.
🔴 Hard (3)¶
1. What is the two-tier buffer pattern in log pipelines, and what happens when both tiers fill up?
Show answer
The first tier is a memory buffer (fast, limited — e.g., 64MB) for normal traffic. The second tier is a file buffer (slower, larger — e.g., 2GB) that catches overflow from memory and survives process restarts. When both are full, the pipeline must either block (stop accepting new logs, risking app stalls) or drop (discard logs, losing data).Remember: Pipeline: Collect→Parse→Enrich→Route→Store→Query. "CPERSQ."
2. What are the main parsing strategies for logs, and what are the tradeoffs of regex vs JSON parsing?
Show answer
Strategies include regex, JSON, key-value, delimiter/CSV, and Grok patterns. Regex is flexible and handles any format but is slow, hard to maintain, and breaks when the format changes. JSON parsing is fast and reliable but only works if the source emits JSON. The best approach is to emit structured logs at the source and only use regex for things you do not control.Remember: Good log pipeline: collect at source, parse centrally, store efficiently, query fast.
3. What metrics should you monitor on the log pipeline itself, and what does a persistent input-rate-exceeding-output-rate indicate?
Show answer
Key metrics: input rate (events/sec), output rate (events/sec), buffer usage (bytes and % full), retry count, drop count, parse error count, and end-to-end latency. When input rate persistently exceeds output rate, the buffer is filling and will eventually overflow, causing data loss or application stalls. Act before it overflows.Remember: Pipeline: Collect→Parse→Enrich→Route→Store→Query. "CPERSQ."