Log Pipelines — Trivia & Interesting Facts¶
Surprising, historical, and little-known facts about log pipelines.
Splunk's founders realized log data was the most underutilized data in IT¶
Rob Das, Michael Baum, and Erik Swan founded Splunk in 2003 based on the insight that organizations generated vast amounts of machine data in logs but had no good way to search or analyze it. The name "Splunk" was inspired by spelunking (cave exploration) — exploring the dark, unknown depths of machine data. Splunk went public in 2012 and was acquired by Cisco in 2023 for $28 billion.
The Elastic Stack (ELK) became the default log pipeline almost by accident¶
Elasticsearch (2010), Logstash (2009), and Kibana (2013) were three independent open-source projects that happened to work well together. Jordan Sissel created Logstash as a personal project for log shipping, and the community combined it with Elasticsearch for storage and Kibana for visualization. This organic combination became so popular that Elastic co. was built around the stack.
A single misconfigured log pipeline can cost more than the application it monitors¶
Cloud logging costs have become a major operational concern. Organizations routinely report Datadog, Splunk, or CloudWatch bills of $100,000-500,000 per month. In extreme cases, debug-level logging left enabled in production has generated bills exceeding the cost of the application's compute and storage combined. Log volume management is now a dedicated discipline.
Fluentd was created to unify 13 different log formats at Treasure Data¶
Sadayuki Furuhashi created Fluentd in 2011 while at Treasure Data because he was dealing with 13 different log formats from various data sources. Fluentd's plugin architecture — with over 900 community plugins — allowed it to parse, transform, and route virtually any log format. It was accepted as a CNCF graduated project in 2019.
Vector was built in Rust because existing log agents were too resource-hungry¶
Timber.io (later acquired by Datadog) built Vector as a high-performance log agent in Rust because existing agents like Logstash (JVM-based) consumed 500 MB+ of RAM just to forward logs. Vector achieves comparable throughput with 10-50 MB of RAM. The choice of Rust for observability tooling started a trend, with several other log pipeline components being rewritten in Rust.
Syslog is over 40 years old and still the backbone of enterprise logging¶
The syslog protocol was developed by Eric Allman in the early 1980s as part of Sendmail. It was not formally standardized until RFC 3164 (2001) and RFC 5424 (2009). Despite its age and significant limitations (no authentication, no encryption, lossy UDP transport, inconsistent timestamp formats), syslog remains the most widely used log transport protocol in enterprise infrastructure.
Log sampling can reduce costs by 90% with minimal observability loss¶
Organizations that implement intelligent log sampling — keeping 100% of error logs, 10% of warning logs, and 1% of info/debug logs — typically reduce their log storage costs by 80-90% while retaining the ability to detect and debug most issues. This approach, popularized by Honeycomb and other observability vendors, challenges the "store everything" assumption that drove log pipeline costs sky-high.
OpenTelemetry's log signal was the last of the three pillars to be standardized¶
OpenTelemetry standardized traces first (2020), then metrics (2021), and logs last (reaching stability in 2023). Logs were intentionally deprioritized because the ecosystem already had established log collection tools, while tracing and metrics lacked widely adopted open standards. When the OTel log specification finally stabilized, it focused on log correlation with traces rather than replacing existing log collection.
A log pipeline outage is often worse than an application outage¶
When a log pipeline goes down, teams lose visibility into all their systems simultaneously. Multiple organizations have reported incidents where a log pipeline failure caused a cascading operational crisis — engineers could not diagnose application issues because they had no logs. This has led to the practice of treating log infrastructure with the same SLO rigor as production applications.