Infrastructure Forensics — Trivia & Interesting Facts¶
Surprising, historical, and little-known facts about infrastructure forensics and incident investigation.
Most breaches go undetected for 204 days on average¶
According to IBM's Cost of a Data Breach Report 2023, the average time to identify a data breach is 204 days. This means attackers typically have nearly 7 months of undetected access before discovery. Forensic investigations must often reconstruct months of activity, not just the events of a single day.
Log retention policies have destroyed evidence in major investigations¶
Multiple high-profile forensic investigations have been hampered by short log retention policies. CloudTrail logs, syslog, and application logs with 30-90 day retention windows mean that evidence from the initial compromise is often deleted before the breach is even detected. Forensic investigators frequently arrive to find the crime scene has been cleaned by automated log rotation.
Volatile evidence in RAM disappears the moment you power off¶
When a compromised server is discovered, the instinct to "turn it off immediately" destroys volatile evidence: running processes, network connections, loaded kernel modules, and in-memory encryption keys. Forensic best practice is to capture a memory dump first. The Volatility Framework, created in 2007, is the standard tool for analyzing these memory dumps.
File timestamps can be trivially forged by attackers¶
Forensic investigators can't trust filesystem timestamps (created, modified, accessed) because attackers routinely modify them using the touch command or similar tools. Reliable forensic timelines require correlating multiple independent evidence sources: network logs, authentication records, filesystem journals, and DNS query logs.
Container forensics is exponentially harder than VM forensics¶
When a container is terminated, everything inside it disappears — filesystem, processes, network connections, and logs (unless externally persisted). This ephemeral nature makes container forensics extremely difficult. By the time an incident is detected, the compromised container may have been replaced dozens of times by the orchestrator.
The SolarWinds attack was discovered by a suspicious MFA enrollment¶
The SolarWinds supply chain compromise (discovered December 2020) was detected by a Mandiant security analyst who noticed a suspicious MFA device enrollment for a user account. This single anomalous event led to the discovery that Russian intelligence had compromised SolarWinds' build pipeline and infected 18,000 organizations — one of the largest supply chain attacks in history.
Dead-box forensics is becoming obsolete in the cloud era¶
Traditional "dead-box" forensics — taking an offline copy of a hard drive and analyzing it in a lab — is increasingly impossible in cloud environments. You can't physically seize an AWS EC2 instance. Cloud forensics requires EBS snapshots, CloudTrail logs, VPC Flow Logs, and API-level investigation techniques that are fundamentally different from traditional methods.
Chain of custody requirements make cloud forensics legally challenging¶
In legal proceedings, evidence must have an unbroken chain of custody — documented proof that evidence hasn't been tampered with. In cloud environments, proving chain of custody for a disk image that only exists as an API-accessible blob is legally untested territory. This gap has complicated prosecution of cloud-based crimes.
eBPF has become the most powerful forensic tool on modern Linux¶
Extended Berkeley Packet Filter (eBPF) allows forensic investigators to attach probes to running kernel functions without modifying the system. Tools like bpftrace, Tracee, and Falco use eBPF to observe system calls, network activity, and file access in real-time. eBPF-based forensics provides a level of visibility that was previously impossible without custom kernel modules.
The first 48 hours of an incident determine the investigation's success¶
Forensic investigators call the first 48 hours the "golden window" — analogous to the concept in criminal investigations. During this period, volatile evidence is still available, logs haven't rotated, and attacker infrastructure may still be active. Organizations that have practiced incident response playbooks recover evidence in this window; those without practiced playbooks usually miss it.