Skip to content

Pattern: tmpfs Consuming Hidden RAM

ID: FP-007 Family: Resource Exhaustion Frequency: Uncommon Blast Radius: Single Host Detection Difficulty: Subtle

The Shape

tmpfs filesystems (including /tmp, /dev/shm, /run, and container overlay layers) are backed by RAM and swap, not disk. Large files written to these locations consume physical memory, but this consumption is invisible to standard memory monitoring tools that only watch process RSS and slab cache. The host can OOM without any single process showing high memory usage.

How You'll See It

In Linux/Infrastructure

$ free -h
              total        used        free      shared  buff/cache   available
Mem:            16G          2G          1G          8G          5G          6G
              ↑ used looks low but...   ↑ "shared" is tmpfs/shm usage

$ df -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           7.8G  7.5G  300M  97% /dev/shm
The "shared" column in free is the memory used by tmpfs. A process writing to /dev/shm for IPC (databases, video processing, scientific computing) can fill RAM without showing high RSS in ps or top.

In Kubernetes

Kubernetes mounts an emptyDir with medium: Memory as a RAM-backed tmpfs. If the application writes more than expected to this volume, memory limit accounting includes it. Pods get OOMKilled even though container RSS looks normal because tmpfs usage counts against the cgroup memory limit.

In CI/CD

Docker build layer caches, temporary build directories, or test databases written to /tmp during CI fill the tmpfs. The CI node appears to have disk space but runs out of RAM, causing OOM kills of unrelated services on the same host.

The Tell

free -h shows high "shared" column. df -h /dev/shm or df -h /tmp shows high utilization. No single process has high RSS, but the host is memory-constrained.

Common Misdiagnosis

Looks Like But Actually How to Tell the Difference
Memory leak in process tmpfs accumulation Process RSS low; df -h /dev/shm high
Kernel memory leak tmpfs cache slabtop shows low slab; df -h /run /tmp /dev/shm shows culprit
Disk full RAM full via tmpfs df -h shows free disk but free -h shows low available RAM

The Fix (Generic)

  1. Immediate: Identify which files are in /dev/shm or /tmp and delete unnecessary ones. ls -laSh /dev/shm and du -sh /tmp/*.
  2. Short-term: Set tmpfs size limits in /etc/fstab: tmpfs /dev/shm tmpfs defaults,size=2G 0 0. Restart processes that use /dev/shm.
  3. Long-term: Monitor the shared column from free or the node_memory_Shmem_bytes Prometheus metric; alert at 50% of RAM used by tmpfs.

Real-World Examples

  • Example 1: PostgreSQL using shared_buffers allocated via /dev/shm. After a misconfiguration set shared_buffers=12GB on a 16GB server, /dev/shm consumed 12GB; other processes OOM killed.
  • Example 2: A machine learning pipeline wrote intermediate tensors to /tmp (a 16GB tmpfs). The pipeline processed large batches overnight; host OOMed at 3am killing the training job.

War Story

The host had 32GB RAM and nothing in top showed more than 2GB RSS. Total process RSS added up to about 8GB. But the system was OOMing. Someone finally ran vmstat -s | grep shm and found 22GB in shared memory — a legacy real-time data feed was writing its ring buffer to /dev/shm and had grown to an unexpected size after a config change. We capped /dev/shm to 8GB in /etc/fstab and the OOMs stopped.

Cross-References