Skip to content

Decision Tree: Memory Usage Is High

Category: Incident Triage Starting Question: "Memory usage is high on a host or in a pod — why?" Estimated traversal: 2-4 minutes Domains: linux-performance, kubernetes, postgresql, observability

The Tree

Memory usage is high on a host or in a pod — why?
│
├── Where are you investigating? (Set your context first)
│   │
│   ├── Kubernetes pod → `kubectl top pod <name> -n <namespace>`
│   │   and `kubectl describe pod <name> | grep -A6 "Limits\|Requests"`
│   │
│   └── Host / node → SSH to node
│       `ssh ubuntu@<node-ip>`
│
├── On a host: `free -h`
│   │
│   ├── Parse the output carefully:
│   │   total = total RAM
│   │   used  = actually used by processes (total - free - buffers/cache)
│   │   buff/cache = OS using it as disk cache (reclaimed instantly if needed)
│   │   available = what processes can actually get
│   │   │
│   │   ├── available > 20% of total → memory is actually fine
│   │   │   (OS is using RAM as cache, which is normal and healthy)
│   │   │   └── ✅ No action needed — alert threshold may need tuning
│   │   │
│   │   └── available < 10% of total → investigate further below
│   │
├── Is a single process consuming most of the memory?
│   `ps aux --sort=-%mem | head -10`
│   `smem -s rss -r | head -10`  (if available)
│   │
│   ├── Yes — one process is dominant
│   │   │
│   │   ├── What type of process?
│   │   │   │
│   │   │   ├── Java / JVM process
│   │   │   │   `jcmd <pid> VM.native_memory` (if JDK tools available in pod)
│   │   │   │   `kubectl exec -it <pod> -- curl localhost:8080/actuator/metrics/jvm.memory.used`
│   │   │   │   │
│   │   │   │   ├── Heap growing over time → memory leak
│   │   │   │   │   └── ✅ ACTION: Take Heap Dump / Investigate Java Memory Leak
│   │   │   │   │
│   │   │   │   ├── Heap at limit, GC running constantly (GC overhead)
│   │   │   │   │   └── ✅ ACTION: Increase JVM Heap / -Xmx
│   │   │   │   │
│   │   │   │   └── Off-heap (Metaspace / Direct buffers) growing
│   │   │   │       └── ✅ ACTION: Tune JVM Non-Heap Limits
│   │   │   │
│   │   │   ├── Go process
│   │   │   │   `kubectl exec -it <pod> -- wget -qO- localhost:6060/debug/pprof/heap > /tmp/heap.out`
│   │   │   │   `go tool pprof /tmp/heap.out`
│   │   │   │   │
│   │   │   │   ├── Goroutine leak → spawning goroutines that never exit
│   │   │   │   │   `GET /debug/pprof/goroutine?debug=2`
│   │   │   │   │   └── ✅ ACTION: Fix Goroutine Leak / Deploy Fix
│   │   │   │   │
│   │   │   │   └── Large in-memory cache growing unbounded
│   │   │   │       └── ✅ ACTION: Add Cache Eviction / Bound Cache Size
│   │   │   │
│   │   │   ├── Node.js process
│   │   │   │   Check V8 heap: `process.memoryUsage()` via `kubectl exec`
│   │   │   │   │
│   │   │   │   └── Event listeners or closures not being freed
│   │   │   │       └── ✅ ACTION: Heap Snapshot / Fix EventEmitter Leak
│   │   │   │
│   │   │   └── PostgreSQL / database process
│   │   │       `SELECT pg_size_pretty(sum(size)) FROM pg_shmem_allocations;`
│   │   │       │
│   │   │       ├── shared_buffers too large for host → reduce it
│   │   │       ├── work_mem * many connections = OOM risk
│   │   │       └── ✅ ACTION: Tune PostgreSQL Memory Parameters
│   │   │
│   │   └── Is the process memory growing over time or stable at a high level?
│   │       `kubectl top pod <name> -w` (watch for 5 min)
│   │       │
│   │       ├── Growing steadily → memory leak confirmed
│   │       │   └── ✅ ACTION: Restart Process (emergency) + Fix Leak (permanent)
│   │       │
│   │       └── Stable but high → high legitimate load
│   │           → is the load expected? If yes, scale out
│   │           → if not expected, check for abuse / unexpected request volume
│   │
│   ├── No — memory is distributed across many processes → pod / node is overloaded
│   │   │
│   │   ├── Is this a Kubernetes node?
│   │   │   `kubectl top pods -n <namespace> --sort-by=memory | head -10`
│   │   │   │
│   │   │   ├── Pod memory limits are not set → pods can consume all node memory
│   │   │   │   └── ✅ ACTION: Set Memory Limits on Pods / LimitRange
│   │   │   │
│   │   │   └── Pods near their limits → need more capacity
│   │   │       └── ✅ ACTION: Scale Node Group / Reschedule Pods
│   │   │
│   │   └── Is swap in use?
│   │       `free -h` — check Swap row
│   │       `vmstat 1 5` — look for si/so (swap in/swap out)
│   │       │
│   │       ├── Swap active (si/so > 0) → severe memory pressure, process paging
│   │       │   └── ✅ ACTION: Add Memory / Reduce Memory Consumers (URGENT)
│   │       │
│   │       └── No swap → either OS killed processes already or about to
│   │           `dmesg | grep -i "oom\|killed"`
│   │           └── OOM kills in dmesg → ✅ ACTION: Add Memory / Adjust Limits
│   │
├── Is the pod approaching its Kubernetes memory limit?
│   `kubectl top pod <pod>` vs `kubectl get pod <pod> -o jsonpath='{.spec.containers[0].resources.limits.memory}'`
│   │
│   ├── Usage > 90% of limit
│   │   `kubectl get pod <pod> -w` — watch for OOMKilled restarts
│   │   │
│   │   ├── Pod restarts with OOMKilled → ✅ ACTION: Increase Memory Limit
│   │   │   Runbook: [oomkilled.md](../../runbooks/kubernetes/oom-kill.md)
│   │   │
│   │   └── Pod not restarting yet → preemptively increase limit
│   │       → ✅ ACTION: Increase Memory Limit (proactive)
│   │
│   └── Usage well below limit but host is out of memory?
│       → Pods' *requests* are too low vs actual usage
│       → K8s scheduled too many pods on this node
│       → ✅ ACTION: Align Memory Requests with Actual Usage
│
└── Check for memory pressure indicators in kernel
    `dmesg -T | grep -iE "oom|killed|low memory|swap"`
    │
    ├── OOM killer fired → process was killed
    │   → identify victim process and adjust limits
    │   └── ✅ ACTION: Adjust Memory Limits / Fix Leak
    │
    └── No kernel events → memory high but stable, no crisis yet
        → schedule investigation during business hours

Node Details

Check 1: Interpreting free -h correctly

Command: free -h and then cat /proc/meminfo | grep -E "MemAvailable|MemFree|SwapTotal|SwapFree|Cached|Buffers" What you're looking for: The available column in free -h is the number that matters — it includes pages that the OS will reclaim from its buffer cache to give to processes. used alone is misleading. Common pitfall: A used value of 14GB on a 16GB host looks alarming. But if available shows 10GB, the OS is simply using 10GB as disk cache, which is normal and healthy behavior. Only alert when available drops below 15-20% of total RAM.

Check 2: Process-level memory consumption

Command: ps aux --sort=-%mem | head -15 and top -o %MEM -b -n1 | head -20. For inside a pod: kubectl exec -it <pod> -- ps aux --sort=-%mem. What you're looking for: The %MEM and RSS (resident set size) columns. A process with high %MEM growing over time is a leak; stable-high is legitimate load. Common pitfall: VSZ (virtual size) includes memory-mapped files and unused heap space — it's always much larger than RSS and is not a meaningful indicator of actual memory consumption. Use RSS.

Check 3: Swap activity

Command: vmstat 1 10 — watch the si (swap in) and so (swap out) columns over 10 seconds. Any non-zero value means the system is paging. What you're looking for: so > 0 means the OS is writing memory pages to disk — the system is memory-starved. Even a small so value (1-5 pages/sec) causes significant application latency. Common pitfall: Kubernetes nodes should have swap disabled (Kubernetes requires it). If swap is active on a k8s node, kubelet will refuse to start unless failSwapOn: false is set. Check: swapon --show.

Check 4: JVM heap vs non-heap

Command: If running Spring Boot with actuator: curl localhost:8080/actuator/metrics/jvm.memory.used?tag=area:heap and ?tag=area:nonheap. Or: jstat -gc <pid> 5s 5 for GC statistics. What you're looking for: Heap growing toward -Xmx limit triggers aggressive GC, then OOM. Non-heap (Metaspace) growing indicates class loading issues (often in frameworks that generate code at runtime). Common pitfall: JVM reports its heap to the OS as COMMITTED (allocated from OS) even if only a fraction is USED (actually holding live objects). kubectl top pod shows committed memory, which includes Java's idle heap reservation. Set -XX:MinRAMPercentage and -XX:MaxRAMPercentage for container-aware JVM tuning.

Check 5: OOM kill events

Command: dmesg -T | grep -i "out of memory\|killed process\|oom_kill" and journalctl -k --since "1 hour ago" | grep -i oom. What you're looking for: Lines like "Out of memory: Kill process 12345 (java) score 800 or sacrifice child". The score tells you what the OOM killer targeted. Common pitfall: The OOM killer may kill a different process than the one causing the problem (e.g., it kills a small service to free memory for a runaway process). The killed process is a victim, not the root cause.

Check 6: Memory requests vs limits alignment

Command: kubectl get pods -n <namespace> -o json | jq '.items[] | {name: .metadata.name, req: .spec.containers[0].resources.requests.memory, lim: .spec.containers[0].resources.limits.memory}' What you're looking for: Pods where requests is set much lower than actual usage (from kubectl top pods). The scheduler places pods based on requests — if requests are 100Mi but actual usage is 1Gi, the node becomes oversubscribed. Common pitfall: Setting requests = 0 (no request) makes pods Burstable class and allows unlimited memory usage, causing unexpected node memory pressure.

Terminal Actions

Action: Restart Process (emergency) + Fix Leak (permanent)

Do: 1. Emergency: kubectl rollout restart deployment/<name> — this restores service but doesn't fix the leak 2. Permanent: capture a heap dump before restart for analysis: kubectl exec -it <pod> -- jmap -dump:format=b,file=/tmp/heap.hprof <pid> (Java) 3. Copy out: kubectl cp <pod>:/tmp/heap.hprof ./heap.hprof 4. Analyze with Eclipse MAT or VisualVM 5. File a bug with the heap dump attached; set maxUnavailable=1 on HPA to buy time Verify: Pod memory stable after restart. Leak investigation underway. Runbook: oomkilled.md

Action: Increase Memory Limit

Do: 1. Check current limit: kubectl get pod <pod> -o jsonpath='{.spec.containers[0].resources.limits.memory}' 2. Increase by 50%: kubectl set resources deployment <name> --limits=memory=2Gi --requests=memory=1Gi 3. Watch for restart: kubectl get pods -l app=<name> -w Verify: New pod runs without OOMKilled. kubectl top pod shows usage below 80% of new limit. Runbook: oomkilled.md

Action: Take Heap Dump / Investigate Java Memory Leak

Do: 1. Capture heap (non-destructive): kubectl exec -it <pod> -- jcmd <pid> GC.heap_dump /tmp/heap.hprof 2. Or via kill (only if jcmd unavailable): kubectl exec -it <pod> -- kill -3 <pid> (thread dump to stdout) 3. Extract: kubectl cp <pod>:/tmp/heap.hprof ./heap.hprof 4. Open with Eclipse MAT — look for "Problem Suspect" report Verify: Leak identified. Code fix deployed. Memory stabilizes after the fix.

Action: Tune JVM Non-Heap Limits

Do: 1. Add to JAVA_OPTS: -XX:MaxMetaspaceSize=256m -XX:CompressedClassSpaceSize=64m 2. For direct buffer leaks: -XX:MaxDirectMemorySize=256m 3. Redeploy: kubectl rollout restart deployment/<name> Verify: Non-heap metrics stabilize. No more OOM on Metaspace.

Action: Fix Goroutine Leak / Deploy Fix

Do: 1. Capture goroutine profile: kubectl exec -it <pod> -- curl -s "localhost:6060/debug/pprof/goroutine?debug=1" > goroutines.txt 2. Identify goroutines stuck in the same state for a long time 3. Fix the code path that spawns goroutines without guaranteed cleanup 4. Add context cancellation to all goroutines that call external services Verify: GET /debug/pprof/goroutine shows stable, low goroutine count over time.

Action: Tune PostgreSQL Memory Parameters

Do: 1. Connect: kubectl exec -it <pg-pod> -- psql -U postgres 2. Check current: SHOW shared_buffers; SHOW work_mem; SHOW max_connections; 3. Calculate risk: work_mem * max_connections * 2 should be << total RAM 4. Reduce: ALTER SYSTEM SET work_mem = '8MB'; SELECT pg_reload_conf(); 5. For shared_buffers > 25% of RAM: ALTER SYSTEM SET shared_buffers = '2GB'; SELECT pg_reload_conf(); Verify: free -h on the DB host shows memory recovering. No OOM kills.

Action: Set Memory Limits on Pods / LimitRange

Do: 1. Add a namespace-level LimitRange as a safety net: 

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
spec:
  limits:
  - default: {memory: 512Mi, cpu: 500m}
    defaultRequest: {memory: 128Mi, cpu: 100m}
    type: Container
2. kubectl apply -f limitrange.yaml -n <namespace> 3. Verify new pods get default limits: kubectl describe pod <new-pod> | grep -A4 "Limits" Verify: kubectl get limitrange -n <namespace> shows the new LimitRange.

Action: Add Memory / Reduce Memory Consumers (URGENT)

Do: 1. If in cloud: scale node group up immediately (add node capacity) 2. Identify and kill low-priority pods to free memory now: kubectl delete pod <low-priority-pod> -n <namespace> 3. Identify what changed recently (new deployment, traffic spike) 4. Roll back if memory spike caused by new deployment Verify: free -h shows available increasing. vmstat 1 5 shows so = 0.

Escalation: Infrastructure Team for Memory Analysis

When: Memory leak is confirmed but cannot be fixed quickly, and the service is critical. Who: Application team lead + SRE on-call Include in page: Pod name, memory growth rate (MB/hour), heap dump location, time of onset, last deployment details

Edge Cases

  • Memory high but no single process to blame: Many small containers on a single node may collectively exceed capacity. Check node requests vs actual usage: kubectl describe node | grep -A10 "Allocated resources".
  • Memory drops after pod restart but grows back within hours: Classic memory leak. Implement an automated restart schedule (Kubernetes CronJob to rolling-restart) as a stopgap while the real fix ships.
  • JVM reports low heap usage, but pod memory is high: JVM's off-heap memory (thread stacks, JNI, direct buffers, Metaspace) can consume significant memory outside the heap. A Java process with -Xmx1g can easily use 2GB total.
  • Memory high only at specific times (batch jobs): Not a leak — it's expected high-water-mark usage. Set limits to accommodate the peak or scale up during batch windows.
  • Redis memory growing without bound: Check maxmemory and maxmemory-policy config. Without maxmemory set, Redis will consume all available memory. Set it and choose an appropriate eviction policy.

Cross-References