--- tags: - k8s - l1 - flashcard-deck - oomkilled --- [Portal](../../../../library/portal/index.md) | **Level:** [L1: Foundations](../../../../library/portal/levels.md) | **Topics:** [OOMKilled (alias)](../../../../library/portal/topics.md) | **Domain:** Kubernetes id category difficulty tags question answer source_path oomkilled/a1b2c3d4e5f6 oomkilled easy oomkilled,kubernetes,memory What does OOMKilled mean in Kubernetes? OOMKilled means a container exceeded its memory limit and the kernel's Out-Of-Memory killer terminated it. The container's exit code is 137 (128 + SIGKILL signal 9). The pod restarts per its restartPolicy, and the restart count increments.\n\nRemember: OOMKilled = exceeded memory limit. Exit 137 = 128+SIGKILL(9).\n\nGotcha: Can be container limit OR node OOM. Check both.\n\nName origin: OOM = Out Of Memory. The Linux OOM killer was introduced to prevent the entire system from hanging when memory is exhausted.\n\nNumber anchor: Exit code 137 = 128 + 9 (SIGKILL). This math is universal in Unix: exit code = 128 + signal number. training/library/topics/oomkilled/primer.md oomkilled/b2c3d4e5f6a7 oomkilled easy oomkilled,diagnosis How do you confirm a pod was OOMKilled? kubectl describe pod — look for Last State: Terminated, Reason: OOMKilled, Exit Code: 137. Also check kubectl get pod -o jsonpath='{.status.containerStatuses[0].lastState}'. Node-level: dmesg | grep -i oom on the node shows the kernel OOM killer invocation.\n\nRemember: OOMKilled = exceeded memory limit. Exit 137 = 128+SIGKILL(9).\n\nGotcha: Can be container limit OR node OOM. Check both. training/library/topics/oomkilled/primer.md oomkilled/c3d4e5f6a7b8 oomkilled medium oomkilled,limits What is the difference between memory requests and limits in the OOM context? Requests determine scheduling (which node has enough). Limits set the hard ceiling — exceeding the limit triggers OOMKill. A pod can use more than its request (if the node has slack) but never more than its limit. If the node is under memory pressure, pods using more than their request are evicted first.\n\nRemember: OOMKilled = exceeded memory limit. Exit 137 = 128+SIGKILL(9).\n\nGotcha: Can be container limit OR node OOM. Check both. training/library/topics/oomkilled/primer.md oomkilled/d4e5f6a7b8c9 oomkilled medium oomkilled,java Why are Java applications especially prone to OOMKilled in containers? The JVM allocates a heap plus off-heap memory (metaspace, thread stacks, JIT code cache, direct buffers). If the container limit only accounts for heap (-Xmx), off-heap can push total RSS over the limit. Fix: set -Xmx to ~75% of container limit, or use -XX:MaxRAMPercentage=75.0 which respects cgroup limits.\n\nRemember: OOMKilled = exceeded memory limit. Exit 137 = 128+SIGKILL(9).\n\nGotcha: Can be container limit OR node OOM. Check both. training/library/topics/oomkilled/primer.md oomkilled/e5f6a7b8c9d0 oomkilled hard oomkilled,kernel How does the Linux OOM killer choose which process to kill? The kernel calculates an oom_score for each process based on memory usage, process age, and oom_score_adj. Higher score = more likely to be killed. In containers, the kernel kills processes within the cgroup that exceeded its limit. oom_score_adj can protect critical processes (-1000 = never kill) or prioritize killing (1000 = kill first).\n\nRemember: OOMKilled = out of memory. Exit 137. Check pod limits AND node memory.\n\nGotcha: Prevent: set proper limits, monitor trends, fix leaks early. training/library/topics/oomkilled/primer.md oomkilled/f6a7b8c9d0e1 oomkilled medium oomkilled,debugging How do you determine the right memory limit for a container? 1) Run the application under realistic load. 2) Monitor container_memory_working_set_bytes in Prometheus (not RSS — working set excludes cache). 3) Set limit to 1.5-2x the observed peak. 4) Set request to the average steady-state usage. 5) Use VPA (Vertical Pod Autoscaler) recommendations as a starting point.\n\nRemember: OOMKilled = out of memory. Exit 137. Check pod limits AND node memory.\n\nGotcha: Prevent: set proper limits, monitor trends, fix leaks early. training/library/topics/oomkilled/primer.md oomkilled/a7b8c9d0e1f2 oomkilled easy oomkilled,troubleshooting What is the difference between OOMKilled and eviction? OOMKilled: container exceeded its cgroup memory limit — killed by kernel immediately. Eviction: kubelet proactively removes pods when node memory pressure is high (memory.available < threshold). Evicted pods are rescheduled; OOMKilled containers restart in place. Eviction is gentler (SIGTERM first).\n\nRemember: OOMKilled = exceeded memory limit. Exit 137 = 128+SIGKILL(9).\n\nGotcha: Can be container limit OR node OOM. Check both. training/library/topics/oomkilled/primer.md oomkilled/b8c9d0e1f2a3 oomkilled hard oomkilled,cgroup How do cgroup memory limits relate to OOMKilled? Kubernetes sets cgroup memory.limit_in_bytes (cgroup v1) or memory.max (cgroup v2) to the container's memory limit. When the cgroup's memory usage hits this limit and the kernel cannot reclaim pages, it invokes the OOM killer within that cgroup. The container sees SIGKILL — no graceful shutdown, no chance to catch the signal.\n\nRemember: OOMKilled = exceeded memory limit. Exit 137 = 128+SIGKILL(9).\n\nGotcha: Can be container limit OR node OOM. Check both. training/library/topics/oomkilled/primer.md oomkilled/c9d0e1f2a3b4 oomkilled medium oomkilled,prevention What are strategies to prevent OOMKilled in production? 1) Right-size limits based on profiling, not guessing. 2) For JVM apps, align -Xmx with container limit. 3) Enable swap (cgroup v2 memory.swap.max) for burst tolerance. 4) Use VPA for automatic recommendations. 5) Monitor memory trends and alert before hitting limits. 6) Fix memory leaks — growing RSS over time indicates a leak.\n\nRemember: OOMKilled = exceeded memory limit. Exit 137 = 128+SIGKILL(9).\n\nGotcha: Can be container limit OR node OOM. Check both. training/library/topics/oomkilled/primer.md oomkilled/d0e1f2a3b4c5 oomkilled medium oomkilled,metrics What Prometheus metrics help diagnose OOM risk? "container_memory_working_set_bytes / container_spec_memory_limit_bytes shows utilization ratio — alert at 80-90%. kube_pod_container_status_last_terminated_reason{reason=""OOMKilled""} counts OOM events. container_memory_rss tracks resident set size. node_memory_MemAvailable_bytes tracks node-level pressure.\n\nRemember: OOMKilled = exceeded memory limit. Exit 137 = 128+SIGKILL(9).\n\nGotcha: Can be container limit OR node OOM. Check both." training/library/topics/oomkilled/primer.md --- ## Wiki Navigation ### Related Content - [OOMKilled](../../../../library/topics/oomkilled/index.md) (Topic Pack, L1) — OOMKilled (alias)