Container Memory Limits and OOM Behavior

Container memory limits constrain how much memory a container can use, and exceeding an enforced limit can cause the kernel to terminate processes, commonly surfaced as `OOMKilled`. For a Java backend engineer, diagnosing OOM events requires separating container limits from JVM heap sizing and other native memory consumers.

A container memory limit is an operating-system-enforced ceiling on memory charged to that container. Depending on the container runtime and Linux configuration, memory usage can include the JVM heap plus native allocations, thread stacks, direct buffers, class metadata, garbage-collector structures, and other process memory.

For Java, `-Xmx` limits the Java heap, not total process memory. A container can therefore be killed even when heap usage is below `Xmx`, because non-heap memory and other charged allocations can push total container usage beyond its limit.

When the kernel's memory control mechanism cannot reclaim enough memory to stay within the applicable limit, it can invoke the OOM killer. In Kubernetes, a container terminated because it exceeded its memory limit is commonly reported with reason `OOMKilled` and exit code `137` is commonly observed because the process receives `SIGKILL`.

The practical fix is not always to increase `Xmx`. First compare the container memory limit with JVM heap sizing and inspect native-memory consumers, garbage-collection behavior, allocation patterns, thread counts, and direct-buffer usage. A reasonable memory budget leaves headroom outside the heap for the JVM and application runtime.

Kubernetes also uses memory requests and limits for scheduling and enforcement. A memory request primarily influences placement and resource accounting, while a memory limit establishes the container's permitted memory ceiling when enforced by the node's resource controls.

Key points

• A container memory limit applies to total memory charged to the container, not merely the JVM heap.
• `-Xmx` constrains heap capacity but does not cap total JVM process memory.
• Exceeding an enforced memory limit can result in an OOM kill; Kubernetes commonly reports `OOMKilled`.
• Exit code `137` commonly indicates termination by `SIGKILL`, but it is not by itself proof of a container memory-limit OOM.
• Leave memory headroom for metaspace, thread stacks, direct buffers, GC structures, native libraries, and other non-heap usage.

Interview questions

What is the relationship between Kubernetes memory requests and limits?

A memory request represents the amount of memory Kubernetes uses for scheduling and resource accounting. A memory limit establishes the upper memory boundary for the container when the node's resource controls enforce it. A pod can therefore have a request lower than its limit and receive more memory when available, up to the limit. Setting these values appropriately affects both scheduling behavior and what happens under memory pressure.

What happens when a Java container exceeds its memory limit?

The operating system's memory-control mechanisms can invoke the OOM killer when the container cannot remain within its memory budget. The affected process may receive `SIGKILL` and cannot clean up normally. Kubernetes commonly reports the terminated container with reason `OOMKilled`. An observed exit code of `137` is also common because it corresponds to termination by signal 9.

Why can a JVM be OOM-killed when heap usage is below `-Xmx`?

`-Xmx` controls the maximum Java heap, not the entire memory footprint of the JVM process. Memory can also be consumed by metaspace, thread stacks, direct byte buffers, garbage-collector structures, JIT/compiler memory, native libraries, and other allocations. If the combined usage exceeds the container's memory limit, the container can still be terminated. Therefore JVM heap sizing must be considered together with the container's total memory budget.

How would you diagnose repeated `OOMKilled` Java pods in Kubernetes?

I would first confirm that the termination reason is actually `OOMKilled` and compare the pod's memory limit with its observed usage. Next I would inspect heap configuration, GC logs, heap occupancy, thread counts, direct-buffer usage, metaspace, and other native-memory consumers. I would also check whether the workload has a memory leak or simply has a limit that is too small for its normal workload. Finally, I would adjust the memory budget or application behavior based on measured peak usage rather than arbitrarily increasing `Xmx`.

Is exit code 137 proof that a Java application was OOM-killed?

No. Exit code 137 commonly results from a process receiving `SIGKILL`, because 128 plus signal 9 equals 137. An OOM kill is one possible reason for `SIGKILL`, but another component could also have sent the signal. I would confirm the container termination reason and inspect Kubernetes and node-level events before concluding that memory exhaustion caused the termination.