Static 1 source · 39m ago

Swap and the Go garbage collector

A Go application running in a container experiences frequent Out-of-Memory (OOM) kills despite appearing to have stable memory usage. The issue arises during peak load when the application downloads, decompresses, and unmarshals large data payloads, simultaneously holding both the raw byte slice and the unmarshalled object graph in memory. This triggers Go's garbage collector (GC) to scan memory pages containing pointers. Concurrently, the Linux kernel, under memory pressure, attempts to reclaim memory by swapping out anonymous pages. However, Go's GC interacts poorly with this process. When the GC needs to scan pointer-containing pages that have been swapped out, it causes major page faults, forcing the kernel to bring those pages back into RAM. These pages are actively being used by the GC but contain live data, meaning they are not reclaimed. This leads to a feedback loop where the GC's activity exacerbates memory pressure, causing more swapping and more page faults, ultimately failing to free memory and leading to OOM kills. The `GOMEMLIMIT` setting can worsen this behavior by increasing GC frequency.

AI Signal Decode

The core problem lies in the interaction between Go's tracing garbage collector and the Linux kernel's memory management, specifically swap. When the Go process experiences a memory spike due to data decompression and unmarshalling, it holds both the large byte slice and the resulting object graph. Go's allocator separates these into `noscan` spans (for the byte slice, ignored by GC) and `scan` spans (for the object graph, scanned by GC). Under memory pressure, the kernel prioritizes swapping out "cold" anonymous memory pages. However, Go's GC necessitates scanning "hot" pages containing pointers. If these hot pages are swapped out, the GC triggers major page faults to bring them back into RAM, not to collect them, but to scan them. This leads to a cycle of continuous page faults, as GC activity pulls swapped pages back into memory, which in turn causes the kernel to swap out other pages.

The market implications are significant for cloud-native applications relying on Go. Frequent OOM kills lead to service instability and unreliability, impacting user experience and potentially incurring higher infrastructure costs due to increased resource allocation or frequent restarts. The analysis highlights a performance bottleneck where memory pressure translates directly into increased latency (via disk I/O for file-backed swap) or CPU overhead (for zram). Developers might need to rethink memory management strategies or avoid certain patterns that trigger this GC-swap interaction, especially in resource-constrained environments. The inability of `GOMEMLIMIT` to mitigate this issue further complicates optimization efforts.

Technically, the issue stems from the fundamental design of tracing GCs and the kernel's page reclamation policies. Go's GC needs to read all reachable objects, and if those objects reside on swapped-out pages, the page fault mechanism inefficiently brings them back into memory only to be scanned, not freed. This is compounded by the kernel's cold/hot page eviction strategy, which targets pages not actively accessed, while Go's GC actively accesses pages containing live objects. This creates a conflict where essential GC operations trigger performance-degrading page faults. The experiment clearly demonstrates the performance cost, with tens of thousands of faults per GC cycle at a significant per-fault cost.

Looking ahead, potential solutions lie in both application-level changes and infrastructure-level innovations. On the application side, avoiding large, transient memory allocations, such as using bounded buffers or streaming unmarshalling, could prevent the simultaneous presence of large byte slices and object graphs. However, this often requires modifying third-party libraries. At the infrastructure level, Kubernetes or container runtime features that provide more granular control over page swapping, or perhaps alternative memory management strategies within Go itself that are more swap-aware, could be explored. The author explicitly seeks input on Kubernetes-level solutions.

Swap

Covered by: reddit_programming