Swap, memory pressure and the Go GC

AI Signal Decode

The Go process's memory spikes, leading to OOM kills, are caused by reading and unmarshalling large data blobs. During these operations, both the raw downloaded data (allocated in 'noscan' spans by Go's allocator) and the resulting unmarshalled structure (allocated in 'scan' spans) occupy memory simultaneously. This dual-allocation exacerbates memory pressure. The kernel's response to this pressure is to reclaim memory by moving anonymous pages to swap. This eviction process is governed by the kernel's LRU lists, which prioritize older, less-accessed pages. However, the author suspects that the Go process's recent memory allocations are quickly marked for potential eviction, as nothing immediately re-accesses them until the GC runs.

The market implications are significant for Go applications deployed in resource-limited environments like Kubernetes pods. When Go processes swap their anonymous memory and the GC subsequently needs to process these pages, it triggers a cascade of page faults. Each page fault requires the kernel to fetch data from swap, incurring substantial I/O latency (for file-backed swap) or CPU overhead (for compressed swap like zram). This drastically increases GC pause times, potentially leading to application unresponsiveness or further OOM situations if memory pressure intensifies. Developers must now consider the interaction between their Go application's memory usage patterns and the underlying operating system's memory management and swapping behavior.

The technical significance lies in the fundamental mismatch between the Go runtime's garbage collection strategy and the Linux kernel's memory reclamation mechanisms. Go's GC, like many tracing collectors, traverses the heap, touching potentially swapped-out pages. The kernel, unaware of the GC's intent, may evict these pages based on its own LRU logic. This leads to the 'Garbage Collection Without Paging' problem described in academic literature, where GC operations cause excessive paging, dramatically increasing pause times. The author's experiments quantify this, demonstrating a tenfold increase in GC latency when pages are swapped. Future solutions likely require tighter integration, potentially through kernel notifications to the GC about impending page evictions or vice-versa.

Moving forward, developers need to explore strategies to mitigate this issue. While GOMEMLIMIT can increase GC frequency, it doesn't solve the underlying problem of swapped pages. Potential solutions could involve optimizing data processing to reduce peak memory usage, explicitly managing memory allocation to avoid concurrent large allocations, or investigating infrastructure-level solutions at the Kubernetes level. Understanding how Go's memory allocator categorizes spans (noscan vs. scan) is crucial. The author is seeking input on potential Kubernetes-level fixes, indicating that a multi-layered approach might be necessary to address this complex interaction between application-level memory management and kernel-level resource control.