40ms Go GC stop-the-world pauses caused by swap

Using swap in production can introduce severe, unpredictable latency spikes to your Go applications.

Running swap in production caused a significant 40ms stop-the-world pause in Go's garbage collector. This occurred because the Go GC reads its metadata, which can be swapped out by the kernel under memory pressure, pointer by pointer. When the GC needs this swapped-out metadata, it triggers major page faults. A BPF script revealed that 39 of the 40ms pause were spent handling 228 page faults related to GC bookkeeping. These pauses happen during both sweep and mark terminations. Additionally, the process of building Go data structures from messages saw a dramatic increase in latency, from milliseconds to hundreds of milliseconds on swapped-out systems, though this cost is localized to the goroutine performing the allocation.

The discovery that Go's garbage collector metadata can reside in swapped-out memory, leading to extreme stop-the-world pauses, highlights a critical interaction between operating system memory management and runtime internals. This failure mode directly impacts applications relying on predictable GC behavior, as even seemingly small metadata operations can balloon into significant delays when page faults occur. The severity of these pauses suggests that environments where memory pressure is common and swap is enabled present a substantial risk for Go services.

This issue means that while swap might be intended as a buffer for memory spikes, its interaction with Go's GC can negate the benefits of memory availability by introducing global application stalls. Developers must now consider the potential for swap-induced GC pauses and evaluate whether the risk outweighs the benefit of memory expansion. Future Go versions or runtime configurations may need to address how GC metadata is handled to prevent this interaction, or users will need to implement strict memory management strategies to avoid swap entirely.