Can gzip be a language model?

Compression tools you already use can now generate text that mimics existing data.

Nathan Barry has demonstrated a method to use the gzip compression algorithm as a language model, termed "GziPT." Instead of neural networks and training, GziPT leverages the compression-prediction equivalence described in information theory. The process involves priming gzip with a corpus of text and then providing it with a prompt. Gzipt then generates continuations by searching for byte sequences that compress most effectively within gzip's DEFLATE algorithm, which uses a 32 KiB sliding window. A candidate continuation is scored by measuring the compressed length of the context plus the candidate; a smaller compressed length indicates higher predictability. To improve generation quality, Gzipt employs a beam search over byte sequences, evaluating potential continuations by compressing context and candidate, and selecting the most compressible spans over a defined horizon. The implementation uses Python's zlib library, which employs the DEFLATE algorithm, to avoid spawning external gzip processes.

This work suggests that even traditional, non-neural compression algorithms possess inherent predictive capabilities applicable to language generation. By framing compression as prediction, Gzipt bypasses the need for large datasets and complex training architectures, demonstrating that statistical regularities in data can be exploited directly by compressors like gzip. This approach may inspire alternative methods for content generation, particularly in scenarios where computational resources or data availability are limited.

The success of Gzipt, even with its current limitations in coherence, indicates a potential shift in how we consider generative models. Future research could explore optimizing the beam search parameters, larger context windows, or combinations with other simple statistical models to enhance the quality and controllability of text generated by compressors. This could lead to novel applications in areas like data augmentation, content summarization, or even specialized creative writing tools that leverage existing system utilities.