# LLMZip: Lossless Text Compression using Large Language Models

arXiv: https://arxiv.org/abs/2306.04050 · HTML: https://mlsanity.com/paper/2306.04050/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 11 references that are in the map.
- First posted on arXiv 6 Jun 2023.

## Builds on (most cited first, 4 of 4)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [DeepZip: Lossless Data Compression using Recurrent Neural Networks](https://mlsanity.com/paper/1811.08162/) (2018)
- [Focus Your Attention (with Adaptive IIR Filters)](https://mlsanity.com/paper/2305.14952/) (2023)

## Built on by (most cited first, 30 of 44)

- [Language Modeling Is Compression](https://mlsanity.com/paper/2309.10668/) (2023)
- [A Survey of Resource-efficient LLM and Multimodal Foundation Models](https://mlsanity.com/paper/2401.08092/) (2024)
- [EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices](https://mlsanity.com/paper/2308.14352/) (2023)
- [SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models](https://mlsanity.com/paper/2310.18859/) (2023)
- [Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models](https://mlsanity.com/paper/2401.17139/) (2024)
- [Training LLMs over Neurally Compressed Text](https://mlsanity.com/paper/2404.03626/) (2024)
- [Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data](https://mlsanity.com/paper/2410.05078/) (2024)
- [SWIFT: Semantic Watermarking for Image Forgery Thwarting](https://mlsanity.com/paper/2407.18995/) (2024)
- [In-context learning and Occam's razor](https://mlsanity.com/paper/2410.14086/) (2024)
- [Towards Optimal Learning of Language Models](https://mlsanity.com/paper/2402.17759/) (2024)
- [AlphaZip: Neural Network-Enhanced Lossless Text Compression](https://mlsanity.com/paper/2409.15046/) (2024)
- [Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training](https://mlsanity.com/paper/2508.15390/) (2025)
- [Joint Lossless Compression and Steganography for Medical Images via Large Language Models](https://mlsanity.com/paper/2508.01782/) (2025)
- [Evaluating Large Language Models for Generalization and Robustness via Data Compression](https://mlsanity.com/paper/2402.00861/) (2024)
- [The Information of Large Language Model Geometry](https://mlsanity.com/paper/2402.03471/) (2024)
- [The KoLMogorov Test: Compression by Code Generation](https://mlsanity.com/paper/2503.13992/) (2025)
- [LMM-driven Semantic Image-Text Coding for Ultra Low-bitrate Learned Image Compression](https://mlsanity.com/paper/2411.13033/) (2024)
- [Nacrith: Neural Lossless Compression via Ensemble Context Modeling and High-Precision CDF Coding](https://mlsanity.com/paper/2602.19626/) (2026)
- [Prediction-Powered Communication with Distortion Guarantees](https://mlsanity.com/paper/2509.24373/) (2025)
- [Fast, Secure, and High-Capacity Image Watermarking with Autoencoded Text Vectors](https://mlsanity.com/paper/2510.00799/) (2025)
- [Large Language Model Evaluation via Matrix Nuclear-Norm](https://mlsanity.com/paper/2410.10672/) (2024)
- [An Enhanced Text Compression Approach Using Transformer-based Language Models](https://mlsanity.com/paper/2412.15250/) (2024)
- [DualComp: End-to-End Learning of a Unified Dual-Modality Lossless Compressor](https://mlsanity.com/paper/2505.16256/) (2025)
- [Revisiting Data Compression with Language Modeling](https://mlsanity.com/paper/2601.02875/) (2026)
- [OmniZip: Learning a Unified and Lightweight Lossless Compressor for Multi-Modal Data](https://mlsanity.com/paper/2602.22286/) (2026)
- [Coding for Intelligence from the Perspective of Category](https://mlsanity.com/paper/2407.01017/) (2024)
- [The Lossy Horizon: Error-Bounded Predictive Coding for Lossy Text Compression (Episode I)](https://mlsanity.com/paper/2510.22207/) (2025)
- [A high-capacity linguistic steganography based on entropy-driven rank-token mapping](https://mlsanity.com/paper/2510.23035/) (2025)
- [In-Context Source and Channel Coding](https://mlsanity.com/paper/2601.10267/) (2026)
- [Semantic Chunking and the Entropy of Natural Language](https://mlsanity.com/paper/2602.13194/) (2026)

## Abstract (arXiv)

We provide new estimates of an asymptotic upper bound on the entropy of English using the large language model LLaMA-7B as a predictor for the next token given a window of past tokens. This estimate is significantly smaller than currently available estimates in citecover1978convergent, citelutati2023focus. A natural byproduct is an algorithm for lossless compression of English text which combines the prediction from the large language model with a lossless compression scheme. Preliminary results from limited experiments suggest that our scheme outperforms state-of-the-art text compression schemes such as BSC, ZPAQ, and paq8h.