mlsanityData to 7 Oct 2026

Language Modeling Is Compression

arXiv 2309.10668 ↗First posted 19 Sep 2023cs.LG cs.AI cs.CLcited by 225

Not on the map. Why?

Abstract (arXiv)

It has long been established that predictive models can be transformed into lossless compressors and vice versa. Incidentally, in recent years, the machine learning community has focused on training increasingly large and powerful self-supervised (language) models. Since these large language models exhibit impressive predictive capabilities, they are well-positioned to be strong compressors. In this work, we advocate for viewing the prediction problem through the lens of compression and evaluate the compression capabilities of large (foundation) models. We show that large language models are powerful general-purpose predictors and that the compression viewpoint provides novel insights into scaling laws, tokenization, and in-context learning. For example, Chinchilla 70B, while trained primarily on text, compresses ImageNet patches to 43.4% and LibriSpeech samples to 16.4% of their raw size, beating domain-specific compressors like PNG (58.5%) or FLAC (30.3%), respectively. Finally, we show that the prediction-compression equivalence allows us to use any compressor (like gzip) to build a conditional generative model.

Key facts. Cited by 225 papers in the map (data through 7 Oct 2026). Builds on 41 of its 81 references that are in the map. First posted on arXiv 19 Sep 2023.

Builds on

41 of 81 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  12. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
29 more
  1. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  2. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  3. 2017Exploring Generalization in Deep Learningcited by 595
  4. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
  5. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  6. 2022Neural Networks and the Chomsky Hierarchycited by 214
  7. 2023Large Language Models as General Pattern Machinescited by 196
  8. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
  9. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  10. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  11. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  12. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 97
  13. 2018Practical Full Resolution Learned Lossless Image Compressioncited by 93
  14. 2019Practical Lossless Compression with Latent Variables using Bits Back Codingcited by 85
  15. 2019Integer Discrete Flows and Lossless Compressioncited by 82
  16. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  17. 2018The Description Length of Deep Learning Modelscited by 71
  18. 2021Shaking the foundations: delusions in sequence models for interaction and controlcited by 61
  19. 2019Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variablescited by 51
  20. 2023LLMZip: Lossless Text Compression using Large Language Modelscited by 44
  21. 2020Learning Better Lossless Compression Using Lossy Compressioncited by 33
  22. 2018DeepZip: Lossless Data Compression using Recurrent Neural Networkscited by 32
  23. 2022A Fast Transformer-based General-Purpose Lossless Compressorcited by 26
  24. 2023Memory-Based Meta-Learning on Non-Stationary Distributionscited by 14
  25. 2021LC-FDNet: Learned Lossless Image Compression with Frequency Decomposition Networkcited by 12
  26. 2014Compress and Controlcited by 7
  27. 2016Syntactically Informed Text Compression with Recurrent Neural Networkscited by 6
  28. 2022Few-Shot Non-Parametric Learning with Deep Latent Variable Modelcited by 5
  29. 2022Accelerated Deep Lossless Image Coding with Unified Paralleleized GPU Coding Architecturecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 70; 2025: 66; 2026: 81.

225 papers.

Sort:

By area:

  1. 2025Large Language Diffusion Modelscited by 872
  2. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 491
  3. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  4. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  5. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  6. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  7. 2024Image and Video Tokenization with Binary Spherical Quantizationcited by 110
  8. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  9. 2024Fine-Tuned Language Models Generate Stable Inorganic Materials as Textcited by 92
  10. 2024Large Language Models are Geographically Biasedcited by 91
  11. 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 91
  12. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 79
38 more
  1. 2023GeoLLM: Extracting Geospatial Knowledge from Large Language Modelscited by 62
  2. 2024Compression Represents Intelligence Linearlycited by 56
  3. 2024Emergence of a High-Dimensional Abstraction Phase in Language Transformerscited by 56
  4. 2023Non-Vacuous Generalization Bounds for Large Language Modelscited by 53
  5. 2024Language Models Resist Alignment: Evidence From Data Compressioncited by 50
  6. 2024Reverse Training to Nurse the Reversal Cursecited by 48
  7. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklistcited by 48
  8. 2023Neural Retrievers are Biased Towards LLM-Generated Contentcited by 37
  9. 2024Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Modelscited by 35
  10. 2025Token Communications: A Large Model-Driven Framework for Cross-modal Context-aware Semantic Communicationscited by 32
  11. 2024How Much Can RAG Help the Reasoning of LLM?cited by 31
  12. 2024Easy Problems That LLMs Get Wrongcited by 29
  13. 2024Security of AI Agentscited by 24
  14. 2023Tailored Visions: Enhancing Text-to-Image Generation with Personalized Prompt Rewritingcited by 23
  15. 2025SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMscited by 23
  16. 2025Evaluating Morphological Alignment of Tokenizers in 70 Languagescited by 22
  17. 2026SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Passcited by 22
  18. 2024Beyond Language Models: Byte Models are Digital World Simulatorscited by 21
  19. 2024SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modelingcited by 21
  20. 2024Subobject-level Image Tokenizationcited by 20
  21. 2024Lossless data compression by large modelscited by 20
  22. 2025Predictive Data Selection: The Data That Predicts Is the Data That Teachescited by 20
  23. 2025LeMat-GenBench: A Unified Evaluation Framework for Crystal Generative Modelscited by 20
  24. 2024MatExpert: Decomposing Materials Discovery by Mimicking Human Expertscited by 18
  25. 2025Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in Its Latent Thoughtscited by 18
  26. 2023Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspectivecited by 17
  27. 2024Training LLMs over Neurally Compressed Textcited by 16
  28. 2025Large Language Model for Lossless Image Compression with Visual Promptscited by 15
  29. 2026Training Language Models via Neural Cellular Automatacited by 15
  30. 2023Grokking as Compression: A Nonlinear Complexity Perspectivecited by 14
  31. 2024gzip Predicts Data-dependent Scaling Lawscited by 14
  32. 2024UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspectivecited by 14
  33. 2025Latent learning: episodic memory complements parametric learning by enabling flexible reuse of experiencescited by 14
  34. 2023In-Context Learning Dynamics with Random Binary Sequencescited by 13
  35. 2024Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Datacited by 13
  36. 2025Mitigating Visual Knowledge Forgetting in MLLM Instruction-tuning via Modality-decoupled Gradient Descentcited by 13
  37. 2024Tele-FLM Technical Reportcited by 12
  38. 2025R3Mem: Bridging Memory Retention and Retrieval via Reversible Compressioncited by 12

50 most cited of 225 All 225 (JSON)