mlsanityData to 7 Oct 2026

Bridging Information-Theoretic and Geometric Compression in Language Models

arXiv 2310.13620 ↗First posted 20 Oct 2023cs.CLcited by 26

Not on the map. Why?

Abstract (arXiv)

For a language model (LM) to faithfully model human language, it must compress vast, potentially infinite information into relatively few dimensions. We propose analyzing compression in (pre-trained) LMs from two points of view: geometric and information-theoretic. We demonstrate that the two views are highly correlated, such that the intrinsic geometric dimension of linguistic data predicts their coding length under the LM. We then show that, in turn, high compression of a linguistic dataset predicts rapid adaptation to that dataset, confirming that being able to compress linguistic information is an important part of successful LM performance. As a practical byproduct of our analysis, we evaluate a battery of intrinsic dimension estimators for the first time on linguistic data, showing that only some encapsulate the relationship between information-theoretic compression, geometric compression, and ease-of-adaptation.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 34 of its 69 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

34 of 69 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  10. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  11. 2015Teaching Machines to Read and Comprehendcited by 1,039
  12. 2020A Primer in BERTology: What we know about how BERT workscited by 807
22 more
  1. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  2. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  3. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  4. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  5. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
  6. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  7. 2020A Survey on Neural Network Interpretabilitycited by 291
  8. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
  9. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 246
  10. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  11. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  12. 2018The Description Length of Deep Learning Modelscited by 71
  13. 2020Emergence of Separable Manifolds in Deep Language Representationscited by 39
  14. 2021The Low-Dimensional Linear Geometry of Contextualized Word Representationscited by 39
  15. 2019Dimensionality compression and expansion in Deep Neural Networkscited by 38
  16. 2020On Information Plane Analyses of Neural Network Classifiers -- A Reviewcited by 35
  17. 2018Scalable Mutual Information Estimation using Dependence Graphscited by 22
  18. 2021Rissanen Data Analysis: Examining Dataset Characteristics via Description Lengthcited by 22
  19. 2022Intrinsic Dimensionality Estimation within Tight Localities: A Theoretical and Experimental Analysiscited by 22
  20. 2019Estimating the effective dimension of large biological datasets using Fisher separability analysiscited by 19
  21. 2019Adaptive Estimators Show Information Compression in Deep Neural Networkscited by 14
  22. 2019Intrinsic dimension estimation for locally undersampled datacited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 10; 2026: 6.

26 papers.

Sort:

By area:

  1. 2025EQ-VAE: Equivariance Regularized Latent Space for Improved Generative Image Modelingcited by 85
  2. 2024Emergence of a High-Dimensional Abstraction Phase in Language Transformerscited by 56
  3. 2025The Intrinsic Dimension of Prompts in Internal Representations of Large Language Modelscited by 17
  4. 2024The representation landscape of few-shot learning and fine-tuning in large language modelscited by 14
  5. 2024Geometric Signatures of Compositionality Across a Language Model's Lifetimecited by 14
  6. 2025Intrinsic Entropy of Context Length Scaling in LLMscited by 14
  7. 2024Persistent Topological Features in Large Language Modelscited by 11
  8. 2025Geometry of Decision Making in Language Modelscited by 8
  9. 2024GeLoRA: Geometric Adaptive Ranks For Efficient LoRA Fine-tuningcited by 7
  10. 2025On the Predictive Power of Representation Dispersion in Language Modelscited by 6
  11. 2024Interpretability of Language Models via Task Spacescited by 4
  12. 2024Intrinsic Dimension Correlation: uncovering nonlinear connections in multimodal representationscited by 4
14 more
  1. 2024ResiDual Transformer Alignment with Spectral Decompositioncited by 4
  2. 2025Prediction hubs are context-informed frequent tokens in LLMscited by 3
  3. 2025Bridging the Dimensional Chasm: Uncover Layer-wise Dimensional Reduction in Transformers through Token Correlationcited by 3
  4. 2025TRACE for Tracking the Emergence of Semantic Representations in Transformerscited by 3
  5. 2025On Entity Identification in Language Modelscited by 3
  6. 2025Controlling the Risk of Corrupted Contexts for Language Models via Early-Exitingcited by 3
  7. 2026Tracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of LLM representationscited by 3
  8. 2024Unsupervised detection of semantic correlations in big datacited by 2
  9. 2026Abstraction Induces the Brain Alignment of Language and Speech Modelscited by 2
  10. 2024A Comparative Study of Learning Paradigms in Large Language Models via Intrinsic Dimensioncited by 1
  11. 2026Geometric and Information Compression of Representations in Deep Learningcited by 0
  12. 2026Evidence for feature-specific error correction in LLMscited by 0
  13. 2026What Makes Linguistic Representations Good Models of High-Level Visual Perception in the Human Brain?cited by 0
  14. 2026The Changing Geometry of Grammar: Dimensionality and Neighborhood Reorganization across Transformer Layerscited by 0