mlsanityData to 14 Oct 2026

Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory

arXiv 2510.12077 ↗First posted 14 Oct 2025stat.ML cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

We study neural network compressibility by using singular learning theory to extend the minimum description length (MDL) principle to singular models like neural networks. Through extensive experiments on the Pythia suite with quantization, factorization, and other compression techniques, we find that complexity estimates based on the local learning coefficient (LLC) are closely, and in some cases, linearly correlated with compressibility. Our results provide a path toward rigorously evaluating the limits of model compression.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 37 of its 56 references that are in the map. First posted on arXiv 14 Oct 2025.

Builds on

37 of 56 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  6. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  8. 2022Toy Models of Superpositioncited by 971
  9. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  10. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  11. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  12. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
25 more
  1. 2018Stronger generalization bounds for deep nets via a compression approachcited by 307
  2. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 301
  3. 2023A Survey on Model Compression for Large Language Modelscited by 269
  4. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  5. 2013Predicting Parameters in Deep Learningcited by 224
  6. 2021Accelerating Sparse Deep Neural Networkscited by 223
  7. 2013A General Framework for Updating Belief Distributionscited by 175
  8. 2015Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networkscited by 121
  9. 2024Scaling Laws for Precisioncited by 95
  10. 2025Distillation Scaling Lawscited by 67
  11. 2014Consistency and fluctuations for stochastic gradient Langevin dynamicscited by 65
  12. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  13. 2020Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisitedcited by 51
  14. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 51
  15. 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 40
  16. 2019Minimum Description Length Revisitedcited by 37
  17. 2016On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integratorscited by 35
  18. 2023Dynamical versus Bayesian Phase Transitions in a Toy Model of Superpositioncited by 29
  19. 2024Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokenscited by 18
  20. 2025Dynamics of Transient Structure in In-Context Linear Regression Transformerscited by 14
  21. 2025Compression Scaling Laws:Unifying Sparsity and Quantizationcited by 11
  22. 2024Characterizing the Accuracy -- Efficiency Trade-off of Low-rank Decomposition in Language Modelscited by 4
  23. 2024Scaling Laws for Post Training Quantized Large Language Modelscited by 4
  24. 2025Structure Development in List-Sorting Transformerscited by 4
  25. 2025Modes of Sequence Models and Learning Coefficientscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Patterning: The Dual of Interpretabilitycited by 7
  2. 2026Self-Improvement as Coherence Optimization: A Theoretical Accountcited by 1
  3. 2026Grokking From Abstraction to Intelligencecited by 1
  4. 2026Dead Directions: Geometric Singular Learningcited by 1
  5. 2026When Updating Stops Being Learning: Rethinking LLM Self-Evolution via learnable information gaincited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient compression of neural networks and datasets
  2. 2026Algorithmic Simplification of Neural Networks with Mosaic-of-Motifs
  3. 2022Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Models
  4. 2025Unified Scaling Laws for Compressed Representations
  5. 2026Big2Small: A Unifying Neural Network Framework for Model Compression
  6. 2026Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data
  7. 2025Theoretical Compression Bounds for Wide Multilayer Perceptrons
  8. 2026Dynamics as Code: On Model Compression via Dynamic System