mlsanityData to 7 Oct 2026

The Quantization Model of Neural Scaling

arXiv 2303.13506 ↗First posted 23 Mar 2023cs.LG cond-mat.dis-nncited by 137

Not on the map. Why?

Abstract (arXiv)

We propose the Quantization Model of neural scaling laws, explaining both the observed power law dropoff of loss with model and data size, and also the sudden emergence of new capabilities with scale. We derive this model from what we call the Quantization Hypothesis, where network knowledge and skills are "quantized" into discrete chunks (quanta). We show that when quanta are learned in order of decreasing use frequency, then a power law in use frequencies explains observed power law scaling of loss. We validate this prediction on toy datasets, then study how scaling curves decompose for large language models. Using language model gradients, we automatically decompose model behavior into a diverse set of skills (quanta). We tentatively find that the frequency at which these quanta are used in the training distribution roughly follows a power law corresponding with the empirical scaling exponent for language models, a prediction of our theory.

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). Builds on 27 of its 54 references that are in the map. First posted on arXiv 23 Mar 2023.

Builds on

27 of 54 references.

References · 27

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  7. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  8. 2021Scaling Vision Transformerscited by 993
  9. 2022In-context Learning and Induction Headscited by 930
  10. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  11. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  12. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
15 more
  1. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  2. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  3. 2021Explaining Neural Scaling Lawscited by 353
  4. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  5. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 241
  6. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 215
  7. 2022Predictability and Surprise in Large Generative Modelscited by 209
  8. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  9. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  10. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  11. 2022A Solvable Model of Neural Scaling Lawscited by 101
  12. 2021Learning Curve Theorycited by 79
  13. 2022Understanding Scaling Laws for Recommendation Modelscited by 38
  14. 2021Scaling Laws for Acoustic Modelscited by 20
  15. 2022Precision Machine Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 38; 2025: 33; 2026: 49.

137 papers.

Sort:

By area:

  1. 2023Large Language Modelscited by 774
  2. 2024KAN: Kolmogorov-Arnold Networkscited by 634
  3. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  4. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  5. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  6. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  7. 2023Language Models Represent Space and Timecited by 306
  8. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  9. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 155
  10. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  11. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  12. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 127
38 more
  1. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  2. 2023Explaining grokking through circuit efficiencycited by 105
  3. 2024A Dynamical Model of Neural Scaling Lawscited by 105
  4. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 78
  5. 2024Chinchilla Scaling: A replication attemptcited by 65
  6. 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 60
  7. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 54
  8. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  9. 2026When Models Manipulate Manifolds: The Geometry of a Counting Taskcited by 43
  10. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 40
  11. 2023Seeing is Believing: Brain-Inspired Modular Training for Mechanistic Interpretabilitycited by 38
  12. 2024Scaling Optimal LR Across Token Horizonscited by 35
  13. 2026When Single-Agent with Skills Replace Multi-Agent Systems and When They Failcited by 34
  14. 2025Tracing the Representation Geometry of Language Models from Pretraining to Post-trainingcited by 33
  15. 2023Scaling Laws for Associative Memoriescited by 30
  16. 2025Superposition Yields Robust Neural Scalingcited by 30
  17. 2024Scaling Law with Learning Rate Annealingcited by 29
  18. 2024A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Languagecited by 26
  19. 2024Compute Better Spent: Replacing Dense Layers with Structured Matricescited by 24
  20. 2024Concept-skill Transferability-based Data Selection for Large Vision-Language Modelscited by 24
  21. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  22. 2024Unified View of Grokking, Double Descent and Emergent Abilities: A Perspective from Circuits Competitioncited by 23
  23. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  24. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  25. 2026Deriving Neural Scaling Laws from the statistics of natural languagecited by 18
  26. 2025Learning curves theory for hierarchically compositional data with power-law distributed featurescited by 17
  27. 2024Loss-to-Loss Prediction: Scaling Laws for All Datasetscited by 15
  28. 2024Neural Scaling Laws Rooted in the Data Distributioncited by 14
  29. 2025Intrinsic Entropy of Context Length Scaling in LLMscited by 14
  30. 2025Efficient Prediction of Pass@k Scaling in Large Language Modelscited by 14
  31. 2023Pareto Frontiers in Neural Feature Learning: Data, Compute, Width, and Luckcited by 13
  32. 2023In-Context Learning Dynamics with Random Binary Sequencescited by 13
  33. 2024Sparse Autoencoders Enable Scalable and Reliable Circuit Identification in Language Modelscited by 13
  34. 2024Revisiting the Superficial Alignment Hypothesiscited by 12
  35. 2026On the origin of neural scaling laws: from random graphs to natural languagecited by 12
  36. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  37. 2025Understanding sparse autoencoder scaling in the presence of feature manifoldscited by 11
  38. 2025Decomposing Elements of Problem Solving: What "Math" Does RL Teach?cited by 10

50 most cited of 137 All 137 (JSON)