mlsanityData to 14 Oct 2026

Neural Scaling Laws Rooted in the Data Distribution

arXiv 2412.07942 ↗First posted 10 Dec 2024cs.LG cond-mat.dis-nncited by 15

Not on the map. Why?

Abstract (arXiv)

Deep neural networks exhibit empirical neural scaling laws, with error decreasing as a power law with increasing model or data size, across a wide variety of architectures, tasks, and datasets. This universality suggests that scaling laws may result from general properties of natural learning tasks. We develop a mathematical model intended to describe natural datasets using percolation theory. Two distinct criticality regimes emerge, each yielding optimal power-law neural scaling laws. These regimes, corresponding to power-law-distributed discrete subtasks and a dominant data manifold, can be associated with previously proposed theories of neural scaling, thereby grounding and unifying prior works. We test the theory by training regression models on toy datasets derived from percolation theory simulations. We suggest directions for quantitatively predicting language model scaling.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 40 of its 67 references that are in the map. First posted on arXiv 10 Dec 2024.

Builds on

40 of 67 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  7. 2022Emergent Abilities of Large Language Modelscited by 2,512
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  9. 2022In-context Learning and Induction Headscited by 964
  10. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  11. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  12. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
28 more
  1. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  2. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  3. 2024The Platonic Representation Hypothesiscited by 382
  4. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  5. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  6. 2022Transformers Learn Shortcuts to Automatacited by 268
  7. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 210
  8. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  9. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  10. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  11. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 152
  12. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 144
  13. 2023The Quantization Model of Neural Scalingcited by 138
  14. 2022Revisiting Neural Scaling Laws in Language and Visioncited by 132
  15. 2022A Solvable Model of Neural Scaling Lawscited by 116
  16. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 113
  17. 2021Scaling Laws for Neural Machine Translationcited by 111
  18. 2022Broken Neural Scaling Lawscited by 109
  19. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 93
  20. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 89
  21. 2021Learning Curve Theorycited by 85
  22. 2024Chinchilla Scaling: A replication attemptcited by 72
  23. 2021Scaling Scaling Laws with Board Gamescited by 63
  24. 2023Scaling MLPs: A Tale of Inductive Biascited by 37
  25. 2023Scaling Laws for Associative Memoriescited by 35
  26. 2023The Exact Sample Complexity Gain from Invariances for Kernel Regressioncited by 28
  27. 2024A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Languagecited by 26
  28. 2021Why is AI hard and Physics simple?cited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 9.

15 papers.

Sort:

By area:

  1. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 43
  2. 2025Superposition Yields Robust Neural Scalingcited by 31
  3. 2025Understanding sparse autoencoder scaling in the presence of feature manifoldscited by 11
  4. 2026Universal One-third Time Scaling in Learning Peaked Distributionscited by 10
  5. 2025X-Factor: Quality Is a Dataset-Intrinsic Propertycited by 3
  6. 2026Towards Worst-Case Guarantees with Scale-Aware Interpretabilitycited by 3
  7. 2025Representation Learning on a Random Latticecited by 2
  8. 2025Evaluating the Robustness of Chinchilla Compute-Optimal Scalingcited by 2
  9. 2026Baby Scale: Investigating Models Trained on Individual Children's Language Inputcited by 2
  10. 2026Faithful-Patchscopes: Understanding and Mitigating Model Bias in Hidden Representations Explanation of Large Language Modelscited by 1
  11. 2026Smooth Scaling Laws Hide Stepwise Token Learningcited by 1
  12. 2026Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained Worldcited by 0
3 more
  1. 2026Critical Percolation as a Synthetic Data Model for Interpretabilitycited by 0
  2. 2026Symmetry without a manifold: intrinsic dimension on orbitscited by 0
  3. 2026Emergent One-Third Scaling Law as Attention Tries to Concentratecited by 0