mlsanityData to 7 Oct 2026

Explaining Neural Scaling Laws

arXiv 2102.06701 ↗First posted 12 Feb 2021cs.LG cond-mat.dis-nn stat.MLcited by 353

Not on the map. Why?

Abstract (arXiv)

The population loss of trained deep neural networks often follows precise power-law scaling relations with either the size of the training dataset or the number of parameters in the network. We propose a theory that explains the origins of and connects these scaling laws. We identify variance-limited and resolution-limited scaling behavior for both dataset and model size, for a total of four scaling regimes. The variance-limited scaling follows simply from the existence of a well-behaved infinite data or infinite width limit, while the resolution-limited regime can be explained by positing that models are effectively resolving a smooth data manifold. In the large width limit, this can be equivalently obtained from the spectrum of certain kernels, and we present evidence that large width and large dataset resolution-limited scaling exponents are related by a duality. We exhibit all four scaling regimes in the controlled setting of large random feature and pretrained models and test the predictions empirically on a range of standard architectures and datasets. We also observe several empirical relationships between datasets and scaling exponents under modifications of task and architecture aspect ratio. Our work provides a taxonomy for classifying different scaling regimes, underscores that there can be different mechanisms driving improvements in loss, and lends insight into the microscopic origins of and relationships between scaling exponents.

Key facts. Cited by 353 papers in the map (data through 7 Oct 2026). Builds on 51 of its 99 references that are in the map. First posted on arXiv 12 Feb 2021.

Builds on

51 of 99 references.

References · 51

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2016Wide Residual Networkscited by 3,057
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  9. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  10. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  11. 2018On Lazy Training in Differentiable Programmingcited by 671
  12. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
38 more
  1. 2017Deep Neural Networks as Gaussian Processescited by 617
  2. 2018Do Better ImageNet Models Transfer Better?cited by 560
  3. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  4. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  5. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  6. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  7. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 402
  8. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  9. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  10. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  11. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  12. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  13. 2016Deep Information Propagationcited by 194
  14. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  15. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  16. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  17. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  18. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  19. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  20. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
  21. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  22. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  23. 2017Sobolev Norm Learning Rates for Regularized Least-Squares Algorithmcited by 113
  24. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  25. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  26. 2019Scaling description of generalization with number of parameters in deep learningcited by 83
  27. 2021Learning Curve Theorycited by 79
  28. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  29. 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
  30. 2020Understanding Double Descent Requires a Fine-Grained Bias-Variance Decompositioncited by 75
  31. 2018A Continuous-Time View of Early Stopping for Least Squarescited by 67
  32. 2020Neural Kernels Without Tangentscited by 56
  33. 2020A Neural Scaling Law from the Dimension of the Data Manifoldcited by 48
  34. 2020Distributional Generalization: A New Kind of Generalizationcited by 41
  35. 2019More Data Can Hurt for Linear Regression: Sample-wise Double Descentcited by 37
  36. 2020On the Predictability of Pruning Across Scalescited by 33
  37. 2020Asymptotics of Wide Convolutional Neural Networkscited by 15
  38. 2019Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspectivecited by 11

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 33; 2023: 26; 2024: 83; 2025: 108; 2026: 88.

353 papers.

Sort:

By area:

  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  2. 2023Large Language Modelscited by 774
  3. 2024KAN: Kolmogorov-Arnold Networkscited by 634
  4. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  5. 2023Scaling Data-Constrained Language Modelscited by 391
  6. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  7. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  8. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  9. 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Aheadcited by 155
  10. 2023The Quantization Model of Neural Scalingcited by 137
  11. 2022Revisiting Neural Scaling Laws in Language and Visioncited by 129
  12. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 128
38 more
  1. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  2. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 111
  3. 2021Scaling Laws for Neural Machine Translationcited by 110
  4. 2024A Dynamical Model of Neural Scaling Lawscited by 105
  5. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  6. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  7. 2022Broken Neural Scaling Lawscited by 102
  8. 2022A Solvable Model of Neural Scaling Lawscited by 101
  9. 2024Scaling Laws for Precisioncited by 92
  10. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
  11. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 87
  12. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  13. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 74
  14. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  15. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  16. 2022Data Feedback Loops: Model-driven Amplification of Dataset Biasescited by 59
  17. 2025Distillation Scaling Lawscited by 59
  18. 2024Scaling Wearable Foundation Modelscited by 57
  19. 2022Data Scaling Laws in NMT: The Effect of Noise and Architecturecited by 55
  20. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  21. 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 54
  22. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 54
  23. 2022Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsingcited by 50
  24. 2023Unifying Grokking and Double Descentcited by 49
  25. 2024Scaling Laws for Downstream Task Performance of Large Language Modelscited by 49
  26. 2024Does equivariance matter at scale?cited by 49
  27. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  28. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  29. 2025Scaling Laws of Synthetic Data for Language Modelscited by 44
  30. 2023Feature-Learning Networks Are Consistent Across Widths At Realistic Scalescited by 40
  31. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 40
  32. 2023Scaling laws for single-agent reinforcement learningcited by 39
  33. 2022Understanding Scaling Laws for Recommendation Modelscited by 38
  34. 2024Towards a theory of how the structure of language is acquired by deep neural networkscited by 38
  35. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 38
  36. 2023Scaling MLPs: A Tale of Inductive Biascited by 37
  37. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  38. 2024On the Scalability of GNNs for Molecular Graphscited by 36

50 most cited of 353 All 353 (JSON)