mlsanityData to 7 Oct 2026

A Constructive Prediction of the Generalization Error Across Scales

arXiv 1909.12673 ↗First posted 27 Sep 2019cs.LG cs.CL cs.CVcited by 202

Not on the map. Why?

Abstract (arXiv)

The dependency of the generalization error of neural networks on model and dataset size is of critical importance both in practice and for understanding the theory of neural networks. Nevertheless, the functional form of this dependency remains elusive. In this work, we present a functional form which approximates well the generalization error in practice. Capitalizing on the successful concept of model scaling (e.g., width, depth), we are able to simultaneously construct such a form and specify the exact models which can attain it across model/data scales. Our construction follows insights obtained from observations conducted over a range of model/data scales, in various model types and datasets, in vision and language tasks. We show that the form both fits the observations well across scales, and provides accurate predictions from small- to large-scale models and data.

Key facts. Cited by 202 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 31; 2023: 38; 2024: 42; 2025: 46; 2026: 26.

202 papers.

Sort:

By area:

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  4. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  5. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  6. 2021Explaining Neural Scaling Lawscited by 353
  7. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  8. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  9. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  10. 2021Scaling Laws for Transfercited by 265
  11. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  12. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
38 more
  1. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  2. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
  3. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
  4. 2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
  5. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 155
  6. 2023The Quantization Model of Neural Scalingcited by 137
  7. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 137
  8. 2022Revisiting Neural Scaling Laws in Language and Visioncited by 129
  9. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 127
  10. 2021Scaling Laws for Neural Machine Translationcited by 110
  11. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  12. 2022Broken Neural Scaling Lawscited by 102
  13. 2022A Solvable Model of Neural Scaling Lawscited by 101
  14. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  15. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 84
  16. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 81
  17. 2021Learning Curve Theorycited by 79
  18. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  19. 2021The Shape of Learning Curves: a Reviewcited by 67
  20. 2021Scaling Scaling Laws with Board Gamescited by 62
  21. 2022Data Scaling Laws in NMT: The Effect of Noise and Architecturecited by 55
  22. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 54
  23. 2022Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsingcited by 50
  24. 2024Does equivariance matter at scale?cited by 49
  25. 2023kNN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inferencecited by 47
  26. 2022On Data Scaling in Masked Image Modelingcited by 45
  27. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  28. 2025Scaling Laws of Synthetic Data for Language Modelscited by 44
  29. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 40
  30. 2024Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMscited by 39
  31. 2022Scaling Laws for a Multi-Agent Reinforcement Learning Modelcited by 37
  32. 2023Scaling MLPs: A Tale of Inductive Biascited by 37
  33. 2024Reconciling Kaplan and Chinchilla Scaling Lawscited by 36
  34. 2024A Simple Model of Inference Scaling Lawscited by 36
  35. 2022Limitations of the NTK for Understanding Generalization in Deep Learningcited by 34
  36. 2023Pretraining on the Test Set Is All You Needcited by 33
  37. 2024Selecting Large Language Model to Fine-tune via Rectified Scaling Lawcited by 33
  38. 2024Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Datacited by 30

50 most cited of 202 All 202 (JSON)