mlsanityData to 14 Oct 2026

Learning Curve Theory

arXiv 2102.04074 ↗First posted 8 Feb 2021cs.LG stat.MLcited by 85

Not on the map. Why?

Abstract (arXiv)

Recently a number of empirical "universal" scaling law papers have been published, most notably by OpenAI. `Scaling laws' refers to power-law decreases of training or test error w.r.t. more data, larger neural networks, and/or more compute. In this work we focus on scaling w.r.t. data size n. Theoretical understanding of this phenomenon is largely lacking, except in finite-dimensional models for which error typically decreases with n-1/2 or n-1, where n is the sample size. We develop and theoretically analyse the simplest possible (toy) model that can exhibit n-β learning curves for arbitrary power β>0, and determine whether power laws are universal or depend on the data distribution.

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). Builds on 12 of its 33 references that are in the map. First posted on arXiv 8 Feb 2021.

Builds on

12 of 33 references.

References · 12

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  3. 2019Introduction to Online Convex Optimizationcited by 803
  4. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  5. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  6. 2021Scaling Laws for Transfercited by 272
  7. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 210
  8. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  9. 2018Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolatecited by 99
  10. 2020A Neural Scaling Law from the Dimension of the Data Manifoldcited by 52
  11. 2019Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigmcited by 20
  12. 2020Learning Curves for Analysis of Deep Networkscited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 6; 2023: 10; 2024: 18; 2025: 23; 2026: 21.

85 papers.

Sort:

By area:

  1. 2021Explaining Neural Scaling Lawscited by 363
  2. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  3. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 184
  4. 2023The Quantization Model of Neural Scalingcited by 138
  5. 2022Revisiting Neural Scaling Laws in Language and Visioncited by 132
  6. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 130
  7. 2022A Solvable Model of Neural Scaling Lawscited by 116
  8. 2021Scaling Laws for Neural Machine Translationcited by 111
  9. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 89
  10. 2021The Shape of Learning Curves: a Reviewcited by 70
  11. 2021Scaling Scaling Laws with Board Gamescited by 63
  12. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 57
38 more
  1. 2024Scaling and renormalization in high-dimensional regressioncited by 50
  2. 2024Scaling Laws for Downstream Task Performance of Large Language Modelscited by 49
  3. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 47
  4. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 43
  5. 2024Reconciling Kaplan and Chinchilla Scaling Lawscited by 39
  6. 2024A Simple Model of Inference Scaling Lawscited by 36
  7. 2023Scaling Laws for Associative Memoriescited by 35
  8. 2025Superposition Yields Robust Neural Scalingcited by 31
  9. 2025Scaling Laws Are Unreliable for Downstream Tasks: A Reality Checkcited by 29
  10. 2023Scaling Laws for Multilingual Neural Machine Translationcited by 27
  11. 2021Scaling Law for Recommendation Models: Towards General-purpose User Representationscited by 26
  12. 2022How Much More Data Do I Need? Estimating Requirements for Downstream Taskscited by 23
  13. 2024Scaling Laws for Pre-training Agents and World Modelscited by 23
  14. 2026Deriving Neural Scaling Laws from the statistics of natural languagecited by 23
  15. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  16. 2026Olmo Hybrid: From Theory to Practice and Backcited by 22
  17. 2025Scaling Pre-training to One Hundred Billion Data for Vision Language Modelscited by 19
  18. 2022Error Scaling Laws for Kernel Classification under Source and Capacity Conditionscited by 17
  19. 2025Learning curves theory for hierarchically compositional data with power-law distributed featurescited by 17
  20. 2025Efficient Prediction of Pass@k Scaling in Large Language Modelscited by 16
  21. 2021A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?cited by 15
  22. 2024Loss-to-Loss Prediction: Scaling Laws for All Datasetscited by 15
  23. 2024Neural Scaling Laws Rooted in the Data Distributioncited by 15
  24. 2026On the origin of neural scaling laws: from random graphs to natural languagecited by 14
  25. 2023Pareto Frontiers in Neural Feature Learning: Data, Compute, Width, and Luckcited by 13
  26. 2025Scaling Laws for Upcycling Mixture-of-Experts Language Modelscited by 13
  27. 2025When Models Don't Collapse: On the Consistency of Iterative MLEcited by 13
  28. 2022Languages You Know Influence Those You Learn: Impact of Language Characteristics on Multi-Lingual Text-to-Text Transfercited by 11
  29. 2025Improved Scaling Laws in Linear Regression via Data Reusecited by 11
  30. 2023Delegated Classificationcited by 10
  31. 2024Scaling Laws for the Value of Individual Data Points in Machine Learningcited by 10
  32. 2025Recursive Inference Scaling: A Winning Path to Scalable Inference in Language and Multimodal Systemscited by 10
  33. 2026Universal One-third Time Scaling in Learning Peaked Distributionscited by 10
  34. 2023Multiperiodic Processes: Ergodic Sources with a Sublinear Entropycited by 9
  35. 2024Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectracited by 9
  36. 2025Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)cited by 9
  37. 2026Inverse Depth Scaling From Most Layers Being Similarcited by 9
  38. 2023Model-agnostic Measure of Generalization Difficultycited by 7

50 most cited of 85 All 85 (JSON)