# The Shape of Learning Curves: a Review

arXiv: https://arxiv.org/abs/2103.10948 · HTML: https://mlsanity.com/paper/2103.10948/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 214 references that are in the map.
- First posted on arXiv 19 Mar 2021.

## Builds on (most cited first, 30 of 33)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [Characterizing and Avoiding Negative Transfer](https://mlsanity.com/paper/1811.09751/) (2018)
- [A Constructive Prediction of the Generalization Error Across Scales](https://mlsanity.com/paper/1909.12673/) (2019)
- [Learning Visual Features from Large Weakly Supervised Data](https://mlsanity.com/paper/1511.02251/) (2015)
- [A review of domain adaptation without target labels](https://mlsanity.com/paper/1901.05335/) (2019)
- [Optimal Regularization Can Mitigate Double Descent](https://mlsanity.com/paper/2003.01897/) (2020)
- [Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It](https://mlsanity.com/paper/1412.3730/) (2014)
- [Learning Curve Theory](https://mlsanity.com/paper/2102.04074/) (2021)
- [A jamming transition from under- to over-parametrization affects loss landscape and generalization](https://mlsanity.com/paper/1810.09665/) (2018)
- [Triple descent and the two kinds of overfitting: Where & why do they appear?](https://mlsanity.com/paper/2006.03509/) (2020)
- [A Theory of Universal Learning](https://mlsanity.com/paper/2011.04483/) (2020)
- [Multiple Descent: Design Your Own Generalization Curve](https://mlsanity.com/paper/2008.01036/) (2020)
- [A Brief Prehistory of Double Descent](https://mlsanity.com/paper/2004.04328/) (2020)
- [More Data Can Hurt for Linear Regression: Sample-wise Double Descent](https://mlsanity.com/paper/1912.07242/) (2019)
- [Small Data, Big Decisions: Model Selection in the Small-Data Regime](https://mlsanity.com/paper/2009.12583/) (2020)
- [Learning Curves for Analysis of Deep Networks](https://mlsanity.com/paper/2010.11029/) (2020)
- [A critical analysis of metrics used for measuring progress in artificial intelligence](https://mlsanity.com/paper/2008.02577/) (2020)
- [Introducing Geometry in Active Learning for Image Segmentation](https://mlsanity.com/paper/1508.04955/) (2015)
- [Posterior Variance Analysis of Gaussian Processes with Application to Average Learning Curves](https://mlsanity.com/paper/1906.01404/) (2019)
- [Minimizers of the Empirical Risk and Risk Monotonicity](https://mlsanity.com/paper/1907.05476/) (2019)
- [Risk-Monotonicity in Statistical Learning](https://mlsanity.com/paper/2011.14126/) (2020)
- [Making Learners (More) Monotone](https://mlsanity.com/paper/1911.11030/) (2019)
- [The Peaking Phenomenon in Semi-supervised Learning](https://mlsanity.com/paper/1610.05160/) (2016)

## Built on by (most cited first, 30 of 70)

- [Learning Curves for Decision Making in Supervised Machine Learning: A Survey](https://mlsanity.com/paper/2201.12150/) (2022)
- [How Much More Data Do I Need? Estimating Requirements for Downstream Tasks](https://mlsanity.com/paper/2207.01725/) (2022)
- [DCI-ES: An Extended Disentanglement Framework with Connections to Identifiability](https://mlsanity.com/paper/2210.00364/) (2022)
- [Learning curves for the multi-class teacher-student perceptron](https://mlsanity.com/paper/2203.12094/) (2022)
- [Learning curves for Gaussian process regression with power-law priors and targets](https://mlsanity.com/paper/2110.12231/) (2021)
- [Fast and Informative Model Selection using Learning Curve Cross-Validation](https://mlsanity.com/paper/2111.13914/) (2021)
- [Optimizing Data Collection for Machine Learning](https://mlsanity.com/paper/2210.01234/) (2022)
- [The Data Addition Dilemma](https://mlsanity.com/paper/2408.04154/) (2024)
- [Multiclass Learnability Beyond the PAC Framework: Universal Rates and Partial Concept Classes](https://mlsanity.com/paper/2210.02297/) (2022)
- [PASHA: Efficient HPO and NAS with Progressive Resource Allocation](https://mlsanity.com/paper/2207.06940/) (2022)
- [Reinforcement Learning with Automated Auxiliary Loss Search](https://mlsanity.com/paper/2210.06041/) (2022)
- [Building a Llama2-finetuned LLM for Odia Language Utilizing Domain Knowledge Instruction Set](https://mlsanity.com/paper/2312.12624/) (2023)
- [Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes](https://mlsanity.com/paper/2210.07612/) (2022)
- [Do Construction Distributions Shape Formal Language Learning In German BabyLMs?](https://mlsanity.com/paper/2503.11593/) (2025)
- [Delegated Classification](https://mlsanity.com/paper/2306.11475/) (2023)
- [Monotone Learning](https://mlsanity.com/paper/2202.05246/) (2022)
- [Model Reporting for Certifiable AI: A Proposal from Merging EU Regulation into AI Development](https://mlsanity.com/paper/2307.11525/) (2023)
- [Are Ensembles Getting Better all the Time?](https://mlsanity.com/paper/2311.17885/) (2023)
- [Subword models struggle with word learning, but surprisal hides it](https://mlsanity.com/paper/2502.12835/) (2025)
- [The Unreasonable Effectiveness Of Early Discarding After One Epoch In Neural Network Hyperparameter Optimization](https://mlsanity.com/paper/2404.04111/) (2024)
- [Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learning](https://mlsanity.com/paper/2506.05447/) (2025)
- [Turing-Universal Learners with Optimal Scaling Laws](https://mlsanity.com/paper/2111.05321/) (2021)
- [Impact of dataset size and long-term ECoG-based BCI usage on deep learning decoders performance](https://mlsanity.com/paper/2209.03789/) (2022)
- [Unraveling overoptimism and publication bias in ML-driven science](https://mlsanity.com/paper/2405.14422/) (2024)
- [MD tree: a model-diagnostic tree grown on loss landscape](https://mlsanity.com/paper/2406.16988/) (2024)
- [On Learning-Curve Monotonicity for Maximum Likelihood Estimators](https://mlsanity.com/paper/2512.10220/) (2025)
- [In-Context Probing for Membership Inference in Fine-Tuned Language Models](https://mlsanity.com/paper/2512.16292/) (2025)
- [Improving Controllable Generation: Faster Training and Better Performance via x0-Supervision](https://mlsanity.com/paper/2604.05761/) (2026)
- [Parameter-Efficient Quantum-Inspired Fast Weight Programmers for Traffic-Matrix Forecasting](https://mlsanity.com/paper/2606.27821/) (2026)
- [A Survey of Learning Curves with Bad Behavior: or How More Data Need Not Lead to Better Performance](https://mlsanity.com/paper/2211.14061/) (2022)

## Abstract (arXiv)

Learning curves provide insight into the dependence of a learner's generalization performance on the training set size. This important tool can be used for model selection, to predict the effect of more training data, and to reduce the computational complexity of model training and hyperparameter tuning. This review recounts the origins of the term, provides a formal definition of the learning curve, and briefly covers basics such as its estimation. Our main contribution is a comprehensive overview of the literature regarding the shape of learning curves. We discuss empirical and theoretical evidence that supports well-behaved curves that often have the shape of a power law or an exponential. We consider the learning curves of Gaussian processes, the complex shapes they can display, and the factors influencing them. We draw specific attention to examples of learning curves that are ill-behaved, showing worse learning performance with more training data. To wrap up, we point out various open problems that warrant deeper empirical and theoretical investigation. All in all, our review underscores that learning curves are surprisingly diverse and no universal model can be identified.