# Learning Curves for Decision Making in Supervised Machine Learning: A Survey

arXiv: https://arxiv.org/abs/2201.12150 · HTML: https://mlsanity.com/paper/2201.12150/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 87 references that are in the map.
- First posted on arXiv 28 Jan 2022.

## Builds on (most cited first, 18 of 18)

- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Non-stochastic Best Arm Identification and Hyperparameter Optimization](https://mlsanity.com/paper/1502.07943/) (2015)
- [Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets](https://mlsanity.com/paper/1605.07079/) (2016)
- [Optimal Regularization Can Mitigate Double Descent](https://mlsanity.com/paper/2003.01897/) (2020)
- [Freeze-Thaw Bayesian Optimization](https://mlsanity.com/paper/1406.3896/) (2014)
- [The Shape of Learning Curves: a Review](https://mlsanity.com/paper/2103.10948/) (2021)
- [How much data is needed to train a medical image deep learning system to achieve necessary high accuracy?](https://mlsanity.com/paper/1511.06348/) (2015)
- [Small Data, Big Decisions: Model Selection in the Small-Data Regime](https://mlsanity.com/paper/2009.12583/) (2020)
- [Fast and Informative Model Selection using Learning Curve Cross-Validation](https://mlsanity.com/paper/2111.13914/) (2021)
- [Selecting Near-Optimal Learners via Incremental Data Allocation](https://mlsanity.com/paper/1601.00024/) (2015)
- [Minimizers of the Empirical Risk and Risk Monotonicity](https://mlsanity.com/paper/1907.05476/) (2019)
- [Risk-Monotonicity in Statistical Learning](https://mlsanity.com/paper/2011.14126/) (2020)
- [Making Learners (More) Monotone](https://mlsanity.com/paper/1911.11030/) (2019)
- [Progressive Sampling-Based Bayesian Optimization for Efficient and Automatic Machine Learning Model Selection](https://mlsanity.com/paper/1812.02855/) (2018)
- [Recommending Training Set Sizes for Classification](https://mlsanity.com/paper/2102.09382/) (2021)
- [Inductive Transfer for Neural Architecture Optimization](https://mlsanity.com/paper/1903.03536/) (2019)

## Built on by (most cited first, 27 of 27)

- [Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks](https://mlsanity.com/paper/2310.20447/) (2023)
- [Artificial intelligence to advance Earth observation: : A review of models, recent trends, and pathways forward](https://mlsanity.com/paper/2305.08413/) (2023)
- [Scaling Laws for Hyperparameter Optimization](https://mlsanity.com/paper/2302.00441/) (2023)
- [PASHA: Efficient HPO and NAS with Progressive Resource Allocation](https://mlsanity.com/paper/2207.06940/) (2022)
- [Optimizing Hyperparameters with Conformal Quantile Regression](https://mlsanity.com/paper/2305.03623/) (2023)
- [Interactive Hyperparameter Optimization in Multi-Objective Problems via Preference Learning](https://mlsanity.com/paper/2309.03581/) (2023)
- [Hyperparameter Importance of Quantum Neural Networks Across Small Datasets](https://mlsanity.com/paper/2206.09992/) (2022)
- [Multi-Fidelity Multi-Armed Bandits Revisited](https://mlsanity.com/paper/2306.07761/) (2023)
- [Is One Epoch All You Need For Multi-Fidelity Hyperparameter Optimization?](https://mlsanity.com/paper/2307.15422/) (2023)
- [Keeping Deep Learning Models in Check: A History-Based Approach to Mitigate Overfitting](https://mlsanity.com/paper/2401.10359/) (2024)
- [Beyond the Single-Best Model: Rashomon Partial Dependence Profile for Trustworthy Explanations in AutoML](https://mlsanity.com/paper/2507.14744/) (2025)
- [The Unreasonable Effectiveness Of Early Discarding After One Epoch In Neural Network Hyperparameter Optimization](https://mlsanity.com/paper/2404.04111/) (2024)
- [Unraveling overoptimism and publication bias in ML-driven science](https://mlsanity.com/paper/2405.14422/) (2024)
- [MD tree: a model-diagnostic tree grown on loss landscape](https://mlsanity.com/paper/2406.16988/) (2024)
- [Towards Meta-learned Algorithm Selection using Implicit Fidelity Information](https://mlsanity.com/paper/2206.03130/) (2022)
- [A Survey of Learning Curves with Bad Behavior: or How More Data Need Not Lead to Better Performance](https://mlsanity.com/paper/2211.14061/) (2022)
- [AGTCNet: A Graph-Temporal Approach for Principled Motor Imagery EEG Classification](https://mlsanity.com/paper/2506.21338/) (2025)
- [Zero-Shot Performance Prediction for Probabilistic Scaling Laws](https://mlsanity.com/paper/2510.16743/) (2025)
- [Meta-learning from Learning Curves Challenge: Lessons learned from the First Round and Design of the Second Round](https://mlsanity.com/paper/2208.02821/) (2022)
- [Learning to Rank Normalized Entropy Curves with Differentiable Window Transformation](https://mlsanity.com/paper/2301.10443/) (2023)
- [LCDB 1.1: A Database Illustrating Learning Curves Are More Ill-Behaved Than Previously Thought](https://mlsanity.com/paper/2505.15657/) (2025)
- [Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks](https://mlsanity.com/paper/2505.23032/) (2025)
- [Characterization of Transfer Using Multi-task Learning Curves](https://mlsanity.com/paper/2512.24866/) (2025)
- [AI Competitions and Benchmarks: Dataset Development](https://mlsanity.com/paper/2404.09703/) (2024)
- [Meta-Learning from Learning Curves for Budget-Limited Algorithm Selection](https://mlsanity.com/paper/2410.07696/) (2024)
- [PyAWD: A Library for Generating Large Synthetic Datasets of Acoustic Wave Propagation](https://mlsanity.com/paper/2411.12636/) (2024)
- [Towards Scaling Law Analysis For Spatiotemporal Weather Data](https://mlsanity.com/paper/2604.05068/) (2026)

## Abstract (arXiv)

Learning curves are a concept from social sciences that has been adopted in the context of machine learning to assess the performance of a learning algorithm with respect to a certain resource, e.g., the number of training examples or the number of training iterations. Learning curves have important applications in several machine learning contexts, most notably in data acquisition, early stopping of model training, and model selection. For instance, learning curves can be used to model the performance of the combination of an algorithm and its hyperparameter configuration, providing insights into their potential suitability at an early stage and often expediting the algorithm selection process. Various learning curve models have been proposed to use learning curves for decision making. Some of these models answer the binary decision question of whether a given algorithm at a certain budget will outperform a certain reference performance, whereas more complex models predict the entire learning curve of an algorithm. We contribute a framework that categorises learning curve approaches using three criteria: the decision-making situation they address, the intrinsic learning curve question they answer and the type of resources they use. We survey papers from the literature and classify them into this framework.