# Fast and Informative Model Selection using Learning Curve Cross-Validation

arXiv: https://arxiv.org/abs/2111.13914 · HTML: https://mlsanity.com/paper/2111.13914/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 50 references that are in the map.
- First posted on arXiv 27 Nov 2021.

## Builds on (most cited first, 13 of 13)

- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Non-stochastic Best Arm Identification and Hyperparameter Optimization](https://mlsanity.com/paper/1502.07943/) (2015)
- [Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning](https://mlsanity.com/paper/2007.04074/) (2020)
- [Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets](https://mlsanity.com/paper/1605.07079/) (2016)
- [An Open Source AutoML Benchmark](https://mlsanity.com/paper/1907.00909/) (2019)
- [Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science](https://mlsanity.com/paper/1603.06212/) (2016)
- [The Shape of Learning Curves: a Review](https://mlsanity.com/paper/2103.10948/) (2021)
- [OpenML-Python: an extensible Python API for OpenML](https://mlsanity.com/paper/1911.02490/) (2019)
- [CMA-ES for Hyperparameter Optimization of Deep Neural Networks](https://mlsanity.com/paper/1604.07269/) (2016)
- [AutoDS: Towards Human-Centered Automation of Data Science](https://mlsanity.com/paper/2101.05273/) (2021)
- [Selecting Near-Optimal Learners via Incremental Data Allocation](https://mlsanity.com/paper/1601.00024/) (2015)
- [Naive Automated Machine Learning -- A Late Baseline for AutoML](https://mlsanity.com/paper/2103.10496/) (2021)

## Built on by (most cited first, 15 of 15)

- [Artificial intelligence to advance Earth observation: : A review of models, recent trends, and pathways forward](https://mlsanity.com/paper/2305.08413/) (2023)
- [Learning Curves for Decision Making in Supervised Machine Learning: A Survey](https://mlsanity.com/paper/2201.12150/) (2022)
- [Don't Waste Your Time: Early Stopping Cross-Validation](https://mlsanity.com/paper/2405.03389/) (2024)
- [The Unreasonable Effectiveness Of Early Discarding After One Epoch In Neural Network Hyperparameter Optimization](https://mlsanity.com/paper/2404.04111/) (2024)
- [Towards Meta-learned Algorithm Selection using Implicit Fidelity Information](https://mlsanity.com/paper/2206.03130/) (2022)
- [Learning to Rank Normalized Entropy Curves with Differentiable Window Transformation](https://mlsanity.com/paper/2301.10443/) (2023)
- [Selection, Ensemble, and Adaptation: Advancing Multi-Source-Free Domain Adaptation via Architecture Zoo](https://mlsanity.com/paper/2403.01582/) (2024)
- [LCDB 1.1: A Database Illustrating Learning Curves Are More Ill-Behaved Than Previously Thought](https://mlsanity.com/paper/2505.15657/) (2025)
- [Evidence-Driven Decision Support for AI Model Selection in Research Software Engineering](https://mlsanity.com/paper/2512.11984/) (2025)
- [Surfing the modeling of PoS taggers in low-resource scenarios](https://mlsanity.com/paper/2402.02449/) (2024)
- [AI Competitions and Benchmarks: Dataset Development](https://mlsanity.com/paper/2404.09703/) (2024)
- [Meta-Learning from Learning Curves for Budget-Limited Algorithm Selection](https://mlsanity.com/paper/2410.07696/) (2024)
- [Prediction by Machine Learning Analysis of Genomic Data Phenotypic Frost Tolerance in Perccottus glenii](https://mlsanity.com/paper/2410.08867/) (2024)
- [DCS-ST for Classification of Breast Cancer Histopathology Images with Limited Annotations](https://mlsanity.com/paper/2505.03204/) (2025)
- [Early-Stopping Thresholds for ES-HyperNEAT: A Data-Driven Approach from Fitness Dynamics](https://mlsanity.com/paper/2609.13533/) (2026)

## Abstract (arXiv)

Common cross-validation (CV) methods like k-fold cross-validation or Monte-Carlo cross-validation estimate the predictive performance of a learner by repeatedly training it on a large portion of the given data and testing on the remaining data. These techniques have two major drawbacks. First, they can be unnecessarily slow on large datasets. Second, beyond an estimation of the final performance, they give almost no insights into the learning process of the validated algorithm. In this paper, we present a new approach for validation based on learning curves (LCCV). Instead of creating train-test splits with a large portion of training data, LCCV iteratively increases the number of instances used for training. In the context of model selection, it discards models that are very unlikely to become competitive. We run a large scale experiment on the 67 datasets from the AutoML benchmark and empirically show that in over 90% of the cases using LCCV leads to similar performance (at most 1.5% difference) as using 5/10-fold CV. However, it yields substantial runtime reductions of over 20% on average. Additionally, it provides important insights, which for example allow assessing the benefits of acquiring more data. These results are orthogonal to other advances in the field of AutoML.