# Small Data, Big Decisions: Model Selection in the Small-Data Regime

arXiv: https://arxiv.org/abs/2009.12583 · HTML: https://mlsanity.com/paper/2009.12583/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Sep 2020.

## Built on by (most cited first, 23 of 23)

- [The Shape of Learning Curves: a Review](https://mlsanity.com/paper/2103.10948/) (2021)
- [GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasets](https://mlsanity.com/paper/2204.02782/) (2022)
- [Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing](https://mlsanity.com/paper/2205.12253/) (2022)
- [Learning Curves for Decision Making in Supervised Machine Learning: A Survey](https://mlsanity.com/paper/2201.12150/) (2022)
- [PASHA: Efficient HPO and NAS with Progressive Resource Allocation](https://mlsanity.com/paper/2207.06940/) (2022)
- [Sequential Learning Of Neural Networks for Prequential MDL](https://mlsanity.com/paper/2210.07931/) (2022)
- [Generative Discovery of Partial Differential Equations by Learning from Math Handbooks](https://mlsanity.com/paper/2505.05869/) (2025)
- [Machine learning on small size samples: A synthetic knowledge synthesis](https://mlsanity.com/paper/2103.01002/) (2021)
- [Archangel: A Hybrid UAV-based Human Detection Benchmark with Position and Pose Metadata](https://mlsanity.com/paper/2209.00128/) (2022)
- [Keeping Deep Learning Models in Check: A History-Based Approach to Mitigate Overfitting](https://mlsanity.com/paper/2401.10359/) (2024)
- [The Effect of Model Size on Worst-Group Generalization](https://mlsanity.com/paper/2112.04094/) (2021)
- [ChimeraMix: Image Classification on Small Datasets via Masked Feature Mixing](https://mlsanity.com/paper/2202.11616/) (2022)
- [Image Classification with Small Datasets: Overview and Benchmark](https://mlsanity.com/paper/2212.12478/) (2022)
- [Prequential MDL for Causal Structure Learning with Neural Networks](https://mlsanity.com/paper/2107.05481/) (2021)
- [Evaluating Representations with Readout Model Switching](https://mlsanity.com/paper/2302.09579/) (2023)
- [No Data Augmentation? Alternative Regularizations for Effective Training on Small Datasets](https://mlsanity.com/paper/2309.01694/) (2023)
- [Semivalue-based data valuation is arbitrary and gameable](https://mlsanity.com/paper/2506.12619/) (2025)
- [A Simple and Fast Baseline for Tuning Large XGBoost Models](https://mlsanity.com/paper/2111.06924/) (2021)
- [On the Effectiveness of Neural Ensembles for Image Classification with Small Datasets](https://mlsanity.com/paper/2111.14493/) (2021)
- [Weakly Supervised Learning with Automated Labels from Radiology Reports for Glioma Change Detection](https://mlsanity.com/paper/2210.09698/) (2022)
- [Something for (almost) nothing: Improving deep ensemble calibration using unlabeled data](https://mlsanity.com/paper/2310.02885/) (2023)
- [You Only Train Once](https://mlsanity.com/paper/2506.04349/) (2025)
- [Small Data Explainer -- The impact of small data methods in everyday life](https://mlsanity.com/paper/2507.11773/) (2025)

## Abstract (arXiv)

Highly overparametrized neural networks can display curiously strong generalization performance - a phenomenon that has recently garnered a wealth of theoretical and empirical research in order to better understand it. In contrast to most previous work, which typically considers the performance as a function of the model size, in this paper we empirically study the generalization performance as the size of the training set varies over multiple orders of magnitude. These systematic experiments lead to some interesting and potentially very useful observations; perhaps most notably that training on smaller subsets of the data can lead to more reliable model selection decisions whilst simultaneously enjoying smaller computational costs. Our experiments furthermore allow us to estimate Minimum Description Lengths for common datasets given modern neural network architectures, thereby paving the way for principled model selection taking into account Occams-razor.