# A Probabilistic Method to Predict Classifier Accuracy on Larger Datasets given Small Pilot Data

arXiv: https://arxiv.org/abs/2311.18025 · HTML: https://mlsanity.com/paper/2311.18025/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 28 references that are in the map.
- First posted on arXiv 29 Nov 2023.

## Builds on (most cited first, 11 of 11)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [A Constructive Prediction of the Generalization Error Across Scales](https://mlsanity.com/paper/1909.12673/) (2019)
- [How Much More Data Do I Need? Estimating Requirements for Downstream Tasks](https://mlsanity.com/paper/2207.01725/) (2022)
- [Learning Curves for Analysis of Deep Networks](https://mlsanity.com/paper/2010.11029/) (2020)
- [A New Semi-supervised Learning Benchmark for Classifying View and Diagnosing Aortic Stenosis from Echocardiograms](https://mlsanity.com/paper/2108.00080/) (2021)
- [Optimizing Data Collection for Machine Learning](https://mlsanity.com/paper/2210.01234/) (2022)
- [A Meta-Learning Approach to Predicting Performance and Data Requirements](https://mlsanity.com/paper/2303.01598/) (2023)

## Built on by (most cited first, 3 of 3)

- [Data Scaling Laws for End-to-End Autonomous Driving](https://mlsanity.com/paper/2504.04338/) (2025)
- [Synthetic Data Reveals Generalization Gaps in Correlated Multiple Instance Learning](https://mlsanity.com/paper/2510.25759/) (2025)
- [A Multi-Dataset Benchmark of Multiple Instance Learning for 3D Neuroimage Classification](https://mlsanity.com/paper/2604.26807/) (2026)

## Abstract (arXiv)

Practitioners building classifiers often start with a smaller pilot dataset and plan to grow to larger data in the near future. Such projects need a toolkit for extrapolating how much classifier accuracy may improve from a 2x, 10x, or 50x increase in data size. While existing work has focused on finding a single "best-fit" curve using various functional forms like power laws, we argue that modeling and assessing the uncertainty of predictions is critical yet has seen less attention. In this paper, we propose a Gaussian process model to obtain probabilistic extrapolations of accuracy or similar performance metrics as dataset size increases. We evaluate our approach in terms of error, likelihood, and coverage across six datasets. Though we focus on medical tasks and image modalities, our open source approach generalizes to any kind of classifier.