# Exploring the Impact of Dataset Statistical Effect Size on Model Performance and Data Sample Size Sufficiency

arXiv: https://arxiv.org/abs/2501.02673 · HTML: https://mlsanity.com/paper/2501.02673/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 16 references that are in the map.
- First posted on arXiv 5 Jan 2025.

## Builds on (most cited first, 4 of 4)

- [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](https://mlsanity.com/paper/2002.06305/) (2020)
- [Unravelling Small Sample Size Problems in the Deep Learning World](https://mlsanity.com/paper/2008.03522/) (2020)
- [Exploring Cross-model Neuronal Correlations in the Context of Predicting Model Performance and Generalizability](https://mlsanity.com/paper/2408.08448/) (2024)
- [Similarities and Differences between Machine Learning and Traditional Advanced Statistical Modeling in Healthcare Analytics](https://mlsanity.com/paper/2201.02469/) (2022)

## Abstract (arXiv)

Having a sufficient quantity of quality data is a critical enabler of training effective machine learning models. Being able to effectively determine the adequacy of a dataset prior to training and evaluating a model's performance would be an essential tool for anyone engaged in experimental design or data collection. However, despite the need for it, the ability to prospectively assess data sufficiency remains an elusive capability. We report here on two experiments undertaken in an attempt to better ascertain whether or not basic descriptive statistical measures can be indicative of how effective a dataset will be at training a resulting model. Leveraging the effect size of our features, this work first explores whether or not a correlation exists between effect size, and resulting model performance (theorizing that the magnitude of the distinction between classes could correlate to a classifier's resulting success). We then explore whether or not the magnitude of the effect size will impact the rate of convergence of the learning curve, (theorizing again that a greater effect size may indicate that the model will converge more rapidly, and with a smaller sample size needed). Our results appear to indicate that this is not an effective heuristic for determining adequate sample size or projecting model performance, and therefore that additional work is still needed to better prospectively assess adequacy of data.