# Cross-validation failure: small sample sizes lead to large error bars

arXiv: https://arxiv.org/abs/1706.07581 · HTML: https://mlsanity.com/paper/1706.07581/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jun 2017.

## Built on by (most cited first, 30 of 43)

- [How I failed machine learning in medical imaging -- shortcomings and recommendations](https://mlsanity.com/paper/2103.10292/) (2021)
- [Generalization Through The Lens Of Leave-One-Out Error](https://mlsanity.com/paper/2203.03443/) (2022)
- [Is K-fold cross validation the best model selection method for Machine Learning?](https://mlsanity.com/paper/2401.16407/) (2024)
- [False Promises in Medical Imaging AI? Assessing Validity of Outperformance Claims](https://mlsanity.com/paper/2505.04720/) (2025)
- [Reproducibility in machine learning for medical imaging](https://mlsanity.com/paper/2209.05097/) (2022)
- [How precise are performance estimates for typical medical image segmentation tasks?](https://mlsanity.com/paper/2210.14677/) (2022)
- [Deep Learning in current Neuroimaging: a multivariate approach with power and type I error control but arguable generalization ability](https://mlsanity.com/paper/2103.16685/) (2021)
- [Then and Now: Quantifying the Longitudinal Validity of Self-Disclosed Depression Diagnoses](https://mlsanity.com/paper/2206.11155/) (2022)
- [Promises and pitfalls of deep neural networks in neuroimaging-based psychiatric research](https://mlsanity.com/paper/2301.08525/) (2023)
- [Learning Against Distributional Uncertainty: On the Trade-off Between Robustness and Specificity](https://mlsanity.com/paper/2301.13565/) (2023)
- [Small Effect Sizes in Malware Detection? Make Harder Train/Test Splits!](https://mlsanity.com/paper/2312.15813/) (2023)
- [Learn2Synth: Learning Optimal Data Synthesis Using Hypergradients for Brain Image Segmentation](https://mlsanity.com/paper/2411.16719/) (2024)
- [Accelerated functional brain aging in major depressive disorder: evidence from a large scale fMRI analysis of Chinese participants](https://mlsanity.com/paper/2205.04871/) (2022)
- [A hypothesis-driven method based on machine learning for neuroimaging data analysis](https://mlsanity.com/paper/2202.04397/) (2022)
- [Efficient Learning of Minimax Risk Classifiers in High Dimensions](https://mlsanity.com/paper/2306.06649/) (2023)
- [Moving Beyond Functional Connectivity: Time-Series Modeling for fMRI-Based Brain Disorder Classification](https://mlsanity.com/paper/2602.08262/) (2026)
- [Muddling Labels for Regularization, a novel approach to generalization](https://mlsanity.com/paper/2102.08769/) (2021)
- [Data Augmentation Through Monte Carlo Arithmetic Leads to More Generalizable Classification in Connectomics](https://mlsanity.com/paper/2109.09649/) (2021)
- [Confidence Intervals for Performance Estimates in Brain MRI Segmentation](https://mlsanity.com/paper/2307.10926/) (2023)
- [Introducing 3DCNN ResNets for ASD full-body kinematic assessment: a comparison with hand-crafted features](https://mlsanity.com/paper/2311.14533/) (2023)
- [A Multi-Probe Audit of Clinical-Interview Depression Detection Benchmarks](https://mlsanity.com/paper/2605.23977/) (2026)
- [Building Robust Machine Learning Models for Small Chemical Science Data: The Case of Shear Viscosity](https://mlsanity.com/paper/2208.10784/) (2022)
- [Statistical Agnostic Regression: a machine learning method to validate regression models](https://mlsanity.com/paper/2402.15213/) (2024)
- [A Machine Learning Approach for Identifying Anatomical Biomarkers of Early Mild Cognitive Impairment](https://mlsanity.com/paper/2407.00040/) (2024)
- [Lifespan tree of brain anatomy: diagnostic values for motor and cognitive neurodegenerative diseases](https://mlsanity.com/paper/2502.09682/) (2025)
- [Intervention Efficiency and Perturbation Validation Framework: Capacity-Aware and Robust Clinical Model Selection under the Rashomon Effect](https://mlsanity.com/paper/2511.14317/) (2025)
- [Assessing Low Back Movement with Motion Tape Sensor Data Through Deep Learning](https://mlsanity.com/paper/2602.11465/) (2026)
- [Which Leakage Types Matter? A Quantitative Landscape Across 2,047 Benchmark Datasets](https://mlsanity.com/paper/2604.04199/) (2026)
- [Leakage-Audited Benchmarking Reveals Limited Evidence for Cross-Subject Auditory-Evoked EEG Vowel Perception Decoding](https://mlsanity.com/paper/2605.00865/) (2026)
- [Rhamba: Region-Aware Hybrid Attention-Mamba Framework for Self-Supervised Learning in Resting-State fMRI](https://mlsanity.com/paper/2605.01240/) (2026)

## Abstract (arXiv)

Predictive models ground many state-of-the-art developments in statistical brain image analysis: decoding, MVPA, searchlight, or extraction of biomarkers. The principled approach to establish their validity and usefulness is cross-validation, testing prediction on unseen data. Here, I would like to raise awareness on error bars of cross-validation, which are often underestimated. Simple experiments show that sample sizes of many neuroimaging studies inherently lead to large error bars, eg ±10% for 100 samples. The standard error across folds strongly underestimates them. These large error bars compromise the reliability of conclusions drawn with predictive models, such as biomarkers or methods developments where, unlike with cognitive neuroimaging MVPA approaches, more samples cannot be acquired by repeating the experiment across many subjects. Solutions to increase sample size must be investigated, tackling possible increases in heterogeneity of the data.