# Is K-fold cross validation the best model selection method for Machine Learning?

arXiv: https://arxiv.org/abs/2401.16407 · HTML: https://mlsanity.com/paper/2401.16407/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 63 references that are in the map.
- First posted on arXiv 29 Jan 2024.

## Builds on (most cited first, 10 of 10)

- [Model-Agnostic Interpretability of Machine Learning](https://mlsanity.com/paper/1606.05386/) (2016)
- [Cross-validation: what does it estimate and how well does it do it?](https://mlsanity.com/paper/2104.00673/) (2021)
- [A PAC-Bayesian Tutorial with A Dropout Bound](https://mlsanity.com/paper/1307.2118/) (2013)
- [Cross-validation failure: small sample sizes lead to large error bars](https://mlsanity.com/paper/1706.07581/) (2017)
- [Can parametric statistical methods be trusted for fMRI based group studies?](https://mlsanity.com/paper/1511.01863/) (2015)
- [Better-Than-Chance Classification for Signal Detection](https://mlsanity.com/paper/1608.08873/) (2016)
- [Ensemble Deep Learning on Large, Mixed-Site fMRI Datasets in Autism and Other Tasks](https://mlsanity.com/paper/2002.07874/) (2020)
- [Statistical Agnostic Mapping: a Framework in Neuroimaging based on Concentration Inequalities](https://mlsanity.com/paper/1912.12274/) (2019)
- [The Same Analysis Approach: Practical protection against the pitfalls of novel neuroimaging analysis methods](https://mlsanity.com/paper/1703.06670/) (2017)
- [A hypothesis-driven method based on machine learning for neuroimaging data analysis](https://mlsanity.com/paper/2202.04397/) (2022)

## Built on by (most cited first, 10 of 10)

- [Perturbing the Derivative: Doubly Wild Refitting for Model-Free Evaluation of Opaque Machine Learning Predictors](https://mlsanity.com/paper/2511.18789/) (2025)
- [Statistical Agnostic Regression: a machine learning method to validate regression models](https://mlsanity.com/paper/2402.15213/) (2024)
- [Uncertainty-Driven Modeling of Microporosity and Permeability in Clastic Reservoirs Using Random Forest](https://mlsanity.com/paper/2503.16957/) (2025)
- [The use of cross validation in the analysis of designed experiments](https://mlsanity.com/paper/2506.14593/) (2025)
- [Harnessing Near-Infrared Spectroscopy and Machine Learning for Traceable Classification of Hanwoo and Holstein Beef](https://mlsanity.com/paper/2507.02903/) (2025)
- [Jet Image Tagging Using Deep Learning: An Ensemble Model](https://mlsanity.com/paper/2508.10034/) (2025)
- [Deep Learning Framework for Early Detection of Pancreatic Cancer Using Multi-Modal Medical Imaging Analysis](https://mlsanity.com/paper/2508.20877/) (2025)
- [Perturbing the Derivative: Wild Refitting for Model-Free Evaluation of Machine Learning Models under Bregman Losses](https://mlsanity.com/paper/2509.02476/) (2025)
- [A machine learning framework integrating seed traits and plasma parameters for predicting germination uplift in crops](https://mlsanity.com/paper/2510.23657/) (2025)
- [Enhancing Language Models for Robust Greenwashing Detection](https://mlsanity.com/paper/2601.21722/) (2026)

## Abstract (arXiv)

As a technique that can compactly represent complex patterns, machine learning has significant potential for predictive inference. K-fold cross-validation (CV) is the most common approach to ascertaining the likelihood that a machine learning outcome is generated by chance, and it frequently outperforms conventional hypothesis testing. This improvement uses measures directly obtained from machine learning classifications, such as accuracy, that do not have a parametric description. To approach a frequentist analysis within machine learning pipelines, a permutation test or simple statistics from data partitions (i.e., folds) can be added to estimate confidence intervals. Unfortunately, neither parametric nor non-parametric tests solve the inherent problems of partitioning small sample-size datasets and learning from heterogeneous data sources. The fact that machine learning strongly depends on the learning parameters and the distribution of data across folds recapitulates familiar difficulties around excess false positives and replication. A novel statistical test based on K-fold CV and the Upper Bound of the actual risk (K-fold CUBV) is proposed, where uncertain predictions of machine learning with CV are bounded by the worst case through the evaluation of concentration inequalities. Probably Approximately Correct-Bayesian upper bounds for linear classifiers in combination with K-fold CV are derived and used to estimate the actual risk. The performance with simulated and neuroimaging datasets suggests that K-fold CUBV is a robust criterion for detecting effects and validating accuracy values obtained from machine learning and classical CV schemes, while avoiding excess false positives.