# Nested cross-validation when selecting classifiers is overzealous for most practical applications

arXiv: https://arxiv.org/abs/1809.09446 · HTML: https://mlsanity.com/paper/1809.09446/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Sep 2018.

## Built on by (most cited first, 19 of 19)

- [A Bayesian Bradley-Terry model to compare multiple ML algorithms on multiple data sets](https://mlsanity.com/paper/2208.04935/) (2022)
- [An LLM Feature-based Framework for Dialogue Constructiveness Assessment](https://mlsanity.com/paper/2406.14760/) (2024)
- [Pseudo-online framework for BCI evaluation: A MOABB perspective](https://mlsanity.com/paper/2308.11656/) (2023)
- [Video-based automatic lameness detection of dairy cows using pose estimation and multiple locomotion traits](https://mlsanity.com/paper/2401.05202/) (2024)
- [Unveiling Processing--Property Relationships in Laser Powder Bed Fusion: The Synergy of Machine Learning and High-throughput Experiments](https://mlsanity.com/paper/2409.00248/) (2024)
- [Utilizing Semantic Textual Similarity for Clinical Survey Data Feature Selection](https://mlsanity.com/paper/2308.09892/) (2023)
- [An Introduction to Natural Language Processing Techniques and Framework for Clinical Implementation in Radiation Oncology](https://mlsanity.com/paper/2311.02205/) (2023)
- [Predicting Parkinson's disease trajectory using clinical and functional MRI features: a reproduction and replication study](https://mlsanity.com/paper/2403.15405/) (2024)
- [Classifying Simulated Gait Impairments using Privacy-preserving Explainable Artificial Intelligence and Mobile Phone Videos](https://mlsanity.com/paper/2412.01056/) (2024)
- [Automatic Classification of OSA related Snoring Signals from Nocturnal Audio Recordings](https://mlsanity.com/paper/2102.12829/) (2021)
- [Extract Dynamic Information To Improve Time Series Modeling: a Case Study with Scientific Workflow](https://mlsanity.com/paper/2205.09703/) (2022)
- [Assessing the impact of emergency department short stay units using length-of-stay prediction and discrete event simulation](https://mlsanity.com/paper/2308.02730/) (2023)
- [Generating Hierarchical Structures for Improved Time Series Classification Using Stochastic Splitting Functions](https://mlsanity.com/paper/2309.11963/) (2023)
- [Lameness detection in dairy cows using pose estimation and bidirectional LSTMs](https://mlsanity.com/paper/2508.10643/) (2025)
- [Intervention Efficiency and Perturbation Validation Framework: Capacity-Aware and Robust Clinical Model Selection under the Rashomon Effect](https://mlsanity.com/paper/2511.14317/) (2025)
- [Deep Learning-Based Meat Freshness Detection with Segmentation and OOD-Aware Classification](https://mlsanity.com/paper/2603.00368/) (2026)
- [An explainable hierarchical self attention-based approach for tremor detection in the time domain](https://mlsanity.com/paper/2606.00461/) (2026)
- [AutoMatBench: An Automatic Optimization Toolkit for the Acceleration of Material Properties Prediction Benchmarking](https://mlsanity.com/paper/2607.11526/) (2026)
- [PPG-Bench: A Unified Benchmark for PPG Foundation Models](https://mlsanity.com/paper/2610.13435/) (2026)

## Abstract (arXiv)

When selecting a classification algorithm to be applied to a particular problem, one has to simultaneously select the best algorithm for that dataset and the best set of hyperparameters for the chosen model. The usual approach is to apply a nested cross-validation procedure; hyperparameter selection is performed in the inner cross-validation, while the outer cross-validation computes an unbiased estimate of the expected accuracy of the algorithm with cross-validation based hyperparameter tuning. The alternative approach, which we shall call `flat cross-validation', uses a single cross-validation step both to select the optimal hyperparameter values and to provide an estimate of the expected accuracy of the algorithm, that while biased may nevertheless still be used to select the best learning algorithm. We tested both procedures using 12 different algorithms on 115 real life binary datasets and conclude that using the less computationally expensive flat cross-validation procedure will generally result in the selection of an algorithm that is, for all practical purposes, of similar quality to that selected via nested cross-validation, provided the learning algorithms have relatively few hyperparameters to be optimised.