# Bootstrapping the Out-of-sample Predictions for Efficient and Accurate Cross-Validation

arXiv: https://arxiv.org/abs/1708.07180 · HTML: https://mlsanity.com/paper/1708.07180/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Aug 2017.

## Built on by (most cited first, 12 of 12)

- [Pan-Cancer Integrative Histology-Genomic Analysis via Interpretable Multimodal Deep Learning](https://mlsanity.com/paper/2108.02278/) (2021)
- [Adversarial Causal Tuning for Realistic Time-series Generation](https://mlsanity.com/paper/2506.02084/) (2025)
- [Post-Selection Confidence Bounds for Prediction Performance](https://mlsanity.com/paper/2210.13206/) (2022)
- [Confidence Interval Estimation of Predictive Performance in the Context of AutoML](https://mlsanity.com/paper/2406.08099/) (2024)
- [The leap to ordinal: detailed functional prognosis after traumatic brain injury with a flexible modelling approach](https://mlsanity.com/paper/2202.04801/) (2022)
- [A Meta-Level Learning Algorithm for Sequential Hyper-Parameter Space Reduction in AutoML](https://mlsanity.com/paper/2312.06305/) (2023)
- [Reproducibility, energy efficiency and performance of pseudorandom number generators in machine learning: a comparative study of python, numpy, tensorflow, and pytorch implementations](https://mlsanity.com/paper/2401.17345/) (2024)
- [Towards Automated Causal Discovery: a case study on 5G telecommunication data](https://mlsanity.com/paper/2402.14481/) (2024)
- [Statistical Quality and Reproducibility of Pseudorandom Number Generators in Machine Learning technologies](https://mlsanity.com/paper/2507.03007/) (2025)
- [Revisiting Non-Specific Syndromic Surveillance](https://mlsanity.com/paper/2101.12246/) (2021)
- [Mining the contribution of intensive care clinical course to outcome after traumatic brain injury](https://mlsanity.com/paper/2303.04630/) (2023)
- [Which Leakage Types Matter? A Quantitative Landscape Across 2,047 Benchmark Datasets](https://mlsanity.com/paper/2604.04199/) (2026)

## Abstract (arXiv)

Cross-Validation (CV), and out-of-sample performance-estimation protocols in general, are often employed both for (a) selecting the optimal combination of algorithms and values of hyper-parameters (called a configuration) for producing the final predictive model, and (b) estimating the predictive performance of the final model. However, the cross-validated performance of the best configuration is optimistically biased. We present an efficient bootstrap method that corrects for the bias, called Bootstrap Bias Corrected CV (BBC-CV). BBC-CV's main idea is to bootstrap the whole process of selecting the best-performing configuration on the out-of-sample predictions of each configuration, without additional training of models. In comparison to the alternatives, namely the nested cross-validation and a method by Tibshirani and Tibshirani, BBC-CV is computationally more efficient, has smaller variance and bias, and is applicable to any metric of performance (accuracy, AUC, concordance index, mean squared error). Subsequently, we employ again the idea of bootstrapping the out-of-sample predictions to speed up the CV process. Specifically, using a bootstrap-based hypothesis test we stop training of models on new folds of statistically-significantly inferior configurations. We name the method Bootstrap Corrected with Early Dropping CV (BCED-CV) that is both efficient and provides accurate performance estimates.