# Bootstrapping the Cross-Validation Estimate

arXiv: https://arxiv.org/abs/2307.00260 · HTML: https://mlsanity.com/paper/2307.00260/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 38 references that are in the map.
- First posted on arXiv 1 Jul 2023.

## Builds on (most cited first, 5 of 5)

- [Cross-validation: what does it estimate and how well does it do it?](https://mlsanity.com/paper/2104.00673/) (2021)
- [Cross-validation Confidence Intervals for Test Error](https://mlsanity.com/paper/2007.12671/) (2020)
- [Cross-Validation with Confidence](https://mlsanity.com/paper/1703.07904/) (2017)
- [Estimation and Validation of Ratio-based Conditional Average Treatment Effects Using Observational Data](https://mlsanity.com/paper/1912.06977/) (2019)
- [Estimating the standard error of cross-Validation-Based estimators of classifier performance](https://mlsanity.com/paper/1908.00325/) (2019)

## Built on by (most cited first, 1 of 1)

- [A Honest Cross-Validation Estimator for Prediction Performance](https://mlsanity.com/paper/2510.07649/) (2025)

## Abstract (arXiv)

Cross-validation is a widely used technique for evaluating the performance of prediction models, ranging from simple binary classification to complex precision medicine strategies. It helps correct for optimism bias in error estimates, which can be significant for models built using complex statistical learning algorithms. However, since the cross-validation estimate is a random value dependent on observed data, it is essential to accurately quantify the uncertainty associated with the estimate. This is especially important when comparing the performance of two models using cross-validation, as one must determine whether differences in estimated error are due to chance. Although various methods have been developed to make inferences on cross-validation estimates, they often have many limitations, such as requiring stringent model assumptions. This paper proposes a fast bootstrap method that quickly estimates the standard error of the cross-validation estimate and produces valid confidence intervals for a population parameter measuring average model performance. Our method overcomes the computational challenges inherent in bootstrapping a cross-validation estimate by estimating the variance component within a random-effects model. It is also as flexible as the cross-validation procedure itself. To showcase the effectiveness of our approach, we conducted comprehensive simulations and real-data analysis across two applications.