# Cross-validation Confidence Intervals for Test Error

arXiv: https://arxiv.org/abs/2007.12671 · HTML: https://mlsanity.com/paper/2007.12671/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jul 2020.

## Built on by (most cited first, 30 of 33)

- [Testing for Outliers with Conformal p-values](https://mlsanity.com/paper/2104.08279/) (2021)
- [Cross-validation: what does it estimate and how well does it do it?](https://mlsanity.com/paper/2104.00673/) (2021)
- [Cross-Prediction-Powered Inference](https://mlsanity.com/paper/2309.16598/) (2023)
- [Automatic Termination for Hyperparameter Optimization](https://mlsanity.com/paper/2104.08166/) (2021)
- [Debiased Machine Learning without Sample-Splitting for Stable Estimators](https://mlsanity.com/paper/2206.01825/) (2022)
- [Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimization](https://mlsanity.com/paper/2405.15393/) (2024)
- [Overtuning in Hyperparameter Optimization](https://mlsanity.com/paper/2506.19540/) (2025)
- [Imputation-Powered Inference](https://mlsanity.com/paper/2509.13778/) (2025)
- [Meta-Analysis of Randomized Experiments with Applications to Heavy-Tailed Response Data](https://mlsanity.com/paper/2112.07602/) (2021)
- [Extrapolated cross-validation for randomized ensembles](https://mlsanity.com/paper/2302.13511/) (2023)
- [Hypothesis Transfer Learning with Surrogate Classification Losses: Generalization Bounds through Algorithmic Stability](https://mlsanity.com/paper/2305.19694/) (2023)
- [Failures and Successes of Cross-Validation for Early-Stopped Gradient Descent](https://mlsanity.com/paper/2402.16793/) (2024)
- [Approximate Leave-one-out Cross Validation for Regression with ℓ1 Regularizers (extended version)](https://mlsanity.com/paper/2310.17629/) (2023)
- [The Limits of Assumption-free Tests for Algorithm Performance](https://mlsanity.com/paper/2402.07388/) (2024)
- [Quantifying Data Similarity Using Cross Learning](https://mlsanity.com/paper/2510.10866/) (2025)
- [Pushing the Accuracy-Group Robustness Frontier with Introspective Self-play](https://mlsanity.com/paper/2302.05807/) (2023)
- [Confidence intervals for the random forest generalization error](https://mlsanity.com/paper/2112.06101/) (2021)
- [Bootstrapping the Cross-Validation Estimate](https://mlsanity.com/paper/2307.00260/) (2023)
- [Is Algorithmic Stability Testable? A Unified Framework under Computational Constraints](https://mlsanity.com/paper/2405.15107/) (2024)
- [The Relative Instability of Model Comparison with Cross-validation](https://mlsanity.com/paper/2508.04409/) (2025)
- [Confidence intervals for the Cox model test error from cross-validation](https://mlsanity.com/paper/2201.10770/) (2022)
- [Predictive Performance Test based on the Exhaustive Nested Cross-Validation for High-dimensional data](https://mlsanity.com/paper/2408.03138/) (2024)
- [SOAK: Same/Other/All K-fold cross-validation for estimating similarity of patterns in data subsets](https://mlsanity.com/paper/2410.08643/) (2024)
- [Stability Regularized Cross-Validation](https://mlsanity.com/paper/2505.06927/) (2025)
- [Evaluation and Optimization of Leave-one-out Cross-validation for the Lasso](https://mlsanity.com/paper/2508.14368/) (2025)
- [Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)](https://mlsanity.com/paper/2508.15141/) (2025)
- [A Honest Cross-Validation Estimator for Prediction Performance](https://mlsanity.com/paper/2510.07649/) (2025)
- [PAIR-CI: Calibrated Conditional Independence Testing for Causal Discovery with Incomplete Data](https://mlsanity.com/paper/2605.04838/) (2026)
- [Minimax Limits of k-Fold Cross-Validation via Majority](https://mlsanity.com/paper/2605.25859/) (2026)
- [Multipath Adaptive Gated Bottleneck Latent ODE with Raman Data Fusion for Cell Culture Process Forecasting](https://mlsanity.com/paper/2606.26520/) (2026)

## Abstract (arXiv)

This work develops central limit theorems for cross-validation and consistent estimators of its asymptotic variance under weak stability conditions on the learning algorithm. Together, these results provide practical, asymptotically-exact confidence intervals for k-fold test error and valid, powerful hypothesis tests of whether one learning algorithm has smaller k-fold test error than another. These results are also the first of their kind for the popular choice of leave-one-out cross-validation. In our real-data experiments with diverse learning algorithms, the resulting intervals and tests outperform the most popular alternative methods from the literature.