mlsanityData to 14 Oct 2026

Cross-validation Confidence Intervals for Test Error

arXiv 2007.12671 ↗First posted 24 Jul 2020stat.ML cs.LG math.STcited by 33

Not on the map. Why?

Abstract (arXiv)

This work develops central limit theorems for cross-validation and consistent estimators of its asymptotic variance under weak stability conditions on the learning algorithm. Together, these results provide practical, asymptotically-exact confidence intervals for k-fold test error and valid, powerful hypothesis tests of whether one learning algorithm has smaller k-fold test error than another. These results are also the first of their kind for the popular choice of leave-one-out cross-validation. In our real-data experiments with diverse learning algorithms, the resulting intervals and tests outperform the most popular alternative methods from the literature.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 2; 2023: 6; 2024: 6; 2025: 8; 2026: 6.

33 papers.

Sort:

  1. 2021Testing for Outliers with Conformal p-valuescited by 134
  2. 2021Cross-validation: what does it estimate and how well does it do it?cited by 65
  3. 2023Cross-Prediction-Powered Inferencecited by 58
  4. 2021Automatic Termination for Hyperparameter Optimizationcited by 21
  5. 2022Debiased Machine Learning without Sample-Splitting for Stable Estimatorscited by 13
  6. 2024Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimizationcited by 7
  7. 2025Overtuning in Hyperparameter Optimizationcited by 5
  8. 2025Imputation-Powered Inferencecited by 5
  9. 2021Meta-Analysis of Randomized Experiments with Applications to Heavy-Tailed Response Datacited by 4
  10. 2023Extrapolated cross-validation for randomized ensemblescited by 4
  11. 2023Hypothesis Transfer Learning with Surrogate Classification Losses: Generalization Bounds through Algorithmic Stabilitycited by 4
  12. 2024Failures and Successes of Cross-Validation for Early-Stopped Gradient Descentcited by 4
21 more
  1. 2023Approximate Leave-one-out Cross Validation for Regression with ℓ1 Regularizers (extended version)cited by 3
  2. 2024The Limits of Assumption-free Tests for Algorithm Performancecited by 3
  3. 2025Quantifying Data Similarity Using Cross Learningcited by 3
  4. 2023Pushing the Accuracy-Group Robustness Frontier with Introspective Self-playcited by 2
  5. 2021Confidence intervals for the random forest generalization errorcited by 1
  6. 2023Bootstrapping the Cross-Validation Estimatecited by 1
  7. 2024Is Algorithmic Stability Testable? A Unified Framework under Computational Constraintscited by 1
  8. 2025The Relative Instability of Model Comparison with Cross-validationcited by 1
  9. 2022Confidence intervals for the Cox model test error from cross-validationcited by 0
  10. 2024Predictive Performance Test based on the Exhaustive Nested Cross-Validation for High-dimensional datacited by 0
  11. 2024SOAK: Same/Other/All K-fold cross-validation for estimating similarity of patterns in data subsetscited by 0
  12. 2025Stability Regularized Cross-Validationcited by 0
  13. 2025Evaluation and Optimization of Leave-one-out Cross-validation for the Lassocited by 0
  14. 2025Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)cited by 0
  15. 2025A Honest Cross-Validation Estimator for Prediction Performancecited by 0
  16. 2026PAIR-CI: Calibrated Conditional Independence Testing for Causal Discovery with Incomplete Datacited by 0
  17. 2026Minimax Limits of k-Fold Cross-Validation via Majoritycited by 0
  18. 2026Multipath Adaptive Gated Bottleneck Latent ODE with Raman Data Fusion for Cell Culture Process Forecastingcited by 0
  19. 2026Cross-Audit Projection for Model Risk Predictioncited by 0
  20. 2026Algorithmic stability via ensemblingcited by 0
  21. 2026Cross-Fitting Under Nonregularity: Normality and Inference via Localitycited by 0