# Cross-Validation with Confidence

arXiv: https://arxiv.org/abs/1703.07904 · HTML: https://mlsanity.com/paper/1703.07904/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Mar 2017.

## Built on by (most cited first, 14 of 14)

- [Cross-validation: what does it estimate and how well does it do it?](https://mlsanity.com/paper/2104.00673/) (2021)
- [Bayesian subset selection and variable importance for interpretable prediction and classification](https://mlsanity.com/paper/2104.10150/) (2021)
- [Network Estimation by Mixing: Adaptivity and More](https://mlsanity.com/paper/2106.02803/) (2021)
- [Meta-Analysis of Randomized Experiments with Applications to Heavy-Tailed Response Data](https://mlsanity.com/paper/2112.07602/) (2021)
- [Extrapolated cross-validation for randomized ensembles](https://mlsanity.com/paper/2302.13511/) (2023)
- [Failures and Successes of Cross-Validation for Early-Stopped Gradient Descent](https://mlsanity.com/paper/2402.16793/) (2024)
- [Bootstrapping the Cross-Validation Estimate](https://mlsanity.com/paper/2307.00260/) (2023)
- [Online Estimation with Rolling Validation: Adaptive Nonparametric Estimation with Streaming Data](https://mlsanity.com/paper/2310.12140/) (2023)
- [The Relative Instability of Model Comparison with Cross-validation](https://mlsanity.com/paper/2508.04409/) (2025)
- [Throwing Vines at the Wall: Structure Learning via Random Search](https://mlsanity.com/paper/2510.20035/) (2025)
- [Targeted Cross-Validation](https://mlsanity.com/paper/2109.06949/) (2021)
- [Inconsistency of cross-validation for structure learning in Gaussian graphical models](https://mlsanity.com/paper/2312.17047/) (2023)
- [Predictive Performance Test based on the Exhaustive Nested Cross-Validation for High-dimensional data](https://mlsanity.com/paper/2408.03138/) (2024)
- [Cross-Fitting Under Nonregularity: Normality and Inference via Locality](https://mlsanity.com/paper/2610.02944/) (2026)

## Abstract (arXiv)

Cross-validation is one of the most popular model selection methods in statistics and machine learning. Despite its wide applicability, traditional cross validation methods tend to select overfitting models, due to the ignorance of the uncertainty in the testing sample. We develop a new, statistically principled inference tool based on cross-validation that takes into account the uncertainty in the testing sample. This new method outputs a set of highly competitive candidate models containing the best one with guaranteed probability. As a consequence, our method can achieve consistent variable selection in a classical linear regression setting, for which existing cross-validation methods require unconventional split ratios. When used for regularizing tuning parameter selection, the method can provide a further trade-off between prediction accuracy and model interpretability. We demonstrate the performance of the proposed method in several simulated and real data examples.