# Cross-validation: what does it estimate and how well does it do it?

arXiv: https://arxiv.org/abs/2104.00673 · HTML: https://mlsanity.com/paper/2104.00673/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 66 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 11 of 11)

- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [On the marginal likelihood and cross-validation](https://mlsanity.com/paper/1905.08737/) (2019)
- [Cross-validation Confidence Intervals for Test Error](https://mlsanity.com/paper/2007.12671/) (2020)
- [Floodgate: inference for model-free variable importance](https://mlsanity.com/paper/2007.01283/) (2020)
- [Ridge Regression: Structure, Cross-Validation, and Sketching](https://mlsanity.com/paper/1910.02373/) (2019)
- [Cross-Validation with Confidence](https://mlsanity.com/paper/1703.07904/) (2017)
- [From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation](https://mlsanity.com/paper/1704.08160/) (2017)
- [Cross-Validation for Correlated Data](https://mlsanity.com/paper/1904.02438/) (2019)
- [Stability revisited: new generalisation bounds for the Leave-one-Out](https://mlsanity.com/paper/1608.06412/) (2016)
- [Estimating the standard error of cross-Validation-Based estimators of classifier performance](https://mlsanity.com/paper/1908.00325/) (2019)
- [Uncertainty in Bayesian Leave-One-Out Cross-Validation Based Model Comparison](https://mlsanity.com/paper/2008.10296/) (2020)

## Built on by (most cited first, 30 of 65)

- [More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalize](https://mlsanity.com/paper/2203.06176/) (2022)
- [Cross-Prediction-Powered Inference](https://mlsanity.com/paper/2309.16598/) (2023)
- [Conformal Prediction: A Data Perspective](https://mlsanity.com/paper/2410.06494/) (2024)
- [Relating the Partial Dependence Plot and Permutation Feature Importance to the Data Generating Process](https://mlsanity.com/paper/2109.01433/) (2021)
- [Position: Why We Must Rethink Empirical Research in Machine Learning](https://mlsanity.com/paper/2405.02200/) (2024)
- [Model Comparison and Calibration Assessment: User Guide for Consistent Scoring Functions in Machine Learning and Actuarial Practice](https://mlsanity.com/paper/2202.12780/) (2022)
- [Is K-fold cross validation the best model selection method for Machine Learning?](https://mlsanity.com/paper/2401.16407/) (2024)
- [Towards Explainable Artificial Intelligence (XAI): A Data Mining Perspective](https://mlsanity.com/paper/2401.04374/) (2024)
- [What Do Machine Learning Researchers Mean by "Reproducible"?](https://mlsanity.com/paper/2412.03854/) (2024)
- [Risk and cross validation in ridge regression with correlated samples](https://mlsanity.com/paper/2408.04607/) (2024)
- [S4Sleep: Elucidating the design space of deep-learning-based sleep stage classification models](https://mlsanity.com/paper/2310.06715/) (2023)
- [On Leakage in Machine Learning Pipelines](https://mlsanity.com/paper/2311.04179/) (2023)
- [Improving the Validity and Practical Usefulness of AI/ML Evaluations Using an Estimands Framework](https://mlsanity.com/paper/2406.10366/) (2024)
- [Overtuning in Hyperparameter Optimization](https://mlsanity.com/paper/2506.19540/) (2025)
- [Wild refitting for black box prediction](https://mlsanity.com/paper/2506.21460/) (2025)
- [Extrapolated cross-validation for randomized ensembles](https://mlsanity.com/paper/2302.13511/) (2023)
- [Concentration inequalities for leave-one-out cross validation](https://mlsanity.com/paper/2211.02478/) (2022)
- [The Limits of Assumption-free Tests for Algorithm Performance](https://mlsanity.com/paper/2402.07388/) (2024)
- [Post-Selection Confidence Bounds for Prediction Performance](https://mlsanity.com/paper/2210.13206/) (2022)
- [Training of Neural Networks with Uncertain Data: A Mixture of Experts Approach](https://mlsanity.com/paper/2312.08083/) (2023)
- [Two-Stage Surrogate Modeling for Data-Driven Design Optimization with Application to Composite Microstructure Generation](https://mlsanity.com/paper/2401.02008/) (2024)
- [Empirical investigation of multi-source cross-validation in clinical ECG classification](https://mlsanity.com/paper/2403.15012/) (2024)
- [Learning Counterfactual Distributions via Kernel Nearest Neighbors](https://mlsanity.com/paper/2410.13381/) (2024)
- [GeoConformal prediction: a model-agnostic framework of measuring the uncertainty of spatial prediction](https://mlsanity.com/paper/2412.08661/) (2024)
- [Predicting Loss Risks for B2B Tendering Processes](https://mlsanity.com/paper/2109.06815/) (2021)
- [GRASP: A Goodness-of-Fit Test for Classification Learning](https://mlsanity.com/paper/2209.02064/) (2022)
- [The out-of-sample R2: estimation and inference](https://mlsanity.com/paper/2302.05131/) (2023)
- [Deep Predictive Coding with Bi-directional Propagation for Classification and Reconstruction](https://mlsanity.com/paper/2305.18472/) (2023)
- [Bootstrapping the Cross-Validation Estimate](https://mlsanity.com/paper/2307.00260/) (2023)
- [Online Estimation with Rolling Validation: Adaptive Nonparametric Estimation with Streaming Data](https://mlsanity.com/paper/2310.12140/) (2023)

## Abstract (arXiv)

Cross-validation is a widely-used technique to estimate prediction error, but its behavior is complex and not fully understood. Ideally, one would like to think that cross-validation estimates the prediction error for the model at hand, fit to the training data. We prove that this is not the case for the linear model fit by ordinary least squares; rather it estimates the average prediction error of models fit on other unseen training sets drawn from the same population. We further show that this phenomenon occurs for most popular estimates of prediction error, including data splitting, bootstrapping, and Mallow's Cp. Next, the standard confidence intervals for prediction error derived from cross-validation may have coverage far below the desired level. Because each data point is used for both training and testing, there are correlations among the measured accuracies for each fold, and so the usual estimate of variance is too small. We introduce a nested cross-validation scheme to estimate this variance more accurately, and we show empirically that this modification leads to intervals with approximately correct coverage in many examples where traditional cross-validation intervals fail.