# Predictive Performance Test based on the Exhaustive Nested Cross-Validation for High-dimensional data

arXiv: https://arxiv.org/abs/2408.03138 · HTML: https://mlsanity.com/paper/2408.03138/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 114 references that are in the map.
- First posted on arXiv 6 Aug 2024.

## Builds on (most cited first, 18 of 18)

- [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [High-Dimensional Asymptotics of Prediction: Ridge Regression and Classification](https://mlsanity.com/paper/1507.03003/) (2015)
- [On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regression](https://mlsanity.com/paper/2006.05800/) (2020)
- [Cross-validation: what does it estimate and how well does it do it?](https://mlsanity.com/paper/2104.00673/) (2021)
- [Asymptotics of Ridge (less) Regression under General Source Condition](https://mlsanity.com/paper/2006.06386/) (2020)
- [Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularization](https://mlsanity.com/paper/1805.10939/) (2018)
- [Cross-validation Confidence Intervals for Test Error](https://mlsanity.com/paper/2007.12671/) (2020)
- [False Discoveries Occur Early on the Lasso Path](https://mlsanity.com/paper/1511.01957/) (2015)
- [A significance test for the lasso](https://mlsanity.com/paper/1301.7161/) (2013)
- [Approximate Leave-One-Out for Fast Parameter Tuning in High Dimensions](https://mlsanity.com/paper/1807.02694/) (2018)
- [Cross-Validation with Confidence](https://mlsanity.com/paper/1703.07904/) (2017)
- [From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation](https://mlsanity.com/paper/1704.08160/) (2017)
- [In Defense of the Indefensible: A Very Naive Approach to High-Dimensional Inference](https://mlsanity.com/paper/1705.05543/) (2017)
- [Bayesian leave-one-out cross-validation approximations for Gaussian latent variable models](https://mlsanity.com/paper/1412.7461/) (2014)
- [Confidence Intervals for the Generalisation Error of Random Forests](https://mlsanity.com/paper/2201.11210/) (2022)
- [Modified Cross-Validation for Penalized High-Dimensional Linear Regression Models](https://mlsanity.com/paper/1309.2068/) (2013)
- [Ridge-penalized adaptive Mantel test and its application in imaging genetics](https://mlsanity.com/paper/2103.02156/) (2021)

## Abstract (arXiv)

It is crucial to assess the predictive performance of a model to establish its practicality and relevance in real-world scenarios, particularly for high-dimensional data analysis. Among data splitting or resampling methods, cross-validation (CV) is extensively used for several tasks such as estimating the prediction error, tuning the regularization parameter, and selecting the most suitable predictive model among competing alternatives. The K-fold cross-validation is a popular CV method but its limitation is that the risk estimates are highly dependent on the partitioning of the data (for training and testing). Here, the issues regarding the reproducibility of the K-fold CV estimator are demonstrated in hypothesis testing wherein different partitions lead to notably disparate conclusions. This study presents a novel predictive performance test and valid confidence intervals based on exhaustive nested cross-validation for determining the difference in prediction error between two model-fitting algorithms. A naive implementation of the exhaustive nested cross-validation is computationally costly. Here, we address concerns regarding computational complexity by devising a computationally tractable closed-form expression for the proposed cross-validation estimator. Our study also investigates strategies aimed at enhancing statistical power within high-dimensional scenarios while controlling the Type I error rate. trThrough comprehensive numerical experiments, we demonstrate that Ridge-based methods using bias to measure uncertainty of CV estimates and adaptive hyperparameter selection provide the most reliable approach for high-dimensional predictive performance testing. To illustrate the practical utility of our method, we apply it to an RNA sequencing study and demonstrate its effectiveness in the context of biological data analysis.