# ROTI-GCV: Generalized Cross-Validation for right-ROTationally Invariant Data

arXiv: https://arxiv.org/abs/2406.11666 · HTML: https://mlsanity.com/paper/2406.11666/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 85 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Builds on (most cited first, 24 of 24)

- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [High-Dimensional Asymptotics of Prediction: Ridge Regression and Classification](https://mlsanity.com/paper/1507.03003/) (2015)
- [On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regression](https://mlsanity.com/paper/2006.05800/) (2020)
- [A modern maximum-likelihood theory for high-dimensional logistic regression](https://mlsanity.com/paper/1803.06964/) (2018)
- [Divide and Conquer Kernel Ridge Regression: A Distributed Algorithm with Minimax Optimal Rates](https://mlsanity.com/paper/1305.5029/) (2013)
- [Scaling and renormalization in high-dimensional regression](https://mlsanity.com/paper/2405.00592/) (2024)
- [Products of Many Large Random Matrices and Gradients in Deep Neural Networks](https://mlsanity.com/paper/1812.05994/) (2018)
- [A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-ℓ1-Norm Interpolated Classifiers](https://mlsanity.com/paper/2002.01586/) (2020)
- [Approximate Message Passing algorithms for rotationally invariant matrices](https://mlsanity.com/paper/2008.11892/) (2020)
- [On Robustness of Principal Component Regression](https://mlsanity.com/paper/1902.10920/) (2019)
- [Asymptotic Errors for Teacher-Student Convex Generalized Linear Models (or : How to Prove Kabashima's Replica Formula)](https://mlsanity.com/paper/2006.06581/) (2020)
- [Estimation in Rotationally Invariant Generalized Linear Models via Approximate Message Passing](https://mlsanity.com/paper/2112.04330/) (2021)
- [The Likelihood Ratio Test in High-Dimensional Logistic Regression Is Asymptotically a Rescaled Chi-Square](https://mlsanity.com/paper/1706.01191/) (2017)
- [Consistent Risk Estimation in Moderately High-Dimensional Linear Regression](https://mlsanity.com/paper/1902.01753/) (2019)
- [A scalable estimate of the extra-sample prediction error via approximate leave-one-out](https://mlsanity.com/paper/1801.10243/) (2018)
- [Distributed Generalized Cross-Validation for Divide-and-Conquer Kernel Ridge Regression and its Asymptotic Optimality](https://mlsanity.com/paper/1612.05907/) (2016)
- [Memory AMP](https://mlsanity.com/paper/2012.10861/) (2020)
- [Approximate Leave-One-Out for High-Dimensional Non-Differentiable Learning Problems](https://mlsanity.com/paper/1810.02716/) (2018)
- [Projected principal component analysis in factor models](https://mlsanity.com/paper/1406.3836/) (2014)
- [High-dimensional analysis of ridge regression for non-identically distributed data with a variance profile](https://mlsanity.com/paper/2403.20200/) (2024)
- [Canonical thresholding for non-sparse high-dimensional linear regression](https://mlsanity.com/paper/2007.12313/) (2020)
- [Approximate Message Passing for Multi-Layer Estimation in Rotationally Invariant Models](https://mlsanity.com/paper/2212.01572/) (2022)
- [Large-dimensional Factor Analysis without Moment Constraints](https://mlsanity.com/paper/1908.04957/) (2019)
- [Envelopes and principal component regression](https://mlsanity.com/paper/2207.05574/) (2022)

## Built on by (most cited first, 4 of 4)

- [Risk and cross validation in ridge regression with correlated samples](https://mlsanity.com/paper/2408.04607/) (2024)
- [Asymptotics of Linear Regression with Linearly Dependent Data](https://mlsanity.com/paper/2412.03702/) (2024)
- [Optimal and Provable Calibration in High-Dimensional Binary Classification: Angular Calibration and Platt Scaling](https://mlsanity.com/paper/2502.15131/) (2025)
- [Generalisation and benign over-fitting for linear regression onto random functional covariates](https://mlsanity.com/paper/2508.13895/) (2025)

## Abstract (arXiv)

Two key tasks in high-dimensional regularized regression are tuning the regularization strength for accurate predictions and estimating the out-of-sample risk. It is known that the standard approach -- k-fold cross-validation -- is inconsistent in modern high-dimensional settings. While leave-one-out and generalized cross-validation remain consistent in some high-dimensional cases, they become inconsistent when samples are dependent or contain heavy-tailed covariates. As a first step towards modeling structured sample dependence and heavy tails, we use right-rotationally invariant covariate distributions -- a crucial concept from compressed sensing. In the proportional asymptotics regime where the number of features and samples grow comparably, which is known to better reflect the empirical behavior in moderately sized datasets, we introduce a new framework, ROTI-GCV, for reliably performing cross-validation under these challenging conditions. Along the way, we propose new estimators for the signal-to-noise ratio and noise variance. We conduct experiments that demonstrate the accuracy of our approach in a variety of synthetic and semi-synthetic settings.