# Risk and cross validation in ridge regression with correlated samples

arXiv: https://arxiv.org/abs/2408.04607 · HTML: https://mlsanity.com/paper/2408.04607/

## Key facts

- Cited by 8 papers in the map (data through 7 Oct 2026).
- Builds on 34 of its 77 references that are in the map.
- First posted on arXiv 8 Aug 2024.

## Builds on (most cited first, 30 of 34)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks](https://mlsanity.com/paper/2002.02561/) (2020)
- [Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks](https://mlsanity.com/paper/2006.13198/) (2020)
- [High-Dimensional Asymptotics of Prediction: Ridge Regression and Classification](https://mlsanity.com/paper/1507.03003/) (2015)
- [Generalisation error in learning with random features and the hidden manifold model](https://mlsanity.com/paper/2002.09339/) (2020)
- [Learning curves of generic features maps for realistic datasets with a teacher-student model](https://mlsanity.com/paper/2102.08127/) (2021)
- [A Dynamical Model of Neural Scaling Laws](https://mlsanity.com/paper/2402.01092/) (2024)
- [A Solvable Model of Neural Scaling Laws](https://mlsanity.com/paper/2210.16859/) (2022)
- [Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime](https://mlsanity.com/paper/2105.15004/) (2021)
- [Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition](https://mlsanity.com/paper/2011.03321/) (2020)
- [Universality of empirical risk minimization](https://mlsanity.com/paper/2202.08832/) (2022)
- [Cross-validation: what does it estimate and how well does it do it?](https://mlsanity.com/paper/2104.00673/) (2021)
- [Kernel Alignment Risk Estimator: Risk Prediction from Training Data](https://mlsanity.com/paper/2006.09796/) (2020)
- [Asymptotic theory of in-context learning by linear attention](https://mlsanity.com/paper/2405.11751/) (2024)
- [High-dimensional analysis of double descent for linear regression with random projections](https://mlsanity.com/paper/2303.01372/) (2023)
- [Scaling and renormalization in high-dimensional regression](https://mlsanity.com/paper/2405.00592/) (2024)
- [What causes the test error? Going beyond bias-variance via ANOVA](https://mlsanity.com/paper/2010.05170/) (2020)
- [Covariate Shift in High-Dimensional Random Feature Regression](https://mlsanity.com/paper/2111.08234/) (2021)
- [A non-asymptotic theory of Kernel Ridge Regression: deterministic equivalents, test error, and GCV estimator](https://mlsanity.com/paper/2403.08938/) (2024)
- [Out-of-Distribution Generalization in Kernel Regression](https://mlsanity.com/paper/2106.02261/) (2021)
- [Optimal Ridge Regularization for Out-of-Distribution Prediction](https://mlsanity.com/paper/2404.01233/) (2024)
- [Statistical Mechanics of High-Dimensional Inference](https://mlsanity.com/paper/1601.04650/) (2016)
- [Learning curves for deep structured Gaussian feature models](https://mlsanity.com/paper/2303.00564/) (2023)
- [Cross-Validation for Correlated Data](https://mlsanity.com/paper/1904.02438/) (2019)
- [Asymptotically free sketched ridge ensembles: Risks, cross-validation, and tuning](https://mlsanity.com/paper/2310.04357/) (2023)

## Built on by (most cited first, 8 of 8)

- [On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective](https://mlsanity.com/paper/2505.18346/) (2025)
- [Two-Point Deterministic Equivalence for Stochastic Gradient Dynamics in Linear Models](https://mlsanity.com/paper/2502.05074/) (2025)
- [Pretrain-Test Task Alignment Governs Generalization in In-Context Learning](https://mlsanity.com/paper/2509.26551/) (2025)
- [Asymptotics of Linear Regression with Linearly Dependent Data](https://mlsanity.com/paper/2412.03702/) (2024)
- [Generalisation and benign over-fitting for linear regression onto random functional covariates](https://mlsanity.com/paper/2508.13895/) (2025)
- [A Ridge Too Far: Correcting Over-Shrinkage via Negative Regularization](https://mlsanity.com/paper/2508.17412/) (2025)
- [Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch](https://mlsanity.com/paper/2607.03660/) (2026)
- [A Simple Approximation to the Distribution of the Ridge Regression Estimator](https://mlsanity.com/paper/2608.02539/) (2026)

## Abstract (arXiv)

Recent years have seen substantial advances in our understanding of high-dimensional ridge regression, but existing theories assume that training examples are independent. By leveraging techniques from random matrix theory and free probability, we provide sharp asymptotics for the in- and out-of-sample risks of ridge regression when the data points have arbitrary correlations. We demonstrate that in this setting, the generalized cross validation estimator (GCV) fails to correctly predict the out-of-sample risk. However, in the case where the noise residuals have the same correlations as the data points, one can modify the GCV to yield an efficiently-computable unbiased estimator that concentrates in the high-dimensional limit, which we dub CorrGCV. We further extend our asymptotic analysis to the case where the test point has nontrivial correlations with the training set, a setting often encountered in time series forecasting. Assuming knowledge of the correlation structure of the time series, this again yields an extension of the GCV estimator, and sharply characterizes the degree to which such test points yield an overly optimistic prediction of long-time risk. We validate the predictions of our theory across a variety of high dimensional data.