# Consistent Risk Estimation in Moderately High-Dimensional Linear Regression

arXiv: https://arxiv.org/abs/1902.01753 · HTML: https://mlsanity.com/paper/1902.01753/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Feb 2019.

## Built on by (most cited first, 14 of 14)

- [Failure and success of the spectral bias prediction for Kernel Ridge Regression: the case of low-dimensional data](https://mlsanity.com/paper/2202.03348/) (2022)
- [Derivatives and residual distribution of regularized M-estimators with application to adaptive tuning](https://mlsanity.com/paper/2107.05143/) (2021)
- [Overparameterized random feature regression with nearly orthogonal data](https://mlsanity.com/paper/2211.06077/) (2022)
- [A New Central Limit Theorem for the Augmented IPW Estimator: Variance Inflation, Cross-Fit Covariance and Beyond](https://mlsanity.com/paper/2205.10198/) (2022)
- [Corrected generalized cross-validation for finite ensembles of penalized estimators](https://mlsanity.com/paper/2310.01374/) (2023)
- [Theoretical Analysis of Leave-one-out Cross Validation for Non-differentiable Penalties under High-dimensional Settings](https://mlsanity.com/paper/2402.08543/) (2024)
- [ROTI-GCV: Generalized Cross-Validation for right-ROTationally Invariant Data](https://mlsanity.com/paper/2406.11666/) (2024)
- [Approximate Leave-one-out Cross Validation for Regression with ℓ1 Regularizers (extended version)](https://mlsanity.com/paper/2310.17629/) (2023)
- [RandALO: Out-of-sample risk estimation in no time flat](https://mlsanity.com/paper/2409.09781/) (2024)
- [Simultaneous analysis of approximate leave-one-out cross-validation and mean-field inference](https://mlsanity.com/paper/2501.02624/) (2025)
- [Ridge interpolators in correlated factor regression models -- exact risk analysis](https://mlsanity.com/paper/2406.09183/) (2024)
- [Precise analysis of ridge interpolators under heavy correlations -- a Random Duality Theory view](https://mlsanity.com/paper/2406.09199/) (2024)
- [Training NTK to Generalize with KARE](https://mlsanity.com/paper/2505.11347/) (2025)
- [Accelerating Conformal Prediction via Approximate Leave-One-Out](https://mlsanity.com/paper/2606.31915/) (2026)

## Abstract (arXiv)

Risk estimation is at the core of many learning systems. The importance of this problem has motivated researchers to propose different schemes, such as cross validation, generalized cross validation, and Bootstrap. The theoretical properties of such estimates have been extensively studied in the low-dimensional settings, where the number of predictors p is much smaller than the number of observations n. However, a unifying methodology accompanied with a rigorous theory is lacking in high-dimensional settings. This paper studies the problem of risk estimation under the moderately high-dimensional asymptotic setting n,p rightarrow ∞ and n/p rightarrow δ>1 (δ is a fixed number), and proves the consistency of three risk estimates that have been successful in numerical studies, i.e., leave-one-out cross validation (LOOCV), approximate leave-one-out (ALO), and approximate message passing (AMP)-based techniques. A corner stone of our analysis is a bound that we obtain on the discrepancy of the `residuals' obtained from AMP and LOOCV. This connection not only enables us to obtain a more refined information on the estimates of AMP, ALO, and LOOCV, but also offers an upper bound on the convergence rate of each estimate.