# High-dimensional ridge regression with random features for non-identically distributed data with a variance profile

arXiv: https://arxiv.org/abs/2504.03035 · HTML: https://mlsanity.com/paper/2504.03035/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 33 references that are in the map.
- First posted on arXiv 3 Apr 2025.

## Builds on (most cited first, 15 of 15)

- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [High-Dimensional Asymptotics of Prediction: Ridge Regression and Classification](https://mlsanity.com/paper/1507.03003/) (2015)
- [The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization](https://mlsanity.com/paper/2008.06786/) (2020)
- [Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime](https://mlsanity.com/paper/2003.01054/) (2020)
- [A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descent](https://mlsanity.com/paper/2006.05013/) (2020)
- [Triple descent and the two kinds of overfitting: Where & why do they appear?](https://mlsanity.com/paper/2006.03509/) (2020)
- [Dimension free ridge regression](https://mlsanity.com/paper/2210.08571/) (2022)
- [Eigenvalue distribution of nonlinear models of random matrices](https://mlsanity.com/paper/1904.03090/) (2019)
- [Dimension-free deterministic equivalents and scaling laws for random feature regression](https://mlsanity.com/paper/2405.15699/) (2024)
- [A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learning](https://mlsanity.com/paper/1912.00827/) (2019)
- [Multiple Descent in the Multiple Random Feature Model](https://mlsanity.com/paper/2208.09897/) (2022)
- [Applications of Realizations (aka Linearizations) to Free Probability](https://mlsanity.com/paper/1511.05330/) (2015)
- [High-dimensional analysis of ridge regression for non-identically distributed data with a variance profile](https://mlsanity.com/paper/2403.20200/) (2024)

## Built on by (most cited first, 2 of 2)

- [Supervised Learning of Random Neural Architectures Structured by Latent Random Fields on Compact Boundaryless Multiply-Connected Manifolds](https://mlsanity.com/paper/2512.10407/) (2025)
- [The Zero Pattern of a Design Matrix Drives Multiple Descent in Over-parameterized Regression](https://mlsanity.com/paper/2607.24041/) (2026)

## Abstract (arXiv)

Random feature ridge regression is often analyzed in the high-dimensional regime under the homogeneous sampling model xi=Sigma1/2xi', where the vectors xi' have iid entries and the same covariance matrix Sigma is shared by all samples. In this paper, we move beyond this setting and study non-identically distributed data through a variance-profile model in which the training and test covariates have row-dependent diagonal covariance matrices Sigmai=diag(γi12,ldots,γip2) and widetildeSigmai=diag(tildeγi12,ldots,tildeγip2). Our main contribution is the derivation of asymptotic equivalents for the training and test risks of ridge regression with random features when n, p, and m grow proportionally. The first set of equivalents is obtained by combining the linear-plus-chaos approximation with traffic-probability arguments, whereas the second set is deterministic and follows from operator-valued free probability through an amalgamation-over-the-diagonal argument. These equivalents are sharp in numerical experiments. They also reveal how heterogeneous variance profiles, including mixture-type profiles inspired by MNIST, can modify generalization and exhibit double-descent behavior when the ridge parameter is small.