# Concentration of Random Feature Matrices in High-Dimensions

arXiv: https://arxiv.org/abs/2204.06935 · HTML: https://mlsanity.com/paper/2204.06935/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 14 Apr 2022.

## Builds on (most cited first, 21 of 21)

- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [Two models of double descent for weak features](https://mlsanity.com/paper/1903.07571/) (2019)
- [To understand deep learning we need to understand kernel learning](https://mlsanity.com/paper/1802.01396/) (2018)
- [Generalization Properties of Learning with Random Features](https://mlsanity.com/paper/1602.04474/) (2016)
- [Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond](https://mlsanity.com/paper/2004.11154/) (2020)
- [A Random Matrix Approach to Neural Networks](https://mlsanity.com/paper/1702.05419/) (2017)
- [Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks](https://mlsanity.com/paper/2005.11879/) (2020)
- [Random Fourier Features for Kernel Ridge Regression: Approximation Bounds and Statistical Guarantees](https://mlsanity.com/paper/1804.09893/) (2018)
- [A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descent](https://mlsanity.com/paper/2006.05013/) (2020)
- [Generalization Bounds for Sparse Random Feature Expansions](https://mlsanity.com/paper/2103.03191/) (2021)
- [Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks](https://mlsanity.com/paper/2109.09304/) (2021)
- [On Random Matrices Arising in Deep Neural Networks. Gaussian Case](https://mlsanity.com/paper/2001.06188/) (2020)
- [HARFE: Hard-Ridge Random Feature Expansion](https://mlsanity.com/paper/2202.02877/) (2022)
- [On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case](https://mlsanity.com/paper/2011.11439/) (2020)
- [SHRIMP: Sparser Random Feature Models via Iterative Magnitude Pruning](https://mlsanity.com/paper/2112.04002/) (2021)
- [Conditioning of Random Feature Matrices: Double Descent and Generalization Error](https://mlsanity.com/paper/2110.11477/) (2021)
- [SRMD: Sparse Random Mode Decomposition](https://mlsanity.com/paper/2204.06108/) (2022)
- [Sparse Recovery With Non-Linear Fourier Features](https://mlsanity.com/paper/2002.04985/) (2020)
- [Sparse Recovery for Orthogonal Polynomial Transforms](https://mlsanity.com/paper/1907.08362/) (2019)

## Built on by (most cited first, 9 of 9)

- [Error Bounds for Learning with Vector-Valued Random Features](https://mlsanity.com/paper/2305.17170/) (2023)
- [Random Feature Models for Learning Interacting Dynamical Systems](https://mlsanity.com/paper/2212.05591/) (2022)
- [Overparameterized random feature regression with nearly orthogonal data](https://mlsanity.com/paper/2211.06077/) (2022)
- [Cauchy Random Features for Operator Learning in Sobolev Space](https://mlsanity.com/paper/2503.00300/) (2025)
- [Differentially Private Random Feature Model](https://mlsanity.com/paper/2412.04785/) (2024)
- [Regularized Random Fourier Features and Finite Element Reconstruction for Operator Learning in Sobolev Space](https://mlsanity.com/paper/2512.17884/) (2025)
- [KITE: A Kernel-based Improved Transferability Estimation Method](https://mlsanity.com/paper/2405.01603/) (2024)
- [Spectral Convergence of Random Feature Method in Multiple Dimensions](https://mlsanity.com/paper/2609.03401/) (2026)
- [Random Feature Pruning for Multi-Class Classification](https://mlsanity.com/paper/2610.15101/) (2026)

## Abstract (arXiv)

The spectra of random feature matrices provide essential information on the conditioning of the linear system used in random feature regression problems and are thus connected to the consistency and generalization of random feature models. Random feature matrices are asymmetric rectangular nonlinear matrices depending on two input variables, the data and the weights, which can make their characterization challenging. We consider two settings for the two input variables, either both are random variables or one is a random variable and the other is well-separated, i.e. there is a minimum distance between points. With conditions on the dimension, the complexity ratio, and the sampling variance, we show that the singular values of these matrices concentrate near their full expectation and near one with high-probability. In particular, since the dimension depends only on the logarithm of the number of random weights or the number of data points, our complexity bounds can be achieved even in moderate dimensions for many practical setting. The theoretical results are verified with numerical experiments.