mlsanityData to 14 Oct 2026

Concentration of Random Feature Matrices in High-Dimensions

arXiv 2204.06935 ↗First posted 14 Apr 2022stat.ML cs.LG cs.NAcited by 9

Not on the map. Why?

Abstract (arXiv)

The spectra of random feature matrices provide essential information on the conditioning of the linear system used in random feature regression problems and are thus connected to the consistency and generalization of random feature models. Random feature matrices are asymmetric rectangular nonlinear matrices depending on two input variables, the data and the weights, which can make their characterization challenging. We consider two settings for the two input variables, either both are random variables or one is a random variable and the other is well-separated, i.e. there is a minimum distance between points. With conditions on the dimension, the complexity ratio, and the sampling variance, we show that the singular values of these matrices concentrate near their full expectation and near one with high-probability. In particular, since the dimension depends only on the logarithm of the number of random weights or the number of data points, our complexity bounds can be achieved even in moderate dimensions for many practical setting. The theoretical results are verified with numerical experiments.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 21 of its 41 references that are in the map. First posted on arXiv 14 Apr 2022.

Builds on

21 of 41 references.

References · 21

  1. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  2. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  3. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  4. 2019Two models of double descent for weak featurescited by 195
  5. 2018To understand deep learning we need to understand kernel learningcited by 187
  6. 2016Generalization Properties of Learning with Random Featurescited by 164
  7. 2020Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyondcited by 109
  8. 2017A Random Matrix Approach to Neural Networkscited by 80
  9. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 77
  10. 2018Random Fourier Features for Kernel Ridge Regression: Approximation Bounds and Statistical Guaranteescited by 71
  11. 2020A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descentcited by 70
  12. 2021Generalization Bounds for Sparse Random Feature Expansionscited by 22
9 more
  1. 2021Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networkscited by 21
  2. 2020On Random Matrices Arising in Deep Neural Networks. Gaussian Casecited by 14
  3. 2022HARFE: Hard-Ridge Random Feature Expansioncited by 12
  4. 2020On Random Matrices Arising in Deep Neural Networks: General I.I.D. Casecited by 10
  5. 2021SHRIMP: Sparser Random Feature Models via Iterative Magnitude Pruningcited by 9
  6. 2021Conditioning of Random Feature Matrices: Double Descent and Generalization Errorcited by 7
  7. 2022SRMD: Sparse Random Mode Decompositioncited by 6
  8. 2020Sparse Recovery With Non-Linear Fourier Featurescited by 3
  9. 2019Sparse Recovery for Orthogonal Polynomial Transformscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 1; 2024: 2; 2025: 2; 2026: 2.

9 papers.

  1. 2023Error Bounds for Learning with Vector-Valued Random Featurescited by 23
  2. 2022Random Feature Models for Learning Interacting Dynamical Systemscited by 10
  3. 2022Overparameterized random feature regression with nearly orthogonal datacited by 7
  4. 2025Cauchy Random Features for Operator Learning in Sobolev Spacecited by 5
  5. 2024Differentially Private Random Feature Modelcited by 2
  6. 2025Regularized Random Fourier Features and Finite Element Reconstruction for Operator Learning in Sobolev Spacecited by 2
  7. 2024KITE: A Kernel-based Improved Transferability Estimation Methodcited by 1
  8. 2026Spectral Convergence of Random Feature Method in Multiple Dimensionscited by 0
  9. 2026Random Feature Pruning for Multi-Class Classificationcited by 0