mlsanityData to 14 Oct 2026

Overparameterized random feature regression with nearly orthogonal data

arXiv 2211.06077 ↗First posted 11 Nov 2022math.ST cs.LG math.PRcited by 7

Not on the map. Why?

Abstract (arXiv)

We investigate the properties of random feature ridge regression (RFRR) given by a two-layer neural network with random Gaussian initialization. We study the non-asymptotic behaviors of the RFRR with nearly orthogonal deterministic unit-length input data vectors in the overparameterized regime, where the width of the first layer is much larger than the sample size. Our analysis shows high-probability non-asymptotic concentration results for the training errors, cross-validations, and generalization errors of RFRR centered around their respective values for a kernel ridge regression (KRR). This KRR is derived from an expected kernel generated by a nonlinear random feature map. We then approximate the performance of the KRR by a polynomial kernel matrix obtained from the Hermite polynomial expansion of the activation function, whose degree only depends on the orthogonality among different data points. This polynomial kernel determines the asymptotic behavior of the RFRR and the KRR. Our results hold for a wide variety of activation functions and input data sets that exhibit nearly orthogonal properties. Based on these approximations, we obtain a lower bound for the generalization error of the RFRR for a nonlinear student-teacher model.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 52 of its 86 references that are in the map. First posted on arXiv 11 Nov 2022.

Builds on

52 of 86 references.

References · 52

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  3. 2017Deep Neural Networks as Gaussian Processescited by 622
  4. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  5. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  6. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  7. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  8. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  9. 2021Deep learning: a statistical viewpointcited by 240
  10. 2016Deep Information Propagationcited by 195
  11. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  12. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
38 more
  1. 2016Generalization Properties of Learning with Random Featurescited by 164
  2. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  3. 2019Linearized two-layers neural networks in high dimensioncited by 155
  4. 2015On the Equivalence between Kernel Quadrature Rules and Random Feature Expansionscited by 138
  5. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  6. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 127
  7. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  8. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  9. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  10. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  11. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 91
  12. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  13. 2017A Random Matrix Approach to Neural Networkscited by 80
  14. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 77
  15. 2018Random Fourier Features for Kernel Ridge Regression: Approximation Bounds and Statistical Guaranteescited by 71
  16. 2020A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descentcited by 70
  17. 2018A Continuous-Time View of Early Stopping for Least Squarescited by 69
  18. 2020Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topologycited by 66
  19. 2020The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networkscited by 58
  20. 2020Implicit Regularization of Random Feature Modelscited by 54
  21. 2020Kernel Alignment Risk Estimator: Risk Prediction from Training Datacited by 54
  22. 2020Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixturescited by 47
  23. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  24. 2022Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regressioncited by 45
  25. 2020Kernel regression in high dimensions: Refined analysis beyond double descentcited by 39
  26. 2021How rotational invariance of common kernels prevents generalization in high dimensionscited by 32
  27. 2018On the Spectrum of Random Features Maps of High Dimensional Datacited by 28
  28. 2020What causes the test error? Going beyond bias-variance via ANOVAcited by 28
  29. 2019Eigenvalue distribution of nonlinear models of random matricescited by 27
  30. 2022An Equivalence Principle for the Spectrum of Random Inner-Product Kernel Matrices with Polynomial Scalingscited by 27
  31. 2022Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regressioncited by 22
  32. 2021Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networkscited by 21
  33. 2022Sharp Asymptotics of Kernel Ridge Regression Beyond the Linear Regimecited by 19
  34. 2019Ridge Regression: Structure, Cross-Validation, and Sketchingcited by 15
  35. 2019Consistent Risk Estimation in Moderately High-Dimensional Linear Regressioncited by 14
  36. 2022Largest Eigenvalues of the Conjugate Kernel of Single-Layered Neural Networkscited by 13
  37. 2022Efficient Hyperparameter Tuning for Large Scale Kernel Ridge Regressioncited by 11
  38. 2022Concentration of Random Feature Matrices in High-Dimensionscited by 9

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 2.

7 papers.

  1. 2024Random features and polynomial rulescited by 6
  2. 2025A general technique for approximating high-dimensional empirical kernel matricescited by 4
  3. 2026Power-Law Spectrum of the Random Feature Modelcited by 4
  4. 2025Restoring Pruned Large Language Models via Lost Component Compensationcited by 3
  5. 2024On the Pinsker bound of inner product kernel regression in large dimensionscited by 2
  6. 2025Balancing Stylization and Truth via Disentangled Representation Steeringcited by 2
  7. 2026Random Feature Pruning for Multi-Class Classificationcited by 0