mlsanityData to 7 Oct 2026

Precise Asymptotic Analysis of Deep Random Feature Models

arXiv 2302.06210 ↗First posted 13 Feb 2023stat.ML cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

We provide exact asymptotic expressions for the performance of regression by an L-layer deep random feature (RF) model, where the input is mapped through multiple random embedding and non-linear activation functions. For this purpose, we establish two key steps: First, we prove a novel universality result for RF models and deterministic data, by which we demonstrate that a deep random feature model is equivalent to a deep linear Gaussian model that matches it in the first and second moments, at each layer. Second, we make use of the convex Gaussian Min-Max theorem multiple times to obtain the exact behavior of deep RF models. We further characterize the variation of the eigendistribution in different layers of the equivalent Gaussian model, demonstrating that depth has a tangible effect on model performance despite the fact that only the last layer of the model is being trained.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 22 of its 53 references that are in the map. First posted on arXiv 13 Feb 2023.

Builds on

22 of 53 references.

References · 22

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  3. 2017Deep Neural Networks as Gaussian Processescited by 617
  4. 2019Benign Overfitting in Linear Regressioncited by 544
  5. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  6. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  7. 2019Two models of double descent for weak featurescited by 194
  8. 2019Linearized two-layers neural networks in high dimensioncited by 155
  9. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  10. 2015On the Equivalence between Kernel Quadrature Rules and Random Feature Expansionscited by 136
  11. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  12. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
10 more
  1. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  2. 2020Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyondcited by 107
  3. 2017A Random Matrix Approach to Neural Networkscited by 80
  4. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 76
  5. 2022Universality of empirical risk minimizationcited by 73
  6. 2017SGD Learns the Conjugate Kernel Class of the Networkcited by 54
  7. 2020Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixturescited by 47
  8. 2020Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networkscited by 40
  9. 2015Universality laws for randomized dimension reduction, with applicationscited by 21
  10. 2021On the Inherent Regularization Effects of Noise Injection During Trainingcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 10; 2025: 4; 2026: 4.

22 papers.

Sort:

By area:

  1. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  2. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 46
  3. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  4. 2024Nonlinear spiked covariance matrices and signal propagation in deep neural networkscited by 25
  5. 2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
  6. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 14
  7. 2023Learning curves for deep structured Gaussian feature modelscited by 13
  8. 2024Asymptotics of Learning with Deep Structured (Random) Featurescited by 13
  9. 2023Fundamental limits of overparametrized shallow neural networks for supervised learningcited by 12
  10. 2024High-dimensional learning of narrow neural networkscited by 12
  11. 2025Learning dynamical systems with hit-and-run random feature mapscited by 7
  12. 2024Random features and polynomial rulescited by 6
10 more
  1. 2025Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regimecited by 6
  2. 2024Towards Understanding the Word Sensitivity of Attention Layers: A Study via Random Featurescited by 5
  3. 2025Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structurecited by 4
  4. 2024MUSO: Achieving Exact Machine Unlearning in Over-Parameterized Regimescited by 3
  5. 2024Input-Label Correlation Governs a Linear-to-Nonlinear Transition in Random Features under Spiked Covariancecited by 2
  6. 2025One-Bit Quantization for Random Features Modelscited by 0
  7. 2026A Gaussian Comparison Theorem for Training Dynamics in Machine Learningcited by 0
  8. 2026Characterizing the Generalization Error of Random Feature Regression with Arbitrary Data-Augmentationcited by 0
  9. 2026Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descentcited by 0
  10. 2026Geometric Dyson Brownian Motions and the Free Log-Normal Limit for a Non-Square Gaussian Matrix Productcited by 0