mlsanityData to 7 Oct 2026

Deterministic equivalent and error universality of deep random features learning

arXiv 2302.00401 ↗First posted 1 Feb 2023stat.ML cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

This manuscript considers the problem of learning a random Gaussian network function using a fully connected network with frozen intermediate layers and trainable readout layer. This problem can be seen as a natural generalization of the widely studied random features model to deeper architectures. First, we prove Gaussian universality of the test error in a ridge regression setting where the learner and target networks share the same intermediate layers, and provide a sharp asymptotic formula for it. Establishing this result requires proving a deterministic equivalent for traces of the deep random features sample covariance matrices which can be of independent interest. Second, we conjecture the asymptotic Gaussian universality of the test error in the more general setting of arbitrary convex losses and generic learner/target architectures. We provide extensive numerical evidence for this conjecture, which requires the derivation of closed-form expressions for the layer-wise post-activation population covariances. In light of our results, we investigate the interplay between architecture design and implicit regularization.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 44 of its 74 references that are in the map. First posted on arXiv 1 Feb 2023.

Builds on

44 of 74 references.

References · 44

  1. 2018On Lazy Training in Differentiable Programmingcited by 671
  2. 2017Deep Neural Networks as Gaussian Processescited by 617
  3. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  4. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  5. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  6. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  7. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  8. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  9. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  10. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  11. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  12. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
32 more
  1. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  2. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  3. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  4. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  5. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  6. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
  7. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 76
  8. 2020Understanding Double Descent Requires a Fine-Grained Bias-Variance Decompositioncited by 75
  9. 2018Phase Retrieval Under a Generative Priorcited by 73
  10. 2022Universality of empirical risk minimizationcited by 73
  11. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  12. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  13. 2020Triple descent and the two kinds of overfitting: Where & why do they appear?cited by 64
  14. 2020Implicit Regularization of Random Feature Modelscited by 54
  15. 2018Entropy and mutual information in models of deep neural networkscited by 50
  16. 2020A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-ℓ1-Norm Interpolated Classifierscited by 44
  17. 2022Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regressioncited by 42
  18. 2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
  19. 2018On the Spectrum of Random Features Maps of High Dimensional Datacited by 27
  20. 2019Eigenvalue distribution of nonlinear models of random matricescited by 27
  21. 2022Homogenization of SGD in high-dimensions: Exact dynamics and generalization propertiescited by 26
  22. 2022Gaussian Universality of Perceptrons with Random Labelscited by 26
  23. 2022Contrasting random and learned features in deep Bayesian linear regressioncited by 24
  24. 2019The spiked matrix model with generative priorscited by 20
  25. 2019Exact asymptotics for phase retrieval and compressed sensing with random generative priorscited by 20
  26. 2022Fluctuations, Bias, Variance & Ensemble of Learners: Exact Asymptotics for Convex Losses in High-Dimensioncited by 19
  27. 2022Sharp Asymptotics of Kernel Ridge Regression Beyond the Linear Regimecited by 18
  28. 2021Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature modelcited by 17
  29. 2017Multi-Layer Generalized Linear Estimationcited by 12
  30. 2021On the interplay between data structure and loss function in classification problemscited by 12
  31. 2022Quantitative deterministic equivalent of sample covariance matrices with a general dependence structurecited by 9
  32. 2019Large deviations for the perceptron model and consequences for active learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 13; 2025: 8; 2026: 4.

35 papers.

Sort:

  1. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  2. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  3. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  4. 2023Universality laws for Gaussian mixtures in generalized linear modelscited by 28
  5. 2023Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimationcited by 25
  6. 2024Nonlinear spiked covariance matrices and signal propagation in deep neural networkscited by 25
  7. 2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
  8. 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
  9. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 14
  10. 2023Learning curves for deep structured Gaussian feature modelscited by 13
  11. 2024Asymptotics of Learning with Deep Structured (Random) Featurescited by 13
  12. 2023Fundamental limits of overparametrized shallow neural networks for supervised learningcited by 12
23 more
  1. 2024High-dimensional learning of narrow neural networkscited by 12
  2. 2024Random features and polynomial rulescited by 6
  3. 2025Two-Point Deterministic Equivalence for Stochastic Gradient Dynamics in Linear Modelscited by 6
  4. 2025Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regimecited by 6
  5. 2023Deterministic equivalent of the Conjugate Kernel matrix associated to Artificial Neural Networkscited by 5
  6. 2024Towards Understanding the Word Sensitivity of Attention Layers: A Study via Random Featurescited by 5
  7. 2024A High Dimensional Statistical Model for Adversarial Training: Geometry and Trade-Offscited by 5
  8. 2023Universality of max-margin classifierscited by 4
  9. 2024The Breakdown of Gaussian Universality in Classification of High-dimensional Linear Factor Mixturescited by 4
  10. 2025Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structurecited by 4
  11. 2026Power-Law Spectrum of the Random Feature Modelcited by 3
  12. 2024Input-Label Correlation Governs a Linear-to-Nonlinear Transition in Random Features under Spiked Covariancecited by 2
  13. 2024On the Geometry of Regularization in Adversarial Training: High-Dimensional Asymptotics and Generalization Boundscited by 2
  14. 2025Eigenvalue distribution of the Neural Tangent Kernel in the quadratic scalingcited by 2
  15. 2023Orthogonal Random Features: Explicit Forms and Sharp Inequalitiescited by 1
  16. 2025On the Existence of Consistent Adversarial Attacks in High-Dimensional Linear Classificationcited by 1
  17. 2025Understanding and Enhancing Mask-Based Pretraining towards Universal Representationscited by 1
  18. 2023Spatially heterogeneous learning by a deep student machinecited by 0
  19. 2025One-Bit Quantization for Random Features Modelscited by 0
  20. 2025Generalization in Representation Models via Random Matrix Theory: Application to Recurrent Networkscited by 0
  21. 2026A Random Matrix Theory of Masked Self-Supervised Regressioncited by 0
  22. 2026Characterizing the Generalization Error of Random Feature Regression with Arbitrary Data-Augmentationcited by 0
  23. 2026Geometric Dyson Brownian Motions and the Free Log-Normal Limit for a Non-Square Gaussian Matrix Productcited by 0