mlsanityData to 13 Oct 2026

Asymptotics of Learning with Deep Structured (Random) Features

arXiv 2402.13999 ↗First posted 21 Feb 2024stat.ML cond-mat.dis-nn cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

For a large class of feature maps we provide a tight asymptotic characterisation of the test error associated with learning the readout layer, in the high-dimensional limit where the input dimension, hidden layer widths, and number of training samples are proportionally large. This characterization is formulated in terms of the population covariance of the features. Our work is partially motivated by the problem of learning with Gaussian rainbow neural networks, namely deep non-linear fully-connected networks with random but structured weights, whose row-wise covariances are further allowed to depend on the weights of previous layers. For such networks we also derive a closed-form formula for the feature covariance in terms of the weight matrices. We further find that in some cases our results can capture feature maps learned by deep, finite-width neural networks trained under gradient descent.

Key facts. Cited by 13 papers in the map (data through 13 Oct 2026). Builds on 50 of its 76 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

50 of 76 references.

References · 50

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 920
  3. 2017Deep Neural Networks as Gaussian Processescited by 618
  4. 2019Benign Overfitting in Linear Regressioncited by 545
  5. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 411
  6. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 316
  7. 2021Deep learning: a statistical viewpointcited by 240
  8. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 213
  9. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  10. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 147
  11. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 147
  12. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
38 more
  1. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  2. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 115
  3. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  4. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  5. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  6. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  7. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 98
  8. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  9. 2017A Random Matrix Approach to Neural Networkscited by 80
  10. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 76
  11. 2018Phase Retrieval Under a Generative Priorcited by 73
  12. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  13. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 69
  14. 2020Implicit Regularization of Random Feature Modelscited by 54
  15. 2018Entropy and mutual information in models of deep neural networkscited by 50
  16. 2022Dimension free ridge regressioncited by 50
  17. 2023High-dimensional analysis of double descent for linear regression with random projectionscited by 47
  18. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 46
  19. 2020A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-ℓ1-Norm Interpolated Classifierscited by 44
  20. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  21. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 37
  22. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  23. 2020Predicting the outputs of finite deep neural networks trained with noisy gradientscited by 31
  24. 2018On the Spectrum of Random Features Maps of High Dimensional Datacited by 27
  25. 2019Eigenvalue distribution of nonlinear models of random matricescited by 27
  26. 2022Gaussian Universality of Perceptrons with Random Labelscited by 26
  27. 2023Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimationcited by 25
  28. 2022Contrasting random and learned features in deep Bayesian linear regressioncited by 24
  29. 2023Precise Asymptotic Analysis of Deep Random Feature Modelscited by 22
  30. 2022Random matrix analysis of deep neural network weight matricescited by 21
  31. 2019The spiked matrix model with generative priorscited by 20
  32. 2019Exact asymptotics for phase retrieval and compressed sensing with random generative priorscited by 20
  33. 2023A Rainbow in Deep Network Black Boxescited by 18
  34. 2023Learning curves for deep structured Gaussian feature modelscited by 13
  35. 2017Multi-Layer Generalized Linear Estimationcited by 12
  36. 2022Quantitative deterministic equivalent of sample covariance matrices with a general dependence structurecited by 9
  37. 2023Deterministic equivalent of the Conjugate Kernel matrix associated to Artificial Neural Networkscited by 5
  38. 2022Random Features Model with General Convex Regularization: A Fine Grained Analysis with Precise Asymptotic Learning Curvescited by 3

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 5; 2025: 5; 2026: 3.

13 papers.

Sort:

  1. 2024Scaling and renormalization in high-dimensional regressioncited by 46
  2. 2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
  3. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 14
  4. 2024High-dimensional learning of narrow neural networkscited by 12
  5. 2025Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regimecited by 6
  6. 2024On the Geometry of Regularization in Adversarial Training: High-Dimensional Asymptotics and Generalization Boundscited by 4
  7. 2026Power-Law Spectrum of the Random Feature Modelcited by 4
  8. 2025A statistical physics framework for optimal learningcited by 3
  9. 2025On the Existence of Consistent Adversarial Attacks in High-Dimensional Linear Classificationcited by 2
  10. 2025One-Bit Quantization for Random Features Modelscited by 0
  11. 2025Supervised Learning of Random Neural Architectures Structured by Latent Random Fields on Compact Boundaryless Multiply-Connected Manifoldscited by 0
  12. 2026A Random Matrix Theory of Masked Self-Supervised Regressioncited by 0
1 more
  1. 2026Characterizing the Generalization Error of Random Feature Regression with Arbitrary Data-Augmentationcited by 0