mlsanityData to 7 Oct 2026

Asymptotics of feature learning in two-layer networks after one gradient-step

arXiv 2402.04980 ↗First posted 7 Feb 2024stat.ML cond-mat.dis-nn cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

In this manuscript, we investigate the problem of how two-layer neural networks learn features from data, and improve over the kernel regime, after being trained with a single gradient descent step. Leveraging the insight from (Ba et al., 2022), we model the trained network by a spiked Random Features (sRF) model. Further building on recent progress on Gaussian universality (Dandi et al., 2023), we provide an exact asymptotic description of the generalization error of the sRF in the high-dimensional limit where the number of samples, the width, and the input dimension grow at a proportional rate. The resulting characterization for sRFs also captures closely the learning curves of the original network model. This enables us to understand how adapting to the data is crucial for the network to efficiently learn non-linear functions in the direction of the gradient -- where at initialization it can only express linear functions in this regime.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 52 of its 68 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

52 of 68 references.

References · 52

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2018On Lazy Training in Differentiable Programmingcited by 671
  4. 2017Deep Neural Networks as Gaussian Processescited by 617
  5. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  6. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  7. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  8. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  9. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  10. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  11. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  12. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 141
38 more
  1. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  2. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  3. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  4. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  5. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  6. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  7. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
  8. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
  9. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  10. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  11. 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
  12. 2022Universality of empirical risk minimizationcited by 73
  13. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
  14. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  15. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  16. 2020The role of regularization in classification of high-dimensional noisy Gaussian mixturecited by 63
  17. 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
  18. 2023Learning time-scales in two-layers neural networkscited by 57
  19. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  20. 2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 52
  21. 2020Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimizationcited by 51
  22. 2022A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Featurescited by 49
  23. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 46
  24. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 42
  25. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
  26. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  27. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  28. 2021A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNscited by 33
  29. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  30. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  31. 2023Universality laws for Gaussian mixtures in generalized linear modelscited by 28
  32. 2023Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimationcited by 25
  33. 2023Precise Asymptotic Analysis of Deep Random Feature Modelscited by 22
  34. 2023High-dimensional robust regression under heavy-tailed data: Asymptotics and Universalitycited by 15
  35. 2013Upper-bounding ℓ1-optimization weak thresholdscited by 13
  36. 2023Learning curves for deep structured Gaussian feature modelscited by 13
  37. 2023Classification of Heavy-tailed Features in High Dimensions: a Superstatistical Approachcited by 11
  38. 2023Kernel Regression with Infinite-Width Neural Networks on Millions of Examplescited by 9

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 10; 2026: 11.

35 papers.

Sort:

By area:

  1. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 46
  2. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  3. 2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
  4. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  5. 2025Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regimecited by 18
  6. 2025Survey on Algorithms for multi-index modelscited by 15
  7. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 14
  8. 2024Asymptotics of Learning with Deep Structured (Random) Featurescited by 13
  9. 2024High-dimensional learning of narrow neural networkscited by 12
  10. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  11. 2024From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networkscited by 9
  12. 2025Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networkscited by 9
23 more
  1. 2024Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methodscited by 7
  2. 2025On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspectivecited by 7
  3. 2024Restoring balance: principled under/oversampling of data for optimal classificationcited by 6
  4. 2025Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structurecited by 4
  5. 2025Low Rank Gradients and Where to Find Themcited by 4
  6. 2024Signal-Plus-Noise Decomposition of Nonlinear Spiked Random Matrix Modelscited by 2
  7. 2024Spring-block theory of feature learning in deep neural networkscited by 2
  8. 2024Input-Label Correlation Governs a Linear-to-Nonlinear Transition in Random Features under Spiked Covariancecited by 2
  9. 2026When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index modelscited by 2
  10. 2026Statistical Properties of Training & Generalizationcited by 2
  11. 2024Learning Hierarchical Polynomials of Multiple Nonlinear Features with Three-Layer Networkscited by 1
  12. 2025The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Modelcited by 1
  13. 2026A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Modelscited by 1
  14. 2026Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regimecited by 1
  15. 2024Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theoriescited by 0
  16. 2025Deep Neural Nets as Hamiltonianscited by 0
  17. 2025Gaussian Equivalence for Self-Attention: Asymptotic Spectral Analysis of Attention Matrixcited by 0
  18. 2026Learning Beyond the Gaussian Data: Learning Dynamics of Neural Networks on an Expressive and Cumulant-Controllable Data Modelcited by 0
  19. 2026Dichotomy of Feature Learning and Unlearning: Fast-Slow Analysis on Neural Networks with Stochastic Gradient Descentcited by 0
  20. 2026Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descentcited by 0
  21. 2026Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probingcited by 0
  22. 2026Feature Learning for the High Dimensional Stationary Sch"odinger Equation with Deep Ritz Methodcited by 0
  23. 2026Dimension-Corrected Hitting Times for Heavy-Tailed Spectral Emergence in Neural Optimizer Dynamicscited by 0