mlsanityData to 7 Oct 2026

Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descent

arXiv 2605.17767 ↗First posted 18 May 2026stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We study feature learning in two-layer neural networks within the linear-width regime, where the number of hidden neurons, sample size, and input dimension scale proportionally. While recent work has analyzed feature learning via a single step of gradient descent on the first layer weights in this regime, such one-step update schemes are fundamentally limited: the update to the weights is approximately rank-one, captures only a single direction, and requires the target function to have an information exponent of one. In this paper, we go beyond one-step updates to provide a full characterization of the features learned during the textitsecond step of gradient descent with step-sizes η1asymp Nα1 and η2 asymp Nα2 for α1, α2 ∈ [0,0.5), where N is the number of hidden neurons. We derive a spectral characterization of the updated weights, demonstrating they behave as a spiked random matrix with multiple outliers, each corresponding to a learned direction. We show that the number of the outliers is determined by the parameters α1, α2 through lfloor fracα21/2 - α1 rfloor. Furthermore, by analyzing the alignment between the learned directions and the target function, we identify a gap between training with independent versus reused batches. While independent batches restrict learning to directions with an information exponent of one, batch reuse enables the second update to capture directions even when the information exponent exceeds one, provided that α1, α2 are chosen properly. This shows that the benefits of batch reuse, previously observed in narrow-width regimes, persist in the linear-width limit as well. By characterizing these early-phase evolutions, our work proposes a tractable framework for studying optimization and feature learning phenomenology in modern overparameterized networks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 54 of its 77 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

54 of 77 references.

References · 54

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  3. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  4. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  5. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  6. 2016Adversarial Feature Learningcited by 350
  7. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  8. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  9. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  10. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  11. 2019Linearized two-layers neural networks in high dimensioncited by 155
  12. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
38 more
  1. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  2. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  3. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  4. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 113
  5. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  6. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  7. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
  8. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  9. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  10. 2017A Random Matrix Approach to Neural Networkscited by 80
  11. 2020Understanding Double Descent Requires a Fine-Grained Bias-Variance Decompositioncited by 75
  12. 2022Universality of empirical risk minimizationcited by 73
  13. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  14. 2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 50
  15. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 46
  16. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 45
  17. 2018Moving Beyond Sub-Gaussianity in High-Dimensional Statistics: Applications in Covariance Estimation and Linear Regressioncited by 42
  18. 2022The curse of overparametrization in adversarial training: Precise analysis of robust generalization for random features regressioncited by 40
  19. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  20. 2024Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functionscited by 35
  21. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  22. 2020What causes the test error? Going beyond bias-variance via ANOVAcited by 28
  23. 2019Eigenvalue distribution of nonlinear models of random matricescited by 27
  24. 2024Nonlinear spiked covariance matrices and signal propagation in deep neural networkscited by 25
  25. 2023Precise Asymptotic Analysis of Deep Random Feature Modelscited by 22
  26. 2019A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learningcited by 20
  27. 2023Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networkscited by 18
  28. 2024Fundamental computational limits of weak learnability in high-dimensional multi-index modelscited by 17
  29. 2022On double-descent in uncertainty quantification in overparametrized modelscited by 15
  30. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 14
  31. 2023Beyond the Universal Law of Robustness: Sharper Laws for Random Features and Neural Tangent Kernelscited by 13
  32. 2024Learning sum of diverse features: computational hardness and efficient gradient-based training for ridge combinationscited by 13
  33. 2025Weak-to-Strong Generalization Even in Random Feature Networks, Provablycited by 13
  34. 2023Demystifying Disagreement-on-the-Line in High Dimensionscited by 12
  35. 2024High-dimensional learning of narrow neural networkscited by 12
  36. 2025On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learningcited by 12
  37. 2023How Spurious Features Are Memorized: Precise Analysis for Random and NTK Featurescited by 11
  38. 2023Learning Hierarchical Polynomials with Three-Layer Neural Networkscited by 8

50 most cited of 54 All 54 (JSON)

Built on by

Not cited yet.