mlsanityData to 14 Oct 2026

A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks

arXiv 2310.07891 ↗First posted 11 Oct 2023stat.ML cs.LGcited by 49

Not on the map. Why?

Abstract (arXiv)

Feature learning is thought to be one of the fundamental reasons for the success of deep neural networks. It is rigorously known that in two-layer fully-connected neural networks under certain conditions, one step of gradient descent on the first layer can lead to feature learning; characterized by the appearance of a separated rank-one component -- spike -- in the spectrum of the feature matrix. However, with a constant gradient descent step size, this spike only carries information from the linear component of the target function and therefore learning non-linear components is impossible. We show that with a learning rate that grows with the sample size, such training in fact introduces multiple rank-one components, each corresponding to a specific polynomial feature. We further prove that the limiting large-dimensional and large sample training and test errors of the updated neural networks are fully characterized by these spikes. By precisely analyzing the improvement in the training and test errors, we demonstrate that these non-linear features can enhance learning.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Builds on 35 of its 86 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

35 of 86 references.

References · 35

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  3. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  4. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  5. 2016Adversarial Feature Learningcited by 350
  6. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  7. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  8. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  9. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  10. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 148
  11. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  12. 2020Spectral Methods for Data Science: A Statistical Perspectivecited by 112
23 more
  1. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  2. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  3. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  4. 2020Understanding Double Descent Requires a Fine-Grained Bias-Variance Decompositioncited by 76
  5. 2022Universality of empirical risk minimizationcited by 75
  6. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 68
  7. 2022A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Featurescited by 49
  8. 2022Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regressioncited by 45
  9. 2018Moving Beyond Sub-Gaussianity in High-Dimensional Statistics: Applications in Covariance Estimation and Linear Regressioncited by 42
  10. 2022The curse of overparametrization in adversarial training: Precise analysis of robust generalization for random features regressioncited by 41
  11. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  12. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  13. 2020What causes the test error? Going beyond bias-variance via ANOVAcited by 28
  14. 2020Concentration Inequalities for Statistical Inferencecited by 24
  15. 2023Precise Asymptotic Analysis of Deep Random Feature Modelscited by 22
  16. 2019A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learningcited by 20
  17. 2023Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networkscited by 19
  18. 2022On double-descent in uncertainty quantification in overparametrized modelscited by 16
  19. 2021On the interplay between data structure and loss function in classification problemscited by 13
  20. 2022Largest Eigenvalues of the Conjugate Kernel of Single-Layered Neural Networkscited by 13
  21. 2023Beyond the Universal Law of Robustness: Sharper Laws for Random Features and Neural Tangent Kernelscited by 13
  22. 2023Demystifying Disagreement-on-the-Line in High Dimensionscited by 12
  23. 2023How Spurious Features Are Memorized: Precise Analysis for Random and NTK Featurescited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 21; 2025: 14; 2026: 13.

49 papers.

Sort:

  1. 2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 52
  2. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 38
  3. 2024Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functionscited by 37
  4. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  5. 2024Nonlinear spiked covariance matrices and signal propagation in deep neural networkscited by 26
  6. 2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
  7. 2025Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regimecited by 20
  8. 2024Fundamental computational limits of weak learnability in high-dimensional multi-index modelscited by 19
  9. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 16
  10. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 14
  11. 2024Asymptotics of Learning with Deep Structured (Random) Featurescited by 13
  12. 2024High-dimensional learning of narrow neural networkscited by 12
37 more
  1. 2024On Regularization via Early Stopping for Least Squares Regressioncited by 11
  2. 2024Feature learning as alignment: a structural property of gradient descent in non-linear neural networkscited by 10
  3. 2024From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networkscited by 9
  4. 2025Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterizationcited by 9
  5. 2025Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networkscited by 9
  6. 2025On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspectivecited by 8
  7. 2025Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Regioncited by 7
  8. 2025Understanding the Mechanisms of Fast Hyperparameter Transfercited by 7
  9. 2026Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfercited by 6
  10. 2024Towards Understanding the Word Sensitivity of Attention Layers: A Study via Random Featurescited by 5
  11. 2024Robust Feature Learning for Multi-Index Models in High Dimensionscited by 5
  12. 2024Generalization Analysis of Online Stochastic Gradient Descent for Overparameterized Two-Layer Neural Networkscited by 4
  13. 2025Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structurecited by 4
  14. 2025Low Rank Gradients and Where to Find Themcited by 4
  15. 2024Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffscited by 3
  16. 2025Bayes optimal learning of attention-indexed modelscited by 3
  17. 2025A Law of Data Reconstruction for Random Features (and Beyond)cited by 3
  18. 2025Learning Multi-Index Models with Hyper-Kernel Ridge Regressioncited by 3
  19. 2024Signal-Plus-Noise Decomposition of Nonlinear Spiked Random Matrix Modelscited by 2
  20. 2024Input-Label Correlation Governs a Linear-to-Nonlinear Transition in Random Features under Spiked Covariancecited by 2
  21. 2024Generalization for Least Squares Regression With Simple Spiked Covariancescited by 2
  22. 2026A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Modelscited by 2
  23. 2026The Intruder Threshold: A Spectral Law for LoRA Fine-Tuningcited by 2
  24. 2024Learning Hierarchical Polynomials of Multiple Nonlinear Features with Three-Layer Networkscited by 1
  25. 2025Gaussian Equivalence for Self-Attention: Asymptotic Spectral Analysis of Attention Matrixcited by 1
  26. 2025How Data Mixing Shapes In-Context Learning: Asymptotic Equivalence for Transformers with MLPscited by 1
  27. 2026MSign: An Optimizer Preventing Training Instability in Large Language Models via Stable Rank Restorationcited by 1
  28. 2026The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledgecited by 1
  29. 2026How does feature learning reshape the function space?cited by 1
  30. 2024Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theoriescited by 0
  31. 2025One-Bit Quantization for Random Features Modelscited by 0
  32. 2026A Random Matrix Theory of Masked Self-Supervised Regressioncited by 0
  33. 2026Learning Beyond the Gaussian Data: Learning Dynamics of Neural Networks on an Expressive and Cumulant-Controllable Data Modelcited by 0
  34. 2026Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descentcited by 0
  35. 2026Eigen-Spike Emergence and Quadratic Equivalents for Conjugate Kernels on Nonlinearly Separable Datacited by 0
  36. 2026Feature Learning for the High Dimensional Stationary Sch"odinger Equation with Deep Ritz Methodcited by 0
  37. 2026Dimension-Corrected Hitting Times for Heavy-Tailed Spectral Emergence in Neural Optimizer Dynamicscited by 0