mlsanityData to 14 Oct 2026

Provable Guarantees for Neural Networks via Gradient Feature Learning

arXiv 2310.12408 ↗First posted 19 Oct 2023cs.LG cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

Neural networks have achieved remarkable empirical performance, while the current theoretical analysis is not adequate for understanding their success, e.g., the Neural Tangent Kernel approach fails to capture their key feature learning ability, while recent analyses on feature learning are typically problem-specific. This work proposes a unified analysis framework for two-layer networks trained by gradient descent. The framework is centered around the principle of feature learning from gradients, and its effectiveness is demonstrated by applications in several prototypical problems, such as mixtures of Gaussians and parity functions. The framework also sheds light on interesting network learning phenomena such as feature learning beyond kernels and the lottery ticket hypothesis.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 63 of its 124 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

63 of 124 references.

References · 63

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  5. 2018On Lazy Training in Differentiable Programmingcited by 675
  6. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  7. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  8. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
  9. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
  10. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  11. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  12. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
38 more
  1. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  2. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  3. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  4. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
  5. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 142
  6. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
  7. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 115
  8. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  9. 2019Are All Layers Created Equal?cited by 94
  10. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  11. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  12. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 91
  13. 2017Implicit Regularization in Deep Learningcited by 90
  14. 2018Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?cited by 88
  15. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  16. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 83
  17. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  18. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 78
  19. 2019Nonparametric Regression on Low-Dimensional Manifolds using Deep ReLU Networks : Function Approximation and Statistical Recoverycited by 73
  20. 2019Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobiancited by 67
  21. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 67
  22. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 63
  23. 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
  24. 2022A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Featurescited by 49
  25. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 49
  26. 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
  27. 2020Towards Understanding Hierarchical Learning: Benefits of Neural Representationscited by 46
  28. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  29. 2021Quantifying the Benefit of Using Differentiable Learning over Tangent Kernelscited by 41
  30. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  31. 2019Stabilizing the Lottery Ticket Hypothesiscited by 37
  32. 2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 35
  33. 2020The Hidden Convex Optimization Landscape of Two-Layer ReLU Neural Networks: an Exact Characterization of the Optimal Solutionscited by 30
  34. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  35. 2020Approximate is Good Enough: Probabilistic Variants of Dimensional and Margin Complexitycited by 25
  36. 2022Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn featurescited by 25
  37. 2021Overparameterization of deep ResNet: zero loss and mean-field analysiscited by 24
  38. 2023The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networkscited by 23

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 4; 2026: 2.

14 papers.

Sort:

By area:

  1. 2024Why Larger Language Models Do In-context Learning Differently?cited by 45
  2. 2024Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuningcited by 31
  3. 2024Unraveling the Smoothness Properties of Diffusion Models: A Gaussian Mixture Perspectivecited by 24
  4. 2024BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Modelcited by 21
  5. 2024Looped ReLU MLPs May Be All You Need as Practical Programmable Computerscited by 19
  6. 2024Differential Privacy Mechanisms in Neural Tangent Kernel Regressioncited by 14
  7. 2025Theoretical Guarantees for High Order Trajectory Refinement in Generative Flowscited by 10
  8. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Expertscited by 2
  9. 2025Unifying Learning Dynamics and Generalization in Transformers Scaling Lawcited by 2
  10. 2024Robust Fourier Neural Networkscited by 0
  11. 2024Sequencing the Neurome: Towards Scalable Exact Parameter Reconstruction of Black-Box Neural Networkscited by 0
  12. 2025Theoretical Foundations of Representation Learning using Unlabeled Data: Statistics and Optimizationcited by 0
2 more
  1. 2026Neuron Block Dynamics for XOR Classification with Zero-Margincited by 0
  2. 2026Training Dynamics of Transformers on Unbounded Inputs: Attention Learning Can Lag Output Layer Learningcited by 0