mlsanityData to 7 Oct 2026

A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Network

arXiv 2102.02410 ↗First posted 4 Feb 2021cs.LG stat.MLcited by 47

Not on the map. Why?

Abstract (arXiv)

While over-parameterization is widely believed to be crucial for the success of optimization for the neural networks, most existing theories on over-parameterization do not fully explain the reason -- they either work in the Neural Tangent Kernel regime where neurons don't move much, or require an enormous number of neurons. In practice, when the data is generated using a teacher neural network, even mildly over-parameterized neural networks can achieve 0 loss and recover the directions of teacher neurons. In this paper we develop a local convergence theory for mildly over-parameterized two-layer neural net. We show that as long as the loss is already lower than a threshold (polynomial in relevant parameters), all student neurons in an over-parameterized two-layer neural network will converge to one of teacher neurons, and the loss will go to 0. Our result holds for any number of student neurons as long as it is at least as large as the number of teacher neurons, and our convergence rate is independent of the number of student neurons. A key component of our analysis is the new characterization of local optimization landscape -- we show the gradient satisfies a special case of Lojasiewicz property which is different from local strong convexity or PL conditions used in previous work.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 31 of its 41 references that are in the map. First posted on arXiv 4 Feb 2021.

Builds on

31 of 41 references.

References · 31

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  7. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  8. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  9. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  10. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  11. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  12. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
19 more
  1. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  2. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  3. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  4. 2019Linearized two-layers neural networks in high dimensioncited by 155
  5. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  6. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  7. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  8. 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
  9. 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
  10. 2020A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depthcited by 56
  11. 2017Stochastic Particle Gradient Descent for Infinite Ensemblescited by 52
  12. 2018Learning One-hidden-layer ReLU Networks via Gradient Descentcited by 50
  13. 2019Sparse Optimization on Measures with Over-parameterized Gradient Descentcited by 50
  14. 2020Algorithms and SQ Lower Bounds for PAC Learning One-Hidden-Layer ReLU Networkscited by 49
  15. 2020Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networkscited by 44
  16. 2020The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networkscited by 33
  17. 2018Learning Two-layer Neural Networks with Symmetric Inputscited by 22
  18. 2018Learning One Convolutional Layer with Overlapping Patchescited by 21
  19. 2020Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTKcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 15; 2023: 10; 2024: 4; 2025: 8; 2026: 3.

47 papers.

Sort:

  1. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  2. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  3. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  4. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  5. 2022A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Featurescited by 49
  6. 2023On the Optimization and Generalization of Multi-head Attentioncited by 37
  7. 2023Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuroncited by 25
  8. 2021Parameter identifiability of a deep feedforward ReLU neural networkcited by 24
  9. 2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21
  10. 2022Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization)cited by 18
  11. 2021On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Settingcited by 16
  12. 2021Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalizationcited by 15
35 more
  1. 2023Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networkscited by 15
  2. 2024Differential Privacy Mechanisms in Neural Tangent Kernel Regressioncited by 14
  3. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 13
  4. 2022On Feature Learning in Neural Networks with Global Convergence Guaranteescited by 12
  5. 2021The Convex Geometry of Backpropagation: Neural Network Gradient Flows Converge to Extreme Points of the Dual Convex Programcited by 11
  6. 2024Convex Relaxations of ReLU Neural Networks Approximate Global Optima in Polynomial Timecited by 11
  7. 2022Excess Risk of Two-Layer ReLU Neural Networks in Teacher-Student Settings and its Superiority to Kernel Methodscited by 10
  8. 2023Efficient Uncertainty Quantification and Reduction for Over-Parameterized Neural Networkscited by 10
  9. 2021Sparse Bayesian Deep Learning for Dynamic System Identificationcited by 8
  10. 2022When Expressivity Meets Trainability: Fewer than n Neurons Can Workcited by 8
  11. 2023Depth Separation with Multilayer Mean-Field Networkscited by 8
  12. 2021Deep Networks Provably Classify Data on Curvescited by 6
  13. 2022Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networkscited by 5
  14. 2022Global Convergence of SGD On Two Layer Neural Netscited by 5
  15. 2022Finite Sample Identification of Wide Shallow Neural Networks with Biasescited by 5
  16. 2024"Yes, My LoRD." Guiding Language Model Extraction with Locality Reinforced Distillationcited by 5
  17. 2022Optimizing the Performative Risk under Weak Convexity Assumptionscited by 4
  18. 2023Toward L_∞-recovery of Nonlinear Functions: A Polynomial Sample Complexity Bound for Gaussian Random Fieldscited by 4
  19. 2024How Does Gradient Descent Learn Features -- A Local Analysis for Regularized Two-Layer Neural Networkscited by 4
  20. 2022Intersection of Parallels as an Early Stopping Criterioncited by 3
  21. 2023Global Convergence of SGD For Logistic Loss on Two Layer Neural Netscited by 3
  22. 2025Convergence of Shallow ReLU Networks on Weakly Interacting Datacited by 2
  23. 2025Deep Learning Optimization Using Self-Adaptive Weighted Auxiliary Variablescited by 2
  24. 2022Parameter Convex Neural Networkscited by 1
  25. 2023SAfER: Layer-Level Sensitivity Assessment for Efficient and Robust Neural Network Inferencecited by 1
  26. 2023Convergence Analysis for Learning Orthonormal Deep Linear Neural Networkscited by 1
  27. 2025Curse of Dimensionality in Neural Network Optimizationcited by 1
  28. 2025Langevin Monte-Carlo Provably Learns Depth Two Neural Nets at Any Size and Datacited by 1
  29. 2025From Sublinear to Linear: Local Convergence in Finite-Width Networks via Locally Polyak-Lojasiewicz Regionscited by 1
  30. 2025Graph Coloring for Multi-Task Learningcited by 1
  31. 2025Unifying Learning Dynamics and Generalization in Transformers Scaling Lawcited by 1
  32. 2026Hidden Gauge Controls Feature Specialization in ReLU Networkscited by 1
  33. 2025A Derandomization Framework for Structure Discovery: Applications in Neural Networks and Beyondcited by 0
  34. 2026A Layer Separation Optimization Framework for Cross-Entropy Training in Deep Learningcited by 0
  35. 2026Removing spurious minima for planar features by skip connectionscited by 0