mlsanityData to 14 Oct 2026

Global Convergence of SGD On Two Layer Neural Nets

arXiv 2210.11452 ↗First posted 20 Oct 2022cs.LG math.OC stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

In this note, we consider appropriately regularized ℓ2-empirical risk of depth 2 nets with any number of gates and show bounds on how the empirical loss evolves for SGD iterates on it -- for arbitrary data and if the activation is adequately smooth and bounded like sigmoid and tanh. This in turn leads to a proof of global convergence of SGD for a special class of initializations. We also prove an exponentially fast convergence rate for continuous time SGD that also applies to smooth unbounded activations like SoftPlus. Our key idea is to show the existence of Frobenius norm regularized loss functions on constant-sized neural nets which are "Villani functions" and thus be able to build on recent progress with analyzing SGD on such objectives. Most critically the amount of regularization required for our analysis is independent of the size of the net.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 57 of its 71 references that are in the map. First posted on arXiv 20 Oct 2022.

Builds on

57 of 71 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  5. 2018On Lazy Training in Differentiable Programmingcited by 675
  6. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  7. 2017Deep Neural Networks as Gaussian Processescited by 622
  8. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  9. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  10. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  11. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  12. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
38 more
  1. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  2. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 243
  3. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  4. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
  5. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  6. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  7. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 121
  8. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 121
  9. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 115
  10. 2020Optimal Regularization Can Mitigate Double Descentcited by 109
  11. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  12. 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
  13. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  14. 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
  15. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  16. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  17. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  18. 2018On the Convergence Rate of Training Recurrent Neural Networkscited by 82
  19. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  20. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  21. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  22. 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
  23. 2022Mean-Field Langevin Dynamics: Exponential Convergence and Annealingcited by 61
  24. 2020Superpolynomial Lower Bounds for Learning One-Layer Neural Networks using Gradient Descentcited by 58
  25. 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
  26. 2020Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networkscited by 44
  27. 2019Mean Field Analysis of Deep Neural Networkscited by 42
  28. 2019The Local Elasticity of Neural Networkscited by 37
  29. 2020On Learning Rates and Schr"odinger Operatorscited by 33
  30. 2015Randomized sketches for kernels: Fast and optimal non-parametric regressioncited by 31
  31. 2019Gradient Descent Finds Global Minima for Generalizable Deep Neural Networks of Practical Sizescited by 29
  32. 2019On Learning Over-parameterized Neural Networks: A Functional Approximation Perspectivecited by 28
  33. 2019Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Networkcited by 27
  34. 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
  35. 2022Convergence of gradient descent for deep neural networkscited by 24
  36. 2018Learning Two-layer Neural Networks with Symmetric Inputscited by 22
  37. 2021Efficient Algorithms for Learning Depth-2 Neural Networks with General ReLU Activationscited by 15
  38. 2018Theory of Deep Learning IIb: Optimization Properties of SGDcited by 12

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 2; 2026: 2.

5 papers.

  1. 2023Training a Two Layer ReLU Network Analyticallycited by 1
  2. 2025Langevin Monte-Carlo Provably Learns Depth Two Neural Nets at Any Size and Datacited by 1
  3. 2026Convergent Stochastic Training of Multi-Headed Attention and Understanding LoRAcited by 1
  4. 2025Gradient Flow Convergence Guarantee for General Neural Network Architecturescited by 0
  5. 2026Emergent Low-Rank Training Dynamics in MLPs with Smooth Activationscited by 0