Global Convergence of SGD For Logistic Loss on Two Layer Neural Nets
Not on the map. Why?
Abstract (arXiv)
In this note, we demonstrate a first-of-its-kind provable convergence of SGD to the global minima of appropriately regularized logistic empirical risk of depth 2 nets -- for arbitrary data and with any number of gates with adequately smooth and bounded activations like sigmoid and tanh. We also prove an exponentially fast convergence rate for continuous time SGD that also applies to smooth unbounded activations like SoftPlus. Our key idea is to show the existence of Frobenius norm regularized logistic loss functions on constant-sized neural nets which are "Villani functions" and thus be able to build on recent progress with analyzing SGD on such objectives.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 40 of its 49 references that are in the map. First posted on arXiv 17 Sep 2023.
Builds on
40 of 49 references.
References · 40
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
- 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
- 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
- 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
- 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
- 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
- 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
- 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
28 more
- 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
- 2020Optimal Regularization Can Mitigate Double Descentcited by 107
- 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
- 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
- 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
- 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
- 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
- 2022Mean-Field Langevin Dynamics: Exponential Convergence and Annealingcited by 60
- 2020Superpolynomial Lower Bounds for Learning One-Layer Neural Networks using Gradient Descentcited by 57
- 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
- 2020Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networkscited by 44
- 2019Mean Field Analysis of Deep Neural Networkscited by 42
- 2019The Local Elasticity of Neural Networkscited by 37
- 2020On Learning Rates and Schr"odinger Operatorscited by 33
- 2015Randomized sketches for kernels: Fast and optimal non-parametric regressioncited by 31
- 2019On Learning Over-parameterized Neural Networks: A Functional Approximation Perspectivecited by 28
- 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
- 2018Learning Two-layer Neural Networks with Symmetric Inputscited by 22
- 2021Efficient Algorithms for Learning Depth-2 Neural Networks with General ReLU Activationscited by 15
- 2018Theory of Deep Learning IIb: Optimization Properties of SGDcited by 12
- 2022Entropic Fictitious Play for Mean Field Optimization Problemcited by 12
- 2021Edge of chaos as a guiding principle for modern neural network trainingcited by 10
- 2021When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?cited by 9
- 2020Depth-2 Neural Networks Under a Data-Poisoning Attackcited by 6
- 2021Efficiently Learning Any One Hidden Layer ReLU Network From Queriescited by 6
- 2020Function approximation by neural nets in the mean-field regime: Entropic regularization and controlled McKean-Vlasov dynamicscited by 5
- 2022Gradient descent provably escapes saddle points in the training of shallow ReLU networkscited by 5
- 2023Classification of Data Generated by Gaussian Mixture Models Using Deep ReLU Networkscited by 4
Built on by
Citing papers per year: 2024: 1; 2025: 1; 2026: 1.
3 papers.