Improved Overparametrization Bounds for Global Convergence of Stochastic Gradient Descent for Shallow Neural Networks
Not on the map. Why?
Abstract (arXiv)
We study the overparametrization bounds required for the global convergence of stochastic gradient descent algorithm for a class of one hidden layer feed-forward neural networks, considering most of the activation functions used in practice, including ReLU. We improve the existing state-of-the-art results in terms of the required hidden layer width. We introduce a new proof technique combining nonlinear analysis with properties of random initializations of the network. First, we establish the global convergence of continuous solutions of the differential inclusion being a nonsmooth analogue of the gradient flow for the MSE loss. Second, we provide a technical result (working also for general approximators) relating solutions of the aforementioned differential inclusion to the (discrete) stochastic gradient descent sequences, hence establishing linear convergence towards zero loss for the stochastic gradient descent iterations.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 30 of its 48 references that are in the map. First posted on arXiv 28 Jan 2022.
Builds on
30 of 48 references.
References · 30
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2016Understanding deep learning requires rethinking generalizationcited by 1,657
- 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
- 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
- 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
- 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
- 2014The Loss Surfaces of Multilayer Networkscited by 278
- 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
- 2016Deep Learning without Poor Local Minimacited by 233
- 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
18 more
- 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
- 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
- 2018Stochastic subgradient method converges on tame functionscited by 120
- 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
- 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
- 2016Identity Matters in Deep Learningcited by 107
- 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
- 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
- 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
- 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
- 2018Small ReLU networks are powerful memorizers: a tight analysis of memorization capacitycited by 76
- 2015Provable approximation properties for deep neural networkscited by 60
- 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
- 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
- 2016Diverse Neural Network Learns True Target Functionscited by 33
- 2019Gradient Descent Finds Global Minima for Generalizable Deep Neural Networks of Practical Sizescited by 29
- 2020Convergence of constant step stochastic gradient descent for non-smooth non-convex functionscited by 27
- 2019The capacity of feedforward neural networkscited by 14
Built on by
Citing papers per year: 2022: 1.
1 paper.