Convergence of stochastic gradient descent under a local Lojasiewicz condition for deep neural networks
Not on the map. Why?
Abstract (arXiv)
We study the convergence of stochastic gradient descent (SGD) for non-convex objective functions. We establish the local convergence with positive probability under the local Lojasiewicz condition introduced by Chatterjee in citechatterjee2022convergence and an additional local structural assumption of the loss function landscape. A key component of our proof is to ensure that the whole trajectories of SGD stay inside the local region with a positive probability. We also provide examples of neural networks with finite widths such that our assumptions hold.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 27 of its 35 references that are in the map. First posted on arXiv 18 Apr 2023.
Builds on
27 of 35 references.
References · 27
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
- 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
- 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
- 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
- 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
- 2019SGD: General Analysis and Improved Ratescited by 207
- 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
- 2016Stochastic Variance Reduction for Nonconvex Optimizationcited by 145
- 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
15 more
- 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 120
- 2019Unified Optimal Analysis of the (Stochastic) Gradient Methodcited by 82
- 2020On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problemscited by 80
- 2013Accelerated Gradient Methods for Nonconvex Nonlinear and Stochastic Programmingcited by 75
- 2020SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolationcited by 67
- 2019Convergence rates for the stochastic gradient descent method for non-convex objective functionscited by 66
- 2017Non-convex Finite-Sum Optimization Via SCSG Methodscited by 61
- 2018How To Make the Gradients Small Stochastically: Even Faster Convex and Nonconvex SGDcited by 57
- 2017Natasha 2: Faster Non-Convex Optimization Than SGDcited by 49
- 2017Stochastic Methods for Composite and Weakly Convex Optimization Problemscited by 48
- 2015Incremental Gradient, Subgradient, and Proximal Methods for Convex Optimization: A Surveycited by 45
- 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 39
- 2022Convergence of gradient descent for deep neural networkscited by 24
- 2021On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networkscited by 12
- 2022From Gradient Flow on Population Loss to Learning with Stochastic Gradient Descentcited by 5
Built on by
Citing papers per year: 2024: 2; 2025: 1.
3 papers.