mlsanityData to 14 Oct 2026

A Local Convergence Theory for the Stochastic Gradient Descent Method in Non-Convex Optimization With Non-isolated Local Minima

arXiv 2203.10973 ↗First posted 21 Mar 2022cs.LG cs.NA math.NAcited by 0

Not on the map. Why?

Abstract (arXiv)

Loss functions with non-isolated minima have emerged in several machine learning problems, creating a gap between theory and practice. In this paper, we formulate a new type of local convexity condition that is suitable to describe the behavior of loss functions near non-isolated minima. We show that such condition is general enough to encompass many existing conditions. In addition we study the local convergence of the SGD under this mild condition by adopting the notion of stochastic stability. The corresponding concentration inequalities from the convergence analysis help to interpret the empirical observation from some practical training results.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 44 references that are in the map. First posted on arXiv 21 Mar 2022.

Builds on

24 of 44 references.

References · 24

  1. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  2. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  3. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  4. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  5. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  6. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
  7. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  8. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  9. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
  10. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
  11. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 156
  12. 2020The Heavy-Tail Phenomenon in SGDcited by 124
12 more
  1. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 120
  2. 2019Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronizationcited by 107
  3. 2020On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problemscited by 80
  4. 2019Optimization for deep learning: theory and algorithmscited by 70
  5. 2019Convergence rates for the stochastic gradient descent method for non-convex objective functionscited by 66
  6. 2019Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptionscited by 48
  7. 2019Near-Optimal Methods for Minimizing Star-Convex Functions and Beyondcited by 36
  8. 2018The Goldilocks zone: Towards better understanding of neural network loss landscapescited by 32
  9. 2021Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapescited by 28
  10. 2013Gradient methods for convex minimization: better rates under weaker conditionscited by 23
  11. 2021Global Convergence and Stability of Stochastic Gradient Descentcited by 17
  12. 2018On the Benefit of Width for Neural Networks: Disappearance of Bad Basinscited by 16

Built on by

Not cited yet.