mlsanityData to 7 Oct 2026

Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapes

arXiv 2102.09385 ↗First posted 16 Feb 2021cs.LG math.PR math.STcited by 27

Not on the map. Why?

Abstract (arXiv)

In this article, we consider convergence of stochastic gradient descent schemes (SGD), including momentum stochastic gradient descent (MSGD), under weak assumptions on the underlying landscape. More explicitly, we show that on the event that the SGD stays bounded we have convergence of the SGD if there is only a countable number of critical points or if the objective function satisfies Lojasiewicz-inequalities around all critical levels as all analytic functions do. In particular, we show that for neural networks with analytic activation function such as softplus, sigmoid and the hyperbolic tangent, SGD converges on the event of staying bounded, if the random variables modelling the signal and response in the training are compactly supported.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 23 of its 74 references that are in the map. First posted on arXiv 16 Feb 2021.

Builds on

23 of 74 references.

References · 23

  1. 2018On Lazy Training in Differentiable Programmingcited by 671
  2. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  3. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  4. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  5. 2017Implicit Regularization in Matrix Factorizationcited by 318
  6. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  7. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 213
  8. 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 176
  9. 2020Better Theory for SGD in the Nonconvex Worldcited by 156
  10. 2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
  11. 2020The Heavy-Tail Phenomenon in SGDcited by 122
  12. 2013Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)cited by 110
11 more
  1. 2020On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problemscited by 79
  2. 2019Convergence rates for the stochastic gradient descent method for non-convex objective functionscited by 66
  3. 2020SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolationcited by 66
  4. 2019On Connected Sublevel Sets in Deep Learningcited by 51
  5. 2019Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptionscited by 48
  6. 2019Non-asymptotic Analysis of Biased Stochastic Approximation Schemecited by 45
  7. 2018On exponential convergence of SGD in non-convex over-parametrized learningcited by 33
  8. 2019Linear Convergence of Adaptive Stochastic Gradient Descentcited by 29
  9. 2020Non-convergence of stochastic gradient descent in the training of deep neural networkscited by 14
  10. 2020Overall error analysis for the training of deep neural networks via stochastic gradient descent with random initialisationcited by 7
  11. 2018Reducing Parameter Space for Neural Network Trainingcited by 2

Built on by

Cited by year

Citing papers per year: 2021: 7; 2022: 2; 2023: 5; 2024: 7; 2025: 6.

27 papers.

Sort:

  1. 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 38
  2. 2024Convergence rates for the Adam optimizercited by 14
  3. 2021On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networkscited by 12
  4. 2024Mask in the Mirror: Implicit Sparsificationcited by 12
  5. 2021A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functionscited by 10
  6. 2021A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functionscited by 9
  7. 2021Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activationcited by 9
  8. 2021Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functionscited by 9
  9. 2023Exponential convergence rates for momentum stochastic gradient descent in the overparametrized settingcited by 7
  10. 2025Sharp higher order convergence rates for the Adam optimizercited by 7
  11. 2023On the existence of minimizers in shallow residual ReLU neural network optimization landscapescited by 6
  12. 2023On the existence of optimal shallow feedforward networks with ReLU activationcited by 5
15 more
  1. 2023A Normal Map-Based Proximal Stochastic Gradient Method: Convergence and Identification Propertiescited by 5
  2. 2024Singular-limit analysis of gradient descent with noise injectioncited by 5
  3. 2024Almost sure convergence rates of stochastic gradient methods under gradient dominationcited by 5
  4. 2025SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structurescited by 5
  5. 2023A New Random Reshuffling Method for Nonsmooth Nonconvex Finite-sum Optimizationcited by 4
  6. 2025Controlling the Flow: Stability and Convergence for Stochastic Gradient Descent with Decaying Regularizationcited by 3
  7. 2024Polyak's Heavy Ball Method Achieves Accelerated Local Rate of Convergence under Polyak-Lojasiewicz Inequalitycited by 2
  8. 2021A Retrospective Approximation Approach for Smooth Stochastic Optimizationcited by 1
  9. 2025Training Dynamics of Learning 3D-Rotational Equivariancecited by 1
  10. 2022A Local Convergence Theory for the Stochastic Gradient Descent Method in Non-Convex Optimization With Non-isolated Local Minimacited by 0
  11. 2022On bounds for norms of reparameterized ReLU artificial neural network parameters: sums of fractional powers of the Lipschitz norm control the network parameter vectorcited by 0
  12. 2024Stochastic Gradient Descent Revisitedcited by 0
  13. 2024Momentum-based minimization of the Ginzburg-Landau functional on Euclidean spaces and graphscited by 0
  14. 2025In almost all shallow analytic neural network optimization landscapes, efficient minimizers have strongly convex neighborhoodscited by 0
  15. 2025The Interplay of Statistics and Noisy Optimization: Learning Linear Predictors with Random Data Weightscited by 0