# Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapes arXiv: https://arxiv.org/abs/2102.09385 ยท HTML: https://mlsanity.com/paper/2102.09385/ ## Key facts - Cited by 28 papers in the map (data through 14 Oct 2026). - Builds on 23 of its 74 references that are in the map. - First posted on arXiv 16 Feb 2021. ## Builds on (most cited first, 23 of 23) - [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018) - [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017) - [Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Condition](https://mlsanity.com/paper/1608.04636/) (2016) - [Implicit Regularization in Deep Matrix Factorization](https://mlsanity.com/paper/1905.13655/) (2019) - [Implicit Regularization in Matrix Factorization](https://mlsanity.com/paper/1705.09280/) (2017) - [Kernel and Rich Regimes in Overparametrized Models](https://mlsanity.com/paper/2002.09277/) (2020) - [A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks](https://mlsanity.com/paper/1901.06053/) (2019) - [Optimal approximation of piecewise smooth functions using deep ReLU neural networks](https://mlsanity.com/paper/1709.05289/) (2017) - [Better Theory for SGD in the Nonconvex World](https://mlsanity.com/paper/2002.03329/) (2020) - [Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptron](https://mlsanity.com/paper/1810.07288/) (2018) - [The Heavy-Tail Phenomenon in SGD](https://mlsanity.com/paper/2006.04740/) (2020) - [Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)](https://mlsanity.com/paper/1306.2119/) (2013) - [On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problems](https://mlsanity.com/paper/2006.11144/) (2020) - [SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolation](https://mlsanity.com/paper/2006.10311/) (2020) - [Convergence rates for the stochastic gradient descent method for non-convex objective functions](https://mlsanity.com/paper/1904.01517/) (2019) - [On Connected Sublevel Sets in Deep Learning](https://mlsanity.com/paper/1901.07417/) (2019) - [Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptions](https://mlsanity.com/paper/1902.00908/) (2019) - [Non-asymptotic Analysis of Biased Stochastic Approximation Scheme](https://mlsanity.com/paper/1902.00629/) (2019) - [On exponential convergence of SGD in non-convex over-parametrized learning](https://mlsanity.com/paper/1811.02564/) (2018) - [Linear Convergence of Adaptive Stochastic Gradient Descent](https://mlsanity.com/paper/1908.10525/) (2019) - [Non-convergence of stochastic gradient descent in the training of deep neural networks](https://mlsanity.com/paper/2006.07075/) (2020) - [Overall error analysis for the training of deep neural networks via stochastic gradient descent with random initialisation](https://mlsanity.com/paper/2003.01291/) (2020) - [Reducing Parameter Space for Neural Network Training](https://mlsanity.com/paper/1805.08340/) (2018) ## Built on by (most cited first, 28 of 28) - [Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysis](https://mlsanity.com/paper/2105.01650/) (2021) - [Convergence rates for the Adam optimizer](https://mlsanity.com/paper/2407.21078/) (2024) - [Mask in the Mirror: Implicit Sparsification](https://mlsanity.com/paper/2408.09966/) (2024) - [On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks](https://mlsanity.com/paper/2112.09684/) (2021) - [A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functions](https://mlsanity.com/paper/2108.04620/) (2021) - [A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions](https://mlsanity.com/paper/2104.00277/) (2021) - [Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activation](https://mlsanity.com/paper/2108.08106/) (2021) - [Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functions](https://mlsanity.com/paper/2112.07369/) (2021) - [Exponential convergence rates for momentum stochastic gradient descent in the overparametrized setting](https://mlsanity.com/paper/2302.03550/) (2023) - [Sharp higher order convergence rates for the Adam optimizer](https://mlsanity.com/paper/2504.19426/) (2025) - [Singular-limit analysis of gradient descent with noise injection](https://mlsanity.com/paper/2404.12293/) (2024) - [On the existence of minimizers in shallow residual ReLU neural network optimization landscapes](https://mlsanity.com/paper/2302.14690/) (2023) - [On the existence of optimal shallow feedforward networks with ReLU activation](https://mlsanity.com/paper/2303.03950/) (2023) - [A Normal Map-Based Proximal Stochastic Gradient Method: Convergence and Identification Properties](https://mlsanity.com/paper/2305.05828/) (2023) - [Almost sure convergence rates of stochastic gradient methods under gradient domination](https://mlsanity.com/paper/2405.13592/) (2024) - [SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures](https://mlsanity.com/paper/2505.09572/) (2025) - [A New Random Reshuffling Method for Nonsmooth Nonconvex Finite-sum Optimization](https://mlsanity.com/paper/2312.01047/) (2023) - [Polyak's Heavy Ball Method Achieves Accelerated Local Rate of Convergence under Polyak-Lojasiewicz Inequality](https://mlsanity.com/paper/2410.16849/) (2024) - [Controlling the Flow: Stability and Convergence for Stochastic Gradient Descent with Decaying Regularization](https://mlsanity.com/paper/2505.11434/) (2025) - [A Retrospective Approximation Approach for Smooth Stochastic Optimization](https://mlsanity.com/paper/2103.04392/) (2021) - [Training Dynamics of Learning 3D-Rotational Equivariance](https://mlsanity.com/paper/2512.02303/) (2025) - [Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks](https://mlsanity.com/paper/2607.04233/) (2026) - [A Local Convergence Theory for the Stochastic Gradient Descent Method in Non-Convex Optimization With Non-isolated Local Minima](https://mlsanity.com/paper/2203.10973/) (2022) - [On bounds for norms of reparameterized ReLU artificial neural network parameters: sums of fractional powers of the Lipschitz norm control the network parameter vector](https://mlsanity.com/paper/2206.13646/) (2022) - [Stochastic Gradient Descent Revisited](https://mlsanity.com/paper/2412.06070/) (2024) - [Momentum-based minimization of the Ginzburg-Landau functional on Euclidean spaces and graphs](https://mlsanity.com/paper/2501.00389/) (2024) - [In almost all shallow analytic neural network optimization landscapes, efficient minimizers have strongly convex neighborhoods](https://mlsanity.com/paper/2504.08867/) (2025) - [The Interplay of Statistics and Noisy Optimization: Learning Linear Predictors with Random Data Weights](https://mlsanity.com/paper/2512.10188/) (2025) ## Abstract (arXiv) In this article, we consider convergence of stochastic gradient descent schemes (SGD), including momentum stochastic gradient descent (MSGD), under weak assumptions on the underlying landscape. More explicitly, we show that on the event that the SGD stays bounded we have convergence of the SGD if there is only a countable number of critical points or if the objective function satisfies Lojasiewicz-inequalities around all critical levels as all analytic functions do. In particular, we show that for neural networks with analytic activation function such as softplus, sigmoid and the hyperbolic tangent, SGD converges on the event of staying bounded, if the random variables modelling the signal and response in the training are compactly supported.