# Convergence rates for the stochastic gradient descent method for non-convex objective functions

arXiv: https://arxiv.org/abs/1904.01517 · HTML: https://mlsanity.com/paper/1904.01517/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2019.

## Built on by (most cited first, 30 of 66)

- [What Happens after SGD Reaches Zero Loss? --A Mathematical Framework](https://mlsanity.com/paper/2110.06914/) (2021)
- [Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction](https://mlsanity.com/paper/2206.07085/) (2022)
- [Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models](https://mlsanity.com/paper/2210.14199/) (2022)
- [Tackling the Curse of Dimensionality with Physics-Informed Neural Networks](https://mlsanity.com/paper/2307.12306/) (2023)
- [How Does Sharpness-Aware Minimization Minimize Sharpness?](https://mlsanity.com/paper/2211.05729/) (2022)
- [Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysis](https://mlsanity.com/paper/2105.01650/) (2021)
- [Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalization](https://mlsanity.com/paper/2307.11007/) (2023)
- [Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapes](https://mlsanity.com/paper/2102.09385/) (2021)
- [A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions](https://mlsanity.com/paper/2102.09924/) (2021)
- [Improving Generalization and Convergence by Enhancing Implicit Regularization](https://mlsanity.com/paper/2405.20763/) (2024)
- [Mask in the Mirror: Implicit Sparsification](https://mlsanity.com/paper/2408.09966/) (2024)
- [On minimal representations of shallow ReLU networks](https://mlsanity.com/paper/2108.05643/) (2021)
- [Stationary Behavior of Constant Stepsize SGD Type Algorithms: An Asymptotic Characterization](https://mlsanity.com/paper/2111.06328/) (2021)
- [Landscape analysis for shallow neural networks: complete classification of critical points for affine target functions](https://mlsanity.com/paper/2103.10922/) (2021)
- [Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation](https://mlsanity.com/paper/2107.04479/) (2021)
- [Increasing Both Batch Size and Learning Rate Accelerates Stochastic Gradient Descent](https://mlsanity.com/paper/2409.08770/) (2024)
- [A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functions](https://mlsanity.com/paper/2108.04620/) (2021)
- [A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions](https://mlsanity.com/paper/2104.00277/) (2021)
- [Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activation](https://mlsanity.com/paper/2108.08106/) (2021)
- [Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functions](https://mlsanity.com/paper/2112.07369/) (2021)
- [Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networks](https://mlsanity.com/paper/2402.05155/) (2024)
- [Existence and Estimation of Critical Batch Size for Training Generative Adversarial Networks with Two Time-Scale Update Rule](https://mlsanity.com/paper/2201.11989/) (2022)
- [Exponential convergence rates for momentum stochastic gradient descent in the overparametrized setting](https://mlsanity.com/paper/2302.03550/) (2023)
- [Characterizing Dynamical Stability of Stochastic Gradient Descent in Overparameterized Learning](https://mlsanity.com/paper/2407.20209/) (2024)
- [Singular-limit analysis of gradient descent with noise injection](https://mlsanity.com/paper/2404.12293/) (2024)
- [Iteration and Stochastic First-order Oracle Complexities of Stochastic Gradient Descent using Constant and Decaying Learning Rates](https://mlsanity.com/paper/2402.15344/) (2024)
- [Almost sure convergence rates of stochastic gradient methods under gradient domination](https://mlsanity.com/paper/2405.13592/) (2024)
- [Theoretical analysis of Adam using hyperparameters close to one without Lipschitz smoothness](https://mlsanity.com/paper/2206.13290/) (2022)
- [Scalable Set Encoding with Universal Mini-Batch Consistency and Unbiased Full Set Gradient Approximation](https://mlsanity.com/paper/2208.12401/) (2022)
- [The Fragility of Noise Estimation in Kalman Filter: Optimization Can Handle Model-Misspecification](https://mlsanity.com/paper/2104.02372/) (2021)

## Abstract (arXiv)

We prove the local convergence to minima and estimates on the rate of convergence for the stochastic gradient descent method in the case of not necessarily globally convex nor contracting objective functions. In particular, the results are applicable to simple objective functions arising in machine learning.