mlsanityData to 14 Oct 2026

Convergence rates for the stochastic gradient descent method for non-convex objective functions

arXiv 1904.01517 ↗First posted 2 Apr 2019math.NA cs.LG cs.NAcited by 66

Not on the map. Why?

Abstract (arXiv)

We prove the local convergence to minima and estimates on the rate of convergence for the stochastic gradient descent method in the case of not necessarily globally convex nor contracting objective functions. In particular, the results are applicable to simple objective functions arising in machine learning.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). First posted on arXiv 2 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 17; 2022: 10; 2023: 13; 2024: 17; 2025: 5; 2026: 4.

66 papers.

Sort:

By area:

  1. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  2. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  3. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  4. 2023Tackling the Curse of Dimensionality with Physics-Informed Neural Networkscited by 79
  5. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 65
  6. 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 39
  7. 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
  8. 2021Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapescited by 28
  9. 2021A proof of convergence for gradient descent in the training of artificial neural networks for constant target functionscited by 14
  10. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  11. 2024Mask in the Mirror: Implicit Sparsificationcited by 14
  12. 2021On minimal representations of shallow ReLU networkscited by 13
38 more
  1. 2021Stationary Behavior of Constant Stepsize SGD Type Algorithms: An Asymptotic Characterizationcited by 13
  2. 2021Landscape analysis for shallow neural networks: complete classification of critical points for affine target functionscited by 11
  3. 2021Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activationcited by 11
  4. 2024Increasing Both Batch Size and Learning Rate Accelerates Stochastic Gradient Descentcited by 11
  5. 2021A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functionscited by 10
  6. 2021A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functionscited by 9
  7. 2021Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activationcited by 9
  8. 2021Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functionscited by 9
  9. 2024Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networkscited by 9
  10. 2022Existence and Estimation of Critical Batch Size for Training Generative Adversarial Networks with Two Time-Scale Update Rulecited by 8
  11. 2023Exponential convergence rates for momentum stochastic gradient descent in the overparametrized settingcited by 8
  12. 2024Characterizing Dynamical Stability of Stochastic Gradient Descent in Overparameterized Learningcited by 8
  13. 2024Singular-limit analysis of gradient descent with noise injectioncited by 7
  14. 2024Iteration and Stochastic First-order Oracle Complexities of Stochastic Gradient Descent using Constant and Decaying Learning Ratescited by 5
  15. 2024Almost sure convergence rates of stochastic gradient methods under gradient dominationcited by 5
  16. 2022Theoretical analysis of Adam using hyperparameters close to one without Lipschitz smoothnesscited by 4
  17. 2022Scalable Set Encoding with Universal Mini-Batch Consistency and Unbiased Full Set Gradient Approximationcited by 4
  18. 2021The Fragility of Noise Estimation in Kalman Filter: Optimization Can Handle Model-Misspecificationcited by 3
  19. 2023Convergence of stochastic gradient descent under a local Lojasiewicz condition for deep neural networkscited by 3
  20. 2023Deep Stochastic Mechanicscited by 3
  21. 2023Gradient is All You Need? How Consensus-Based Optimization can be Interpreted as a Stochastic Relaxation of Gradient Descentcited by 3
  22. 2023Learning Unorthogonalized Matrices for Rotation Estimationcited by 3
  23. 2024Federated reinforcement learning for robot motion planning with zero-shot generalizationcited by 3
  24. 2024Optimal time sampling in physics-informed neural networkscited by 3
  25. 2024Polyak's Heavy Ball Method Achieves Accelerated Local Rate of Convergence under Polyak-Lojasiewicz Inequalitycited by 3
  26. 2025A Dynamical Systems Perspective on the Analysis of Neural Networkscited by 3
  27. 2021Exact Recovery of Community Structures Using DeepWalk and Node2veccited by 2
  28. 2023On the Stability and Generalization of Triplet Learningcited by 2
  29. 2023Demystifying the Myths and Legends of Nonconvex Convergence of SGDcited by 2
  30. 2023Using Stochastic Gradient Descent to Smooth Nonconvex Functions: Analysis of Implicit Graduated Optimizationcited by 2
  31. 2024Stochastic Modified Flows for Riemannian Stochastic Gradient Descentcited by 2
  32. 2024Convergence of a L2 regularized Policy Gradient Algorithm for the Multi Armed Banditcited by 2
  33. 2025Dropout Neural Network Training Viewed from a Percolation Perspectivecited by 2
  34. 2021The Number of Steps Needed for Nonconvex Optimization of a Deep Learning Optimizer is a Rational Function of Batch Sizecited by 1
  35. 2022Critical Bach Size Minimizes Stochastic First-Order Oracle Complexity of Deep Learning Optimizer using Hyperparameters Close to Onecited by 1
  36. 2022Convergence analysis of a quasi-Monte Carlo-based deep learning algorithm for solving partial differential equationscited by 1
  37. 2023Spatial Deep Learning for Site-Specific Movement Optimization of Aerial Base Stationscited by 1
  38. 2024Learn What You Need in Personalized Federated Learningcited by 1

50 most cited of 66 All 66 (JSON)