mlsanityData to 7 Oct 2026

Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptions

arXiv 1902.00908 ↗First posted 3 Feb 2019cs.LG math.OC stat.MLcited by 48

Not on the map. Why?

Abstract (arXiv)

Stochastic gradient descent (SGD) is a popular and efficient method with wide applications in training deep neural nets and other nonconvex models. While the behavior of SGD is well understood in the convex learning setting, the existing theoretical results for SGD applied to nonconvex objective functions are far from mature. For example, existing results require to impose a nontrivial assumption on the uniform boundedness of gradients for all iterates encountered in the learning process, which is hard to verify in practical implementations. In this paper, we establish a rigorous theoretical foundation for SGD in nonconvex learning by showing that this boundedness assumption can be removed without affecting convergence rates. In particular, we establish sufficient conditions for almost sure convergence as well as optimal convergence rates for SGD applied to both general nonconvex objective functions and gradient-dominated objective functions. A linear convergence is further derived in the case with zero variances.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 6; 2023: 8; 2024: 9; 2025: 7; 2026: 3.

48 papers.

Sort:

By area:

  1. 2021Decentralized Federated Averagingcited by 114
  2. 2023Tackling the Curse of Dimensionality with Physics-Informed Neural Networkscited by 78
  3. 2021A general sample complexity analysis of vanilla policy gradientcited by 74
  4. 2022Byzantine Machine Learning Made Easy by Resilient Averaging of Momentumscited by 63
  5. 2023A Guide Through the Zoo of Biased SGDcited by 39
  6. 2025MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoningcited by 37
  7. 2021Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapescited by 27
  8. 2023Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methodscited by 23
  9. 2021Global Convergence and Stability of Stochastic Gradient Descentcited by 17
  10. 2021A Bregman Learning Framework for Sparse Neural Networkscited by 15
  11. 2021A proof of convergence for gradient descent in the training of artificial neural networks for constant target functionscited by 14
  12. 2023Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Casecited by 13
36 more
  1. 2023Differentially Private SGD Without Clipping Bias: An Error-Feedback Approachcited by 13
  2. 2022A Unified Convergence Theorem for Stochastic Optimization Methodscited by 12
  3. 2021A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functionscited by 10
  4. 2025Sharpness-Aware Minimization: General Analysis and Improved Ratescited by 10
  5. 2021A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functionscited by 9
  6. 2021Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activationcited by 9
  7. 2021Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functionscited by 9
  8. 2024Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networkscited by 9
  9. 2021Robustness through Data Augmentation Loss Consistencycited by 8
  10. 2024Large Deviation Upper Bounds and Improved MSE Rates of Nonlinear SGD: Heavy-tailed Noise and Power of Symmetrycited by 7
  11. 2022Gradient descent provably escapes saddle points in the training of shallow ReLU networkscited by 5
  12. 2024Almost sure convergence rates of stochastic gradient methods under gradient dominationcited by 5
  13. 2025Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgerycited by 5
  14. 2023Distributed Stochastic Optimization under a General Variance Conditioncited by 4
  15. 2024Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversioncited by 4
  16. 2021Guided parallelized stochastic gradient descent for delay compensationcited by 3
  17. 2023Learning Unorthogonalized Matrices for Rotation Estimationcited by 3
  18. 2025Second-order Optimization under Heavy-Tailed Noise: Hessian Clipping and Sample Complexity Limitscited by 3
  19. 2021Towards Sharper Utility Bounds for Differentially Private Pairwise Learningcited by 2
  20. 2023Demystifying the Myths and Legends of Nonconvex Convergence of SGDcited by 2
  21. 2024Parallel Momentum Methods Under Biased Gradient Estimationscited by 2
  22. 2025Collaborative Representation Learning for Alignment of Tactile, Language, and Vision Modalitiescited by 2
  23. 2026Beyond Student: An Asymmetric Network for Neural Network Inheritancecited by 1
  24. 2021Spatio-Temporal Neural Network for Fitting and Forecasting COVID-19cited by 0
  25. 2021FastAdaBelief: Improving Convergence Rate for Belief-based Adaptive Optimizers by Exploiting Strong Convexitycited by 0
  26. 2022A Local Convergence Theory for the Stochastic Gradient Descent Method in Non-Convex Optimization With Non-isolated Local Minimacited by 0
  27. 2022FedVeca: Federated Vectorized Averaging on Non-IID Data with Adaptive Bi-directional Global Objectivecited by 0
  28. 2022Convergence of ease-controlled Random Reshuffling gradient Algorithms under Lipschitz smoothnesscited by 0
  29. 2024Accelerating Federated Learning by Selecting Beneficial Herd of Local Gradientscited by 0
  30. 2024Dynamic Decoupling of Placid Terminal Attractor-based Gradient Descent Algorithmcited by 0
  31. 2024SGD Jittering: A Training Strategy for Robust and Accurate Model-Based Architecturescited by 0
  32. 2024Stochastic Gradient Descent Revisitedcited by 0
  33. 2025Observability conditions for neural state-space models with eigenvalues and their roots of unitycited by 0
  34. 2025Quantitative Convergence Analysis of Projected Stochastic Gradient Descent for Non-Convex Losses via the Goldstein Subdifferentialcited by 0
  35. 2026Achieving Linear Speedup for Composite Federated Learningcited by 0
  36. 2026Parameterized Representations via Implicit Stochastic Modulation for High-Dimensional and High-Order Neural PDE Solverscited by 0