mlsanityData to 7 Oct 2026

Convergence of a Stochastic Gradient Method with Momentum for Non-Smooth Non-Convex Optimization

arXiv 2002.05466 ↗First posted 13 Feb 2020math.OC cs.LGcited by 32

Not on the map. Why?

Abstract (arXiv)

Stochastic gradient methods with momentum are widely used in applications and at the core of optimization subroutines in many popular machine learning libraries. However, their sample complexities have not been obtained for problems beyond those that are convex or smooth. This paper establishes the convergence rate of a stochastic subgradient method with a momentum term of Polyak type for a broad class of non-smooth, non-convex, and constrained optimization problems. Our key innovation is the construction of a special Lyapunov function for which the proven complexity can be achieved without any tuning of the momentum parameter. For smooth problems, we extend the known complexity bound to the constrained case and demonstrate how the unconstrained case can be analyzed under weaker assumptions than the state-of-the-art. Numerical results confirm our theoretical developments.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 6; 2023: 4; 2024: 7; 2025: 5; 2026: 2.

32 papers.

Sort:

  1. 2021Minibatch optimal transport distances; analysis and applicationscited by 62
  2. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  3. 2021Non-convex Distributionally Robust Optimization: Non-asymptotic Analysiscited by 39
  4. 2021Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothnesscited by 37
  5. 2022Policy Gradient in Robust MDPs with Global Convergence Guaranteecited by 35
  6. 2022On Almost Sure Convergence Rates of Stochastic Gradient Methodscited by 32
  7. 2024Random Scaling and Momentum for Non-smooth Non-convex Optimizationcited by 18
  8. 2021Minibatch and Momentum Model-based Methods for Stochastic Weakly Convex Optimizationcited by 10
  9. 2023Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noisecited by 9
  10. 2024Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networkscited by 9
  11. 2023High Probability Convergence of Adam Under Unbounded Gradients and Affine Variance Noisecited by 8
  12. 2024On the Performance Analysis of Momentum Method: A Frequency Domain Perspectivecited by 8
20 more
  1. 2022On the Convergence of mSGD and AdaGrad for Stochastic Optimizationcited by 7
  2. 2022Convergence of First-Order Methods for Constrained Nonconvex Optimization with Dependent Datacited by 7
  3. 2023Acceleration of stochastic gradient descent with momentum by averaging: finite-sample rates and asymptotic normalitycited by 6
  4. 2024Towards Exact Gradient-based Training on Analog In-memory Computingcited by 6
  5. 2024Almost sure convergence rates of stochastic gradient methods under gradient dominationcited by 5
  6. 2024Stochastic Weakly Convex Optimization Beyond Lipschitz Continuitycited by 3
  7. 2024Reevaluating Theoretical Analysis Methods for Optimization in Deep Learningcited by 3
  8. 2021Bandwidth-based Step-Sizes for Non-Convex Stochastic Optimizationcited by 2
  9. 2025Increasing Batch Size Improves Convergence of Stochastic Gradient Descent with Momentumcited by 2
  10. 2026Lower Bounds and Proximally Anchored SGD for Non-Convex Minimization Under Unbounded Variancecited by 2
  11. 2021OpReg-Boost: Learning to Accelerate Online Algorithms with Operator Regressioncited by 1
  12. 2022Personalized incentives as feedback design in generalized Nash equilibrium problemscited by 1
  13. 2025Accelerating SGDM via Learning Rate and Batch Size Schedules: A Lyapunov-Based Analysiscited by 1
  14. 2025Adaptive Memory Momentum via a Model-Based Framework for Deep Learning Optimizationcited by 1
  15. 2021Scale Invariant Monte Carlo under Linear Function Approximation with Curvature based step-sizecited by 0
  16. 2021Robust Regression via Model Based Methodscited by 0
  17. 2023A Unified Momentum-based Paradigm of Decentralized SGD for Non-Convex Models and Heterogeneous Datacited by 0
  18. 2025Decentralized Nonconvex Composite Federated Learning with Gradient Tracking and Momentumcited by 0
  19. 2025HOME-3: High-Order Momentum Estimator with Third-Power Gradient for Convex and Smooth Nonconvex Optimizationcited by 0
  20. 2026Vanilla SGD with Momentum Survives Heavy-Tailed Noise: Convergence Analysis without Gradient Clipping or Normalizationcited by 0