mlsanityData to 7 Oct 2026

Stochastic Subgradient Methods with Guaranteed Global Stability in Nonsmooth Nonconvex Optimization

arXiv 2307.10053 ↗First posted 19 Jul 2023math.OC cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

In this paper, we focus on providing convergence guarantees for stochastic subgradient methods in minimizing nonsmooth nonconvex functions. We first investigate the global stability of a general framework for stochastic subgradient methods, where the corresponding differential inclusion admits a coercive Lyapunov function. We prove that, for any sequence of sufficiently small stepsizes and approximation parameters, coupled with sufficiently controlled noises, the iterates are uniformly bounded and asymptotically stabilize around the stable set of its corresponding differential inclusion. Moreover, we develop an improved analysis to apply our proposed framework to establish the global stability of a wide range of stochastic subgradient methods, where the corresponding Lyapunov functions are possibly non-coercive. These theoretical results illustrate the promising potential of our proposed framework for establishing the global stability of various stochastic subgradient methods.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 37 of its 72 references that are in the map. First posted on arXiv 19 Jul 2023.

Builds on

37 of 72 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019On the Convergence of Adam and Beyondcited by 834
  4. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  5. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  6. 2017Large Batch Training of Convolutional Networkscited by 529
  7. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  8. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 282
  9. 2020Momentum Improves Normalized SGDcited by 136
  10. 2018Stochastic subgradient method converges on tame functionscited by 120
  11. 2022Adam Can Converge Without Any Modification On Update Rulescited by 112
  12. 2020Improved Analysis of Clipping Algorithms for Non-convex Optimizationcited by 88
25 more
  1. 2019Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learningcited by 71
  2. 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
  3. 2017Stochastic Methods for Composite and Weakly Convex Optimization Problemscited by 48
  4. 2018Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimizationcited by 47
  5. 2020A mathematical model for automatic differentiation in machine learningcited by 44
  6. 2021Nonsmooth Implicit Differentiation for Machine Learning and Optimizationcited by 43
  7. 2018Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov accelerationcited by 39
  8. 2019Conformal Symplectic and Relativistic Optimizationcited by 28
  9. 2020Optimization with Momentum: Dynamical, Control-Theoretic, and Symplectic Perspectivescited by 27
  10. 2020Convergence of constant step stochastic gradient descent for non-smooth non-convex functionscited by 27
  11. 2023Adam-family Methods for Nonsmooth Optimization with Convergence Guaranteescited by 22
  12. 2020A Stochastic Subgradient Method for Nonsmooth Nonconvex Multi-Level Composition Optimizationcited by 16
  13. 2019An Inertial Newton Algorithm for Deep Learningcited by 14
  14. 2022Dissolving Constraints for Riemannian Optimizationcited by 12
  15. 2022An Improved Unconstrained Approach for Bilevel Optimizationcited by 11
  16. 2022Born-Infeld (BI) for AI: Energy-Conserving Descent (ECD) for Optimizationcited by 10
  17. 2019Convergence of a Stochastic Subgradient Method with Averaging for Nonsmooth Nonconvex Constrained Optimizationcited by 9
  18. 2023Nonsmooth nonconvex stochastic heavy ballcited by 8
  19. 2022A Constraint Dissolving Approach for Nonsmooth Optimization over the Stiefel Manifoldcited by 7
  20. 2020Long term dynamics of the subgradient method for Lipschitz path differentiable functionscited by 6
  21. 2022Lyapunov stability of the subgradient method with constant step sizecited by 5
  22. 2020Incremental Without Replacement Sampling in Nonconvex Optimizationcited by 4
  23. 2021A closed-measure approach to stochastic approximationcited by 4
  24. 2019Hamiltonian descent for composite objectivescited by 3
  25. 2023Convergence of the momentum method for semialgebraic functions with locally Lipschitz gradientscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 2.

3 papers.

  1. 2024Central Limit Theorem for Two-Timescale Stochastic Approximation with Markovian Noise: Theory and Applicationscited by 7
  2. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 4
  3. 2026When and Why SignSGD Outperforms SGD: A Theoretical Study Based on ℓ1-norm Lower Boundscited by 2