mlsanityData to 14 Oct 2026

Adam-family Methods for Nonsmooth Optimization with Convergence Guarantees

arXiv 2305.03938 ↗First posted 6 May 2023math.OC cs.LG stat.MLcited by 22

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a comprehensive study on the convergence properties of Adam-family methods for nonsmooth optimization, especially in the training of nonsmooth neural networks. We introduce a novel two-timescale framework that adopts a two-timescale updating scheme, and prove its convergence properties under mild assumptions. Our proposed framework encompasses various popular Adam-family methods, providing convergence guarantees for these methods in training nonsmooth neural networks. Furthermore, we develop stochastic subgradient methods that incorporate gradient clipping techniques for training nonsmooth neural networks with heavy-tailed noise. Through our framework, we show that our proposed methods converge even when the evaluation noises are only assumed to be integrable. Extensive numerical experiments demonstrate the high efficiency and robustness of our proposed methods.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 25 of its 45 references that are in the map. First posted on arXiv 6 May 2023.

Builds on

25 of 45 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019On the Convergence of Adam and Beyondcited by 841
  4. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  5. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
  6. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
  7. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 126
  8. 2018Stochastic subgradient method converges on tame functionscited by 124
  9. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  10. 2020Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clippingcited by 104
  11. 2019Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learningcited by 72
  12. 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
13 more
  1. 2018Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimizationcited by 49
  2. 2021Nonsmooth Implicit Differentiation for Machine Learning and Optimizationcited by 43
  3. 2018Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov accelerationcited by 39
  4. 2021Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothnesscited by 37
  5. 2022Provable Adaptivity of Adam under Non-uniform Smoothnesscited by 30
  6. 2018A general system of differential equations to model first order adaptive algorithmscited by 28
  7. 2020Convergence of constant step stochastic gradient descent for non-smooth non-convex functionscited by 27
  8. 2021Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Accelerationcited by 25
  9. 2023Variance-reduced Clipping for Non-convex Optimizationcited by 22
  10. 2017Linearly convergent stochastic heavy ball method for minimizing generalization errorcited by 14
  11. 2019An Inertial Newton Algorithm for Deep Learningcited by 14
  12. 2023Nonsmooth nonconvex stochastic heavy ballcited by 8
  13. 2023U-Clip: On-Average Unbiased Stochastic Gradient Clippingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 2; 2026: 12.

22 papers.

Sort:

  1. 2024Convergence rates for the Adam optimizercited by 15
  2. 2025ODE approximation for the Adam algorithm: General and overparametrized settingcited by 10
  3. 2024A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGDcited by 9
  4. 2023Adam-family Methods with Decoupled Weight Decay in Deep Learningcited by 5
  5. 2026Dynamic Regret via Discounted-to-Dynamic Reduction with Applications to Curved Losses and Adam Optimizercited by 4
  6. 2023Stochastic Subgradient Methods with Guaranteed Global Stability in Nonsmooth Nonconvex Optimizationcited by 3
  7. 2025On exploration of an interior mirror descent flow for stochastic nonconvex constrained problemcited by 3
  8. 2024Developing Lagrangian-based Methods for Nonsmooth Nonconvex Optimizationcited by 2
  9. 2026A unified convergence theory for adaptive first-order methods in the nonconvex case, including AdaNorm, full and diagonal AdaGrad and Muoncited by 2
  10. 2023Supervised Contrastive Learning for Fine-grained Chromosome Recognitioncited by 1
  11. 2024Convergence of Decentralized Stochastic Subgradient-based Methods for Nonsmooth Nonconvex functionscited by 1
  12. 2024Tissue-Contrastive Semi-Masked Autoencoders for Segmentation Pretraining on Chest CTcited by 1
10 more
  1. 2026Rethinking Adam for Time Series Forecasting: A Simple Heuristic to Improve Optimization under Distribution Shiftscited by 1
  2. 2026Analysis of Adam Algorithms for Stochastic Dynamic Systemscited by 1
  3. 2026Combining Adam and its Inverse Counterpart to Enhance Generalization of Deep Learning Optimizerscited by 0
  4. 2026AdamFlow: Adam-based Wasserstein Gradient Flows for Surface Registration in Medical Imagingcited by 0
  5. 2026Efficient and provably convergent end-to-end training of deep neural networks with linear constraintscited by 0
  6. 2026Stochastic convergence of parallel asynchronous adaptive first-order methodscited by 0
  7. 2026Decentralized Stochastic Subgradient-type Methods with Communication Compression for Nonsmooth Nonconvex Optimizationcited by 0
  8. 2026An Inertial Block Proximal Linearized Method with Adaptive Momentum for Nonconvex and Nonsmooth Optimizationcited by 0
  9. 2026Nonsmooth Optimization via Orthogonalized Momentumcited by 0
  10. 2026Generalized Geometry Block Proximal Linearized Method for Multiblock Nonconvex and Nonsmooth Optimizationcited by 0