mlsanityData to 7 Oct 2026

Adam with model exponential moving average is effective for nonconvex optimization

arXiv 2405.18199 ↗First posted 28 May 2024cs.LG math.OCcited by 22

Not on the map. Why?

Abstract (arXiv)

In this work, we offer a theoretical analysis of two modern optimization techniques for training large and complex models: (i) adaptive optimization algorithms, such as Adam, and (ii) the model exponential moving average (EMA). Specifically, we demonstrate that a clipped version of Adam with model EMA achieves the optimal convergence rates in various nonconvex optimization settings, both smooth and nonsmooth. Moreover, when the scale varies significantly across different coordinates, we demonstrate that the coordinate-wise adaptivity of Adam is provably advantageous. Notably, unlike previous analyses of Adam, our analysis crucially relies on its core elements -- momentum and discounting factors -- as well as model EMA, motivating their wide applications in practice.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 45 of its 62 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

45 of 62 references.

References · 45

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2020Denoising Diffusion Implicit Modelscited by 9,655
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  6. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  7. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  8. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  9. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  10. 2019On the Convergence of Adam and Beyondcited by 834
  11. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  12. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
33 more
  1. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  2. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 303
  3. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 173
  4. 2020A Simple Convergence Proof of Adam and Adagradcited by 169
  5. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 124
  6. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  7. 2018Stochastic subgradient method converges on tame functionscited by 120
  8. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  9. 2022Adam Can Converge Without Any Modification On Update Rulescited by 112
  10. 2023Convergence of Adam Under Relaxed Assumptionscited by 97
  11. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 97
  12. 2016Scale-Free Online Learningcited by 83
  13. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 80
  14. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 78
  15. 2018The Unusual Effectiveness of Averaging in GAN Trainingcited by 73
  16. 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
  17. 2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 50
  18. 2022Understanding AdamW through Proximal Methods and Scale-Freenesscited by 46
  19. 2023Optimal Stochastic Non-smooth Non-convex Optimization through Online-to-Non-convex Conversioncited by 42
  20. 2020Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitationscited by 39
  21. 2023Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexitycited by 38
  22. 2021A gradient sampling method with complexity guarantees for Lipschitz functions in high and low dimensionscited by 37
  23. 2020A new regret analysis for Adam-type algorithmscited by 35
  24. 2018Subgradient methods for sharp weakly convex functionscited by 33
  25. 2021On the Finite-Time Complexity and Practical Computation of Approximate Stationarity Concepts of Lipschitz Functionscited by 30
  26. 2023Deterministic Nonsmooth Nonconvex Optimizationcited by 25
  27. 2018AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methodscited by 24
  28. 2022How Does Adaptive Optimization Impact Local Neural Network Geometry?cited by 24
  29. 2024Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguisecited by 23
  30. 2024Random Scaling and Momentum for Non-smooth Non-convex Optimizationcited by 18
  31. 2022No Dimension-Free Deterministic Algorithm Computes Approximate Stationarities of Lipschitzianscited by 15
  32. 2023Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregressioncited by 14
  33. 2021Escaping strict saddle points of the Moreau envelope in nonsmooth optimizationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 11.

22 papers.

Sort:

By area:

  1. 2024General framework for online-to-nonconvex conversion: Schedule-free SGD is also effective for nonconvex optimizationcited by 14
  2. 2025Online Convex Optimization with Heavy Tails: Old Algorithms, New Regrets, and Applicationscited by 13
  3. 2026Adaptive Matrix Online Learning through Smoothing with Guarantees for Nonsmooth Nonconvex Optimizationcited by 9
  4. 2025On the O(frac√dK1/4) Convergence Rate of AdamW Measured by ℓ1 Normcited by 8
  5. 2025Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Trainingcited by 8
  6. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 4
  7. 2026Muon with Finite Newton-Schulz: The Smoothing Benefit in Nonsmooth Nonconvex Optimizationcited by 4
  8. 2025Improving Online-to-Nonconvex Conversion for Smooth Optimization via Double Optimismcited by 3
  9. 2025Elucidating Subspace Perturbation in Zeroth-Order Optimization: Theory and Practice at Scalecited by 2
  10. 2025PANDORA: Diffusion Policy Learning for Dexterous Robotic Piano Playingcited by 2
  11. 2025How to Set β1, β2 in Adam: An Online Learning Perspectivecited by 2
  12. 2025Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regimecited by 2
10 more
  1. 2026Dynamic Regret via Discounted-to-Dynamic Reduction with Applications to Curved Losses and Adam Optimizercited by 2
  2. 2024Towards Simple and Provable Parameter-Free Adaptive Gradient Methodscited by 1
  3. 2025PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learningcited by 1
  4. 2026Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?cited by 1
  5. 2026The Convergence Behavior of Adam under Heavy-Tailed Noisecited by 1
  6. 2026Stochastic Non-Smooth Convex Optimization with Unbounded Gradientscited by 0
  7. 2026Central limit theorem for the averaged Adam optimizercited by 0
  8. 2026Adam Converges in Nonsmooth Nonconvex Optimizationcited by 0
  9. 2026Optimizing the Preconditioner: A Black-box Online-to-Nonconvex Conversion with Static Regret Minimization Oraclescited by 0
  10. 2026The Exponential Price of Determinism in Nonsmooth Nonconvex Optimizationcited by 0