mlsanityData to 7 Oct 2026

A new regret analysis for Adam-type algorithms

arXiv 2003.09729 ↗First posted 21 Mar 2020stat.ML cs.LG math.OCcited by 35

Not on the map. Why?

Abstract (arXiv)

In this paper, we focus on a theory-practice gap for Adam and its variants (AMSgrad, AdamNC, etc.). In practice, these algorithms are used with a constant first-order moment parameter β1 (typically between 0.9 and 0.99). In theory, regret guarantees for online convex optimization require a rapidly decaying β1→0 schedule. We show that this is an artifact of the standard analysis and propose a novel framework that allows us to derive optimal, data-dependent regret bounds with a constant β1, without further assumptions. We also demonstrate the flexibility of our analysis on a wide range of different algorithms and settings.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 9; 2023: 3; 2024: 6; 2025: 6; 2026: 4.

35 papers.

Sort:

  1. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  2. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 74
  3. 2022High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsizecited by 40
  4. 2025In Search of Adam's Secret Saucecited by 40
  5. 2023SGD with AdaGrad Stepsizes: Full Adaptivity with High Probability to Unknown Parameters, Unbounded Gradients and Affine Variancecited by 30
  6. 2025SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Trainingcited by 27
  7. 2024Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguisecited by 23
  8. 2024Adam with model exponential moving average is effective for nonconvex optimizationcited by 22
  9. 2022Differentially Private Adaptive Optimization with Delayed Preconditionerscited by 18
  10. 2024Provable Complexity Improvement of AdaGrad over SGD: Upper and Lower Bounds in Stochastic Non-Convex Optimizationcited by 16
  11. 2022Maximizing Communication Efficiency for Large-scale Training via 0/1 Adamcited by 15
  12. 2021Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimizationcited by 12
23 more
  1. 2024How Free is Parameter-Free Stochastic Optimization?cited by 12
  2. 2022AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimizationcited by 10
  3. 2021On the Last Iterate Convergence of Momentum Methodscited by 9
  4. 2022Divergence Results and Convergence of a Variance Reduced Version of ADAMcited by 8
  5. 2021The Role of Momentum Parameters in the Optimal Convergence of Adaptive Polyak's Heavy-ball Methodscited by 7
  6. 2023Bidirectional Looking with A Novel Double Exponential Moving Average to Adaptive and Non-adaptive Momentum Optimizerscited by 6
  7. 2021Momentum Centering and Asynchronous Update for Adaptive Gradient Methodscited by 4
  8. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 4
  9. 2022Analysis of Dual-Based PID Controllers through Convolutional Mirror Descentcited by 3
  10. 2022Optimistic Optimisation of Composite Objective with Exponentiated Updatecited by 3
  11. 2022Dynamic Regret of Adaptive Gradient Methods for Strongly Convex Problemscited by 3
  12. 2025Scale-Invariant Regret Matching and Online Learning with Optimal Convergence: Bridging Theory and Practice in Zero-Sum Gamescited by 3
  13. 2025How to Set β1, β2 in Adam: An Online Learning Perspectivecited by 2
  14. 2026Dynamic Regret via Discounted-to-Dynamic Reduction with Applications to Curved Losses and Adam Optimizercited by 2
  15. 2022Communication-Efficient Adam-Type Algorithms for Distributed Data Miningcited by 1
  16. 2024CAdam: Confidence-Based Optimization for Online Learningcited by 1
  17. 2024Towards Simple and Provable Parameter-Free Adaptive Gradient Methodscited by 1
  18. 2025ADAM Optimization with Adaptive Batch Selectioncited by 1
  19. 2026On the Convergence of Adam, Revisitedcited by 1
  20. 2021AdaL: Adaptive Gradient Transformation Contributes to Convergences and Generalizationscited by 0
  21. 2021A New Adaptive Gradient Method with Gradient Decompositioncited by 0
  22. 2025Learning Rate Annealing Improves Tuning Robustness in Stochastic Optimizationcited by 0
  23. 2026Analysis of Adam Algorithms for Stochastic Dynamic Systemscited by 0