mlsanityData to 7 Oct 2026

Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguise

arXiv 2402.01567 ↗First posted 2 Feb 2024cs.LG math.OCcited by 23

Not on the map. Why?

Abstract (arXiv)

Despite the success of the Adam optimizer in practice, the theoretical understanding of its algorithmic components still remains limited. In particular, most existing analyses of Adam show the convergence rate that can be simply achieved by non-adative algorithms like SGD. In this work, we provide a different perspective based on online learning that underscores the importance of Adam's algorithmic components. Inspired by Cutkosky et al. (2023), we consider the framework called online learning of updates/increments, where we choose the updates/increments of an optimizer based on an online learner. With this framework, the design of a good optimizer is reduced to the design of a good online learner. Our main observation is that Adam corresponds to a principled online learning framework called Follow-the-Regularized-Leader (FTRL). Building on this observation, we study the benefits of its algorithmic components from the online learning perspective.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 26 of its 53 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

26 of 53 references.

References · 26

  1. 2019On the Convergence of Adam and Beyondcited by 834
  2. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  3. 2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 407
  4. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  5. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  6. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 173
  7. 2020A Simple Convergence Proof of Adam and Adagradcited by 169
  8. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 155
  9. 2018Adaptive Online Learning in Dynamic Environmentscited by 130
  10. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 124
  11. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  12. 2022Adam Can Converge Without Any Modification On Update Rulescited by 112
14 more
  1. 2015Strongly Adaptive Online Learningcited by 102
  2. 2023Convergence of Adam Under Relaxed Assumptionscited by 97
  3. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 97
  4. 2016Scale-Free Online Learningcited by 83
  5. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 80
  6. 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
  7. 2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 50
  8. 2022Parameter-free Mirror Descentcited by 45
  9. 2023Optimal Stochastic Non-smooth Non-convex Optimization through Online-to-Non-convex Conversioncited by 42
  10. 2023Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexitycited by 38
  11. 2020A new regret analysis for Adam-type algorithmscited by 35
  12. 2018AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methodscited by 24
  13. 2022How Does Adaptive Optimization Impact Local Neural Network Geometry?cited by 24
  14. 2023Unconstrained Dynamic Regret via Sparse Codingcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 5; 2026: 11.

23 papers.

Sort:

  1. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  2. 2024Does SGD really happen in tiny subspaces?cited by 35
  3. 2024Adam with model exponential moving average is effective for nonconvex optimizationcited by 22
  4. 2024Random Scaling and Momentum for Non-smooth Non-convex Optimizationcited by 18
  5. 2024General framework for online-to-nonconvex conversion: Schedule-free SGD is also effective for nonconvex optimizationcited by 14
  6. 2024Discounted Adaptive Online Learning: Towards Better Regularizationcited by 12
  7. 2025A Fine-Tuning Approach for T5 Using Knowledge Graphs to Address Complex Taskscited by 6
  8. 2025On the Dynamic Regret of Following the Regularized Leader: Optimism with History Pruningcited by 5
  9. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 4
  10. 2026Muon with Finite Newton-Schulz: The Smoothing Benefit in Nonsmooth Nonconvex Optimizationcited by 4
  11. 2024Reevaluating Theoretical Analysis Methods for Optimization in Deep Learningcited by 3
  12. 2025Improving Online-to-Nonconvex Conversion for Smooth Optimization via Double Optimismcited by 3
11 more
  1. 2025Discounted Online Convex Optimization: Uniform Regret Across a Continuous Intervalcited by 2
  2. 2025How to Set β1, β2 in Adam: An Online Learning Perspectivecited by 2
  3. 2026Dynamic Regret via Discounted-to-Dynamic Reduction with Applications to Curved Losses and Adam Optimizercited by 2
  4. 2026On the Convergence of Adam, Revisitedcited by 1
  5. 2026The Convergence Behavior of Adam under Heavy-Tailed Noisecited by 1
  6. 2026Finite-Horizon Input-Output Dynamics of Minibatch Perturbations in AdamWcited by 1
  7. 2026Stochastic Non-Smooth Convex Optimization with Unbounded Gradientscited by 0
  8. 2026SEArch: Optimistic Policy Selection Between Scene Noise and Drift for UAV Radar Searchcited by 0
  9. 2026FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoocited by 0
  10. 2026Adam Converges in Nonsmooth Nonconvex Optimizationcited by 0
  11. 2026Optimizing the Preconditioner: A Black-box Online-to-Nonconvex Conversion with Static Regret Minimization Oraclescited by 0