mlsanityData to 7 Oct 2026

Towards Understanding Adam Convergence on Highly Degenerate Polynomials

arXiv 2603.09581 ↗First posted 10 Mar 2026cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Adam is a widely used optimization algorithm in deep learning, yet the specific class of objective functions where it exhibits inherent advantages remains underexplored. Unlike prior studies requiring external schedulers and β2 near 1 for convergence, this work investigates the ``natural'' auto-convergence properties of Adam. We identify a class of highly degenerate polynomials where Adam converges automatically without additional schedulers. Specifically, we derive theoretical conditions for local asymptotic stability on degenerate polynomials and demonstrate strong alignment between theoretical bounds and experimental results. We prove that Adam achieves local linear convergence on these degenerate functions, significantly outperforming the sub-linear convergence of Gradient Descent and Momentum. This acceleration stems from a decoupling mechanism between the second moment vt and squared gradient gt2, which exponentially amplifies the effective learning rate. Finally, we characterize Adam's hyperparameter phase diagram, identifying three distinct behavioral regimes: stable convergence, spikes, and SignGD-like oscillation.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 32 of its 41 references that are in the map. First posted on arXiv 10 Mar 2026.

Builds on

32 of 41 references.

References · 32

  1. 2019On the Convergence of Adam and Beyondcited by 834
  2. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  3. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  4. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 173
  5. 2020A Simple Convergence Proof of Adam and Adagradcited by 169
  6. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 155
  7. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  8. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 124
  9. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  10. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  11. 2022Adam Can Converge Without Any Modification On Update Rulescited by 112
  12. 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
20 more
  1. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  2. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 78
  3. 2018Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimizationcited by 47
  4. 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 36
  5. 2020A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithmscited by 31
  6. 2019Semi-flat minima and saddle points by embedding neural networks to overparameterizationcited by 29
  7. 2019Linear Convergence of Adaptive Stochastic Gradient Descentcited by 29
  8. 2018A general system of differential equations to model first order adaptive algorithmscited by 28
  9. 2024Adam Exploits ℓ_∞-geometry of Loss Landscape via Coordinate-wise Adaptivitycited by 20
  10. 2018A Unified Analysis of AdaGrad with Weighted Aggregation and Momentum Accelerationcited by 15
  11. 2024Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning ratescited by 14
  12. 2024Convergence rates for the Adam optimizercited by 14
  13. 2025Adaptive Preconditioners Trigger Loss Spikes in Adamcited by 10
  14. 2022Non-Convergence and Limit Cycles in the Adam optimizercited by 9
  15. 2024Gradient descent with adaptive stepsize converges (nearly) linearly under fourth-order growthcited by 9
  16. 2025ODE approximation for the Adam algorithm: General and overparametrized settingcited by 8
  17. 2022Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networkscited by 7
  18. 2025Sharp higher order convergence rates for the Adam optimizercited by 7
  19. 2025Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizercited by 5
  20. 2023Geometry and Local Recovery of Global Minima of Two-layer Neural Networks at Overparameterizationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026On the Convergence of Adam, Revisitedcited by 1
  2. 2026On the Principles Behind Neural Network Optimizerscited by 0
  3. 2026Beyond Quadratic Loss: The Stability Phase Diagram of Adamcited by 0