mlsanityData to 7 Oct 2026

Understanding Adam Requires Better Rotation Dependent Assumptions

arXiv 2410.19964 ↗First posted 25 Oct 2024cs.LG cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Despite its widespread adoption, Adam's advantage over Stochastic Gradient Descent (SGD) lacks a comprehensive theoretical explanation. This paper investigates Adam's sensitivity to rotations of the parameter space. We observe that Adam's performance in training transformers degrades under random rotations of the parameter space, indicating a crucial sensitivity to the choice of basis in practice. This reveals that conventional rotation-invariant assumptions are insufficient to capture Adam's advantages theoretically. To better understand the rotation-dependent properties that benefit Adam, we also identify structured rotations that preserve or even enhance its empirical performance. We then examine the rotation-dependent assumptions in the literature and find that they fall short in explaining Adam's behaviour across various rotation types. In contrast, we verify the orthogonality of the update as a promising indicator of Adam's basis sensitivity, suggesting it may be the key quantity for developing rotation-dependent theoretical frameworks that better explain its empirical success.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 41 of its 63 references that are in the map. First posted on arXiv 25 Oct 2024.

Builds on

41 of 63 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2019On the Convergence of Adam and Beyondcited by 834
  8. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  9. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  10. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  11. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 303
  12. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 264
29 more
  1. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  2. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 173
  3. 2020A Simple Convergence Proof of Adam and Adagradcited by 169
  4. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 163
  5. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  6. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 124
  7. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  8. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 121
  9. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  10. 2022Adam Can Converge Without Any Modification On Update Rulescited by 112
  11. 2020The Cost of Training NLP Models: A Concise Overviewcited by 97
  12. 2022Better plain ViT baselines for ImageNet-1kcited by 97
  13. 2023Convergence of Adam Under Relaxed Assumptionscited by 97
  14. 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
  15. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 80
  16. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 78
  17. 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
  18. 2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 50
  19. 2023Convex and Non-convex Optimization Under Generalized Smoothnesscited by 44
  20. 2020The Geometry of Sign Gradient Descentcited by 39
  21. 2023Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexitycited by 38
  22. 2024Hype, Sustainability, and the Price of the Bigger-is-Better Paradigm in AIcited by 35
  23. 2023Parameter-Agnostic Optimization under Relaxed Smoothnesscited by 29
  24. 2020A Study of Condition Numbers for First-Order Optimizationcited by 10
  25. 2023No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Pathscited by 7
  26. 2022Gradient Descent Is Optimal Under Lower Restricted Secant Inequality And Upper Error Boundcited by 5
  27. 2022Optimal first-order methods for convex functions with a quadratic upper boundcited by 4
  28. 2024Large Language Models (LLMs): Deployment, Tokenomics and Sustainabilitycited by 3
  29. 2018Exploration of Numerical Precision in Deep Neural Networkscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditionercited by 21
  2. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13
  3. 2026Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectorycited by 5
  4. 2025Adam or Gauss-Newton? A Comparative Study In Terms of Basis Alignment and SGD Noisecited by 2
  5. 2025Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regimecited by 2
  6. 2026Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotationcited by 2
  7. 2026PolarAdamW: Disentangling Spectral Control and Schur Gauge-Equivariance in Matrix Optimisationcited by 2
  8. 2025How do simple rotations affect the implicit bias of Adam?cited by 1
  9. 2026Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrencecited by 1
  10. 2026The Role of Symmetry in Optimizing Overparameterized Networkscited by 0