mlsanityData to 14 Oct 2026

Adam Exploits ℓ_∞-geometry of Loss Landscape via Coordinate-wise Adaptivity

arXiv 2410.08198 ↗First posted 10 Oct 2024cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

Adam outperforms SGD when training language models. Yet this advantage is not well-understood theoretically -- previous convergence analysis for Adam and SGD mainly focuses on the number of steps T and is already minimax-optimal in non-convex cases, which are both widetildeO(T-1/4). In this work, we argue that the exploitation of nice ℓ_∞-geometry is the key advantage of Adam over SGD. More specifically, we give a new convergence analysis for Adam under novel assumptions that loss is smooth under ℓ_∞-geometry rather than the more common ℓ2-geometry, which yields a much better empirical smoothness constant for GPT-2 and ResNet models. Our experiments confirm that Adam performs much worse when the favorable ℓ_∞-geometry is changed while SGD provably remains unaffected. We also extend the convergence analysis to blockwise Adam under novel blockwise smoothness assumptions.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 31 of its 44 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

31 of 44 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  9. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  10. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 307
  11. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  12. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
19 more
  1. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  2. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 139
  3. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  4. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 126
  5. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  6. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  7. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  8. 2023Convergence of Adam Under Relaxed Assumptionscited by 99
  9. 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
  10. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 80
  11. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
  12. 2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 53
  13. 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
  14. 2024Deconstructing What Makes a Good Optimizer for Language Modelscited by 41
  15. 2018Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov accelerationcited by 39
  16. 2024AdaGrad under Anisotropic Smoothnesscited by 22
  17. 2020Adaptive Gradient Methods for Constrained Convex Optimization and Variational Inequalitiescited by 18
  18. 2022VectorAdam for Rotation Equivariant Geometry Optimizationcited by 18
  19. 2020AdaSGD: Bridging the gap between SGD and Adamcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 11.

23 papers.

Sort:

By area:

  1. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 57
  2. 2025Structured Preconditioners in Adaptive Optimization: A Unified Analysiscited by 33
  3. 2025Is your batch size the problem? Revisiting the Adam-SGD gap in language modelingcited by 15
  4. 2025Generalization of Diffusion Models Arises with a Balanced Representation Spacecited by 11
  5. 2026Enhancing LLM Training via Spectral Clippingcited by 10
  6. 2025On the O(frac√dK1/4) Convergence Rate of AdamW Measured by ℓ1 Normcited by 8
  7. 2026Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectorycited by 8
  8. 2025On Learning Verifiers and Implications to Chain-of-Thought Reasoningcited by 6
  9. 2025Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Schedulingcited by 6
  10. 2025Conda: Column-Normalized Adam for Training Large Language Models Fastercited by 5
  11. 2025A Tale of Two Geometries: Adaptive Optimizers and Non-Euclidean Descentcited by 5
  12. 2025Provable Benefit of SignGD: A Minimal Model Under Heavy-Tailed Class Imbalancecited by 5
11 more
  1. 2026Towards Understanding Adam Convergence on Highly Degenerate Polynomialscited by 4
  2. 2025How do simple rotations affect the implicit bias of Adam?cited by 3
  3. 2025Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regimecited by 2
  4. 2026Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotationcited by 2
  5. 2026When and Why SignSGD Outperforms SGD: A Theoretical Study Based on ℓ1-norm Lower Boundscited by 2
  6. 2026Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrencecited by 1
  7. 2026Stabilizing Transformer Training Through Consensuscited by 0
  8. 2026Uniform Scaling Limits in AdamW-Trained Transformerscited by 0
  9. 2026The Loss Does Not See the Basis, but Adam Doescited by 0
  10. 2026MercerFlow: Flow Matching in a Kernel-Induced Latent Space for Probabilistic Forecastingcited by 0
  11. 2026Which Geometry on Which Layer? Measured Sign-Spectral Assignment in Mixed-Optimizer Trainingcited by 0