mlsanityData to 7 Oct 2026

Why Transformers Need Adam: A Hessian Perspective

arXiv 2402.16788 ↗First posted 26 Feb 2024cs.LG cs.AIcited by 123

Not on the map. Why?

Abstract (arXiv)

SGD performs worse than Adam by a significant margin on Transformers, but the reason remains unclear. In this work, we provide an explanation through the lens of Hessian: (i) Transformers are "heterogeneous": the Hessian spectrum across parameter blocks vary dramatically, a phenomenon we call "block heterogeneity"; (ii) Heterogeneity hampers SGD: SGD performs worse than Adam on problems with block heterogeneity. To validate (i) and (ii), we check various Transformers, CNNs, MLPs, and quadratic problems, and find that SGD can perform on par with Adam on problems without block heterogeneity, but performs worse than Adam when the heterogeneity exists. Our initial theoretical analysis indicates that SGD performs worse because it applies one single learning rate to all blocks, which cannot handle the heterogeneity among blocks. This limitation could be ameliorated if we use coordinate-wise learning rates, as designed in Adam.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). Builds on 74 of its 114 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

74 of 114 references.

References · 74

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
38 more
  1. 2020On Layer Normalization in the Transformer Architecturecited by 906
  2. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  3. 2019On the Convergence of Adam and Beyondcited by 834
  4. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  5. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  6. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  7. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  8. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  9. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  10. 2021Early Convolutions Help Transformers See Bettercited by 395
  11. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  12. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  13. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  14. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  15. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  16. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  17. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  18. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  19. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 260
  20. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  21. 2020Understanding the Difficulty of Training Transformerscited by 210
  22. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  23. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  24. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 181
  25. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  26. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 173
  27. 2020A Simple Convergence Proof of Adam and Adagradcited by 169
  28. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  29. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  30. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  31. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  32. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  33. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 124
  34. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 116
  35. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  36. 2022Adam Can Converge Without Any Modification On Update Rulescited by 112
  37. 2018The Best of Both Worlds: Combining Recent Advances in Neural Machine Translationcited by 101
  38. 2023Convergence of Adam Under Relaxed Assumptionscited by 97

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 25; 2025: 40; 2026: 58.

123 papers.

Sort:

By area:

  1. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  2. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 78
  3. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
  4. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 55
  5. 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 50
  6. 2024Deconstructing What Makes a Good Optimizer for Language Modelscited by 40
  7. 2025In Search of Adam's Secret Saucecited by 40
  8. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 38
  9. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 36
  10. 2024Does SGD really happen in tiny subspaces?cited by 35
  11. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 33
  12. 2025Peri-LN: Revisiting Normalization Layer in the Transformer Architecturecited by 32
38 more
  1. 2025Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?cited by 30
  2. 2025COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMscited by 28
  3. 2026Preconditioning Benefits of Spectral Orthogonalization in Muoncited by 28
  4. 2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Trainingcited by 27
  5. 2025The emergence of sparse attention: impact of data distribution and benefits of repetitioncited by 25
  6. 2024Adam with model exponential moving average is effective for nonconvex optimizationcited by 22
  7. 2024AdaGrad under Anisotropic Smoothnesscited by 22
  8. 2024Adam Exploits ℓ_∞-geometry of Loss Landscape via Coordinate-wise Adaptivitycited by 20
  9. 2024SWAN: SGD with Normalization and Whitening Enables Stateless LLM Trainingcited by 20
  10. 2024Encryption-Friendly LLM Architecturecited by 15
  11. 2025Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learningcited by 15
  12. 2025FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Modelscited by 15
  13. 2024AdaFisher: Adaptive Second Order Optimization via Fisher Informationcited by 14
  14. 2025Is your batch size the problem? Revisiting the Adam-SGD gap in language modelingcited by 14
  15. 2024MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuningcited by 13
  16. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13
  17. 2025Unveiling the Basin-Like Loss Landscape in Large Language Modelscited by 13
  18. 2025The Rich and the Simple: On the Implicit Bias of Adam and SGDcited by 12
  19. 2024Addax: Utilizing Zeroth-Order Gradients to Improve Memory Efficiency and Performance of SGD for Fine-Tuning Language Modelscited by 11
  20. 2024Convergence Rate Analysis of LIONcited by 11
  21. 2026RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimizationcited by 11
  22. 2024Understanding Adam Requires Better Rotation Dependent Assumptionscited by 10
  23. 2024What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysiscited by 9
  24. 2025Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methodscited by 9
  25. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  26. 2025PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflictscited by 8
  27. 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancementcited by 8
  28. 2024HyperINF: Unleashing the HyperPower of the Schulz's Method for Data Influence Estimationcited by 7
  29. 2025On Provable Benefits of Muon in Federated Learningcited by 7
  30. 2024No More Adam: Learning Rate Scaling at Initialization is All You Needcited by 6
  31. 2025Mechanistic Insights into Grokking from the Embedding Layercited by 6
  32. 2025GradPower: Powering Gradients for Faster Language Model Pre-Trainingcited by 6
  33. 2026Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMscited by 6
  34. 2026A Defense of the Quadratic Modelcited by 6
  35. 2024Exact Risk Curves of signSGD in High-Dimensions: Quantifying Preconditioning and Noise-Compression Effectscited by 5
  36. 2025On the Origin of Algorithmic Progress in AIcited by 5
  37. 2026On Surprising Effectiveness of Masking Updates in Adaptive Optimizerscited by 5
  38. 2026MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-trainingcited by 5

50 most cited of 123 All 123 (JSON)