mlsanityData to 7 Oct 2026

Deconstructing What Makes a Good Optimizer for Language Models

arXiv 2407.07972 ↗First posted 10 Jul 2024cs.LG cs.AIcited by 40

Not on the map. Why?

Abstract (arXiv)

Training language models becomes increasingly expensive with scale, prompting numerous attempts to improve optimization efficiency. Despite these efforts, the Adam optimizer remains the most widely used, due to a prevailing view that it is the most effective approach. We aim to compare several optimization algorithms, including SGD, Adafactor, Adam, Lion, and Sophia in the context of autoregressive language modeling across a range of model sizes, hyperparameters, and architecture variants. Our findings indicate that, except for SGD, these algorithms all perform comparably both in their optimal performance and also in terms of how they fare across a wide range of hyperparameter choices. Our results suggest to practitioners that the choice of optimizer can be guided by practical considerations like memory constraints and ease of implementation, as no single algorithm emerged as a clear winner in terms of performance or stability to hyperparameter misspecification. Given our findings, we further dissect these approaches, examining two simplified versions of Adam: a) signed momentum (Signum) which we see recovers both the performance and hyperparameter stability of Adam and b) Adalayer, a layerwise variant of Adam which we introduce to study the impact on Adam's preconditioning for different layers of the network. Examining Adalayer leads us to the conclusion that, perhaps surprisingly, adaptivity on both the last layer and LayerNorm parameters in particular are necessary for retaining performance and stability to learning rate.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 36 of its 45 references that are in the map. First posted on arXiv 10 Jul 2024.

Builds on

36 of 45 references.

References · 36

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2021Scaling Vision Transformerscited by 993
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  10. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  11. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  12. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
24 more
  1. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  2. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  3. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  4. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  5. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  6. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 136
  7. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  8. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  9. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 118
  10. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  11. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 97
  12. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 84
  13. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 78
  14. 2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 50
  15. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  16. 2020The Geometry of Sign Gradient Descentcited by 39
  17. 2020Disentangling Adaptive Gradient Methods from Learning Ratescited by 39
  18. 2019DeepOBS: A Deep Learning Optimizer Benchmark Suitecited by 37
  19. 2020Benchmarking in Optimization: Best Practice and Open Issuescited by 35
  20. 2022How Does Adaptive Optimization Impact Local Neural Network Geometry?cited by 24
  21. 2021Learning by Turning: Neural Architecture Aware Optimisationcited by 23
  22. 2022How to Fine-Tune Vision Models with SGDcited by 23
  23. 2020AdaSGD: Bridging the gap between SGD and Adamcited by 7
  24. 2019Blockwise Adaptivity: Faster Training and Better Generalization in Deep Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 17; 2026: 11.

40 papers.

Sort:

By area:

  1. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 179
  2. 2024Old Optimizer, New Norm: An Anthologycited by 173
  3. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 59
  4. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  5. 2024The AdEMAMix Optimizer: Better, Faster, Oldercited by 47
  6. 2024Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?cited by 39
  7. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 38
  8. 2025COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMscited by 28
  9. 2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Trainingcited by 27
  10. 2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scalingcited by 21
  11. 2024Adam Exploits ℓ_∞-geometry of Loss Landscape via Coordinate-wise Adaptivitycited by 20
  12. 2024SWAN: SGD with Normalization and Whitening Enables Stateless LLM Trainingcited by 20
28 more
  1. 2025The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newtoncited by 17
  2. 2024Loss-to-Loss Prediction: Scaling Laws for All Datasetscited by 15
  3. 2025A Stable Whitening Optimizer for Efficient Neural Network Trainingcited by 15
  4. 2025What Really Matters in Matrix-Whitening Optimizers?cited by 15
  5. 2025Is your batch size the problem? Revisiting the Adam-SGD gap in language modelingcited by 14
  6. 2025Sign Operator for Coping with Heavy-Tailed Noise in Non-Convex Optimization: High Probability Bounds Under (L0, L1)-Smoothnesscited by 12
  7. 2026Mano: Restriking Manifold Optimization for LLM Trainingcited by 12
  8. 2025Gradient Multi-Normalization for Stateless and Scalable LLM Trainingcited by 9
  9. 2024FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Trainingcited by 7
  10. 2025FOCUS: First Order Concentrated Updating Schemecited by 7
  11. 2026Deriving Hyperparameter Scaling Laws via Modern Optimization Theorycited by 7
  12. 2025GradPower: Powering Gradients for Faster Language Model Pre-Trainingcited by 6
  13. 2026Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMscited by 6
  14. 2024Exact Risk Curves of signSGD in High-Dimensions: Quantifying Preconditioning and Noise-Compression Effectscited by 5
  15. 2025Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Schedulingcited by 5
  16. 2025LOTION: Smoothing the Optimization Landscape for Quantized Trainingcited by 4
  17. 2025When Can You Get Away with Low Memory Adam?cited by 3
  18. 2025Noise-Adaptive Layerwise Learning Rates: Accelerating Geometry-Aware Optimization for Deep Neural Network Trainingcited by 3
  19. 2025Adam or Gauss-Newton? A Comparative Study In Terms of Basis Alignment and SGD Noisecited by 2
  20. 2026Why is Normalization Preferred? A Worst-Case Complexity Theory for Stochastically Preconditioned SGD under Heavy-Tailed Noisecited by 2
  21. 2026Prototype Transformer: Towards Language Model Architectures Interpretable by Designcited by 1
  22. 2026Muon Learns More Robust and Transferable Features than Adamcited by 1
  23. 2025A multilevel approach to accelerate the training of Transformerscited by 0
  24. 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergycited by 0
  25. 2026Same Error, Different Function: The Optimizer as an Implicit Prior in Financial Time Seriescited by 0
  26. 2026PowerStep: Memory-Efficient Adaptive Optimization via ℓp-Norm Steepest Descentcited by 0
  27. 2026Muon Meets Mamba: Spectral Optimization for State Space Modelscited by 0
  28. 2026Relative Generalization Invariance of LLM Pretrainingcited by 0