mlsanityData to 7 Oct 2026

A Theory on Adam Instability in Large-Scale Machine Learning

arXiv 2304.09871 ↗First posted 19 Apr 2023cs.LG cs.AI math.OCcited by 54

Not on the map. Why?

Abstract (arXiv)

We present a theory for the previously unexplained divergent behavior noticed in the training of large language models. We argue that the phenomenon is an artifact of the dominant optimization algorithm used for training, called Adam. We observe that Adam can enter a state in which the parameter update vector has a relatively large norm and is essentially uncorrelated with the direction of descent on the training loss landscape, leading to divergence. This artifact is more likely to be observed in the training of a deep model with a large batch size, which is the typical setting of large-scale language model training. To argue the theory, we present observations from the training runs of the language models of different scales: 7 billion, 30 billion, 65 billion, and 546 billion parameters.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 18 of its 30 references that are in the map. First posted on arXiv 19 Apr 2023.

Builds on

18 of 30 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2020GLU Variants Improve Transformercited by 1,724
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  10. 2019On the Convergence of Adam and Beyondcited by 834
  11. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  12. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
6 more
  1. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 173
  2. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 124
  3. 2020A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularizationcited by 44
  4. 2021Hessian Eigenspectra of More Realistic Nonlinear Modelscited by 33
  5. 2018AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methodscited by 24
  6. 2022Divergence Results and Convergence of a Variance Reduced Version of ADAMcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 14; 2025: 19; 2026: 13.

54 papers.

Sort:

By area:

  1. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  2. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 637
  3. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  4. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  5. 2024Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathologycited by 144
  6. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  7. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  8. 2023ReContrast: Domain-Specific Anomaly Detection via Contrastive Reconstructioncited by 62
  9. 2025Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspectivecited by 55
  10. 2024Scaling Laws For Diffusion Transformerscited by 41
  11. 2023Jointly Training Large Autoregressive Multimodal Modelscited by 32
  12. 2025SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Trainingcited by 27
38 more
  1. 2023On the Implicit Bias of Adamcited by 25
  2. 2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scalingcited by 21
  3. 2024SWAN: SGD with Normalization and Whitening Enables Stateless LLM Trainingcited by 20
  4. 2025SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Modelscited by 19
  5. 2024Methods of improving LLM training stabilitycited by 13
  6. 2023XGen-7B Technical Reportcited by 11
  7. 2024Falcon2-11B Technical Reportcited by 11
  8. 2024Is Flash Attention Stable?cited by 10
  9. 2025Stochastic Rounding for LLM Training: Theory and Practicecited by 10
  10. 2025Adaptive Preconditioners Trigger Loss Spikes in Adamcited by 10
  11. 2024MomentumSMoE: Integrating Momentum into Sparse Mixture of Expertscited by 9
  12. 2025On the O(frac√dK1/4) Convergence Rate of AdamW Measured by ℓ1 Normcited by 8
  13. 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 8
  14. 2024Anytime-Valid Inference for Double/Debiased Machine Learning of Causal Parameterscited by 5
  15. 2024YuLan-Mini: An Open Data-efficient Language Modelcited by 5
  16. 2025BlueLM-2.5-3B Technical Reportcited by 5
  17. 2025AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-trainingcited by 4
  18. 2025Characterization and Mitigation of Training Instabilities in Microscaling Formatscited by 4
  19. 2026Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikescited by 4
  20. 2026PowLU: An Activation Function for Stable Pre-Training of LLMscited by 4
  21. 2025BSFA: Leveraging the Subspace Dichotomy to Accelerate Neural Network Trainingcited by 3
  22. 2025K2-V2: A 360-Open, Reasoning-Enhanced LLMcited by 3
  23. 2026Proxy Compression for Language Modelingcited by 3
  24. 2026Exploring Silent Data Corruption as a Reliability Challenge in LLM Trainingcited by 2
  25. 2024GEB-1.3B: Open Lightweight Large Language Modelcited by 1
  26. 2024Geometrical structures of digital fluctuations in parameter space of neural networks trained with adaptive momentum optimizationcited by 1
  27. 2025Revisiting Transformers through the Lens of Low Entropy and Dynamic Sparsitycited by 1
  28. 2025Taming LLMs by Scaling Learning Rates with Gradient Groupingcited by 1
  29. 2026Leveraging Second-Order Curvature for Efficient Learned Image Compression: Theory and Empirical Evidencecited by 1
  30. 2026Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decaycited by 1
  31. 2023SING: A Plug-and-Play DNN Learning Techniquecited by 0
  32. 2023Understanding Optimization of Deep Learning via Jacobian Matrix and Lipschitz Constantcited by 0
  33. 2025GWT: Scalable Optimizer State Compression for Large Language Model Trainingcited by 0
  34. 2025Frankenstein Optimizer: Harnessing the Potential by Revisiting Optimization Trickscited by 0
  35. 2025SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adamcited by 0
  36. 2026Automated Customization of LLMs for Enterprise Code Repositories Using Semantic Scopescited by 0
  37. 2026A Mechanism Study of Delayed Loss Spikes in Batch-Normalized Linear Modelscited by 0
  38. 2026Learn-by-Wire Training Control Governance: Bounded Autonomous Training Under Stress for Stability and Efficiencycited by 0

50 most cited of 54 All 54 (JSON)