mlsanityData to 14 Oct 2026

Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be

arXiv 2304.13960 ↗First posted 27 Apr 2023cs.LG math.OCcited by 114

Not on the map. Why?

Abstract (arXiv)

The success of the Adam optimizer on a wide array of architectures has made it the default in settings where stochastic gradient descent (SGD) performs poorly. However, our theoretical understanding of this discrepancy is lagging, preventing the development of significant improvements on either algorithm. Recent work advances the hypothesis that Adam and other heuristics like gradient clipping outperform SGD on language tasks because the distribution of the error induced by sampling has heavy tails. This suggests that Adam outperform SGD because it uses a more robust gradient estimate. We evaluate this hypothesis by varying the batch size, up to the entire dataset, to control for stochasticity. We present evidence that stochasticity and heavy-tailed noise are not major factors in the performance gap between SGD and Adam. Rather, Adam performs better as the batch size increases, while SGD is less effective at taking advantage of the reduction in noise. This raises the question as to why Adam outperforms SGD in the full-batch setting. Through further investigation of simpler variants of SGD, we find that the behavior of Adam with large batches is similar to sign descent with momentum.

Key facts. Cited by 114 papers in the map (data through 14 Oct 2026). Builds on 35 of its 48 references that are in the map. First posted on arXiv 27 Apr 2023.

Builds on

35 of 48 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2016Layer Normalizationcited by 5,669
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2019On the Convergence of Adam and Beyondcited by 841
  12. 2019Introduction to Online Convex Optimizationcited by 803
23 more
  1. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  2. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  3. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  4. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  5. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  6. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  7. 2020Understanding the Difficulty of Training Transformerscited by 210
  8. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
  9. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  10. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 139
  11. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 117
  12. 2018Convergence of Gradient Descent on Separable Datacited by 110
  13. 2020Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clippingcited by 104
  14. 2020Improved Analysis of Clipping Algorithms for Non-convex Optimizationcited by 89
  15. 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 82
  16. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 81
  17. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  18. 2019Stochastic Sign Descent Methods: New Algorithms and Better Theorycited by 49
  19. 2020The Geometry of Sign Gradient Descentcited by 41
  20. 2020A new regret analysis for Adam-type algorithmscited by 35
  21. 2015From Averaging to Acceleration, There is Only a Step-sizecited by 24
  22. 2021A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizescited by 20
  23. 2019Memory-Efficient Adaptive Optimizationcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 13; 2024: 41; 2025: 29; 2026: 31.

114 papers.

Sort:

By area:

  1. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  2. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  3. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  4. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 98
  5. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 80
  6. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
  7. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 57
  8. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 51
  9. 2024Deconstructing What Makes a Good Optimizer for Language Modelscited by 41
  10. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 41
  11. 2025In Search of Adam's Secret Saucecited by 40
  12. 2024Does SGD really happen in tiny subspaces?cited by 36
38 more
  1. 2024The Implicit Bias of Adam on Separable Datacited by 36
  2. 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
  3. 2023Scaling Laws for Associative Memoriescited by 35
  4. 2025MuonBP: Faster Muon via Block-Periodic Orthogonalizationcited by 35
  5. 2025The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Trainingcited by 33
  6. 2024Implicit Bias and Fast Convergence Rates for Self-attentioncited by 31
  7. 2024Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguisecited by 26
  8. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
  9. 2025Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Lawcited by 24
  10. 2024Adam with model exponential moving average is effective for nonconvex optimizationcited by 23
  11. 2024Adam Exploits ℓ_∞-geometry of Loss Landscape via Coordinate-wise Adaptivitycited by 23
  12. 2023On the Convergence of Black-Box Variational Inferencecited by 22
  13. 2024AdaGrad under Anisotropic Smoothnesscited by 22
  14. 2024On Convergence of Adam for Stochastic Optimization under Relaxed Assumptionscited by 21
  15. 2024SWAN: SGD with Normalization and Whitening Enables Stateless LLM Trainingcited by 20
  16. 2023When and Why Momentum Accelerates SGD:An Empirical Studycited by 19
  17. 2025SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Trainingcited by 19
  18. 2024FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher informationcited by 18
  19. 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 18
  20. 2023Don't be so Monotone: Relaxing Stochastic Line Search in Over-Parameterized Modelscited by 15
  21. 2025Is your batch size the problem? Revisiting the Adam-SGD gap in language modelingcited by 15
  22. 2023Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimizationcited by 13
  23. 2025Memory-Efficient LLM Pretraining via Minimalist Optimizer Designcited by 13
  24. 2025Overcoming the Communication-Performance Tradeoff in LLM Pretrainingcited by 12
  25. 2024Understanding Adam Requires Better Rotation Dependent Assumptionscited by 11
  26. 2024Convergence Rate Analysis of LIONcited by 11
  27. 2024FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Trainingcited by 10
  28. 2023Profit: Benchmarking Personalization and Robustness Trade-off in Federated Prompt Tuningcited by 9
  29. 2024From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networkscited by 9
  30. 2025Gradient Multi-Normalization for Stateless and Scalable LLM Trainingcited by 9
  31. 2024AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methodscited by 8
  32. 2024No More Adam: Learning Rate Scaling at Initialization is All You Needcited by 8
  33. 2026Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectorycited by 8
  34. 2024Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformerscited by 7
  35. 2024Exact Risk Curves of signSGD in High-Dimensions: Quantifying Preconditioning and Noise-Compression Effectscited by 7
  36. 2025Optimization Benchmark for Diffusion Models on Dynamical Systemscited by 7
  37. 2026Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMscited by 7
  38. 2025How Memory in Optimization Algorithms Implicitly Modifies the Losscited by 6

50 most cited of 114 All 114 (JSON)