mlsanityData to 14 Oct 2026

Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients

arXiv 1705.07774 ↗First posted 22 May 2017cs.LG stat.MLcited by 139

Not on the map. Why?

Abstract (arXiv)

The ADAM optimizer is exceedingly popular in the deep learning community. Often it works very well, sometimes it doesn't. Why? We interpret ADAM as a combination of two aspects: for each weight, the update direction is determined by the sign of stochastic gradients, whereas the update magnitude is determined by an estimate of their relative variance. We disentangle these two aspects and analyze them in isolation, gaining insight into the mechanisms underlying ADAM. This analysis also extends recent results on adverse effects of ADAM on generalization, isolating the sign aspect as the problematic one. Transferring the variance adaptation to SGD gives rise to a novel method, completing the practitioner's toolbox for problems where ADAM fails.

Key facts. Cited by 139 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 13; 2022: 12; 2023: 22; 2024: 20; 2025: 24; 2026: 48.

139 papers.

Sort:

By area:

  1. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  2. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  3. 2024Old Optimizer, New Norm: An Anthologycited by 193
  4. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  5. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 101
  6. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 88
  7. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 81
  8. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 80
  9. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 72
  10. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
  11. 2025Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspectivecited by 56
  12. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 51
38 more
  1. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 49
  2. 2021GradInit: Learning to Initialize Neural Networks for Stable and Efficient Trainingcited by 44
  3. 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 43
  4. 2024Deconstructing What Makes a Good Optimizer for Language Modelscited by 41
  5. 2025In Search of Adam's Secret Saucecited by 40
  6. 2024The Implicit Bias of Adam on Separable Datacited by 36
  7. 2023Scaling Laws for Associative Memoriescited by 35
  8. 2026Toto 2.0: Time Series Forecasting Enters the Scaling Eracited by 30
  9. 2021Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimizationcited by 28
  10. 2023DP-AdamBC: Your DP-Adam Is Actually DP-SGD (Unless You Apply Bias Correction)cited by 28
  11. 2025An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variantscited by 28
  12. 2021Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Accelerationcited by 25
  13. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
  14. 2023Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methodscited by 23
  15. 2024Adam Exploits ℓ_∞-geometry of Loss Landscape via Coordinate-wise Adaptivitycited by 23
  16. 2024Surge Phenomenon in Optimal Learning Rate and Batch Size Scalingcited by 22
  17. 2023Dropout Reduces Underfittingcited by 21
  18. 2024Towards Quantifying the Preconditioning Effect of Adamcited by 19
  19. 2022Cutting Some Slack for SGD with Adaptive Polyak Stepsizescited by 18
  20. 2025What Really Matters in Matrix-Whitening Optimizers?cited by 16
  21. 2022Maximizing Communication Efficiency for Large-scale Training via 0/1 Adamcited by 15
  22. 2023Magnitude Invariant Parametrizations Improve Hypernetwork Learningcited by 14
  23. 2026Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentumcited by 14
  24. 2023Temporal Difference Learning with Compressed Updates: Error-Feedback meets Reinforcement Learningcited by 12
  25. 2025Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelismcited by 12
  26. 2026On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfercited by 12
  27. 2021Polygonal Unadjusted Langevin Algorithms: Creating stable and efficient adaptive algorithms for neural networkscited by 11
  28. 2022Optimization without Backpropagationcited by 11
  29. 2025Forget Forgetting: Continual Learning in a World of Abundant Memorycited by 11
  30. 2024FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Trainingcited by 10
  31. 2026Deriving Hyperparameter Scaling Laws via Modern Optimization Theorycited by 10
  32. 2024Rethinking Weight Decay for Robust Fine-Tuning of Foundation Modelscited by 9
  33. 2025Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Ordercited by 9
  34. 2022Logit Normalization for Long-tail Object Detectioncited by 8
  35. 2023Understanding the robustness difference between stochastic gradient descent and adaptive gradient methodscited by 8
  36. 2026Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectorycited by 8
  37. 2023SignSVRG: fixing SignSGD via variance reductioncited by 7
  38. 2024Exact Risk Curves of signSGD in High-Dimensions: Quantifying Preconditioning and Noise-Compression Effectscited by 7

50 most cited of 139 All 139 (JSON)