mlsanityData to 14 Oct 2026

Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?

arXiv 2606.23676 ↗First posted 22 Jun 2026cs.LG cs.AI math.OCcited by 1

Not on the map. Why?

Abstract (arXiv)

AdamW is the de facto optimizer for training large language models (LLMs), yet the theory behind it still lives mostly in finite-variance regimes. This is increasingly unsatisfying, as empirical evidence indicates that stochastic gradient noise in LLM pretraining is typically heavy-tailed. Recent work shows that sign-based optimizers such as Lion and Muon achieve sharp heavy-tailed rates, and that AdaGrad can also converge under heavy-tailed noise. However, no rigorous convergence theory for AdamW has yet been established in this regime. Can AdamW converge under the same heavy-tailed assumptions, or does its second-moment accumulator create a genuine obstruction? We formulate this as an open problem, prove a positive weighted-metric benchmark, and give a corridor lower-bound mechanism showing how denominator memory can hide large gradients.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 37 of its 45 references that are in the map. First posted on arXiv 22 Jun 2026.

Builds on

37 of 45 references.

References · 37

  1. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  2. 2019On the Convergence of Adam and Beyondcited by 841
  3. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  4. 2025Kimi K2: Open Agentic Intelligencecited by 567
  5. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  6. 2025Muon is Scalable for LLM Trainingcited by 428
  7. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
  8. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 139
  9. 2020The Heavy-Tail Phenomenon in SGDcited by 124
  10. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  11. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  12. 2025Fantastic Pretraining Optimizers and Where to Find Themcited by 100
25 more
  1. 2023Convergence of Adam Under Relaxed Assumptionscited by 99
  2. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 98
  3. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 80
  4. 2025Practical Efficiency of Muon for Pretrainingcited by 80
  5. 2025Benchmarking Optimizers for Large Language Model Pretrainingcited by 68
  6. 2023Convex and Non-convex Optimization Under Generalized Smoothnesscited by 44
  7. 2024Nonconvex Stochastic Optimization under Heavy-Tailed Noises: Optimal Convergence without Gradient Clippingcited by 42
  8. 2020The Geometry of Sign Gradient Descentcited by 41
  9. 2023Beyond Uniform Smoothness: A Stopped Analysis of Adaptive SGDcited by 41
  10. 2023Revisiting the Noise Model of Stochastic Gradient Descentcited by 36
  11. 2025Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Lawcited by 24
  12. 2024Adam with model exponential moving average is effective for nonconvex optimizationcited by 23
  13. 2024AdaGrad under Anisotropic Smoothnesscited by 22
  14. 2024On Convergence of Adam for Stochastic Optimization under Relaxed Assumptionscited by 21
  15. 2024Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailedcited by 18
  16. 2026Sign-Based Optimizers Are Effective Under Heavy-Tailed Noisecited by 15
  17. 2025On the O(frac√dK1/4) Convergence Rate of AdamW Measured by ℓ1 Normcited by 8
  18. 2025Better Convergence Guarantees for Sign-Based Momentum Methodscited by 6
  19. 2025Mirror Descent Under Generalized Smoothnesscited by 5
  20. 2025Provable Benefit of SignGD: A Minimal Model Under Heavy-Tailed Class Imbalancecited by 5
  21. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 5
  22. 2026Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Lesscited by 5
  23. 2025Simple Convergence Proof of Adam From a Sign-like Descent Perspectivecited by 4
  24. 2026Can Adaptive Gradient Methods Converge under Heavy-Tailed Noise? A Case Study of AdaGradcited by 3
  25. 2026When and Why SignSGD Outperforms SGD: A Theoretical Study Based on ℓ1-norm Lower Boundscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Second-Moment Memory in Coordinatewise Adamcited by 1