mlsanityData to 7 Oct 2026

Why are Adaptive Methods Good for Attention Models?

arXiv 1912.03194 ↗First posted 6 Dec 2019math.OC cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

While stochastic gradient descent (SGD) is still the de facto algorithm in deep learning, adaptive methods like Clipped SGD/Adam have been observed to outperform SGD across important tasks, such as attention models. The settings under which SGD performs poorly in comparison to adaptive methods are not well understood yet. In this paper, we provide empirical and theoretical evidence that a heavy-tailed distribution of the noise in stochastic gradients is one cause of SGD's poor performance. We provide the first tight upper and lower convergence bounds for adaptive gradient methods under heavy-tailed noise. Further, we demonstrate how gradient clipping plays a key role in addressing heavy-tailed gradient noise. Subsequently, we show how clipping can be applied in practice by developing an adaptive coordinate-wise clipping algorithm (ACClip) and demonstrate its superior performance on BERT pretraining and finetuning tasks.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 8; 2023: 10; 2024: 4; 2025: 2; 2026: 2.

35 papers.

Sort:

By area:

  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  2. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  3. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  4. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  5. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  6. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
  7. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-trainingcited by 67
  8. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  9. 2022ABINet++: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Spottingcited by 40
  10. 2021Non-convex Distributionally Robust Optimization: Non-asymptotic Analysiscited by 39
  11. 2023A Survey on Efficient Training of Transformerscited by 37
  12. 2022Faster Adaptive Federated Learningcited by 35
23 more
  1. 2023In-Context Learning for Attention Scheme: from Single Softmax Regression to Multiple Softmax Regression via a Tensor Trickcited by 32
  2. 2021On Proximal Policy Optimization's Heavy-tailed Gradientscited by 30
  3. 2022Provable Adaptivity of Adam under Non-uniform Smoothnesscited by 30
  4. 2022On the Impossible Safety of Large AI Modelscited by 27
  5. 2022Accelerated Federated Learning with Decoupled Adaptive Optimizationcited by 23
  6. 2024Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Timecited by 22
  7. 2022Federated Learning and Meta Learning: Approaches, Applications, and Directionscited by 12
  8. 2025The Rich and the Simple: On the Implicit Bias of Adam and SGDcited by 12
  9. 2023Unmasking Transformers: A Theoretical Approach to Data Recovery via Attention Weightscited by 11
  10. 2024Memory-Efficient Optimization with Factorized Hamiltonian Descentcited by 11
  11. 2024On Goodhart's law, with an application to value alignmentcited by 9
  12. 2021Synthetic Dataset Generation of Driver Telematicscited by 8
  13. 2023A Theoretical Insight into Attack and Defense of Gradient Leakage in Transformercited by 6
  14. 2026Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMscited by 6
  15. 2023High Probability Analysis for Non-Convex Stochastic Optimization with Clippingcited by 5
  16. 2023On Suppressing Range of Adaptive Stepsizes of Adam to Improve Generalisation Performancecited by 3
  17. 2024Communication-Efficient Federated Group Distributionally Robust Optimizationcited by 3
  18. 2021Extending AdamW by Leveraging Its Second Moment and Magnitudecited by 1
  19. 2022A DNN Optimizer that Improves over AdaBelief by Suppression of the Adaptive Stepsize Rangecited by 1
  20. 2025Detecting AI Assistance in Abstract Complex Taskscited by 1
  21. 2023Adaptive Strategies in Non-convex Optimizationcited by 0
  22. 2023Preconditioned Federated Learningcited by 0
  23. 2026Across the Loss Landscape with Progressive Growthcited by 0