mlsanityData to 7 Oct 2026

On the Variance of the Adaptive Learning Rate and Beyond

arXiv 1908.03265 ↗First posted 8 Aug 2019cs.LG cs.CL stat.MLcited by 763

Not on the map. Why?

Cited by 41 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam. Here, we study its mechanism in details. Pursuing the theory behind warmup, we identify a problem of the adaptive learning rate (i.e., it has problematically large variance in the early stage), suggest warmup works as a variance reduction technique, and provide both empirical and theoretical evidence to verify our hypothesis. We further propose RAdam, a new variant of Adam, by introducing a term to rectify the variance of the adaptive learning rate. Extensive experimental results on image classification, language modeling, and neural machine translation verify our intuition and demonstrate the effectiveness and robustness of our proposed method. All implementations are available at: https://github.com/LiyuanLucasLiu/RAdam.

Key facts. Cited by 763 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 166; 2022: 145; 2023: 136; 2024: 148; 2025: 98; 2026: 70.

763 papers.

Sort:

By area:

  1. 2023Consistency Modelscited by 1,778
  2. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 826
  3. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  4. 2023Improved Techniques for Training Consistency Modelscited by 430
  5. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  6. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  7. 2021DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentationcited by 299
  8. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 276
  9. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  10. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  11. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  12. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 175
38 more
  1. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 172
  2. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  3. 2022SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Imagecited by 165
  4. 2021GDRNPP: A Geometry-guided and Fully Learning-based Object Pose Estimatorcited by 148
  5. 2021Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalizationcited by 141
  6. 2024Consistency Models Made Easycited by 140
  7. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  8. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  9. 2022Class-Incremental Continual Learning into the eXtended DER-versecited by 121
  10. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  11. 2022Adam Can Converge Without Any Modification On Update Rulescited by 112
  12. 2022Narcissus: A Practical Clean-Label Backdoor Attack with Limited Informationcited by 102
  13. 2023METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokenscited by 98
  14. 2022Particle Transformer for Jet Taggingcited by 95
  15. 2022GPV-Pose: Category-level Object Pose Estimation via Geometry-guided Point-wise Votingcited by 94
  16. 2021Only a Matter of Style: Age Transformation Using a Style-Based Regression Modelcited by 92
  17. 2023Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Predictioncited by 92
  18. 2023Improving Statistical Fidelity for Neural Image Compression with Implicit Local Likelihood Modelscited by 90
  19. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  20. 2021SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimationcited by 88
  21. 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 87
  22. 2021Iterative Filter Adaptive Network for Single Image Defocus Deblurringcited by 85
  23. 2021HyperInverter: Improving StyleGAN Inversion via Hypernetworkcited by 85
  24. 2022ZeroEGGS: Zero-shot Example-based Gesture Generation from Speechcited by 83
  25. 2021RotoGrad: Gradient Homogenization in Multitask Learningcited by 82
  26. 2022ESS: Learning Event-based Semantic Segmentation from Still Imagescited by 81
  27. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  28. 2021High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tailscited by 80
  29. 2024Distilling Diffusion Models into Conditional GANscited by 78
  30. 2024Fully Explicit Dynamic Gaussian Splattingcited by 75
  31. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  32. 2021Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systemscited by 73
  33. 2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 73
  34. 2024Simplifying Deep Temporal Difference Learningcited by 71
  35. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  36. 2023Residual Denoising Diffusion Modelscited by 68
  37. 2022CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classificationcited by 65
  38. 2022Motion Puzzle: Arbitrary Motion Style Transfer by Body Partcited by 64

50 most cited of 763 All 763 (JSON)