# On the adequacy of untuned warmup for adaptive optimization

arXiv: https://arxiv.org/abs/1910.04209 · HTML: https://mlsanity.com/paper/1910.04209/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2019.

## Built on by (most cited first, 30 of 35)

- [Neural Networks Learn Statistics of Increasing Complexity](https://mlsanity.com/paper/2402.04362/) (2024)
- [Is High Variance Unavoidable in RL? A Case Study in Continuous Control](https://mlsanity.com/paper/2110.11222/) (2021)
- [Ranger21: a synergistic deep learning optimizer](https://mlsanity.com/paper/2106.13731/) (2021)
- [Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training](https://mlsanity.com/paper/2410.23922/) (2024)
- [Class-Level Confidence Based 3D Semi-Supervised Learning](https://mlsanity.com/paper/2210.10138/) (2022)
- [Phase Recognition in Contrast-Enhanced CT Scans based on Deep Learning and Random Sampling](https://mlsanity.com/paper/2203.11206/) (2022)
- [Hierarchical Mixture of Experts: Generalizable Learning for High-Level Synthesis](https://mlsanity.com/paper/2410.19225/) (2024)
- [Efficient Distributed Optimization under Heavy-Tailed Noise](https://mlsanity.com/paper/2502.04164/) (2025)
- [A DNN Framework for Learning Lagrangian Drift With Uncertainty](https://mlsanity.com/paper/2204.05891/) (2022)
- [Investigating Neural Architectures by Synthetic Dataset Design](https://mlsanity.com/paper/2204.11045/) (2022)
- [DynMuon: A Dynamic Spectral Shaping View of Muon](https://mlsanity.com/paper/2605.17109/) (2026)
- [Artist Similarity with Graph Neural Networks](https://mlsanity.com/paper/2107.14541/) (2021)
- [Matrix Completion with Heterogonous Cost](https://mlsanity.com/paper/2203.12120/) (2022)
- [A Distance-Geometric Method for Recovering Robot Joint Angles From an RGB Image](https://mlsanity.com/paper/2301.02051/) (2023)
- [Synaptic Stripping: How Pruning Can Bring Dead Neurons Back To Life](https://mlsanity.com/paper/2302.05818/) (2023)
- [Reflection-Equivariant Diffusion for 3D Structure Determination from Isotopologue Rotational Spectra in Natural Abundance](https://mlsanity.com/paper/2310.11609/) (2023)
- [No Need to Sacrifice Data Quality for Quantity: Crowd-Informed Machine Annotation for Cost-Effective Understanding of Visual Data](https://mlsanity.com/paper/2409.00048/) (2024)
- [Balancing Label Quantity and Quality for Scalable Elicitation](https://mlsanity.com/paper/2410.13215/) (2024)
- [Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rules](https://mlsanity.com/paper/2509.18396/) (2025)
- [DeepGANTT: A Scalable Deep Learning Scheduler for Backscatter Networks](https://mlsanity.com/paper/2112.12985/) (2021)
- [Feels Bad Man: Dissecting Automated Hateful Meme Detection Through the Lens of Facebook's Challenge](https://mlsanity.com/paper/2202.08492/) (2022)
- [Transformer Models for Type Inference in the Simply Typed Lambda Calculus: A Case Study in Deep Learning for Code](https://mlsanity.com/paper/2304.10500/) (2023)
- [A ground-based dataset and a diffusion model for on-orbit low-light image enhancement](https://mlsanity.com/paper/2306.14227/) (2023)
- [Stochastic Subnetwork Annealing: A Regularization Technique for Fine Tuning Pruned Subnetworks](https://mlsanity.com/paper/2401.08830/) (2024)
- [DyConfidMatch: Dynamic Thresholding and Re-sampling for 3D Semi-supervised Learning](https://mlsanity.com/paper/2411.08340/) (2024)
- [Revisiting the Initial Steps in Adaptive Gradient Descent Optimization](https://mlsanity.com/paper/2412.02153/) (2024)
- [On the Distributional Properties of Adaptive Gradients](https://mlsanity.com/paper/2105.07222/) (2021)
- [Training BERT Models to Carry Over a Coding System Developed on One Corpus to Another](https://mlsanity.com/paper/2308.03742/) (2023)
- [Resource Efficient Perception for Vision Systems](https://mlsanity.com/paper/2405.07166/) (2024)
- [Robust Generalization of Graph Neural Networks for Carrier Scheduling](https://mlsanity.com/paper/2407.08479/) (2024)

## Abstract (arXiv)

Adaptive optimization algorithms such as Adam are widely used in deep learning. The stability of such algorithms is often improved with a warmup schedule for the learning rate. Motivated by the difficulty of choosing and tuning warmup schedules, recent work proposes automatic variance rectification of Adam's adaptive learning rate, claiming that this rectified approach ("RAdam") surpasses the vanilla Adam algorithm and reduces the need for expensive tuning of Adam with warmup. In this work, we refute this analysis and provide an alternative explanation for the necessity of warmup based on the magnitude of the update term, which is of greater relevance to training stability. We then provide some "rule-of-thumb" warmup schedules, and we demonstrate that simple untuned warmup of Adam performs more-or-less identically to RAdam in typical practical settings. We conclude by suggesting that practitioners stick to linear warmup with Adam, with a sensible default being linear warmup over 2 / (1 - β2) training iterations.