mlsanityData to 14 Oct 2026

Why Warmup the Learning Rate? Underlying Mechanisms and Improvements

arXiv 2406.09405 ↗First posted 13 Jun 2024cs.LG cond-mat.dis-nn stat.MLcited by 46

Not on the map. Why?

Abstract (arXiv)

It is common in deep learning to warm up the learning rate η, often by a linear schedule between ηinit = 0 and a predetermined target ηtrgt. In this paper, we show through systematic experiments using SGD and Adam that the overwhelming benefit of warmup arises from allowing the network to tolerate larger ηtrgt by forcing the network to more well-conditioned areas of the loss landscape. The ability to handle larger ηtrgt makes hyperparameter tuning more robust while improving the final performance. We uncover different regimes of operation during the warmup period, depending on whether training starts off in a progressive sharpening or sharpness reduction phase, which in turn depends on the initialization and parameterization. Using these insights, we show how ηinit can be properly chosen by utilizing the loss catapult mechanism, which saves on the number of warmup steps, in some cases completely eliminating the need for warmup. We also suggest an initialization for the variance in Adam which provides benefits similar to warmup.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Builds on 25 of its 42 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

25 of 42 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2016Wide Residual Networkscited by 3,070
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  10. 2021Convex optimizationcited by 2,515
  11. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  12. 2020On Layer Normalization in the Transformer Architecturecited by 922
13 more
  1. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  2. 2018On Lazy Training in Differentiable Programmingcited by 675
  3. 2014Convex Optimization: Algorithms and Complexitycited by 575
  4. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  5. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  6. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  7. 2021The Principles of Deep Learning Theorycited by 173
  8. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  9. 2018A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillationcited by 110
  10. 2020On the infinite width limit of neural networks with a standard parameterizationcited by 38
  11. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 22
  12. 2023Effective Theory of Transformers at Initializationcited by 21
  13. 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 22; 2026: 20.

46 papers.

Sort:

By area:

  1. 2025REVE: A Foundation Model for EEG -- Adapting to Any Setup with Large-Scale Pretraining on 25,000 Subjectscited by 80
  2. 2025Mid-Training of Large Language Models: A Surveycited by 21
  3. 2025OvercookedV2: Rethinking Overcooked for Zero-Shot Coordinationcited by 16
  4. 2025Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Schedulercited by 12
  5. 2026Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averagingcited by 10
  6. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 10
  7. 2026A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMscited by 8
  8. 2025Efficient Distributed Optimization under Heavy-Tailed Noisecited by 7
  9. 2024Simulating Dynamic Tumor Contrast Enhancement in Breast MRI using Conditional Generative Adversarial Networkscited by 6
  10. 2025RICO: Two Realistic Benchmarks and an In-Depth Analysis for Incremental Learning in Object Detectioncited by 5
  11. 2025Accelerating Training Speed of Tiny Recursive Models with Curriculum Guided Adaptive Recursioncited by 5
  12. 2026Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizerscited by 5
34 more
  1. 2025Theoretical Analysis on how Learning Rate Warmup Accelerates Convergencecited by 4
  2. 2024Structured and Balanced Multi-Component and Multi-Layer Neural Networkscited by 3
  3. 2025When Can You Get Away with Low Memory Adam?cited by 3
  4. 2025A statistical physics framework for optimal learningcited by 3
  5. 2025SpeechLLMs for Large-scale Contextualized Zero-shot Slot Fillingcited by 3
  6. 2026Anytime Training with Schedule-Free Spectral Optimizationcited by 3
  7. 2026Demystifying Data Organization for Enhanced LLM Trainingcited by 3
  8. 2025Fractal-IR: A Unified Framework for Efficient and Scalable Image Restorationcited by 2
  9. 2026Prototype Transformer: Towards Language Model Architectures Interpretable by Designcited by 2
  10. 2026Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Ratecited by 2
  11. 2026Scale Weight Decay and Train Bettercited by 2
  12. 2024Hierarchical Information Flow for Generalized Efficient Image Restorationcited by 1
  13. 2025A Privacy-Preserving Domain Adversarial Federated learning for multi-site brain functional connectivity analysiscited by 1
  14. 2025The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergencecited by 1
  15. 2025RNN Generalization to Omega-Regular Languagescited by 1
  16. 2025Intrinsic Dimension Estimating Autoencoder (IDEA) Using CancelOut Layer and a Projected Losscited by 1
  17. 2025Topological Invariance and Breakdown in Learningcited by 1
  18. 2025Adaptive Memory Momentum via a Model-Based Framework for Deep Learning Optimizationcited by 1
  19. 2025Gradient Descent with Provably Tuned Learning-rate Schedulescited by 1
  20. 2026Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networkscited by 1
  21. 2026Mitigating Forgetting in Continual Learning with Selective Gradient Projectioncited by 1
  22. 2026Fine-Tuning Without Forgetting via Loss-Adaptive Learning Ratescited by 1
  23. 2024Investigating Long-term Training for Remote Sensing Object Detectioncited by 0
  24. 2025Some Optimizers are More Equal: Understanding the Role of Optimizers in Group Fairnesscited by 0
  25. 2025ADP-VRSGP: Decentralized Learning with Adaptive Differential Privacy via Variance-Reduced Stochastic Gradient Pushcited by 0
  26. 2025Yahtzee: Reinforcement Learning Techniques for Stochastic Combinatorial Gamescited by 0
  27. 2026CoMa: Contextual Massing Generation with Vision-Language Modelscited by 0
  28. 2026Stabilizing Transformer Training Through Consensuscited by 0
  29. 2026From Vision to Text: A Compact Multimodal Approach for Robust, Cross-Domain Presentation Attack Detection on ID Cardscited by 0
  30. 2026Directional Curvature from Armijo Backtracking: A Low-Cost Sharpness Probe and a Calibration-Free Learning-Rate Safeguard for Adamcited by 0
  31. 2026Invariant Pretraining for Robust Code Representationscited by 0
  32. 2026A Journey to the Edge of Stabilitycited by 0
  33. 2026Balancing Early Performance Sacrifices with Long-Term Gains: Scaling Learning-Rate Warmup Duration Across Training Horizonscited by 0
  34. 2026Can You Trust Your Preference Models? Stackelberg Games for Alignmentcited by 0