mlsanityData to 7 Oct 2026

Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training

arXiv 2410.23922 ↗First posted 31 Oct 2024cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Learning Rate Warmup is a popular heuristic for training neural networks, especially at larger batch sizes, despite limited understanding of its benefits. Warmup decreases the update size Delta wt = ηt ut early in training by using lower values for the learning rate ηt. In this work we argue that warmup benefits training by keeping the overall size of Delta wt limited, counteracting large initial values of ut. Focusing on small-scale GPT training with AdamW/Lion, we explore the following question: Why and by which criteria are early updates ut too large? We analyze different metrics for the update size including the ℓ2-norm, resulting directional change, and impact on the representations of the network, providing a new perspective on warmup. In particular, we find that warmup helps counteract large angular updates as well as a limited critical batch size early in training. Finally, we show that the need for warmup can be significantly reduced or eliminated by modifying the optimizer to explicitly normalize ut based on the aforementioned metrics.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 35 of its 48 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

35 of 48 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2016Layer Normalizationcited by 5,579
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  8. 2018Group Normalizationcited by 1,514
  9. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  10. 2020On Layer Normalization in the Transformer Architecturecited by 906
  11. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  12. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
23 more
  1. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  2. 2017Large Batch Training of Convolutional Networkscited by 529
  3. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  4. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  5. 2018An Empirical Model of Large-Batch Trainingcited by 268
  6. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  7. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  8. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 116
  9. 2018A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillationcited by 110
  10. 2023A Spectral Condition for Feature Learningcited by 101
  11. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 100
  12. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  13. 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 81
  14. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  15. 2024Scaling Exponents Across Parameterizations and Optimizerscited by 71
  16. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  17. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 50
  18. 2020On the distance between two neural networks and the stability of learningcited by 46
  19. 2019On the adequacy of untuned warmup for adaptive optimizationcited by 35
  20. 2019Online Normalization for Training Neural Networkscited by 29
  21. 2023When and Why Momentum Accelerates SGD:An Empirical Studycited by 17
  22. 2021Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updatescited by 16
  23. 2023The Marginal Value of Momentum for Small Learning Rate SGDcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 9.

22 papers.

Sort:

By area:

  1. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 41
  2. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 38
  3. 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 34
  4. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  5. 2026Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimizationcited by 12
  6. 2025Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Schedulercited by 9
  7. 2025Taming Transformer Without Using Learning Rate Warmupcited by 8
  8. 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 8
  9. 2025Efficient Distributed Optimization under Heavy-Tailed Noisecited by 7
  10. 2025Theoretical Analysis on how Learning Rate Warmup Accelerates Convergencecited by 4
  11. 2026Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectorscited by 3
  12. 2025Gradient Shaping Beyond Clipping: A Functional Perspective on Update Magnitude Controlcited by 1
10 more
  1. 2025Adaptive Memory Momentum via a Model-Based Framework for Deep Learning Optimizationcited by 1
  2. 2026Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networkscited by 1
  3. 2026Progressive Residual Warmup for Language Model Pretrainingcited by 1
  4. 2026Demystifying Manifold Constraints in LLM Pre-trainingcited by 1
  5. 2026Fine-Tuning Without Forgetting via Loss-Adaptive Learning Ratescited by 1
  6. 2025Mpemba Effect in Large-Language Model Training Dynamics: A Minimal Analysis of the Valley-River modelcited by 0
  7. 2025Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMscited by 0
  8. 2026When is Warmstarting Effective for Scaling Language Models?cited by 0
  9. 2026Balancing Early Performance Sacrifices with Long-Term Gains: Scaling Learning-Rate Warmup Duration Across Training Horizonscited by 0
  10. 2026How Does Local Landscape Geometry Evolve in Language Model Pre-Training?cited by 0