mlsanityData to 7 Oct 2026

How to decay your learning rate

arXiv 2103.12682 ↗First posted 23 Mar 2021cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Complex learning rate schedules have become an integral part of deep learning. We find empirically that common fine-tuned schedules decay the learning rate after the weight norm bounces. This leads to the proposal of ABEL: an automatic scheduler which decays the learning rate by keeping track of the weight norm. ABEL's performance matches that of tuned schedules and is more robust with respect to its parameters. Through extensive experiments in vision, NLP, and RL, we show that if the weight norm does not bounce, we can simplify schedules even further with no loss in performance. In such cases, a complex schedule has similar performance to a constant learning rate with a decay at the end of training.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 26 of its 30 references that are in the map. First posted on arXiv 23 Mar 2021.

Builds on

26 of 30 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  7. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  8. 2017Large Batch Training of Convolutional Networkscited by 529
  9. 2017Meta-SGD: Learning to Learn Quickly for Few-Shot Learningcited by 377
  10. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  11. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  12. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
14 more
  1. 2016Learning to Optimizecited by 195
  2. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  3. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  4. 2017L2 Regularization versus Batch and Weight Normalizationcited by 121
  5. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  6. 2017Learned Optimizers that Scale and Generalizecited by 109
  7. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 93
  8. 2020MLRSNet: A Multi-label High Spatial Resolution Remote Sensing Dataset for Semantic Scene Understandingcited by 59
  9. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  10. 2018Fluctuation-dissipation relations for stochastic gradient descentcited by 47
  11. 2020Disentangling Adaptive Gradient Methods from Learning Ratescited by 39
  12. 2020On the training dynamics of deep networks with L2 regularizationcited by 38
  13. 2018L4: Practical loss-based stepsize adaptation for deep learningcited by 26
  14. 2020Learning Rate Annealing Can Provably Help Generalization, Even for Convex Problemscited by 11

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 2; 2023: 5; 2024: 2; 2025: 1.

11 papers.

  1. 2024Sapiens: Foundation for Human Vision Modelscited by 229
  2. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
  3. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 47
  4. 2023Building a Llama2-finetuned LLM for Odia Language Utilizing Domain Knowledge Instruction Setcited by 12
  5. 2023An Automatic Learning Rate Schedule Algorithm for Achieving Faster Convergence and Steeper Descentcited by 9
  6. 2023Time-sensitive Learning for Heterogeneous Federated Edge Intelligencecited by 8
  7. 2022Optimal learning rate schedules in high-dimensional non-convex optimization problemscited by 4
  8. 2023Using Stochastic Gradient Descent to Smooth Nonconvex Functions: Analysis of Implicit Graduated Optimizationcited by 2
  9. 2025Analyzing Generalization in Pre-Trained Symbolic Regressioncited by 2
  10. 2022Learning Rate Perturbation: A Generic Plugin of Learning Rate Schedule towards Flatter Local Minimacited by 1
  11. 2024Carrying over algorithm in transformerscited by 0