mlsanityData to 14 Oct 2026

Optimal Linear Decay Learning Rate Schedules and Further Refinements

arXiv 2310.07831 ↗First posted 11 Oct 2023cs.LG cs.AI stat.MLcited by 43

Not on the map. Why?

Abstract (arXiv)

Learning rate schedules used in practice bear little resemblance to those recommended by theory. We close much of this theory/practice gap, and as a consequence are able to derive new problem-adaptive learning rate schedules. Our main technical contribution is a refined analysis of learning rate schedules for a wide class of optimization algorithms (including SGD). When considering only worst-case analysis, our theory predicts that the optimal choice is the linear decay schedule where the step-size is set proportional to 1 - t/T, where t is the current iteration and T is the total number of steps. To go beyond this worst-case analysis, we use the observed gradient norms to derive schedules refined for any particular task. These refined schedules exhibit learning rate warm-up and rapid learning rate annealing near the end of training. Ours is the first systematic approach to automatically yield both of these properties. We perform the most comprehensive evaluation of learning rate schedules to date, evaluating across 10 diverse deep learning problems, a series of LLMs, and a suite of logistic regression problems. We validate that overall, the linear-decay schedule outperforms all commonly used default schedules including cosine annealing. Our adaptive schedule refinement method gives further improvements.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 39 of its 72 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

39 of 72 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  8. 2016Densely Connected Convolutional Networkscited by 6,720
  9. 2016Wide Residual Networkscited by 3,070
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  11. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  12. 2018fastMRI: An Open Dataset and Benchmarks for Accelerated MRIcited by 470
27 more
  1. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  2. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 234
  3. 2016Hyperparameter optimization with approximate gradientcited by 233
  4. 2020End-to-End Variational Networks for Accelerated MRI Reconstructioncited by 202
  5. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  6. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 156
  7. 2016Coin Betting and Parameter-Free Online Learningcited by 133
  8. 2016Sequence-to-Sequence Learning as Beam-Search Optimizationcited by 127
  9. 2018Black-Box Reductions for Parameter-free Online Learning in Banach Spacescited by 125
  10. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 118
  11. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  12. 2019The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squarescited by 68
  13. 2020Lipschitz and Comparator-Norm Adaptivity in Online Learningcited by 57
  14. 2022Making SGD Parameter-Freecited by 50
  15. 2019The Complexity of Finding Stationary Points with Stochastic Gradient Descentcited by 35
  16. 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
  17. 2019Learning an Adaptive Learning Rate Schedulecited by 29
  18. 2022PDE-Based Optimal Strategy for Unconstrained Online Learningcited by 27
  19. 2023Exact convergence rate of the last iterate in subgradient methodscited by 25
  20. 2018WNGrad: Learn the Learning Rate in Gradient Descentcited by 24
  21. 2023Mechanic: A Learning Rate Tunercited by 20
  22. 2021Eigencurve: Optimal Learning Rate Schedule for SGD on Quadratic Objectives with Skewed Hessian Spectrumscited by 15
  23. 2021AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Flycited by 14
  24. 2019Using Statistics to Automate Stochastic Optimizationcited by 12
  25. 2021Automated Learning Rate Scheduler for Large-batch Trainingcited by 10
  26. 2015Iterative Regularization for Learning with Convex Loss Functionscited by 9
  27. 2020Statistical Adaptive Stochastic Gradient Methodscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 18; 2026: 19.

43 papers.

Sort:

By area:

  1. 2025The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Trainingcited by 33
  2. 2025WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-trainingcited by 23
  3. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  4. 2025Mid-Training of Large Language Models: A Surveycited by 21
  5. 2025Online Convex Optimization with Heavy Tails: Old Algorithms, New Regrets, and Applicationscited by 17
  6. 2024General framework for online-to-nonconvex conversion: Schedule-free SGD is also effective for nonconvex optimizationcited by 15
  7. 2026A Defense of the Quadratic Modelcited by 14
  8. 2026Replaying pre-training data improves fine-tuningcited by 13
  9. 2024Fully Unconstrained Online Learningcited by 10
  10. 2024An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizescited by 10
  11. 2026Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averagingcited by 10
  12. 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 9
31 more
  1. 2026Non-Euclidean Gradient Descent Operates at the Edge of Stabilitycited by 9
  2. 2025Optimal Rates in Continual Linear Regression via Increasing Regularizationcited by 8
  3. 2026Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decaycited by 8
  4. 2025Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedule for Continual Pre-trainingcited by 7
  5. 2025Optimization Benchmark for Diffusion Models on Dynamical Systemscited by 7
  6. 2026Theory of Optimal Learning Rate Schedules and Scaling Laws for a Random Feature Modelcited by 7
  7. 2026ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Modelscited by 7
  8. 2026Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Lawscited by 6
  9. 2025Smoothing DiLoCo with Primal Averaging for Faster Training of LLMscited by 5
  10. 2025Stochastic Approximation with Block Coordinate Optimal Stepsizescited by 3
  11. 2026Convex Dominance in Deep Learning I: A Scaling Law of Loss and Learning Ratecited by 3
  12. 2024Cyclic Sparse Training: Is it Enough?cited by 1
  13. 2025Stepsize anything: A unified learning rate schedule for budgeted-iteration trainingcited by 1
  14. 2025Online Learning-guided Learning Rate Adaptation via Gradient Alignmentcited by 1
  15. 2025Scaling depth capacity via zero/one-layer model expansioncited by 1
  16. 2025Scaling and Transferability of Annealing Strategies in Large Language Model Trainingcited by 1
  17. 2026Towards joint scaling laws with optimal batch size schedulescited by 1
  18. 2024Learning Rate-Free Reinforcement Learning: A Case for Model Selection with Non-Stationary Objectivescited by 0
  19. 2024LiMTR: Time Series Motion Prediction for Diverse Road Users through Multimodal Feature Integrationcited by 0
  20. 2025Learning Rate Annealing Improves Tuning Robustness in Stochastic Optimizationcited by 0
  21. 2025Optimization as a Dynamical System: Generative Schedules from Latent ODEscited by 0
  22. 2025Yahtzee: Reinforcement Learning Techniques for Stochastic Combinatorial Gamescited by 0
  23. 2026Step-Size Stability in Stochastic Optimization: A Theoretical Perspectivecited by 0
  24. 2026DongYuan: An LLM-Based Framework for Integrative Chinese and Western Medicine Spleen-Stomach Disorders Diagnosiscited by 0
  25. 2026Sharp asymptotic theory for Q-learning with LDTZ learning rate and its generalizationcited by 0
  26. 2026A Closed-Form Upper Bound for Admissible Learning-Rate Steps in Belief-Space Dynamicscited by 0
  27. 2026Adaptive Weighted Averagingcited by 0
  28. 2026WSqD: A Horizon-Free Learning Rate Schedule for Large Model Trainingcited by 0
  29. 2026BrachistoneLR: A Brachistochrone-Inspired Learning-Rate Schedule and a Controlled Benchmark of Scheduling Policiescited by 0
  30. 2026Last-Iterate Convergence Rate of Normalized Gradient Descent under H"older Smoothnesscited by 0
  31. 2026Asymptotic Risk of Learning-Rate Schedules for Least-Squares Regression with Discretely Distributed Featurescited by 0