mlsanityData to 14 Oct 2026

The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training

arXiv 2501.18965 ↗First posted 31 Jan 2025cs.LG math.OC stat.MLcited by 33

Not on the map. Why?

Abstract (arXiv)

We show that learning-rate schedules for large model training behave surprisingly similar to a performance bound from non-smooth convex optimization theory. We provide a bound for the constant schedule with linear cooldown; in particular, the practical benefit of cooldown is reflected in the bound due to the absence of logarithmic terms. Further, we show that this surprisingly close match between optimization theory and practice can be exploited for learning-rate tuning: we achieve noticeable improvements for training 124M and 210M Llama-type models by (i) extending the schedule for continued training with optimal learning-rate, and (ii) transferring the optimal learning-rate across schedules.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 27 of its 44 references that are in the map. First posted on arXiv 31 Jan 2025.

Builds on

27 of 44 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  7. 2021Scaling Vision Transformerscited by 999
  8. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  9. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  10. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  11. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  12. 2016Gradient Descent Learns Linear Dynamical Systemscited by 79
15 more
  1. 2019Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learningcited by 72
  2. 2024Chinchilla Scaling: A replication attemptcited by 72
  3. 2023Optimal Linear Decay Learning Rate Schedules and Further Refinementscited by 43
  4. 2015Smooth Strongly Convex Interpolation and Exact Worst-case Performance of First-order Methodscited by 37
  5. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 36
  6. 2020A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performancecited by 26
  7. 2023Exact convergence rate of the last iterate in subgradient methodscited by 25
  8. 2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scalingcited by 23
  9. 2023Aiming towards the minimizers: fast convergence of SGD for overparametrized problemscited by 22
  10. 2023Training trajectories, mini-batch losses and the curious role of the learning ratecited by 21
  11. 2015Exact Worst-case Performance of First-order Methods for Composite Convex Optimizationcited by 16
  12. 2022PEPit: computer-assisted worst-case analyses of first-order optimization methods in Pythoncited by 11
  13. 2024Loss Landscape Characterization of Neural Networks without Over-Parametrizationcited by 8
  14. 2024No More Adam: Learning Rate Scaling at Initialization is All You Needcited by 8
  15. 2024Why you don't overfit, and don't need Bayes if you only train for one epochcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 23.

33 papers.

Sort:

By area:

  1. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  2. 2025Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networkscited by 22
  3. 2026A Defense of the Quadratic Modelcited by 14
  4. 2026Replaying pre-training data improves fine-tuningcited by 13
  5. 2025Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Schedulercited by 12
  6. 2026Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averagingcited by 10
  7. 2026Deriving Hyperparameter Scaling Laws via Modern Optimization Theorycited by 10
  8. 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 9
  9. 2026Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decaycited by 8
  10. 2025Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedule for Continual Pre-trainingcited by 7
  11. 2025Optimization Benchmark for Diffusion Models on Dynamical Systemscited by 7
  12. 2026On the Role of Batch Size in Stochastic Conditional Gradient Methodscited by 7
21 more
  1. 2026ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Modelscited by 7
  2. 2026The Design Space of Tri-Modal Masked Diffusion Modelscited by 6
  3. 2026Universal Dynamics of Warmup Stable Decay: understanding WSD beyond Transformerscited by 5
  4. 2025AdaLRS: Loss-Guided Adaptive Learning Rate Search for Efficient Foundation Model Pretrainingcited by 3
  5. 2026Convex Dominance in Deep Learning I: A Scaling Law of Loss and Learning Ratecited by 3
  6. 2026Optimistic Dual Averaging Unifies Modern Optimizerscited by 3
  7. 2026Anytime Training with Schedule-Free Spectral Optimizationcited by 3
  8. 2026Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectorscited by 3
  9. 2025Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMscited by 2
  10. 2025Scaling depth capacity via zero/one-layer model expansioncited by 1
  11. 2025Scaling and Transferability of Annealing Strategies in Large Language Model Trainingcited by 1
  12. 2026Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Expertscited by 1
  13. 2026How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Sizecited by 1
  14. 2026Towards joint scaling laws with optimal batch size schedulescited by 1
  15. 2026Step-Size Stability in Stochastic Optimization: A Theoretical Perspectivecited by 0
  16. 2026WSqD: A Horizon-Free Learning Rate Schedule for Large Model Trainingcited by 0
  17. 2026Same Loss, Same Noise, Opposite Schedules: Noise Structure and Optimizer Normalization Jointly Determine Whether Learning-Rate Cooldown Helpscited by 0
  18. 2026Understanding the Subspace Stabilization of the Hessian and Gradient Covariance Matrixcited by 0
  19. 2026Balancing Early Performance Sacrifices with Long-Term Gains: Scaling Learning-Rate Warmup Duration Across Training Horizonscited by 0
  20. 2026Last-Iterate Convergence Rate of Normalized Gradient Descent under H"older Smoothnesscited by 0
  21. 2026Asymptotic Risk of Learning-Rate Schedules for Least-Squares Regression with Discretely Distributed Featurescited by 0