mlsanityData to 14 Oct 2026

A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules

arXiv 2503.12811 ↗First posted 17 Mar 2025cs.LG cs.AI cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

Training large models is both resource-intensive and time-consuming, making it crucial to understand the quantitative relationship between model performance and hyperparameters. In this paper, we present an empirical law that describes how the pretraining loss of large language models evolves under different learning rate schedules, such as constant, cosine, and step decay schedules. Our proposed law takes a multi-power form, combining a power law based on the sum of learning rates and additional power laws to account for a loss reduction effect induced by learning rate decay. We extensively validate this law on various model sizes and architectures, and demonstrate that after fitting on a few learning rate schedules, the law accurately predicts the loss curves for unseen schedules of different shapes and horizons. Moreover, by minimizing the predicted final pretraining loss across learning rate schedules, we are able to find a schedule that outperforms the widely used cosine learning rate schedule. Interestingly, this automatically discovered schedule bears some resemblance to the recently proposed Warmup-Stable-Decay (WSD) schedule (Hu et al, 2024) but achieves a slightly lower final loss. We believe these results could offer valuable insights for understanding the dynamics of pretraining and designing learning rate schedules to improve efficiency.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 83 of its 106 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

83 of 106 references.

References · 83

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
38 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  3. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  4. 2021Scaling Vision Transformerscited by 999
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  6. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  7. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 699
  8. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  9. 2015Cyclical Learning Rates for Training Neural Networkscited by 629
  10. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  11. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  12. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  13. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  14. 2023Scaling Data-Constrained Language Modelscited by 403
  15. 2021Explaining Neural Scaling Lawscited by 363
  16. 20242 OLMo 2 Furiouscited by 340
  17. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  18. 2021Scaling Laws for Transfercited by 272
  19. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  20. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 184
  21. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
  22. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  23. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  24. 2024The Road Less Scheduledcited by 163
  25. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  26. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  27. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  28. 2023The Quantization Model of Neural Scalingcited by 138
  29. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 134
  30. 2022Revisiting Neural Scaling Laws in Language and Visioncited by 132
  31. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 127
  32. 2023Scaling Laws of RoPE-based Extrapolationcited by 125
  33. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  34. 2022A Solvable Model of Neural Scaling Lawscited by 116
  35. 2024A Dynamical Model of Neural Scaling Lawscited by 111
  36. 2022Broken Neural Scaling Lawscited by 109
  37. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
  38. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 96

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 11.

22 papers.

Sort:

By area:

  1. 2025Intern-S1: A Scientific Multimodal Foundation Modelcited by 53
  2. 2025Scaling Laws for Optimal Data Mixturescited by 52
  3. 2025Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Durationcited by 27
  4. 2025Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networkscited by 22
  5. 2025How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretrainingcited by 14
  6. 2026Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averagingcited by 10
  7. 2025Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Trainingcited by 9
  8. 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 9
  9. 2026Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decaycited by 8
  10. 2026Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Lawscited by 6
  11. 2025Learning Dynamics in Continual Pre-Training for Large Language Modelscited by 5
  12. 2025ZenFlow: Enabling Stall-Free Offloading for LLM Trainingcited by 5
10 more
  1. 2025Efficient Hyperparameter Tuning via Trajectory Invariance Principlecited by 4
  2. 2026Configuration-to-Performance Scaling Law with Neural Ansatzcited by 2
  3. 2026On the Nonlinearity of Learning Rate Scaling for LLM Trainingcited by 2
  4. 2025Stepsize anything: A unified learning rate schedule for budgeted-iteration trainingcited by 1
  5. 2026Effective Frontiers: A Unification of Neural Scaling Lawscited by 1
  6. 2026TuneAhead: Predicting Fine-tuning Performance Before Full Training Beginscited by 1
  7. 2026Smooth Scaling Laws Hide Stepwise Token Learningcited by 1
  8. 2026How to Set the Batch Size for Large-Scale Pre-training?cited by 0
  9. 2026Predicting Large Model Test Losses with a Noisy Quadratic Systemcited by 0
  10. 2026Balancing Early Performance Sacrifices with Long-Term Gains: Scaling Learning-Rate Warmup Duration Across Training Horizonscited by 0