mlsanityData to 14 Oct 2026

Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining

arXiv 2608.24814 ↗First posted 25 Aug 2026cs.LG stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

We uncover ELR collapse in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR). When ELR is matched across runs, their loss trajectories collapse throughout training despite substantially different LRs and parameter norms. Across optimizers, architectures, datasets, and model scales, mean collapse errors are typically a few x 10^-3, below the seed-to-seed variation measured in a representative configuration. Systematic ablations identify normalization design and the timescale of LR-norm variation as key determinants of collapse precision. Controlled interventions further show that weight decay and Hyperball shape loss dynamics primarily through the ELR schedules they induce. Replacing LR with ELR enables a fitted functional scaling law (FSL) to transfer across norm-control methods. The resulting ELR-based FSL also explains delayed acceleration, a recurring effect of norm control. Together, these results establish ELR as a common coordinate linking LR scheduling, norm control, and loss dynamics.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 37 of its 44 references that are in the map. First posted on arXiv 25 Aug 2026.

Builds on

37 of 44 references.

References · 37

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2019Root Mean Square Layer Normalizationcited by 1,480
  8. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  9. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  10. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  11. 2025Muon is Scalable for LLM Trainingcited by 428
  12. 2020Query-Key Normalization for Transformerscited by 280
25 more
  1. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  2. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  3. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  4. 2017L2 Regularization versus Batch and Weight Normalizationcited by 124
  5. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  6. 2018Three Mechanisms of Weight Decay Regularizationcited by 113
  7. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 72
  8. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  9. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  10. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 53
  11. 2026Controlled LLM Training on Spectral Spherecited by 31
  12. 2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Trainingcited by 30
  13. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  14. 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
  15. 2025Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedulescited by 17
  16. 2026Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimizationcited by 16
  17. 2025Cautious Weight Decaycited by 13
  18. 2025GradPower: Powering Gradients for Faster Language Model Pre-Trainingcited by 8
  19. 2026Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decaycited by 8
  20. 2025Understanding the Mechanisms of Fast Hyperparameter Transfercited by 7
  21. 2026Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Lawscited by 6
  22. 2023Weight Norm Controlcited by 4
  23. 2025Efficient Hyperparameter Tuning via Trajectory Invariance Principlecited by 4
  24. 2026Hyperball May Not Be a Free Lunchcited by 4
  25. 2026On the Nonlinearity of Learning Rate Scaling for LLM Trainingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretrainingcited by 0
  2. 2026HyperTransfer: Understanding the Equivalence between Base Optimizer and Hyperballcited by 0
  3. 2026Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantizationcited by 0