mlsanityData to 7 Oct 2026

Scaling and Transferability of Annealing Strategies in Large Language Model Training

arXiv 2512.13705 ↗First posted 5 Dec 2025cs.LG cs.AIcited by 1

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Learning rate scheduling is crucial for training large language models, yet understanding the optimal annealing strategies across different model configurations remains challenging. In this work, we investigate the transferability of annealing dynamics in large language model training and refine a generalized predictive framework for optimizing annealing strategies under the Warmup-Steady-Decay (WSD) scheduler. Our improved framework incorporates training steps, maximum learning rate, and annealing behavior, enabling more efficient optimization of learning rate schedules. Our work provides a practical guidance for selecting optimal annealing strategies without exhaustive hyperparameter searches, demonstrating that smaller models can serve as reliable proxies for optimizing the training dynamics of larger models. We validate our findings on extensive experiments using both Dense and Mixture-of-Experts (MoE) models, demonstrating that optimal annealing ratios follow consistent patterns and can be transferred across different training configurations.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 29 of its 35 references that are in the map. First posted on arXiv 5 Dec 2025.

Builds on

29 of 35 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  10. 2021Scaling Vision Transformerscited by 993
  11. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  12. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
17 more
  1. 2017Large Batch Training of Convolutional Networkscited by 529
  2. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  3. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  4. 2018An Empirical Model of Large-Batch Trainingcited by 268
  5. 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 94
  6. 2024Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecited by 75
  7. 2025LongCat-Flash Technical Reportcited by 54
  8. 2020Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Trainingcited by 50
  9. 2023Optimal Linear Decay Learning Rate Schedules and Further Refinementscited by 41
  10. 2025The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Trainingcited by 31
  11. 2024Scaling Law with Learning Rate Annealingcited by 29
  12. 2023Exact convergence rate of the last iterate in subgradient methodscited by 24
  13. 2024Surge Phenomenon in Optimal Learning Rate and Batch Size Scalingcited by 22
  14. 2025From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scalingcited by 5
  15. 2017ImageNet Training in Minutescited by 4
  16. 2018Large Scale Language Modeling: Converging on 40GB of Text in Four Hourscited by 4
  17. 2019How Does Learning Rate Decay Help Modern Neural Networks?cited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026TopoCurate:Modeling Interaction Topology for Tool-Use Agent Trainingcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025Model Merging in Pre-training of Large Language Models
  2. 2026SOLAR: A State-Driven Online Learning Rate Scheduler for LLM Pretraining
  3. 2024Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models
  4. 2025Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
  5. 2025Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining
  6. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Models
  7. 2026Understanding and Accelerating the Training of Masked Diffusion Language Models
  8. 2024A critical review of methods and challenges in large language models