mlsanityData to 14 Oct 2026

ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models

arXiv 2605.19095 ↗First posted 18 May 2026cs.LG cs.AI stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

Schedule-Free Learning has shown promise as a practical anytime training method for machine learning, showing success across dozens of standard benchmark problems. However, strong performance for LLM training has only been demonstrated at small scales. We identify a number of fixes necessary to scale up Schedule-Free Learning to larger batch sizes and model sizes, and present a learning-rate-free and schedule-free method (ScheduleFree+) for training large language models which greatly outperforms Warmup-Stable-Decay (WSD) schedules. We also demonstrate that Schedule-Free Learning is most effective for long duration training, and at 1000 tokens per parameter, it outperforms SOTA schedules by 31%. Schedule-Free Learning provides a theoretical foundation for the use of model averaging and checkpoint merging during pretraining.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 35 of its 46 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

35 of 46 references.

References · 35

  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  2. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  4. 2021Scaling Vision Transformerscited by 999
  5. 2021Robust fine-tuning of zero-shot modelscited by 797
  6. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  7. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  8. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  9. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
  10. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  11. 2022Patching open-vocabulary models by interpolating weightscited by 190
  12. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
23 more
  1. 2024The Road Less Scheduledcited by 163
  2. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 133
  3. 2017L2 Regularization versus Batch and Weight Normalizationcited by 124
  4. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 121
  5. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 117
  6. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  7. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  8. 2024How Does Critical Batch Size Scale in Pre-training?cited by 73
  9. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 53
  10. 2023Optimal Linear Decay Learning Rate Schedules and Further Refinementscited by 43
  11. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 41
  12. 2025Model Merging in Pre-training of Large Language Modelscited by 37
  13. 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
  14. 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 35
  15. 2025The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Trainingcited by 33
  16. 2025Why Gradients Rapidly Increase Near the End of Trainingcited by 22
  17. 2023Training trajectories, mini-batch losses and the curious role of the learning ratecited by 21
  18. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  19. 2025Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variantscited by 10
  20. 2025Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Trainingcited by 9
  21. 2020The Power of Factorial Powers: New Parameter settings for (Stochastic) Optimizationcited by 8
  22. 2026Universal Dynamics of Warmup Stable Decay: understanding WSD beyond Transformerscited by 5
  23. 2025Taking the Road Less Scheduled with Adaptive Polyak Stepscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026A Defense of the Quadratic Modelcited by 14
  2. 2026AMUSE: Anytime Muon with Stable Gradient Evaluationcited by 4
  3. 2026Hyperball May Not Be a Free Lunchcited by 4
  4. 2026Optimizer Memory Schedules for Outscaling the Overtraining Axiscited by 1
  5. 2026WSqD: A Horizon-Free Learning Rate Schedule for Large Model Trainingcited by 0
  6. 2026Blog: Survey of Optimizerscited by 0
  7. 2026Optimizers for Diffusion Models: A Controlled Benchmarkcited by 0