mlsanityData to 14 Oct 2026

Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations

arXiv 2405.18392 ↗First posted 28 May 2024cs.LGcited by 170

Not on the map. Why?

Abstract (arXiv)

Scale has become a main ingredient in obtaining strong machine learning models. As a result, understanding a model's scaling properties is key to effectively designing both the right training setup as well as future generations of architectures. In this work, we argue that scale and training research has been needlessly complex due to reliance on the cosine schedule, which prevents training across different lengths for the same model size. We investigate the training behavior of a direct alternative -- constant learning rate and cooldowns -- and find that it scales predictably and reliably similar to cosine. Additionally, we show that stochastic weight averaging yields improved performance along the training trajectory, without additional training costs, across different scales. Importantly, with these findings we demonstrate that scaling experiments can be performed with significantly reduced compute and GPU hours by utilizing fewer but reusable training runs. Our code is available at urlhttps://github.com/epfml/schedules-and-scaling/.

Key facts. Cited by 170 papers in the map (data through 14 Oct 2026). Builds on 51 of its 72 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

51 of 72 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  12. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
38 more
  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  6. 2022Emergent Abilities of Large Language Modelscited by 2,512
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  10. 2020GLU Variants Improve Transformercited by 1,804
  11. 2019Root Mean Square Layer Normalizationcited by 1,480
  12. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  13. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  14. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  15. 2023PaLM 2 Technical Reportcited by 1,162
  16. 2017Mixed Precision Trainingcited by 1,156
  17. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  18. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  19. 2023Scaling Data-Constrained Language Modelscited by 403
  20. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  21. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
  22. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  23. 2024The Road Less Scheduledcited by 163
  24. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  25. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 137
  26. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 134
  27. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 127
  28. 2024Language models scale reliably with over-training and on downstream taskscited by 113
  29. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 106
  30. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  31. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 77
  32. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 72
  33. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Modelscited by 67
  34. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollarscited by 57
  35. 2022Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averagingcited by 54
  36. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  37. 2023Early Weight Averaging meets High Learning Rates for LLM Pre-trainingcited by 37
  38. 2023Training trajectories, mini-batch losses and the curious role of the learning ratecited by 21

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 20; 2025: 68; 2026: 82.

170 papers.

Sort:

By area:

  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  2. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  3. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  4. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 100
  5. 2024Scaling Laws for Precisioncited by 95
  6. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 88
  7. 2025REVE: A Foundation Model for EEG -- Adapting to Any Setup with Large-Scale Pretraining on 25,000 Subjectscited by 80
  8. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 72
  9. 2025GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generationcited by 57
  10. 2024The AdEMAMix Optimizer: Better, Faster, Oldercited by 54
  11. 2025Scaling Laws for Native Multimodal Modelscited by 50
  12. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 47
38 more
  1. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 43
  2. 2024Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMscited by 40
  3. 2024Reconciling Kaplan and Chinchilla Scaling Lawscited by 39
  4. 2025Model Merging in Pre-training of Large Language Modelscited by 37
  5. 2025Walrus: A Cross-Domain Foundation Model for Continuum Dynamicscited by 36
  6. 2025Scaling Laws for Forgetting during Finetuning with Pretraining Data Injectioncited by 35
  7. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 35
  8. 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 35
  9. 2025MuonBP: Faster Muon via Block-Periodic Orthogonalizationcited by 35
  10. 2025Scaling Behavior of Discrete Diffusion Language Modelscited by 35
  11. 2025The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Trainingcited by 33
  12. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficientcited by 31
  13. 2024Scaling Law with Learning Rate Annealingcited by 29
  14. 2024Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Schedulercited by 28
  15. 2025BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLPcited by 28
  16. 2025(Mis)Fitting: A Survey of Scaling Lawscited by 26
  17. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 23
  18. 2024Scaling Laws for Pre-training Agents and World Modelscited by 23
  19. 2025WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-trainingcited by 23
  20. 2025Implicit Language Models are RNNs: Balancing Parallelization and Expressivitycited by 22
  21. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  22. 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 22
  23. 2025A Survey on LLM Mid-Trainingcited by 21
  24. 2025Gemstones: A Model Suite for Multi-Faceted Scaling Lawscited by 20
  25. 2025Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenizationcited by 20
  26. 2024Improving Text-To-Audio Models with Synthetic Captionscited by 19
  27. 2025MuLoCo: Muon is a practical inner optimizer for DiLoCocited by 18
  28. 2025Closing the Gap Between Text and Speech Understanding in LLMscited by 17
  29. 2025Data Scaling Laws for End-to-End Autonomous Drivingcited by 15
  30. 2025Is your batch size the problem? Revisiting the Adam-SGD gap in language modelingcited by 15
  31. 2025Training Dynamics Impact Post-Training Quantization Robustnesscited by 15
  32. 2025How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretrainingcited by 14
  33. 2026A Defense of the Quadratic Modelcited by 14
  34. 2025Scaling Laws for Upcycling Mixture-of-Experts Language Modelscited by 13
  35. 2025Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Modelscited by 13
  36. 2026Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuningcited by 13
  37. 2024Small Languages, Big Models: A Study of Continual Training on Languages of Norwaycited by 12
  38. 2025Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Schedulercited by 12

50 most cited of 170 All 170 (JSON)