mlsanityData to 14 Oct 2026

Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training

arXiv 2505.13738 ↗First posted 19 May 2025cs.LG cs.AI cs.CLcited by 43

Not on the map. Why?

Abstract (arXiv)

Efficient LLM pre-training requires well-tuned hyperparameters (HPs), including learning rate η and weight decay λ. We study scaling laws for HPs: formulas for how to scale HPs as we scale model size N, dataset size D, and batch size B. Recent work suggests the AdamW timescale, τ = B/(η λ D), should remain constant across training settings, and we verify the implication that optimal λ scales linearly with B, for a fixed N and D. However, as N and D scale, we show optimal τ obeys a precise power law in the tokens-per-parameter ratio, D/N. This law thus provides a method to accurately predict λopt in advance of large-scale training. We also study scaling laws for optimal batch size Bopt (the B enabling lowest loss at a given N,D) and critical batch size Bcrit (the B beyond which further data parallelism becomes ineffective). In contrast to prior work, we find both Bopt and Bcrit scale as power laws in D, independent of model size, N. Finally, we analyze how these findings inform the real-world selection of Pareto-optimal N and D under dual training time and compute objectives. All experiments were run on Cerebras CS-3 systems.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 61 of its 82 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

61 of 82 references.

References · 61

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2020GLU Variants Improve Transformercited by 1,804
  8. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  11. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  12. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
38 more
  1. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  2. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  3. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  4. 2017Large Batch Training of Convolutional Networkscited by 535
  5. 2023The Falcon Series of Open Language Modelscited by 535
  6. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  7. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  8. 2018An Empirical Model of Large-Batch Trainingcited by 281
  9. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  10. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  11. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  12. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
  13. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
  14. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  15. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  16. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
  17. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  18. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  19. 2017Revisiting Distributed Synchronous SGDcited by 142
  20. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 135
  21. 2017L2 Regularization versus Batch and Weight Normalizationcited by 124
  22. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  23. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 117
  24. 2018Three Mechanisms of Weight Decay Regularizationcited by 113
  25. 2024Language models scale reliably with over-training and on downstream taskscited by 113
  26. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 109
  27. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
  28. 2024Scaling Laws for Precisioncited by 95
  29. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 88
  30. 2024Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecited by 81
  31. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 78
  32. 2024How Does Critical Batch Size Scale in Pre-training?cited by 73
  33. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 72
  34. 2024Chinchilla Scaling: A replication attemptcited by 72
  35. 2023Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Modelscited by 69
  36. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  37. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 53
  38. 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 43

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 30.

43 papers.

Sort:

By area:

  1. 2025Scaling Behavior of Discrete Diffusion Language Modelscited by 35
  2. 2025Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Durationcited by 27
  3. 2025Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Trainingcited by 23
  4. 2025Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networkscited by 22
  5. 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 21
  6. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  7. 2025MuLoCo: Muon is a practical inner optimizer for DiLoCocited by 18
  8. 2026Hyperparameter Transfer with Mixture-of-Expert Layerscited by 13
  9. 2026Rethinking Language Model Scaling under Transferable Hypersphere Optimizationcited by 12
  10. 2026Deriving Hyperparameter Scaling Laws via Modern Optimization Theorycited by 10
  11. 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 9
  12. 2025Optimal Scaling Needs Optimal Normcited by 8
31 more
  1. 2026On the Role of Batch Size in Stochastic Conditional Gradient Methodscited by 7
  2. 2025Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Schedulingcited by 6
  3. 2025Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Trainingcited by 6
  4. 2026The Design Space of Tri-Modal Masked Diffusion Modelscited by 6
  5. 2026Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficientscited by 5
  6. 2025Efficient Hyperparameter Tuning via Trajectory Invariance Principlecited by 4
  7. 2026Weight Decay Improves Language Model Plasticitycited by 4
  8. 2026Hyperball May Not Be a Free Lunchcited by 4
  9. 2026Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectorscited by 3
  10. 2025Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMscited by 2
  11. 2026Configuration-to-Performance Scaling Law with Neural Ansatzcited by 2
  12. 2026Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Ratecited by 2
  13. 2026On the Nonlinearity of Learning Rate Scaling for LLM Trainingcited by 2
  14. 2026Small-Scale Experiments: Are We There Yet?cited by 2
  15. 2026GQA-μP: The maximal parameterization update for grouped query attentioncited by 1
  16. 2026Asymmetric Scaling Laws from Sparse Featurescited by 1
  17. 2026On the Optimizer Dependence of Neural Scaling Lawscited by 1
  18. 2026Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Losscited by 1
  19. 2026How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Sizecited by 1
  20. 2026Towards joint scaling laws with optimal batch size schedulescited by 1
  21. 2026Abra: Scaling Diffusion Image Trainingcited by 1
  22. 2026Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Losscited by 1
  23. 2026Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolationcited by 0
  24. 2026Predicting Large Model Test Losses with a Noisy Quadratic Systemcited by 0
  25. 2026Validity Threats for Foundation Model Researchcited by 0
  26. 2026Amortizing Scaling Law Construction Costscited by 0
  27. 2026Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inferencecited by 0
  28. 2026Hyperparameter Scaling Laws Across MoE Sparsitycited by 0
  29. 2026Emergent One-Third Scaling Law as Attention Tries to Concentratecited by 0
  30. 2026Balancing Early Performance Sacrifices with Long-Term Gains: Scaling Learning-Rate Warmup Duration Across Training Horizonscited by 0
  31. 2026Optimizer-dependent training dynamics converge to the same one-third optimal data scalingcited by 0