Towards joint scaling laws with optimal batch size schedules
Not on the map. Why?
Abstract (arXiv)
Modern deep learning typically keeps the batch size static throughout training, thus overlooking the joint effect of learning rate and batch size on the training dynamics. In this paper, we study the deep learning dynamics through the lens of convex optimization and derive a joint characterization of loss in terms of both schedules, applicable to general optimizers and model architectures. This characterization yields a closed-form optimal batch size schedule for any prescribed learning rate schedule, and further leads to joint scaling laws that consistently outperform static batch size baselines, highlighting the significance of dynamic batch size schedule in large language model training.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 36 of its 41 references that are in the map. First posted on arXiv 30 Jul 2026.
Builds on
36 of 41 references.
References · 36
- 2017Attention Is All You Needcited by 60,361
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2025Qwen3 Technical Reportcited by 8,820
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
- 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
- 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
24 more
- 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
- 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
- 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
- 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
- 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
- 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
- 2018An Empirical Model of Large-Batch Trainingcited by 281
- 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
- 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
- 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 97
- 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 78
- 2024How Does Critical Batch Size Scale in Pre-training?cited by 73
- 2023Optimal Linear Decay Learning Rate Schedules and Further Refinementscited by 43
- 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 43
- 2024Scaling Optimal LR Across Token Horizonscited by 40
- 2025The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Trainingcited by 33
- 2016Coupling Adaptive Batch Sizes with Learning Ratescited by 30
- 2018On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descentcited by 25
- 2025Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Trainingcited by 23
- 2024Scaling Law for Language Models Training Considering Batch Sizecited by 22
- 2025Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networkscited by 22
- 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 9
- 2025Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Schedulingcited by 6
- 2026Convex Dominance in Deep Learning I: A Scaling Law of Loss and Learning Ratecited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.