mlsanityData to 14 Oct 2026

How Does Critical Batch Size Scale in Pre-training?

arXiv 2410.21676 ↗First posted 29 Oct 2024cs.LG cs.AI math.OCcited by 73

Not on the map. Why?

Abstract (arXiv)

Training large-scale models under given resources requires careful design of parallelism strategies. In particular, the efficiency notion of critical batch size (CBS), concerning the compromise between time and compute, marks the threshold beyond which greater data parallelism leads to diminishing returns. To operationalize it, we propose a measure of CBS and pre-train a series of auto-regressive language models, ranging from 85 million to 1.2 billion parameters, on the C4 dataset. Through extensive hyper-parameter sweeps and careful control of factors such as batch size, momentum, and learning rate along with its scheduling, we systematically investigate the impact of scale on CBS. Then we fit scaling laws with respect to model and data sizes to decouple their effects. Overall, our results demonstrate that CBS scales primarily with data size rather than model size, a finding we justify theoretically through the analysis of infinite-width limits of neural networks and infinite-dimensional least squares regression. Of independent interest, we highlight the importance of common hyper-parameter choices and strategies for studying large-scale pre-training beyond fixed training durations.

Key facts. Cited by 73 papers in the map (data through 14 Oct 2026). Builds on 51 of its 66 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

51 of 66 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  9. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  10. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  11. 2021Scaling Vision Transformerscited by 999
  12. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
38 more
  1. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  2. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  3. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  4. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  5. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  6. 2017Large Batch Training of Convolutional Networkscited by 535
  7. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  8. 2023Scaling Data-Constrained Language Modelscited by 403
  9. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 402
  10. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  11. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  12. 2018An Empirical Model of Large-Batch Trainingcited by 281
  13. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
  14. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  15. 2024The Road Less Scheduledcited by 163
  16. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
  17. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  18. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 135
  19. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 117
  20. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  21. 2024A Dynamical Model of Neural Scaling Lawscited by 111
  22. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 109
  23. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
  24. 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 97
  25. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 88
  26. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  27. 2024Chinchilla Scaling: A replication attemptcited by 72
  28. 2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
  29. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 68
  30. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 66
  31. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 57
  32. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 53
  33. 2023Feature-Learning Networks Are Consistent Across Widths At Realistic Scalescited by 41
  34. 2024Infinite Limits of Multi-head Transformer Dynamicscited by 39
  35. 2024Reconciling Kaplan and Chinchilla Scaling Lawscited by 39
  36. 2021Batch size-invariance for policy optimizationcited by 35
  37. 2018On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descentcited by 25
  38. 2023BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Modelcited by 19

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 30; 2026: 42.

73 papers.

Sort:

By area:

  1. 2025Practical Efficiency of Muon for Pretrainingcited by 80
  2. 2025Dion: Distributed Orthonormalized Updatescited by 68
  3. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 45
  4. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 43
  5. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 41
  6. 2025In Search of Adam's Secret Saucecited by 40
  7. 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 35
  8. 2025Scaling Behavior of Discrete Diffusion Language Modelscited by 35
  9. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 35
  10. 2025Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretrainingcited by 34
  11. 2025Convergence Bound and Critical Batch Size of Muon Optimizercited by 28
  12. 2025Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Trainingcited by 23
38 more
  1. 2025WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-trainingcited by 23
  2. 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 19
  3. 2025The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newtoncited by 17
  4. 2025Is your batch size the problem? Revisiting the Adam-SGD gap in language modelingcited by 15
  5. 2026A Defense of the Quadratic Modelcited by 14
  6. 2025Towards a Comprehensive Scaling Law of Mixture-of-Expertscited by 10
  7. 2026Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averagingcited by 10
  8. 2026Deriving Hyperparameter Scaling Laws via Modern Optimization Theorycited by 10
  9. 2025Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Trainingcited by 9
  10. 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 9
  11. 2025Training and Inference Efficiency of Encoder-Decoder Speech Modelscited by 8
  12. 2025Optimal Scaling Needs Optimal Normcited by 8
  13. 2026On the Role of Batch Size in Stochastic Conditional Gradient Methodscited by 7
  14. 2026ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Modelscited by 7
  15. 2025Optimal Embedding Learning Rate in LLMs: The Effect of Vocabulary Sizecited by 6
  16. 2025Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Schedulingcited by 6
  17. 2025Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Trainingcited by 6
  18. 2026Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Lawscited by 6
  19. 2026IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RLcited by 6
  20. 2026VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenizationcited by 6
  21. 2026The Recurrent Transformer: Greater Effective Depth and Efficient Decodingcited by 6
  22. 2026Benchmarking Optimizers for MLPs in Tabular Deep Learningcited by 5
  23. 2024Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelismcited by 4
  24. 2025DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Modelscited by 4
  25. 2025Efficient Hyperparameter Tuning via Trajectory Invariance Principlecited by 4
  26. 2026The Effect of Mini-Batch Noise on the Implicit Bias of Adamcited by 4
  27. 2026Beware of the Batch Size: Hyperparameter Bias in Evaluating LoRAcited by 4
  28. 2026Weight Decay Improves Language Model Plasticitycited by 4
  29. 2025EMA Without the Lag: Bias-Corrected Iterate Averaging Schemescited by 3
  30. 2025A Proof of Learning Rate Transfer under μPcited by 3
  31. 2026Anytime Training with Schedule-Free Spectral Optimizationcited by 3
  32. 2025Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMscited by 2
  33. 2025MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updatescited by 2
  34. 2026Configuration-to-Performance Scaling Law with Neural Ansatzcited by 2
  35. 2026Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Ratescited by 2
  36. 2026Compute Efficiency and Serial Runtime Tradeoffs for Stochastic Momentum Methodscited by 2
  37. 2026On the Nonlinearity of Learning Rate Scaling for LLM Trainingcited by 2
  38. 2025Understanding SGD with Exponential Moving Average: A Case Study in Linear Regressioncited by 1

50 most cited of 73 All 73 (JSON)