mlsanityData to 7 Oct 2026

An Empirical Model of Large-Batch Training

arXiv 1812.06162 ↗First posted 14 Dec 2018cs.LG stat.MLcited by 268

Not on the map. Why?

Abstract (arXiv)

In an increasing number of domains it has been demonstrated that deep learning models can be trained using relatively large batch sizes without sacrificing data efficiency. However the limits of this massive data parallelism seem to differ from domain to domain, ranging from batches of tens of thousands in ImageNet to batches of millions in RL agents that play the game Dota 2. To our knowledge there is limited conceptual understanding of why these limits to batch size differ or how we might choose the correct batch size in a new domain. In this paper, we demonstrate that a simple and easy-to-measure statistic called the gradient noise scale predicts the largest useful batch size across many domains and applications, including a number of supervised learning datasets (MNIST, SVHN, CIFAR-10, ImageNet, Billion Word), reinforcement learning domains (Atari and Dota), and even generative model training (autoencoders on SVHN). We find that the noise scale increases as the loss decreases over a training run and depends on the model size primarily through improved model performance. Our empirically-motivated theory also describes the tradeoff between compute-efficiency and time-efficiency, and provides a rough model of the benefits of adaptive batch-size training.

Key facts. Cited by 268 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 25; 2023: 22; 2024: 47; 2025: 80; 2026: 73.

268 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  6. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  7. 2023A Comprehensive Overview of Large Language Modelscited by 639
  8. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  9. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  10. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  11. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  12. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
38 more
  1. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  2. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  3. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  4. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 106
  5. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  6. 2024Stable LM 2 1.6B Technical Reportcited by 86
  7. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 84
  8. 2022Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Trainingcited by 78
  9. 2025Practical Efficiency of Muon for Pretrainingcited by 74
  10. 2021On Large-Cohort Training for Federated Learningcited by 73
  11. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  12. 2024How Does Critical Batch Size Scale in Pre-training?cited by 68
  13. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  14. 2021Scaling Scaling Laws with Board Gamescited by 62
  15. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  16. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 59
  17. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  18. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 50
  19. 2025Intern-S1: A Scientific Multimodal Foundation Modelcited by 49
  20. 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 42
  21. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 41
  22. 2025AlphaFlow: Understanding and Improving MeanFlow Modelscited by 40
  23. 2021A Loss Curvature Perspective on Training Instability in Deep Learningcited by 39
  24. 2022Self-Supervised Learning with an Information Maximization Criterioncited by 38
  25. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 38
  26. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  27. 2025Model Merging in Pre-training of Large Language Modelscited by 36
  28. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 36
  29. 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 34
  30. 2021Batch size-invariance for policy optimizationcited by 33
  31. 2025Scaling Behavior of Discrete Diffusion Language Modelscited by 32
  32. 2024The Future of Large Language Model Pre-training is Federatedcited by 30
  33. 2025Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretrainingcited by 30
  34. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficientcited by 29
  35. 2021What can linear interpolation of neural network loss landscapes tell us?cited by 27
  36. 2026PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competencecited by 26
  37. 2024Unraveling the Mystery of Scaling Laws: Part Icited by 24
  38. 2024Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Modelscited by 24

50 most cited of 268 All 268 (JSON)