mlsanityData to 14 Oct 2026

Measuring the Effects of Data Parallelism on Neural Network Training

arXiv 1811.03600 ↗First posted 8 Nov 2018cs.LG stat.MLcited by 193

Not on the map. Why?

Abstract (arXiv)

Recent hardware developments have dramatically increased the scale of data parallelism available for neural network training. Among the simplest ways to harness next-generation hardware is to increase the batch size in standard mini-batch neural network training algorithms. In this work, we aim to experimentally characterize the effects of increasing the batch size on training time, as measured by the number of steps necessary to reach a goal out-of-sample error. We study how this relationship varies with the training algorithm, model, and data set, and find extremely large variation between workloads. Along the way, we show that disagreements in the literature on how batch size affects model quality can largely be explained by differences in metaparameter tuning and compute budgets at different batch sizes. We find no evidence that larger batch sizes degrade out-of-sample performance. Finally, we discuss the implications of our results on efforts to train neural networks much faster in the future. Our experimental data is publicly available as a database of 71,638,836 loss measurements taken over the course of training for 168,160 individual models across 35 workloads.

Key facts. Cited by 193 papers in the map (data through 14 Oct 2026). First posted on arXiv 8 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 39; 2022: 30; 2023: 31; 2024: 34; 2025: 27; 2026: 32.

193 papers.

Sort:

By area:

  1. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  2. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  3. 2021Federated Learning on Non-IID Data: A Surveycited by 346
  4. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  5. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  6. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
  7. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 190
  8. 2023Efficient Large Language Models: A Surveycited by 186
  9. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  10. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  11. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  12. 2021Large-Scale Differentially Private BERTcited by 118
38 more
  1. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  2. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 109
  3. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 88
  4. 2022Pathways: Asynchronous Distributed Dataflow for MLcited by 82
  5. 2025Practical Efficiency of Muon for Pretrainingcited by 80
  6. 2021On Large-Cohort Training for Federated Learningcited by 74
  7. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  8. 2024How Does Critical Batch Size Scale in Pre-training?cited by 73
  9. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  10. 2023Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Modelscited by 61
  11. 2023Elo Uncovered: Robustness and Best Practices in Language Model Evaluationcited by 61
  12. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 60
  13. 2021Randomness In Neural Network Training: Characterizing The Impact of Toolingcited by 59
  14. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  15. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 53
  16. 2021Consensus Control for Decentralized Deep Learningcited by 49
  17. 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 43
  18. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 43
  19. 2024Deconstructing What Makes a Good Optimizer for Language Modelscited by 41
  20. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 41
  21. 2021A Loss Curvature Perspective on Training Instability in Deep Learningcited by 40
  22. 2023CAME: Confidence-guided Adaptive Memory Efficient Optimizationcited by 36
  23. 2023Small batch deep reinforcement learningcited by 36
  24. 2021Batch size-invariance for policy optimizationcited by 35
  25. 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 35
  26. 2022On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Modelscited by 34
  27. 2023A Path to Simpler Models Starts With Noisecited by 33
  28. 2021Implicit Gradient Alignment in Distributed and Federated Learningcited by 29
  29. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  30. 2025Convergence Bound and Critical Batch Size of Muon Optimizercited by 28
  31. 2023CASSINI: Network-Aware Job Scheduling in Machine Learning Clusterscited by 26
  32. 2022On the Generalization Mystery in Deep Learningcited by 25
  33. 2024Unraveling the Mystery of Scaling Laws: Part Icited by 24
  34. 2022Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensionscited by 23
  35. 2024Surge Phenomenon in Optimal Learning Rate and Batch Size Scalingcited by 22
  36. 2024Analyzing & Reducing the Need for Learning Rate Warmup in GPT Trainingcited by 22
  37. 2022Reproducibility in Optimization: Theoretical Framework and Limitscited by 21
  38. 2021A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizescited by 20

50 most cited of 193 All 193 (JSON)