mlsanityData to 14 Oct 2026

DropCompute: simple and more robust distributed synchronous training via compute variance reduction

arXiv 2306.10598 ↗First posted 18 Jun 2023cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Background: Distributed training is essential for large scale training of deep neural networks (DNNs). The dominant methods for large scale DNN training are synchronous (e.g. All-Reduce), but these require waiting for all workers in each step. Thus, these methods are limited by the delays caused by straggling workers. Results: We study a typical scenario in which workers are straggling due to variability in compute time. We find an analytical relation between compute time properties and scalability limitations, caused by such straggling workers. With these findings, we propose a simple yet effective decentralized method to reduce the variation among workers and thus improve the robustness of synchronous training. This method can be integrated with the widely used All-Reduce. Our findings are validated on large-scale training tasks using 200 Gaudi Accelerators.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 41 of its 66 references that are in the map. First posted on arXiv 18 Jun 2023.

Builds on

41 of 66 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  12. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
29 more
  1. 2019Introduction to Online Convex Optimizationcited by 803
  2. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  3. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  4. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  5. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  6. 2017Large Batch Training of Convolutional Networkscited by 535
  7. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 502
  8. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  9. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
  10. 2018Scaling Neural Machine Translationcited by 204
  11. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
  12. 2020The Hardware Lotterycited by 160
  13. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 156
  14. 2018Stochastic Gradient Push for Distributed Deep Learningcited by 152
  15. 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 145
  16. 2017Revisiting Distributed Synchronous SGDcited by 142
  17. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  18. 2019MLPerf Training Benchmarkcited by 115
  19. 2014Deep learning with Elastic Averaging SGDcited by 90
  20. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 75
  21. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  22. 2017Collaborative Deep Learning in Fixed Topology Networkscited by 60
  23. 2017Online to Offline Conversions, Universality and Adaptive Minibatch Sizescited by 39
  24. 2016Asynchrony begets Momentum, with an Application to Deep Learningcited by 30
  25. 2021ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Trainingcited by 22
  26. 2019At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?cited by 16
  27. 2020Breaking (Global) Barriers in Parallel Stochastic Optimization with Wait-Avoiding Group Averagingcited by 9
  28. 2020Accelerated Large Batch Optimization of BERT Pretraining in 54 minutescited by 7
  29. 2019Learning Rate Dropoutcited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2025Understanding Stragglers in Large Model Training Using What-if Analysiscited by 12
  2. 2024From promise to practice: realizing high-performance decentralized trainingcited by 4
  3. 2025Efficient AllReduce with Stragglerscited by 1
  4. 2025Pseudo-Asynchronous Local SGD: Robust and Efficient Data-Parallel Trainingcited by 0