mlsanityData to 7 Oct 2026

BAGUA: Scaling up Distributed Learning with System Relaxations

arXiv 2107.01499 ↗First posted 3 Jul 2021cs.LG cs.DCcited by 13

Not on the map. Why?

Abstract (arXiv)

Recent years have witnessed a growing list of systems for distributed data-parallel training. Existing systems largely fit into two paradigms, i.e., parameter server and MPI-style collective operations. On the algorithmic side, researchers have proposed a wide range of techniques to lower the communication via system relaxations: quantization, decentralization, and communication delay. However, most, if not all, existing systems only rely on standard synchronous and asynchronous stochastic gradient (SG) based optimization, therefore, cannot take advantage of all possible optimizations that the machine learning community has been developing recently. Given this emerging gap between the current landscapes of systems and theory, we build BAGUA, a MPI-style communication library, providing a collection of primitives, that is both flexible and modular to support state-of-the-art system relaxation techniques of distributed training. Powered by this design, BAGUA has a great ability to implement and extend various state-of-the-art distributed learning algorithms. In a production cluster with up to 16 machines (128 GPUs), BAGUA can outperform PyTorch-DDP, Horovod and BytePS in the end-to-end training time by a significant margin (up to 2 times) across a diverse range of tasks. Moreover, we conduct a rigorous tradeoff exploration showing that different algorithms and system relaxations achieve the best performance over different network conditions.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 51 of its 89 references that are in the map. First posted on arXiv 3 Jul 2021.

Builds on

51 of 89 references.

References · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  9. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  10. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
  11. 2018Sparsified SGD with Memorycited by 373
  12. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
38 more
  1. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  2. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 253
  3. 2018The Convergence of Sparsified Gradient Methodscited by 228
  4. 2019Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communicationcited by 218
  5. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
  6. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 199
  7. 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 179
  8. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  9. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  10. 2017Asynchronous Decentralized Parallel Stochastic Gradient Descentcited by 159
  11. 2018D2: Decentralized Training over Decentralized Datacited by 140
  12. 2020On Biased Compression for Distributed Learningcited by 140
  13. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 135
  14. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 133
  15. 2018AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scalecited by 125
  16. 2019Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronizationcited by 106
  17. 2016Asynchronous Stochastic Gradient Descent with Delay Compensationcited by 98
  18. 2019DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compressioncited by 98
  19. 2018SGD and Hogwild! Convergence Without the Bounded Gradients Assumptioncited by 94
  20. 2018Communication Compression for Decentralized Trainingcited by 87
  21. 2019Communication-efficient distributed SGD with Sketchingcited by 81
  22. 2018Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGDcited by 76
  23. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 74
  24. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 74
  25. 2017Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusterscited by 55
  26. 2018Supporting Very Large Models using Automatic Dataflow Graph Partitioningcited by 43
  27. 2019Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Toolscited by 40
  28. 2016Efficient Distributed Learning with Sparsitycited by 38
  29. 2018Rafiki: Machine Learning as an Analytics Service Systemcited by 33
  30. 2018Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Trainingcited by 28
  31. 2021PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformerscited by 23
  32. 2021Distributed Learning Systems with First-order Methodscited by 17
  33. 2015SparkNet: Training Deep Networks in Sparkcited by 13
  34. 2019Declarative Recursive Computation on an RDBMS, or, Why You Should Use a Database For Distributed Machine Learningcited by 12
  35. 2020Tensor Relational Algebra for Machine Learning System Designcited by 11
  36. 2014High-Performance Distributed ML at Scale through Parameter Server Consistency Modelscited by 10
  37. 2017Asynchronous Distributed Variational Gaussian Processes for Regressioncited by 7
  38. 2017A Cost-based Optimizer for Gradient Descent Optimizationcited by 4

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 4; 2023: 2; 2024: 3; 2025: 0; 2026: 1.

13 papers.

Sort:

By area:

  1. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  2. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 29
  3. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  4. 2021Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameterscited by 17
  5. 2021BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learningcited by 16
  6. 2022Data Debugging with Shapley Importance over End-to-End Machine Learning Pipelinescited by 16
  7. 2022Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Modelscited by 11
  8. 2024Training and Serving System of Foundation Models: A Comprehensive Surveycited by 11
  9. 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Servingcited by 7
  10. 2021CGX: Adaptive System Support for Communication-Efficient Deep Learningcited by 4
  11. 2022Optimal Complexity in Non-Convex Decentralized Learning over Time-Varying Networkscited by 4
  12. 2024Investigation of Energy-efficient AI Model Architectures and Compression Techniques for "Green" Fetal Brain Segmentationcited by 1
1 more
  1. 2024Fully First-Order Methods for Decentralized Bilevel Optimizationcited by 1