mlsanityData to 14 Oct 2026

On the Utility of Gradient Compression in Distributed Training Systems

arXiv 2103.00543 ↗First posted 28 Feb 2021cs.DC cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

A rich body of prior work has highlighted the existence of communication bottlenecks in synchronous data-parallel training. To alleviate these bottlenecks, a long line of recent work proposes gradient and model compression methods. In this work, we evaluate the efficacy of gradient compression methods and compare their scalability with optimized implementations of synchronous data-parallel SGD across more than 200 different setups. Surprisingly, we observe that only in 6 cases out of more than 200, gradient compression methods provide speedup over optimized synchronous data-parallel training in the typical data-center setting. We conduct an extensive investigation to identify the root causes of this phenomenon, and offer a performance model that can be used to identify the benefits of gradient compression for a variety of system setups. Based on our analysis, we propose a list of desirable properties that gradient compression methods should satisfy, in order for them to provide a meaningful end-to-end speedup.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 65 of its 118 references that are in the map. First posted on arXiv 28 Feb 2021.

Builds on

65 of 118 references.

References · 65

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  9. 2017Mixed Precision Trainingcited by 1,156
  10. 2019Towards Federated Learning at Scale: System Designcited by 832
  11. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  12. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 664
38 more
  1. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  2. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  3. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 502
  4. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
  5. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  6. 2019How to Fine-Tune BERT for Text Classification?cited by 411
  7. 2018Sparsified SGD with Memorycited by 376
  8. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  9. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 310
  10. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
  11. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 256
  12. 2017Sparse Communication for Distributed Gradient Descentcited by 249
  13. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
  14. 2018The Convergence of Sparsified Gradient Methodscited by 229
  15. 2019Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communicationcited by 221
  16. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 201
  17. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
  18. 2018D2: Decentralized Training over Decentralized Datacited by 140
  19. 2016Distributed Mean Estimation with Limited Communicationcited by 136
  20. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 136
  21. 2019MLPerf Training Benchmarkcited by 115
  22. 2019Decentralized Deep Learning with Arbitrary Communication Compressioncited by 111
  23. 2019DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compressioncited by 99
  24. 2019Natural Compression for Distributed Deep Learningcited by 90
  25. 2018Communication Compression for Decentralized Trainingcited by 88
  26. 2018Hessian-based Analysis of Large Batch Training and Robustness to Adversariescited by 87
  27. 2019Machine Learning at the Network Edge: A Surveycited by 84
  28. 2019Communication-efficient distributed SGD with Sketchingcited by 81
  29. 2018Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimizationcited by 76
  30. 2019Scaling Distributed Machine Learning with In-Network Aggregationcited by 65
  31. 20158-Bit Approximations for Parallelism in Deep Learningcited by 56
  32. 2019Understanding Top-k Sparsification in Distributed Deep Learningcited by 56
  33. 2018Sparse Binary Compression: Towards Distributed Deep Learning with minimal Communicationcited by 51
  34. 2019Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedbackcited by 50
  35. 2019A Distributed Synchronous SGD Algorithm with Global Top-k Sparsification for Low Bandwidth Networkscited by 37
  36. 2021An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systemscited by 32
  37. 2017AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networkscited by 29
  38. 2018TicTac: Accelerating Distributed Deep Learning with Communication Schedulingcited by 27

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 4; 2023: 11; 2024: 3; 2025: 1; 2026: 4.

26 papers.

Sort:

By area:

  1. 2021A Field Guide to Federated Optimizationcited by 315
  2. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  3. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  4. 2023Just One Byte (per gradient): A Note on Low-Bandwidth Decentralized Language Model Finetuning Using Shared Randomnesscited by 18
  5. 2022FedLite: A Scalable Approach for Federated Learning on Resource-constrained Clientscited by 15
  6. 2023Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compressioncited by 15
  7. 2023THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compressioncited by 15
  8. 2023Does compressing activations help model parallel training?cited by 12
  9. 2021JUWELS Booster -- A Supercomputer for Large-Scale AI Researchcited by 10
  10. 2023GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Trainingcited by 10
  11. 2023Adaptive Message Quantization and Parallelization for Distributed Full-graph GNN Trainingcited by 10
  12. 2022Distributed Learning With Sparsified Gradient Differencescited by 9
14 more
  1. 2023Evaluation and Optimization of Gradient Compression for Distributed Deep Learningcited by 7
  2. 2023Accelerating Distributed ML Training via Selective Synchronizationcited by 7
  3. 2024Beyond Throughput and Compression Ratios: Towards High End-to-end Utility of Gradient Compressioncited by 6
  4. 2025OmniLearn: A Framework for Distributed Deep Learning over Heterogeneous Clusterscited by 5
  5. 2021CGX: Adaptive System Support for Communication-Efficient Deep Learningcited by 4
  6. 2026DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reducecited by 4
  7. 2023Flexible Communication for Optimal Distributed Learning over Unpredictable Networkscited by 3
  8. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applicationscited by 2
  9. 2022ByteComp: Revisiting Gradient Compression in Distributed Trainingcited by 1
  10. 2023Quantize Once, Train Fast: Allreduce-Compatible Compression with Provable Guaranteescited by 0
  11. 2024Breaking MLPerf Training: A Case Study on Optimizing BERTcited by 0
  12. 2024Approximated Likelihood Ratio: A Forward-Only and Parallel Framework for Boosting Neural Network Trainingcited by 0
  13. 2026Prof-K: Probabilistic One-Pass Filtering for Efficient Top-k Selectioncited by 0
  14. 2026CIERA: Cross-Iteration Exponent Reuse for Lossless Allgather in Sharded MoE Trainingcited by 0