mlsanityData to 7 Oct 2026

Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

arXiv 1712.01887 ↗First posted 5 Dec 2017cs.CV cs.DC cs.LGcited by 471

Not on the map. Why?

Abstract (arXiv)

Large-scale distributed training requires significant communication bandwidth for gradient exchange that limits the scalability of multi-node training, and requires expensive high-bandwidth network infrastructure. The situation gets even worse with distributed training on mobile devices (federated learning), which suffers from higher latency, lower throughput, and intermittent poor connections. In this paper, we find 99.9% of the gradient exchange in distributed SGD is redundant, and propose Deep Gradient Compression (DGC) to greatly reduce the communication bandwidth. To preserve accuracy during compression, DGC employs four methods: momentum correction, local gradient clipping, momentum factor masking, and warm-up training. We have applied Deep Gradient Compression to image classification, speech recognition, and language modeling with multiple datasets including Cifar10, ImageNet, Penn Treebank, and Librispeech Corpus. On these scenarios, Deep Gradient Compression achieves a gradient compression ratio from 270x to 600x without losing accuracy, cutting the gradient size of ResNet-50 from 97MB to 0.35MB, and for DeepSpeech from 488MB to 0.74MB. Deep gradient compression enables large-scale distributed training on inexpensive commodity 1Gbps Ethernet and facilitates distributed training on mobile. Code is available at: https://github.com/synxlin/deep-gradient-compression.

Key facts. Cited by 471 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 120; 2022: 78; 2023: 76; 2024: 80; 2025: 57; 2026: 59.

471 papers.

Sort:

By area:

  1. 2021Towards Personalized Federated Learningcited by 462
  2. 2021A Field Guide to Federated Optimizationcited by 312
  3. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  4. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 140
  5. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
  6. 2022Vertical Federated Learning: Concepts, Advances and Challengescited by 104
  7. 2021Gradient Inversion with Generative Image Priorcited by 101
  8. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 92
  9. 2023Multimodal Federated Learning via Contrastive Representation Ensemblecited by 85
  10. 2022Survey on Federated Learning Threats: concepts, taxonomy on attacks and defences, experimental study and challengescited by 81
  11. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  12. 2022Auditing Privacy Defenses in Federated Learning via Generative Gradient Leakagecited by 72
38 more
  1. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  2. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  3. 2023AI-Generated Content (AIGC): A Surveycited by 59
  4. 2021Distributed Deep Learning in Open Collaborationscited by 58
  5. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 58
  6. 2021FedLab: A Flexible Federated Learning Frameworkcited by 54
  7. 2021EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learningcited by 53
  8. 2023Federated Continual Learning via Knowledge Fusion: A Surveycited by 53
  9. 2021Federated Learning: A Signal Processing Perspectivecited by 49
  10. 2022Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environmentscited by 49
  11. 2021SparseFed: Mitigating Model Poisoning Attacks in Federated Learning with Sparsificationcited by 46
  12. 2021Unity Perception: Generate Synthetic Data for Computer Visioncited by 43
  13. 2021Rethinking gradient sparsification as total error minimizationcited by 41
  14. 2021Local Stochastic Gradient Descent Ascent: Convergence Analysis and Communication Efficiencycited by 40
  15. 2021Gradient-Leakage Resilient Federated Learningcited by 40
  16. 2021Node Selection Toward Faster Convergence for Federated Learning on Non-IID Datacited by 39
  17. 2022The Fundamental Price of Secure Aggregation in Differentially Private Federated Learningcited by 39
  18. 2021Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devicescited by 36
  19. 2022No One Left Behind: Inclusive Federated Learning over Heterogeneous Devicescited by 36
  20. 2022Compressed-VFL: Communication-Efficient Learning with Vertically Partitioned Datacited by 36
  21. 2021GRNN: Generative Regression Neural Network -- A Data Leakage Attack for Federated Learningcited by 35
  22. 2022Differentially Private Federated Learning with Local Regularization and Sparsificationcited by 35
  23. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 35
  24. 2021SynthASR: Unlocking Synthetic Data for Speech Recognitioncited by 34
  25. 2022Joint Privacy Enhancement and Quantization in Federated Learningcited by 34
  26. 2022Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compressioncited by 33
  27. 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
  28. 2021An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systemscited by 32
  29. 2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
  30. 2022Sparse Random Networks for Communication-Efficient Federated Learningcited by 32
  31. 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
  32. 2023Momentum Provably Improves Error Feedback!cited by 31
  33. 2021PRECODE - A Generic Model Extension to Prevent Deep Gradient Leakagecited by 28
  34. 2021FEDZIP: A Compression Framework for Communication-Efficient Federated Learningcited by 27
  35. 2021Distribution Adaptive INT8 Quantization for Training CNNscited by 27
  36. 2021Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneitycited by 27
  37. 2021ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Trainingcited by 27
  38. 20223PC: Three Point Compressors for Communication-Efficient Distributed Training and a Better Theory for Lazy Aggregationcited by 27

50 most cited of 471 All 471 (JSON)