mlsanityData to 7 Oct 2026

Sparse Communication for Distributed Gradient Descent

arXiv 1704.05021 ↗First posted 17 Apr 2017cs.CL cs.DC cs.LGcited by 243

Not on the map. Why?

Abstract (arXiv)

We make distributed stochastic gradient descent faster by exchanging sparse updates instead of dense updates. Gradient updates are positively skewed as most updates are near zero, so we map the 99% smallest updates (by absolute value) to zero then exchange sparse matrices. This method can be combined with quantization to further improve the compression. We explore different configurations and apply them to neural machine translation and MNIST image classification tasks. Most configurations work on MNIST, whereas different configurations reduce convergence rate on the more complex translation task. Our experiments show that we can achieve up to 49% speed up on MNIST and 22% on NMT without damaging the final accuracy or BLEU.

Key facts. Cited by 243 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Apr 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 64; 2022: 39; 2023: 44; 2024: 31; 2025: 42; 2026: 22.

243 papers.

Sort:

By area:

  1. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  2. 2021Distributed Learning in Wireless Networks: Recent Progress and Future Challengescited by 125
  3. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
  4. 2022Vertical Federated Learning: Concepts, Advances and Challengescited by 104
  5. 2021Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applicationscited by 96
  6. 2023Towards Accurate Post-Training Quantization for Vision Transformercited by 70
  7. 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 65
  8. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  9. 2021FedLab: A Flexible Federated Learning Frameworkcited by 54
  10. 2023GIFD: A Generative Gradient Inversion Method with Feature Domain Optimizationcited by 45
  11. 2021Rethinking gradient sparsification as total error minimizationcited by 41
  12. 2022The Fundamental Price of Secure Aggregation in Differentially Private Federated Learningcited by 39
38 more
  1. 2023A Guide Through the Zoo of Biased SGDcited by 39
  2. 2023FedML-HE: An Efficient Homomorphic-Encryption-Based Privacy-Preserving Federated Learning Systemcited by 38
  3. 2022Gradient Obfuscation Gives a False Sense of Security in Federated Learningcited by 37
  4. 2022Joint Privacy Enhancement and Quantization in Federated Learningcited by 34
  5. 2021An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systemscited by 32
  6. 2022Sparse Random Networks for Communication-Efficient Federated Learningcited by 32
  7. 2022Learning to Invert: Simple Adaptive Attacks for Gradient Inversion in Federated Learningcited by 30
  8. 2021FEDZIP: A Compression Framework for Communication-Efficient Federated Learningcited by 27
  9. 2021On the Utility of Gradient Compression in Distributed Training Systemscited by 26
  10. 2021QLSD: Quantised Langevin stochastic dynamics for Bayesian federated learningcited by 26
  11. 2021Time-Correlated Sparsification for Communication-Efficient Federated Learningcited by 25
  12. 2021An Information-Theoretic Justification for Model Pruningcited by 23
  13. 2022Near-Optimal Sparse Allreduce for Distributed Deep Learningcited by 23
  14. 2023Reducing Communication for Split Learning by Randomized Top-k Sparsificationcited by 22
  15. 2021Learned Gradient Compression for Distributed Deep Learningcited by 20
  16. 2022On Distributed Adaptive Optimization with Gradient Compressioncited by 19
  17. 2021Pervasive AI for IoT applications: A Survey on Resource-efficient Distributed Artificial Intelligencecited by 18
  18. 2021ResIST: Layer-Wise Decomposition of ResNets for Distributed Trainingcited by 18
  19. 2021FedDQ: Communication-Efficient Federated Learning with Descending Quantizationcited by 18
  20. 2024Communication Efficient Distributed Training with Distributed Lioncited by 18
  21. 2021On the Convergence of Decentralized Adaptive Gradient Methodscited by 17
  22. 2021Revealing and Protecting Labels in Distributed Trainingcited by 17
  23. 2021Communication-Efficient Federated Learning via Quantized Compressed Sensingcited by 17
  24. 2024Training Neural Networks from Scratch with Parallel Low-Rank Adapterscited by 17
  25. 2021Masked Training of Neural Networks with Partial Gradientscited by 15
  26. 2021FLASHE: Additively Symmetric Homomorphic Encryption for Cross-Silo Federated Learningcited by 15
  27. 2021Neural Tangent Kernel Empowered Federated Learningcited by 15
  28. 2021Citadel: Protecting Data Privacy and Model Confidentiality for Collaborative Learning with SGXcited by 14
  29. 2022Optimizing the Communication-Accuracy Trade-off in Federated Learning with Rate-Distortion Theorycited by 14
  30. 2022Correlated quantization for distributed mean estimation and optimizationcited by 14
  31. 2023A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Servicescited by 14
  32. 2021Joint Optimization of Communications and Federated Learning Over the Aircited by 13
  33. 2021Communication-Efficient Federated Learning with Binary Neural Networkscited by 13
  34. 2024GI-NAS: Boosting Gradient Inversion Attacks Through Adaptive Neural Architecture Searchcited by 13
  35. 2022Efficient and Light-Weight Federated Learning via Asynchronous Distributed Dropoutcited by 12
  36. 2023Temporal Difference Learning with Compressed Updates: Error-Feedback meets Reinforcement Learningcited by 12
  37. 2023Towards Green AI in Fine-tuning Large Language Models via Adaptive Backpropagationcited by 12
  38. 2021Federated Learning over Wireless Networks: A Band-limited Coordinated Descent Approachcited by 11

50 most cited of 243 All 243 (JSON)