mlsanityData to 14 Oct 2026

Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computations

arXiv 1906.02367 ↗First posted 6 Jun 2019stat.ML cs.DC cs.LGcited by 181

Not on the map. Why?

Abstract (arXiv)

Communication bottleneck has been identified as a significant issue in distributed optimization of large-scale learning models. Recently, several approaches to mitigate this problem have been proposed, including different forms of gradient compression or computing local models and mixing them iteratively. In this paper, we propose Qsparse-local-SGD algorithm, which combines aggressive sparsification with quantization and local computation along with error compensation, by keeping track of the difference between the true and compressed gradients. We propose both synchronous and asynchronous implementations of Qsparse-local-SGD. We analyze convergence for Qsparse-local-SGD in the distributed setting for smooth non-convex and convex objective functions. We demonstrate that Qsparse-local-SGD converges at the same rate as vanilla distributed SGD for many important classes of sparsifiers and quantizers. We use Qsparse-local-SGD to train ResNet-50 on ImageNet and show that it results in significant savings over the state-of-the-art, in the number of bits transmitted to reach target accuracy.

Key facts. Cited by 181 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 45; 2022: 40; 2023: 43; 2024: 27; 2025: 10; 2026: 15.

181 papers.

Sort:

By area:

  1. 2021A Field Guide to Federated Optimizationcited by 315
  2. 2021Personalized Federated Learning using Hypernetworkscited by 212
  3. 2022Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challengescited by 138
  4. 2021EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedbackcited by 133
  5. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 101
  6. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  7. 2021MARINA: Faster Non-Convex Distributed Learning with Compressioncited by 78
  8. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 75
  9. 2021On Large-Cohort Training for Federated Learningcited by 74
  10. 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
  11. 2021Fast Federated Learning in the Presence of Arbitrary Device Unavailabilitycited by 72
  12. 2021What Do We Mean by Generalization in Federated Learning?cited by 62
38 more
  1. 2022FedNest: Federated Bilevel, Minimax, and Compositional Optimizationcited by 60
  2. 2022Communication-Efficient Adaptive Federated Learningcited by 55
  3. 2021Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learningcited by 52
  4. 2021Federated Learning over Wireless Device-to-Device Networks: Algorithms and Convergence Analysiscited by 51
  5. 2021Rethinking gradient sparsification as total error minimizationcited by 41
  6. 2021Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devicescited by 36
  7. 2022Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compressioncited by 33
  8. 2021QuPeD: Quantized Personalization via Distillation with Applications to Federated Learningcited by 32
  9. 2022FedSynth: Gradient Compression via Synthetic Data in Federated Learningcited by 31
  10. 2023On the Convergence of Federated Averaging with Cyclic Client Participationcited by 31
  11. 20211-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speedcited by 24
  12. 2022Communication-Efficient Federated Learning with Accelerated Client Gradientcited by 24
  13. 2021Learned Gradient Compression for Distributed Deep Learningcited by 20
  14. 2021Robust Training in High Dimensions via Block Coordinate Geometric Median Descentcited by 20
  15. 2022A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networkscited by 20
  16. 2022SplitGP: Achieving Both Generalization and Personalization in Federated Learningcited by 20
  17. 2022On Distributed Adaptive Optimization with Gradient Compressioncited by 19
  18. 2022Federated Optimization Algorithms with Random Reshuffling and Gradient Compressioncited by 18
  19. 2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
  20. 2025MuLoCo: Muon is a practical inner optimizer for DiLoCocited by 18
  21. 2021Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameterscited by 17
  22. 2022EF-BV: A Unified Theory of Error Feedback and Variance Reduction Mechanisms for Biased and Unbiased Compression in Distributed Optimizationcited by 17
  23. 2022A General Theory for Federated Optimization with Asynchronous and Heterogeneous Clients Updatescited by 17
  24. 2023z-SignFedAvg: A Unified Stochastic Sign-based Compression for Federated Learningcited by 17
  25. 2023On the Role of Server Momentum in Federated Learningcited by 16
  26. 2022Maximizing Communication Efficiency for Large-scale Training via 0/1 Adamcited by 15
  27. 2023Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compressioncited by 15
  28. 2021MURANA: A Generic Framework for Stochastic Variance-Reduced Optimizationcited by 14
  29. 2022Correlated quantization for distributed mean estimation and optimizationcited by 14
  30. 2023SimFBO: Towards Simple, Flexible and Communication-efficient Federated Bilevel Learningcited by 14
  31. 2023Benchmarking Algorithms for Federated Domain Generalizationcited by 14
  32. 2024SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Trainingcited by 14
  33. 2021Neural Distributed Source Codingcited by 13
  34. 2021BAGUA: Scaling up Distributed Learning with System Relaxationscited by 13
  35. 2023Communication-Efficient Federated Hypergradient Computation via Aggregated Iterative Differentiationcited by 13
  36. 2021Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimizationcited by 12
  37. 2021Edge-Native Intelligence for 6G Communications Driven by Federated Learning: A Survey of Trends and Challengescited by 12
  38. 2022Federated Random Reshuffling with Compression and Variance Reductioncited by 12

50 most cited of 181 All 181 (JSON)