mlsanityData to 14 Oct 2026

Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization

arXiv 1806.08054 ↗First posted 21 Jun 2018cs.CV cs.DCcited by 76

Not on the map. Why?

Abstract (arXiv)

Large-scale distributed optimization is of great importance in various applications. For data-parallel based distributed learning, the inter-node gradient communication often becomes the performance bottleneck. In this paper, we propose the error compensated quantized stochastic gradient descent algorithm to improve the training efficiency. Local gradients are quantized to reduce the communication overhead, and accumulated quantization error is utilized to speed up the convergence. Furthermore, we present theoretical analysis on the convergence behaviour, and demonstrate its advantage over competitors. Extensive experiments indicate that our algorithm can compress gradients by a factor of up to two magnitudes without performance degradation.

Key facts. Cited by 76 papers in the map (data through 14 Oct 2026). First posted on arXiv 21 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 26; 2022: 14; 2023: 14; 2024: 13; 2025: 6; 2026: 3.

76 papers.

Sort:

  1. 2021A Field Guide to Federated Optimizationcited by 315
  2. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 101
  3. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  4. 2021Federated Learning over Wireless Device-to-Device Networks: Algorithms and Convergence Analysiscited by 51
  5. 2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
  6. 2022Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compressioncited by 33
  7. 2021An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systemscited by 32
  8. 2021Permutation Compressors for Provably Faster Distributed Nonconvex Optimizationcited by 32
  9. 2021Time-Correlated Sparsification for Communication-Efficient Federated Learningcited by 26
  10. 2021On the Utility of Gradient Compression in Distributed Training Systemscited by 26
  11. 2023FwdLLM: Efficient FedLLM using Forward Gradientcited by 26
  12. 2022FedAdapter: Efficient Federated Learning for Modern NLPcited by 25
38 more
  1. 2021ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Trainingcited by 22
  2. 2021Smoothness Matrices Beat Smoothness Constants: Better Communication Compression Techniques for Distributed Optimizationcited by 21
  3. 2021Preserved central model for faster bidirectional compression in distributed settingscited by 21
  4. 2022On Distributed Adaptive Optimization with Gradient Compressioncited by 19
  5. 2021Fast Federated Learning by Balancing Communication Trade-Offscited by 18
  6. 2021Distributed Learning Systems with First-order Methodscited by 17
  7. 2021Towards Demystifying Serverless Machine Learning Trainingcited by 16
  8. 2022Federated Few-Shot Learning for Mobile NLPcited by 15
  9. 2023Unbiased Compression Saves Communication in Distributed Optimization: When and How Much?cited by 14
  10. 2021BROADCAST: Reducing Both Stochastic and Compression Noise to Robustify Communication-Efficient Federated Learningcited by 13
  11. 2022Federated Random Reshuffling with Compression and Variance Reductioncited by 12
  12. 2025Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelismcited by 12
  13. 2021Federated Learning over Wireless Networks: A Band-limited Coordinated Descent Approachcited by 11
  14. 2023Lower Bounds and Accelerated Algorithms in Distributed Stochastic Optimization with Communication Compressioncited by 11
  15. 2023Clip21: Error Feedback for Gradient Clippingcited by 11
  16. 2023Adaptive Message Quantization and Parallelization for Distributed Full-graph GNN Trainingcited by 10
  17. 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 10
  18. 2021Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategiescited by 9
  19. 2022GlueFL: Reconciling Client Sampling and Model Masking for Bandwidth Efficient Federated Learningcited by 7
  20. 2022Personalized Federated Learning with Communication Compressioncited by 6
  21. 2021The Gradient Convergence Bound of Federated Multi-Agent Reinforcement Learning with Efficient Communicationcited by 5
  22. 2022Towards Practical Few-shot Federated NLPcited by 5
  23. 2023Data-Aware Gradient Compression for FL in Communication-Constrained Mobile Computingcited by 5
  24. 2024FedAQ: Communication-Efficient Federated Edge Learning via Joint Uplink and Downlink Adaptive Quantizationcited by 5
  25. 2021Distributed Adaptive Learning Under Communication Constraintscited by 4
  26. 2021Collaborative Learning over Wireless Networks: An Introductory Overviewcited by 4
  27. 2023FedREP: A Byzantine-Robust, Communication-Efficient and Privacy-Preserving Framework for Federated Learningcited by 4
  28. 2024Error Feedback under (L0,L1)-Smoothness: Normalization and Momentumcited by 4
  29. 2021Decentralized Multi-Task Stochastic Optimization With Compressed Communicationscited by 3
  30. 2022Analysis of Error Feedback in Federated Non-Convex Optimization with Biased Compressioncited by 3
  31. 2025Smoothed Normalization for Efficient Distributed Private Optimizationcited by 3
  32. 2021DQ-SGD: Dynamic Quantization in SGD for Communication-Efficient Distributed Learningcited by 2
  33. 2021Compressing gradients by exploiting temporal correlation in momentum-SGDcited by 2
  34. 2022Adaptive Differential Filters for Fast and Communication-Efficient Federated Learningcited by 2
  35. 2023Magnitude Matters: Fixing SIGNSGD Through Magnitude-Aware Sparsification in the Presence of Data Heterogeneitycited by 2
  36. 2023Compressed Regression over Adaptive Networkscited by 2
  37. 2024Communication-Efficient Distributed Learning with Local Immediate Error Compensationcited by 2
  38. 2024Communication Efficient ConFederated Learning: An Event-Triggered SAGA Approachcited by 2

50 most cited of 76 All 76 (JSON)