mlsanityData to 7 Oct 2026

Distributed learning with compressed gradients

arXiv 1806.06573 ↗First posted 18 Jun 2018math.OC stat.MLcited by 53

Not on the map. Why?

Abstract (arXiv)

Asynchronous computation and gradient compression have emerged as two key techniques for achieving scalability in distributed optimization for large-scale machine learning. This paper presents a unified analysis framework for distributed gradient methods operating with staled and compressed gradients. Non-asymptotic bounds on convergence rates and information exchange are derived for several optimization algorithms. These bounds give explicit expressions for step-sizes and characterize how the amount of asynchrony and the compression accuracy affect iteration and communication complexity guarantees. Numerical results highlight convergence properties of different gradient compression algorithms and confirm that fast convergence under limited information exchange is indeed possible.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 13; 2023: 13; 2024: 8; 2025: 7; 2026: 1.

53 papers.

Sort:

  1. 2021EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedbackcited by 131
  2. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
  3. 2021FedNL: Making Newton-Type Methods Applicable to Federated Learningcited by 50
  4. 2022BEER: Fast O(1/T) Rate for Decentralized Nonconvex Optimization with Communication Compressioncited by 41
  5. 2023A Guide Through the Zoo of Biased SGDcited by 39
  6. 2022SoteriaFL: A Unified Framework for Private Federated Learning with Communication Compressioncited by 34
  7. 2022Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learningcited by 33
  8. 20223PC: Three Point Compressors for Communication-Efficient Distributed Training and a Better Theory for Lazy Aggregationcited by 27
  9. 2022Finite-Sum Coupled Compositional Stochastic Optimization: Theory and Applicationscited by 27
  10. 2021Distributed Second Order Methods with Fast Rates and Compressed Communicationcited by 25
  11. 2021CANITA: Faster Rates for Distributed Convex Optimization with Communication Compressioncited by 23
  12. 2022FedNew: A Communication-Efficient and Privacy-Preserving Newton-Type Method for Federated Learningcited by 23
38 more
  1. 2021Smoothness Matrices Beat Smoothness Constants: Better Communication Compression Techniques for Distributed Optimizationcited by 21
  2. 2022Federated Optimization Algorithms with Random Reshuffling and Gradient Compressioncited by 18
  3. 2023AsGrad: A Sharp Unified Analysis of Asynchronous-SGD Algorithmscited by 16
  4. 2021Faster Rates for Compressed Federated Learning with Client-Variance Reductioncited by 14
  5. 2022Adaptive Compression for Communication-Efficient Distributed Trainingcited by 14
  6. 2023Unbiased Compression Saves Communication in Distributed Optimization: When and How Much?cited by 14
  7. 2023EControl: Fast Distributed Optimization with Compression and Error Controlcited by 14
  8. 2023Temporal Difference Learning with Compressed Updates: Error-Feedback meets Reinforcement Learningcited by 12
  9. 2023Communication Compression for Byzantine Robust Learning: New Efficient Algorithms and Improved Ratescited by 12
  10. 2022FL_PyTorch: optimization research simulator for federated learningcited by 10
  11. 2024DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Modelscited by 10
  12. 2021Theoretically Better and Numerically Faster Distributed Optimization with Smoothness-Aware Quantization Techniquescited by 9
  13. 2022Linear Stochastic Bandits over a Bit-Constrained Channelcited by 9
  14. 2021IntSGD: Adaptive Floatless Compression of Stochastic Gradientscited by 8
  15. 2023Improving Accelerated Federated Learning with Compression and Importance Samplingcited by 7
  16. 2021EF21 with Bells & Whistles: Six Algorithmic Extensions of Modern Error Feedbackcited by 6
  17. 2022Shifted Compression Framework: Generalizations and Improvementscited by 6
  18. 2022Personalized Federated Learning with Communication Compressioncited by 6
  19. 2023Towards a Better Theoretical Understanding of Independent Subnetwork Trainingcited by 6
  20. 2023Byzantine Robustness and Partial Participation Can Be Achieved at Once: Just Clip Gradient Differencescited by 5
  21. 2024Rendering Wireless Environments Useful for Gradient Estimators: A Zero-Order Stochastic Federated Learning Methodcited by 4
  22. 2025Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEscited by 4
  23. 2025Theoretical Analysis on how Learning Rate Warmup Accelerates Convergencecited by 4
  24. 2024On the Convergence of FedProx with Extrapolation and Inexact Proxcited by 3
  25. 2024Problem-dependent convergence bounds for randomized linear gradient compressioncited by 3
  26. 2025The Ball-Proximal (="Broximal") Point Method: a New Algorithm, Convergence Theory, and Applicationscited by 3
  27. 2025Delayed Momentum Aggregation: Communication-efficient Byzantine-robust Federated Learning with Partial Participationcited by 3
  28. 2021Comfetch: Federated Learning of Large Networks on Constrained Clients via Sketchingcited by 2
  29. 2023Convergence of Sign-based Random Reshuffling Algorithms for Nonconvex Optimizationcited by 2
  30. 2023Federated Learning is Better with Non-Homomorphic Encryptioncited by 2
  31. 2024Temporal Predictive Coding for Gradient Compression in Distributed Learningcited by 2
  32. 2024Accelerated Methods with Compressed Communications for Distributed Optimization Problems under Data Similaritycited by 2
  33. 2023CORE: Common Random Reconstruction for Distributed Optimization with Provable Low Communication Complexitycited by 1
  34. 2024Communication and Energy Efficient Federated Learning using Zero-Order Optimization Techniquecited by 1
  35. 2025Tight analyses of first-order methods with error feedbackcited by 1
  36. 2025Communication-Efficient Zero-Order and First-Order Federated Learning Methods over Wireless Networkscited by 1
  37. 2022Better Methods and Theory for Federated Learning: Compression, Client Selection and Heterogeneitycited by 0
  38. 2023Unified analysis of SGD-type methodscited by 0

50 most cited of 53 All 53 (JSON)