mlsanityData to 7 Oct 2026

Preserved central model for faster bidirectional compression in distributed settings

arXiv 2102.12528 ↗First posted 24 Feb 2021cs.LG cs.DC math.STcited by 21

Not on the map. Why?

Abstract (arXiv)

We develop a new approach to tackle communication constraints in a distributed learning problem with a central server. We propose and analyze a new algorithm that performs bidirectional compression and achieves the same convergence rate as algorithms using only uplink (from the local workers to the central server) compression. To obtain this improvement, we design MCM, an algorithm such that the downlink compression only impacts local models, while the global model is preserved. As a result, and contrary to previous works, the gradients on local servers are computed on perturbed models. Consequently, convergence proofs are more challenging and require a precise control of this perturbation. To ensure it, MCM additionally combines model compression with a memory mechanism. This analysis opens new doors, e.g. incorporating worker dependent randomized-models and partial participation.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 33 of its 72 references that are in the map. First posted on arXiv 24 Feb 2021.

Builds on

33 of 72 references.

References · 33

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2019Advances and Open Problems in Federated Learningcited by 3,021
  4. 2018LEAF: A Benchmark for Federated Settingscited by 780
  5. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
  6. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  7. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 415
  8. 2018Sparsified SGD with Memorycited by 373
  9. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 253
  10. 2018The Convergence of Sparsified Gradient Methodscited by 228
  11. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 199
  12. 2018cpSGD: Communication-efficient and differentially-private distributed SGDcited by 157
21 more
  1. 2018D2: Decentralized Training over Decentralized Datacited by 140
  2. 2020On Biased Compression for Distributed Learningcited by 140
  3. 2019Distributed Learning with Compressed Gradient Differencescited by 110
  4. 2019DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compressioncited by 98
  5. 2020Acceleration for Compressed Gradient Descent in Distributed and Federated Optimizationcited by 90
  6. 2019Communication-efficient distributed SGD with Sketchingcited by 81
  7. 2018Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimizationcited by 76
  8. 2016Decentralized Collaborative Learning of Personalized Models over Networkscited by 73
  9. 2015Perturbed Iterate Analysis for Asynchronous Stochastic Optimizationcited by 67
  10. 2019Stochastic Distributed Learning with Gradient Quantization and Variance Reductioncited by 64
  11. 2020Linearly Converging Error Compensated SGDcited by 61
  12. 2020Free-rider Attacks on Model Aggregation in Federated Learningcited by 57
  13. 2019Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedbackcited by 49
  14. 2020A Better Alternative to Error Feedback for Communication-Efficient Distributed Learningcited by 42
  15. 2018Optimal Algorithms for Non-Smooth Distributed Optimization in Networkscited by 41
  16. 2019A Double Residual Compression Algorithm for Efficient Distributed Learningcited by 31
  17. 2019RATQ: A Universal Fixed-Length Quantizer for Stochastic Optimizationcited by 23
  18. 2016Gossip Dual Averaging for Decentralized Optimization of Pairwise Functionscited by 21
  19. 2019Privacy for Free: Communication-Efficient Learning with Differential Privacy Using Sketchescited by 17
  20. 2020A flexible framework for communication-efficient machine learning: from HPC to IoTcited by 2
  21. 2019Theoretical Limits of Pipeline Parallel Optimization and Application to Distributed Deep Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 6; 2023: 6; 2024: 4; 2025: 1; 2026: 2.

21 papers.

Sort:

  1. 2022Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compressioncited by 33
  2. 2023DoCoFL: Downlink Compression for Cross-Device Federated Learningcited by 27
  3. 2022FedShuffle: Recipes for Better Use of Local Work in Federated Learningcited by 19
  4. 2021Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimizationcited by 12
  5. 2023Communication Compression for Byzantine Robust Learning: New Efficient Algorithms and Improved Ratescited by 12
  6. 2022CompressedScaffnew: The First Theoretical Double Acceleration of Communication from Local Training and Compression in Distributed Optimizationcited by 11
  7. 2023Lower Bounds and Accelerated Algorithms in Distributed Stochastic Optimization with Communication Compressioncited by 11
  8. 2023Knowledge Distillation Performs Partial Variance Reductioncited by 11
  9. 2024LoCoDL: Communication-Efficient Distributed Learning with Local Training and Compressioncited by 11
  10. 2023Stochastic Approximation Beyond Gradient for Signal Processing and Machine Learningcited by 10
  11. 2023Compression with Exact Error Distribution for Federated Learningcited by 9
  12. 2021Federated Expectation Maximization with heterogeneity mitigation and variance reductioncited by 5
9 more
  1. 2022Reconciling Security and Communication Efficiency in Federated Learningcited by 4
  2. 2024Quantized and Asynchronous Federated Learningcited by 3
  3. 2024Communication-Efficient Distributed Learning with Local Immediate Error Compensationcited by 2
  4. 2025Tight analyses of first-order methods with error feedbackcited by 1
  5. 2026BiCoLoR: Communication-Efficient Optimization with Bidirectional Compression and Local Trainingcited by 1
  6. 2022Variance Reduction is an Antidote to Byzantines: Better Rates, Weaker Assumptions and Communication Compression as a Cherry on the Topcited by 0
  7. 2022Downlink Compression Improves TopK Sparsificationcited by 0
  8. 2024MARINA-P: Superior Performance in Non-smooth Federated Optimization with Adaptive Stepsizescited by 0
  9. 2026A Tight Theory of Error Feedback Algorithms in Distributed Optimizationcited by 0