mlsanityData to 14 Oct 2026

Adaptive Compression for Communication-Efficient Distributed Training

arXiv 2211.00188 ↗First posted 31 Oct 2022cs.LG cs.DC cs.ITcited by 16

Not on the map. Why?

Abstract (arXiv)

We propose Adaptive Compressed Gradient Descent (AdaCGD) - a novel optimization algorithm for communication-efficient training of supervised machine learning models with adaptive compression level. Our approach is inspired by the recently proposed three point compressor (3PC) framework of Richtarik et al. (2022), which includes error feedback (EF21), lazily aggregated gradient (LAG), and their combination as special cases, and offers the current state-of-the-art rates for these methods under weak assumptions. While the above mechanisms offer a fixed compression level, or adapt between two extremes only, our proposal is to perform a much finer adaptation. In particular, we allow the user to choose any number of arbitrarily chosen contractive compression mechanisms, such as Top-K sparsification with a user-defined selection of sparsification levels K, or quantization with a user-defined selection of quantization levels, or their combination. AdaCGD chooses the appropriate compressor and compression level adaptively during the optimization process. Besides i) proposing a theoretically-grounded multi-adaptive communication compression mechanism, we further ii) extend the 3PC framework to bidirectional compression, i.e., we allow the server to compress as well, and iii) provide sharp convergence bounds in the strongly convex, convex and nonconvex settings. The convex regime results are new even for several key special cases of our general mechanism, including 3PC and EF21. In all regimes, our rates are superior compared to all existing adaptive compression methods.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 37 of its 53 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

37 of 53 references.

References · 37

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  4. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  5. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  6. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
  7. 2018Sparsified SGD with Memorycited by 376
  8. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 356
  9. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 310
  10. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  11. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 256
  12. 2018The Convergence of Sparsified Gradient Methodscited by 229
25 more
  1. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  2. 2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
  3. 2020On Biased Compression for Distributed Learningcited by 140
  4. 2016Distributed Mean Estimation with Limited Communicationcited by 136
  5. 2021EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedbackcited by 133
  6. 2020PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimizationcited by 119
  7. 2019DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compressioncited by 99
  8. 2018LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learningcited by 96
  9. 2021Adaptive Quantization of Model Updates for Communication-Efficient Federated Learningcited by 63
  10. 2020SGD with shuffling: optimal rates without component convexity and large epoch requirementscited by 58
  11. 2020Closing the convergence gap of SGD without replacementcited by 57
  12. 2018Distributed learning with compressed gradientscited by 55
  13. 2020Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guaranteescited by 46
  14. 2020Adaptive Gradient Quantization for Data-Parallel SGDcited by 45
  15. 2020A Better Alternative to Error Feedback for Communication-Efficient Distributed Learningcited by 43
  16. 2021DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learningcited by 41
  17. 2016Randomized Distributed Mean Estimation: Accuracy vs Communicationcited by 39
  18. 2019Communication-Efficient Distributed Learning via Lazily Aggregated Quantized Gradientscited by 32
  19. 2019On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learningcited by 32
  20. 2021Permutation Compressors for Provably Faster Distributed Nonconvex Optimizationcited by 32
  21. 20223PC: Three Point Compressors for Communication-Efficient Distributed Training and a Better Theory for Lazy Aggregationcited by 27
  22. 2021FedDQ: Communication-Efficient Federated Learning with Descending Quantizationcited by 20
  23. 2020Accordion: Adaptive Gradient Communication via Critical Learning Regime Identificationcited by 19
  24. 2021IntSGD: Adaptive Floatless Compression of Stochastic Gradientscited by 9
  25. 2021Compressed Communication for Distributed Training: Adaptive Methods and Systemcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 3; 2025: 4; 2026: 3.

16 papers.

Sort:

By area:

  1. 2023Momentum Provably Improves Error Feedback!cited by 31
  2. 2023EControl: Fast Distributed Optimization with Compression and Error Controlcited by 15
  3. 2024Loss Landscape Characterization of Neural Networks without Over-Parametrizationcited by 8
  4. 2023Accelerating Distributed ML Training via Selective Synchronizationcited by 7
  5. 2023Towards a Better Theoretical Understanding of Independent Subnetwork Trainingcited by 6
  6. 2024Queuing dynamics of asynchronous Federated Learningcited by 6
  7. 2023Private Federated Learning with Autotuned Compressioncited by 5
  8. 2024Towards Faster Decentralized Stochastic Optimization with Communication Compressioncited by 5
  9. 2025Optimizing Asynchronous Federated Learning: A Delicate Trade-Off Between Model-Parameter Staleness and Update Frequencycited by 2
  10. 2023FAVANO: Federated AVeraging with Asynchronous NOdescited by 1
  11. 2025Layer-wise Quantization for Quantized Optimistic Dual Averagingcited by 1
  12. 2025Tight analyses of first-order methods with error feedbackcited by 1
4 more
  1. 2026Byzantine-Robust and Differentially Private Federated Optimization under Weaker Assumptionscited by 1
  2. 2025Gradient Projection onto Historical Descent Directions for Communication-Efficient Federated Learningcited by 0
  3. 2026Optimization Trade-offs in Asynchronous Federated Learning: A Stochastic Networks Approachcited by 0
  4. 2026CIERA: Cross-Iteration Exponent Reuse for Lossless Allgather in Sharded MoE Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient Distributed Training through Gradient Compression with Sparsification and Quantization Techniques
  2. 2024Beyond Throughput and Compression Ratios: Towards High End-to-end Utility of Gradient Compression
  3. 2023Compressed and Sparse Models for Non-Convex Decentralized Learning
  4. 2023Evaluation and Optimization of Gradient Compression for Distributed Deep Learning
  5. 2021Learned Gradient Compression for Distributed Deep Learning
  6. 2021Unbiased Single-scale and Multi-scale Quantizers for Distributed Optimization
  7. 2021Theoretically Better and Numerically Faster Distributed Optimization with Smoothness-Aware Quantization Techniques
  8. 2022Fundamental Limits of Communication Efficiency for Model Aggregation in Distributed Learning: A Rate-Distortion Approach