mlsanityData to 7 Oct 2026

Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimization

arXiv 2111.00705 ↗First posted 1 Nov 2021cs.LG cs.AI cs.DCcited by 12

Not on the map. Why?

Abstract (arXiv)

Due to the explosion in the size of the training datasets, distributed learning has received growing interest in recent years. One of the major bottlenecks is the large communication cost between the central server and the local workers. While error feedback compression has been proven to be successful in reducing communication costs with stochastic gradient descent (SGD), there are much fewer attempts in building communication-efficient adaptive gradient methods with provable guarantees, which are widely used in training large-scale machine learning models. In this paper, we propose a new communication-compressed AMSGrad for distributed nonconvex optimization problem, which is provably efficient. Our proposed distributed learning framework features an effective gradient compression strategy and a worker-side model update design. We prove that the proposed communication-efficient distributed adaptive gradient method converges to the first-order stationary point with the same iteration complexity as uncompressed vanilla AMSGrad in the stochastic nonconvex optimization setting. Experiments on various benchmarks back up our theory.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 33 of its 51 references that are in the map. First posted on arXiv 1 Nov 2021.

Builds on

33 of 51 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2016Wide Residual Networkscited by 3,057
  7. 2019On the Convergence of Adam and Beyondcited by 834
  8. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  9. 2018Sparsified SGD with Memorycited by 373
  10. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 306
  11. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 181
  12. 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 179
21 more
  1. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 173
  2. 2020On Biased Compression for Distributed Learningcited by 140
  3. 2021EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedbackcited by 131
  4. 2019Distributed Learning with Compressed Gradient Differencescited by 110
  5. 2019Decentralized Deep Learning with Arbitrary Communication Compressioncited by 109
  6. 2016Asynchronous Stochastic Gradient Descent with Delay Compensationcited by 98
  7. 2019DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compressioncited by 98
  8. 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
  9. 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 89
  10. 2019Communication-efficient distributed SGD with Sketchingcited by 81
  11. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 74
  12. 2019Stochastic Distributed Learning with Gradient Quantization and Variance Reductioncited by 64
  13. 2020Linearly Converging Error Compensated SGDcited by 61
  14. 2019Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedbackcited by 49
  15. 2020A new regret analysis for Adam-type algorithmscited by 35
  16. 2019A Double Residual Compression Algorithm for Efficient Distributed Learningcited by 31
  17. 2020Quantized Adam with Error Feedbackcited by 27
  18. 2021Preserved central model for faster bidirectional compression in distributed settingscited by 21
  19. 2020CADA: Communication-Adaptive Distributed Adamcited by 10
  20. 2020Accelerated Large Batch Optimization of BERT Pretraining in 54 minutescited by 7
  21. 2021Compressed Communication for Distributed Training: Adaptive Methods and Systemcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 3; 2024: 4; 2025: 2; 2026: 1.

12 papers.

  1. 2022Communication-Efficient Adaptive Federated Learningcited by 55
  2. 2023Momentum Provably Improves Error Feedback!cited by 31
  3. 2022Efficient-Adam: Communication-Efficient Distributed Adamcited by 11
  4. 2024FADAS: Towards Federated Adaptive Asynchronous Optimizationcited by 10
  5. 2024FEDMEKI: A Benchmark for Scaling Medical Foundation Models via Federated Knowledge Injectioncited by 8
  6. 2023Locally Adaptive Federated Learningcited by 7
  7. 2024FEDKIM: Adaptive Federated Knowledge Injection into Medical Foundation Modelscited by 5
  8. 2025Compressed Decentralized Momentum Stochastic Gradient Methods for Nonconvex Optimizationcited by 2
  9. 2023A Lightweight Method for Tackling Unknown Participation Statistics in Federated Averagingcited by 1
  10. 2024Sketched Adaptive Federated Deep Learning: A Sharp Convergence Analysiscited by 1
  11. 2025Tight analyses of first-order methods with error feedbackcited by 1
  12. 2026LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communicationcited by 0