mlsanityData to 14 Oct 2026

On Distributed Adaptive Optimization with Gradient Compression

arXiv 2205.05632 ↗First posted 11 May 2022stat.ML cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

We study COMP-AMS, a distributed optimization framework based on gradient averaging and adaptive AMSGrad algorithm. Gradient compression with error feedback is applied to reduce the communication cost in the gradient transmission process. Our convergence analysis of COMP-AMS shows that such compressed gradient averaging strategy yields same convergence rate as standard AMSGrad, and also exhibits the linear speedup effect w.r.t. the number of local workers. Compared with recently proposed protocols on distributed adaptive methods, COMP-AMS is simple and convenient. Numerical experiments are conducted to justify the theoretical findings, and demonstrate that the proposed method can achieve same test accuracy as the full-gradient AMSGrad with substantial communication savings. With its simplicity and efficiency, COMP-AMS can serve as a useful distributed training framework for adaptive gradient methods.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 46 of its 78 references that are in the map. First posted on arXiv 11 May 2022.

Builds on

46 of 78 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  7. 2020Adaptive Federated Optimizationcited by 1,046
  8. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  9. 2019On the Convergence of Adam and Beyondcited by 841
  10. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  11. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  12. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
34 more
  1. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  2. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
  3. 2018Sparsified SGD with Memorycited by 376
  4. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 310
  5. 2017Recent Trends in Deep Learning Based Natural Language Processingcited by 263
  6. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 256
  7. 2017Sparse Communication for Distributed Gradient Descentcited by 249
  8. 2020Revisiting Few-sample BERT Fine-tuningcited by 238
  9. 2018The Convergence of Sparsified Gradient Methodscited by 229
  10. 2019Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communicationcited by 221
  11. 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 181
  12. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 180
  13. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
  14. 2018cpSGD: Communication-efficient and differentially-private distributed SGDcited by 158
  15. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 154
  16. 2020On Biased Compression for Distributed Learningcited by 140
  17. 2018Deep Learning for Sentiment Analysis : A Surveycited by 139
  18. 2021EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedbackcited by 133
  19. 2019On Empirical Comparisons of Optimizers for Deep Learningcited by 103
  20. 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
  21. 2019Communication-efficient distributed SGD with Sketchingcited by 81
  22. 2018Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimizationcited by 76
  23. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 75
  24. 20158-Bit Approximations for Parallelism in Deep Learningcited by 56
  25. 2019Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedbackcited by 50
  26. 2019SWALP : Stochastic Weight Averaging in Low-Precision Trainingcited by 42
  27. 2019DADAM: A Consensus-based Distributed Adaptive Gradient Method for Online Optimizationcited by 30
  28. 2020Quantized Adam with Error Feedbackcited by 27
  29. 2021Toward Communication Efficient Adaptive Gradient Methodcited by 25
  30. 2020FedSKETCH: Communication-Efficient and Private Federated Learning via Sketchingcited by 20
  31. 2021On the Convergence of Decentralized Adaptive Gradient Methodscited by 17
  32. 2018Towards More Efficient Stochastic Decentralized Learning: Faster Convergence and Sparse Communicationcited by 13
  33. 2019Exploring Fast and Communication-Efficient Algorithms in Large-scale Distributed Networkscited by 8
  34. 2022Communication-Efficient TeraByte-Scale Model Training Framework for Online Advertisingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 6; 2024: 6; 2025: 4; 2026: 1.

19 papers.

Sort:

  1. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 101
  2. 2022On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness and Beyondcited by 38
  3. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 38
  4. 2023Momentum Provably Improves Error Feedback!cited by 31
  5. 2023AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networkscited by 21
  6. 2023FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUscited by 20
  7. 2024MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergencecited by 19
  8. 2023THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compressioncited by 15
  9. 2024FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compressioncited by 7
  10. 2023Error Feedback Can Accurately Compress Preconditionerscited by 6
  11. 2025DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Modelscited by 4
  12. 2022Analysis of Error Feedback in Federated Non-Convex Optimization with Biased Compressioncited by 3
7 more
  1. 2024Improving training time and GPU utilization in geo-distributed language model trainingcited by 3
  2. 2025FFT-based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of Large Language Modelscited by 2
  3. 2025Compressed Decentralized Momentum Stochastic Gradient Methods for Nonconvex Optimizationcited by 2
  4. 2024Studying the Impact of TensorFlow and PyTorch Bindings on Machine Learning Software Qualitycited by 1
  5. 2024Sketched Adaptive Federated Deep Learning: A Sharp Convergence Analysiscited by 1
  6. 2025Analysis of Asynchronous Federated Learning: Unraveling the Interactions between Gradient Compression, Delay, and Data Heterogeneitycited by 0
  7. 2026LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communicationcited by 0