mlsanityData to 14 Oct 2026

Toward Communication Efficient Adaptive Gradient Method

arXiv 2109.05109 ↗First posted 10 Sep 2021cs.LG cs.DC stat.MLcited by 25

Not on the map. Why?

Abstract (arXiv)

In recent years, distributed optimization is proven to be an effective approach to accelerate training of large scale machine learning models such as deep neural networks. With the increasing computation power of GPUs, the bottleneck of training speed in distributed training is gradually shifting from computation to communication. Meanwhile, in the hope of training machine learning models on mobile devices, a new distributed training paradigm called ``federated learning'' has become popular. The communication time in federated learning is especially important due to the low bandwidth of mobile devices. While various approaches to improve the communication efficiency have been proposed for federated learning, most of them are designed with SGD as the prototype training algorithm. While adaptive gradient methods have been proven effective for training neural nets, the study of adaptive gradient methods in federated learning is scarce. In this paper, we propose an adaptive gradient method that can guarantee both the convergence and the communication efficiency for federated learning.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 27 of its 45 references that are in the map. First posted on arXiv 10 Sep 2021.

Builds on

27 of 45 references.

References · 27

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  4. 2020Adaptive Federated Optimizationcited by 1,046
  5. 2019On the Convergence of Adam and Beyondcited by 841
  6. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  7. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  8. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  9. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
  10. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 356
  11. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
  12. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 256
15 more
  1. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 201
  2. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  3. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  4. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 180
  5. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
  6. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 156
  7. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 126
  8. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 125
  9. 2017On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimizationcited by 92
  10. 2020Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systemscited by 60
  11. 2019Escaping Saddle Points with Adaptive Gradient Methodscited by 34
  12. 2019Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Ratescited by 26
  13. 2020FedSKETCH: Communication-Efficient and Private Federated Learning via Sketchingcited by 20
  14. 2021On the Convergence of Decentralized Adaptive Gradient Methodscited by 17
  15. 2018Universal Stagewise Learning for Non-Convex Problems with Convergence on Averaged Solutionscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 9; 2023: 6; 2024: 4; 2025: 2; 2026: 3.

25 papers.

Sort:

  1. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 101
  2. 2022Communication-Efficient Adaptive Federated Learningcited by 55
  3. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  4. 2022On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness and Beyondcited by 38
  5. 2022Faster Adaptive Federated Learningcited by 35
  6. 2023Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedupcited by 23
  7. 2022On Distributed Adaptive Optimization with Gradient Compressioncited by 19
  8. 2025FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Modelscited by 15
  9. 2022Fast Adaptive Federated Bilevel Optimizationcited by 7
  10. 2023Locally Adaptive Federated Learningcited by 7
  11. 2024TurboSVM-FL: Boosting Federated Learning through SVM Aggregation for Lazy Clientscited by 7
  12. 2024ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Trainingcited by 6
13 more
  1. 2022Communication-Efficient TeraByte-Scale Model Training Framework for Online Advertisingcited by 5
  2. 2022Adaptive Federated Minimax Optimization with Lower Complexitiescited by 5
  3. 2021Layer-wise and Dimension-wise Locally Adaptive Federated Learningcited by 3
  4. 2022Analysis of Error Feedback in Federated Non-Convex Optimization with Biased Compressioncited by 3
  5. 2022Faster Adaptive Momentum-Based Federated Methods for Distributed Composition Optimizationcited by 2
  6. 2023FedDA: Faster Framework of Local Adaptive Gradient Methods via Restarted Dual Averagingcited by 2
  7. 2023FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Datacited by 2
  8. 2024Efficient Adaptive Federated Optimizationcited by 2
  9. 2024Non-Convex Optimization in Federated Learning via Variance Reduction and Adaptive Learningcited by 2
  10. 2025Compressed Decentralized Momentum Stochastic Gradient Methods for Nonconvex Optimizationcited by 2
  11. 2026DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Modelscited by 1
  12. 2026FedAdaVR: Adaptive Variance Reduction for Robust Federated Learning under Limited Client Participationcited by 0
  13. 2026LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communicationcited by 0