mlsanityData to 14 Oct 2026

Accelerated Federated Learning with Decoupled Adaptive Optimization

arXiv 2207.07223 ↗First posted 14 Jul 2022cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

The federated learning (FL) framework enables edge clients to collaboratively learn a shared inference model while keeping privacy of training data on clients. Recently, many heuristics efforts have been made to generalize centralized adaptive optimization methods, such as SGDM, Adam, AdaGrad, etc., to federated settings for improving convergence and accuracy. However, there is still a paucity of theoretical principles on where to and how to design and utilize adaptive optimization methods in federated settings. This work aims to develop novel adaptive optimization methods for FL from the perspective of dynamics of ordinary differential equations (ODEs). First, an analytic framework is established to build a connection between federated optimization methods and decompositions of ODEs of corresponding centralized optimizers. Second, based on this analytic framework, a momentum decoupling adaptive optimization method, FedDA, is developed to fully utilize the global momentum on each local iteration and accelerate the training convergence. Last but not least, full batch gradients are utilized to mimic centralized optimization in the end of the training process to ensure the convergence and overcome the possible inconsistency caused by adaptive optimization methods.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 53 of its 101 references that are in the map. First posted on arXiv 14 Jul 2022.

Builds on

53 of 101 references.

References · 53

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2018Group Normalizationcited by 1,524
  5. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  6. 2020Adaptive Federated Optimizationcited by 1,046
  7. 2018LEAF: A Benchmark for Federated Settingscited by 792
  8. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
  9. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  10. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  11. 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
  12. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 182
38 more
  1. 2020FetchSGD: Communication-Efficient Federated Learning with Sketchingcited by 179
  2. 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 163
  3. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 154
  4. 2020Minibatch vs Local SGD for Heterogeneous Distributed Learningcited by 147
  5. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
  6. 2020Byzantine-Robust Learning on Heterogeneous Datasets via Bucketingcited by 128
  7. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 123
  8. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  9. 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
  10. 2021Robbing the Fed: Directly Obtaining Private Data in Federated Learning with Modified Modelscited by 108
  11. 2018Federated Learning for Keyword Spottingcited by 95
  12. 2020Acceleration for Compressed Gradient Descent in Distributed and Federated Optimizationcited by 90
  13. 2020Federated Bayesian Optimization via Thompson Samplingcited by 77
  14. 2019Accelerating Federated Learning via Momentum Gradient Descentcited by 76
  15. 2020Fast-Convergent Federated Learning with Adaptive Weightingcited by 67
  16. 2021From Distributed Machine Learning to Federated Learning: A Surveycited by 66
  17. 2021What Do We Mean by Generalization in Federated Learning?cited by 62
  18. 2020Federated Composite Optimizationcited by 49
  19. 2021STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learningcited by 43
  20. 2021Minibatch vs Local SGD with Shuffling: Tight Convergence Bounds and Beyondcited by 36
  21. 2019Why are Adaptive Methods Good for Attention Models?cited by 35
  22. 2021Bayesian Framework for Gradient Leakagecited by 34
  23. 2020Communication-Efficient Federated Learning with Compensated Overlap-FedAvgcited by 31
  24. 2021Local Adaptivity in Federated Learning: Convergence and Consistencycited by 31
  25. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillationcited by 31
  26. 2021Efficient Device Scheduling with Multi-Job Federated Learningcited by 29
  27. 2022Acceleration of Federated Learning with Alleviated Forgetting in Local Trainingcited by 29
  28. 2022FedDUAP: Federated Learning with Dynamic Update and Adaptive Pruning Using Shared Data on the Servercited by 29
  29. 2021ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Trainingcited by 27
  30. 2019Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Ratescited by 26
  31. 2020FedADC: Accelerated Federated Learning with Drift Controlcited by 26
  32. 2019A Communication Efficient Collaborative Learning Framework for Distributed Featurescited by 21
  33. 2022Improving Federated Learning Face Recognition via Privacy-Agnostic Clusterscited by 18
  34. 2020Accelerating Federated Learning over Reliability-Agnostic Clients in Mobile Edge Computing Systemscited by 17
  35. 2020Fast Convergence Algorithm for Analog Federated Learningcited by 17
  36. 2022Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank?cited by 16
  37. 2021On Bridging Generic and Personalized Federated Learning for Image Classificationcited by 15
  38. 2021H-FL: A Hierarchical Communication-Efficient and Privacy-Protected Architecture for Federated Learningcited by 14

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 10; 2024: 8; 2025: 3; 2026: 1.

23 papers.

Sort:

  1. 2023Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimizationcited by 65
  2. 2023Momentum Benefits Non-IID Federated Learning Simply and Provablycited by 38
  3. 2023Reducing Communication for Split Learning by Randomized Top-k Sparsificationcited by 22
  4. 2022Multi-Job Intelligent Scheduling with Cross-Device Federated Learningcited by 20
  5. 2023Federated Fine-Tuning of LLMs on the Very Edge: The Good, the Bad, the Uglycited by 17
  6. 2023AEDFL: Efficient Asynchronous Decentralized Federated Learning with Heterogeneous Devicescited by 15
  7. 2024Trustworthy Federated Learning: Privacy, Security, and Beyondcited by 11
  8. 2024FADAS: Towards Federated Adaptive Asynchronous Optimizationcited by 10
  9. 2023Efficient Asynchronous Federated Learning with Sparsification and Quantizationcited by 9
  10. 2024Enhancing Trust and Privacy in Distributed Networks: A Comprehensive Survey on Blockchain-based Federated Learningcited by 9
  11. 2023Making Batch Normalization Great in Federated Deep Learningcited by 8
  12. 2023Federated Zeroth-Order Optimization using Trajectory-Informed Surrogate Gradientscited by 8
11 more
  1. 2025Bridging Symmetry and Robustness: On the Role of Equivariance in Enhancing Adversarial Robustnesscited by 8
  2. 2024Federated Motor Imagery Classification for Privacy-Preserving Brain-Computer Interfacescited by 7
  3. 2024FedLion: Faster Adaptive Federated Optimization with Fewer Communicationcited by 6
  4. 2024Efficient Federated Learning Using Dynamic Update and Adaptive Pruning with Momentum on Shared Server Datacited by 6
  5. 2024Fisher Information-based Efficient Curriculum Federated Learning with Large Language Modelscited by 6
  6. 2023V2X-Boosted Federated Learning for Cooperative Intelligent Transportation Systems with Contextual Client Selectioncited by 5
  7. 2024Federated brain tumor segmentation: an extensive benchmarkcited by 5
  8. 2025PDSL: Privacy-Preserved Decentralized Stochastic Learning with Heterogeneous Data Distributioncited by 2
  9. 2025Efficient Federated Learning with Heterogeneous Data and Adaptive Dropoutcited by 2
  10. 2023Tackling Computational Heterogeneity in FL: A Few Theoretical Insightscited by 0
  11. 2026Pushing the (Decision) Boundaries: Dynamically Calibrating Differentially Private Noise to Explainability in Federated Learningcited by 0