mlsanityData to 14 Oct 2026

Faster Adaptive Federated Learning

arXiv 2212.00974 ↗First posted 2 Dec 2022cs.LG cs.DCcited by 35

Not on the map. Why?

Abstract (arXiv)

Federated learning has attracted increasing attention with the emergence of distributed data. While extensive federated learning algorithms have been proposed for the non-convex distributed problem, federated learning in practice still faces numerous challenges, such as the large training iterations to converge since the sizes of models and datasets keep increasing, and the lack of adaptivity by SGD-based model updates. Meanwhile, the study of adaptive methods in federated learning is scarce and existing works either lack a complete theoretical convergence guarantee or have slow sample complexity. In this paper, we propose an efficient adaptive algorithm (i.e., FAFED) based on the momentum-based variance-reduced technique in cross-silo FL. We first explore how to design the adaptive algorithm in the FL setting. By providing a counter-example, we prove that a simple combination of FL and adaptive methods could lead to divergence. More importantly, we provide a convergence analysis for our method and prove that our algorithm is the first adaptive FL algorithm to reach the best-known samples O(ε-3) and O(ε-2) communication rounds to find an ε-stationary point without large batches. The experimental results on the language modeling task and image classification task with heterogeneous data demonstrate the efficiency of our algorithms.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 30 of its 56 references that are in the map. First posted on arXiv 2 Dec 2022.

Builds on

30 of 56 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2020Adaptive Federated Optimizationcited by 1,046
  6. 2019On the Convergence of Adam and Beyondcited by 841
  7. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 431
  8. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  9. 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 287
  10. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
  11. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  12. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
18 more
  1. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  2. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 180
  3. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 154
  4. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  5. 2019Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methodscited by 148
  6. 2022Communication-Efficient Adaptive Federated Learningcited by 55
  7. 2020Faster Non-Convex Federated Learning via Global and Local Momentumcited by 48
  8. 2021STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learningcited by 43
  9. 2022Closing the Generalization Gap of Cross-silo Federated Medical Image Segmentationcited by 42
  10. 2021SUPER-ADAM: Faster and Universal Framework of Adaptive Gradientscited by 40
  11. 2019Why are Adaptive Methods Good for Attention Models?cited by 35
  12. 2019Escaping Saddle Points with Adaptive Gradient Methodscited by 34
  13. 2020Fast OSCAR and OWL Regression via Safe Screening Rulescited by 25
  14. 2021Toward Communication Efficient Adaptive Gradient Methodcited by 25
  15. 2022RetrievalGuard: Provably Robust 1-Nearest Neighbor Image Retrievalcited by 15
  16. 2021Coordinating Momenta for Cross-silo Federated Learningcited by 9
  17. 2020Multi-Classifier Interactive Learning for Ambiguous Speech Emotion Recognitioncited by 7
  18. 2020Inflating Topic Relevance with Ideology: A Case Study of Political Ideology Bias in Social Topic Detection Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 15; 2025: 8; 2026: 4.

35 papers.

Sort:

By area:

  1. 2024AdaptSFL: Adaptive Split Federated Learning in Resource-constrained Edge Networkscited by 64
  2. 2023Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reductioncited by 24
  3. 2022Adversarial Weight Perturbation Improves Generalization in Graph Neural Networkscited by 21
  4. 2024Auto-Train-Once: Controller Network Guided Automatic Network Pruning from Scratchcited by 20
  5. 2024Leveraging Foundation Models for Multi-modal Federated Learning with Incomplete Modalitycited by 14
  6. 2023Solving a Class of Non-Convex Minimax Optimization in Federated Learningcited by 10
  7. 2024FADAS: Towards Federated Adaptive Asynchronous Optimizationcited by 10
  8. 2023Decentralized Riemannian Algorithm for Nonconvex Minimax Problemscited by 7
  9. 2025On Provable Benefits of Muon in Federated Learningcited by 7
  10. 2024FedLion: Faster Adaptive Federated Optimization with Fewer Communicationcited by 6
  11. 2024FedCAda: Adaptive Client-Side Optimization for Accelerated and Stable Federated Learningcited by 5
  12. 2023Federated Conditional Stochastic Optimizationcited by 4
23 more
  1. 2024Exploring the Practicality of Federated Learning: A Survey Towards the Communication Perspectivecited by 4
  2. 2024Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learningcited by 4
  3. 2023Serverless Federated AUPRC Optimization for Multi-Party Collaborative Imbalanced Data Miningcited by 3
  4. 2023On-ramp and Off-ramp Traffic Flows Estimation Based on A Data-driven Transfer Learning Frameworkcited by 3
  5. 2024Boosting Federated Domain Generalization: Understanding the Role of Advanced Pre-Trained Architecturescited by 3
  6. 2025Research in Collaborative Learning Does Not Serve Cross-Silo Federated Learning in Practicecited by 3
  7. 2026Rethinking LoRA for Data Heterogeneous Federated Learning: Subspace and State Alignmentcited by 2
  8. 2023Performance and Energy Consumption of Parallel Machine Learning Algorithmscited by 1
  9. 2024Federated Knowledge Graph Unlearning via Diffusion Modelcited by 1
  10. 2024Addressing Heterogeneity in Federated Load Forecasting with Personalization Layerscited by 1
  11. 2025Addressing Label Shift in Distributed Learning via Entropy Regularizationcited by 1
  12. 2025Adaptive Federated Distillation for Multi-Domain Non-IID Textual Datacited by 1
  13. 2025A Byzantine-Robust Federated Learning with Nesterov Momentumcited by 1
  14. 2026DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Modelscited by 1
  15. 2024FedSiKD: Clients Similarity and Knowledge Distillation: Addressing Non-i.i.d. and Constraints in Federated Learningcited by 0
  16. 2024A Whole-Process Certifiably Robust Aggregation Method Against Backdoor Attacks in Federated Learningcited by 0
  17. 2024Painless Federated Learning: An Interplay of Line-Search and Extrapolationcited by 0
  18. 2024FedGTST: Boosting Global Transferability of Federated Models via Statistics Tuningcited by 0
  19. 2025FedX: Adaptive Model Decomposition and Quantization for IoT Federated Learningcited by 0
  20. 2025Adaptive Federated Learning via Dynamical System Modelcited by 0
  21. 2025SGFusion: Stochastic Geographic Gradient Fusion in Federated Learningcited by 0
  22. 2026MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuningcited by 0
  23. 2026Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deploymentcited by 0