mlsanityData to 14 Oct 2026

Adaptive Federated Optimization

arXiv 2003.00295 ↗First posted 29 Feb 2020cs.LG cs.DC math.OCcited by 1,046

Not on the map. Why?

Cited by 22 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Federated learning is a distributed machine learning paradigm in which a large number of clients coordinate with a central server to learn a model without sharing their own training data. Standard federated optimization methods such as Federated Averaging (FedAvg) are often difficult to tune and exhibit unfavorable convergence behavior. In non-federated settings, adaptive optimization methods have had notable success in combating such issues. In this work, we propose federated versions of adaptive optimizers, including Adagrad, Adam, and Yogi, and analyze their convergence in the presence of heterogeneous data for general non-convex settings. Our results highlight the interplay between client heterogeneity and communication efficiency. We also perform extensive experiments on these methods and show that the use of adaptive optimizers can significantly improve the performance of federated learning.

Key facts. Cited by 1,046 papers in the map (data through 14 Oct 2026). First posted on arXiv 29 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 135; 2022: 192; 2023: 219; 2024: 180; 2025: 176; 2026: 144.

1,046 papers.

Sort:

By area:

  1. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
  2. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  3. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  4. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  5. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  6. 2021Federated Learning on Non-IID Data: A Surveycited by 346
  7. 2021A Field Guide to Federated Optimizationcited by 315
  8. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  9. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
  10. 2021Federated Multi-Task Learning under a Mixture of Distributionscited by 192
  11. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 190
  12. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
38 more
  1. 2022FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correctioncited by 181
  2. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 170
  3. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  4. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
  5. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 136
  6. 2021The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregationcited by 128
  7. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 128
  8. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 127
  9. 2022FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settingscited by 125
  10. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 121
  11. 2021Back to the Drawing Board: A Critical Evaluation of Poisoning Attacks on Production Federated Learningcited by 118
  12. 2021FairFed: Enabling Group Fairness in Federated Learningcited by 118
  13. 2021Decentralized Federated Averagingcited by 114
  14. 2022Federated Learning with Partial Model Personalizationcited by 114
  15. 2022Federated Learning with Label Distribution Skew via Logits Calibrationcited by 109
  16. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 101
  17. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 99
  18. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  19. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 93
  20. 2023Deep Model Fusion: A Surveycited by 91
  21. 2021Papaya: Practical, Private, and Scalable Federated Learningcited by 90
  22. 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 85
  23. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 81
  24. 2022NVIDIA FLARE: Federated Learning from Simulation to Real-Worldcited by 80
  25. 2021Federated Reconstruction: Partially Local Federated Learningcited by 79
  26. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  27. 2021On Large-Cohort Training for Federated Learningcited by 74
  28. 2021Differentially Private Federated Learning on Heterogeneous Datacited by 74
  29. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  30. 2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 71
  31. 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 69
  32. 2021Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacycited by 69
  33. 2023FedExP: Speeding Up Federated Averaging via Extrapolationcited by 69
  34. 2021The Skellam Mechanism for Differentially Private Federated Learningcited by 68
  35. 2023FedDisco: Federated Learning with Discrepancy-Aware Collaborationcited by 67
  36. 2022FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learningcited by 66
  37. 2021LightSecAgg: a Lightweight and Versatile Design for Secure Aggregation in Federated Learningcited by 64
  38. 2021HarmoFL: Harmonizing Local and Global Drifts in Federated Learning on Heterogeneous Medical Imagescited by 64

50 most cited of 1,046 All 1,046 (JSON)