mlsanityData to 7 Oct 2026

On the Convergence of FedAvg on Non-IID Data

arXiv 1907.02189 ↗First posted 4 Jul 2019stat.ML cs.LG math.OCcited by 1,220

Not on the map. Why?

Abstract (arXiv)

Federated learning enables a large amount of edge computing devices to jointly learn a model without data sharing. As a leading algorithm in this setting, Federated Averaging (textttFedAvg) runs Stochastic Gradient Descent (SGD) in parallel on a small subset of the total devices and averages the sequences only once in a while. Despite its simplicity, it lacks theoretical guarantees under realistic settings. In this paper, we analyze the convergence of textttFedAvg on non-iid data and establish a convergence rate of O(frac1T) for strongly convex and smooth problems, where T is the number of SGDs. Importantly, our bound demonstrates a trade-off between communication-efficiency and convergence rate. As user devices may be disconnected from the server, we relax the assumption of full device participation to partial device participation and study different averaging schemes; low device participation rate can be achieved without severely slowing down the learning. Our results indicate that heterogeneity of data slows down the convergence, which matches empirical observations. Furthermore, we provide a necessary condition for textttFedAvg on non-iid data: the learning rate η must decay, even if full-gradient is used; otherwise, the solution will be Omega (η) away from the optimal.

Key facts. Cited by 1,220 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 204; 2022: 234; 2023: 238; 2024: 241; 2025: 192; 2026: 110.

1,220 papers.

Sort:

By area:

  1. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  2. 2021Model-Contrastive Federated Learningcited by 762
  3. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  4. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  5. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  6. 2021Towards Personalized Federated Learningcited by 462
  7. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  8. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  9. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  10. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  11. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  12. 2021Trustworthy AI: From Principles to Practicescited by 206
38 more
  1. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
  2. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
  3. 2022FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correctioncited by 181
  4. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  5. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  6. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 144
  7. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
  8. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
  9. 2021Decentralized Federated Averagingcited by 114
  10. 2022Federated Learning with Partial Model Personalizationcited by 114
  11. 2021FedMix: Approximation of Mixup under Mean Augmented Federated Learningcited by 108
  12. 2021Distributionally Robust Federated Averagingcited by 104
  13. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 100
  14. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
  15. 2021Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applicationscited by 96
  16. 2021Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated Learningcited by 93
  17. 2021Trustworthy AI: A Computational Perspectivecited by 93
  18. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  19. 2021CRFL: Certifiably Robust Federated Learning against Backdoor Attackscited by 91
  20. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 89
  21. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  22. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  23. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  24. 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 79
  25. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  26. 2023Federated Learning for Medical Image Analysis: A Surveycited by 77
  27. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 73
  28. 2021Fast Federated Learning in the Presence of Arbitrary Device Unavailabilitycited by 72
  29. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  30. 2022An Improved Analysis of Gradient Tracking for Decentralized Machine Learningcited by 68
  31. 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 67
  32. 2021Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Samplingcited by 66
  33. 2023FedDisco: Federated Learning with Discrepancy-Aware Collaborationcited by 66
  34. 2021LightSecAgg: a Lightweight and Versatile Design for Secure Aggregation in Federated Learningcited by 63
  35. 2021What Do We Mean by Generalization in Federated Learning?cited by 62
  36. 2021HarmoFL: Harmonizing Local and Global Drifts in Federated Learning on Heterogeneous Medical Imagescited by 62
  37. 2022FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learningcited by 62
  38. 2022FedCorr: Multi-Stage Federated Learning for Label Noise Correctioncited by 61

50 most cited of 1,220 All 1,220 (JSON)