mlsanityData to 14 Oct 2026

Can We Theoretically Quantify the Impacts of Local Updates on the Generalization Performance of Federated Learning?

arXiv 2409.03863 ↗First posted 5 Sep 2024cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) has gained significant popularity due to its effectiveness in training machine learning models across diverse sites without requiring direct data sharing. While various algorithms along with their optimization analyses have shown that FL with local updates is a communication-efficient distributed learning framework, the generalization performance of FL with local updates has received comparatively less attention. This lack of investigation can be attributed to the complex interplay between data heterogeneity and infrequent communication due to the local updates within the FL framework. This motivates us to investigate a fundamental question in FL: Can we quantify the impact of data heterogeneity and local updates on the generalization performance for FL as the learning process evolves? To this end, we conduct a comprehensive theoretical study of FL's generalization performance using a linear model as the first step, where the data heterogeneity is considered for both the stationary and online/non-stationary cases. By providing closed-form expressions of the model error, we rigorously quantify the impact of the number of the local updates (denoted as K) under three settings (K=1, K<∞, and K=∞) and show how the generalization performance evolves with the number of rounds t. Our investigation also provides a comprehensive understanding of how different configurations (including the number of model parameters p and the number of training samples n) contribute to the overall generalization performance, thus shedding new insights (such as benign overfitting) for implementing FL over networks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 39 of its 56 references that are in the map. First posted on arXiv 5 Sep 2024.

Builds on

39 of 56 references.

References · 39

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2019Advances and Open Problems in Federated Learningcited by 3,046
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  4. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  5. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  6. 2018Federated Learning with Non-IID Datacited by 1,108
  7. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  8. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  9. 2019Benign Overfitting in Linear Regressioncited by 549
  10. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  11. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  12. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
27 more
  1. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
  2. 2019Two models of double descent for weak featurescited by 195
  3. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
  4. 2018To understand deep learning we need to understand kernel learningcited by 187
  5. 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
  6. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 154
  7. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  8. 2021Federated Learning for Open Bankingcited by 141
  9. 2020Federated Learning for Internet of Things: Recent Advances, Taxonomy, and Open Challengescited by 139
  10. 2019Harmless interpolation of noisy data in regressioncited by 121
  11. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  12. 2023Theory on Forgetting and Generalization of Continual Learningcited by 72
  13. 2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 71
  14. 2021What Do We Mean by Generalization in Federated Learning?cited by 62
  15. 2021Anarchic Federated Learningcited by 42
  16. 2023Make Landscape Flatter in Differentially Private Federated Learningcited by 33
  17. 2023Understanding Generalization of Federated Learning via Stability: Heterogeneity Matterscited by 27
  18. 2023Why (and When) does Local SGD Generalize Better than SGD?cited by 26
  19. 2021Trade-offs of Local SGD at Scale: An Empirical Studycited by 20
  20. 2022Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learningcited by 14
  21. 2021On the Generalization Power of Overfitted Two-Layer Neural Tangent Kernel Modelscited by 13
  22. 2021Meta-learning with negative learning ratescited by 11
  23. 2021The Dynamics of Gradient Descent for Overparametrized Neural Networkscited by 11
  24. 2023More Communication Does Not Result in Smaller Generalization Error in Federated Learningcited by 10
  25. 2023Theoretical Characterization of the Generalization Performance of Overfitted Meta-Learningcited by 4
  26. 2023Generalization Performance of Transfer Learning: Overparameterized and Underparameterized Regimescited by 4
  27. 2023Federated PAC-Bayesian Learning on Non-IID datacited by 2

Built on by

Not cited yet.