mlsanityData to 14 Oct 2026

Understanding Generalization of Federated Learning via Stability: Heterogeneity Matters

arXiv 2306.03824 ↗First posted 6 Jun 2023cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

Generalization performance is a key metric in evaluating machine learning models when applied to real-world applications. Good generalization indicates the model can predict unseen data correctly when trained under a limited number of data. Federated learning (FL), which has emerged as a popular distributed learning framework, allows multiple devices or clients to train a shared model without violating privacy requirements. While the existing literature has studied extensively the generalization performances of centralized machine learning algorithms, similar analysis in the federated settings is either absent or with very restrictive assumptions on the loss functions. In this paper, we aim to analyze the generalization performances of federated learning by means of algorithmic stability, which measures the change of the output model of an algorithm when perturbing one data point. Three widely-used algorithms are studied, including FedAvg, SCAFFOLD, and FedProx, under convex and non-convex loss functions. Our analysis shows that the generalization performances of models trained by these three algorithms are closely related to the heterogeneity of clients' datasets as well as the convergence behaviors of the algorithms. Particularly, in the i.i.d. setting, our results recover the classical results of stochastic gradient descent (SGD).

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 34 of its 62 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

34 of 62 references.

References · 34

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2019Advances and Open Problems in Federated Learningcited by 3,046
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  4. 2018Federated Learning with Non-IID Datacited by 1,108
  5. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  6. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  7. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  8. 2019Agnostic Federated Learningcited by 456
  9. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 402
  10. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
  11. 2017Three Factors Influencing Minima in SGDcited by 273
  12. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
22 more
  1. 2016Katyusha: The First Direct Acceleration of Stochastic Gradient Methodscited by 135
  2. 2018Rademacher Complexity for Adversarially Robust Generalizationcited by 132
  3. 2020The Heavy-Tail Phenomenon in SGDcited by 124
  4. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 123
  5. 2020Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descentcited by 119
  6. 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
  7. 2017Data-Dependent Stability of Stochastic Gradient Descentcited by 92
  8. 2020Federated Accelerated Stochastic Gradient Descentcited by 85
  9. 2017Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpointscited by 82
  10. 2020Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithmscited by 78
  11. 2020FedDANE: A Federated Newton-Type Methodcited by 76
  12. 2014A Stochastic Quasi-Newton Method for Large-Scale Optimizationcited by 69
  13. 2018Stability and Convergence Trade-off of Iterative Optimization Algorithmscited by 53
  14. 2020Federated Composite Optimizationcited by 49
  15. 2022On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness and Beyondcited by 38
  16. 2021Generalization of Model-Agnostic Meta-Learning Algorithms: Recurring and Unseen Taskscited by 36
  17. 2021Generalization Bounds for Noisy Iterative Algorithms Using Properties of Additive Noise Channelscited by 15
  18. 2021Minimax Estimation for Personalized Federated Learning: An Alternative between FedAvg and Local Training?cited by 15
  19. 2018Graph-Dependent Implicit Regularisation for Distributed Stochastic Subgradient Descentcited by 14
  20. 2022Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learningcited by 14
  21. 2022A Communication-efficient Algorithm with Linear Convergence for Federated Minimax Learningcited by 9
  22. 2021FedHybrid: A Hybrid Primal-Dual Algorithm Framework for Federated Optimizationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 7; 2025: 13; 2026: 3.

27 papers.

Sort:

By area:

  1. 2025FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averagingcited by 25
  2. 2024Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directionscited by 13
  3. 2024Federated Learning with MMD-based Early Stopping for Adaptive GNSS Interference Classificationcited by 7
  4. 2023Minibatch and Local SGD: Algorithmic Stability and Linear Speedup in Generalizationcited by 5
  5. 2024Differentiated Aggregation to Improve Generalization in Federated Learningcited by 4
  6. 2025Revisiting LocalSGD and SCAFFOLD: Improved Rates and Missing Analysiscited by 4
  7. 2025FedMVP: Federated Multimodal Visual Prompt Tuning for Vision-Language Modelscited by 4
  8. 2023Towards Understanding Generalization and Stability Gaps between Centralized and Decentralized Federated Learningcited by 3
  9. 2023Federated Learning with Nonvacuous Generalisation Boundscited by 2
  10. 2023Information-Theoretic Generalization Analysis for Topology-aware Heterogeneous Federated Edge Learning over Noisy Channelscited by 2
  11. 2025Generalization in Federated Learning: A Conditional Mutual Information Frameworkcited by 2
  12. 2025Tight Stability Bounds for Robust Distributed Learning: Byzantine Failures Hurt Generalization More than Data Poisoningcited by 2
15 more
  1. 2025An Information-Theoretic Analysis for Federated Learning under Concept Driftcited by 2
  2. 2025Heterogeneity Matters even More in Distributed Learning: Study from Generalization Perspectivecited by 1
  3. 2025Generalization Error Analysis for Attack-Free and Byzantine-Resilient Decentralized Learning with Data Heterogeneitycited by 1
  4. 2025Bridging Generalization Gap of Heterogeneous Federated Clients Using Generative Modelscited by 1
  5. 2025BadFU: Backdoor Federated Learning through Adversarial Machine Unlearningcited by 1
  6. 2025On the Convergence and Stability of Distributed Sub-model Trainingcited by 1
  7. 2026Stability beyond Bounded Differences: Sharp Generalization Bounds under Finite Lp Momentscited by 1
  8. 2024Can We Theoretically Quantify the Impacts of Local Updates on the Generalization Performance of Federated Learning?cited by 0
  9. 2024A Statistical Analysis of Deep Federated Learning for Intrinsically Low-dimensional Datacited by 0
  10. 2024Understanding Generalization of Federated Learning: the Trade-off between Model Stability and Optimizationcited by 0
  11. 2024How Does the Smoothness Approximation Method Facilitate Generalization for Federated Adversarial Learning?cited by 0
  12. 2025FedEve: On Bridging the Client Drift and Period Drift for Cross-device Federated Learningcited by 0
  13. 2025Local Performance vs. Out-of-Distribution Generalization: An Empirical Analysis of Personalized Federated Learning in Heterogeneous Data Environmentscited by 0
  14. 2026Learning Theory of the SVRG: Generalization and Convergence Analysiscited by 0
  15. 2026Dangerous Liaisons of Convex Learning and Non-Affine Aggregationcited by 0