mlsanityData to 14 Oct 2026

Understanding Generalization of Federated Learning: the Trade-off between Model Stability and Optimization

arXiv 2411.16303 ↗First posted 25 Nov 2024cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) is a distributed learning approach that trains machine learning models across multiple devices while keeping their local data private. However, FL often faces challenges due to data heterogeneity, leading to inconsistent local optima among clients. These inconsistencies can cause unfavorable convergence behavior and generalization performance degradation. Existing studies often describe this issue through textitconvergence analysis on gradient norms, focusing on how well a model fits training data, or through textitalgorithmic stability, which examines the generalization gap. However, neither approach precisely captures the generalization performance of FL algorithms, especially for non-convex neural network training. In response, this paper introduces an innovative generalization dynamics analysis framework, namely textitLibra, for algorithm-dependent excess risk minimization, highlighting the trade-offs between model stability and gradient norms. We present Libra towards a standard federated optimization framework and its variants using server momentum. Through this framework, we show that larger local steps or momentum accelerate convergence of gradient norms, while worsening model stability, yielding better excess risk. Experimental results on standard FL settings prove the insights of our theories. These insights can guide hyperparameter tuning and future algorithm design to achieve stronger generalization.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 56 of its 76 references that are in the map. First posted on arXiv 25 Nov 2024.

Builds on

56 of 76 references.

References · 56

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2019Advances and Open Problems in Federated Learningcited by 3,046
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  6. 2018Group Normalizationcited by 1,524
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  8. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  10. 2020Adaptive Federated Optimizationcited by 1,046
  11. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  12. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
38 more
  1. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  2. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  3. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  4. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  5. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 367
  6. 2021Federated Learning for Internet of Things: A Comprehensive Surveycited by 252
  7. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
  8. 2020Reasoning About Generalization via Conditional Mutual Informationcited by 147
  9. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  10. 2021Federated Learning for Open Bankingcited by 141
  11. 2017Non-convex Optimization for Machine Learningcited by 133
  12. 2018Rademacher Complexity for Adversarially Robust Generalizationcited by 132
  13. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 126
  14. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 121
  15. 2020Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descentcited by 119
  16. 2020Robust Federated Learning: The Case of Affine Distribution Shiftscited by 102
  17. 2017Stability and Generalization of Learning Algorithms that Converge to Global Optimacited by 100
  18. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 99
  19. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  20. 2017Data-Dependent Stability of Stochastic Gradient Descentcited by 92
  21. 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
  22. 2018Generalization Error Bounds for Noisy, Iterative Algorithmscited by 82
  23. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 80
  24. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  25. 2022FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learningcited by 64
  26. 2019On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learningcited by 57
  27. 2018Stability and Convergence Trade-off of Iterative Optimization Algorithmscited by 53
  28. 2018Privacy-preserving Predictioncited by 48
  29. 2023Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscapecited by 46
  30. 2020Toward Better Generalization Bounds with Locally Elastic Stabilitycited by 42
  31. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  32. 2017Algorithmic stability and hypothesis complexitycited by 39
  33. 2022On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Datacited by 38
  34. 2023Make Landscape Flatter in Differentially Private Federated Learningcited by 33
  35. 2023Understanding Generalization of Federated Learning via Stability: Heterogeneity Matterscited by 27
  36. 2022Stability and Generalization of Stochastic Optimization with Nonconvex and Nonsmooth Problemscited by 25
  37. 2023A Survey of Trustworthy Federated Learning with Perspectives on Security, Robustness, and Privacycited by 24
  38. 2019Hypothesis Set Stability and Generalizationcited by 23

50 most cited of 56 All 56 (JSON)

Built on by

Not cited yet.