mlsanityData to 14 Oct 2026

Towards Understanding Generalization and Stability Gaps between Centralized and Decentralized Federated Learning

arXiv 2310.03461 ↗First posted 5 Oct 2023cs.LG math.OCcited by 3

Not on the map. Why?

Abstract (arXiv)

As two mainstream frameworks in federated learning (FL), both centralized and decentralized approaches have shown great application value in practical scenarios. However, existing studies do not provide sufficient evidence and clear guidance for analysis of which performs better in the FL community. Although decentralized methods have been proven to approach the comparable convergence of centralized with less communication, their test performance always falls short of expectations in empirical studies. To comprehensively and fairly compare their efficiency gaps in FL, in this paper, we explore their stability and generalization efficiency. Specifically, we prove that on the general smooth non-convex objectives, 1) centralized FL (CFL) always generalizes better than decentralized FL (DFL); 2) CFL achieves the best performance via adopting partial participation instead of full participation; and, 3) there is a necessary requirement for the topology in DFL to avoid performance collapse as the training scale increases. We also conduct extensive experiments on several common setups in FL to validate that our theoretical analysis is consistent with experimental phenomena and contextually valid in several general and practical scenarios.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 53 of its 81 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

53 of 81 references.

References · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  4. 2020Adaptive Federated Optimizationcited by 1,046
  5. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  6. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  7. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  8. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 502
  9. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 367
  10. 2019Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communicationcited by 221
  11. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
  12. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 180
38 more
  1. 2020The Lipschitz Constant of Self-Attentioncited by 174
  2. 2014EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimizationcited by 160
  3. 2017Asynchronous Decentralized Parallel Stochastic Gradient Descentcited by 159
  4. 2019On the Convergence of Local Descent Methods in Federated Learningcited by 153
  5. 2018Stochastic Gradient Push for Distributed Deep Learningcited by 152
  6. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 144
  7. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  8. 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
  9. 2021Decentralized Federated Averagingcited by 114
  10. 2020Robust Federated Learning: The Case of Affine Distribution Shiftscited by 102
  11. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  12. 2020Local SGD: Unified Theory and New Efficient Methodscited by 88
  13. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  14. 2021On Large-Cohort Training for Federated Learningcited by 74
  15. 2019Decentralized Federated Learning: A Segmented Gossip Approachcited by 64
  16. 2021What Do We Mean by Generalization in Federated Learning?cited by 62
  17. 2021Exponential Graph is Provably Efficient for Decentralized Deep Trainingcited by 56
  18. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  19. 2023Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscapecited by 46
  20. 2023Improving the Model Consistency of Decentralized Federated Learningcited by 45
  21. 2021A Unified and Refined Convergence Analysis for Non-Convex Decentralized Learningcited by 44
  22. 2022Topology-aware Generalization of Decentralized SGDcited by 38
  23. 2021Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothnesscited by 37
  24. 2019On the Influence of Bias-Correction on Distributed Stochastic Optimizationcited by 34
  25. 2021Stability of SGD: Tightness Analysis and Improved Boundscited by 33
  26. 2022Stability Analysis and Generalization Bounds of Adversarial Trainingcited by 30
  27. 2023Understanding Generalization of Federated Learning via Stability: Heterogeneity Matterscited by 27
  28. 2022FedADMM: A Robust Federated Deep Learning Framework with Adaptivity to System Heterogeneitycited by 25
  29. 2023Local Exact-Diffusion for Decentralized Optimization and Learningcited by 24
  30. 2023Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedupcited by 23
  31. 2020Information-Theoretic Bounds on the Generalization Error and Privacy Leakage in Federated Learningcited by 21
  32. 2022On the Performance of Gradient Tracking with Local Updatescited by 21
  33. 2023Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initializationcited by 21
  34. 2022Beyond Uniform Lipschitz Condition in Differentially Private Optimizationcited by 19
  35. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15
  36. 2023Fusion of Global and Local Knowledge for Personalized Federated Learningcited by 14
  37. 2022Gradient Descent in the Absence of Global Lipschitz Continuity of the Gradientscited by 11
  38. 2023Improved Stability and Generalization Guarantees of the Decentralized SGD Algorithmcited by 10

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 2.

3 papers.

  1. 2024A Unified Analysis for Finite Weight Averagingcited by 3
  2. 2026OmniISR: A Unified Framework for Centralized and Federated Learning via Intermediate Supervision and Regularizationcited by 0
  3. 2026SPADE-DFL: Communication-Efficient Decentralized Federated Learning via Derivative-Free Linearized ADMMcited by 0