mlsanityData to 7 Oct 2026

Improving the Model Consistency of Decentralized Federated Learning

arXiv 2302.04083 ↗First posted 8 Feb 2023cs.LG cs.DC math.OCcited by 45

Not on the map. Why?

Abstract (arXiv)

To mitigate the privacy leakages and communication burdens of Federated Learning (FL), decentralized FL (DFL) discards the central server and each client only communicates with its neighbors in a decentralized communication network. However, existing DFL suffers from high inconsistency among local clients, which results in severe distribution shift and inferior performance compared with centralized FL (CFL), especially on heterogeneous data or sparse communication topology. To alleviate this issue, we propose two DFL algorithms named DFedSAM and DFedSAM-MGS to improve the performance of DFL. Specifically, DFedSAM leverages gradient perturbation to generate local flat models via Sharpness Aware Minimization (SAM), which searches for models with uniformly low loss values. DFedSAM-MGS further boosts DFedSAM by adopting Multiple Gossip Steps (MGS) for better model consistency, which accelerates the aggregation of local flat models and better balances communication complexity and generalization. Theoretically, we present improved convergence rates small Obig(frac1√KT+frac1T+frac1K1/2T3/2(1-λ)2big) and small Obig(frac1√KT+frac1T+fracλQ+1K1/2T3/2(1-λQ)2big) in non-convex setting for DFedSAM and DFedSAM-MGS, respectively, where 1-λ is the spectral gap of gossip matrix and Q is the number of MGS. Empirically, our methods can achieve competitive performance compared with CFL methods and outperform existing DFL methods.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 44 of its 68 references that are in the map. First posted on arXiv 8 Feb 2023.

Builds on

44 of 68 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2019Advances and Open Problems in Federated Learningcited by 3,021
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  7. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  8. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  9. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  10. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  11. 2020Adaptive Federated Optimizationcited by 1,036
  12. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
32 more
  1. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  2. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  3. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  4. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
  5. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 361
  6. 2020Adaptive Personalized Federated Learningcited by 353
  7. 2021A Field Guide to Federated Optimizationcited by 312
  8. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  9. 2020Personalized Cross-Silo Federated Learning on Non-IID Datacited by 282
  10. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
  11. 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
  12. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
  13. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  14. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  15. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  16. 2022Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challengescited by 137
  17. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  18. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  19. 2019BrainTorrent: A Peer-to-Peer Environment for Decentralized Federated Learningcited by 115
  20. 2021Decentralized Federated Averagingcited by 114
  21. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  22. 2019Peer-to-peer Federated Learning on Graphscited by 85
  23. 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
  24. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  25. 2022Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learningcited by 57
  26. 2021Blockchain Assisted Decentralized Federated Learning (BLADE-FL): Performance Analysis and Resource Allocationcited by 55
  27. 2022Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Modelscited by 42
  28. 2022Topology-aware Generalization of Decentralized SGDcited by 37
  29. 2022FedADMM: A Federated Primal-Dual Algorithm Allowing Partial Participationcited by 35
  30. 2021DeEPCA: Decentralized Exact PCA with Linear Convergence Ratecited by 18
  31. 2022Blockchain-based Federated Learning for Industrial Metaverses: Incentive Scheme with Optimal AoIcited by 16
  32. 2022NET-FLEET: Achieving Linear Convergence Speedup for Fully Decentralized Federated Learning with Heterogeneous Datacited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 13; 2025: 12; 2026: 6.

45 papers.

Sort:

By area:

  1. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 160
  2. 2023Decentralized Federated Learning: A Survey and Perspectivecited by 86
  3. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  4. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  5. 2023Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscapecited by 46
  6. 2023Make Landscape Flatter in Differentially Private Federated Learningcited by 33
  7. 2023Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initializationcited by 21
  8. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15
  9. 2024Decentralized Directed Collaboration for Personalized Federated Learningcited by 9
  10. 2023Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Trainingcited by 8
  11. 2024One-Shot Sequential Federated Learning for Non-IID Data by Enhancing Local Model Diversitycited by 8
  12. 2024Byzantine-Robust Aggregation for Securing Decentralized Federated Learningcited by 7
33 more
  1. 2023DFedADMM: Dual Constraints Controlled Model Inconsistency for Decentralized Federated Learningcited by 6
  2. 2025DeepSeek-Inspired Exploration of RL-based LLMs and Synergy with Wireless Networks: A Surveycited by 6
  3. 2025Exact and Linear Convergence for Federated Learning under Arbitrary Client Participation is Attainablecited by 6
  4. 2023Towards the Flatter Landscape and Better Generalization in Federated Learning under Client-level Differential Privacycited by 5
  5. 2023Asymmetrically Decentralized Federated Learningcited by 5
  6. 2024NTK-DFL: Enhancing Decentralized Federated Learning in Heterogeneous Settings via Neural Tangent Kernelcited by 4
  7. 2023Towards Understanding Generalization and Stability Gaps between Centralized and Decentralized Federated Learningcited by 3
  8. 2024Decentralized Personalized Federated Learning based on a Conditional Sparse-to-Sparser Schemecited by 3
  9. 2024Review of Mathematical Optimization in Federated Learningcited by 3
  10. 2023Sparse Decentralized Federated Learningcited by 2
  11. 2024Privacy Preserved Blood Glucose Level Cross-Prediction: An Asynchronous Decentralized Federated Learning Approachcited by 2
  12. 2024Boosting the Performance of Decentralized Federated Learning via Catalyst Accelerationcited by 2
  13. 2025FedDAG: Federated Domain Adversarial Generation Towards Generalizable Medical Image Analysiscited by 2
  14. 2025FedBM: Stealing Knowledge from Pre-trained Language Models for Heterogeneous Federated Learningcited by 2
  15. 2025PDSL: Privacy-Preserved Decentralized Stochastic Learning with Heterogeneous Data Distributioncited by 2
  16. 2024OledFL: Unleashing the Potential of Decentralized Federated Learning via Opposite Lookahead Enhancementcited by 1
  17. 2025DBA-DFL: Towards Distributed Backdoor Attacks with Network Detection in Decentralized Federated Learningcited by 1
  18. 2025Unveiling the Power of Multiple Gossip Steps: A Stability-Based Generalization Analysis in Decentralized Trainingcited by 1
  19. 2026Decentralized Federated Learning by Partial Message Exchangecited by 1
  20. 2024MAPL: Model Agnostic Peer-to-peer Learningcited by 0
  21. 2024SADDLe: Sharpness-Aware Decentralized Deep Learning with Heterogeneous Datacited by 0
  22. 2024P4: Towards private, personalized, and Peer-to-Peer learningcited by 0
  23. 2024Equitable Federated Learning with Activation Clusteringcited by 0
  24. 2025Lossless Privacy-Preserving Aggregation for Decentralized Federated Learningcited by 0
  25. 2025Gradient Purification: Defense Against Poisoning Attack in Decentralized Federated Learningcited by 0
  26. 2025Client Clustering Meets Knowledge Sharing: Enhancing Privacy and Robustness in Personalized Peer-to-Peer Learningcited by 0
  27. 2025ADP-VRSGP: Decentralized Learning with Adaptive Differential Privacy via Variance-Reduced Stochastic Gradient Pushcited by 0
  28. 2025Communication-Efficient Neural Tangent Kernels for Heterogeneous Decentralized Federated Learningcited by 0
  29. 2026ERIS: Enhancing Privacy and Scalability in Federated Learning via Federated Shard Aggregationcited by 0
  30. 2026Rethinking the Personalized Relaxed Initialization in the Federated Learning: Consistency and Generalizationcited by 0
  31. 2026DECA: Decentralizing Block-Wise Adam for Efficient LLM Full-Parameter Fine-Tuning on Non-IID Datacited by 0
  32. 2026FedA2L: Adaptive layer-wise learning rate adjustment in decentralized federated learningcited by 0
  33. 2026Model-Consistent Byzantine-Resilient Decentralized Federated Learning for Collaborative Missionscited by 0