mlsanityData to 14 Oct 2026

Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Training

arXiv 2305.15157 ↗First posted 24 May 2023cs.LG cs.DC math.OCcited by 8

Not on the map. Why?

Abstract (arXiv)

Personalized federated learning (PFL) aims to produce the greatest personalized model for each client to face an insurmountable problem--data heterogeneity in real FL systems. However, almost all existing works have to face large communication burdens and the risk of disruption if the central server fails. Only limited efforts have been used in a decentralized way but still suffers from inferior representation ability due to sharing the full model with its neighbors. Therefore, in this paper, we propose a personalized FL framework with a decentralized partial model training called DFedAlt. It personalizes the "right" components in the modern deep models by alternately updating the shared and personal parameters to train partially personalized models in a peer-to-peer manner. To further promote the shared parameters aggregation process, we propose DFedSalt integrating the local Sharpness Aware Minimization (SAM) optimizer to update the shared parameters. It adds proper perturbation in the direction of the gradient to overcome the shared model inconsistency across clients. Theoretically, we provide convergence analysis of both algorithms in the general non-convex setting for decentralized partial model training in PFL. Our experiments on several real-world data with various data partition settings demonstrate that (i) decentralized training is more suitable for partial personalization, which results in state-of-the-art (SOTA) accuracy compared with the SOTA PFL baselines; (ii) the shared parameters with proper perturbation make partial personalized FL more suitable for decentralized training, where DFedSalt achieves most competitive performance.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 58 of its 73 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

58 of 73 references.

References · 58

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  5. 2018Group Normalizationcited by 1,524
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  7. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  8. 2020Adaptive Federated Optimizationcited by 1,046
  9. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  10. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  11. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  12. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
38 more
  1. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  2. 2019Federated Learning with Personalization Layerscited by 541
  3. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  4. 2020An Efficient Framework for Clustered Federated Learningcited by 511
  5. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 503
  6. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 502
  7. 2021Towards Personalized Federated Learningcited by 464
  8. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  9. 2020Adaptive Personalized Federated Learningcited by 358
  10. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  11. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  12. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  13. 2020Personalized Cross-Silo Federated Learning on Non-IID Datacited by 283
  14. 2019Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communicationcited by 221
  15. 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
  16. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
  17. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 182
  18. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  19. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  20. 2022Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challengescited by 138
  21. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  22. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  23. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  24. 2019Overcoming Forgetting in Federated Learning on Non-IID Datacited by 120
  25. 2019BrainTorrent: A Peer-to-Peer Environment for Decentralized Federated Learningcited by 117
  26. 2021Decentralized Federated Averagingcited by 114
  27. 2022Federated Learning with Partial Model Personalizationcited by 114
  28. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  29. 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
  30. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  31. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
  32. 2022Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learningcited by 58
  33. 2021Blockchain Assisted Decentralized Federated Learning (BLADE-FL): Performance Analysis and Resource Allocationcited by 56
  34. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  35. 2023Improving the Model Consistency of Decentralized Federated Learningcited by 45
  36. 2022Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Modelscited by 42
  37. 2022Topology-aware Generalization of Decentralized SGDcited by 38
  38. 2016Gossip training for deep learningcited by 35

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 4; 2025: 1.

8 papers.

  1. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15
  2. 2024Decentralized Directed Collaboration for Personalized Federated Learningcited by 9
  3. 2023DFedADMM: Dual Constraints Controlled Model Inconsistency for Decentralized Federated Learningcited by 6
  4. 2024FedStaleWeight: Buffered Asynchronous Federated Learning with Fair Aggregation via Staleness Reweightingcited by 5
  5. 2023Mitigating Communications Threats in Decentralized Federated Learning through Moving Target Defensecited by 1
  6. 2024MAPL: Model Agnostic Peer-to-peer Learningcited by 0
  7. 2024P4: Towards private, personalized, and Peer-to-Peer learningcited by 0
  8. 2025Client Clustering Meets Knowledge Sharing: Enhancing Privacy and Robustness in Personalized Peer-to-Peer Learningcited by 0