mlsanityData to 7 Oct 2026

Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer

arXiv 2109.08119 ↗First posted 16 Sep 2021cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

Personalized federated learning (FL) aims to train model(s) that can perform well for individual clients that are highly data and system heterogeneous. Most work in personalized FL, however, assumes using the same model architecture at all clients and increases the communication cost by sending/receiving models. This may not be feasible for realistic scenarios of FL. In practice, clients have highly heterogeneous system-capabilities and limited communication resources. In our work, we propose a personalized FL framework, PerFed-CKT, where clients can use heterogeneous model architectures and do not directly communicate their model parameters. PerFed-CKT uses clustered co-distillation, where clients use logits to transfer their knowledge to other clients that have similar data-distributions. We theoretically show the convergence and generalization properties of PerFed-CKT and empirically show that PerFed-CKT achieves high test accuracy with several orders of magnitude lower communication cost compared to the state-of-the-art personalized FL schemes.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 16 Sep 2021.

Builds on

41 of 55 references.

References · 41

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2019Advances and Open Problems in Federated Learningcited by 3,021
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  7. 2020Adaptive Federated Optimizationcited by 1,036
  8. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  9. 2019Towards Federated Learning at Scale: System Designcited by 826
  10. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  11. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  12. 2017Deep Mutual Learningcited by 621
29 more
  1. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  2. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  3. 2020An Efficient Framework for Clustered Federated Learningcited by 505
  4. 2021Towards Personalized Federated Learningcited by 462
  5. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  6. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 361
  7. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 322
  8. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 321
  9. 2021A Field Guide to Federated Optimizationcited by 312
  10. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  11. 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
  12. 2020Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategiescited by 196
  13. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  14. 2020Privacy and Robustness in Federated Learning: Attacks and Defensescited by 163
  15. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  16. 2019On the Convergence of Local Descent Methods in Federated Learningcited by 150
  17. 2018Large scale distributed neural network training through online distillationcited by 144
  18. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 143
  19. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  20. 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
  21. 2020Distilled One-Shot Federated Learningcited by 108
  22. 2018Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithmscited by 106
  23. 2020Practical One-Shot Federated Learning for Cross-Silo Settingcited by 76
  24. 2020From Local SGD to Local Fixed-Point Methods for Federated Learningcited by 74
  25. 2020Federated Model Distillation with Noise-Free Differential Privacycited by 40
  26. 2020Faster On-Device Training Using New Federated Momentum Algorithmcited by 25
  27. 2021Federated Learning with Positive and Unlabeled Datacited by 10
  28. 2020A Closer Look at Codistillation for Distributed Trainingcited by 8
  29. 2020Adaptive Distillation for Decentralized Learning from Heterogeneous Clientscited by 5

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 8; 2024: 6; 2025: 3.

23 papers.

Sort:

  1. 2023Personalized Federated Learning with Feature Alignment and Classifier Collaborationcited by 97
  2. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  3. 2022On Privacy and Personalization in Cross-Silo Federated Learningcited by 36
  4. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  5. 2022Towards Fair Federated Recommendation Learning: Characterizing the Inter-Dependence of System and Data Heterogeneitycited by 17
  6. 2023FedICT: Federated Multi-task Distillation for Multi-access Edge Computingcited by 17
  7. 2023Visual Prompt Based Personalized Federated Learningcited by 16
  8. 2022Personalized Federated Learning via Convex Clusteringcited by 10
  9. 2022Straggler-Resilient Personalized Federated Learningcited by 9
  10. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutionscited by 8
  11. 2023A Practical Recipe for Federated Learning Under Statistical Heterogeneity Experimental Designcited by 5
  12. 2023Leveraging Foundation Models to Improve Lightweight Clients in Federated Learningcited by 5
11 more
  1. 2023FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learningcited by 5
  2. 2022Modular Federated Learningcited by 4
  3. 2025Towards Personalized Quantum Federated Learning for Anomaly Detectioncited by 4
  4. 2024Understanding Server-Assisted Federated Learning in the Presence of Incomplete Client Participationcited by 3
  5. 2024Distributionally Robust Clustered Federated Learning: A Case Study in Healthcarecited by 2
  6. 2024Personalized Federated Learning with Adaptive Feature Aggregation and Knowledge Transfercited by 2
  7. 2023Plug In and Learn: Federated Intelligence over a Smart Grid of Modelscited by 1
  8. 2023Tackling Computational Heterogeneity in FL: A Few Theoretical Insightscited by 0
  9. 2024Centralized and Federated Heart Disease Classification Models Using UCI Dataset and their Shapley-value Based Interpretabilitycited by 0
  10. 2025Whom to Trust? Adaptive Collaboration in Personalized Federated Learningcited by 0
  11. 2025Towards a Larger Model via One-Shot Federated Learning on Heterogeneous Client Modelscited by 0