# Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer

arXiv: https://arxiv.org/abs/2109.08119 · HTML: https://mlsanity.com/paper/2109.08119/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 55 references that are in the map.
- First posted on arXiv 16 Sep 2021.

## Builds on (most cited first, 30 of 41)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Ditto: Fair and Robust Federated Learning Through Personalization](https://mlsanity.com/paper/2012.04221/) (2020)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Personalized Federated Learning with Moreau Envelopes](https://mlsanity.com/paper/2006.08848/) (2020)
- [Local SGD Converges Fast and Communicates Little](https://mlsanity.com/paper/1805.09767/) (2018)
- [An Efficient Framework for Clustered Federated Learning](https://mlsanity.com/paper/2006.04088/) (2020)
- [Towards Personalized Federated Learning](https://mlsanity.com/paper/2103.00710/) (2021)
- [FedMD: Heterogenous Federated Learning via Model Distillation](https://mlsanity.com/paper/1910.03581/) (2019)
- [A Unified Theory of Decentralized SGD with Changing Topology and Local Updates](https://mlsanity.com/paper/2003.10422/) (2020)
- [Think Locally, Act Globally: Federated Learning with Local and Global Representations](https://mlsanity.com/paper/2001.01523/) (2020)
- [Three Approaches for Personalization with Applications to Federated Learning](https://mlsanity.com/paper/2002.10619/) (2020)
- [A Field Guide to Federated Optimization](https://mlsanity.com/paper/2107.06917/) (2021)
- [Tighter Theory for Local SGD on Identical and Heterogeneous Data](https://mlsanity.com/paper/1909.04746/) (2019)
- [Personalized Federated Learning with First Order Model Optimization](https://mlsanity.com/paper/2012.08565/) (2020)
- [Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategies](https://mlsanity.com/paper/2010.01243/) (2020)
- [Knowledge Distillation by On-the-Fly Native Ensemble](https://mlsanity.com/paper/1806.04606/) (2018)
- [Privacy and Robustness in Federated Learning: Attacks and Defenses](https://mlsanity.com/paper/2012.06337/) (2020)
- [On the Convergence of Local Descent Methods in Federated Learning](https://mlsanity.com/paper/1910.14425/) (2019)
- [Is Local SGD Better than Minibatch SGD?](https://mlsanity.com/paper/2002.07839/) (2020)
- [Large scale distributed neural network training through online distillation](https://mlsanity.com/paper/1804.03235/) (2018)
- [FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data](https://mlsanity.com/paper/2005.11418/) (2020)

## Built on by (most cited first, 23 of 23)

- [Personalized Federated Learning with Feature Alignment and Classifier Collaboration](https://mlsanity.com/paper/2306.11867/) (2023)
- [Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning](https://mlsanity.com/paper/2204.12703/) (2022)
- [On Privacy and Personalization in Cross-Silo Federated Learning](https://mlsanity.com/paper/2206.07902/) (2022)
- [PrE-Text: Training Language Models on Private Federated Data in the Age of LLMs](https://mlsanity.com/paper/2406.02958/) (2024)
- [Towards Fair Federated Recommendation Learning: Characterizing the Inter-Dependence of System and Data Heterogeneity](https://mlsanity.com/paper/2206.02633/) (2022)
- [FedICT: Federated Multi-task Distillation for Multi-access Edge Computing](https://mlsanity.com/paper/2301.00389/) (2023)
- [Visual Prompt Based Personalized Federated Learning](https://mlsanity.com/paper/2303.08678/) (2023)
- [Personalized Federated Learning via Convex Clustering](https://mlsanity.com/paper/2202.00718/) (2022)
- [Straggler-Resilient Personalized Federated Learning](https://mlsanity.com/paper/2206.02078/) (2022)
- [Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions](https://mlsanity.com/paper/2406.10861/) (2024)
- [A Practical Recipe for Federated Learning Under Statistical Heterogeneity Experimental Design](https://mlsanity.com/paper/2307.15245/) (2023)
- [Leveraging Foundation Models to Improve Lightweight Clients in Federated Learning](https://mlsanity.com/paper/2311.08479/) (2023)
- [FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learning](https://mlsanity.com/paper/2311.16584/) (2023)
- [Modular Federated Learning](https://mlsanity.com/paper/2209.03090/) (2022)
- [Towards Personalized Quantum Federated Learning for Anomaly Detection](https://mlsanity.com/paper/2511.07471/) (2025)
- [Understanding Server-Assisted Federated Learning in the Presence of Incomplete Client Participation](https://mlsanity.com/paper/2405.02745/) (2024)
- [Distributionally Robust Clustered Federated Learning: A Case Study in Healthcare](https://mlsanity.com/paper/2410.07039/) (2024)
- [Personalized Federated Learning with Adaptive Feature Aggregation and Knowledge Transfer](https://mlsanity.com/paper/2410.15073/) (2024)
- [Plug In and Learn: Federated Intelligence over a Smart Grid of Models](https://mlsanity.com/paper/2302.04363/) (2023)
- [Tackling Computational Heterogeneity in FL: A Few Theoretical Insights](https://mlsanity.com/paper/2307.06283/) (2023)
- [Centralized and Federated Heart Disease Classification Models Using UCI Dataset and their Shapley-value Based Interpretability](https://mlsanity.com/paper/2408.06183/) (2024)
- [Whom to Trust? Adaptive Collaboration in Personalized Federated Learning](https://mlsanity.com/paper/2507.00259/) (2025)
- [Towards a Larger Model via One-Shot Federated Learning on Heterogeneous Client Models](https://mlsanity.com/paper/2508.13625/) (2025)

## Abstract (arXiv)

Personalized federated learning (FL) aims to train model(s) that can perform well for individual clients that are highly data and system heterogeneous. Most work in personalized FL, however, assumes using the same model architecture at all clients and increases the communication cost by sending/receiving models. This may not be feasible for realistic scenarios of FL. In practice, clients have highly heterogeneous system-capabilities and limited communication resources. In our work, we propose a personalized FL framework, PerFed-CKT, where clients can use heterogeneous model architectures and do not directly communicate their model parameters. PerFed-CKT uses clustered co-distillation, where clients use logits to transfer their knowledge to other clients that have similar data-distributions. We theoretically show the convergence and generalization properties of PerFed-CKT and empirically show that PerFed-CKT achieves high test accuracy with several orders of magnitude lower communication cost compared to the state-of-the-art personalized FL schemes.