# Personalized Decentralized Federated Learning with Knowledge Distillation

arXiv: https://arxiv.org/abs/2302.12156 · HTML: https://mlsanity.com/paper/2302.12156/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation.
- Builds on 24 of its 30 references that are in the map.
- First posted on arXiv 23 Feb 2023.

## Placed in

- Knowledge distillation / [Federated, robust and private distillation](https://mlsanity.com/distill/federated-robust-and-private-distillation/)

## Builds on (most cited first, 24 of 24)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Inverting Gradients -- How easy is it to break privacy in federated learning?](https://mlsanity.com/paper/2003.14053/) (2020)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Robust and Communication-Efficient Federated Learning from Non-IID Data](https://mlsanity.com/paper/1903.02891/) (2019)
- [Three Approaches for Personalization with Applications to Federated Learning](https://mlsanity.com/paper/2002.10619/) (2020)
- [Personalized Federated Learning: A Meta-Learning Approach](https://mlsanity.com/paper/2002.07948/) (2020)
- [Improving Federated Learning Personalization via Model Agnostic Meta Learning](https://mlsanity.com/paper/1909.12488/) (2019)
- [Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data](https://mlsanity.com/paper/1811.11479/) (2018)
- [Federated Multi-Task Learning under a Mixture of Distributions](https://mlsanity.com/paper/2108.10252/) (2021)
- [Large scale distributed neural network training through online distillation](https://mlsanity.com/paper/1804.03235/) (2018)
- [Federated Learning for Wireless Communications: Motivation, Opportunities and Challenges](https://mlsanity.com/paper/1908.06847/) (2019)
- [Decentralized Collaborative Learning of Personalized Models over Networks](https://mlsanity.com/paper/1610.05202/) (2016)
- [Personalized Federated Learning through Local Memorization](https://mlsanity.com/paper/2111.09360/) (2021)
- [Personalized and Private Peer-to-Peer Machine Learning](https://mlsanity.com/paper/1705.08435/) (2017)
- [Fully Decentralized Joint Learning of Personalized Models and Collaboration Graphs](https://mlsanity.com/paper/1901.08460/) (2019)
- [Motley: Benchmarking Heterogeneity and Personalization in Federated Learning](https://mlsanity.com/paper/2206.09262/) (2022)
- [Decentralized Personalized Federated Learning: Lower Bounds and Optimal Algorithm for All Personalization Modes](https://mlsanity.com/paper/2107.07190/) (2021)
- [Decentralized Personalized Federated Learning for Min-Max Problems](https://mlsanity.com/paper/2106.07289/) (2021)
- [Unifying Distillation with Personalization in Federated Learning](https://mlsanity.com/paper/2105.15191/) (2021)

## Built on by (most cited first, 21 of 21)

- [Green Federated Learning: A new era of Green Aware AI](https://mlsanity.com/paper/2409.12626/) (2024)
- [Decentralized Directed Collaboration for Personalized Federated Learning](https://mlsanity.com/paper/2405.17876/) (2024)
- [Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Training](https://mlsanity.com/paper/2305.15157/) (2023)
- [Boosting Asynchronous Decentralized Learning with Model Fragmentation](https://mlsanity.com/paper/2410.12918/) (2024)
- [DRACO: Decentralized Asynchronous Federated Learning over Row-Stochastic Wireless Networks](https://mlsanity.com/paper/2406.13533/) (2024)
- [FedSPD: A Soft-clustering Approach for Personalized Decentralized Federated Learning](https://mlsanity.com/paper/2410.18862/) (2024)
- [Smart Sampling: Helping from Friendly Neighbors for Decentralized Federated Learning](https://mlsanity.com/paper/2407.04460/) (2024)
- [Learning to Collaborate: An Orchestrated-Decentralized Framework for Peer-to-Peer LLM Federation](https://mlsanity.com/paper/2601.17133/) (2026)
- [CRAD: Class-wise Reliability-Aware Distillation for Decentralized Heterogeneous Federated Learning](https://mlsanity.com/paper/2609.00446/) (2026)
- [MAPL: Model Agnostic Peer-to-peer Learning](https://mlsanity.com/paper/2403.19792/) (2024)
- [P4: Towards private, personalized, and Peer-to-Peer learning](https://mlsanity.com/paper/2405.17697/) (2024)
- [Trust-free Personalized Decentralized Learning](https://mlsanity.com/paper/2410.11378/) (2024)
- [A Novel Algorithm for Personalized Federated Learning: Knowledge Distillation with Weighted Combination Loss](https://mlsanity.com/paper/2504.04642/) (2025)
- [Harmonizing Generalization and Personalization in Ring-topology Decentralized Federated Learning](https://mlsanity.com/paper/2504.19103/) (2025)
- [Client Clustering Meets Knowledge Sharing: Enhancing Privacy and Robustness in Personalized Peer-to-Peer Learning](https://mlsanity.com/paper/2506.20413/) (2025)
- [Whom to Trust? Adaptive Collaboration in Personalized Federated Learning](https://mlsanity.com/paper/2507.00259/) (2025)
- [Cost-Free Personalization via Information-Geometric Projection in Bayesian Federated Learning](https://mlsanity.com/paper/2509.10132/) (2025)
- [Feature-Based Semantics-Aware Scheduling for Energy-Harvesting Federated Learning](https://mlsanity.com/paper/2512.01983/) (2025)
- [CALM: Class-wise Agreement and Label-gated Disagreement Modulation for Decentralized Federated Learning](https://mlsanity.com/paper/2609.05884/) (2026)
- [Robust Decentralized Federated Distillation via Multi-Modality Knowledge Collaboration](https://mlsanity.com/paper/2609.07230/) (2026)
- [Robust Decentralized Personalized Federated Learning via Prediction-Constrained Neighborhood Collaboration](https://mlsanity.com/paper/2609.07312/) (2026)

## Abstract (arXiv)

Personalization in federated learning (FL) functions as a coordinator for clients with high variance in data or behavior. Ensuring the convergence of these clients' models relies on how closely users collaborate with those with similar patterns or preferences. However, it is generally challenging to quantify similarity under limited knowledge about other users' models given to users in a decentralized network. To cope with this issue, we propose a personalized and fully decentralized FL algorithm, leveraging knowledge distillation techniques to empower each device so as to discern statistical distances between local models. Each client device can enhance its performance without sharing local data by estimating the similarity between two intermediate outputs from feeding local samples as in knowledge distillation. Our empirical studies demonstrate that the proposed algorithm improves the test accuracy of clients in fewer iterations under highly non-independent and identically distributed (non-i.i.d.) data distributions and is beneficial to agents with small datasets, even without the need for a central server.