# FedDKD: Federated Learning with Decentralized Knowledge Distillation

arXiv: https://arxiv.org/abs/2205.00706 · HTML: https://mlsanity.com/paper/2205.00706/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation.
- Builds on 28 of its 48 references that are in the map.
- First posted on arXiv 2 May 2022.

## Placed in

- Knowledge distillation / [Federated, robust and private distillation](https://mlsanity.com/distill/federated-robust-and-private-distillation/)

## Builds on (most cited first, 28 of 28)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [FedBN: Federated Learning on Non-IID Features via Local Batch Normalization](https://mlsanity.com/paper/2102.07623/) (2021)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [Federated Learning on Non-IID Data: A Survey](https://mlsanity.com/paper/2106.06843/) (2021)
- [Federated Optimization:Distributed Optimization Beyond the Datacenter](https://mlsanity.com/paper/1511.03575/) (2015)
- [Adversarially Robust Distillation](https://mlsanity.com/paper/1905.09747/) (2019)
- [Preservation of the Global Knowledge by Not-True Distillation in Federated Learning](https://mlsanity.com/paper/2106.03097/) (2021)
- [Adaptive Multi-Teacher Multi-level Knowledge Distillation](https://mlsanity.com/paper/2103.04062/) (2021)
- [Astraea: Self-balancing Federated Learning for Improving Classification Accuracy of Mobile Deep Learning Applications](https://mlsanity.com/paper/1907.01132/) (2019)
- [Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation](https://mlsanity.com/paper/1710.06169/) (2017)
- [Multi-Task Federated Learning for Personalised Deep Neural Networks in Edge Computing](https://mlsanity.com/paper/2007.09236/) (2020)
- [Federated Knowledge Distillation](https://mlsanity.com/paper/2011.02367/) (2020)
- [Dynamic Fusion based Federated Learning for COVID-19 Detection](https://mlsanity.com/paper/2009.10401/) (2020)
- [Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data](https://mlsanity.com/paper/2107.00051/) (2021)
- [Mix2FLD: Downlink Federated Learning After Uplink Federated Distillation With Two-Way Mixup](https://mlsanity.com/paper/2006.09801/) (2020)
- [Adversarial Teacher-Student Learning for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1804.00644/) (2018)
- [Unifying Heterogeneous Classifiers with Distillation](https://mlsanity.com/paper/1904.06062/) (2019)
- [FedMAX: Mitigating Activation Divergence for Accurate and Communication-Efficient Federated Learning](https://mlsanity.com/paper/2004.03657/) (2020)

## Built on by (most cited first, 9 of 9)

- [Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challenges](https://mlsanity.com/paper/2211.08413/) (2022)
- [Federated Distillation: A Survey](https://mlsanity.com/paper/2404.08564/) (2024)
- [Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions](https://mlsanity.com/paper/2406.10861/) (2024)
- [UnifyFL: Enabling Decentralized Cross-Silo Federated Learning](https://mlsanity.com/paper/2504.18916/) (2025)
- [FedBKD: Distilled Federated Learning to Embrace Gerneralization and Personalization on Non-IID Data](https://mlsanity.com/paper/2506.20245/) (2025)
- [Adaptive Federated Distillation for Multi-Domain Non-IID Textual Data](https://mlsanity.com/paper/2508.20557/) (2025)
- [Heterogeneous Federated Learning via Personalized Generative Networks](https://mlsanity.com/paper/2308.13265/) (2023)
- [GeFL: Model-Agnostic Federated Learning with Generative Models](https://mlsanity.com/paper/2412.18460/) (2024)
- [Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network](https://mlsanity.com/paper/2609.01952/) (2026)

## Abstract (arXiv)

The performance of federated learning in neural networks is generally influenced by the heterogeneity of the data distribution. For a well-performing global model, taking a weighted average of the local models, as done by most existing federated learning algorithms, may not guarantee consistency with local models in the space of neural network maps. In this paper, we propose a novel framework of federated learning equipped with the process of decentralized knowledge distillation (FedDKD) (i.e., without data on the server). The FedDKD introduces a module of decentralized knowledge distillation (DKD) to distill the knowledge of the local models to train the global model by approaching the neural network map average based on the metric of divergence defined in the loss function, other than only averaging parameters as done in literature. Numeric experiments on various heterogeneous datasets reveal that FedDKD outperforms the state-of-the-art methods with more efficient communication and training in a few DKD steps, especially on some extremely heterogeneous datasets.