# Knowledge Distillation in Federated Edge Learning: A Survey

arXiv: https://arxiv.org/abs/2301.05849 · HTML: https://mlsanity.com/paper/2301.05849/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation.
- Builds on 35 of its 49 references that are in the map.
- First posted on arXiv 14 Jan 2023.

## Placed in

- Knowledge distillation / [Federated, robust and private distillation](https://mlsanity.com/distill/federated-robust-and-private-distillation/)

## Builds on (most cited first, 30 of 35)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [FedMD: Heterogenous Federated Learning via Model Distillation](https://mlsanity.com/paper/1910.03581/) (2019)
- [Federated Learning in Mobile Edge Networks: A Comprehensive Survey](https://mlsanity.com/paper/1909.11875/) (2019)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Correlation Congruence for Knowledge Distillation](https://mlsanity.com/paper/1904.01802/) (2019)
- [Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data](https://mlsanity.com/paper/1811.11479/) (2018)
- [Large scale distributed neural network training through online distillation](https://mlsanity.com/paper/1804.03235/) (2018)
- [Federated Class-Incremental Learning](https://mlsanity.com/paper/2203.11473/) (2022)
- [Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Data](https://mlsanity.com/paper/2008.06180/) (2020)
- [Parameterized Knowledge Transfer for Personalized Federated Learning](https://mlsanity.com/paper/2111.02862/) (2021)
- [Knowledge Adaptation for Efficient Semantic Segmentation](https://mlsanity.com/paper/1903.04688/) (2019)
- [Preservation of the Global Knowledge by Not-True Distillation in Federated Learning](https://mlsanity.com/paper/2106.03097/) (2021)
- [Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning](https://mlsanity.com/paper/2204.12703/) (2022)
- [Multi-Task Federated Learning for Personalised Deep Neural Networks in Edge Computing](https://mlsanity.com/paper/2007.09236/) (2020)
- [FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusion](https://mlsanity.com/paper/2110.11027/) (2021)
- [Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environments](https://mlsanity.com/paper/2208.11311/) (2022)
- [Unsupervised Multi-Target Domain Adaptation Through Knowledge Distillation](https://mlsanity.com/paper/2007.07077/) (2020)
- [Decentralized Federated Learning via Mutual Knowledge Transfer](https://mlsanity.com/paper/2012.13063/) (2020)
- [Wireless Federated Distillation for Distributed Edge Learning with Heterogeneous Data](https://mlsanity.com/paper/1907.02745/) (2019)
- [MetaFed: Federated Learning among Federations with Cyclic Knowledge Distillation for Personalized Healthcare](https://mlsanity.com/paper/2206.08516/) (2022)
- [FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Models](https://mlsanity.com/paper/2109.03775/) (2021)
- [Knowledge Distillation for Federated Learning: a Practical Guide](https://mlsanity.com/paper/2211.04742/) (2022)
- [Mix2FLD: Downlink Federated Learning After Uplink Federated Distillation With Two-Way Mixup](https://mlsanity.com/paper/2006.09801/) (2020)
- [Federated Edge Learning : Design Issues and Challenges](https://mlsanity.com/paper/2009.00081/) (2020)
- [Towards Federated Learning against Noisy Labels via Local Self-Regularization](https://mlsanity.com/paper/2208.12807/) (2022)
- [Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillation](https://mlsanity.com/paper/2204.07028/) (2022)
- [Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Space](https://mlsanity.com/paper/2104.00352/) (2021)

## Built on by (most cited first, 5 of 5)

- [Federated Skewed Label Learning with Logits Fusion](https://mlsanity.com/paper/2311.08202/) (2023)
- [Logits Poisoning Attack in Federated Distillation](https://mlsanity.com/paper/2401.03685/) (2024)
- [Data-Free Black-Box Federated Learning via Zeroth-Order Gradient Estimation](https://mlsanity.com/paper/2503.06028/) (2025)
- [Cyberattack Detection in Virtualized Microgrids Using LightGBM and Knowledge-Distilled Classifiers](https://mlsanity.com/paper/2601.03495/) (2026)
- [SCENE OTA-FD: Self-Centering Noncoherent Estimator for Over-the-Air Federated Distillation](https://mlsanity.com/paper/2602.15326/) (2026)

## Abstract (arXiv)

The increasing demand for intelligent services and privacy protection of mobile and Internet of Things (IoT) devices motivates the wide application of Federated Edge Learning (FEL), in which devices collaboratively train on-device Machine Learning (ML) models without sharing their private data. Limited by device hardware, diverse user behaviors and network infrastructure, the algorithm design of FEL faces challenges related to resources, personalization and network environments. Fortunately, Knowledge Distillation (KD) has been leveraged as an important technique to tackle the above challenges in FEL. In this paper, we investigate the works that KD applies to FEL, discuss the limitations and open problems of existing KD-based FEL approaches, and provide guidance for their real deployment.