mlsanityData to 14 Oct 2026

Knowledge Distillation in Federated Edge Learning: A Survey

arXiv 2301.05849 ↗First posted 14 Jan 2023cs.LGcited by 5

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

The increasing demand for intelligent services and privacy protection of mobile and Internet of Things (IoT) devices motivates the wide application of Federated Edge Learning (FEL), in which devices collaboratively train on-device Machine Learning (ML) models without sharing their private data. Limited by device hardware, diverse user behaviors and network infrastructure, the algorithm design of FEL faces challenges related to resources, personalization and network environments. Fortunately, Knowledge Distillation (KD) has been leveraged as an important technique to tackle the above challenges in FEL. In this paper, we investigate the works that KD applies to FEL, discuss the limitations and open problems of existing KD-based FEL approaches, and provide guidance for their real deployment.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 35 of its 49 references that are in the map. First posted on arXiv 14 Jan 2023.

Builds on

35 of 49 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023FedICT: Federated Multi-task Distillation for Multi-access Edge Computing1 Jan
  2. 2022Knowledge Distillation for Federated Learning: a Practical Guide9 Nov
  3. 2022MetaFed: Federated Learning among Federations with Cyclic Knowledge Distillation for Personalized Healthcare17 Jun
  4. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learning20 May

Other references · 31

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2020Knowledge Distillation: A Surveycited by 1,603
  4. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  5. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 408
  6. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 316
  7. 2019Correlation Congruence for Knowledge Distillationcited by 285
  8. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 243
  9. 2018Large scale distributed neural network training through online distillationcited by 146
  10. 2022Federated Class-Incremental Learningcited by 127
  11. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  12. 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 109
19 more
  1. 2019Knowledge Adaptation for Efficient Semantic Segmentationcited by 105
  2. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  3. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  4. 2020Multi-Task Federated Learning for Personalised Deep Neural Networks in Edge Computingcited by 52
  5. 2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49
  6. 2022Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environmentscited by 49
  7. 2020Unsupervised Multi-Target Domain Adaptation Through Knowledge Distillationcited by 47
  8. 2020Decentralized Federated Learning via Mutual Knowledge Transfercited by 43
  9. 2019Wireless Federated Distillation for Distributed Edge Learning with Heterogeneous Datacited by 39
  10. 2021FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Modelscited by 31
  11. 2020Mix2FLD: Downlink Federated Learning After Uplink Federated Distillation With Two-Way Mixupcited by 29
  12. 2020Federated Edge Learning : Design Issues and Challengescited by 26
  13. 2022Towards Federated Learning against Noisy Labels via Local Self-Regularizationcited by 26
  14. 2022Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillationcited by 22
  15. 2021Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Spacecited by 17
  16. 2020Cooperative Learning via Federated Distillation over Fading Channelscited by 16
  17. 2022Enhancing Heterogeneous Federated Learning with Knowledge Extraction and Multi-Model Fusioncited by 10
  18. 2021Spirit Distillation: A Model Compression Method with Multi-domain Knowledge Transfercited by 7
  19. 2021Spirit Distillation: Precise Real-time Semantic Segmentation of Road Scenes with Insufficient Datacited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 1; 2026: 2.

5 papers.

  1. 2023Federated Skewed Label Learning with Logits Fusioncited by 5
  2. 2024Logits Poisoning Attack in Federated Distillationcited by 2
  3. 2025Data-Free Black-Box Federated Learning via Zeroth-Order Gradient Estimationcited by 1
  4. 2026Cyberattack Detection in Virtualized Microgrids Using LightGBM and Knowledge-Distilled Classifierscited by 0
  5. 2026SCENE OTA-FD: Self-Centering Noncoherent Estimator for Over-the-Air Federated Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Federated Distillation on Edge Devices: Efficient Client-Side Filtering for Non-IID Data
  2. 2025Enabling Weak Client Participation via On-device Knowledge Distillation in Heterogeneous Federated Learning
  3. 2021Efficient Federated Learning for AIoT Applications Using Knowledge Distillation
  4. 2024FedSiKD: Clients Similarity and Knowledge Distillation: Addressing Non-i.i.d. and Constraints in Federated Learning
  5. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions
  6. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillation
  7. 2026RIFLE: Robust Distillation-based FL for Deep Model Deployment on Resource-Constrained IoT Networks
  8. 2022Label driven Knowledge Distillation for Federated Learning with non-IID Data