mlsanityData to 14 Oct 2026

Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions

arXiv 2406.10861 ↗First posted 16 Jun 2024cs.LG cs.DCcited by 9

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

Federated Learning (FL) is a distributed and privacy-preserving machine learning paradigm that coordinates multiple clients to train a model while keeping the raw data localized. However, this traditional FL poses some challenges, including privacy risks, data heterogeneity, communication bottlenecks, and system heterogeneity issues. To tackle these challenges, knowledge distillation (KD) has been widely applied in FL since 2020. KD is a validated and efficacious model compression and enhancement algorithm. The core concept of KD involves facilitating knowledge transfer between models by exchanging logits at intermediate or output layers. These properties make KD an excellent solution for the long-lasting challenges in FL. Up to now, there have been few reviews that summarize and analyze the current trend and methods for how KD can be applied in FL efficiently. This article aims to provide a comprehensive survey of KD-based FL, focusing on addressing the above challenges. First, we provide an overview of KD-based FL, including its motivation, basics, taxonomy, and a comparison with traditional FL and where KD should execute. We also analyze the critical factors in KD-based FL in the appendix, including teachers, knowledge, data, and methods. We discuss how KD can address the challenges in FL, including privacy protection, data heterogeneity, communication efficiency, and personalization. Finally, we discuss the challenges facing KD-based FL algorithms and future research directions. We hope this survey can provide insights and guidance for researchers and practitioners in the FL area.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 125 of its 248 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

125 of 248 references.

Closest ancestors · 17

Same direction, newest first

  1. 2023Federated Learning on Non-iid Data via Local and Global Distillation26 Jun
  2. 2023Selective Knowledge Sharing for Privacy-Preserving Federated Distillation without A Good Teacher4 Apr
  3. 2023The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillation21 Jan
  4. 2022Knowledge Distillation for Federated Learning: a Practical Guide9 Nov
  5. 2022Domain Discrepancy Aware Distillation for Model Aggregation in Federated Learning4 Oct
  6. 2022Label driven Knowledge Distillation for Federated Learning with non-IID Data29 Sep
  7. 2022Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillation10 Sep
  8. 2022FedX: Unsupervised Federated Learning with Cross Knowledge Distillation19 Jul
  9. 2022MetaFed: Federated Learning among Federations with Cyclic Knowledge Distillation for Personalized Healthcare17 Jun
  10. 2022FedDKD: Federated Learning with Decentralized Knowledge Distillation2 May
  11. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning17 Mar
  12. 2022FedDTG:Federated Data-Free Knowledge Distillation via Three-Player Generative Adversarial Networks10 Jan
5 more
  1. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillation28 Oct
  2. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillation30 Aug
  3. 2021QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning29 Jul
  4. 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data30 Jun
  5. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learning20 May

Other references · 108

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  6. 2019Advances and Open Problems in Federated Learningcited by 3,046
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  8. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  9. 2020Knowledge Distillation: A Surveycited by 1,603
  10. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  11. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  12. 2018Federated Learning with Non-IID Datacited by 1,108
38 more
  1. 2019Deep Leakage from Gradientscited by 1,021
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  3. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 712
  4. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  5. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  6. 2017Deep Mutual Learningcited by 628
  7. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  8. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  9. 2019Federated Learning with Personalization Layerscited by 541
  10. 2018Object Detection with Deep Learning: A Reviewcited by 528
  11. 2020An Efficient Framework for Clustered Federated Learningcited by 511
  12. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  13. 2021Towards Personalized Federated Learningcited by 464
  14. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  15. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  16. 2019Federated Machine Learning: Concept and Applicationscited by 412
  17. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 408
  18. 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 395
  19. 2020iDLG: Improved Deep Leakage from Gradientscited by 382
  20. 2019On the Efficacy of Knowledge Distillationcited by 381
  21. 2020Adaptive Personalized Federated Learningcited by 358
  22. 2021Federated Learning on Non-IID Data: A Surveycited by 346
  23. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  24. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 325
  25. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 316
  26. 2018Taking A Closer Look at Domain Shift: Category-level Adversaries for Semantics Consistent Domain Adaptationcited by 295
  27. 2021Distilling Knowledge via Knowledge Reviewcited by 288
  28. 2020Personalized Cross-Silo Federated Learning on Non-IID Datacited by 283
  29. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 243
  30. 2018Dataset Distillationcited by 230
  31. 2018A Hybrid Approach to Privacy-Preserving Federated Learningcited by 210
  32. 2018Statistical Aspects of Wasserstein Distancescited by 203
  33. 2021Evaluating Gradient Inversion Attacks and Defenses in Federated Learningcited by 196
  34. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  35. 2019FedHealth: A Federated Transfer Learning Framework for Wearable Healthcarecited by 182
  36. 2020FetchSGD: Communication-Efficient Federated Learning with Sketchingcited by 179
  37. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  38. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 175

50 most cited of 108 All 108 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 1.

9 papers.

  1. 2025A Survey on Privacy Risks and Protection in Large Language Modelscited by 21
  2. 2024Towards Personalized Federated Learning via Comprehensive Knowledge Distillationcited by 1
  3. 2025Tackling Data Heterogeneity in Federated Learning through Knowledge Distillation with Inequitable Aggregationcited by 1
  4. 2025Query-based Knowledge Transfer for Heterogeneous Learning Environmentscited by 0
  5. 2025From Privacy to Trust in the Agentic Era: A Taxonomy of Challenges in Trustworthy Federated Learning Through the Lens of Trust Report 2.0cited by 0
  6. 2025Federated Distillation on Edge Devices: Efficient Client-Side Filtering for Non-IID Datacited by 0
  7. 2025Reviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learningcited by 0
  8. 2025Prediction-space knowledge markets for communication-efficient federated learning on multimedia taskscited by 0
  9. 2026Beyond Weights and Gradients: A Taxonomy of Federated Learning Messagescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Federated Distillation: A Survey
  2. 2024KnFu: Effective Knowledge Fusion
  3. 2024FedSiKD: Clients Similarity and Knowledge Distillation: Addressing Non-i.i.d. and Constraints in Federated Learning
  4. 2024Practical Insights into Knowledge Distillation for Pre-Trained Models
  5. 2024HYDRA-FL: Hybrid Knowledge Distillation for Robust and Accurate Federated Learning
  6. 2025FedBKD: Distilled Federated Learning to Embrace Gerneralization and Personalization on Non-IID Data
  7. 2023FedSDD: Scalable and Diversity-enhanced Distillation for Model Aggregation in Federated Learning
  8. 2024Training Heterogeneous Client Models using Knowledge Distillation in Serverless Federated Learning