mlsanityData to 14 Oct 2026

Federated Distillation: A Survey

arXiv 2404.08564 ↗First posted 2 Apr 2024cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) seeks to train a model collaboratively without sharing private training data from individual clients. Despite its promise, FL encounters challenges such as high communication costs for large-scale models and the necessity for uniform model architectures across all clients and the server. These challenges severely restrict the practical applications of FL. To address these limitations, the integration of knowledge distillation (KD) into FL has been proposed, forming what is known as Federated Distillation (FD). FD enables more flexible knowledge transfer between clients and the server, surpassing the mere sharing of model parameters. By eliminating the need for identical model architectures across clients and the server, FD mitigates the communication costs associated with training large-scale models. This paper aims to offer a comprehensive overview of FD, highlighting its latest advancements. It delves into the fundamental principles underlying the design of FD frameworks, delineates FD approaches for tackling various challenges, and provides insights into the diverse applications of FD across different scenarios.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 154 of its 251 references that are in the map. First posted on arXiv 2 Apr 2024.

Builds on

154 of 251 references.

References · 154

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  5. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  6. 2019Advances and Open Problems in Federated Learningcited by 3,046
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  8. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  9. 2020Knowledge Distillation: A Surveycited by 1,603
  10. 2018Federated Learning with Non-IID Datacited by 1,108
  11. 2019Deep Leakage from Gradientscited by 1,021
  12. 2021Transformers in Vision: A Surveycited by 1,013
38 more
  1. 2019Relational Knowledge Distillationcited by 889
  2. 2019Towards Federated Learning at Scale: System Designcited by 832
  3. 2021Model-Contrastive Federated Learningcited by 766
  4. 2021Going deeper with Image Transformerscited by 764
  5. 2019Contrastive Representation Distillationcited by 748
  6. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  7. 2017Federated Multi-Task Learningcited by 672
  8. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
  9. 2018Exploiting Unintended Feature Leakage in Collaborative Learningcited by 520
  10. 2019Similarity-Preserving Knowledge Distillationcited by 516
  11. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 503
  12. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  13. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  14. 2022Decoupled Knowledge Distillationcited by 388
  15. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
  16. 2021Federated Learning on Non-IID Data: A Surveycited by 346
  17. 2021Distilling Knowledge via Knowledge Reviewcited by 288
  18. 2019Correlation Congruence for Knowledge Distillationcited by 285
  19. 2019Structured Knowledge Distillation for Dense Predictioncited by 279
  20. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 269
  21. 2021Federated Learning for Internet of Things: A Comprehensive Surveycited by 252
  22. 2018Learning without Memorizingcited by 250
  23. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 243
  24. 2019Client-Edge-Cloud Hierarchical Federated Learningcited by 202
  25. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  26. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 187
  27. 2021Federated Learning for Smart Healthcare: A Surveycited by 184
  28. 2019FedHealth: A Federated Transfer Learning Framework for Wearable Healthcarecited by 182
  29. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  30. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  31. 2020Channel-wise Knowledge Distillation for Dense Predictioncited by 163
  32. 2019One-Shot Federated Learningcited by 125
  33. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  34. 2021A Survey of Visual Transformerscited by 117
  35. 2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
  36. 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 109
  37. 2021DENSE: Data-Free One-Shot Federated Learningcited by 100
  38. 2019Federated Learning over Wireless Networks: Convergence Analysis and Resource Allocationcited by 94

50 most cited of 154 All 154 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 5.

15 papers.

Sort:

By area:

  1. 2024Towards Diverse Device Heterogeneous Federated Learning via Task Arithmetic Knowledge Integrationcited by 5
  2. 2024Federated Hybrid Training and Self-Adversarial Distillation: Towards Robust Edge Networkscited by 2
  3. 2025Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environmentscited by 2
  4. 2024Peak-Controlled Logits Poisoning Attack in Federated Distillationcited by 1
  5. 2025Who to Trust? Aggregating Client Predictions in Federated Distillationcited by 1
  6. 2026Self-Distillation is Optimal Among Spectral Shrinkage Estimators in Spiked Covariance Modelscited by 1
  7. 2024Framework for Co-distillation Driven Federated Learning to Address Class Imbalance in Healthcarecited by 0
  8. 2025Federated One-Shot Learning with Data Privacy and Objective-Hidingcited by 0
  9. 2025Towards a Small Language Model Lifecycle Frameworkcited by 0
  10. 2025Federated Distillation on Edge Devices: Efficient Client-Side Filtering for Non-IID Datacited by 0
  11. 2025Reviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learningcited by 0
  12. 2026Hybrid Federated Learning for Noise-Robust Trainingcited by 0
3 more
  1. 2026SCENE OTA-FD: Self-Centering Noncoherent Estimator for Over-the-Air Federated Distillationcited by 0
  2. 2026FedEMA-Distill: Exponential Moving Average Guided Knowledge Distillation for Robust Federated Learningcited by 0
  3. 2026Enhancing Robustness of Federated Learning via Server Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions
  2. 2024Training Heterogeneous Client Models using Knowledge Distillation in Serverless Federated Learning
  3. 2024KnFu: Effective Knowledge Fusion
  4. 2023FedSDD: Scalable and Diversity-enhanced Distillation for Model Aggregation in Federated Learning
  5. 2026Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search
  6. 2025SFedKD: Sequential Federated Learning with Discrepancy-Aware Multi-Teacher Knowledge Distillation
  7. 2024FedSiKD: Clients Similarity and Knowledge Distillation: Addressing Non-i.i.d. and Constraints in Federated Learning
  8. 2026FedMTFI: Feature Importance Based Optimized Multi Teacher Knowledge Distillation in Heterogeneous Federated Learning Environment