mlsanityData to 7 Oct 2026

Federated Knowledge Distillation

arXiv 2011.02367 ↗First posted 4 Nov 2020cs.LG cs.DC cs.ITcited by 51

Not on the map. Why?

Abstract (arXiv)

Distributed learning frameworks often rely on exchanging model parameters across workers, instead of revealing their raw data. A prime example is federated learning that exchanges the gradients or weights of each neural network model. Under limited communication resources, however, such a method becomes extremely costly particularly for modern deep neural networks having a huge number of model parameters. In this regard, federated distillation (FD) is a compelling distributed learning solution that only exchanges the model outputs whose dimensions are commonly much smaller than the model sizes (e.g., 10 labels in the MNIST dataset). The goal of this chapter is to provide a deep understanding of FD while demonstrating its communication efficiency and applicability to a variety of tasks. To this end, towards demystifying the operational principle of FD, the first part of this chapter provides a novel asymptotic analysis for two foundational algorithms of FD, namely knowledge distillation (KD) and co-distillation (CD), by exploiting the theory of neural tangent kernel (NTK). Next, the second part elaborates on a baseline implementation of FD for a classification task, and illustrates its performance in terms of accuracy and communication efficiency compared to FL. Lastly, to demonstrate the applicability of FD to various distributed learning tasks and environments, the third part presents two selected applications, namely FD over asymmetric uplink-and-downlink wireless channels and FD for reinforcement learning.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 9; 2023: 12; 2024: 6; 2025: 4; 2026: 3.

51 papers.

Sort:

By area:

  1. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  2. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  3. 2022Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challengescited by 137
  4. 2021Distributed Learning in Wireless Networks: Recent Progress and Future Challengescited by 125
  5. 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 67
  6. 2021FedLab: A Flexible Federated Learning Frameworkcited by 54
  7. 2023Heterogeneous Federated Knowledge Graph Embedding Learning and Unlearningcited by 39
  8. 2022Multi-Level Branched Regularization for Federated Learningcited by 34
  9. 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Datacited by 31
  10. 2021FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Modelscited by 31
  11. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillationcited by 31
  12. 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
38 more
  1. 2022Meta Knowledge Condensation for Federated Learningcited by 26
  2. 2023Selective Knowledge Sharing for Privacy-Preserving Federated Distillation without A Good Teachercited by 25
  3. 2023A Survey of Trustworthy Federated Learning with Perspectives on Security, Robustness, and Privacycited by 24
  4. 2021Semantics-Native Communication with Contextual Reasoningcited by 21
  5. 2023A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiencycited by 21
  6. 2021Federated Learning of Molecular Properties with Graph Neural Networks in a Heterogeneous Settingcited by 19
  7. 2021Neural Tangent Kernel Empowered Federated Learningcited by 15
  8. 2021An Efficient Federated Distillation Learning System for Multi-task Time Series Classificationcited by 14
  9. 2024Federated Distillation: A Surveycited by 14
  10. 2021FedHe: Heterogeneous Models and Communication-Efficient Federated Learningcited by 13
  11. 2022Autonomy and Intelligence in the Computing Continuum: Challenges, Enablers, and Future Directions for Orchestrationcited by 12
  12. 2023FedACK: Federated Adversarial Contrastive Knowledge Distillation for Cross-Lingual and Cross-Model Social Bot Detectioncited by 12
  13. 2022Enhancing Heterogeneous Federated Learning with Knowledge Extraction and Multi-Model Fusioncited by 10
  14. 2022FedDKD: Federated Learning with Decentralized Knowledge Distillationcited by 9
  15. 2023Revisiting Data-Free Knowledge Distillation with Poisoned Teacherscited by 9
  16. 2024A Comprehensive Survey of Federated Transfer Learning: Challenges, Methods and Applicationscited by 8
  17. 2024Real World Federated Learning with a Knowledge Distilled Transformer for Cardiac CT Imagingcited by 8
  18. 2023NeFL: Nested Model Scaling for Federated Learning with System Heterogeneous Clientscited by 7
  19. 2022Federated Distillation based Indoor Localization for IoT Networkscited by 6
  20. 2021Adaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient Distillationcited by 4
  21. 2025Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Datacited by 4
  22. 2023Rethinking Client Drift in Federated Learning: A Logit Perspectivecited by 3
  23. 2021Reward-Based 1-bit Compressed Federated Distillation on Blockchaincited by 2
  24. 2024Evidential Federated Learning for Skin Lesion Image Classificationcited by 2
  25. 2025Federated Learning-Distillation Alternation for Resource-Constrained IoTcited by 2
  26. 2021Communication Optimization in Large Scale Federated Learning using Autoencoder Compressed Weight Updatescited by 1
  27. 2023Temporal Knowledge Distillation for Time-Sensitive Financial Services Applicationscited by 1
  28. 2023LEFL: Low Entropy Client Sampling in Federated Learningcited by 1
  29. 2025Tackling Data Heterogeneity in Federated Learning through Knowledge Distillation with Inequitable Aggregationcited by 1
  30. 2025Who to Trust? Aggregating Client Predictions in Federated Distillationcited by 1
  31. 2026FedRio: Personalized Federated Social Bot Detection via Cooperative Reinforced Contrastive Adversarial Distillationcited by 1
  32. 2021Communication and Energy Efficient Slimmable Federated Learning via Superposition Coding and Successive Decodingcited by 0
  33. 2022Uncertainty Minimization for Personalized Federated Semi-Supervised Learningcited by 0
  34. 2023Federated Learning for Energy Constrained IoT devices: A systematic mapping studycited by 0
  35. 2023Heterogeneous Federated Learning Using Knowledge Codistillationcited by 0
  36. 2024FedBrain-Distill: Communication-Efficient Federated Brain Tumor Classification Using Ensemble Knowledge Distillation on Non-IID Datacited by 0
  37. 2024GeFL: Model-Agnostic Federated Learning with Generative Modelscited by 0
  38. 2026DeepFusion: Accelerating MoE Training via Federated Knowledge Distillation from Heterogeneous Edge Devicescited by 0

50 most cited of 51 All 51 (JSON)