mlsanityData to 7 Oct 2026

Knowledge Distillation for Federated Learning: a Practical Guide

arXiv 2211.04742 ↗First posted 9 Nov 2022cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) enables the training of Deep Learning models without centrally collecting possibly sensitive raw data. The most used algorithms for FL are parameter-averaging based schemes (e.g., Federated Averaging) that, however, have well known limits, i.e., model homogeneity, high communication cost, poor performance in presence of heterogeneous data distributions. Federated adaptations of regular Knowledge Distillation (KD) can solve or mitigate the weaknesses of parameter-averaging FL algorithms while possibly introducing other trade-offs. In this article, we originally present a focused review of the state-of-the-art KD-based algorithms specifically tailored for FL, by providing both a novel classification of the existing approaches and a detailed technical description of their pros, cons, and tradeoffs.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 50 of its 79 references that are in the map. First posted on arXiv 9 Nov 2022.

Builds on

50 of 79 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  6. 2020Knowledge Distillation: A Surveycited by 1,578
  7. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  8. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  9. 2018Federated Learning with Non-IID Datacited by 1,099
  10. 2020Adaptive Federated Optimizationcited by 1,036
  11. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  12. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
38 more
  1. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  2. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  3. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
  4. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  5. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 415
  6. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  7. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 349
  8. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 347
  9. 2021A Field Guide to Federated Optimizationcited by 312
  10. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 242
  11. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  12. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  13. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 174
  14. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  15. 2018Large scale distributed neural network training through online distillationcited by 144
  16. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  17. 2019Overcoming Forgetting in Federated Learning on Non-IID Datacited by 120
  18. 2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
  19. 2019Towards Visually Explaining Variational Autoencoderscited by 113
  20. 2018Snapshot Distillation: Teacher-Student Optimization in One Generationcited by 99
  21. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 91
  22. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  23. 2017Being Robust (in High Dimensions) Can Be Practicalcited by 83
  24. 2019Split Learning for collaborative deep learning in healthcarecited by 72
  25. 2022Federated Unlearning with Knowledge Distillationcited by 70
  26. 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 67
  27. 2020Ternary Compression for Communication-Efficient Federated Learningcited by 57
  28. 2018Sparse Binary Compression: Towards Distributed Deep Learning with minimal Communicationcited by 51
  29. 2020Federated Knowledge Distillationcited by 51
  30. 2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49
  31. 2022Multi-Level Branched Regularization for Federated Learningcited by 34
  32. 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Datacited by 31
  33. 2021FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Modelscited by 31
  34. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillationcited by 31
  35. 2020Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillationcited by 25
  36. 2022Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillationcited by 22
  37. 2021Teacher's pet: understanding and mitigating biases in distillationcited by 20
  38. 2020GradAug: A New Regularization Method for Deep Neural Networkscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 13; 2025: 7; 2026: 5.

31 papers.

Sort:

By area:

  1. 2024Federated Unlearning: A Survey on Methods, Design Guidelines, and Evaluation Metricscited by 25
  2. 2023A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiencycited by 21
  3. 2024Federated Distillation: A Surveycited by 14
  4. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutionscited by 8
  5. 2024Real World Federated Learning with a Knowledge Distilled Transformer for Cardiac CT Imagingcited by 8
  6. 2023Continual Adaptation of Vision Transformers for Federated Learningcited by 6
  7. 2023Adapter-based Selective Knowledge Distillation for Federated Multi-domain Meeting Summarizationcited by 6
  8. 2025One-shot Federated Learning Methods: A Practical Guidecited by 6
  9. 2023Knowledge Distillation in Federated Edge Learning: A Surveycited by 5
  10. 2024Towards Diverse Device Heterogeneous Federated Learning via Task Arithmetic Knowledge Integrationcited by 5
  11. 2023Federated Learning for 6G: Paradigms, Taxonomy, Recent Advances and Insightscited by 4
  12. 2024Federated Class-Incremental Learning with New-Class Augmented Self-Distillationcited by 4
19 more
  1. 2024FedQUIT: On-Device Federated Unlearning via a Quasi-Competent Virtual Teachercited by 2
  2. 2025FedPromo: Federated Lightweight Proxy Models at the Edge Bring New Domains to Foundation Modelscited by 2
  3. 2024Logits Poisoning Attack in Federated Distillationcited by 1
  4. 2024Peak-Controlled Logits Poisoning Attack in Federated Distillationcited by 1
  5. 2025Federated Unlearning Made Practical: Seamless Integration via Negated Pseudo-Gradientscited by 1
  6. 2025Benchmarking Federated Learning for Throughput Prediction in 5G Live Streaming Applicationscited by 1
  7. 2026CRAD: Class-wise Reliability-Aware Distillation for Decentralized Heterogeneous Federated Learningcited by 1
  8. 2023Heterogeneous Federated Learning Using Knowledge Codistillationcited by 0
  9. 2024CoDream: Exchanging dreams instead of models for federated aggregation with heterogeneous modelscited by 0
  10. 2024Federated Learning for Collaborative Inference Systems: The Case of Early Exit Networkscited by 0
  11. 2024Federated Knowledge Transfer Fine-tuning Large Server Model with Resource-Constrained IoT Clientscited by 0
  12. 2024Applied Federated Model Personalisation in the Industrial Domain: A Comparative Studycited by 0
  13. 2025Asynchronous Federated Continual Segmentation with Evolving Clients and Label Spacescited by 0
  14. 2025FedL2T: Personalized Federated Learning with Two-Teacher Distillation for Seizure Predictioncited by 0
  15. 2025SAM-Fed: SAM-Guided Federated Semi-Supervised Learning for Medical Image Segmentationcited by 0
  16. 2026SCENE OTA-FD: Self-Centering Noncoherent Estimator for Over-the-Air Federated Distillationcited by 0
  17. 2026Powerful Teachers Matter: Text-Guided Multi-view Knowledge Distillation with Visual Prior Enhancementcited by 0
  18. 2026Function-Space ADMM for Decentralized Federated Learning: A Control Theoretic Perspectivecited by 0
  19. 2026COSMOS: Model-Agnostic Personalized Federated Learning with Clustered Server Models and Pseudo-Label-Only Communicationcited by 0