mlsanityData to 14 Oct 2026

Federated, robust and private distillation

191 papers0 this weekMedium confidence in this grouping: mostly one topic, with blurry edgesMedium confidence in this grouping: mostly one topic, with blurry edges · About

Keywords: federated, knowledge, adversarial, federated learning, distillation

Key facts. Federated, robust and private distillation: 191 papers on arXiv since 2021 (data through 14 Oct 2026). 36 of its 191 mapped papers (19%) cite “Data-Free Knowledge Distillation for Heterogeneous Federated Learning” (2021, arXiv 2105.10056). 2 new papers 25 Sep–14 Oct.

What’s new

25 Sep – 14 Oct 2026

2 new since 25 Sep

  1. Federated Subspace Guided Vision-Language-Action Policy Distillation for Non-IID Multi-Robot Manipulation also fits: Multimodal distillation
  2. Unknown-Traffic Detection, Calibration and Shortcut Reliance in Distilled Encrypted-Traffic Classifiers over One Year also fits: General knowledge distillation methods

Foundations

Most cited by papers here

  1. Data-Free Knowledge Distillation for Heterogeneous Federated Learning2021cited by 36 here
  2. FedKD: Communication Efficient Federated Learning via Knowledge Distillation2021cited by 22 here
  3. Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning2022cited by 20 here
  4. Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better2021cited by 17 here
  5. Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data2021cited by 15 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 36 here
  2. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 20 here
  3. 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 8 here
21 more it builds on, not in this chain
  1. 2021Adaptive Multi-Teacher Multi-level Knowledge Distillationcited by 3 here
  2. 2021Undistillable: Making A Nasty Teacher That CANNOT teach studentscited by 5 here
  3. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 4 here
  4. 2021Towards Understanding Knowledge Distillationcited by 5 here
  5. 2021Reliable Adversarial Distillation with Unreliable Teacherscited by 14 here
  6. 2021Knowledge distillation: A good teacher is patient and consistentcited by 7 here
  7. 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Datacited by 15 here
  8. 2021QuPeD: Quantized Personalization via Distillation with Applications to Federated Learningcited by 4 here
  9. 2021Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Bettercited by 17 here
  10. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 22 here
  11. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillationcited by 6 here
  12. 2022FedDTG:Federated Data-Free Knowledge Distillation via Three-Player Generative Adversarial Networkscited by 7 here
  13. 2022On the benefits of knowledge distillation for adversarial robustnesscited by 4 here
  14. 2022Knowledge Distillation from A Stronger Teachercited by 4 here
  15. 2022MetaFed: Federated Learning among Federations with Cyclic Knowledge Distillation for Personalized Healthcarecited by 6 here
  16. 2022Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillationcited by 5 here
  17. 2023FedICT: Federated Multi-task Distillation for Multi-access Edge Computingcited by 5 here
  18. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 4 here
  19. 2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learningcited by 7 here
  20. 2024Differentially Private Knowledge Distillation via Synthetic Text Generationcited by 4 here
  21. 2025DOGe: Defensive Output Generation for LLM Protection Against Knowledge Distillationcited by 4 here

The 24 most cited of 145.

Browse

30 of 191, ranked 6–35

  1. 6Reliable Adversarial Distillation with Unreliable Teachers202114
  2. 7Knowledge Distillation for Federated Learning: a Practical Guide20228
  3. 8FedFed: Feature Distillation against Data Heterogeneity in Federated Learning20237
  4. 9FedDTG:Federated Data-Free Knowledge Distillation via Three-Player Generative Adversarial Networks20227
  5. 10MetaFed: Federated Learning among Federations with Cyclic Knowledge Distillation for Personalized Healthcare20226
  6. 11Towards Model Agnostic Federated Learning Using Knowledge Distillation20216
  7. 12FedICT: Federated Multi-task Distillation for Multi-access Edge Computing20235
  8. 13Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillation20225
  9. 14DOGe: Defensive Output Generation for LLM Protection Against Knowledge Distillation20254
  10. 15Differentially Private Knowledge Distillation via Synthetic Text Generation20244
All 30
  1. 16On the benefits of knowledge distillation for adversarial robustness20224
  2. 17QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning20214
  3. 18Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions20243
  4. 19Students Parrot Their Teachers: Membership Inference on Model Distillation20233
  5. 20Selective Knowledge Sharing for Privacy-Preserving Federated Distillation without A Good Teacher20233
  6. 21Revisiting Data-Free Knowledge Distillation with Poisoned Teachers20233
  7. 22Federated Unlearning with Knowledge Distillation20223
  8. 23How and When Adversarial Robustness Transfers in Knowledge Distillation?20213
  9. 24Can LLM Watermarks Robustly Prevent Unauthorized Knowledge Distillation?20252
  10. 25The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillation20232
  11. 26Heterogeneous Federated Knowledge Graph Embedding Learning and Unlearning20232
  12. 27FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learning20232
  13. 28FedDKD: Federated Learning with Decentralized Knowledge Distillation20222
  14. 29Protecting Language Models Against Unauthorized Distillation through Trace Rewriting20261
  15. 30Corrected with the Latest Version: Make Robust Asynchronous Federated Learning Possible20251
  16. 31Unified attacks to large language model watermarks: spoofing and scrubbing in unauthorized knowledge distillation20251
  17. 32How to Backdoor Image Knowledge Distillation20251
  18. 33Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning20251
  19. 34Communication-Aware Knowledge Distillation for Federated LLM Fine-Tuning over Wireless Networks20251
  20. 35Taught Well Learned Ill: Towards Distillation-conditional Backdoor Attack20251