mlsanityData to 7 Oct 2026

HYDRA-FL: Hybrid Knowledge Distillation for Robust and Accurate Federated Learning

arXiv 2409.19912 ↗First posted 30 Sep 2024cs.LG cs.CRcited by 2

Not on the map. Why?

Abstract (arXiv)

Data heterogeneity among Federated Learning (FL) users poses a significant challenge, resulting in reduced global model performance. The community has designed various techniques to tackle this issue, among which Knowledge Distillation (KD)-based techniques are common. While these techniques effectively improve performance under high heterogeneity, they inadvertently cause higher accuracy degradation under model poisoning attacks (known as attack amplification). This paper presents a case study to reveal this critical vulnerability in KD-based FL systems. We show why KD causes this issue through empirical evidence and use it as motivation to design a hybrid distillation technique. We introduce a novel algorithm, Hybrid Knowledge Distillation for Robust and Accurate FL (HYDRA-FL), which reduces the impact of attacks in attack scenarios by offloading some of the KD loss to a shallow layer via an auxiliary classifier. We model HYDRA-FL as a generic framework and adapt it to two KD-based FL algorithms, FedNTD and MOON. Using these two as case studies, we demonstrate that our technique outperforms baselines in attack settings while maintaining comparable performance in benign settings.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 39 of its 53 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

39 of 53 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2019Advances and Open Problems in Federated Learningcited by 3,021
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  6. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  7. 2018Federated Learning with Non-IID Datacited by 1,099
  8. 2020Adaptive Federated Optimizationcited by 1,036
  9. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  10. 2021Model-Contrastive Federated Learningcited by 762
  11. 2018How To Backdoor Federated Learningcited by 745
  12. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 698
27 more
  1. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  2. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  3. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  4. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  5. 2019Local Model Poisoning Attacks to Byzantine-Robust Federated Learningcited by 479
  6. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  7. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  8. 2018Analyzing Federated Learning through an Adversarial Lenscited by 346
  9. 2018The Hidden Vulnerability of Distributed Learning in Byzantiumcited by 324
  10. 2019A Little Is Enough: Circumventing Defenses For Distributed Learningcited by 285
  11. 2017Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimizationcited by 188
  12. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  13. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  14. 2018RSA: Byzantine-Robust Stochastic Aggregation Methods for Distributed Learning from Heterogeneous Datasetscited by 148
  15. 2019Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulationcited by 136
  16. 2022FLDetector: Defending Federated Learning Against Model Poisoning Attacks via Detecting Malicious Clientscited by 119
  17. 2021Back to the Drawing Board: A Critical Evaluation of Poisoning Attacks on Production Federated Learningcited by 116
  18. 2020Distilled One-Shot Federated Learningcited by 108
  19. 2018Generalized Byzantine-tolerant SGDcited by 94
  20. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 91
  21. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  22. 2021Provably Secure Federated Learning against Malicious Clientscited by 60
  23. 2022FedRecover: Recovering from Poisoning Attacks in Federated Learning using Historical Informationcited by 53
  24. 2021Undistillable: Making A Nasty Teacher That CANNOT teach studentscited by 36
  25. 2018Universal Multi-Party Poisoning Attackscited by 17
  26. 2023Revisiting Data-Free Knowledge Distillation with Poisoned Teacherscited by 9
  27. 2022Robust Federated Learning against both Data Heterogeneity and Poisoning Attack via Aggregation Optimizationcited by 3

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025C^2Prompt: Class-aware Client Knowledge Interaction for Federated Continual Learningcited by 2
  2. 2025Stragglers Can Contribute More: Uncertainty-Aware Distillation for Asynchronous Federated Learningcited by 0