mlsanityData to 7 Oct 2026

On the Byzantine-Resilience of Distillation-Based Federated Learning

arXiv 2402.12265 ↗First posted 19 Feb 2024cs.LG cs.AI cs.DCcited by 5

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) algorithms using Knowledge Distillation (KD) have received increasing attention due to their favorable properties with respect to privacy, non-i.i.d. data and communication cost. These methods depart from transmitting model parameters and instead communicate information about a learning task by sharing predictions on a public dataset. In this work, we study the performance of such approaches in the byzantine setting, where a subset of the clients act in an adversarial manner aiming to disrupt the learning process. We show that KD-based FL algorithms are remarkably resilient and analyze how byzantine clients can influence the learning process. Based on these insights, we introduce two new byzantine attacks and demonstrate their ability to break existing byzantine-resilient methods. Additionally, we propose a novel defence method which enhances the byzantine resilience of KD-based FL algorithms. Finally, we provide a general framework to obfuscate attacks, making them significantly harder to detect, thereby improving their effectiveness. Our findings serve as an important building block in the analysis of byzantine FL, contributing through the development of new attacks and new defence mechanisms, further advancing the robustness of KD-based FL algorithms.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 42 of its 59 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

42 of 59 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2016Wide Residual Networkscited by 3,057
  6. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 698
  7. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  8. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  9. 2020The Future of Digital Health with Federated Learningcited by 625
  10. 2018Exploiting Unintended Feature Leakage in Collaborative Learningcited by 509
  11. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  12. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
30 more
  1. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  2. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  3. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 347
  4. 2018The Hidden Vulnerability of Distributed Learning in Byzantiumcited by 324
  5. 2019A Little Is Enough: Circumventing Defenses For Distributed Learningcited by 285
  6. 2021Ensemble deep learning: A reviewcited by 283
  7. 2018Applied Federated Learning: Improving Google Keyboard Query Suggestionscited by 202
  8. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  9. 2018Multi-Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentationcited by 148
  10. 2020Learning from History for Byzantine Robust Optimizationcited by 137
  11. 2019Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulationcited by 136
  12. 2020Byzantine-Robust Learning on Heterogeneous Datasets via Bucketingcited by 125
  13. 2019Federated Learning for Emoji Prediction in a Mobile Keyboardcited by 115
  14. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 91
  15. 2017Being Robust (in High Dimensions) Can Be Practicalcited by 83
  16. 2018Byzantine Stochastic Gradient Descentcited by 81
  17. 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 67
  18. 2023Fixing by Mixing: A Recipe for Optimal Byzantine ML under Heterogeneitycited by 64
  19. 2020Byzantine-Resilient Non-Convex Stochastic Gradient Descentcited by 56
  20. 2020Collaborative Learning in the Jungle (Decentralized, Byzantine, Heterogeneous, Asynchronous and Nonconvex Learning)cited by 54
  21. 2019Federated Learning Of Out-Of-Vocabulary Wordscited by 51
  22. 2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49
  23. 2022Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillationcited by 45
  24. 2022Byzantine-Robust Federated Learning with Optimal Statistical Rates and Privacy Guaranteescited by 30
  25. 2020Communication-Efficient Federated Distillationcited by 28
  26. 2023Sparse Model Soups: A Recipe for Improved Pruning via Model Averagingcited by 20
  27. 2023PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMscited by 14
  28. 2021How I Learned to Stop Worrying and Love Retrainingcited by 12
  29. 2021FedRAD: Federated Robust Adaptive Distillationcited by 11
  30. 2023Collaborative Learning via Prediction Consensuscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2025Robust Federated Inferencecited by 2
  2. 2025A Nesterov-Accelerated Byzantine-Robust Federated Learningcited by 1
  3. 2024Consensus learning: A novel decentralised ensemble learning paradigmcited by 0
  4. 2026FedEMA-Distill: Exponential Moving Average Guided Knowledge Distillation for Robust Federated Learningcited by 0
  5. 2026Robust Decentralized Federated Distillation via Multi-Modality Knowledge Collaborationcited by 0