# FedRAD: Federated Robust Adaptive Distillation

arXiv: https://arxiv.org/abs/2112.01405 · HTML: https://mlsanity.com/paper/2112.01405/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 2 Dec 2021.

## Builds on (most cited first, 14 of 14)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates](https://mlsanity.com/paper/1803.01498/) (2018)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning](https://mlsanity.com/paper/2009.01974/) (2020)
- [Zero-shot Knowledge Transfer via Adversarial Belief Matching](https://mlsanity.com/paper/1905.09768/) (2019)
- [Zero-Shot Knowledge Distillation in Deep Networks](https://mlsanity.com/paper/1905.08114/) (2019)
- [Byzantine-Robust Federated Machine Learning through Adaptive Model Averaging](https://mlsanity.com/paper/1909.05125/) (2019)
- [Learning to Drive from Simulation without Real World Labels](https://mlsanity.com/paper/1812.03823/) (2018)
- [Robust Aggregation for Adaptive Privacy Preserving Federated Learning in Healthcare](https://mlsanity.com/paper/2009.08294/) (2020)

## Built on by (most cited first, 11 of 11)

- [Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmark](https://mlsanity.com/paper/2311.06750/) (2023)
- [On the Vulnerability of Backdoor Defenses for Federated Learning](https://mlsanity.com/paper/2301.08170/) (2023)
- [Federated Distillation: A Survey](https://mlsanity.com/paper/2404.08564/) (2024)
- [A Survey on Vulnerability of Federated Learning: A Learning Algorithm Perspective](https://mlsanity.com/paper/2311.16065/) (2023)
- [Decentralized Learning with Multi-Headed Distillation](https://mlsanity.com/paper/2211.15774/) (2022)
- [On the Byzantine-Resilience of Distillation-Based Federated Learning](https://mlsanity.com/paper/2402.12265/) (2024)
- [Robust Knowledge Distillation in Federated Learning: Counteracting Backdoor Attacks](https://mlsanity.com/paper/2502.00587/) (2025)
- [Domain Discrepancy Aware Distillation for Model Aggregation in Federated Learning](https://mlsanity.com/paper/2210.02190/) (2022)
- [Venomancer: Towards Imperceptible and Target-on-Demand Backdoor Attacks in Federated Learning](https://mlsanity.com/paper/2407.03144/) (2024)
- [FedSym: Unleashing the Power of Entropy for Benchmarking the Algorithms for Federated Learning](https://mlsanity.com/paper/2310.07807/) (2023)
- [FedEMA-Distill: Exponential Moving Average Guided Knowledge Distillation for Robust Federated Learning](https://mlsanity.com/paper/2603.04422/) (2026)

## Abstract (arXiv)

The robustness of federated learning (FL) is vital for the distributed training of an accurate global model that is shared among large number of clients. The collaborative learning framework by typically aggregating model updates is vulnerable to model poisoning attacks from adversarial clients. Since the shared information between the global server and participants are only limited to model parameters, it is challenging to detect bad model updates. Moreover, real-world datasets are usually heterogeneous and not independent and identically distributed (Non-IID) among participants, which makes the design of such robust FL pipeline more difficult. In this work, we propose a novel robust aggregation method, Federated Robust Adaptive Distillation (FedRAD), to detect adversaries and robustly aggregate local models based on properties of the median statistic, and then performing an adapted version of ensemble Knowledge Distillation. We run extensive experiments to evaluate the proposed method against recently published works. The results show that FedRAD outperforms all other aggregators in the presence of adversaries, as well as in heterogeneous data distributions.