# FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learning

arXiv: https://arxiv.org/abs/2311.16584 · HTML: https://mlsanity.com/paper/2311.16584/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation.
- Builds on 28 of its 58 references that are in the map.
- First posted on arXiv 28 Nov 2023.

## Placed in

- Knowledge distillation / [Federated, robust and private distillation](https://mlsanity.com/distill/federated-robust-and-private-distillation/)

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [FedMD: Heterogenous Federated Learning via Model Distillation](https://mlsanity.com/paper/1910.03581/) (2019)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [FedProto: Federated Prototype Learning across Heterogeneous Clients](https://mlsanity.com/paper/2105.00243/) (2021)
- [Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data](https://mlsanity.com/paper/1811.11479/) (2018)
- [FedKD: Communication Efficient Federated Learning via Knowledge Distillation](https://mlsanity.com/paper/2108.13323/) (2021)
- [Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning](https://mlsanity.com/paper/2203.09249/) (2022)
- [Overcoming Forgetting in Federated Learning on Non-IID Data](https://mlsanity.com/paper/1910.07796/) (2019)
- [Practical One-Shot Federated Learning for Cross-Silo Setting](https://mlsanity.com/paper/2010.01017/) (2020)
- [Multi-Agent Diverse Generative Adversarial Networks](https://mlsanity.com/paper/1704.02906/) (2017)
- [Generalizable Heterogeneous Federated Cross-Correlation and Instance Similarity Learning](https://mlsanity.com/paper/2309.16286/) (2023)
- [Wireless Federated Distillation for Distributed Edge Learning with Heterogeneous Data](https://mlsanity.com/paper/1907.02745/) (2019)
- [QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning](https://mlsanity.com/paper/2107.13892/) (2021)
- [Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data](https://mlsanity.com/paper/2107.00051/) (2021)
- [FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Models](https://mlsanity.com/paper/2109.03775/) (2021)
- [Acceleration of Federated Learning with Alleviated Forgetting in Local Training](https://mlsanity.com/paper/2203.02645/) (2022)
- [Communication-Efficient Federated Distillation](https://mlsanity.com/paper/2012.00632/) (2020)
- [DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2309.13546/) (2023)
- [Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer](https://mlsanity.com/paper/2109.08119/) (2021)
- [Learning from a Lightweight Teacher for Efficient Knowledge Distillation](https://mlsanity.com/paper/2005.09163/) (2020)

## Built on by (most cited first, 5 of 5)

- [Tri-Level Navigator: LLM-Empowered Tri-Level Learning for Time Series OOD Generalization](https://mlsanity.com/paper/2410.07018/) (2024)
- [Provably Convergent Federated Trilevel Learning](https://mlsanity.com/paper/2312.11835/) (2023)
- [FedX: Explanation-Guided Pruning for Communication-Efficient Federated Learning in Remote Sensing](https://mlsanity.com/paper/2508.06256/) (2025)
- [Adaptive Federated Distillation for Multi-Domain Non-IID Textual Data](https://mlsanity.com/paper/2508.20557/) (2025)
- [HFedCKD: Toward Robust Heterogeneous Federated Learning via Data-free Knowledge Distillation and Two-way Contrast](https://mlsanity.com/paper/2503.06511/) (2025)

## Abstract (arXiv)

Knowledge distillation (KD) can enable collaborative learning among distributed clients that have different model architectures and do not share their local data and model parameters with others. Each client updates its local model using the average model output/feature of all client models as the target, known as federated KD. However, existing federated KD methods often do not perform well when clients' local models are trained with heterogeneous local datasets. In this paper, we propose Federated knowledge distillation enabled by Adversarial Learning (FedAL) to address the data heterogeneity among clients. First, to alleviate the local model output divergence across clients caused by data heterogeneity, the server acts as a discriminator to guide clients' local model training to achieve consensus model outputs among clients through a min-max game between clients and the discriminator. Moreover, catastrophic forgetting may happen during the clients' local training and global knowledge transfer due to clients' heterogeneous local data. Towards this challenge, we design the less-forgetting regularization for both local training and global knowledge transfer to guarantee clients' ability to transfer/learn knowledge to/from others. Experimental results show that FedAL and its variants achieve higher accuracy than other federated KD baselines.