mlsanityData to 14 Oct 2026

FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learning

arXiv 2311.16584 ↗First posted 28 Nov 2023cs.LG cs.DCcited by 5

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

Knowledge distillation (KD) can enable collaborative learning among distributed clients that have different model architectures and do not share their local data and model parameters with others. Each client updates its local model using the average model output/feature of all client models as the target, known as federated KD. However, existing federated KD methods often do not perform well when clients' local models are trained with heterogeneous local datasets. In this paper, we propose Federated knowledge distillation enabled by Adversarial Learning (FedAL) to address the data heterogeneity among clients. First, to alleviate the local model output divergence across clients caused by data heterogeneity, the server acts as a discriminator to guide clients' local model training to achieve consensus model outputs among clients through a min-max game between clients and the discriminator. Moreover, catastrophic forgetting may happen during the clients' local training and global knowledge transfer due to clients' heterogeneous local data. Towards this challenge, we design the less-forgetting regularization for both local training and global knowledge transfer to guarantee clients' ability to transfer/learn knowledge to/from others. Experimental results show that FedAL and its variants achieve higher accuracy than other federated KD baselines.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 28 of its 58 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

28 of 58 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning17 Mar
  2. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillation30 Aug
  3. 2021QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning29 Jul
  4. 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data30 Jun
  5. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learning20 May

Other references · 23

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2019Advances and Open Problems in Federated Learningcited by 3,046
  5. 2016Learning without Forgettingcited by 2,496
  6. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  7. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  8. 2013Do Deep Nets Really Need to be Deep?cited by 474
  9. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  10. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
  11. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  12. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 243
11 more
  1. 2019Overcoming Forgetting in Federated Learning on Non-IID Datacited by 120
  2. 2020Practical One-Shot Federated Learning for Cross-Silo Settingcited by 77
  3. 2017Multi-Agent Diverse Generative Adversarial Networkscited by 59
  4. 2023Generalizable Heterogeneous Federated Cross-Correlation and Instance Similarity Learningcited by 40
  5. 2019Wireless Federated Distillation for Distributed Edge Learning with Heterogeneous Datacited by 39
  6. 2021FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Modelscited by 31
  7. 2022Acceleration of Federated Learning with Alleviated Forgetting in Local Trainingcited by 29
  8. 2020Communication-Efficient Federated Distillationcited by 28
  9. 2023DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learningcited by 24
  10. 2021Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfercited by 23
  11. 2020Learning from a Lightweight Teacher for Efficient Knowledge Distillationcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 3.

5 papers.

  1. 2024Tri-Level Navigator: LLM-Empowered Tri-Level Learning for Time Series OOD Generalizationcited by 12
  2. 2023Provably Convergent Federated Trilevel Learningcited by 6
  3. 2025FedX: Explanation-Guided Pruning for Communication-Efficient Federated Learning in Remote Sensingcited by 1
  4. 2025Adaptive Federated Distillation for Multi-Domain Non-IID Textual Datacited by 1
  5. 2025HFedCKD: Toward Robust Heterogeneous Federated Learning via Data-free Knowledge Distillation and Two-way Contrastcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Knowledge Distillation for Federated Learning: a Practical Guide
  2. 2026Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search
  3. 2026FedEMA-Distill: Exponential Moving Average Guided Knowledge Distillation for Robust Federated Learning
  4. 2022Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion
  5. 2022FedDTG:Federated Data-Free Knowledge Distillation via Three-Player Generative Adversarial Networks
  6. 2022Closing the Gap between Client and Global Model Performance in Heterogeneous Federated Learning
  7. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillation
  8. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions