# Communication-Efficient Federated Distillation

arXiv: https://arxiv.org/abs/2012.00632 · HTML: https://mlsanity.com/paper/2012.00632/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Dec 2020.

## Built on by (most cited first, 28 of 28)

- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning](https://mlsanity.com/paper/2102.02514/) (2021)
- [Federated Learning Meets Natural Language Processing: A Survey](https://mlsanity.com/paper/2107.12603/) (2021)
- [Generalizable Heterogeneous Federated Cross-Correlation and Instance Similarity Learning](https://mlsanity.com/paper/2309.16286/) (2023)
- [Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data](https://mlsanity.com/paper/2107.00051/) (2021)
- [FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Models](https://mlsanity.com/paper/2109.03775/) (2021)
- [Towards Model Agnostic Federated Learning Using Knowledge Distillation](https://mlsanity.com/paper/2110.15210/) (2021)
- [Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions](https://mlsanity.com/paper/2406.10861/) (2024)
- [A Comprehensive Survey of Federated Transfer Learning: Challenges, Methods and Applications](https://mlsanity.com/paper/2403.01387/) (2024)
- [Federated Distillation based Indoor Localization for IoT Networks](https://mlsanity.com/paper/2205.11440/) (2022)
- [EvoFed: Leveraging Evolutionary Strategies for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2311.07485/) (2023)
- [Decentralized Learning with Multi-Headed Distillation](https://mlsanity.com/paper/2211.15774/) (2022)
- [FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learning](https://mlsanity.com/paper/2311.16584/) (2023)
- [On the Byzantine-Resilience of Distillation-Based Federated Learning](https://mlsanity.com/paper/2402.12265/) (2024)
- [Robust Asymmetric Heterogeneous Federated Learning with Corrupted Clients](https://mlsanity.com/paper/2503.09206/) (2025)
- [Multiple Access in the Era of Distributed Computing and Edge Intelligence](https://mlsanity.com/paper/2403.07903/) (2024)
- [Federated Distillation for Medical Image Classification: Towards Trustworthy Computer-Aided Diagnosis](https://mlsanity.com/paper/2407.02261/) (2024)
- [Reward-Based 1-bit Compressed Federated Distillation on Blockchain](https://mlsanity.com/paper/2106.14265/) (2021)
- [Communication-Efficient Federated Learning with Adaptive Compression under Dynamic Bandwidth](https://mlsanity.com/paper/2405.03248/) (2024)
- [FedNet2Net: Saving Communication and Computations in Federated Learning with Model Growing](https://mlsanity.com/paper/2207.09568/) (2022)
- [Inclusive Data Representation in Federated Learning: A Novel Approach Integrating Textual and Visual Prompt](https://mlsanity.com/paper/2310.04455/) (2023)
- [Hypernetworks for Model-Heterogeneous Personalized Federated Learning](https://mlsanity.com/paper/2507.22330/) (2025)
- [Federated Learning for Energy Constrained IoT devices: A systematic mapping study](https://mlsanity.com/paper/2301.03720/) (2023)
- [Convergence Visualizer of Decentralized Federated Distillation with Reduced Communication Costs](https://mlsanity.com/paper/2312.11905/) (2023)
- [Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead](https://mlsanity.com/paper/2502.06349/) (2025)
- [Federated Distillation on Edge Devices: Efficient Client-Side Filtering for Non-IID Data](https://mlsanity.com/paper/2508.14769/) (2025)
- [SCENE OTA-FD: Self-Centering Noncoherent Estimator for Over-the-Air Federated Distillation](https://mlsanity.com/paper/2602.15326/) (2026)
- [GCA: Global Centroid Alignment in Federated Learning](https://mlsanity.com/paper/2608.22593/) (2026)

## Abstract (arXiv)

Communication constraints are one of the major challenges preventing the wide-spread adoption of Federated Learning systems. Recently, Federated Distillation (FD), a new algorithmic paradigm for Federated Learning with fundamentally different communication properties, emerged. FD methods leverage ensemble distillation techniques and exchange model outputs, presented as soft labels on an unlabeled public data set, between the central server and the participating clients. While for conventional Federated Learning algorithms, like Federated Averaging (FA), communication scales with the size of the jointly trained model, in FD communication scales with the distillation data set size, resulting in advantageous communication properties, especially when large models are trained. In this work, we investigate FD from the perspective of communication efficiency by analyzing the effects of active distillation-data curation, soft-label quantization and delta-coding techniques. Based on the insights gathered from this analysis, we present Compressed Federated Distillation (CFD), an efficient Federated Distillation method. Extensive experiments on Federated image classification and language modeling problems demonstrate that our method can reduce the amount of communication necessary to achieve fixed performance targets by more than two orders of magnitude, when compared to FD and by more than four orders of magnitude when compared with FA.