# Efficient Federated Learning for AIoT Applications Using Knowledge Distillation

arXiv: https://arxiv.org/abs/2111.14347 · HTML: https://mlsanity.com/paper/2111.14347/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation.
- Builds on 15 of its 38 references that are in the map.
- First posted on arXiv 29 Nov 2021.

## Placed in

- Knowledge distillation / [Federated, robust and private distillation](https://mlsanity.com/distill/federated-robust-and-private-distillation/)

## Builds on (most cited first, 15 of 15)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [Personalized Cross-Silo Federated Learning on Non-IID Data](https://mlsanity.com/paper/2007.03797/) (2020)
- [Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data](https://mlsanity.com/paper/1811.11479/) (2018)
- [Federated learning with hierarchical clustering of local updates to improve training on non-IID data](https://mlsanity.com/paper/2004.11791/) (2020)
- [SCAFFOLD: Stochastic Controlled Averaging for Federated Learning](https://mlsanity.com/paper/1910.06378/) (2019)
- [Large scale distributed neural network training through online distillation](https://mlsanity.com/paper/1804.03235/) (2018)
- [Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Data](https://mlsanity.com/paper/2008.06180/) (2020)
- [Multi-Center Federated Learning: Clients Clustering for Better Personalization](https://mlsanity.com/paper/2108.08647/) (2021)

## Built on by (most cited first, 5 of 5)

- [A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiency](https://mlsanity.com/paper/2307.10655/) (2023)
- [Federated Distillation: A Survey](https://mlsanity.com/paper/2404.08564/) (2024)
- [Adaptive Resolution Inference (ARI): Energy-Efficient Machine Learning for Internet of Things](https://mlsanity.com/paper/2408.14528/) (2024)
- [Cyberattack Detection in Virtualized Microgrids Using LightGBM and Knowledge-Distilled Classifiers](https://mlsanity.com/paper/2601.03495/) (2026)
- [Adaptive Dual-Weighting Framework for Federated Learning via Out-of-Distribution Detection](https://mlsanity.com/paper/2602.01039/) (2026)

## Abstract (arXiv)

As a promising distributed machine learning paradigm, Federated Learning (FL) trains a central model with decentralized data without compromising user privacy, which has made it widely used by Artificial Intelligence Internet of Things (AIoT) applications. However, the traditional FL suffers from model inaccuracy since it trains local models using hard labels of data and ignores useful information of incorrect predictions with small probabilities. Although various solutions try to tackle the bottleneck of the traditional FL, most of them introduce significant communication and memory overhead, making the deployment of large-scale AIoT devices a great challenge. To address the above problem, this paper presents a novel Distillation-based Federated Learning (DFL) architecture that enables efficient and accurate FL for AIoT applications. Inspired by Knowledge Distillation (KD) that can increase the model accuracy, our approach adds the soft targets used by KD to the FL model training, which occupies negligible network resources. The soft targets are generated by local sample predictions of each AIoT device after each round of local training and used for the next round of model training. During the local training of DFL, both soft targets and hard labels are used as approximation objectives of model predictions to improve model accuracy by supplementing the knowledge of soft targets. To further improve the performance of our DFL model, we design a dynamic adjustment strategy for tuning the ratio of two loss functions used in KD, which can maximize the use of both soft targets and hard labels. Comprehensive experimental results on well-known benchmarks show that our approach can significantly improve the model accuracy of FL with both Independent and Identically Distributed (IID) and non-IID data.