# Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data

arXiv: https://arxiv.org/abs/1811.11479 · HTML: https://mlsanity.com/paper/1811.11479/

## Key facts

- Cited by 243 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2018.

## Built on by (most cited first, 30 of 243)

- [Towards Personalized Federated Learning](https://mlsanity.com/paper/2103.00710/) (2021)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [FedProto: Federated Prototype Learning across Heterogeneous Clients](https://mlsanity.com/paper/2105.00243/) (2021)
- [No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data](https://mlsanity.com/paper/2106.05001/) (2021)
- [Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning](https://mlsanity.com/paper/2101.11203/) (2021)
- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [Federated Learning for Open Banking](https://mlsanity.com/paper/2108.10749/) (2021)
- [Distributed Learning in Wireless Networks: Recent Progress and Future Challenges](https://mlsanity.com/paper/2104.02151/) (2021)
- [Federated Graph Classification over Non-IID Graphs](https://mlsanity.com/paper/2106.13423/) (2021)
- [Multi-Center Federated Learning: Clients Clustering for Better Personalization](https://mlsanity.com/paper/2108.08647/) (2021)
- [Parameterized Knowledge Transfer for Personalized Federated Learning](https://mlsanity.com/paper/2111.02862/) (2021)
- [Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmark](https://mlsanity.com/paper/2311.06750/) (2023)
- [Deep Model Fusion: A Survey](https://mlsanity.com/paper/2309.15698/) (2023)
- [FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning](https://mlsanity.com/paper/2102.02514/) (2021)
- [Towards Fair Federated Learning with Zero-Shot Data Augmentation](https://mlsanity.com/paper/2104.13417/) (2021)
- [FedGH: Heterogeneous Federated Learning with Generalized Global Header](https://mlsanity.com/paper/2303.13137/) (2023)
- [FedFed: Feature Distillation against Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2310.05077/) (2023)
- [Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2206.02465/) (2022)
- [FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learning](https://mlsanity.com/paper/2401.03230/) (2024)
- [Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillation](https://mlsanity.com/paper/2209.04599/) (2022)
- [A Survey on Heterogeneous Federated Learning](https://mlsanity.com/paper/2210.04505/) (2022)
- [Federated Learning for Connected and Automated Vehicles: A Survey of Existing Approaches and Challenges](https://mlsanity.com/paper/2308.10407/) (2023)
- [pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuning](https://mlsanity.com/paper/2310.13283/) (2023)
- [Emerging Trends in Federated Learning: From Model Fusion to Federated X Learning](https://mlsanity.com/paper/2102.12920/) (2021)
- [Towards Efficient Replay in Federated Incremental Learning](https://mlsanity.com/paper/2403.05890/) (2024)
- [FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Models](https://mlsanity.com/paper/2109.03775/) (2021)
- [Knowledge Distillation for Federated Learning: a Practical Guide](https://mlsanity.com/paper/2211.04742/) (2022)
- [Integration of Large Language Models and Federated Learning](https://mlsanity.com/paper/2307.08925/) (2023)
- [FedDUAP: Federated Learning with Dynamic Update and Adaptive Pruning Using Shared Data on the Server](https://mlsanity.com/paper/2204.11536/) (2022)
- [Privacy and Fairness in Federated Learning: on the Perspective of Trade-off](https://mlsanity.com/paper/2306.14123/) (2023)

## Abstract (arXiv)

On-device machine learning (ML) enables the training process to exploit a massive amount of user-generated private data samples. To enjoy this benefit, inter-device communication overhead should be minimized. With this end, we propose federated distillation (FD), a distributed model training algorithm whose communication payload size is much smaller than a benchmark scheme, federated learning (FL), particularly when the model size is large. Moreover, user-generated data samples are likely to become non-IID across devices, which commonly degrades the performance compared to the case with an IID dataset. To cope with this, we propose federated augmentation (FAug), where each device collectively trains a generative model, and thereby augments its local data towards yielding an IID dataset. Empirical studies demonstrate that FD with FAug yields around 26x less communication overhead while achieving 95-98% test accuracy compared to FL.