# FedHe: Heterogeneous Models and Communication-Efficient Federated Learning

arXiv: https://arxiv.org/abs/2110.09910 · HTML: https://mlsanity.com/paper/2110.09910/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 22 references that are in the map.
- First posted on arXiv 19 Oct 2021.

## Builds on (most cited first, 19 of 19)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [FedMD: Heterogenous Federated Learning via Model Distillation](https://mlsanity.com/paper/1910.03581/) (2019)
- [Robust and Communication-Efficient Federated Learning from Non-IID Data](https://mlsanity.com/paper/1903.02891/) (2019)
- [Asynchronous Federated Optimization](https://mlsanity.com/paper/1903.03934/) (2019)
- [Paraphrasing Complex Network: Network Compression via Factor Transfer](https://mlsanity.com/paper/1802.04977/) (2018)
- [A Survey on Distributed Machine Learning](https://mlsanity.com/paper/1912.09789/) (2019)
- [Cross-Layer Distillation with Semantic Calibration](https://mlsanity.com/paper/2012.03236/) (2020)
- [Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregation](https://mlsanity.com/paper/1903.07424/) (2019)
- [TiFL: A Tier-based Federated Learning System](https://mlsanity.com/paper/2001.09249/) (2020)
- [Federated Mutual Learning](https://mlsanity.com/paper/2006.16765/) (2020)
- [Federated Knowledge Distillation](https://mlsanity.com/paper/2011.02367/) (2020)
- [FedH2L: Federated Learning with Model and Statistical Heterogeneity](https://mlsanity.com/paper/2101.11296/) (2021)

## Built on by (most cited first, 13 of 13)

- [A Survey on Heterogeneous Federated Learning](https://mlsanity.com/paper/2210.04505/) (2022)
- [Federated Learning for Computationally-Constrained Heterogeneous Devices: A Survey](https://mlsanity.com/paper/2307.09182/) (2023)
- [Overcoming Data and Model Heterogeneities in Decentralized Federated Learning via Synthetic Anchors](https://mlsanity.com/paper/2405.11525/) (2024)
- [Federated Distillation: A Survey](https://mlsanity.com/paper/2404.08564/) (2024)
- [Internal Cross-layer Gradients for Extending Homogeneity to Heterogeneity in Federated Learning](https://mlsanity.com/paper/2308.11464/) (2023)
- [Federated Learning on Non-iid Data via Local and Global Distillation](https://mlsanity.com/paper/2306.14443/) (2023)
- [FedHPL: Efficient Heterogeneous Federated Learning with Prompt Tuning and Logit Distillation](https://mlsanity.com/paper/2405.17267/) (2024)
- [Embracing Federated Learning: Enabling Weak Client Participation via Partial Model Training](https://mlsanity.com/paper/2406.15125/) (2024)
- [FedWSIDD: Federated Whole Slide Image Classification via Dataset Distillation](https://mlsanity.com/paper/2506.15365/) (2025)
- [Exploiting Features and Logits in Heterogeneous Federated Learning](https://mlsanity.com/paper/2210.15527/) (2022)
- [FedIN: Federated Intermediate Layers Learning for Model Heterogeneity](https://mlsanity.com/paper/2304.00759/) (2023)
- [Enabling Weak Client Participation via On-device Knowledge Distillation in Heterogeneous Federated Learning](https://mlsanity.com/paper/2503.11151/) (2025)
- [Federated Distillation for Whole Slide Image via Gaussian-Mixture Feature Alignment and Curriculum Integration](https://mlsanity.com/paper/2605.00578/) (2026)

## Abstract (arXiv)

Federated learning (FL) is able to manage edge devices to cooperatively train a model while maintaining the training data local and private. One common assumption in FL is that all edge devices share the same machine learning model in training, for example, identical neural network architecture. However, the computation and store capability of different devices may not be the same. Moreover, reducing communication overheads can improve the training efficiency though it is still a challenging problem in FL. In this paper, we propose a novel FL method, called FedHe, inspired by knowledge distillation, which can train heterogeneous models and support asynchronous training processes with significantly reduced communication overheads. Our analysis and experimental results demonstrate that the performance of our proposed method is better than the state-of-the-art algorithms in terms of communication overheads and model accuracy.