mlsanityData to 14 Oct 2026

FedHe: Heterogeneous Models and Communication-Efficient Federated Learning

arXiv 2110.09910 ↗First posted 19 Oct 2021cs.LG cs.DCcited by 13

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) is able to manage edge devices to cooperatively train a model while maintaining the training data local and private. One common assumption in FL is that all edge devices share the same machine learning model in training, for example, identical neural network architecture. However, the computation and store capability of different devices may not be the same. Moreover, reducing communication overheads can improve the training efficiency though it is still a challenging problem in FL. In this paper, we propose a novel FL method, called FedHe, inspired by knowledge distillation, which can train heterogeneous models and support asynchronous training processes with significantly reduced communication overheads. Our analysis and experimental results demonstrate that the performance of our proposed method is better than the state-of-the-art algorithms in terms of communication overheads and model accuracy.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 19 of its 22 references that are in the map. First posted on arXiv 19 Oct 2021.

Builds on

19 of 22 references.

References · 19

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  6. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  7. 2019Towards Federated Learning at Scale: System Designcited by 832
  8. 2017Federated Multi-Task Learningcited by 672
  9. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  10. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  11. 2019Asynchronous Federated Optimizationcited by 267
  12. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 260
7 more
  1. 2019A Survey on Distributed Machine Learningcited by 188
  2. 2020Cross-Layer Distillation with Semantic Calibrationcited by 143
  3. 2019Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregationcited by 100
  4. 2020TiFL: A Tier-based Federated Learning Systemcited by 96
  5. 2020Federated Mutual Learningcited by 71
  6. 2020Federated Knowledge Distillationcited by 51
  7. 2021FedH2L: Federated Learning with Model and Statistical Heterogeneitycited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 4; 2024: 4; 2025: 2; 2026: 1.

13 papers.

Sort:

By area:

  1. 2022A Survey on Heterogeneous Federated Learningcited by 42
  2. 2023Federated Learning for Computationally-Constrained Heterogeneous Devices: A Surveycited by 40
  3. 2024Overcoming Data and Model Heterogeneities in Decentralized Federated Learning via Synthetic Anchorscited by 21
  4. 2024Federated Distillation: A Surveycited by 15
  5. 2023Internal Cross-layer Gradients for Extending Homogeneity to Heterogeneity in Federated Learningcited by 5
  6. 2023Federated Learning on Non-iid Data via Local and Global Distillationcited by 4
  7. 2024FedHPL: Efficient Heterogeneous Federated Learning with Prompt Tuning and Logit Distillationcited by 3
  8. 2024Embracing Federated Learning: Enabling Weak Client Participation via Partial Model Trainingcited by 3
  9. 2025FedWSIDD: Federated Whole Slide Image Classification via Dataset Distillationcited by 3
  10. 2022Exploiting Features and Logits in Heterogeneous Federated Learningcited by 2
  11. 2023FedIN: Federated Intermediate Layers Learning for Model Heterogeneitycited by 1
  12. 2025Enabling Weak Client Participation via On-device Knowledge Distillation in Heterogeneous Federated Learningcited by 1
1 more
  1. 2026Federated Distillation for Whole Slide Image via Gaussian-Mixture Feature Alignment and Curriculum Integrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Training Heterogeneous Client Models using Knowledge Distillation in Serverless Federated Learning
  2. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillation
  3. 2026Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search
  4. 2022Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion
  5. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learning
  6. 2022Communication-Efficient Federated Learning With Data and Client Heterogeneity
  7. 2022Closing the Gap between Client and Global Model Performance in Heterogeneous Federated Learning
  8. 2022Enhancing Heterogeneous Federated Learning with Knowledge Extraction and Multi-Model Fusion