mlsanityData to 14 Oct 2026

HierarchyFL: Heterogeneous Federated Learning via Hierarchical Self-Distillation

arXiv 2212.02006 ↗First posted 5 Dec 2022cs.LGcited by 2

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

Federated learning (FL) has been recognized as a privacy-preserving distributed machine learning paradigm that enables knowledge sharing among various heterogeneous artificial intelligence (AIoT) devices through centralized global model aggregation. FL suffers from model inaccuracy and slow convergence due to the model heterogeneity of the AIoT devices involved. Although various existing methods try to solve the bottleneck of the model heterogeneity problem, most of them improve the accuracy of heterogeneous models in a coarse-grained manner, which makes it still a great challenge to deploy large-scale AIoT devices. To alleviate the negative impact of this problem and take full advantage of the diversity of each heterogeneous model, we propose an efficient framework named HierarchyFL, which uses a small amount of public data for efficient and scalable knowledge across a variety of differently structured models. By using self-distillation and our proposed ensemble library, each hierarchical model can intelligently learn from each other on cloud servers. Experimental results on various well-known datasets show that HierarchyFL can not only maximize the knowledge sharing among various heterogeneous models in large-scale AIoT systems, but also greatly improve the model performance of each involved heterogeneous AIoT device.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 9 of its 18 references that are in the map. First posted on arXiv 5 Dec 2022.

Builds on

9 of 18 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learning20 May

Other references · 8

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  5. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  6. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  7. 2022Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customizationcited by 35
  8. 2022Federated Contrastive Learning for Dermatological Disease Diagnosis via On-device Learningcited by 21

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 0; 2025: 1.

2 papers.

  1. 2023AdapterFL: Adaptive Heterogeneous Federated Learning for Resource-constrained Mobile Computing Systemscited by 4
  2. 2025QA-HFL: Quality-Aware Hierarchical Federated Learning for Resource-Constrained Mobile Devices with Heterogeneous Image Qualitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Diverse Device Heterogeneous Federated Learning via Task Arithmetic Knowledge Integration
  2. 2022Enhancing Heterogeneous Federated Learning with Knowledge Extraction and Multi-Model Fusion
  3. 2025Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning
  4. 2022No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices
  5. 2025Dual-Distilled Heterogeneous Federated Learning with Adaptive Margins for Trainable Global Prototypes
  6. 2022Closing the Gap between Client and Global Model Performance in Heterogeneous Federated Learning
  7. 2022Completely Heterogeneous Federated Learning
  8. 2021FedHe: Heterogeneous Models and Communication-Efficient Federated Learning