mlsanityData to 14 Oct 2026

Enabling Weak Client Participation via On-device Knowledge Distillation in Heterogeneous Federated Learning

arXiv 2503.11151 ↗First posted 14 Mar 2025cs.LGcited by 1

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

Online Knowledge Distillation (KD) is recently highlighted to train large models in Federated Learning (FL) environments. Many existing studies adopt the logit ensemble method to perform KD on the server side. However, they often assume that unlabeled data collected at the edge is centralized on the server. Moreover, the logit ensemble method personalizes local models, which can degrade the quality of soft targets, especially when data is highly non-IID. To address these critical limitations,we propose a novel on-device KD-based heterogeneous FL method. Our approach leverages a small auxiliary model to learn from labeled local data. Subsequently, a subset of clients with strong system resources transfers knowledge to a large model through on-device KD using their unlabeled data. Our extensive experiments demonstrate that our on-device KD-based heterogeneous FL method effectively utilizes the system resources of all edge devices as well as the unlabeled data, resulting in higher accuracy compared to SOTA KD-based FL methods.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 29 of its 34 references that are in the map. First posted on arXiv 14 Mar 2025.

Builds on

29 of 34 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning17 Mar

Other references · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2018LEAF: A Benchmark for Federated Settingscited by 792
  6. 2021Model-Contrastive Federated Learningcited by 766
  7. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
  8. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  9. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  10. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  11. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 243
  12. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
16 more
  1. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  2. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  3. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  4. 2020Peer Collaborative Learning for Online Knowledge Distillationcited by 58
  5. 2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49
  6. 2021Layer-wise Adaptive Model Aggregation for Scalable Federated Learningcited by 36
  7. 2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
  8. 2024Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimizationcited by 28
  9. 2023TimelyFL: Heterogeneity-aware Asynchronous Federated Learning with Adaptive Partial Trainingcited by 27
  10. 2023PockEngine: Sparse and Efficient Fine-tuning in a Pocketcited by 23
  11. 2021FedHe: Heterogeneous Models and Communication-Efficient Federated Learningcited by 13
  12. 2022Federated Learning of Large Models at the Edge via Principal Sub-Model Trainingcited by 13
  13. 2021On-edge Multi-task Transfer Learning: Model and Practice with Data-driven Task Allocationcited by 9
  14. 2022Partial Model Averaging in Federated Learning: Performance Guarantees and Benefitscited by 8
  15. 2022SemiPFL: Personalized Semi-Supervised Federated Learning Framework for Edge Intelligencecited by 7
  16. 2024Embracing Federated Learning: Enabling Weak Client Participation via Partial Model Trainingcited by 3

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Overcoming Challenges of Partial Client Participation in Federated Learning : A Comprehensive Reviewcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Closing the Gap between Client and Global Model Performance in Heterogeneous Federated Learning
  2. 2025Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning
  3. 2022Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion
  4. 2024Training Heterogeneous Client Models using Knowledge Distillation in Serverless Federated Learning
  5. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learning
  6. 2023The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillation
  7. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions
  8. 2023FedSDD: Scalable and Diversity-enhanced Distillation for Model Aggregation in Federated Learning