mlsanityData to 14 Oct 2026

HFedCKD: Toward Robust Heterogeneous Federated Learning via Data-free Knowledge Distillation and Two-way Contrast

arXiv 2503.06511 ↗First posted 9 Mar 2025cs.LG cs.AIcited by 0

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

Most current federated learning frameworks are modeled as static processes, ignoring the dynamic characteristics of the learning system. Under the limited communication budget of the central server, the flexible model architecture of a large number of clients participating in knowledge transfer requires a lower participation rate, active clients have uneven contributions, and the client scale seriously hinders the performance of FL. We consider a more general and practical federation scenario and propose a system heterogeneous federation method based on data-free knowledge distillation and two-way contrast (HFedCKD). We apply the Inverse Probability Weighted Distillation (IPWD) strategy to the data-free knowledge transfer framework. The generator completes the data features of the nonparticipating clients. IPWD implements a dynamic evaluation of the prediction contribution of each client under different data distributions. Based on the antibiased weighting of its prediction loss, the weight distribution of each client is effectively adjusted to fairly integrate the knowledge of participating clients. At the same time, the local model is split into a feature extractor and a classifier. Through differential contrast learning, the feature extractor is aligned with the global model in the feature space, while the classifier maintains personalized decision-making capabilities. HFedCKD effectively alleviates the knowledge offset caused by a low participation rate under data-free knowledge distillation and improves the performance and stability of the model. We conduct extensive experiments on image and IoT datasets to comprehensively evaluate and verify the generalization and robustness of the proposed HFedCKD framework.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 26 of its 38 references that are in the map. First posted on arXiv 9 Mar 2025.

Builds on

26 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learning28 Nov

Other references · 25

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  3. 2020Adaptive Federated Optimizationcited by 1,046
  4. 2021Model-Contrastive Federated Learningcited by 766
  5. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  6. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  7. 2021Federated Learning on Non-IID Data: A Surveycited by 346
  8. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  9. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  10. 2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
  11. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  12. 2020Joint Device Scheduling and Resource Allocation for Latency Constrained Wireless Federated Learningcited by 90
13 more
  1. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  2. 2020FedCoin: A Peer-to-Peer Payment System for Federated Learningcited by 39
  3. 2021FedCG: Leverage Conditional GAN for Protecting Privacy and Maintaining Competitive Performance in Federated Learningcited by 39
  4. 2023DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learningcited by 24
  5. 2022Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillationcited by 22
  6. 2021Federated Learning for Internet of Things: A Federated Learning Framework for On-device Anomaly Data Detectioncited by 20
  7. 2024FedPFT: Federated Proxy Fine-Tuning of Foundation Modelscited by 16
  8. 2022Federated Continual Learning through distillation in pervasive computingcited by 10
  9. 2022FedGen: Generalizable Federated Learning for Sequential Datacited by 9
  10. 2024FedSC: Provable Federated Self-supervised Learning with Spectral Contrastive Objective over Non-i.i.d. Datacited by 5
  11. 2023Hierarchical Contrastive Learning for Pattern-Generalizable Image Corruption Detectioncited by 3
  12. 2023Towards Fast and Stable Federated Learning: Confronting Heterogeneity via Knowledge Anchorcited by 3
  13. 2024Watch Your Head: Assembling Projection Heads to Save the Reliability of Federated Modelscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learning
  2. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning
  3. 2022Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion
  4. 2022FedDKD: Federated Learning with Decentralized Knowledge Distillation
  5. 2024Training Heterogeneous Client Models using Knowledge Distillation in Serverless Federated Learning
  6. 2022FedX: Unsupervised Federated Learning with Cross Knowledge Distillation
  7. 2023Heterogeneous Federated Learning Using Knowledge Codistillation
  8. 2025Enabling Weak Client Participation via On-device Knowledge Distillation in Heterogeneous Federated Learning