mlsanityData to 14 Oct 2026

FedeKD: Energy-Based Gating for Robust Federated Knowledge Distillation under Heterogeneous Settings

arXiv 2605.05553 ↗First posted 7 May 2026cs.LGcited by 0

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

Federated learning (FL) operates in heterogeneous environments, where variations in data distributions and asymmetric model design often result in negative transfer. While federated knowledge distillation (FKD) avoids direct model parameter sharing, existing methods typically rely on public datasets or assume that transferred knowledge is uniformly reliable, which limits their robustness in practice. This paper presents FedeKD, a reliability-aware FKD framework that makes sample-wise trust estimation an explicit component of knowledge transfer, without relying on additional public data. Each client maintains a high-capacity private model for local learning and a lightweight shared proxy model for cross-client knowledge exchange. During training, proxy models are aggregated on the server to form a global proxy, which is then used to guide updates of the private models. At the core of FedeKD is an energy-based gating mechanism that converts task-specific private-proxy disagreement into sample-wise trust weights for backward distillation. This mechanism enables sample-wise weighting of knowledge transfer, where the proxy model contributes more to reliable samples while down-weighting unreliable ones. Extensive experiments on six real-world datasets demonstrate that FedeKD significantly reduces negative transfer under heterogeneous settings while maintaining strong predictive performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 14 of its 22 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

14 of 22 references.

References · 14

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  4. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  5. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  6. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  7. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  8. 2023FedGH: Heterogeneous Federated Learning with Generalized Global Headercited by 56
  9. 2023Federated Conformal Predictors for Distributed Uncertainty Quantificationcited by 46
  10. 2023Towards Personalized Federated Learning via Heterogeneous Model Reassemblycited by 31
  11. 2024A comprehensive and easy-to-use multi-domain multi-task medical imaging meta-datasetcited by 12
  12. 2024Bridging Model Heterogeneity in Federated Learning via Uncertainty-based Asymmetrical Reciprocity Learningcited by 11
2 more
  1. 2024Asymmetrical Reciprocity-based Federated Learning for Resolving Disparities in Medical Diagnosiscited by 2
  2. 2026Conformalized Neural Networks for Federated Uncertainty Quantification under Dual Heterogeneitycited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026FedEMA-Distill: Exponential Moving Average Guided Knowledge Distillation for Robust Federated Learning
  2. 2022CDKT-FL: Cross-Device Knowledge Transfer using Proxy Dataset in Federated Learning
  3. 2024HYDRA-FL: Hybrid Knowledge Distillation for Robust and Accurate Federated Learning
  4. 2025Robust Knowledge Distillation in Federated Learning: Counteracting Backdoor Attacks
  5. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillation
  6. 2024FedSiKD: Clients Similarity and Knowledge Distillation: Addressing Non-i.i.d. and Constraints in Federated Learning
  7. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning
  8. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions