mlsanityData to 14 Oct 2026

Dual-Distilled Heterogeneous Federated Learning with Adaptive Margins for Trainable Global Prototypes

arXiv 2508.19009 ↗First posted 26 Aug 2025cs.LG cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Heterogeneous Federated Learning (HFL) has gained significant attention for its capacity to handle both model and data heterogeneity across clients. Prototype-based HFL methods emerge as a promising solution to address statistical and model heterogeneity as well as privacy challenges, paving the way for new advancements in HFL research. This method focuses on sharing class-representative prototypes among heterogeneous clients. However, aggregating these prototypes via standard weighted averaging often yields sub-optimal global knowledge. Specifically, the averaging approach induces a shrinking of the aggregated prototypes' decision margins, thereby degrading model performance in scenarios with model heterogeneity and non-IID data distributions. The propose FedProtoKD in a Heterogeneous Federated Learning setting, utilizing an enhanced dual-knowledge distillation mechanism to enhance system performance by leveraging clients' logits and prototype feature representations. The proposed framework aims to resolve the prototype margin-shrinking problem using a contrastive learning-based trainable server prototype by leveraging a class-wise adaptive prototype margin. Furthermore, the framework assess the importance of public samples using the closeness of the sample's prototype to its class representative prototypes, which enhances learning performance. FedProtoKD improved test accuracy by an average of 1.13% and up to 34.13% across various settings, significantly outperforming existing state-of-the-art HFL methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 27 references that are in the map. First posted on arXiv 26 Aug 2025.

Builds on

20 of 27 references.

References · 20

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  6. 2020Personalized Federated Learning with Moreau Envelopescited by 618
  7. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 503
  8. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  9. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
  10. 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 374
  11. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  12. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 325
8 more
  1. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 325
  2. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 187
  3. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  4. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  5. 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 84
  6. 2021FedProc: Prototypical Contrastive Federated Learning on Non-IID datacited by 81
  7. 2024FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learningcited by 47
  8. 2023GPFL: Simultaneously Learning Global and Personalized Feature Information for Personalized Federated Learningcited by 18

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022HierarchyFL: Heterogeneous Federated Learning via Hierarchical Self-Distillation
  2. 2022Closing the Gap between Client and Global Model Performance in Heterogeneous Federated Learning
  3. 2025Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning
  4. 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data
  5. 2025FedPref: Federated Learning Across Heterogeneous Multi-objective Preferences
  6. 2023Adaptive Self-Distillation for Minimizing Client Drift in Heterogeneous Federated Learning
  7. 2024FedDW: Distilling Weights through Consistency Optimization in Heterogeneous Federated Learning
  8. 2024Federated Learning with a Single Shared Image