mlsanityData to 14 Oct 2026

Personalized Federated Learning through Global Knowledge Distillation and Local Head Adaptation

arXiv 2609.17284 ↗First posted 15 Sep 2026cs.LG stat.ML stat.OTcited by 0

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

Statistical heterogeneity limits federated learning when a single global classifier cannot represent client-specific label distributions. In this work, we propose Personalized Federated Knowledge Distillation with Head Adaptation (pFedKDH), which aggregates only the shared backbone, keeps persistent client-specific heads, and uses a recalibrated global head as a teacher during local training. Across MNIST, Fashion-MNIST, CIFAR10, and CIFAR100 under class-wise Dirichlet partitions, pFedKDH obtains the best accuracy in most settings, with accuracy gaps up to 37.67% over the weakest baseline and consistently low standard deviation across repetitions. Component-wise diagnostics and convergence results support the role of persistent heads and distillation-guided local optimization under label-skewed data.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 17 of its 20 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

17 of 20 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learning8 Oct

Other references · 16

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  6. 2020Personalized Federated Learning with Moreau Envelopescited by 618
  7. 2019Federated Learning with Personalization Layerscited by 541
  8. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  9. 2021Towards Personalized Federated Learningcited by 464
  10. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  11. 2021Federated Learning on Non-IID Data: A Surveycited by 346
  12. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 182
4 more
  1. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 161
  2. 2022FedALA: Adaptive Local Aggregation for Personalized Federated Learningcited by 122
  3. 2024FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Datacited by 10
  4. 2023pFedSim: Similarity-Aware Model Aggregation Towards Personalized Federated Learningcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillation
  2. 2025A Novel Algorithm for Personalized Federated Learning: Knowledge Distillation with Weighted Combination Loss
  3. 2024Personalized Federated Learning via Backbone Self-Distillation
  4. 2024FedD2S: Personalized Data-Free Federated Knowledge Distillation
  5. 2025Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data
  6. 2023Personalized Decentralized Federated Learning with Knowledge Distillation
  7. 2022Closing the Gap between Client and Global Model Performance in Heterogeneous Federated Learning
  8. 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data