mlsanityData to 14 Oct 2026

pFedES: Model Heterogeneous Personalized Federated Learning with Feature Extractor Sharing

arXiv 2311.06879 ↗First posted 12 Nov 2023cs.LG cs.DCcited by 9

Not on the map. Why?

Abstract (arXiv)

As a privacy-preserving collaborative machine learning paradigm, federated learning (FL) has attracted significant interest from academia and the industry alike. To allow each data owner (a.k.a., FL clients) to train a heterogeneous and personalized local model based on its local data distribution, system resources and requirements on model structure, the field of model-heterogeneous personalized federated learning (MHPFL) has emerged. Existing MHPFL approaches either rely on the availability of a public dataset with special characteristics to facilitate knowledge transfer, incur high computation and communication costs, or face potential model leakage risks. To address these limitations, we propose a model-heterogeneous personalized Federated learning approach based on feature Extractor Sharing (pFedES). It incorporates a small homogeneous feature extractor into each client's heterogeneous local model. Clients train them via the proposed iterative learning method to enable the exchange of global generalized knowledge and local personalized knowledge. The small local homogeneous extractors produced after local training are uploaded to the FL server and for aggregation to facilitate easy knowledge sharing among clients. We theoretically prove that pFedES can converge over wall-to-wall time. Extensive experiments on two real-world datasets against six state-of-the-art methods demonstrate that pFedES builds the most accurate model, while incurring low communication and computation costs. Compared with the best-performing baseline, it achieves 1.61% higher test accuracy, while reducing communication and computation costs by 99.6% and 82.9%, respectively.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 42 of its 54 references that are in the map. First posted on arXiv 12 Nov 2023.

Builds on

42 of 54 references.

References · 42

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2019Advances and Open Problems in Federated Learningcited by 3,046
  3. 2016An overview of gradient descent optimization algorithmscited by 1,033
  4. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 997
  5. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  6. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  7. 2021Towards Personalized Federated Learningcited by 464
  8. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  9. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
  10. 2021Federated Learning on Non-IID Data: A Surveycited by 346
  11. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  12. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 325
30 more
  1. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 243
  2. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 222
  3. 2021Personalized Federated Learning using Hypernetworkscited by 212
  4. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  5. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 187
  6. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 182
  7. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 161
  8. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  9. 2020FedVision: An Online Visual Object Detection Platform Powered by Federated Learningcited by 124
  10. 2022Federated Learning with Partial Model Personalizationcited by 114
  11. 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 109
  12. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  13. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 93
  14. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  15. 2020Practical One-Shot Federated Learning for Cross-Silo Settingcited by 77
  16. 2020Federated Mutual Learningcited by 71
  17. 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 69
  18. 2021Towards Fairness-Aware Federated Learningcited by 58
  19. 2023FedGH: Heterogeneous Federated Learning with Generalized Global Headercited by 56
  20. 2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49
  21. 2021Fed2: Feature-Aligned Federated Learningcited by 42
  22. 2019Wireless Federated Distillation for Distributed Edge Learning with Heterogeneous Datacited by 39
  23. 2021FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Modelscited by 31
  24. 2022Architecture Agnostic Federated Learning for Neural Networkscited by 30
  25. 2021FedMatch: Federated Learning Over Heterogeneous Question Answering Datacited by 23
  26. 2022FedClassAvg: Local Representation Learning for Personalized Federated Learning on Heterogeneous Neural Networkscited by 23
  27. 2020Cooperative Learning via Federated Distillation over Fading Channelscited by 16
  28. 2022Completely Heterogeneous Federated Learningcited by 9
  29. 2023Breaching FedMD: Image Recovery via Paired-Logits Inversion Attackcited by 7
  30. 2023Efficient Training of Large-scale Industrial Fault Diagnostic Models through Federated Opportunistic Block Dropoutcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3; 2025: 5.

9 papers.

  1. 2024Federated Model Heterogeneous Matryoshka Representation Learningcited by 21
  2. 2025Ten Challenging Problems in Federated Foundation Modelscited by 20
  3. 2024MH-pFLID: Model Heterogeneous personalized Federated Learning via Injection and Distillation for Medical Data Analysiscited by 14
  4. 2023FedSSA: Semantic Similarity-based Aggregation for Efficient Model-Heterogeneous Personalized Federated Learningcited by 10
  5. 2025dFLMoE: Decentralized Federated Learning via Mixture of Experts for Medical Data Analysiscited by 9
  6. 2025Federated Out-of-Distribution Generalization: A Causal Augmentation Viewcited by 4
  7. 2024pFedAFM: Adaptive Feature Mixture for Batch-Level Personalization in Heterogeneous Federated Learningcited by 3
  8. 2025Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learningcited by 2
  9. 2025Hypernetworks for Model-Heterogeneous Personalized Federated Learningcited by 1