mlsanityData to 14 Oct 2026

pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learning

arXiv 2402.01350 ↗First posted 2 Feb 2024cs.LG cs.DCcited by 24

On the map: Mixture-of-experts → Federated mixture-of-experts

Abstract (arXiv)

Federated learning (FL) has been widely adopted for collaborative training on decentralized data. However, it faces the challenges of data, system, and model heterogeneity. This has inspired the emergence of model-heterogeneous personalized federated learning (MHPFL). Nevertheless, the problem of ensuring data and model privacy, while achieving good model performance and keeping communication and computation costs low remains open in MHPFL. To address this problem, we propose a model-heterogeneous personalized Federated learning with Mixture of Experts (pFedMoE) method. It assigns a shared homogeneous small feature extractor and a local gating network for each client's local heterogeneous large model. Firstly, during local training, the local heterogeneous model's feature extractor acts as a local expert for personalized feature (representation) extraction, while the shared homogeneous small feature extractor serves as a global expert for generalized feature extraction. The local gating network produces personalized weights for extracted representations from both experts on each data sample. The three models form a local heterogeneous MoE. The weighted mixed representation fuses generalized and personalized features and is processed by the local heterogeneous large model's header with personalized prediction information. The MoE and prediction header are updated simultaneously. Secondly, the trained local homogeneous small feature extractors are sent to the server for cross-client information fusion via aggregation. Overall, pFedMoE enhances local model personalization at a fine-grained data level, while supporting model heterogeneity.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in Federated mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 37 of its 53 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

37 of 53 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings14 Jun
  2. 2021Federated Mixture of Experts14 Jul
  3. 2020PFL-MoE: Personalized Federated Learning Based on Mixture of Experts31 Dec
  4. 2020Specialized federated learning using a mixture of experts5 Oct

Other references · 33

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2019Advances and Open Problems in Federated Learningcited by 3,046
  3. 2016An overview of gradient descent optimization algorithmscited by 1,033
  4. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 997
  5. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  6. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  7. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  8. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
  9. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  10. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  11. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 325
  12. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 243
21 more
  1. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  2. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 187
  3. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  4. 2020Multi-site fMRI Analysis Using Privacy-preserving Federated Learning and Domain Adaptation: ABIDE Resultscited by 122
  5. 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 109
  6. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  7. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 93
  8. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  9. 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 69
  10. 2023FedGH: Heterogeneous Federated Learning with Generalized Global Headercited by 56
  11. 2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49
  12. 2021Fed2: Feature-Aligned Federated Learningcited by 42
  13. 2019Wireless Federated Distillation for Distributed Edge Learning with Heterogeneous Datacited by 39
  14. 2023Efficient Personalized Federated Learning via Sparse Model-Adaptationcited by 38
  15. 2023FedCP: Separating Feature Information for Personalized Federated Learning via Conditional Policycited by 36
  16. 2018Differentiable Learning-to-Normalize via Switchable Normalizationcited by 33
  17. 2021FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Modelscited by 31
  18. 2023Towards Personalized Federated Learning via Heterogeneous Model Reassemblycited by 31
  19. 2021FedMatch: Federated Learning Over Heterogeneous Question Answering Datacited by 23
  20. 2022FedClassAvg: Local Representation Learning for Personalized Federated Learning on Heterogeneous Neural Networkscited by 23
  21. 2022Enhancing Heterogeneous Federated Learning with Knowledge Extraction and Multi-Model Fusioncited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 6; 2026: 13.

24 papers.

Sort:

By area:

  1. 2024Advances and Open Challenges in Federated Foundation Modelscited by 36
  2. 2024Federated Model Heterogeneous Matryoshka Representation Learningcited by 21
  3. 2024Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architecturescited by 12
  4. 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Expertscited by 11
  5. 2024On-Device Collaborative Language Modeling via a Mixture of Generalists and Specialistscited by 6
  6. 2026Aggregation Alignment for Federated Learning with Mixture-of-Experts under Data Heterogeneitycited by 4
  7. 2025Efficient Training of Large-Scale AI Models Through Federated Mixture-of-Experts: A System-Level Approachcited by 3
  8. 2025FLEX-MoE: Federated Mixture-of-Experts with Load-balanced Expert Assignment for Edge Computingcited by 2
  9. 2024Towards Personalized Federated Learning via Comprehensive Knowledge Distillationcited by 1
  10. 2025FedSKD: Aggregation-free Model-heterogeneous Federated Learning via Multi-dimensional Similarity Knowledge Distillation for Medical Image Classificationcited by 1
  11. 2025Towards Efficient Federated Learning of Networked Mixture-of-Experts for Mobile Edge Computingcited by 1
  12. 2025Harmonizing Generalization and Personalization in Ring-topology Decentralized Federated Learningcited by 0
12 more
  1. 2025Communication-Efficient Module-Wise Federated Learning for Grasp Pose Detection in Cluttered Environmentscited by 0
  2. 2026DriftGuard: Mitigating Asynchronous Data Drift in Federated Learningcited by 0
  3. 2026MoASE++: Mixture of Activation Sparsity Experts with Domain-Adaptive On-policy Distillation for Continual Test Time Adaptationcited by 0
  4. 2026FedCoE: Bridging Generalization and Personalization via Federated Coordinated Dual-level MoEscited by 0
  5. 2026Large Language Models Should Learn Personalized Rather Than Aggregated Human Preferencescited by 0
  6. 2026Fisher-Routed Mixture of Experts for Federated Class-Incremental Learningcited by 0
  7. 2026Global Federated Learning Strategies for Building Efficient Personalized Modelscited by 0
  8. 2026MoE Router-Guided Clustering for Heterogeneous Federated Instruction Tuningcited by 0
  9. 2026FedGuide: Diffusion Prior Alignment and Value Baseline Guidance for Heterogeneous Federated Reinforcement Learningcited by 0
  10. 2026FedMAD: Modulation-Aware Directional Aggregation for Federated Learning in Remote Sensing Image Classificationcited by 0
  11. 2026Does Every User Need a Private LoRA? Decoupling Personalization from Per-User Adaptationcited by 0
  12. 2026Regime-Aware Router Learning for Multi-Domain Medical Image Generation with Diffusion Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Adaptive Expert Models for Personalization in Federated Learning
  2. 2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts
  3. 2025A Novel Algorithm for Personalized Federated Learning: Knowledge Distillation with Weighted Combination Loss
  4. 2024GAI-Enabled Explainable Personalized Federated Semi-Supervised Learning
  5. 2025FedMerge: Federated Personalization via Model Merging
  6. 2025PM-MOE: Mixture of Experts on Private Model Parameters for Personalized Federated Learning
  7. 2024ADEPT: Hierarchical Bayes Approach to Personalized Federated Unsupervised Learning
  8. 2024FedD2S: Personalized Data-Free Federated Knowledge Distillation