# pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learning

arXiv: https://arxiv.org/abs/2402.01350 · HTML: https://mlsanity.com/paper/2402.01350/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Federated mixture-of-experts, one of 12 directions in Mixture-of-experts.
- Builds on 37 of its 53 references that are in the map.
- First posted on arXiv 2 Feb 2024.

## Placed in

- Mixture-of-experts / [Federated mixture-of-experts](https://mlsanity.com/moe/federated-mixture-of-experts/)

## Builds on (most cited first, 30 of 37)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Exploiting Shared Representations for Personalized Federated Learning](https://mlsanity.com/paper/2102.07078/) (2021)
- [FedMD: Heterogenous Federated Learning via Model Distillation](https://mlsanity.com/paper/1910.03581/) (2019)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [FedProto: Federated Prototype Learning across Heterogeneous Clients](https://mlsanity.com/paper/2105.00243/) (2021)
- [HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients](https://mlsanity.com/paper/2010.01264/) (2020)
- [Think Locally, Act Globally: Federated Learning with Local and Global Representations](https://mlsanity.com/paper/2001.01523/) (2020)
- [Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data](https://mlsanity.com/paper/1811.11479/) (2018)
- [FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout](https://mlsanity.com/paper/2102.13451/) (2021)
- [FedKD: Communication Efficient Federated Learning via Knowledge Distillation](https://mlsanity.com/paper/2108.13323/) (2021)
- [Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Data](https://mlsanity.com/paper/2008.06180/) (2020)
- [Multi-site fMRI Analysis Using Privacy-preserving Federated Learning and Domain Adaptation: ABIDE Results](https://mlsanity.com/paper/2001.05647/) (2020)
- [Parameterized Knowledge Transfer for Personalized Federated Learning](https://mlsanity.com/paper/2111.02862/) (2021)
- [FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction](https://mlsanity.com/paper/2212.01548/) (2022)
- [Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer](https://mlsanity.com/paper/1912.11279/) (2019)
- [Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning](https://mlsanity.com/paper/2204.12703/) (2022)
- [FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning](https://mlsanity.com/paper/2102.02514/) (2021)
- [FedGH: Heterogeneous Federated Learning with Generalized Global Header](https://mlsanity.com/paper/2303.13137/) (2023)
- [FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusion](https://mlsanity.com/paper/2110.11027/) (2021)
- [Fed2: Feature-Aligned Federated Learning](https://mlsanity.com/paper/2111.14248/) (2021)
- [Wireless Federated Distillation for Distributed Edge Learning with Heterogeneous Data](https://mlsanity.com/paper/1907.02745/) (2019)
- [Efficient Personalized Federated Learning via Sparse Model-Adaptation](https://mlsanity.com/paper/2305.02776/) (2023)
- [FedCP: Separating Feature Information for Personalized Federated Learning via Conditional Policy](https://mlsanity.com/paper/2307.01217/) (2023)
- [Differentiable Learning-to-Normalize via Switchable Normalization](https://mlsanity.com/paper/1806.10779/) (2018)
- [FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Models](https://mlsanity.com/paper/2109.03775/) (2021)
- [Towards Personalized Federated Learning via Heterogeneous Model Reassembly](https://mlsanity.com/paper/2308.08643/) (2023)

## Built on by (most cited first, 24 of 24)

- [Advances and Open Challenges in Federated Foundation Models](https://mlsanity.com/paper/2404.15381/) (2024)
- [Federated Model Heterogeneous Matryoshka Representation Learning](https://mlsanity.com/paper/2406.00488/) (2024)
- [Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architectures](https://mlsanity.com/paper/2411.19128/) (2024)
- [HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Experts](https://mlsanity.com/paper/2601.00583/) (2026)
- [On-Device Collaborative Language Modeling via a Mixture of Generalists and Specialists](https://mlsanity.com/paper/2409.13931/) (2024)
- [Aggregation Alignment for Federated Learning with Mixture-of-Experts under Data Heterogeneity](https://mlsanity.com/paper/2603.21276/) (2026)
- [Efficient Training of Large-Scale AI Models Through Federated Mixture-of-Experts: A System-Level Approach](https://mlsanity.com/paper/2507.05685/) (2025)
- [FLEX-MoE: Federated Mixture-of-Experts with Load-balanced Expert Assignment for Edge Computing](https://mlsanity.com/paper/2512.23070/) (2025)
- [Towards Personalized Federated Learning via Comprehensive Knowledge Distillation](https://mlsanity.com/paper/2411.03569/) (2024)
- [FedSKD: Aggregation-free Model-heterogeneous Federated Learning via Multi-dimensional Similarity Knowledge Distillation for Medical Image Classification](https://mlsanity.com/paper/2503.18981/) (2025)
- [Towards Efficient Federated Learning of Networked Mixture-of-Experts for Mobile Edge Computing](https://mlsanity.com/paper/2511.01743/) (2025)
- [Harmonizing Generalization and Personalization in Ring-topology Decentralized Federated Learning](https://mlsanity.com/paper/2504.19103/) (2025)
- [Communication-Efficient Module-Wise Federated Learning for Grasp Pose Detection in Cluttered Environments](https://mlsanity.com/paper/2507.05861/) (2025)
- [DriftGuard: Mitigating Asynchronous Data Drift in Federated Learning](https://mlsanity.com/paper/2603.18872/) (2026)
- [MoASE++: Mixture of Activation Sparsity Experts with Domain-Adaptive On-policy Distillation for Continual Test Time Adaptation](https://mlsanity.com/paper/2605.17743/) (2026)
- [FedCoE: Bridging Generalization and Personalization via Federated Coordinated Dual-level MoEs](https://mlsanity.com/paper/2605.21264/) (2026)
- [Large Language Models Should Learn Personalized Rather Than Aggregated Human Preferences](https://mlsanity.com/paper/2606.07629/) (2026)
- [Fisher-Routed Mixture of Experts for Federated Class-Incremental Learning](https://mlsanity.com/paper/2606.28835/) (2026)
- [Global Federated Learning Strategies for Building Efficient Personalized Models](https://mlsanity.com/paper/2608.15107/) (2026)
- [MoE Router-Guided Clustering for Heterogeneous Federated Instruction Tuning](https://mlsanity.com/paper/2608.15311/) (2026)
- [FedGuide: Diffusion Prior Alignment and Value Baseline Guidance for Heterogeneous Federated Reinforcement Learning](https://mlsanity.com/paper/2609.18964/) (2026)
- [FedMAD: Modulation-Aware Directional Aggregation for Federated Learning in Remote Sensing Image Classification](https://mlsanity.com/paper/2610.00693/) (2026)
- [Does Every User Need a Private LoRA? Decoupling Personalization from Per-User Adaptation](https://mlsanity.com/paper/2610.02353/) (2026)
- [Regime-Aware Router Learning for Multi-Domain Medical Image Generation with Diffusion Experts](https://mlsanity.com/paper/2610.16082/) (2026)

## Abstract (arXiv)

Federated learning (FL) has been widely adopted for collaborative training on decentralized data. However, it faces the challenges of data, system, and model heterogeneity. This has inspired the emergence of model-heterogeneous personalized federated learning (MHPFL). Nevertheless, the problem of ensuring data and model privacy, while achieving good model performance and keeping communication and computation costs low remains open in MHPFL. To address this problem, we propose a model-heterogeneous personalized Federated learning with Mixture of Experts (pFedMoE) method. It assigns a shared homogeneous small feature extractor and a local gating network for each client's local heterogeneous large model. Firstly, during local training, the local heterogeneous model's feature extractor acts as a local expert for personalized feature (representation) extraction, while the shared homogeneous small feature extractor serves as a global expert for generalized feature extraction. The local gating network produces personalized weights for extracted representations from both experts on each data sample. The three models form a local heterogeneous MoE. The weighted mixed representation fuses generalized and personalized features and is processed by the local heterogeneous large model's header with personalized prediction information. The MoE and prediction header are updated simultaneously. Secondly, the trained local homogeneous small feature extractors are sent to the server for cross-client information fusion via aggregation. Overall, pFedMoE enhances local model personalization at a fine-grained data level, while supporting model heterogeneity.