mlsanityData to 7 Oct 2026

FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts

arXiv 2408.11304 ↗First posted 21 Aug 2024cs.LGcited by 27

On the map: Mixture-of-experts → Federated mixture-of-experts

Abstract (arXiv)

As Large Language Models (LLMs) push the boundaries of AI capabilities, their demand for data is growing. Much of this data is private and distributed across edge devices, making Federated Learning (FL) a de-facto alternative for fine-tuning (i.e., FedLLM). However, it faces significant challenges due to the inherent heterogeneity among clients, including varying data distributions and diverse task types. Towards a versatile FedLLM, we replace traditional dense model with a sparsely-activated Mixture-of-Experts (MoE) architecture, whose parallel feed-forward networks enable greater flexibility. To make it more practical in resource-constrained environments, we present FedMoE, the efficient personalized FL framework to address data heterogeneity, constructing an optimal sub-MoE for each client and bringing the knowledge back to global MoE. FedMoE is composed of two fine-tuning stages. In the first stage, FedMoE simplifies the problem by conducting a heuristic search based on observed activation patterns, which identifies a suboptimal submodel for each client. In the second stage, these submodels are distributed to clients for further training and returned for server aggregating through a novel modular aggregation strategy. Meanwhile, FedMoE progressively adjusts the submodels to optimal through global expert recommendation. Experimental results demonstrate the superiority of our method over previous personalized FL methods.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Federated mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 28 of its 55 references that are in the map. First posted on arXiv 21 Aug 2024.

Builds on

28 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Specialized federated learning using a mixture of experts5 Oct

Other references · 27

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2024Mixtral of Expertscited by 1,785
  9. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  10. 2019Towards Federated Learning at Scale: System Designcited by 826
  11. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  12. 2019Federated Learning with Personalization Layerscited by 533
15 more
  1. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  2. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  3. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 187
  4. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
  5. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  6. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  7. 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 84
  8. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  9. 2016Decentralized Collaborative Learning of Personalized Models over Networkscited by 73
  10. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 65
  11. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
  12. 2021QuPeD: Quantized Personalization via Distillation with Applications to Federated Learningcited by 32
  13. 2024M3oE: Multi-Domain Multi-Task Mixture-of Experts Recommendation Frameworkcited by 32
  14. 2023Towards Personalized Federated Learning via Heterogeneous Model Reassemblycited by 31
  15. 2022A Benchmark for Federated Hetero-Task Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 13.

27 papers.

Sort:

By area:

  1. 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Expertscited by 11
  2. 2024Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architecturescited by 10
  3. 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
  4. 2025FFT-MoE: Efficient Federated Fine-Tuning for Foundation Models via Large-scale Sparse MoE under Heterogeneous Edgecited by 7
  5. 2025DynaMo: Runtime Switchable Quantization for MoE with Cross-Dataset Adaptationcited by 4
  6. 2026Aggregation Alignment for Federated Learning with Mixture-of-Experts under Data Heterogeneitycited by 4
  7. 2025Efficient Training of Large-Scale AI Models Through Federated Mixture-of-Experts: A System-Level Approachcited by 3
  8. 2025FedADP: Unified Model Aggregation for Federated Learning with Heterogeneous Model Architecturescited by 2
  9. 2025FLEx: Personalized Federated Learning for Mixture-of-Experts LLMs via Expert Graftingcited by 2
  10. 2025VeriLLM: A Lightweight Framework for Publicly Verifiable Decentralized Inferencecited by 2
  11. 2025FLEX-MoE: Federated Mixture-of-Experts with Load-balanced Expert Assignment for Edge Computingcited by 2
  12. 2025Towards Efficient Federated Learning of Networked Mixture-of-Experts for Mobile Edge Computingcited by 1
15 more
  1. 2025Integration of TinyML and LargeML: A Survey of 6G and Beyondcited by 0
  2. 2025Shift Happens: Mixture of Experts based Continual Adaptation in Federated Learningcited by 0
  3. 2025Closer to Reality: Practical Semi-Supervised Federated Learning for Foundation Model Adaptationcited by 0
  4. 2025Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devicescited by 0
  5. 2026DriftGuard: Mitigating Asynchronous Data Drift in Federated Learningcited by 0
  6. 2026Love Me, Love My Label: Rethinking the Role of Labels in Prompt Retrieval for Visual In-Context Learningcited by 0
  7. 2026FedCoE: Bridging Generalization and Personalization via Federated Coordinated Dual-level MoEscited by 0
  8. 2026Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Expertscited by 0
  9. 2026FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEscited by 0
  10. 2026Fisher-Routed Mixture of Experts for Federated Class-Incremental Learningcited by 0
  11. 2026FedWeave: Rethinking the Unit of Specialization in Heterogeneous Federated MoE-LoRAcited by 0
  12. 2026MedMix: Specialization-Consistent Federated Sparse MoEs under Modality Heterogeneitycited by 0
  13. 2026MoE Router-Guided Clustering for Heterogeneous Federated Instruction Tuningcited by 0
  14. 2026FedGuide: Diffusion Prior Alignment and Value Baseline Guidance for Heterogeneous Federated Reinforcement Learningcited by 0
  15. 2026RoboFL: Federated Expert Assembly for World Action Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregation
  2. 2024pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learning
  3. 2022Adaptive Expert Models for Personalization in Federated Learning
  4. 2025FedPref: Federated Learning Across Heterogeneous Multi-objective Preferences
  5. 2025PM-MOE: Mixture of Experts on Private Model Parameters for Personalized Federated Learning
  6. 2021Federated Mixture of Experts
  7. 2024On-Device Collaborative Language Modeling via a Mixture of Generalists and Specialists
  8. 2026Task-Aware Federated Fine-Tuning for MoE-based Large Language Models