mlsanityData to 7 Oct 2026

FLEx: Personalized Federated Learning for Mixture-of-Experts LLMs via Expert Grafting

arXiv 2506.00965 ↗First posted 1 Jun 2025cs.AIcited by 2

On the map: Mixture-of-experts → Federated mixture-of-experts

Abstract (arXiv)

Federated instruction tuning of large language models (LLMs) is challenged by significant data heterogeneity across clients, demanding robust personalization. The Mixture of Experts (MoE) architecture, where experts can specialize in distinct data patterns, presents a natural architectural solution to this challenge. The inherent sparsity of the MoE architecture, achieved by selectively activating experts, poses a significant challenge to its integration with federated learning (FL). Conventional FL frameworks, designed for dense models, naively aggregate all expert parameters irrespective of their local activation patterns. This naive approach not only undermines MoE's dynamic sparsity but also risks corrupting the world knowledge within pretrained experts. To address this, we propose FLEx (Federated LLMs with Personalized Experts), a novel framework that leverages pretrained MoE-based LLMs for efficient personalization. By aggregating only the shared non-expert parameters, FLEx significantly reduces communication overhead and preserves the world knowledge stored within the frozen pretrained experts. For personalization, we introduce a novel expert grafting mechanism that leverages dynamic sparsity to construct a client-specific expert from selected components of pretrained experts, tailored to local data. This grafted expert is then fine-tuned locally alongside the gating mechanism. This joint training enables the model to learn when to leverage the shared knowledge from frozen experts and when to employ the personalized one. Evaluations on diverse, non-IID instruction tuning datasets show that FLEx consistently outperforms federated baselines on average, while demonstrating strong knowledge preservation on the knowledge-driven benchmark MMLU. Our code is available at hrefhttps://anonymous.4open.science/r/FLEx-8F12texttthttps://anonymous.4open.science/r/FLEx-8F12.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Federated mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 21 of its 27 references that are in the map. First posted on arXiv 1 Jun 2025.

Builds on

21 of 27 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregation4 Nov
  2. 2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts21 Aug

Other references · 19

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  6. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  7. 2020Adaptive Federated Optimizationcited by 1,036
  8. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  9. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  10. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  11. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  12. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
7 more
  1. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  2. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  3. 2024FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuningcited by 57
  4. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 54
  5. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 51
  6. 2024Federated Large Language Models: Current Progress and Future Directionscited by 22
  7. 2024Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Modelscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoEcited by 0
  2. 2026MoE Router-Guided Clustering for Heterogeneous Federated Instruction Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Experts
  2. 2026Aggregation Alignment for Federated Learning with Mixture-of-Experts under Data Heterogeneity
  3. 2022Adaptive Expert Models for Personalization in Federated Learning
  4. 2023Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings
  5. 2026Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Experts
  6. 2026FlexMoRE: A Flexible Mixture of Rank-heterogeneous Experts for Efficient Federatedly-trained Large Language Models
  7. 2026FedCoE: Bridging Generalization and Personalization via Federated Coordinated Dual-level MoEs
  8. 2025Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices