mlsanityData to 14 Oct 2026

Aggregation Alignment for Federated Learning with Mixture-of-Experts under Data Heterogeneity

arXiv 2603.21276 ↗First posted 22 Mar 2026cs.LG cs.AIcited by 4

On the map: Mixture-of-experts → Federated mixture-of-experts

Abstract (arXiv)

The growing demand for on-device large language model (LLM) services on mobile edge devices has driven the adoption of Mixture-of-Experts (MoE) architectures, which scale model capacity with limited computation. Since fine-tuning MoE-based LLMs relies on privacy-sensitive local data, federated learning (FL) offers a natural paradigm for collaborative training without exposing raw data. However, integrating MoE-based LLM fine-tuning into FL faces two critical challenges caused by data heterogeneity across clients: (i) divergent local data distributions drive clients to develop distinct gating preferences, so direct parameter aggregation yields a one-size-fits-none global gating network; and (ii) same-indexed experts develop disparate semantic roles across devices, leading to expert semantic blurring and degraded specialization. To address these challenges, we propose FedAlign-MoE, a federated aggregation alignment framework for edge computing systems that jointly enforces routing consistency and expert semantic alignment. Specifically, FedAlign-MoE aggregates gating behaviors by aligning routing distributions through consistency weighting and optimizes local gating networks through distribution regularization, maintaining cross-client stability while preserving discriminative local gating preferences. Meanwhile, FedAlign-MoE quantifies the semantic consistency of same-indexed experts across devices and selectively aggregates semantically aligned experts, ensuring stable and specialized global experts. Extensive experiments demonstrate that FedAlign-MoE outperforms state-of-the-art benchmarks, achieving faster convergence and higher accuracy in non-IID federated environments with lightweight computation and efficient communication.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Federated mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 38 of its 57 references that are in the map. First posted on arXiv 22 Mar 2026.

Builds on

38 of 57 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Experts2 Jan
  2. 2025FFT-MoE: Efficient Federated Fine-Tuning for Foundation Models via Large-scale Sparse MoE under Heterogeneous Edge26 Aug
  3. 2025dFLMoE: Decentralized Federated Learning via Mixture of Experts for Medical Data Analysis13 Mar
  4. 2025PM-MOE: Mixture of Experts on Private Model Parameters for Personalized Federated Learning1 Feb
  5. 2024FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregation4 Nov
  6. 2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts21 Aug
  7. 2024pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learning2 Feb
  8. 2022Adaptive Expert Models for Personalization in Federated Learning15 Jun
  9. 2020PFL-MoE: Personalized Federated Learning Based on Mixture of Experts31 Dec
  10. 2020Specialized federated learning using a mixture of experts5 Oct

Other references · 28

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  9. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  10. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 503
  11. 2021Federated Learning on Non-IID Data: A Surveycited by 346
  12. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
16 more
  1. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  2. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  3. 2023Pushing Large Language Models to the 6G Edge: Vision, Challenges, and Opportunitiescited by 90
  4. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 69
  5. 2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
  6. 2020FedGroup: Efficient Clustered Federated Learning via Decomposed Data-Driven Measurecited by 46
  7. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 38
  8. 2023FedSN: A Federated Learning Framework over Heterogeneous LEO Satellite Networkscited by 38
  9. 2023FedCP: Separating Feature Information for Personalized Federated Learning via Conditional Policycited by 36
  10. 2025HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 31
  11. 2024Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Modelscited by 25
  12. 2024AgentsCoMerge: Large Language Model Empowered Collaborative Decision Making for Ramp Mergingcited by 22
  13. 2025FedVLA: Federated Vision-Language-Action Learning with Dual Gating Mixture-of-Experts for Robotic Manipulationcited by 18
  14. 2025Dynamic Uncertainty-aware Multimodal Fusion for Outdoor Health Monitoringcited by 10
  15. 2023Optimization Design for Federated Learning in Heterogeneous 6G Networkscited by 7
  16. 2026Conflict-Aware Client Selection for Multi-Server Federated Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026SL-FAC: A Communication-Efficient Split Learning Framework with Frequency-Aware Compressioncited by 1
  2. 2026Physically-Induced Atmospheric Adversarial Perturbations: Enhancing Transferability and Robustness in Remote Sensing Image Classificationcited by 1
  3. 2026Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Expertscited by 0
  4. 2026MedMix: Specialization-Consistent Federated Sparse MoEs under Modality Heterogeneitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Task-Aware Federated Fine-Tuning for MoE-based Large Language Models
  2. 2025FLEx: Personalized Federated Learning for Mixture-of-Experts LLMs via Expert Grafting
  3. 2026FedCoE: Bridging Generalization and Personalization via Federated Coordinated Dual-level MoEs
  4. 2026AlignFed: Alignment-Aware Asynchronous Federated Fine-Tuning for Large Language Models in Heterogeneous Edge Environments
  5. 2026MoE Router-Guided Clustering for Heterogeneous Federated Instruction Tuning
  6. 2025Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices
  7. 2025Efficient Training of Large-Scale AI Models Through Federated Mixture-of-Experts: A System-Level Approach
  8. 2025Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning