mlsanityData to 7 Oct 2026

FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregation

arXiv 2411.02115 ↗First posted 4 Nov 2024cs.LG cs.DCcited by 11

On the map: Mixture-of-experts → Federated mixture-of-experts

Abstract (arXiv)

Federated learning (FL) is a collaborative machine learning approach that enables multiple clients to train models without sharing their private data. With the rise of deep learning, large-scale models have garnered significant attention due to their exceptional performance. However, a key challenge in FL is the limitation imposed by clients with constrained computational and communication resources, which hampers the deployment of these large models. The Mixture of Experts (MoE) architecture addresses this challenge with its sparse activation property, which reduces computational workload and communication demands during inference and updates. Additionally, MoE facilitates better personalization by allowing each expert to specialize in different subsets of the data distribution. To alleviate the communication burdens between the server and clients, we propose FedMoE-DA, a new FL model training framework that leverages the MoE architecture and incorporates a novel domain-aware, fine-grained aggregation strategy to enhance the robustness, personalizability, and communication efficiency simultaneously. Specifically, the correlation between both intra-client expert models and inter-client data heterogeneity is exploited. Moreover, we utilize peer-to-peer (P2P) communication between clients for selective expert model synchronization, thus significantly reducing the server-client transmissions. Experiments demonstrate that our FedMoE-DA achieves excellent performance while reducing the communication pressure on the server.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Federated mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 35 of its 43 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

35 of 43 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Adaptive Expert Models for Personalization in Federated Learning15 Jun
  2. 2021Federated Mixture of Experts14 Jul
  3. 2020PFL-MoE: Personalized Federated Learning Based on Mixture of Experts31 Dec
  4. 2020Specialized federated learning using a mixture of experts5 Oct

Other references · 31

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  7. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  8. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  9. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  10. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  11. 2021Model-Contrastive Federated Learningcited by 762
  12. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
19 more
  1. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  2. 2017Federated Multi-Task Learningcited by 669
  3. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  4. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  5. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  6. 2019Federated Learning with Personalization Layerscited by 533
  7. 2020An Efficient Framework for Clustered Federated Learningcited by 505
  8. 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 408
  9. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 407
  10. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
  11. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 242
  12. 2021Federated Multi-Task Learning under a Mixture of Distributionscited by 189
  13. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  14. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
  15. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 152
  16. 2020Federated Visual Classification with Real-World Data Distributioncited by 126
  17. 2023Personalized Federated Learning with Feature Alignment and Classifier Collaborationcited by 97
  18. 2020LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasetscited by 60
  19. 2022FedClassAvg: Local Representation Learning for Personalized Federated Learning on Heterogeneous Neural Networkscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 6.

11 papers.

  1. 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Expertscited by 11
  2. 2024Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architecturescited by 10
  3. 2025FFT-MoE: Efficient Federated Fine-Tuning for Foundation Models via Large-scale Sparse MoE under Heterogeneous Edgecited by 7
  4. 2026Aggregation Alignment for Federated Learning with Mixture-of-Experts under Data Heterogeneitycited by 4
  5. 2025Efficient Training of Large-Scale AI Models Through Federated Mixture-of-Experts: A System-Level Approachcited by 3
  6. 2025FLEx: Personalized Federated Learning for Mixture-of-Experts LLMs via Expert Graftingcited by 2
  7. 2025FLEX-MoE: Federated Mixture-of-Experts with Load-balanced Expert Assignment for Edge Computingcited by 2
  8. 2026DeepFusion: Accelerating MoE Training via Federated Knowledge Distillation from Heterogeneous Edge Devicescited by 0
  9. 2026FedCoE: Bridging Generalization and Personalization via Federated Coordinated Dual-level MoEscited by 0
  10. 2026Fisher-Routed Mixture of Experts for Federated Class-Incremental Learningcited by 0
  11. 2026MoE Router-Guided Clustering for Heterogeneous Federated Instruction Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts
  2. 2025dFLMoE: Decentralized Federated Learning via Mixture of Experts for Medical Data Analysis
  3. 2026Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Experts
  4. 2025PM-MOE: Mixture of Experts on Private Model Parameters for Personalized Federated Learning
  5. 2023Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings
  6. 2026UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Models
  7. 2022FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformers
  8. 2025Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments