mlsanityData to 14 Oct 2026

Federated Mixture of Experts

arXiv 2107.06724 ↗First posted 14 Jul 2021cs.LG cs.DCcited by 28

On the map: Mixture-of-experts → Federated mixture-of-experts

Abstract (arXiv)

Federated learning (FL) has emerged as the predominant approach for collaborative training of neural network models across multiple users, without the need to gather the data at a central location. One of the important challenges in this setting is data heterogeneity, i.e. different users have different data characteristics. For this reason, training and using a single global model might be suboptimal when considering the performance of each of the individual user's data. In this work, we tackle this problem via Federated Mixture of Experts, FedMix, a framework that allows us to train an ensemble of specialized models. FedMix adaptively selects and trains a user-specific selection of the ensemble members. We show that users with similar data characteristics select the same members and therefore share statistical strength while mitigating the effect of non-i.i.d data. Empirically, we show through an extensive experimental evaluation that FedMix improves performance compared to using a single global model across a variety of different sources of non-i.i.d.-ness.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Federated mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 26 of its 33 references that are in the map. First posted on arXiv 14 Jul 2021.

Builds on

26 of 33 references.

References · 26

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2018Group Normalizationcited by 1,524
  6. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  7. 2020Adaptive Federated Optimizationcited by 1,046
  8. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  9. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  10. 2019Towards Federated Learning at Scale: System Designcited by 832
  11. 2018LEAF: A Benchmark for Federated Settingscited by 792
  12. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 503
14 more
  1. 2019Agnostic Federated Learningcited by 456
  2. 2018Do Deep Generative Models Know What They Don't Know?cited by 443
  3. 2019Fair Resource Allocation in Federated Learningcited by 394
  4. 2020Adaptive Personalized Federated Learningcited by 358
  5. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 325
  6. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 325
  7. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 312
  8. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 269
  9. 2020Federated learning with hierarchical clustering of local updates to improve training on non-IID datacited by 223
  10. 2019Adaptive Gradient-Based Meta-Learning Methodscited by 183
  11. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  12. 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 163
  13. 2018Federated Meta-Learning with Fast Convergence and Efficient Communicationcited by 155
  14. 2020Survey of Personalization Techniques for Federated Learningcited by 145

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 0; 2023: 5; 2024: 9; 2025: 7; 2026: 5.

28 papers.

Sort:

By area:

  1. 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 109
  2. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  3. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  4. 2024pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learningcited by 24
  5. 2024Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Modelscited by 17
  6. 2024Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architecturescited by 12
  7. 2024FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregationcited by 11
  8. 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Expertscited by 11
  9. 2024DiPaCo: Distributed Path Compositioncited by 10
  10. 2023Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settingscited by 7
  11. 2023Federated Learning of Shareable Bases for Personalization-Friendly Image Classificationcited by 6
  12. 2021User-Centric Federated Learningcited by 3
16 more
  1. 2023User-Centric Federated Learning: Trading off Wireless Resources for Personalizationcited by 3
  2. 2023Enhancing Clustered Federated Learning: Integration of Strategies and Improved Methodologiescited by 3
  3. 2025Efficient Training of Large-Scale AI Models Through Federated Mixture-of-Experts: A System-Level Approachcited by 3
  4. 2024Towards Vision Mixture of Experts for Wildlife Monitoring on the Edgecited by 2
  5. 2025FLEX-MoE: Federated Mixture-of-Experts with Load-balanced Expert Assignment for Edge Computingcited by 2
  6. 2024Collaborative and Efficient Personalization with Mixtures of Adaptorscited by 1
  7. 2023Global Layers: Non-IID Tabular Federated Learningcited by 0
  8. 2024Federated Bayesian Network Ensemblescited by 0
  9. 2025Token-Level Prompt Mixture with Parameter-Free Routing for Federated Domain Generalizationcited by 0
  10. 2025Topology-Assisted Spatio-Temporal Pattern Disentangling for Scalable MARL in Large-scale Autonomous Traffic Controlcited by 0
  11. 2025Shift Happens: Mixture of Experts based Continual Adaptation in Federated Learningcited by 0
  12. 2025Closer to Reality: Practical Semi-Supervised Federated Learning for Foundation Model Adaptationcited by 0
  13. 2026FedSQ: Optimized Weight Averaging via Fixed Gatingcited by 0
  14. 2026FedWeave: Rethinking the Unit of Specialization in Heterogeneous Federated MoE-LoRAcited by 0
  15. 2026MedMix: Specialization-Consistent Federated Sparse MoEs under Modality Heterogeneitycited by 0
  16. 2026Regime-Aware Router Learning for Multi-Domain Medical Image Generation with Diffusion Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts
  2. 2022Adaptive Expert Models for Personalization in Federated Learning
  3. 2020Specialized federated learning using a mixture of experts
  4. 2024Harnessing Increased Client Participation with Cohort-Parallel Federated Learning
  5. 2025FedPref: Federated Learning Across Heterogeneous Multi-objective Preferences
  6. 2024Federated Learning with a Single Shared Image
  7. 2026FedCoE: Bridging Generalization and Personalization via Federated Coordinated Dual-level MoEs
  8. 2024Federated Cross-Training Learners for Robust Generalization under Data Heterogeneity