# Federated Mixture of Experts

arXiv: https://arxiv.org/abs/2107.06724 · HTML: https://mlsanity.com/paper/2107.06724/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Federated mixture-of-experts, one of 12 directions in Mixture-of-experts.
- Builds on 26 of its 33 references that are in the map.
- First posted on arXiv 14 Jul 2021.

## Placed in

- Mixture-of-experts / [Federated mixture-of-experts](https://mlsanity.com/moe/federated-mixture-of-experts/)

## Builds on (most cited first, 26 of 26)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints](https://mlsanity.com/paper/1910.01991/) (2019)
- [Agnostic Federated Learning](https://mlsanity.com/paper/1902.00146/) (2019)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [Fair Resource Allocation in Federated Learning](https://mlsanity.com/paper/1905.10497/) (2019)
- [Adaptive Personalized Federated Learning](https://mlsanity.com/paper/2003.13461/) (2020)
- [Think Locally, Act Globally: Federated Learning with Local and Global Representations](https://mlsanity.com/paper/2001.01523/) (2020)
- [Three Approaches for Personalization with Applications to Federated Learning](https://mlsanity.com/paper/2002.10619/) (2020)
- [Personalized Federated Learning: A Meta-Learning Approach](https://mlsanity.com/paper/2002.07948/) (2020)
- [Improving Federated Learning Personalization via Model Agnostic Meta Learning](https://mlsanity.com/paper/1909.12488/) (2019)
- [Federated learning with hierarchical clustering of local updates to improve training on non-IID data](https://mlsanity.com/paper/2004.11791/) (2020)
- [Adaptive Gradient-Based Meta-Learning Methods](https://mlsanity.com/paper/1906.02717/) (2019)
- [Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors](https://mlsanity.com/paper/2005.07186/) (2020)
- [Expanding the Reach of Federated Learning by Reducing Client Resource Requirements](https://mlsanity.com/paper/1812.07210/) (2018)
- [Federated Meta-Learning with Fast Convergence and Efficient Communication](https://mlsanity.com/paper/1802.07876/) (2018)
- [Survey of Personalization Techniques for Federated Learning](https://mlsanity.com/paper/2003.08673/) (2020)

## Built on by (most cited first, 28 of 28)

- [Parameterized Knowledge Transfer for Personalized Federated Learning](https://mlsanity.com/paper/2111.02862/) (2021)
- [A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications](https://mlsanity.com/paper/2503.07137/) (2025)
- [A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning](https://mlsanity.com/paper/2408.07057/) (2024)
- [pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learning](https://mlsanity.com/paper/2402.01350/) (2024)
- [Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2410.10114/) (2024)
- [Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architectures](https://mlsanity.com/paper/2411.19128/) (2024)
- [FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregation](https://mlsanity.com/paper/2411.02115/) (2024)
- [HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Experts](https://mlsanity.com/paper/2601.00583/) (2026)
- [DiPaCo: Distributed Path Composition](https://mlsanity.com/paper/2403.10616/) (2024)
- [Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings](https://mlsanity.com/paper/2306.08586/) (2023)
- [Federated Learning of Shareable Bases for Personalization-Friendly Image Classification](https://mlsanity.com/paper/2304.07882/) (2023)
- [User-Centric Federated Learning](https://mlsanity.com/paper/2110.09869/) (2021)
- [User-Centric Federated Learning: Trading off Wireless Resources for Personalization](https://mlsanity.com/paper/2304.12930/) (2023)
- [Enhancing Clustered Federated Learning: Integration of Strategies and Improved Methodologies](https://mlsanity.com/paper/2310.05397/) (2023)
- [Efficient Training of Large-Scale AI Models Through Federated Mixture-of-Experts: A System-Level Approach](https://mlsanity.com/paper/2507.05685/) (2025)
- [Towards Vision Mixture of Experts for Wildlife Monitoring on the Edge](https://mlsanity.com/paper/2411.07834/) (2024)
- [FLEX-MoE: Federated Mixture-of-Experts with Load-balanced Expert Assignment for Edge Computing](https://mlsanity.com/paper/2512.23070/) (2025)
- [Collaborative and Efficient Personalization with Mixtures of Adaptors](https://mlsanity.com/paper/2410.03497/) (2024)
- [Global Layers: Non-IID Tabular Federated Learning](https://mlsanity.com/paper/2305.19290/) (2023)
- [Federated Bayesian Network Ensembles](https://mlsanity.com/paper/2402.12142/) (2024)
- [Token-Level Prompt Mixture with Parameter-Free Routing for Federated Domain Generalization](https://mlsanity.com/paper/2504.21063/) (2025)
- [Topology-Assisted Spatio-Temporal Pattern Disentangling for Scalable MARL in Large-scale Autonomous Traffic Control](https://mlsanity.com/paper/2506.12453/) (2025)
- [Shift Happens: Mixture of Experts based Continual Adaptation in Federated Learning](https://mlsanity.com/paper/2506.18789/) (2025)
- [Closer to Reality: Practical Semi-Supervised Federated Learning for Foundation Model Adaptation](https://mlsanity.com/paper/2508.16568/) (2025)
- [FedSQ: Optimized Weight Averaging via Fixed Gating](https://mlsanity.com/paper/2604.02990/) (2026)
- [FedWeave: Rethinking the Unit of Specialization in Heterogeneous Federated MoE-LoRA](https://mlsanity.com/paper/2607.26618/) (2026)
- [MedMix: Specialization-Consistent Federated Sparse MoEs under Modality Heterogeneity](https://mlsanity.com/paper/2608.13911/) (2026)
- [Regime-Aware Router Learning for Multi-Domain Medical Image Generation with Diffusion Experts](https://mlsanity.com/paper/2610.16082/) (2026)

## Abstract (arXiv)

Federated learning (FL) has emerged as the predominant approach for collaborative training of neural network models across multiple users, without the need to gather the data at a central location. One of the important challenges in this setting is data heterogeneity, i.e. different users have different data characteristics. For this reason, training and using a single global model might be suboptimal when considering the performance of each of the individual user's data. In this work, we tackle this problem via Federated Mixture of Experts, FedMix, a framework that allows us to train an ensemble of specialized models. FedMix adaptively selects and trains a user-specific selection of the ensemble members. We show that users with similar data characteristics select the same members and therefore share statistical strength while mitigating the effect of non-i.i.d data. Empirically, we show through an extensive experimental evaluation that FedMix improves performance compared to using a single global model across a variety of different sources of non-i.i.d.-ness.