# Specialized federated learning using a mixture of experts

arXiv: https://arxiv.org/abs/2010.02056 · HTML: https://mlsanity.com/paper/2010.02056/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Federated mixture-of-experts, one of 12 directions in Mixture-of-experts.
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 5 Oct 2020.

## Placed in

- Mixture-of-experts / [Federated mixture-of-experts](https://mlsanity.com/moe/federated-mixture-of-experts/)

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Federated Learning with Personalization Layers](https://mlsanity.com/paper/1912.00818/) (2019)
- [Adaptive Personalized Federated Learning](https://mlsanity.com/paper/2003.13461/) (2020)
- [Personalized Federated Learning: A Meta-Learning Approach](https://mlsanity.com/paper/2002.07948/) (2020)
- [Improving Federated Learning Personalization via Model Agnostic Meta Learning](https://mlsanity.com/paper/1909.12488/) (2019)
- [Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learning](https://mlsanity.com/paper/1812.00535/) (2018)
- [Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data](https://mlsanity.com/paper/1811.11479/) (2018)
- [Federated Learning of a Mixture of Global and Local Models](https://mlsanity.com/paper/2002.05516/) (2020)
- [Federated Evaluation of On-device Personalization](https://mlsanity.com/paper/1910.10252/) (2019)
- [Decentralized Collaborative Learning of Personalized Models over Networks](https://mlsanity.com/paper/1610.05202/) (2016)
- [Personalized and Private Peer-to-Peer Machine Learning](https://mlsanity.com/paper/1705.08435/) (2017)
- [Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge](https://mlsanity.com/paper/2007.14513/) (2020)

## Built on by (most cited first, 23 of 23)

- [A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications](https://mlsanity.com/paper/2503.07137/) (2025)
- [Scaling Federated Learning for Fine-tuning of Large Language Models](https://mlsanity.com/paper/2102.00875/) (2021)
- [Advances and Open Challenges in Federated Foundation Models](https://mlsanity.com/paper/2404.15381/) (2024)
- [FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts](https://mlsanity.com/paper/2408.11304/) (2024)
- [Decentralized federated learning of deep neural networks on non-iid data](https://mlsanity.com/paper/2107.08517/) (2021)
- [The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillation](https://mlsanity.com/paper/2301.08968/) (2023)
- [pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learning](https://mlsanity.com/paper/2402.01350/) (2024)
- [On Bridging Generic and Personalized Federated Learning for Image Classification](https://mlsanity.com/paper/2107.00778/) (2021)
- [FLIX: A Simple and Communication-Efficient Alternative to Local Methods in Federated Learning](https://mlsanity.com/paper/2111.11556/) (2021)
- [Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architectures](https://mlsanity.com/paper/2411.19128/) (2024)
- [FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregation](https://mlsanity.com/paper/2411.02115/) (2024)
- [Federated Asymptotics: a model to compare federated learning algorithms](https://mlsanity.com/paper/2108.07313/) (2021)
- [Practical and Secure Federated Recommendation with Personalized Masks](https://mlsanity.com/paper/2109.02464/) (2021)
- [Federated Learning of Shareable Bases for Personalization-Friendly Image Classification](https://mlsanity.com/paper/2304.07882/) (2023)
- [FedMS: Federated Learning with Mixture of Sparsely Activated Foundations Models](https://mlsanity.com/paper/2312.15926/) (2023)
- [Aggregation Alignment for Federated Learning with Mixture-of-Experts under Data Heterogeneity](https://mlsanity.com/paper/2603.21276/) (2026)
- [Towards Vision Mixture of Experts for Wildlife Monitoring on the Edge](https://mlsanity.com/paper/2411.07834/) (2024)
- [FLEX-MoE: Federated Mixture-of-Experts with Load-balanced Expert Assignment for Edge Computing](https://mlsanity.com/paper/2512.23070/) (2025)
- [Demystifying the Effects of Non-Independence in Federated Learning](https://mlsanity.com/paper/2103.11226/) (2021)
- [Friends in Unexpected Places: Enhancing Local Fairness in Federated Learning through Clustering](https://mlsanity.com/paper/2407.19331/) (2024)
- [Shift Happens: Mixture of Experts based Continual Adaptation in Federated Learning](https://mlsanity.com/paper/2506.18789/) (2025)
- [FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs](https://mlsanity.com/paper/2606.19025/) (2026)
- [FedWeave: Rethinking the Unit of Specialization in Heterogeneous Federated MoE-LoRA](https://mlsanity.com/paper/2607.26618/) (2026)

## Abstract (arXiv)

In federated learning, clients share a global model that has been trained on decentralized local client data. Although federated learning shows significant promise as a key approach when data cannot be shared or centralized, current methods show limited privacy properties and have shortcomings when applied to common real-world scenarios, especially when client data is heterogeneous. In this paper, we propose an alternative method to learn a personalized model for each client in a federated setting, with greater generalization abilities than previous methods. To achieve this personalization we propose a federated learning framework using a mixture of experts to combine the specialist nature of a locally trained model with the generalist knowledge of a global model. We evaluate our method on a variety of datasets with different levels of data heterogeneity, and our results show that the mixture of experts model is better suited as a personalized model for devices in these settings, outperforming both fine-tuned global models and local specialists.