mlsanityData to 7 Oct 2026

Specialized federated learning using a mixture of experts

arXiv 2010.02056 ↗First posted 5 Oct 2020cs.LGcited by 23

On the map: Mixture-of-experts → Federated mixture-of-experts

Abstract (arXiv)

In federated learning, clients share a global model that has been trained on decentralized local client data. Although federated learning shows significant promise as a key approach when data cannot be shared or centralized, current methods show limited privacy properties and have shortcomings when applied to common real-world scenarios, especially when client data is heterogeneous. In this paper, we propose an alternative method to learn a personalized model for each client in a federated setting, with greater generalization abilities than previous methods. To achieve this personalization we propose a federated learning framework using a mixture of experts to combine the specialist nature of a locally trained model with the generalist knowledge of a global model. We evaluate our method on a variety of datasets with different levels of data heterogeneity, and our results show that the mixture of experts model is better suited as a personalized model for devices in these settings, outperforming both fine-tuned global models and local specialists.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Federated mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 20 of its 27 references that are in the map. First posted on arXiv 5 Oct 2020.

Builds on

20 of 27 references.

References · 20

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2019Advances and Open Problems in Federated Learningcited by 3,021
  6. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  7. 2018Federated Learning with Non-IID Datacited by 1,099
  8. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  9. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  10. 2019Federated Learning with Personalization Layerscited by 533
  11. 2020Adaptive Personalized Federated Learningcited by 353
  12. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
8 more
  1. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 264
  2. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 242
  3. 2018Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learningcited by 242
  4. 2020Federated Learning of a Mixture of Global and Local Modelscited by 208
  5. 2019Federated Evaluation of On-device Personalizationcited by 147
  6. 2016Decentralized Collaborative Learning of Personalized Models over Networkscited by 73
  7. 2017Personalized and Private Peer-to-Peer Machine Learningcited by 54
  8. 2020Group Knowledge Transfer: Federated Learning of Large CNNs at the Edgecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 0; 2023: 3; 2024: 7; 2025: 3; 2026: 3.

23 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  2. 2021Scaling Federated Learning for Fine-tuning of Large Language Modelscited by 36
  3. 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
  4. 2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Expertscited by 27
  5. 2023The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillationcited by 25
  6. 2021Decentralized federated learning of deep neural networks on non-iid datacited by 23
  7. 2024pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learningcited by 21
  8. 2021On Bridging Generic and Personalized Federated Learning for Image Classificationcited by 15
  9. 2021FLIX: A Simple and Communication-Efficient Alternative to Local Methods in Federated Learningcited by 12
  10. 2024FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregationcited by 11
  11. 2024Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architecturescited by 10
  12. 2021Federated Asymptotics: a model to compare federated learning algorithmscited by 7
11 more
  1. 2021Practical and Secure Federated Recommendation with Personalized Maskscited by 7
  2. 2023Federated Learning of Shareable Bases for Personalization-Friendly Image Classificationcited by 6
  3. 2023FedMS: Federated Learning with Mixture of Sparsely Activated Foundations Modelscited by 4
  4. 2026Aggregation Alignment for Federated Learning with Mixture-of-Experts under Data Heterogeneitycited by 4
  5. 2024Towards Vision Mixture of Experts for Wildlife Monitoring on the Edgecited by 2
  6. 2025FLEX-MoE: Federated Mixture-of-Experts with Load-balanced Expert Assignment for Edge Computingcited by 2
  7. 2021Demystifying the Effects of Non-Independence in Federated Learningcited by 1
  8. 2024Friends in Unexpected Places: Enhancing Local Fairness in Federated Learning through Clusteringcited by 1
  9. 2025Shift Happens: Mixture of Experts based Continual Adaptation in Federated Learningcited by 0
  10. 2026FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEscited by 0
  11. 2026FedWeave: Rethinking the Unit of Specialization in Heterogeneous Federated MoE-LoRAcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Adaptive Expert Models for Personalization in Federated Learning
  2. 2026FedCoE: Bridging Generalization and Personalization via Federated Coordinated Dual-level MoEs
  3. 2025PM-MOE: Mixture of Experts on Private Model Parameters for Personalized Federated Learning
  4. 2020PFL-MoE: Personalized Federated Learning Based on Mixture of Experts
  5. 2026Global Federated Learning Strategies for Building Efficient Personalized Models
  6. 2021Personalised Federated Learning: A Combinational Approach
  7. 2025FedMerge: Federated Personalization via Model Merging
  8. 2025FLEx: Personalized Federated Learning for Mixture-of-Experts LLMs via Expert Grafting