mlsanityData to 7 Oct 2026

Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings

arXiv 2306.08586 ↗First posted 14 Jun 2023cs.LG cs.AI math.OCcited by 7

On the map: Mixture-of-experts → Federated mixture-of-experts

Abstract (arXiv)

Mixture-of-Experts (MoEs) achieve scalability by dynamically activating subsets of their components. Yet, understanding how expertise emerges through joint training of gating mechanisms and experts remains incomplete, especially in scenarios without clear task partitions. Motivated by inference costs and data heterogeneity, we study how joint training of gating functions and experts can dynamically allocate domain-specific expertise across multiple underlying data distributions. As an outcome of our framework, we develop an instance tailored specifically to decentralized training scenarios, introducing textitDynamically Decentralized Orchestration of MoEs or textttDDOME. textttDDOME leverages heterogeneity emerging from distributional shifts across decentralized data sources to specialize experts dynamically. By integrating a pretrained common expert to inform a gating function, textttDDOME achieves personalized expert subset selection on-the-fly, facilitating just-in-time personalization. We empirically validate textttDDOME within a Federated Learning (FL) context: textttDDOME attains from 4% up to an 24% accuracy improvement over state-of-the-art FL baselines in image and text classification tasks, while maintaining competitive zero-shot generalization capabilities. Furthermore, we provide theoretical insights confirming that the joint gating-experts training is critical for achieving meaningful expert specialization.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Federated mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 23 of its 34 references that are in the map. First posted on arXiv 14 Jun 2023.

Builds on

23 of 34 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Federated Mixture of Experts14 Jul

Other references · 22

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  12. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
10 more
  1. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  2. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  3. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  4. 2019Federated Evaluation of On-device Personalizationcited by 147
  5. 2020Salvaging Federated Learning by Local Adaptationcited by 142
  6. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  7. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  8. 2020Scalable Transfer Learning with Expert Modelscited by 48
  9. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Expertscited by 48
  10. 2021FedCV: A Federated Learning Framework for Diverse Computer Vision Taskscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 3.

7 papers.

  1. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  2. 2024pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learningcited by 21
  3. 2024Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Modelscited by 16
  4. 2025Efficient Training of Large-Scale AI Models Through Federated Mixture-of-Experts: A System-Level Approachcited by 3
  5. 2026Federated Inference: Toward Privacy-Preserving Collaborative and Incentivized Model Servingcited by 1
  6. 2026FedCoE: Bridging Generalization and Personalization via Federated Coordinated Dual-level MoEscited by 0
  7. 2026Fisher-Routed Mixture of Experts for Federated Class-Incremental Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FLEx: Personalized Federated Learning for Mixture-of-Experts LLMs via Expert Grafting
  2. 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Experts
  3. 2026Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization
  4. 2026Towards a Statistical Understanding of Mixture-of-Experts
  5. 2022Adaptive Expert Models for Personalization in Federated Learning
  6. 2025FLEX-MoE: Federated Mixture-of-Experts with Load-balanced Expert Assignment for Edge Computing
  7. 2024FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregation
  8. 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts