mlsanityData to 14 Oct 2026

Improving Expert Specialization in Mixture of Experts

arXiv 2302.14703 ↗First posted 28 Feb 2023cs.LG cs.AI cs.NEcited by 17

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Mixture of experts (MoE), introduced over 20 years ago, is the simplest gated modular neural network architecture. There is renewed interest in MoE because the conditional computation allows only parts of the network to be used during each inference, as was recently demonstrated in large scale natural language processing models. MoE is also of potential interest for continual learning, as experts may be reused for new tasks, and new experts introduced. The gate in the MoE architecture learns task decompositions and individual experts learn simpler functions appropriate to the gate's decomposition. In this paper: (1) we show that the original MoE architecture and its training method do not guarantee intuitive task decompositions and good expert utilization, indeed they can fail spectacularly even for simple data such as MNIST and FashionMNIST; (2) we introduce a novel gating architecture, similar to attention, that improves performance and results in a lower entropy task decomposition; and (3) we introduce a novel data-driven regularization that improves expert specialization. We empirically validate our methods on MNIST, FashionMNIST and CIFAR-100 datasets.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 16 of its 25 references that are in the map. First posted on arXiv 28 Feb 2023.

Builds on

16 of 25 references.

References · 16

  1. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  6. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  7. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  8. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  9. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  10. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  11. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  12. 2020Efficient Continual Learning with Modular Networks and Task-Driven Priorscited by 88
4 more
  1. 2017One Model To Learn Them Allcited by 87
  2. 2018Modular Networks: Learning to Decompose Neural Computationcited by 62
  3. 2022Is a Modular Architecture Enough?cited by 36
  4. 2021Mixture-of-Variational-Experts for Continual Learningcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 4; 2026: 5.

17 papers.

Sort:

By area:

  1. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  2. 2024HMoE: Heterogeneous Mixture of Experts for Language Modelingcited by 29
  3. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Expertscited by 16
  4. 2024On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functionscited by 16
  5. 2025Mixture of Experts in Large Language Modelscited by 15
  6. 2025ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specializationcited by 7
  7. 2025Equipping Vision Foundation Model with Mixture of Experts for Out-of-Distribution Detectioncited by 5
  8. 2026Expert Divergence Learning for MoE-based Language Modelscited by 4
  9. 2026SD-MoE: Spectral Decomposition for Effective Expert Specializationcited by 3
  10. 2024Graph Knowledge Distillation to Mixture of Expertscited by 2
  11. 2024Flexible task abstractions emerge in linear networks with fast and bounded unitscited by 2
  12. 2025Video-VoT-R1: An efficient video inference model integrating image packing and AoE architecturecited by 2
5 more
  1. 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Expertscited by 1
  2. 2024Active partitioning: inverting the paradigm of active learningcited by 0
  3. 2026Do Domain-specific Experts exist in MoE-based LLMs?cited by 0
  4. 2026Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlapcited by 0
  5. 2026Distributionally Robust Mixture-of-Experts Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
  2. 2026The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level
  3. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  4. 2025Advancing Expert Specialization for Better MoE
  5. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
  6. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
  7. 2024MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts
  8. 2025Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations