mlsanityData to 7 Oct 2026

Towards an empirical understanding of MoE design choices

arXiv 2402.13089 ↗First posted 20 Feb 2024cs.LG cs.AI cs.CLcited by 20

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

In this study, we systematically evaluate the impact of common design choices in Mixture of Experts (MoEs) on validation performance, uncovering distinct influences at token and sequence levels. We also present empirical evidence showing comparable performance between a learned router and a frozen, randomly initialized router, suggesting that learned routing may not be essential. Our study further reveals that Sequence-level routing can result in topic-specific weak expert specialization, in contrast to syntax specialization observed with Token-level routing.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 13 of its 20 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

13 of 20 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 12

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  4. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  5. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  6. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  7. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  8. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  9. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  10. 2021Hash Layers For Large Sparse Modelscited by 236
  11. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  12. 2020X-Stance: A Multilingual Multi-Target Dataset for Stance Detectioncited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 4; 2026: 8.

20 papers.

Sort:

By area:

  1. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 55
  2. 2024Upcycling Large Language Models into Mixture of Expertscited by 48
  3. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Modelscited by 43
  4. 2024Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrastcited by 19
  5. 2025Mixture of Experts in Large Language Modelscited by 15
  6. 2025TimeExpert: An Expert-Guided Video LLM for Video Temporal Groundingcited by 14
  7. 2025Probing Semantic Routing in Large Mixture-of-Expert Modelscited by 13
  8. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  9. 2024On-Device Collaborative Language Modeling via a Mixture of Generalists and Specialistscited by 6
  10. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
  11. 2026The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Modelscited by 6
  12. 2026Self-Routing: Parameter-Free Expert Routing from Hidden Statescited by 3
8 more
  1. 2024Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsitycited by 2
  2. 2026Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beatcited by 2
  3. 2026Routing Sensitivity Without Controllability: A Diagnostic Study of Fairness in MoE Language Modelscited by 1
  4. 2024A Novel Method to Metigate Demographic and Expert Bias in ICD Coding with Causal Inferencecited by 0
  5. 2025RouterKT: Mixture-of-Experts for Knowledge Tracingcited by 0
  6. 2026Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Expertscited by 0
  7. 2026Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatorycited by 0
  8. 2026MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise
  2. 2026MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
  3. 2025Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules
  4. 2026Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation
  5. 2024GW-MoE: Resolving Uncertainty in MoE Router with Global Workspace Theory
  6. 2026DBES: A Systematic Benchmark and Metric Suite for Evaluating Expert Specialization in Large-Scale MoEs
  7. 2026Safety-Oriented Routing Analysis of Mixtral MoE Under Benign and Harmful Prompts
  8. 2026Slicing and Dicing: Configuring Optimal Mixtures of Experts