mlsanityData to 7 Oct 2026

Mixture-of-Experts with Expert Choice Routing

arXiv 2202.09368 ↗First posted 18 Feb 2022cs.LG cs.AIcited by 611

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Sparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy (e.g. one resulting in load imbalance) can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2x. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.

Key facts. Cited by 611 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 31 of its 39 references that are in the map. First posted on arXiv 18 Feb 2022.

Builds on

31 of 39 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Taming Sparsely Activated Transformer with Stochastic Experts8 Oct
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
17 more
  1. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  2. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  3. 2016Language Modeling with Gated Convolutional Networkscited by 688
  4. 2019Green AIcited by 620
  5. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  6. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  7. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  8. 2019NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detectioncited by 289
  9. 2021Hash Layers For Large Sparse Modelscited by 236
  10. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  11. 2020Conditional Channel Gated Networks for Task-Aware Continual Learningcited by 115
  12. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  13. 2017Hard Mixtures of Experts for Large Scale Weakly Supervised Visioncited by 67
  14. 2020Scalable Transfer Learning with Expert Modelscited by 48
  15. 2014Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learningcited by 24
  16. 2021Tricks for Training Sparse Translation Modelscited by 17
  17. 2019Conditional Computation for Continual Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 57; 2024: 164; 2025: 179; 2026: 203.

611 papers.

Sort:

By area:

  1. 2024Mixtral of Expertscited by 1,785
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  3. 2022Confident Adaptive Language Modelingcited by 297
  4. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  5. 2023From Sparse to Soft Mixtures of Expertscited by 245
  6. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 235
  7. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  8. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  9. 2023Perpetual Humanoid Control for Real-time Simulated Avatarscited by 198
  10. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  11. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 191
  12. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
38 more
  1. 2023Efficient Large Language Models: A Surveycited by 183
  2. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
  3. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  4. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
  5. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  6. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  7. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  8. 2023Chain of Code: Reasoning with a Language Model-Augmented Code Emulatorcited by 120
  9. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  10. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 114
  11. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  12. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 101
  13. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  14. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 95
  15. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
  16. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  17. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  18. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  19. 2023Modular Deep Learningcited by 82
  20. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  21. 2023Alleviating Hallucinations of Large Language Models through Induced Hallucinationscited by 78
  22. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  23. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  24. 2024Complexity Experts are Task-Discriminative Learners for Any Image Restorationcited by 72
  25. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Modelscited by 71
  26. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 71
  27. 2024Mixture of A Million Expertscited by 70
  28. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  29. 2024MVMoE: Multi-Task Vehicle Routing Solver with Mixture-of-Expertscited by 69
  30. 2024Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Expertscited by 69
  31. 2024Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Modelcited by 67
  32. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
  33. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
  34. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 65
  35. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  36. 2024Learning to Decode Collaboratively with Multiple Language Modelscited by 61
  37. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Expertscited by 61
  38. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59

50 most cited of 611 All 611 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Distributionally Robust Mixture-of-Experts Training
  2. 2026Beyond Routing: Decoupling Expert Dispatch and Aggregation in Sparse Mixture-of-Experts
  3. 2026Routing-Free Mixture-of-Experts
  4. 2026Towards a Statistical Understanding of Mixture-of-Experts
  5. 2022SMILE: Scaling Mixture-of-Experts with Efficient Bi-level Routing
  6. 2026Conditional Capacity and Routing in Mixture-of-Experts Particle Transformers
  7. 2026Elbow-Based MoE Routing: A Training-Free Inference Time Plugin for Expert Selection
  8. 2025Binary-Integer-Programming Based Algorithm for Expert Load Balancing in Mixture-of-Experts Models