Data to 7 Oct 2026

StableMoE: Stable Routing Strategy for Mixture of Experts

arXiv 2204.08396 ↗First posted 18 Apr 2022cs.LG cs.CLcited by 98

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

The Mixture-of-Experts (MoE) technique can scale up the model size of Transformers with an affordable computational overhead. We point out that existing learning-to-route MoE methods suffer from the routing fluctuation issue, i.e., the target expert of the same input may change along with training, but only one expert will be activated for the input during inference. The routing fluctuation tends to harm sample efficiency because the same input updates different experts but only one is finally used. In this paper, we propose StableMoE with two training stages to address the routing fluctuation problem. In the first training stage, we learn a balanced and cohesive routing strategy and distill it into a lightweight router decoupled from the backbone model. In the second training stage, we utilize the distilled router to determine the token-to-expert assignment and freeze it for a stable routing strategy. We validate our method on language modeling and multilingual machine translation. The results show that StableMoE outperforms existing MoE methods in terms of both convergence speed and performance.

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 18 of its 31 references that are in the map. First posted on arXiv 18 Apr 2022.

Builds on

18 of 31 references.

Closest ancestors · 1

Same direction, newest first

2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 17

2017Attention Is All You Needcited by 59,932
2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
2014Adam: A Method for Stochastic Optimizationcited by 36,960
2020Language Models are Few-Shot Learnerscited by 32,199
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668

5 more

2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
2021Hash Layers For Large Sparse Modelscited by 236
2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
2020Multi-task Learning for Multilingual Neural Machine Translationcited by 46
2020XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoderscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 10; 2024: 24; 2025: 27; 2026: 31.

98 papers.

Sort:

By area:

2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
2024A Survey on Efficient Inference for Large Language Modelscited by 197
2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
2023Soft Merging of Experts with Adaptive Routingcited by 88
2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 69
2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 67
2024Scaling Diffusion Transformers to 16 Billion Parameterscited by 65
2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 52
2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 51
2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Trainingcited by 47

38 more

2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 47
2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 31
2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizercited by 28
2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
2024HyperMoE: Towards Better Mixture of Experts via Transferring Among Expertscited by 25
2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competitioncited by 23
2024Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Taskscited by 22
2022Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translationcited by 21
2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
2024LocMoE: A Low-Overhead MoE for Large Language Model Trainingcited by 18
2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Modelscited by 17
2022ADMoE: Anomaly Detection with Mixture-of-Experts from Noisy Labelscited by 15
2022MoEC: Mixture of Expert Clusterscited by 14
2024SEUF: Is Unlearning One Expert Enough for Mixture-of-Experts LLMs?cited by 14
2025Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMscited by 12
2025eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inferencecited by 11
2024Decomposing the Neurons: Activation Sparsity via Mixture of Experts for Continual Test Time Adaptationcited by 10
2023Subnetwork-to-go: Elastic Neural Network with Dynamic Training and Customizable Inferencecited by 9
2024MomentumSMoE: Integrating Momentum into Sparse Mixture of Expertscited by 9
2022HMOE: Hypernetwork-based Mixture of Experts for Domain Generalizationcited by 8
2024ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRAcited by 8
2024Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruningcited by 8
2025Tight Clusters Make Specialized Expertscited by 8
2026When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Modelscited by 8
2025Mixture-of-Experts Meets In-Context Reinforcement Learningcited by 7
2023Adapter-based Selective Knowledge Distillation for Federated Multi-domain Meeting Summarizationcited by 6
2023Adaptive Computation Modules: Granular Conditional Computation For Efficient Inferencecited by 6
2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
2025Towards Stable and Effective Reinforcement Learning for Mixture-of-Expertscited by 6
2025ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specializationcited by 6
2023Mixture of Tokens: Continuous MoE through Cross-Example Aggregationcited by 5
2025Route Experts by Sequence, not by Tokencited by 5
2026Grouter: Decoupling Routing from Representation for Accelerated MoE Trainingcited by 5
2024On the Role of Discrete Representation in Sparse Mixture of Expertscited by 4
2025MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognitioncited by 4
2025CAMEx: Curvature-aware Merging of Expertscited by 4
2025Improving Routing in Sparse Mixture of Experts with Graph of Tokenscited by 4

50 most cited of 98 All 98 (JSON)

Similar, not cited

Close in wording, no citation either way