mlsanityData to 7 Oct 2026

ST-MoE: Designing Stable and Transferable Sparse Expert Models

arXiv 2202.08906 ↗First posted 17 Feb 2022cs.CL cs.LGcited by 391

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Scale has opened new frontiers in natural language processing -- but at a high cost. In response, Mixture-of-Experts (MoE) and Switch Transformers have been proposed as an energy efficient path to even larger and more capable language models. But advancing the state-of-the-art across a broad set of natural language tasks has been hindered by training instabilities and uncertain quality during fine-tuning. Our work focuses on these issues and acts as a design guide. We conclude by scaling a sparse model to 269B parameters, with a computational cost comparable to a 32B dense encoder-decoder Transformer (Stable and Transferable Mixture-of-Experts or ST-MoE-32B). For the first time, a sparse model achieves state-of-the-art performance in transfer learning, across a diverse set of tasks including reasoning (SuperGLUE, ARC Easy, ARC Challenge), summarization (XSum, CNN-DM), closed book question answering (WebQA, Natural Questions), and adversarially constructed tasks (Winogrande, ANLI R3).

Key facts. Cited by 391 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 68 of its 91 references that are in the map. First posted on arXiv 17 Feb 2022.

Builds on

68 of 91 references.

Closest ancestors · 6

Same direction, newest first

  1. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  2. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  3. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference24 Sep
  4. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models22 Sep
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2016Layer Normalizationcited by 5,579
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
38 more
  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  9. 2020GLU Variants Improve Transformercited by 1,724
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  12. 2019Root Mean Square Layer Normalizationcited by 1,425
  13. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  14. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  15. 2017Mixed Precision Trainingcited by 1,135
  16. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  17. 2015Teaching Machines to Read and Comprehendcited by 1,039
  18. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  19. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  20. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  21. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  22. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  23. 2016Language Modeling with Gated Convolutional Networkscited by 688
  24. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  25. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  26. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  27. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  28. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  29. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  30. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  31. 2021Hash Layers For Large Sparse Modelscited by 236
  32. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  33. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  34. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  35. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  36. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  37. 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 144
  38. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 32; 2024: 101; 2025: 111; 2026: 137.

391 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  4. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  5. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  6. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  7. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  8. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  9. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  10. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  11. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  12. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
38 more
  1. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  2. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  3. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  4. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  5. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 171
  6. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  7. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  8. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  9. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 136
  10. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  11. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  12. 2024Higher Layers Need More LoRA Expertscited by 101
  13. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  14. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  15. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  16. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 88
  17. 2025ForceVLA: Enhancing VLA Models with a Force-aware MoE for Contact-rich Manipulationcited by 87
  18. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  19. 2023Modular Deep Learningcited by 82
  20. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  21. 2023LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Modelscited by 71
  22. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Modelscited by 71
  23. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
  24. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  25. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  26. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  27. 2022Transcending Scaling Laws with 0.1% Extra Computecited by 55
  28. 2025LongCat-Flash Technical Reportcited by 54
  29. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 54
  30. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 53
  31. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Modelscited by 53
  32. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 52
  33. 2024Upcycling Large Language Models into Mixture of Expertscited by 48
  34. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 48
  35. 2023RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Modelcited by 47
  36. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 47
  37. 2025Scaling Laws for Native Multimodal Modelscited by 47
  38. 2024ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understandingcited by 44

50 most cited of 391 All 391 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficiently Editing Mixture-of-Experts Models with Compressed Experts
  2. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  3. 2026Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints
  4. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  5. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models
  6. 2024U2++ MoE: Scaling 4.7x parameters with minimal impact on RTF
  7. 2024XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection
  8. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition