mlsanityData to 7 Oct 2026

Taming Sparsely Activated Transformer with Stochastic Experts

arXiv 2110.04260 ↗First posted 8 Oct 2021cs.CL cs.LGcited by 112

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Sparsely activated models (SAMs), such as Mixture-of-Experts (MoE), can easily scale to have outrageously large amounts of parameters without significant increase in computational cost. However, SAMs are reported to be parameter inefficient such that larger models do not always lead to better performance. While most on-going research focuses on improving SAMs models by exploring methods of routing inputs to experts, our analysis reveals that such research might not lead to the solution we expect, i.e., the commonly-used routing methods based on gating mechanisms do not work better than randomly routing inputs to experts. In this paper, we propose a new expert-based model, THOR (Transformer witH StOchastic ExpeRts). Unlike classic expert-based models, such as the Switch Transformer, experts in THOR are randomly activated for each input during training and inference. THOR models are trained using a consistency regularized loss, where experts learn not only from training data but also from other experts as teachers, such that all the experts make consistent predictions. We validate the effectiveness of THOR on machine translation tasks. Results show that THOR models are more parameter efficient in that they significantly outperform the Transformer and MoE models across various settings. For example, in multilingual translation, THOR outperforms the Switch Transformer by 2 BLEU scores, and obtains the same BLEU score as that of a state-of-the-art MoE model that is 18 times larger. Our code is publicly available at: https://github.com/microsoft/Stochastic-Mixture-of-Experts.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 34 of its 36 references that are in the map. First posted on arXiv 8 Oct 2021.

Builds on

34 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
21 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  2. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  4. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  5. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  6. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  7. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  8. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  9. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  10. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  11. 2021Hash Layers For Large Sparse Modelscited by 236
  12. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  13. 2020Understanding the Difficulty of Training Transformerscited by 210
  14. 2018Scaling Neural Machine Translationcited by 204
  15. 2019The Evolved Transformercited by 180
  16. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 156
  17. 2019Mixture Models for Diverse Machine Translation: Tricks of the Tradecited by 77
  18. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  19. 2020Very Deep Transformers for Neural Machine Translationcited by 50
  20. 2021M6-T: Exploring Sparse Expert Models and Beyondcited by 22
  21. 2019Depth Growing for Neural Machine Translationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 19; 2023: 26; 2024: 34; 2025: 22; 2026: 11.

112 papers.

Sort:

By area:

  1. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  2. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  4. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  5. 2023Efficient Large Language Models: A Surveycited by 183
  6. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  7. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  8. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  9. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  10. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 101
  11. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  12. 2023Modular Deep Learningcited by 82
38 more
  1. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Modelscited by 71
  2. 2024MVMoE: Multi-Task Vehicle Routing Solver with Mixture-of-Expertscited by 69
  3. 2024Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Modelcited by 67
  4. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
  5. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  6. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  7. 2024MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Modelscited by 54
  8. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 53
  9. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
  10. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Trainingcited by 47
  11. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 47
  12. 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Expertscited by 46
  13. 2022MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptationcited by 37
  14. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Servicescited by 37
  15. 2022Is a Modular Architecture Enough?cited by 35
  16. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  17. 2023Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memoriescited by 34
  18. 2022Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Modelscited by 31
  19. 2023Brainformers: Trading Simplicity for Efficiencycited by 31
  20. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizercited by 28
  21. 2023Enhancing NeRF akin to Enhancing LLMs: Generalizable NeRF Transformer with Mixture-of-View-Expertscited by 27
  22. 2023Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Expertscited by 26
  23. 2024BECoTTA: Input-dependent Online Blending of Experts for Continual Test-time Adaptationcited by 25
  24. 2022A Survey on Dynamic Neural Networks for Natural Language Processingcited by 24
  25. 2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competitioncited by 23
  26. 2022Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformerscited by 22
  27. 2024Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlappingcited by 22
  28. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  29. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
  30. 2022METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signalscited by 20
  31. 2023Chain-of-Skills: A Configurable Model for Open-domain Question Answeringcited by 19
  32. 2024LocMoE: A Low-Overhead MoE for Large Language Model Trainingcited by 18
  33. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Modelscited by 17
  34. 2023Sparse Backpropagation for MoE Trainingcited by 16
  35. 2023Adaptive Gating in Mixture-of-Experts based Language Modelscited by 16
  36. 2024Ada-K Routing: Boosting the Efficiency of MoE-based LLMscited by 16
  37. 2022ADMoE: Anomaly Detection with Mixture-of-Experts from Noisy Labelscited by 15
  38. 2023Image Super-resolution Via Latent Diffusion: A Sampling-space Mixture Of Experts And Frequency-augmented Decoder Approachcited by 13

50 most cited of 112 All 112 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection
  2. 2022StableMoE: Stable Routing Strategy for Mixture of Experts
  3. 2024Mixture of A Million Experts
  4. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models
  5. 2022Efficient Sparsely Activated Transformers
  6. 2025ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restoration
  7. 2024Improving the Downstream Performance of Mixture-of-Experts Transformers via Weak Vanilla Transformers
  8. 2025Union of Experts: Adapting Hierarchical Routing to Equivalently Decomposed Transformer