mlsanityData to 7 Oct 2026

M6-T: Exploring Sparse Expert Models and Beyond

arXiv 2105.15082 ↗First posted 31 May 2021cs.LG cs.CLcited by 22

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) models can achieve promising results with outrageous large amount of parameters but constant computation cost, and thus it has become a trend in model scaling. Still it is a mystery how MoE layers bring quality gains by leveraging the parameters with sparse activation. In this work, we investigate several key factors in sparse expert models. We observe that load imbalance may not be a significant problem affecting model quality, contrary to the perspectives of recent studies, while the number of sparsely activated experts k and expert capacity C in top-k routing can significantly make a difference in this context. Furthermore, we take a step forward to propose a simple method called expert prototyping that splits experts into different prototypes and applies k top-1 routing. This strategy improves the model quality but maintains constant computational costs, and our further exploration on extremely large-scale models reflects that it is more effective in training larger models. We push the model scale to over 1 trillion parameters and implement it on solely 480 NVIDIA V100-32GB GPUs, in comparison with the recent SOTAs on 2048 TPU cores. The proposed giant model achieves substantial speedup in convergence over the same-size baseline.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 30 of its 40 references that are in the map. First posted on arXiv 31 May 2021.

Builds on

30 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021FastMoE: A Fast Mixture-of-Expert Training System24 Mar

Other references · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  12. 2018Deep contextualized word representationscited by 2,285
17 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  4. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  5. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  6. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  7. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  8. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  9. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  10. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  11. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  12. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  13. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  14. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  15. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  16. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  17. 2021M6: A Chinese Multimodal Pretrainercited by 91

Built on by

Cited by year

Citing papers per year: 2021: 6; 2022: 9; 2023: 2; 2024: 4; 2025: 1.

22 papers.

Sort:

By area:

  1. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  2. 2021A Survey of Transformerscited by 399
  3. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  4. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  5. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  6. 2022Chinese CLIP: Contrastive Vision-Language Pretraining in Chinesecited by 108
  7. 2023Modular Deep Learningcited by 82
  8. 2021Go Wider Instead of Deepercited by 56
  9. 2021Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AIcited by 38
  10. 2022MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptationcited by 37
  11. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training Systemcited by 36
  12. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 36
10 more
  1. 2022Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Modelscited by 31
  2. 2024XMoE: Sparse Models with Fine-grained and Adaptive Expert Selectioncited by 27
  3. 2022Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformerscited by 22
  4. 2021Sparse MoEs meet Efficient Ensemblescited by 19
  5. 2025eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inferencecited by 11
  6. 2021Unbiased Gradient Estimation with Balanced Assignments for Mixtures of Expertscited by 10
  7. 2022Automatic Expert Selection for Multi-Scenario and Multi-Task Searchcited by 9
  8. 2022Efficient Language Modeling with Sparse all-MLPcited by 8
  9. 2024LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashingcited by 5
  10. 2024Routing in Sparsely-gated Language Models responds to Contextcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing
  2. 2025MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Experts
  3. 2026WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware
  4. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts
  5. 2024Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation
  6. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
  7. 2026Scalable Training of Mixture-of-Experts Models with Megatron Core
  8. 2024Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules