mlsanityData to 7 Oct 2026

Scalable and Efficient MoE Training for Multitask Multilingual Models

arXiv 2109.10465 ↗First posted 22 Sep 2021cs.CL cs.AI cs.LGcited by 76

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

The Mixture of Experts (MoE) models are an emerging class of sparsely activated deep learning models that have sublinear compute costs with respect to their parameters. In contrast with dense models, the sparse architecture of MoE offers opportunities for drastically growing model size with significant accuracy gain while consuming much lower compute budget. However, supporting large scale MoE training also has its own set of system and modeling challenges. To overcome the challenges and embrace the opportunities of MoE, we first develop a system capable of scaling MoE models efficiently to trillions of parameters. It combines multi-dimensional parallelism and heterogeneous memory technologies harmoniously with MoE to empower 8x larger models on the same hardware compared with existing work. Besides boosting system efficiency, we also present new training methods to improve MoE sample efficiency and leverage expert pruning strategy to improve inference time efficiency. By combining the efficient system and training methods, we are able to significantly scale up large multitask multilingual models for language generation which results in a great improvement in model accuracy. A model trained with 10 billion parameters on 50 languages can achieve state-of-the-art performance in Machine Translation (MT) and multilingual natural language generation tasks. The system support of efficient MoE training has been implemented and open-sourced with the DeepSpeed library.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 23 of its 29 references that are in the map. First posted on arXiv 22 Sep 2021.

Builds on

23 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun

Other references · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  9. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  10. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  11. 2020On Layer Normalization in the Transformer Architecturecited by 906
  12. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
10 more
  1. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  2. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  3. 2019Massively Multilingual Neural Machine Translationcited by 246
  4. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  5. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  6. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  7. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 116
  8. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 104
  9. 2020Multi-task Learning for Multilingual Neural Machine Translationcited by 46
  10. 2021ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generationcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 18; 2023: 19; 2024: 21; 2025: 12; 2026: 4.

76 papers.

Sort:

By area:

  1. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  2. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  4. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  5. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  6. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  7. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  8. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  9. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  10. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  11. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  12. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
38 more
  1. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 101
  2. 2021The Efficiency Misnomercited by 98
  3. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  4. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 88
  5. 2023LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Modelscited by 71
  6. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Modelscited by 71
  7. 2024Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Expertscited by 69
  8. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  9. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
  10. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decompositioncited by 57
  11. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Trainingcited by 47
  12. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains Morecited by 38
  13. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Servicescited by 37
  14. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Modelcited by 36
  15. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 36
  16. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustnesscited by 34
  17. 2022Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Modelscited by 31
  18. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruningcited by 30
  19. 2023SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budgetcited by 27
  20. 2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Expertscited by 27
  21. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routingcited by 26
  22. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Productioncited by 24
  23. 2022Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformerscited by 22
  24. 2023FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMscited by 22
  25. 2025Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observationscited by 19
  26. 2025EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Modelscited by 18
  27. 2023Adaptive Gating in Mixture-of-Experts based Language Modelscited by 16
  28. 2024GRIN: GRadient-INformed MoEcited by 16
  29. 2022Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systemscited by 15
  30. 2024T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectivescited by 15
  31. 2024SEUF: Is Unlearning One Expert Enough for Mixture-of-Experts LLMs?cited by 14
  32. 2022Revisiting Checkpoint Averaging for Neural Machine Translationcited by 13
  33. 2024Machine Translation with Large Language Models: Prompt Engineering for Persian, English, and Russian Directionscited by 12
  34. 2024Learning More Generalized Experts by Merging Experts in Mixture-of-Expertscited by 12
  35. 2025Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Modelscited by 10
  36. 2022AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translationcited by 9
  37. 2024MaskMoE: Boosting Token-Level Learning via Routing Mask in Mixture-of-Expertscited by 9
  38. 2022Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translationcited by 8

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixture of Experts in Large Language Models
  2. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  3. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
  4. 2021Efficient Large Scale Language Modeling with Mixtures of Experts
  5. 2024U2++ MoE: Scaling 4.7x parameters with minimal impact on RTF
  6. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models
  7. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  8. 2025Faster MoE LLM Inference for Extremely Large Models