mlsanityData to 14 Oct 2026

Task-Based MoE for Multitask Multilingual Machine Translation

arXiv 2308.15772 ↗First posted 30 Aug 2023cs.CLcited by 5

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-experts (MoE) architecture has been proven a powerful method for diverse tasks in training deep models in many applications. However, current MoE implementations are task agnostic, treating all tokens from different tasks in the same manner. In this work, we instead design a novel method that incorporates task information into MoE models at different granular levels with shared dynamic task-based adapters. Our experiments and analysis show the advantages of our approaches over the dense and canonical MoE models on multi-task multilingual machine translations. With task-specific adapters, our models can additionally generalize to new tasks efficiently.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 20 of its 27 references that are in the map. First posted on arXiv 30 Aug 2023.

Builds on

20 of 27 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation15 Dec
  2. 2022Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers28 May
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  5. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models22 Sep
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 13

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  4. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  5. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  6. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  7. 2021Hash Layers For Large Sparse Modelscited by 237
  8. 2022A Review of Sparse Expert Models in Deep Learningcited by 148
  9. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  10. 2020Understanding and Improving Information Transfer in Multi-Task Learningcited by 114
  11. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 113
  12. 2022Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learnerscited by 15
1 more
  1. 2023TA-MoE: Topology-Aware Large Scale Mixture-of-Expert Trainingcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 1.

5 papers.

  1. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 303
  2. 2024Multi-Head Mixture-of-Expertscited by 31
  3. 2024MH-MoE: Multi-Head Mixture-of-Expertscited by 4
  4. 2025THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translationcited by 2
  5. 2026LM-Lexicon: Improving Definition Modeling via Harmonizing Semantic Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference
  2. 2025Mixture of Experts in Large Language Models
  3. 2026Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs
  4. 2023Direct Neural Machine Translation with Task-level Mixture of Experts models
  5. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production
  6. 2021Efficient Large Scale Language Modeling with Mixtures of Experts
  7. 2024A Survey on Mixture of Experts in Large Language Models
  8. 2026Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models