mlsanityData to 7 Oct 2026

Direct Neural Machine Translation with Task-level Mixture of Experts models

arXiv 2310.12236 ↗First posted 18 Oct 2023cs.CLcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Direct neural machine translation (direct NMT) is a type of NMT system that translates text between two non-English languages. Direct NMT systems often face limitations due to the scarcity of parallel data between non-English language pairs. Several approaches have been proposed to address this limitation, such as multilingual NMT and pivot NMT (translation between two languages via English). Task-level Mixture of expert models (Task-level MoE), an inference-efficient variation of Transformer-based models, has shown promising NMT performance for a large number of language pairs. In Task-level MoE, different language groups can use different routing strategies to optimize cross-lingual learning and inference speed. In this work, we examine Task-level MoE's applicability in direct NMT and propose a series of high-performing training and evaluation configurations, through which Task-level MoE-based direct NMT systems outperform bilingual and pivot-based models for a large number of low and high-resource direct pairs, and translation directions. Our Task-level MoE with 16 experts outperforms bilingual NMT, Pivot NMT models for 7 language pairs, while pivot-based models still performed better in 9 pairs and directions.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 46 of its 61 references that are in the map. First posted on arXiv 18 Oct 2023.

Builds on

46 of 61 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT24 May
  2. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  4. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference24 Sep
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  8. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  9. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  10. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  11. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  12. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
28 more
  1. 2019Massively Multilingual Neural Machine Translationcited by 246
  2. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  3. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  4. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  5. 2018Achieving Human Parity on Automatic Chinese to English News Translationcited by 119
  6. 2016Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanismcited by 117
  7. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  8. 2019Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlationscited by 61
  9. 2019The Missing Ingredient in Zero-Shot Neural Machine Translationcited by 59
  10. 2020Scalable Transfer Learning with Expert Modelscited by 48
  11. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Expertscited by 48
  12. 2016Zero-Resource Translation with Multi-Lingual Neural Machine Translationcited by 46
  13. 2020Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Expertscited by 43
  14. 2016Multi-Source Neural Translationcited by 37
  15. 2019Pivot-based Transfer Learning for Neural Machine Translation between Non-English Languagescited by 36
  16. 2020Improving Zero-Shot Translation by Disentangling Positional Informationcited by 35
  17. 2021Improving Multilingual Translation by Representation and Gradient Regularizationcited by 31
  18. 2020Complete Multilingual Neural Machine Translationcited by 30
  19. 2018A neural interlingua for multilingual machine translationcited by 29
  20. 2021SpeechMoE2: Mixture-of-Experts Model with Improved Routingcited by 26
  21. 2016Neural Machine Translation with Pivot Languagescited by 25
  22. 2019Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translationcited by 23
  23. 2017A Teacher-Student Framework for Zero-Resource Neural Machine Translationcited by 22
  24. 2019Consistency by Agreement in Zero-shot Neural Machine Translationcited by 22
  25. 2021Rethinking Zero-shot Neural Machine Translation: From a Perspective of Latent Variablescited by 14
  26. 2020Subword Segmentation and a Single Bridge Language Affect Zero-Shot Neural Machine Translationcited by 13
  27. 2018Emerging Language Spaces Learned From Massively Multilingual Corporacited by 9
  28. 2018Zero-Resource Neural Machine Translation with Multi-Agent Communication Gamecited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Task-Based MoE for Multitask Multilingual Machine Translation
  2. 2025THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translation
  3. 2025Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation
  4. 2022AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation
  5. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model
  6. 2026Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs
  7. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models
  8. 2020Learning to Multi-Task Learn for Better Neural Machine Translation