mlsanityData to 7 Oct 2026

Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation

arXiv 2212.07571 ↗First posted 15 Dec 2022cs.CL cs.AIcited by 8

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Sparsely gated Mixture of Experts (MoE) models have been shown to be a compute-efficient method to scale model capacity for multilingual machine translation. However, for low-resource tasks, MoE models severely over-fit. We show effective regularization strategies, namely dropout techniques for MoE layers in EOM and FOM, Conditional MoE Routing and Curriculum Learning methods that prevent over-fitting and improve the performance of MoE models on low-resource tasks without adversely affecting high-resource tasks. On a massively multilingual machine translation benchmark, our strategies result in about +1 chrF++ improvement in very low resource language pairs. We perform an extensive analysis of the learned MoE routing to better understand the impact of our regularization methods and how we can improve them.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 42 of its 60 references that are in the map. First posted on arXiv 15 Dec 2022.

Builds on

42 of 60 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022MoEC: Mixture of Expert Clusters19 Jul
  2. 2022Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers28 May
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  5. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models22 Sep
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2016Layer Normalizationcited by 5,579
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  9. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  12. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
23 more
  1. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  2. 2020On Layer Normalization in the Transformer Architecturecited by 906
  3. 2017Learning multiple visual domains with residual adapterscited by 553
  4. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  5. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  6. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  7. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  8. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  9. 2021Hash Layers For Large Sparse Modelscited by 236
  10. 2019Competence-based Curriculum Learning for Neural Machine Translationcited by 201
  11. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  12. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  13. 2020TICO-19: the Translation Initiative for Covid-19cited by 67
  14. 2017Curriculum Learning and Minibatch Bucketing in Neural Machine Translationcited by 63
  15. 2018An Empirical Exploration of Curriculum Learning for Neural Machine Translationcited by 58
  16. 2022Bitext Mining Using Distilled Sentence Representations for Low-Resource Languagescited by 48
  17. 2021The Effect of Domain and Diacritics in Yor`ub'a-English Neural Machine Translationcited by 38
  18. 2019Reinforcement Learning based Curriculum Optimization for Neural Machine Translationcited by 29
  19. 2021A Large-Scale Study of Machine Translation in the Turkic Languagescited by 21
  20. 2015Dynamic Capacity Networkscited by 16
  21. 2022Multilingual Machine Translation with Hyper-Adapterscited by 15
  22. 2021English-Twi Parallel Corpus for Machine Translationcited by 7
  23. 2018Dynamic Sentence Sampling for Efficient Training of Neural Machine Translationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 2; 2026: 1.

8 papers.

  1. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  2. 2023SiRA: Sparse Mixture of Low Rank Adaptationcited by 32
  3. 2023Task-Based MoE for Multitask Multilingual Machine Translationcited by 5
  4. 2024Investigating Neural Machine Translation for Low-Resource Languages: Using Bavarian as a Case Studycited by 5
  5. 2025THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translationcited by 2
  6. 2024A Unified Framework for Iris Anti-Spoofing: Introducing Iris Anti-Spoofing Cross-Domain-Testing Protocol and Masked-MoE Methodcited by 0
  7. 2025Exploring Performance Variations in Finetuned Translators of Ultra-Low Resource Languages: Do Linguistic Differences Matter?cited by 0
  8. 2026Disentangling Curriculum Learning in NLP: Towards a Unifying Taxonomycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs
  2. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference
  3. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production
  4. 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
  5. 2026Extracting Small Translation Specialists from LLMs by Aggressively Pruning Experts
  6. 2025Faster MoE LLM Inference for Extremely Large Models
  7. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness
  8. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights