# Investigating the potential of Sparse Mixtures-of-Experts for multi-domain neural machine translation

arXiv: https://arxiv.org/abs/2407.01126 · HTML: https://mlsanity.com/paper/2407.01126/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in MoE large language models, one of 12 directions in Mixture-of-experts.
- Builds on 20 of its 29 references that are in the map.
- First posted on arXiv 1 Jul 2024.

## Placed in

- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [Simple, Scalable Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1909.08478/) (2019)
- [Tutel: Adaptive Mixture-of-Experts at Scale](https://mlsanity.com/paper/2206.03382/) (2022)
- [When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?](https://mlsanity.com/paper/1804.06323/) (2018)
- [Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference](https://mlsanity.com/paper/2110.03742/) (2021)
- [Scalable and Efficient MoE Training for Multitask Multilingual Models](https://mlsanity.com/paper/2109.10465/) (2021)
- [Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners](https://mlsanity.com/paper/2204.07689/) (2022)
- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [Domain Control for Neural Machine Translation](https://mlsanity.com/paper/1612.06140/) (2016)
- [Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model](https://mlsanity.com/paper/2212.09811/) (2022)
- [Learning a Multi-Domain Curriculum for Neural Machine Translation](https://mlsanity.com/paper/1908.10940/) (2019)
- [Multi-Domain Neural Machine Translation with Word-Level Adaptive Layer-wise Domain Mixing](https://mlsanity.com/paper/1911.02692/) (2019)
- [A Comprehensive Survey of Multilingual Neural Machine Translation](https://mlsanity.com/paper/2001.01115/) (2020)
- [Machine Translation of Restaurant Reviews: New Corpus for Domain Adaptation and Robustness](https://mlsanity.com/paper/1910.14589/) (2019)
- [Efficient Inference for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2109.06679/) (2021)
- [Efficient Inference For Neural Machine Translation](https://mlsanity.com/paper/2010.02416/) (2020)

## Built on by (most cited first, 1 of 1)

- [Point-MoE: Large-Scale Multi-Dataset Training with Mixture-of-Experts for 3D Semantic Segmentation](https://mlsanity.com/paper/2505.23926/) (2025)

## Abstract (arXiv)

We focus on multi-domain Neural Machine Translation, with the goal of developing efficient models which can handle data from various domains seen during training and are robust to domains unseen during training. We hypothesize that Sparse Mixture-of-Experts (SMoE) models are a good fit for this task, as they enable efficient model scaling, which helps to accommodate a variety of multi-domain data, and allow flexible sharing of parameters between domains, potentially enabling knowledge transfer between similar domains and limiting negative transfer. We conduct a series of experiments aimed at validating the utility of SMoE for the multi-domain scenario, and find that a straightforward width scaling of Transformer is a simpler and surprisingly more efficient approach in practice, and reaches the same performance level as SMoE. We also search for a better recipe for robustness of multi-domain systems, highlighting the importance of mixing-in a generic domain, i.e. Paracrawl, and introducing a simple technique, domain randomization.