# Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation

arXiv: https://arxiv.org/abs/2004.11867 · HTML: https://mlsanity.com/paper/2004.11867/

## Key facts

- Cited by 277 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2020.

## Built on by (most cited first, 30 of 277)

- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Prompting Large Language Model for Machine Translation: A Case Study](https://mlsanity.com/paper/2301.07069/) (2023)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [MADLAD-400: A Multilingual And Document-Level Large Audited Dataset](https://mlsanity.com/paper/2309.04662/) (2023)
- [DeepNet: Scaling Transformers to 1,000 Layers](https://mlsanity.com/paper/2203.00555/) (2022)
- [The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset](https://mlsanity.com/paper/2303.03915/) (2023)
- [Hallucinations in Large Multilingual Translation Models](https://mlsanity.com/paper/2303.16104/) (2023)
- [Accelerating Transformer Inference for Translation via Parallel Decoding](https://mlsanity.com/paper/2305.10427/) (2023)
- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [Contrastive Learning for Many-to-many Multilingual Neural Machine Translation](https://mlsanity.com/paper/2105.09501/) (2021)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference](https://mlsanity.com/paper/2110.03742/) (2021)
- [Survey of Low-Resource Machine Translation](https://mlsanity.com/paper/2109.00486/) (2021)
- [Building Machine Translation Systems for the Next Thousand Languages](https://mlsanity.com/paper/2205.03983/) (2022)
- [Large Language Model Agent for Hyper-Parameter Optimization](https://mlsanity.com/paper/2402.01881/) (2024)
- [Cross-modal Contrastive Learning for Speech Translation](https://mlsanity.com/paper/2205.02444/) (2022)
- [BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languages](https://mlsanity.com/paper/2305.18098/) (2023)
- [Adapting Large Language Models for Document-Level Machine Translation](https://mlsanity.com/paper/2401.06468/) (2024)
- [Facebook AI WMT21 News Translation Task Submission](https://mlsanity.com/paper/2108.03265/) (2021)
- [Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructions](https://mlsanity.com/paper/2305.15083/) (2023)
- [Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task](https://mlsanity.com/paper/2107.05782/) (2021)
- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback](https://mlsanity.com/paper/2406.01771/) (2024)
- [An Open Dataset and Model for Language Identification](https://mlsanity.com/paper/2305.13820/) (2023)
- [Learning Shared Semantic Space for Speech-to-Text Translation](https://mlsanity.com/paper/2105.03095/) (2021)
- [Learning Language Specific Sub-network for Multilingual Machine Translation](https://mlsanity.com/paper/2105.09259/) (2021)
- [End-to-end Speech Translation via Cross-modal Progressive Training](https://mlsanity.com/paper/2104.10380/) (2021)
- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [MURAL: Multimodal, Multitask Retrieval Across Languages](https://mlsanity.com/paper/2109.05125/) (2021)
- [Fine-tuning Large Language Models for Domain-specific Machine Translation](https://mlsanity.com/paper/2402.15061/) (2024)

## Abstract (arXiv)

Massively multilingual models for neural machine translation (NMT) are theoretically attractive, but often underperform bilingual models and deliver poor zero-shot translations. In this paper, we explore ways to improve them. We argue that multilingual NMT requires stronger modeling capacity to support language pairs with varying typological characteristics, and overcome this bottleneck via language-specific components and deepening NMT architectures. We identify the off-target translation issue (i.e. translating into a wrong target language) as the major source of the inferior zero-shot performance, and propose random online backtranslation to enforce the translation of unseen training language pairs. Experiments on OPUS-100 (a novel multilingual dataset with 100 languages) show that our approach substantially narrows the performance gap with bilingual models in both one-to-many and many-to-many settings, and improves zero-shot performance by ~10 BLEU, approaching conventional pivot-based methods.