# Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation

arXiv: https://arxiv.org/abs/2005.04816 · HTML: https://mlsanity.com/paper/2005.04816/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2020.

## Built on by (most cited first, 30 of 51)

- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [Contrastive Learning for Many-to-many Multilingual Neural Machine Translation](https://mlsanity.com/paper/2105.09501/) (2021)
- [Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference](https://mlsanity.com/paper/2110.03742/) (2021)
- [A Benchmark for Learning to Translate a New Language from One Grammar Book](https://mlsanity.com/paper/2309.16575/) (2023)
- [It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning](https://mlsanity.com/paper/2106.12066/) (2021)
- [Building Machine Translation Systems for the Next Thousand Languages](https://mlsanity.com/paper/2205.03983/) (2022)
- [Data Scaling Laws in NMT: The Effect of Noise and Architecture](https://mlsanity.com/paper/2202.01994/) (2022)
- [Learning Language Specific Sub-network for Multilingual Machine Translation](https://mlsanity.com/paper/2105.09259/) (2021)
- [MURAL: Multimodal, Multitask Retrieval Across Languages](https://mlsanity.com/paper/2109.05125/) (2021)
- [Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models](https://mlsanity.com/paper/2103.08849/) (2021)
- [MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages](https://mlsanity.com/paper/2203.08388/) (2022)
- [Counter-Interference Adapter for Multilingual Machine Translation](https://mlsanity.com/paper/2104.08154/) (2021)
- [Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution](https://mlsanity.com/paper/2103.06799/) (2021)
- [Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2105.03953/) (2021)
- [SeaD: End-to-end Text-to-SQL Generation with Schema-aware Denoising](https://mlsanity.com/paper/2105.07911/) (2021)
- [English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddings](https://mlsanity.com/paper/2211.06127/) (2022)
- [Causes and Cures for Interference in Multilingual Translation](https://mlsanity.com/paper/2212.07530/) (2022)
- [Towards the Next 1000 Languages in Multilingual Machine Translation: Exploring the Synergy Between Supervised and Self-Supervised Learning](https://mlsanity.com/paper/2201.03110/) (2022)
- [Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.08570/) (2021)
- [Adapting High-resource NMT Models to Translate Low-resource Related Languages without Parallel Data](https://mlsanity.com/paper/2105.15071/) (2021)
- [Self-Training Sampling with Monolingual Data Uncertainty for Neural Machine Translation](https://mlsanity.com/paper/2106.00941/) (2021)
- [A Survey on Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2107.04239/) (2021)
- [Tencent's Multilingual Machine Translation System for WMT22 Large-Scale African Languages](https://mlsanity.com/paper/2210.09644/) (2022)
- [The Zeno's Paradox of `Low-Resource' Languages](https://mlsanity.com/paper/2410.20817/) (2024)
- [MMTAfrica: Multilingual Machine Translation for African Languages](https://mlsanity.com/paper/2204.04306/) (2022)
- [Multilingual Translation via Grafting Pre-trained Language Models](https://mlsanity.com/paper/2109.05256/) (2021)
- [Assessing Reference-Free Peer Evaluation for Machine Translation](https://mlsanity.com/paper/2104.05146/) (2021)
- [BLISS: Robust Sequence-to-Sequence Learning via Self-Supervised Input Representation](https://mlsanity.com/paper/2204.07837/) (2022)
- [Bilex Rx: Lexical Data Augmentation for Massively Multilingual Machine Translation](https://mlsanity.com/paper/2303.15265/) (2023)
- [Towards Universality in Multilingual Text Rewriting](https://mlsanity.com/paper/2107.14749/) (2021)

## Abstract (arXiv)

Over the last few years two promising research directions in low-resource neural machine translation (NMT) have emerged. The first focuses on utilizing high-resource languages to improve the quality of low-resource languages via multilingual NMT. The second direction employs monolingual data with self-supervision to pre-train translation models, followed by fine-tuning on small amounts of supervised data. In this work, we join these two lines of research and demonstrate the efficacy of monolingual data with self-supervision in multilingual NMT. We offer three major results: (i) Using monolingual data significantly boosts the translation quality of low-resource languages in multilingual models. (ii) Self-supervision improves zero-shot translation quality in multilingual models. (iii) Leveraging monolingual data with self-supervision provides a viable path towards adding new languages to multilingual models, getting up to 33 BLEU on ro-en translation without any parallel data or back-translation.