# Multi-task Learning for Multilingual Neural Machine Translation

arXiv: https://arxiv.org/abs/2010.02523 · HTML: https://mlsanity.com/paper/2010.02523/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 30 of 46)

- [On the Representation Collapse of Sparse Mixture of Experts](https://mlsanity.com/paper/2204.09179/) (2022)
- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [StableMoE: Stable Routing Strategy for Mixture of Experts](https://mlsanity.com/paper/2204.08396/) (2022)
- [Survey of Low-Resource Machine Translation](https://mlsanity.com/paper/2109.00486/) (2021)
- [Scalable and Efficient MoE Training for Multitask Multilingual Models](https://mlsanity.com/paper/2109.10465/) (2021)
- [Multi-Task Learning in Natural Language Processing: An Overview](https://mlsanity.com/paper/2109.09138/) (2021)
- [CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation](https://mlsanity.com/paper/2310.04951/) (2023)
- [DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders](https://mlsanity.com/paper/2106.13736/) (2021)
- [MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services](https://mlsanity.com/paper/2205.10034/) (2022)
- [RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios?](https://mlsanity.com/paper/2404.14397/) (2024)
- [Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task](https://mlsanity.com/paper/2111.02086/) (2021)
- [Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness](https://mlsanity.com/paper/2310.02410/) (2023)
- [Improving Multilingual Translation by Representation and Gradient Regularization](https://mlsanity.com/paper/2109.04778/) (2021)
- [Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization](https://mlsanity.com/paper/2208.09770/) (2022)
- [Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers](https://mlsanity.com/paper/2205.14336/) (2022)
- [GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator](https://mlsanity.com/paper/2212.10218/) (2022)
- [Tencent's Multilingual Machine Translation System for WMT22 Large-Scale African Languages](https://mlsanity.com/paper/2210.09644/) (2022)
- [Multilingual Translation via Grafting Pre-trained Language Models](https://mlsanity.com/paper/2109.05256/) (2021)
- [UM4: Unified Multilingual Multiple Teacher-Student Model for Zero-Resource Neural Machine Translation](https://mlsanity.com/paper/2207.04900/) (2022)
- [On the Pareto Front of Multilingual Neural Machine Translation](https://mlsanity.com/paper/2304.03216/) (2023)
- [Order Matters in the Presence of Dataset Imbalance for Multilingual Learning](https://mlsanity.com/paper/2312.06134/) (2023)
- [Polyglot Prompt: Multilingual Multitask PrompTraining](https://mlsanity.com/paper/2204.14264/) (2022)
- [Improving Multilingual Neural Machine Translation System for Indic Languages](https://mlsanity.com/paper/2209.13279/) (2022)
- [Exploring Italian sentence embeddings properties through multi-tasking](https://mlsanity.com/paper/2409.06622/) (2024)
- [Netmarble AI Center's WMT21 Automatic Post-Editing Shared Task Submission](https://mlsanity.com/paper/2109.06515/) (2021)
- [HLT-MT: High-resource Language-specific Training for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2207.04906/) (2022)
- [A Case Study on Context-Aware Neural Machine Translation with Multi-Task Learning](https://mlsanity.com/paper/2407.03076/) (2024)
- [Exploring Unsupervised Pretraining Objectives for Machine Translation](https://mlsanity.com/paper/2106.05634/) (2021)
- [HintedBT: Augmenting Back-Translation with Quality and Transliteration Hints](https://mlsanity.com/paper/2109.04443/) (2021)
- [PoE: a Panel of Experts for Generalized Automatic Dialogue Assessment](https://mlsanity.com/paper/2212.08992/) (2022)

## Abstract (arXiv)

While monolingual data has been shown to be useful in improving bilingual neural machine translation (NMT), effectively and efficiently leveraging monolingual data for Multilingual NMT (MNMT) systems is a less explored area. In this work, we propose a multi-task learning (MTL) framework that jointly trains the model with the translation task on bitext data and two denoising tasks on the monolingual data. We conduct extensive empirical studies on MNMT systems with 10 language pairs from WMT datasets. We show that the proposed approach can effectively improve the translation quality for both high-resource and low-resource languages with large margin, achieving significantly better results than the individual bilingual models. We also demonstrate the efficacy of the proposed approach in the zero-shot setup for language pairs without bitext training data. Furthermore, we show the effectiveness of MTL over pre-training approaches for both NMT and cross-lingual transfer learning NLU tasks; the proposed approach outperforms massive scale models trained on single task.