# nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?

arXiv: https://arxiv.org/abs/2106.02171 · HTML: https://mlsanity.com/paper/2106.02171/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 3 Jun 2021.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark](https://mlsanity.com/paper/2008.09335/) (2020)
- [WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization](https://mlsanity.com/paper/2010.03093/) (2020)
- [ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora](https://mlsanity.com/paper/2012.15674/) (2020)
- [Explicit Alignment Objectives for Multilingual Bidirectional Encoders](https://mlsanity.com/paper/2010.07972/) (2020)
- [FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding](https://mlsanity.com/paper/2009.05166/) (2020)
- [Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1909.00437/) (2019)

## Built on by (most cited first, 21 of 21)

- [mSLAM: Massively multilingual joint pre-training for speech and text](https://mlsanity.com/paper/2202.01374/) (2022)
- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [Understanding Cross-Lingual Alignment -- A Survey](https://mlsanity.com/paper/2404.06228/) (2024)
- [Enhancing Cross-lingual Transfer by Manifold Mixup](https://mlsanity.com/paper/2205.04182/) (2022)
- [PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining](https://mlsanity.com/paper/2108.01887/) (2021)
- [An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)](https://mlsanity.com/paper/2503.10267/) (2025)
- [Reasoning Beyond Limits: Advances and Open Problems for LLMs](https://mlsanity.com/paper/2503.22732/) (2025)
- [On the Role of Parallel Data in Cross-lingual Transfer Learning](https://mlsanity.com/paper/2212.10173/) (2022)
- [Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resources](https://mlsanity.com/paper/2504.04152/) (2025)
- [DOCmT5: Document-Level Pretraining of Multilingual Language Models](https://mlsanity.com/paper/2112.08709/) (2021)
- [Cross-Lingual Supervision improves Large Language Models Pre-training](https://mlsanity.com/paper/2305.11778/) (2023)
- [Enhancing Translation Accuracy of Large Language Models through Continual Pre-Training on Parallel Data](https://mlsanity.com/paper/2407.03145/) (2024)
- [Can Machine Translation Bridge Multilingual Pretraining and Cross-lingual Transfer Learning?](https://mlsanity.com/paper/2403.16777/) (2024)
- [The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation](https://mlsanity.com/paper/2305.09652/) (2023)
- [Advancing Multilingual Pre-training: TRIP Triangular Document-level Pre-training for Multilingual Language Models](https://mlsanity.com/paper/2212.07752/) (2022)
- [Exploiting Domain-Specific Parallel Data on Multilingual Language Models for Low-resource Language Translation](https://mlsanity.com/paper/2412.19522/) (2024)
- [LEIA: Facilitating Cross-lingual Knowledge Transfer in Language Models with Entity-based Data Augmentation](https://mlsanity.com/paper/2402.11485/) (2024)
- [A Comparison of Language Modeling and Translation as Multilingual Pretraining Objectives](https://mlsanity.com/paper/2407.15489/) (2024)
- [Cross-Lingual Language Model Meta-Pretraining](https://mlsanity.com/paper/2109.11129/) (2021)
- [Smart Bilingual Focused Crawling of Parallel Documents](https://mlsanity.com/paper/2405.14779/) (2024)
- [On the limited utility of parallel data for learning shared multilingual representations](https://mlsanity.com/paper/2603.29026/) (2026)

## Abstract (arXiv)

Recently, mT5 - a massively multilingual version of T5 - leveraged a unified text-to-text format to attain state-of-the-art results on a wide variety of multilingual NLP tasks. In this paper, we investigate the impact of incorporating parallel data into mT5 pre-training. We find that multi-tasking language modeling with objectives such as machine translation during pre-training is a straightforward way to improve performance on downstream multilingual and cross-lingual tasks. However, the gains start to diminish as the model capacity increases, suggesting that parallel data might not be as essential for larger models. At the same time, even at larger model sizes, we find that pre-training with parallel data still provides benefits in the limited labelled data regime.