mlsanityData to 7 Oct 2026

nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?

arXiv 2106.02171 ↗First posted 3 Jun 2021cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

Recently, mT5 - a massively multilingual version of T5 - leveraged a unified text-to-text format to attain state-of-the-art results on a wide variety of multilingual NLP tasks. In this paper, we investigate the impact of incorporating parallel data into mT5 pre-training. We find that multi-tasking language modeling with objectives such as machine translation during pre-training is a straightforward way to improve performance on downstream multilingual and cross-lingual tasks. However, the gains start to diminish as the model capacity increases, suggesting that parallel data might not be as essential for larger models. At the same time, even at larger model sizes, we find that pre-training with parallel data still provides benefits in the limited labelled data regime.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 18 of its 24 references that are in the map. First posted on arXiv 3 Jun 2021.

Builds on

18 of 24 references.

References · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  6. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  7. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  8. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  9. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  10. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  11. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  12. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
6 more
  1. 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 156
  2. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  3. 2020ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corporacited by 63
  4. 2020Explicit Alignment Objectives for Multilingual Bidirectional Encoderscited by 50
  5. 2020FILTER: An Enhanced Fusion Method for Cross-lingual Language Understandingcited by 29
  6. 2019Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translationcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 4; 2023: 2; 2024: 8; 2025: 3; 2026: 1.

21 papers.

Sort:

By area:

  1. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  2. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  3. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  4. 2022Enhancing Cross-lingual Transfer by Manifold Mixupcited by 30
  5. 2021PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretrainingcited by 21
  6. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  7. 2025An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)cited by 15
  8. 2022On the Role of Parallel Data in Cross-lingual Transfer Learningcited by 11
  9. 2025Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resourcescited by 10
  10. 2021DOCmT5: Document-Level Pretraining of Multilingual Language Modelscited by 8
  11. 2023Cross-Lingual Supervision improves Large Language Models Pre-trainingcited by 8
  12. 2024Enhancing Translation Accuracy of Large Language Models through Continual Pre-Training on Parallel Datacited by 7
9 more
  1. 2024Can Machine Translation Bridge Multilingual Pretraining and Cross-lingual Transfer Learning?cited by 6
  2. 2023The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translationcited by 4
  3. 2022Advancing Multilingual Pre-training: TRIP Triangular Document-level Pre-training for Multilingual Language Modelscited by 2
  4. 2024Exploiting Domain-Specific Parallel Data on Multilingual Language Models for Low-resource Language Translationcited by 2
  5. 2024LEIA: Facilitating Cross-lingual Knowledge Transfer in Language Models with Entity-based Data Augmentationcited by 1
  6. 2024A Comparison of Language Modeling and Translation as Multilingual Pretraining Objectivescited by 1
  7. 2021Cross-Lingual Language Model Meta-Pretrainingcited by 0
  8. 2024Smart Bilingual Focused Crawling of Parallel Documentscited by 0
  9. 2026On the limited utility of parallel data for learning shared multilingual representationscited by 0