mlsanityData to 14 Oct 2026

PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining

arXiv 2108.01887 ↗First posted 4 Aug 2021cs.CL cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Despite the success of multilingual sequence-to-sequence pretraining, most existing approaches rely on monolingual corpora, and do not make use of the strong cross-lingual signal contained in parallel data. In this paper, we present PARADISE (PARAllel & Denoising Integration in SEquence-to-sequence models), which extends the conventional denoising objective used to train these models by (i) replacing words in the noised sequence according to a multilingual dictionary, and (ii) predicting the reference translation according to a parallel corpus instead of recovering the original sequence. Our experiments on machine translation and cross-lingual natural language inference show an average improvement of 2.0 BLEU points and 6.7 accuracy points from integrating parallel data into pretraining, respectively, obtaining results that are competitive with several popular models at a fraction of their computational cost.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 30 of its 48 references that are in the map. First posted on arXiv 4 Aug 2021.

Builds on

30 of 48 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  8. 2017Mixed Precision Trainingcited by 1,156
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  10. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  11. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  12. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
18 more
  1. 2019How multilingual is Multilingual BERT?cited by 645
  2. 2017Word Translation Without Parallel Datacited by 487
  3. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 347
  4. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  5. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  6. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 181
  7. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  8. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  9. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  10. 2016Edinburgh Neural Machine Translation Systems for WMT 16cited by 76
  11. 2017The IIT Bombay English-Hindi Parallel Corpuscited by 75
  12. 2020Explicit Alignment Objectives for Multilingual Bidirectional Encoderscited by 50
  13. 2021MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairscited by 41
  14. 2020A Call for More Rigor in Unsupervised Cross-lingual Learningcited by 37
  15. 2019Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translationcited by 27
  16. 2021nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?cited by 21
  17. 2020DICT-MLM: Improved Multilingual Pre-Training using Bilingual Dictionariescited by 20
  18. 2019Polyglot Contextual Representations Improve Crosslingual Transfercited by 16

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 4; 2023: 6; 2024: 6; 2025: 4.

21 papers.

Sort:

  1. 2022Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptationcited by 44
  2. 2024Breaking the Curse of Multilinguality with Cross-lingual Expert Language Modelscited by 38
  3. 2025EuroBERT: Scaling Multilingual Encoders for European Languagescited by 31
  4. 2024Code-Switching Curriculum Learning for Multilingual Transfer in LLMscited by 27
  5. 2021AfroMT: Pretraining Strategies and Reproducible Benchmarks for Translation of 8 African Languagescited by 24
  6. 2023Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarkscited by 14
  7. 2022On the Role of Parallel Data in Cross-lingual Transfer Learningcited by 11
  8. 2024The Role of Language Imbalance in Cross-lingual Generalisation: Insights from Cloned Language Experimentscited by 9
  9. 2023Bilex Rx: Lexical Data Augmentation for Massively Multilingual Machine Translationcited by 8
  10. 2023Cross-Lingual Supervision improves Large Language Models Pre-trainingcited by 8
  11. 2024How Transliterations Improve Crosslingual Alignmentcited by 6
  12. 2023Leveraging Auxiliary Domain Parallel Data in Intermediate Task Fine-tuning for Low-resource Translationcited by 5
9 more
  1. 2023The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translationcited by 4
  2. 2023Code-Switching with Word Senses for Pretraining in Neural Machine Translationcited by 4
  3. 2022Advancing Multilingual Pre-training: TRIP Triangular Document-level Pre-training for Multilingual Language Modelscited by 2
  4. 2024Exploiting Domain-Specific Parallel Data on Multilingual Language Models for Low-resource Language Translationcited by 2
  5. 2025Beyond Vanilla Fine-Tuning: Leveraging Multistage, Multilingual, and Domain-Specific Methods for Low-Resource Machine Translationcited by 2
  6. 2024LEIA: Facilitating Cross-lingual Knowledge Transfer in Language Models with Entity-based Data Augmentationcited by 1
  7. 2022Revamping Multilingual Agreement Bidirectionally via Switched Back-translation for Multilingual Neural Machine Translationcited by 0
  8. 2025TULUN: Transparent and Adaptable Low-resource Machine Translationcited by 0
  9. 2025Pretraining Strategies using Monolingual and Parallel Data for Low-Resource Machine Translationcited by 0