mlsanityData to 7 Oct 2026

On the Role of Parallel Data in Cross-lingual Transfer Learning

arXiv 2212.10173 ↗First posted 20 Dec 2022cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

While prior work has established that the use of parallel data is conducive for cross-lingual learning, it is unclear if the improvements come from the data itself, or if it is the modeling of parallel interactions that matters. Exploring this, we examine the usage of unsupervised machine translation to generate synthetic parallel data, and compare it to supervised machine translation and gold parallel data. We find that even model generated parallel data can be useful for downstream tasks, in both a general setting (continued pretraining) as well as the task-specific setting (translate-train), although our best results are still obtained using real parallel data. Our findings suggest that existing multilingual models do not exploit the full potential of monolingual data, and prompt the community to reconsider the traditional categorization of cross-lingual learning approaches.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 17 of its 21 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

17 of 21 references.

References · 17

  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  2. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  3. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  4. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  5. 2019How multilingual is Multilingual BERT?cited by 642
  6. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  7. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  8. 2017Word Translation Without Parallel Datacited by 480
  9. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  10. 2017Unsupervised Neural Machine Translationcited by 148
  11. 2021Pre-training without Natural Imagescited by 100
  12. 2019A critical analysis of self-supervision, or what we can learn from a single imagecited by 75
5 more
  1. 2021Does Pretraining for Summarization Require Knowledge Transfer?cited by 30
  2. 2021nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?cited by 21
  3. 2021PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretrainingcited by 21
  4. 2022Downstream Datasets Make Surprisingly Good Pretraining Corporacited by 20
  5. 2020Pre-Training a Language Model Without Human Languagecited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 2; 2026: 1.

11 papers.

  1. 2023Do Multilingual Language Models Think Better in English?cited by 87
  2. 2023Lost in Translation: Large Language Models in Non-English Content Analysiscited by 34
  3. 2025EuroBERT: Scaling Multilingual Encoders for European Languagescited by 28
  4. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 27
  5. 2024INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languagescited by 24
  6. 2024Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledgecited by 14
  7. 2024Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative taskscited by 11
  8. 2024CORI: CJKV Benchmark with Romanization Integration -- A step towards Cross-lingual Transfer Beyond Textual Scriptscited by 4
  9. 2025SynthDetoxM: Modern LLMs are Few-Shot Parallel Detoxification Data Annotatorscited by 4
  10. 2024Exploring Cross-lingual Latent Transplantation: Mutual Opportunities and Open Challengescited by 1
  11. 2026Parallel-SFT: Improving Zero-Shot Cross-Programming-Language Transfer for Code RLcited by 1