mlsanityData to 7 Oct 2026

Translation Errors Significantly Impact Low-Resource Languages in Cross-Lingual Learning

arXiv 2402.02080 ↗First posted 3 Feb 2024cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Popular benchmarks (e.g., XNLI) used to evaluate cross-lingual language understanding consist of parallel versions of English evaluation sets in multiple target languages created with the help of professional translators. When creating such parallel data, it is critical to ensure high-quality translations for all target languages for an accurate characterization of cross-lingual transfer. In this work, we find that translation inconsistencies do exist and interestingly they disproportionally impact low-resource languages in XNLI. To identify such inconsistencies, we propose measuring the gap in performance between zero-shot evaluations on the human-translated and machine-translated target text across multiple target languages; relatively large gaps are indicative of translation errors. We also corroborate that translation errors exist for two target languages, namely Hindi and Urdu, by doing a manual reannotation of human-translated test instances in these two languages and finding poor agreement with the original English labels these instances were supposed to inherit.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 18 of its 28 references that are in the map. First posted on arXiv 3 Feb 2024.

Builds on

18 of 28 references.

References · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  8. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  9. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  10. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  11. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  12. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
6 more
  1. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  2. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  3. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 167
  4. 2020Translation Artifacts in Cross-lingual Transfer Learningcited by 95
  5. 2023Revisiting Machine Translation for Cross-lingual Classificationcited by 40
  6. 2023XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languagescited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025Mangosteen: An Open Thai Corpus for Language Model Pretrainingcited by 2
  2. 2025How and Where to Translate? The Impact of Translation Strategies in Cross-lingual LLM Promptingcited by 2
  3. 2025Uncovering Cross-Linguistic Disparities in LLMs using Sparse Autoencoderscited by 1
  4. 2026Quantifying the Impact of Translation Errors on Multilingual LLM Evaluationcited by 1
  5. 2026Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluationcited by 1
  6. 2026Training Language Models with homotokens Leads to Delayed Overfittingcited by 0
  7. 2026BETA-Labeling for Multilingual Dataset Construction in Low-Resource IRcited by 0
  8. 2026Is Textual Similarity Invariant under Machine Translation? Evidence Based on the Political Manifesto Corpuscited by 0
  9. 2026SEA-NLI: Natural Language Inference as a Lens into Southeast Asian Cultural Understandingcited by 0