mlsanityData to 14 Oct 2026

Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation

arXiv 2605.24904 ↗First posted 24 May 2026cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Machine-translated benchmarks are widely used to assess the multilingual capabilities of large language models (LLMs), yet translation errors in these benchmarks remain underexplored, raising concerns about the reliability and comparability of multilingual evaluation. We address two practical gaps: (i) how well automatic MQM-style error spans from LLM judges and a span-aware QE baseline (xCOMET-XXL) match expert human span annotations on benchmark translations, and (ii) how strongly translation errors (as opposed to source-side issues in the English original) explain accuracy drops on translated benchmarks. We find that span agreement is non-trivial on naturally occurring benchmark translations, and that target-side translation errors are consistently associated with measurable, percentage-point drops in translated accuracy even after controlling for English correctness and source-side anomalies.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 22 of its 46 references that are in the map. First posted on arXiv 24 May 2026.

Builds on

22 of 46 references.

References · 22

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  7. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  8. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  9. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  10. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  11. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  12. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
10 more
  1. 2020Translation Artifacts in Cross-lingual Transfer Learningcited by 96
  2. 2024Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translationcited by 62
  3. 2024Towards Multilingual LLM Evaluation for European Languagescited by 30
  4. 2024MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluatorscited by 22
  5. 2024Spanish and LLM Benchmarks: is MMLU Lost in Translation?cited by 20
  6. 2024Machine Translation Meta Evaluation through Translation Accuracy Challenge Setscited by 18
  7. 2024Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translationcited by 16
  8. 2024BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanismcited by 12
  9. 2024Translation Errors Significantly Impact Low-Resource Languages in Cross-Lingual Learningcited by 9
  10. 2024On the Evaluation Practices in Multilingual NLP: Can Machine Translation Offer an Alternative to Human Translations?cited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026XHotpotQA: A Benchmark for Cross-Lingual Knowledge Composition in Multi-Hop Question Answeringcited by 0