The Translation Tax Is Not a Scalar: A Counterfactual Audit of English-Source Cue Inheritance in Chinese Multilingual Benchmarks
Not on the map. Why?
Abstract (arXiv)
The Translation Tax is often treated as a scalar: translated benchmarks are assumed to inflate scores by preserving English-source cues. We audit this claim in an English-to-Chinese setting. Three proxy estimators disagree: back-translation gaps are small and parser-fragile; cue-score calibration does not predict item-level gains; and a six-model native-control comparison shows model-family rather than uniform benchmark effects. We add a same-item LLM-naturalization stress test that holds answer, options, and content fixed while rewriting Chinese surface form. After correcting a prompt-construction bug, this contrast no longer supports a model-family interaction, but it preserves a residue dose-response: high-residue items benefit while low-residue items do not. The result is not a single Translation Tax, but a set of estimator- and item-dependent validity risks. We release per-cell evidence, the naturalization protocol, human QC, and a reporting checklist for translated multilingual benchmark papers.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 10 of its 11 references that are in the map. First posted on arXiv 8 May 2026.
Builds on
10 of 11 references.
References · 10
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
- 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
- 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
- 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
- 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
- 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 145
- 2020Translation Artifacts in Cross-lingual Transfer Learningcited by 95
- 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 78
- 2025The Bitter Lesson Learned from 2,000+ Multilingual Benchmarkscited by 31
Built on by
Not cited yet.