# Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Miss

arXiv: https://arxiv.org/abs/2604.12911 · HTML: https://mlsanity.com/paper/2604.12911/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 57 references that are in the map.
- First posted on arXiv 14 Apr 2026.

## Builds on (most cited first, 30 of 37)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://mlsanity.com/paper/2104.02145/) (2021)
- [Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation](https://mlsanity.com/paper/2412.03304/) (2024)
- [A Survey of Evaluation Metrics Used for NLG Systems](https://mlsanity.com/paper/2008.12009/) (2020)
- [GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4](https://mlsanity.com/paper/2310.13988/) (2023)
- [Prompting PaLM for Translation: Assessing Strategies and Performance](https://mlsanity.com/paper/2211.09102/) (2022)
- [Scaling Laws for Neural Machine Translation](https://mlsanity.com/paper/2109.07740/) (2021)
- [Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation](https://mlsanity.com/paper/1808.07048/) (2018)
- [Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models](https://mlsanity.com/paper/2303.13809/) (2023)
- [INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge](https://mlsanity.com/paper/2411.19799/) (2024)
- [Large language models effectively leverage document-level context for literary translation, but critical errors persist](https://mlsanity.com/paper/2304.03245/) (2023)
- [MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures](https://mlsanity.com/paper/2406.06565/) (2024)
- [PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts](https://mlsanity.com/paper/2504.18428/) (2025)
- [Benchmarking Machine Translation with Cultural Awareness](https://mlsanity.com/paper/2305.14328/) (2023)
- [Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above](https://mlsanity.com/paper/2502.14127/) (2025)
- [A Challenge Set Approach to Evaluating Machine Translation](https://mlsanity.com/paper/1704.07431/) (2017)
- [Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoning](https://mlsanity.com/paper/2502.17407/) (2025)
- [Examining the Tip of the Iceberg: A Data Set for Idiom Translation](https://mlsanity.com/paper/1802.04681/) (2018)
- [The Bitter Lesson Learned from 2,000+ Multilingual Benchmarks](https://mlsanity.com/paper/2504.15521/) (2025)
- [Revisiting Round-Trip Translation for Quality Estimation](https://mlsanity.com/paper/2004.13937/) (2020)

## Built on by (most cited first, 2 of 2)

- [M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models](https://mlsanity.com/paper/2608.03803/) (2026)
- [Building Multilingual Bridges: Data Mixing as the Pillar of Generalization for In-Language Reasoning](https://mlsanity.com/paper/2609.10445/) (2026)

## Abstract (arXiv)

Multilingual benchmarks guide the development of frontier models. Yet multilingual evaluations reported by frontier models are structured similar to popular reasoning and knowledge benchmarks, but across many languages. We show such benchmarks, and consequently multilingual evaluations, measure mathematical reasoning and factual recall, not multilingual proficiency. For example, thinking variants dramatically outperform instruct variants on these benchmarks, yet often perform worse on real-world multilingual tasks, such as LMArena. We propose a simple alternative: evaluate multilingual capability via round-trip translation. Given text in a source language, translate it to a target language and back; semantic gaps between the original and result expose failures in multilingual generation capabilities. Round-trip translation correlates almost perfectly (rho = 0.94) with user ratings on LMArena with our benchmark, requires no human reference translations, and does not require a more capable multilingual judge than tested models. Lastly, we introduce Lost in Translation (LiT), a challenging round-trip translation benchmark spanning widely spoken languages worldwide, for realistic evaluation of multilingual frontier models.