# Languages Still Left Behind: Toward a Better Multilingual Machine Translation Benchmark

arXiv: https://arxiv.org/abs/2508.20511 · HTML: https://mlsanity.com/paper/2508.20511/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 26 references that are in the map.
- First posted on arXiv 28 Aug 2025.

## Builds on (most cited first, 3 of 3)

- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Correcting FLORES Evaluation Dataset for Four African Languages](https://mlsanity.com/paper/2409.00626/) (2024)

## Built on by (most cited first, 7 of 7)

- [Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Miss](https://mlsanity.com/paper/2604.12911/) (2026)
- [Alexandria: A Multi-Domain Dialectal Arabic Machine Translation Dataset for Culturally Inclusive and Linguistically Diverse LLMs](https://mlsanity.com/paper/2601.13099/) (2026)
- [Rosetta at AlexandriaX-2026: LoRA-Adapted NileChat for Context-Aware Dialectal Arabic Dialogue Translation](https://mlsanity.com/paper/2609.10395/) (2026)
- [The Hrunting of AI: Where and How to Improve English Dialectal Fairness](https://mlsanity.com/paper/2603.15187/) (2026)
- [HardMTBench: Stress-Testing Chinese-English Translation on Knowledge-Intensive Domains](https://mlsanity.com/paper/2605.28315/) (2026)
- [Multilinguality of Large Language Models From a Structural Perspective](https://mlsanity.com/paper/2606.01800/) (2026)
- [The Role of Disfluencies in Speech Translation](https://mlsanity.com/paper/2608.02138/) (2026)

## Abstract (arXiv)

Multilingual machine translation (MT) benchmarks play a central role in evaluating the capabilities of modern MT systems. Among them, the FLORES+ benchmark is widely used, offering English-to-many translation data for over 200 languages, curated with strict quality control protocols. However, we study data in four languages (Asante Twi, Japanese, Jinghpaw, and South Azerbaijani) and uncover critical shortcomings in the benchmark's suitability for truly multilingual evaluation. Human assessments reveal that many translations fall below the claimed 90% quality standard, and the annotators report that source sentences are often too domain-specific and culturally biased toward the English-speaking world. We further demonstrate that simple heuristics, such as copying named entities, can yield non-trivial BLEU scores, suggesting vulnerabilities in the evaluation protocol. Notably, we show that MT models trained on high-quality, naturalistic data perform poorly on FLORES+ while achieving significant gains on our domain-relevant evaluation set. Based on these findings, we advocate for multilingual MT benchmarks that use domain-general and culturally neutral source texts rely less on named entities, in order to better reflect real-world translation challenges.