mlsanityData to 14 Oct 2026

HardMTBench: Stress-Testing Chinese-English Translation on Knowledge-Intensive Domains

arXiv 2605.28315 ↗First posted 27 May 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

General-purpose machine translation benchmarks such as FLORES-200 have reached a saturation regime on Chinese-English pairs, where modern large language models cluster within a narrow band of high scores. Across 22 systems, FLORES-200 zh-en GEMBA scores fall in a 7.87-point range with a standard deviation of 2.29, which compresses the separation between systems on knowledge-intensive domains such as finance, healthcare, law, and science and technology. We introduce HardMTBench, a difficulty-aware diagnostic benchmark for bidirectional Chinese-English domain translation. HardMTBench covers 12 domains and contains 10,000 hand-curated source sentences with reference translations, packaged as 20,000 directional test items. A three-stage construction pipeline builds a domain-balanced candidate pool of 84{,}566 pairs, applies an LLM-based multi-signal judge over knowledge density, translation difficulty, terminology load and reference correctness, and assembles the final test set under a hardness fusion rule with per-domain quotas. Across 22 systems spanning general LLMs, commercial engines and specialised MT models, HardMTBench widens the cross-system GEMBA range by roughly a factor of two over FLORES-200, induces visible rank reorderings, and exposes domain-specific terminology and knowledge weaknesses that quality-only metrics tend to flatten. All data and code are open-sourced at https://github.com/jasonNLP/HardMTBench.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 26 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

17 of 26 references.

References · 17

  1. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  2. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  3. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  4. 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 146
  5. 2025WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialectscited by 79
  6. 2019Training Neural Machine Translation To Apply Terminology Constraintscited by 73
  7. 2020TICO-19: the Translation Initiative for Covid-19cited by 68
  8. 2018MTNT: A Testbed for Machine Translation of Noisy Textcited by 54
  9. 2024Findings of the WMT 2024 Shared Task on Discourse-Level Literary Translationcited by 15
  10. 2024Large Language Model for Multi-Domain Translation: Benchmarking and Domain CoT Fine-tuningcited by 11
  11. 2026Hy-MT2: A Family of Fast, Efficient and Powerful Multilingual Translation Models in the Wildcited by 10
  12. 2025Languages Still Left Behind: Toward a Better Multilingual Machine Translation Benchmarkcited by 7
5 more
  1. 2026Translation as a Scalable Proxy for Multilingual Evaluationcited by 6
  2. 2026"Be My Cheese?": Cultural Nuance Benchmarking for Machine Translation in Multilingual LLMscited by 3
  3. 2024Machine Translation Evaluation Benchmark for Wu Chinese: Workflow and Analysiscited by 2
  4. 2026Benchmarking Machine Translation on Chinese Social Media Textscited by 2
  5. 2026Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domainscited by 2

Built on by

Not cited yet.