PatTransBench: an ipc-stratified, multi-dimensional benchmark for patent machine translation evaluation
Not on the map. Why?
Abstract (arXiv)
We present PatTransBench, a 12,000-item Chinese-English/Japanese/Korean patent machine translation (PatMT) benchmark with a four-dimensional evaluation protocol. The benchmark comprises 6,000 short-text (sentence/claim) and 6,000 long-text (paragraph) pairs, evenly split across three language pairs and both translation directions, built via IPC-section-stratified sampling and dualthreshold quality gating from upstream patent-family parallel corpora. The evaluation protocol spans literal n-gram overlap (BLEU, ROUGE-L, chrF), cross-lingual semantic textual similarity (CLSTS), 38-probe-language consistency testing, and serving performance under concurrency. A case study on PatMT2-7B and its fp8/AWQ4/AutoRound-4bit variants reveals direction-dependent quantization effects masked by aggregate scores and non-monotonic language-failure patterns across quantization schemes that are invisible to single-metric evaluation. We plan to release the benchmark, protocol, and case-study results upon publication to support reproducible PatMT evaluation.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 3 of its 8 references that are in the map. First posted on arXiv 12 Oct 2026.
Builds on
3 of 8 references.
References · 3
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
- 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
- 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
Built on by
Not cited yet.