# Improving Term Evaluation in Machine Translation: Variation Matters

arXiv: https://arxiv.org/abs/2609.08779 · HTML: https://mlsanity.com/paper/2609.08779/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 47 references that are in the map.
- First posted on arXiv 8 Sep 2026.

## Builds on (most cited first, 20 of 20)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search](https://mlsanity.com/paper/1704.07138/) (2017)
- [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2209.06243/) (2022)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation](https://mlsanity.com/paper/1804.06609/) (2018)
- [SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings](https://mlsanity.com/paper/2004.08728/) (2020)
- [GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4](https://mlsanity.com/paper/2310.13988/) (2023)
- [MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task](https://mlsanity.com/paper/2410.03983/) (2024)
- [WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects](https://mlsanity.com/paper/2502.12404/) (2025)
- [Training Neural Machine Translation To Apply Terminology Constraints](https://mlsanity.com/paper/1906.01105/) (2019)
- [Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation](https://mlsanity.com/paper/2102.00287/) (2021)
- [Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translation](https://mlsanity.com/paper/1906.12068/) (2019)
- [Post-editese: an Exacerbated Translationese](https://mlsanity.com/paper/1907.00900/) (2019)
- [DelTA: An Online Document-Level Translation Agent Based on Multi-Level Memory](https://mlsanity.com/paper/2410.08143/) (2024)
- [MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators](https://mlsanity.com/paper/2409.14335/) (2024)
- [EuroLLM-9B: Technical Report](https://mlsanity.com/paper/2506.04079/) (2025)
- [On the Evaluation of Machine Translation for Terminology Consistency](https://mlsanity.com/paper/2106.11891/) (2021)
- [EuroLLM-22B: Technical Report](https://mlsanity.com/paper/2602.05879/) (2026)

## Built on by (most cited first, 1 of 1)

- [TermJudge: A Document-Level Metric Judging, Not Counting, Terminology in Machine Translation Evaluation](https://mlsanity.com/paper/2609.35017/) (2026)

## Abstract (arXiv)

Terminology evaluation in machine translation (MT) usually assumes a single correct target form per source term. However, human translators routinely introduce variation that current metrics penalize as inconsistency. We examine how to account for this variation in document-level MT evaluation of English-French scientific translation, combining glossary-based accuracy, translation consistency, and a new cross-term variation (CTV) diagnostic measure that tests whether variation relationships are preserved across languages. Based on analyses of two parallel corpora, translated by four MT systems, we find that (1) MT systems generate less target-side variation than human translators; (2) transfer patterns strongly depend on the variation type; (3) consistency rankings vary with the choice of metric; and (4) constraining MT with a glossary improves accuracy and consistency but degrades CTV by suppressing valid variation. We argue for variation-aware evaluation that conditions consistency penalties on whether target-side variation mirrors source-side variation.