# Enhancing Human Evaluation in Machine Translation with Comparative Judgment

arXiv: https://arxiv.org/abs/2502.17797 · HTML: https://mlsanity.com/paper/2502.17797/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 45 references that are in the map.
- First posted on arXiv 25 Feb 2025.

## Builds on (most cited first, 13 of 13)

- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators](https://mlsanity.com/paper/2403.16950/) (2024)
- [The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation](https://mlsanity.com/paper/2109.06835/) (2021)
- [The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation](https://mlsanity.com/paper/2308.07286/) (2023)
- [Large language models effectively leverage document-level context for literary translation, but critical errors persist](https://mlsanity.com/paper/2304.03245/) (2023)
- [Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration](https://mlsanity.com/paper/2305.14324/) (2023)
- [Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation](https://mlsanity.com/paper/2406.11580/) (2024)
- [Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation](https://mlsanity.com/paper/1808.10432/) (2018)
- [A Set of Recommendations for Assessing Human-Machine Parity in Language Translation](https://mlsanity.com/paper/2004.01694/) (2020)
- [Suri: Multi-constraint Instruction Following for Long-form Text Generation](https://mlsanity.com/paper/2406.19371/) (2024)
- [Finding Replicable Human Evaluations via Stable Ranking Probability](https://mlsanity.com/paper/2404.01474/) (2024)
- [Pairwise Neural Machine Translation Evaluation](https://mlsanity.com/paper/1912.03135/) (2019)

## Built on by (most cited first, 7 of 7)

- [Pearmut: Human Evaluation of Translation Made Trivial](https://mlsanity.com/paper/2601.02933/) (2026)
- [Remedy: Learning Machine Translation Evaluation from Human Preferences with Reward Modeling](https://mlsanity.com/paper/2504.13630/) (2025)
- [Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs](https://mlsanity.com/paper/2512.16378/) (2025)
- [MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation](https://mlsanity.com/paper/2510.24664/) (2025)
- [TransEvalnia: Reasoning-based Evaluation and Ranking of Translations](https://mlsanity.com/paper/2507.12724/) (2025)
- [Contrastive ESA: Human Evaluation of Multiple Translations at Once](https://mlsanity.com/paper/2607.26640/) (2026)
- [Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged](https://mlsanity.com/paper/2610.10918/) (2026)

## Abstract (arXiv)

Human evaluation is crucial for assessing rapidly evolving language models but is influenced by annotator proficiency and task design. This study explores the integration of comparative judgment into human annotation for machine translation (MT) and evaluates three annotation setups-point-wise Multidimensional Quality Metrics (MQM), side-by-side (SxS) MQM, and its simplified version SxS relative ranking (RR). In MQM, annotators mark error spans with categories and severity levels. SxS MQM extends MQM to pairwise error annotation for two translations of the same input, while SxS RR focuses on selecting the better output without labeling errors. Key findings are: (1) the SxS settings achieve higher inter-annotator agreement than MQM; (2) SxS MQM enhances inter-translation error marking consistency compared to MQM by, on average, 38.5% for explicitly compared MT systems and 19.5% for others; (3) all annotation settings return stable system rankings, with SxS RR offering a more efficient alternative to (SxS) MQM; (4) the SxS settings highlight subtle errors overlooked in MQM without altering absolute system evaluations. To spur further research, we will release the triply annotated datasets comprising 377 ZhEn and 104 EnDe annotation examples.