# Contrastive ESA: Human Evaluation of Multiple Translations at Once

arXiv: https://arxiv.org/abs/2607.26640 · HTML: https://mlsanity.com/paper/2607.26640/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 47 references that are in the map.
- First posted on arXiv 29 Jul 2026.

## Builds on (most cited first, 14 of 14)

- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [RankME: Reliable Human Ratings for Natural Language Generation](https://mlsanity.com/paper/1803.05928/) (2018)
- [Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation](https://mlsanity.com/paper/2406.11580/) (2024)
- [Efficient Online Scalar Annotation with Bounded Support](https://mlsanity.com/paper/1806.01170/) (2018)
- [Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreement](https://mlsanity.com/paper/2301.10684/) (2023)
- [AI-Assisted Human Evaluation of Machine Translation](https://mlsanity.com/paper/2406.12419/) (2024)
- [K-Sort Arena: Efficient and Reliable Benchmarking for Generative Models via K-wise Human Preferences](https://mlsanity.com/paper/2408.14468/) (2024)
- [MT-Ranker: Reference-free machine translation evaluation by inter-system ranking](https://mlsanity.com/paper/2401.17099/) (2024)
- [Pearmut: Human Evaluation of Translation Made Trivial](https://mlsanity.com/paper/2601.02933/) (2026)
- [Explaining with Contrastive Phrasal Highlighting: A Case Study in Assisting Humans to Detect Translation Differences](https://mlsanity.com/paper/2312.01582/) (2023)
- [Enhancing Human Evaluation in Machine Translation with Comparative Judgment](https://mlsanity.com/paper/2502.17797/) (2025)
- [COMET-poly: Machine Translation Metric Grounded in Other Candidates](https://mlsanity.com/paper/2508.18549/) (2025)
- [AI translation of literary texts is "fine", but readers still prefer human translations](https://mlsanity.com/paper/2606.26040/) (2026)

## Built on by (most cited first, 1 of 1)

- [Last Translation Benchmark](https://mlsanity.com/paper/2609.04173/) (2026)

## Abstract (arXiv)

Current human evaluation of machine translation typically assesses single outputs in isolation, a paradigm that suffers from high annotator noise and cost. We introduce Contrastive Error Span Annotation (cESA), a protocol that presents multiple translations of the source input (text, video, audio, image). In cESA, the annotator sees multiple translations of the same document, marks major and minor error spans, and then assigns a score from 0% to 100% on absolute scale. By allowing annotators to access the shared context across multiple outputs, cESA facilitates more consistent and efficient judgments. We validate cESA using a large-scale human evaluation of English->Japanese translations of 12 models, demonstrating reductions in annotation time and noise compared to standard pointwise evaluation. Unlike existing contrastive ranking methods, cESA yields absolute quality judgments that enable simple, interpretable non-parametric model rankings without the need for post-hoc corrections.