# Evaluating Optimal Reference Translations

arXiv: https://arxiv.org/abs/2311.16787 · HTML: https://mlsanity.com/paper/2311.16787/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 30 references that are in the map.
- First posted on arXiv 28 Nov 2023.

## Builds on (most cited first, 5 of 5)

- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Achieving Human Parity on Automatic Chinese to English News Translation](https://mlsanity.com/paper/1803.05567/) (2018)
- [The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation](https://mlsanity.com/paper/2109.06835/) (2021)
- [Consistent Human Evaluation of Machine Translation across Language Pairs](https://mlsanity.com/paper/2205.08533/) (2022)
- [A Survey on Document-level Neural Machine Translation: Methods and Evaluation](https://mlsanity.com/paper/1912.08494/) (2019)

## Built on by (most cited first, 1 of 1)

- [Quality and Quantity of Machine Translation References for Automatic Metrics](https://mlsanity.com/paper/2401.01283/) (2024)

## Abstract (arXiv)

The overall translation quality reached by current machine translation (MT) systems for high-resourced language pairs is remarkably good. Standard methods of evaluation are not suitable nor intended to uncover the many translation errors and quality deficiencies that still persist. Furthermore, the quality of standard reference translations is commonly questioned and comparable quality levels have been reached by MT alone in several language pairs. Navigating further research in these high-resource settings is thus difficult. In this article, we propose a methodology for creating more reliable document-level human reference translations, called "optimal reference translations," with the simple aim to raise the bar of what should be deemed "human translation quality." We evaluate the obtained document-level optimal reference translations in comparison with "standard" ones, confirming a significant quality increase and also documenting the relationship between evaluation and translation editing.