# Automatic Reference-Based Evaluation of Pronoun Translation Misses the Point

arXiv: https://arxiv.org/abs/1808.04164 · HTML: https://mlsanity.com/paper/1808.04164/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Aug 2018.

## Built on by (most cited first, 6 of 6)

- [BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation](https://mlsanity.com/paper/2103.11878/) (2021)
- [Extrinsic Evaluation of Machine Translation Metrics](https://mlsanity.com/paper/2212.10297/) (2022)
- [Discourse Centric Evaluation of Machine Translation with a Densely Annotated Parallel Corpus](https://mlsanity.com/paper/2305.11142/) (2023)
- [A Bilingual Parallel Corpus with Discourse Annotations](https://mlsanity.com/paper/2210.14667/) (2022)
- [Evaluating and Improving the Coreference Capabilities of Machine Translation Models](https://mlsanity.com/paper/2302.08464/) (2023)
- [The Blindness of Document-Level Translation Evaluation](https://mlsanity.com/paper/2609.05949/) (2026)

## Abstract (arXiv)

We compare the performance of the APT and AutoPRF metrics for pronoun translation against a manually annotated dataset comprising human judgements as to the correctness of translations of the PROTEST test suite. Although there is some correlation with the human judgements, a range of issues limit the performance of the automated metrics. Instead, we recommend the use of semi-automatic metrics and test suites in place of fully automatic metrics.