# Fine-grained human evaluation of neural versus phrase-based machine translation

arXiv: https://arxiv.org/abs/1706.04389 · HTML: https://mlsanity.com/paper/1706.04389/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2017.

## Built on by (most cited first, 2 of 2)

- [Translating the Unseen? Yoruba-English MT in Low-Resource, Morphologically-Unmarked Settings](https://mlsanity.com/paper/2103.04225/) (2021)
- [Impact of Domain-Adapted Multilingual Neural Machine Translation in the Medical Domain](https://mlsanity.com/paper/2212.02143/) (2022)

## Abstract (arXiv)

We compare three approaches to statistical machine translation (pure phrase-based, factored phrase-based and neural) by performing a fine-grained manual evaluation via error annotation of the systems' outputs. The error types in our annotation are compliant with the multidimensional quality metrics (MQM), and the annotation is performed by two annotators. Inter-annotator agreement is high for such a task, and results show that the best performing system (neural) reduces the errors produced by the worst system (phrase-based) by 54%.