# A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directions

arXiv: https://arxiv.org/abs/1701.02901 · HTML: https://mlsanity.com/paper/1701.02901/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jan 2017.

## Built on by (most cited first, 10 of 10)

- [ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics](https://mlsanity.com/paper/2210.15615/) (2022)
- [PreQuEL: Quality Estimation of Machine Translation Outputs in Advance](https://mlsanity.com/paper/2205.09178/) (2022)
- [Rubrik's Cube: Testing a New Rubric for Evaluating Explanations on the CUBE dataset](https://mlsanity.com/paper/2503.23899/) (2025)
- [Translating the Unseen? Yoruba-English MT in Low-Resource, Morphologically-Unmarked Settings](https://mlsanity.com/paper/2103.04225/) (2021)
- [CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignment](https://mlsanity.com/paper/2503.23777/) (2025)
- [Good, but not always Fair: An Evaluation of Gender Bias for three commercial Machine Translation Systems](https://mlsanity.com/paper/2306.05882/) (2023)
- [Statistical Machine Translation for Indic Languages](https://mlsanity.com/paper/2301.00539/) (2023)
- [Joint Dropout: Improving Generalizability in Low-Resource Neural Machine Translation through Phrase Pair Variables](https://mlsanity.com/paper/2307.12835/) (2023)
- [The Comparison of Translationese in Machine Translation and Human Transation in terms of Translation Relations](https://mlsanity.com/paper/2404.08661/) (2024)
- [Understanding the effects of word-level linguistic annotations in under-resourced neural machine translation](https://mlsanity.com/paper/2401.16078/) (2024)

## Abstract (arXiv)

We aim to shed light on the strengths and weaknesses of the newly introduced neural machine translation paradigm. To that end, we conduct a multifaceted evaluation in which we compare outputs produced by state-of-the-art neural machine translation and phrase-based machine translation systems for 9 language directions across a number of dimensions. Specifically, we measure the similarity of the outputs, their fluency and amount of reordering, the effect of sentence length and performance across different error categories. We find out that translations produced by neural machine translation systems are considerably different, more fluent and more accurate in terms of word order compared to those produced by phrase-based systems. Neural machine translation systems are also more accurate at producing inflected forms, but they perform poorly when translating very long sentences.