# Linguistic evaluation of German-English Machine Translation using a Test Suite

arXiv: https://arxiv.org/abs/1910.07457 · HTML: https://mlsanity.com/paper/1910.07457/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2019.

## Built on by (most cited first, 4 of 4)

- [Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation](https://mlsanity.com/paper/2205.15301/) (2022)
- [Angler: Helping Machine Translation Practitioners Prioritize Model Improvements](https://mlsanity.com/paper/2304.05967/) (2023)
- [GrASP: A Library for Extracting and Exploring Human-Interpretable Textual Patterns](https://mlsanity.com/paper/2104.03958/) (2021)
- [The Box is in the Pen: Evaluating Commonsense Reasoning in Neural Machine Translation](https://mlsanity.com/paper/2503.03308/) (2025)

## Abstract (arXiv)

We present the results of the application of a grammatical test suite for GermanrightarrowEnglish MT on the systems submitted at WMT19, with a detailed analysis for 107 phenomena organized in 14 categories. The systems still translate wrong one out of four test items in average. Low performance is indicated for idioms, modals, pseudo-clefts, multi-word expressions and verb valency. When compared to last year, there has been a improvement of function words, non-verbal agreement and punctuation. More detailed conclusions about particular systems and phenomena are also presented.