# The Effect of Translationese in Machine Translation Test Sets

arXiv: https://arxiv.org/abs/1906.08069 · HTML: https://mlsanity.com/paper/1906.08069/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jun 2019.

## Built on by (most cited first, 30 of 49)

- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation](https://mlsanity.com/paper/2107.10821/) (2021)
- [Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset](https://mlsanity.com/paper/2205.12522/) (2022)
- [A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models](https://mlsanity.com/paper/2309.11674/) (2023)
- [Facebook AI WMT21 News Translation Task Submission](https://mlsanity.com/paper/2108.03265/) (2021)
- [Data Scaling Laws in NMT: The Effect of Noise and Architecture](https://mlsanity.com/paper/2202.01994/) (2022)
- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail](https://mlsanity.com/paper/2110.08300/) (2021)
- [Text Embedding Inversion Security for Multilingual Language Models](https://mlsanity.com/paper/2401.12192/) (2024)
- [URIEL+: Enhancing Linguistic Inclusion and Usability in a Typological and Multilingual Knowledge Base](https://mlsanity.com/paper/2409.18472/) (2024)
- [Akal Badi ya Bias: An Exploratory Study of Gender Bias in Hindi Language Technology](https://mlsanity.com/paper/2405.06346/) (2024)
- [Language Models are Good Translators](https://mlsanity.com/paper/2106.13627/) (2021)
- [On the Copying Behaviors of Pre-Training for Neural Machine Translation](https://mlsanity.com/paper/2107.08212/) (2021)
- [On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation](https://mlsanity.com/paper/2110.01811/) (2021)
- [Understanding Translationese in Cross-Lingual Summarization](https://mlsanity.com/paper/2212.07220/) (2022)
- [Comparing Feature-Engineering and Feature-Learning Approaches for Multilingual Translationese Classification](https://mlsanity.com/paper/2109.07604/) (2021)
- [On the Language Coverage Bias for Neural Machine Translation](https://mlsanity.com/paper/2106.03297/) (2021)
- [The USYD-JD Speech Translation System for IWSLT 2021](https://mlsanity.com/paper/2107.11572/) (2021)
- [Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performance](https://mlsanity.com/paper/2205.02293/) (2022)
- [Lost in Literalism: How Supervised Training Shapes Translationese in LLMs](https://mlsanity.com/paper/2503.04369/) (2025)
- [Lost in Translationese? Reducing Translation Effect Using Abstract Meaning Representation](https://mlsanity.com/paper/2304.11501/) (2023)
- [An Empirical Study of In-context Learning in LLMs for Machine Translation](https://mlsanity.com/paper/2401.12097/) (2024)
- [Beyond Glass-Box Features: Uncertainty Quantification Enhanced Quality Estimation for Neural Machine Translation](https://mlsanity.com/paper/2109.07141/) (2021)
- [Bridging the Data Gap between Training and Inference for Unsupervised Neural Machine Translation](https://mlsanity.com/paper/2203.08394/) (2022)
- [Explaining Translationese: why are Neural Classifiers Better and what do they Learn?](https://mlsanity.com/paper/2210.13391/) (2022)
- [PR-MCS: Perturbation Robust Metric for MultiLingual Image Captioning](https://mlsanity.com/paper/2303.08389/) (2023)
- [Breeding Machine Translations: Evolutionary approach to survive and thrive in the world of automated evaluation](https://mlsanity.com/paper/2305.19330/) (2023)

## Abstract (arXiv)

The effect of translationese has been studied in the field of machine translation (MT), mostly with respect to training data. We study in depth the effect of translationese on test data, using the test sets from the last three editions of WMT's news shared task, containing 17 translation directions. We show evidence that (i) the use of translationese in test sets results in inflated human evaluation scores for MT systems; (ii) in some cases system rankings do change and (iii) the impact translationese has on a translation direction is inversely correlated to the translation quality attainable by state-of-the-art MT systems for that direction.