# On The Evaluation of Machine Translation Systems Trained With Back-Translation

arXiv: https://arxiv.org/abs/1908.05204 · HTML: https://mlsanity.com/paper/1908.05204/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Aug 2019.

## Built on by (most cited first, 30 of 40)

- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [Facebook AI WMT21 News Translation Task Submission](https://mlsanity.com/paper/2108.03265/) (2021)
- [The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation](https://mlsanity.com/paper/2110.03036/) (2021)
- [Transparent Human Evaluation for Image Captioning](https://mlsanity.com/paper/2111.08940/) (2021)
- [Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand](https://mlsanity.com/paper/2112.04139/) (2021)
- [IndicXNLI: Evaluating Multilingual Inference for Indian Languages](https://mlsanity.com/paper/2204.08776/) (2022)
- [Causal Direction of Data Collection Matters: Implications of Causal and Anticausal Learning for NLP](https://mlsanity.com/paper/2110.03618/) (2021)
- [StoryER: Automatic Story Evaluation via Ranking, Rating and Reasoning](https://mlsanity.com/paper/2210.08459/) (2022)
- [GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation](https://mlsanity.com/paper/2101.06561/) (2021)
- [On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation](https://mlsanity.com/paper/2110.01811/) (2021)
- [Understanding Translationese in Cross-Lingual Summarization](https://mlsanity.com/paper/2212.07220/) (2022)
- [Comparing Feature-Engineering and Feature-Learning Approaches for Multilingual Translationese Classification](https://mlsanity.com/paper/2109.07604/) (2021)
- [Learning to Generalize to More: Continuous Semantic Augmentation for Neural Machine Translation](https://mlsanity.com/paper/2204.06812/) (2022)
- [On the Language Coverage Bias for Neural Machine Translation](https://mlsanity.com/paper/2106.03297/) (2021)
- [Cross-lingual Intermediate Fine-tuning improves Dialogue State Tracking](https://mlsanity.com/paper/2109.13620/) (2021)
- [Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performance](https://mlsanity.com/paper/2205.02293/) (2022)
- [AFRIDOC-MT: Document-level MT Corpus for African Languages](https://mlsanity.com/paper/2501.06374/) (2025)
- [A Call for Clarity in Beam Search: How It Works and When It Stops](https://mlsanity.com/paper/2204.05424/) (2022)
- [Comparison of Grammatical Error Correction Using Back-Translation Models](https://mlsanity.com/paper/2104.07848/) (2021)
- [Can Question Generation Debias Question Answering Models? A Case Study on Question-Context Lexical Overlap](https://mlsanity.com/paper/2109.11256/) (2021)
- [Generative linguistics contribution to artificial intelligence: Where this contribution lies?](https://mlsanity.com/paper/2410.20221/) (2024)
- [What fifty-one years of Linguistics and Artificial Intelligence research tell us about their correlation: A scientometric analysis](https://mlsanity.com/paper/2411.19858/) (2024)
- [A Comparison of Approaches to Document-level Machine Translation](https://mlsanity.com/paper/2101.11040/) (2021)
- [Bridging the Data Gap between Training and Inference for Unsupervised Neural Machine Translation](https://mlsanity.com/paper/2203.08394/) (2022)
- [Geographical Distance Is The New Hyperparameter: A Case Study Of Finding The Optimal Pre-trained Language For English-isiZulu Machine Translation](https://mlsanity.com/paper/2205.08621/) (2022)
- [Exploring Diversity in Back Translation for Low-Resource Machine Translation](https://mlsanity.com/paper/2206.00564/) (2022)
- [Rule-Based, Neural and LLM Back-Translation: Comparative Insights from a Variant of Ladin](https://mlsanity.com/paper/2407.08819/) (2024)
- [AUGVIC: Exploiting BiText Vicinity for Low-Resource NMT](https://mlsanity.com/paper/2106.05141/) (2021)
- [On Systematic Style Differences between Unsupervised and Supervised MT and an Application for High-Resource Machine Translation](https://mlsanity.com/paper/2106.15818/) (2021)

## Abstract (arXiv)

Back-translation is a widely used data augmentation technique which leverages target monolingual data. However, its effectiveness has been challenged since automatic metrics such as BLEU only show significant improvements for test examples where the source itself is a translation, or translationese. This is believed to be due to translationese inputs better matching the back-translated training data. In this work, we show that this conjecture is not empirically supported and that back-translation improves translation quality of both naturally occurring text as well as translationese according to professional human translators. We provide empirical evidence to support the view that back-translation is preferred by humans because it produces more fluent outputs. BLEU cannot capture human preferences because references are translationese when source sentences are natural text. We recommend complementing BLEU with a language model score to measure fluency.