# Fully Synthetic Data Improves Neural Machine Translation with Knowledge Distillation

arXiv: https://arxiv.org/abs/2012.15455 · HTML: https://mlsanity.com/paper/2012.15455/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2020.

## Built on by (most cited first, 3 of 3)

- [Data Augmentation With Back translation for Low Resource languages: A case of English and Luganda](https://mlsanity.com/paper/2505.02463/) (2025)
- [KD4MT: A Survey of Knowledge Distillation for Machine Translation](https://mlsanity.com/paper/2602.15845/) (2026)
- [Is Knowledge Distillation Actually Greener? A Case Study in Machine Translation](https://mlsanity.com/paper/2602.09691/) (2026)

## Abstract (arXiv)

This paper explores augmenting monolingual data for knowledge distillation in neural machine translation. Source language monolingual text can be incorporated as a forward translation. Interestingly, we find the best way to incorporate target language monolingual text is to translate it to the source language and round-trip translate it back to the target language, resulting in a fully synthetic corpus. We find that combining monolingual data from both source and target languages yields better performance than a corpus twice as large only in one language. Moreover, experiments reveal that the improvement depends upon the provenance of the test set. If the test set was originally in the source language (with the target side written by translators), then forward translating source monolingual data matters. If the test set was originally in the target language (with the source written by translators), then incorporating target monolingual data matters.