# On Synthetic Data for Back Translation

arXiv: https://arxiv.org/abs/2310.13675 · HTML: https://mlsanity.com/paper/2310.13675/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 58 references that are in the map.
- First posted on arXiv 20 Oct 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [Do CIFAR-10 Classifiers Generalize to CIFAR-10?](https://mlsanity.com/paper/1806.00451/) (2018)
- [On Using Monolingual Corpora in Neural Machine Translation](https://mlsanity.com/paper/1503.03535/) (2015)
- [Unsupervised Neural Machine Translation](https://mlsanity.com/paper/1710.11041/) (2017)
- [Dual Learning for Machine Translation](https://mlsanity.com/paper/1611.00179/) (2016)
- [Data Augmentation for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1705.00440/) (2017)
- [Phrase-Based & Neural Unsupervised Machine Translation](https://mlsanity.com/paper/1804.07755/) (2018)
- [Style Transfer Through Back-Translation](https://mlsanity.com/paper/1804.09000/) (2018)
- [Reward Augmented Maximum Likelihood for Neural Structured Prediction](https://mlsanity.com/paper/1609.00150/) (2016)
- [Tagged Back-Translation](https://mlsanity.com/paper/1906.06442/) (2019)
- [SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation](https://mlsanity.com/paper/1808.07512/) (2018)
- [Regularizing Neural Machine Translation by Target-bidirectional Agreement](https://mlsanity.com/paper/1808.04064/) (2018)
- [On The Evaluation of Machine Translation Systems Trained With Back-Translation](https://mlsanity.com/paper/1908.05204/) (2019)
- [Improving Back-Translation with Uncertainty-based Confidence Estimation](https://mlsanity.com/paper/1909.00157/) (2019)
- [Mask Attention Networks: Rethinking and Strengthen Transformer](https://mlsanity.com/paper/2103.13597/) (2021)
- [Back-Translation Sampling by Targeting Difficult Words in Neural Machine Translation](https://mlsanity.com/paper/1808.09006/) (2018)
- [Generalizing Back-Translation in Neural Machine Translation](https://mlsanity.com/paper/1906.07286/) (2019)
- [Detecting Overfitting via Adversarial Examples](https://mlsanity.com/paper/1903.02380/) (2019)
- [Decoding and Diversity in Machine Translation](https://mlsanity.com/paper/2011.13477/) (2020)
- [Evaluating Explanation Methods for Neural Machine Translation](https://mlsanity.com/paper/2005.01672/) (2020)

## Built on by (most cited first, 4 of 4)

- [Best Practices and Lessons Learned on Synthetic Data](https://mlsanity.com/paper/2404.07503/) (2024)
- [Enhancing LLM Safety Through a Theoretical Minimax Game Lens](https://mlsanity.com/paper/2502.05163/) (2025)
- [Rule-Based, Neural and LLM Back-Translation: Comparative Insights from a Variant of Ladin](https://mlsanity.com/paper/2407.08819/) (2024)
- [Extending Multilingual Machine Translation through Imitation Learning](https://mlsanity.com/paper/2311.08538/) (2023)

## Abstract (arXiv)

Back translation (BT) is one of the most significant technologies in NMT research fields. Existing attempts on BT share a common characteristic: they employ either beam search or random sampling to generate synthetic data with a backward model but seldom work studies the role of synthetic data in the performance of BT. This motivates us to ask a fundamental question: em what kind of synthetic data contributes to BT performance? Through both theoretical and empirical studies, we identify two key factors on synthetic data controlling the back-translation NMT performance, which are quality and importance. Furthermore, based on our findings, we propose a simple yet effective method to generate synthetic data to better trade off both factors so as to yield a better performance for BT. We run extensive experiments on WMT14 DE-EN, EN-DE, and RU-EN benchmark tasks. By employing our proposed method to generate synthetic data, our BT model significantly outperforms the standard BT baselines (i.e., beam and sampling based methods for data generation), which proves the effectiveness of our proposed methods.