# Meta Back-translation

arXiv: https://arxiv.org/abs/2102.07847 · HTML: https://mlsanity.com/paper/2102.07847/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 37 references that are in the map.
- First posted on arXiv 15 Feb 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Meta Pseudo Labels](https://mlsanity.com/paper/2003.10580/) (2020)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [Revisiting Self-Training for Neural Sequence Generation](https://mlsanity.com/paper/1909.13788/) (2019)
- [On Using Monolingual Corpora in Neural Machine Translation](https://mlsanity.com/paper/1503.03535/) (2015)
- [Unsupervised Neural Machine Translation](https://mlsanity.com/paper/1710.11041/) (2017)
- [Dual Learning for Machine Translation](https://mlsanity.com/paper/1611.00179/) (2016)
- [Choosing Transfer Languages for Cross-Lingual Learning](https://mlsanity.com/paper/1905.12688/) (2019)
- [When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?](https://mlsanity.com/paper/1804.06323/) (2018)
- [Online Learning Rate Adaptation with Hypergradient Descent](https://mlsanity.com/paper/1703.04782/) (2017)
- [Meta-Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1808.08437/) (2018)
- [Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data](https://mlsanity.com/paper/1912.07768/) (2019)
- [Rapid Adaptation of Neural Machine Translation to New Languages](https://mlsanity.com/paper/1808.04189/) (2018)
- [Balancing Training for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2004.06748/) (2020)
- [Optimizing Data Usage via Differentiable Rewards](https://mlsanity.com/paper/1911.10088/) (2019)
- [compare-mt: A Tool for Holistic Comparison of Language Generation Systems](https://mlsanity.com/paper/1903.07926/) (2019)
- [Semi-Supervised Learning for Neural Machine Translation](https://mlsanity.com/paper/1606.04596/) (2016)
- [Dynamic Data Selection and Weighting for Iterative Back-Translation](https://mlsanity.com/paper/2004.03672/) (2020)
- [Multilingual Neural Machine Translation With Soft Decoupled Encoding](https://mlsanity.com/paper/1902.03499/) (2019)
- [Target Conditioned Sampling: Optimizing Data Selection for Multilingual Neural Machine Translation](https://mlsanity.com/paper/1905.08212/) (2019)
- [Selecting Backtranslated Data from Multiple Sources for Improved Neural Machine Translation](https://mlsanity.com/paper/2005.00308/) (2020)

## Built on by (most cited first, 16 of 16)

- [Best Practices and Lessons Learned on Synthetic Data](https://mlsanity.com/paper/2404.07503/) (2024)
- [BERT Learns to Teach: Knowledge Distillation with Meta Learning](https://mlsanity.com/paper/2106.04570/) (2021)
- [Meta Learning for Natural Language Processing: A Survey](https://mlsanity.com/paper/2205.01500/) (2022)
- [GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents](https://mlsanity.com/paper/2505.23671/) (2025)
- [FOAM: A Follower-aware Speaker Model For Vision-and-Language Navigation](https://mlsanity.com/paper/2206.04294/) (2022)
- [Enhancing LLM Safety Through a Theoretical Minimax Game Lens](https://mlsanity.com/paper/2502.05163/) (2025)
- [Learning to Perturb Word Embeddings for Out-of-distribution QA](https://mlsanity.com/paper/2105.02692/) (2021)
- [Learning to Generalize to More: Continuous Semantic Augmentation for Neural Machine Translation](https://mlsanity.com/paper/2204.06812/) (2022)
- [A Feature-space Multimodal Data Augmentation Technique for Text-video Retrieval](https://mlsanity.com/paper/2208.02080/) (2022)
- [Back Translation for Speech-to-text Translation Without Transcripts](https://mlsanity.com/paper/2305.08709/) (2023)
- [Overcoming Data Scarcity in Generative Language Modelling for Low-Resource Languages: A Systematic Review](https://mlsanity.com/paper/2505.04531/) (2025)
- [Text Style Transfer Back-Translation](https://mlsanity.com/paper/2306.01318/) (2023)
- [Asynchronous and Segmented Bidirectional Encoding for NMT](https://mlsanity.com/paper/2402.14849/) (2024)
- [End-to-End Training for Back-Translation with Categorical Reparameterization Trick](https://mlsanity.com/paper/2202.08465/) (2022)
- [INarIG: Iterative Non-autoregressive Instruct Generation Model For Word-Level Auto Completion](https://mlsanity.com/paper/2311.18200/) (2023)
- [Reducing and Exploiting Data Augmentation Noise through Meta Reweighting Contrastive Learning for Text Classification](https://mlsanity.com/paper/2409.17474/) (2024)

## Abstract (arXiv)

Back-translation is an effective strategy to improve the performance of Neural Machine Translation~(NMT) by generating pseudo-parallel data. However, several recent works have found that better translation quality of the pseudo-parallel data does not necessarily lead to better final translation models, while lower-quality but more diverse data often yields stronger results. In this paper, we propose a novel method to generate pseudo-parallel data from a pre-trained back-translation model. Our method is a meta-learning algorithm which adapts a pre-trained back-translation model so that the pseudo-parallel data it generates would train a forward-translation model to do well on a validation set. In our evaluations in both the standard datasets WMT En-De'14 and WMT En-Fr'14, as well as a multilingual translation setting, our method leads to significant improvements over strong baselines. Our code will be made available.