# Boosting Unsupervised Machine Translation with Pseudo-Parallel Data

arXiv: https://arxiv.org/abs/2310.14262 · HTML: https://mlsanity.com/paper/2310.14262/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 38 references that are in the map.
- First posted on arXiv 22 Oct 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [Unsupervised Neural Machine Translation](https://mlsanity.com/paper/1710.11041/) (2017)
- [Phrase-Based & Neural Unsupervised Machine Translation](https://mlsanity.com/paper/1804.07755/) (2018)
- [On the Language Neutrality of Pre-trained Multilingual Representations](https://mlsanity.com/paper/2004.05160/) (2020)
- [Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings](https://mlsanity.com/paper/1811.01136/) (2018)
- [Cross-lingual Retrieval for Iterative Self-Supervised Training](https://mlsanity.com/paper/2006.09526/) (2020)
- [Unsupervised Statistical Machine Translation](https://mlsanity.com/paper/1809.01272/) (2018)
- [An Effective Approach to Unsupervised Machine Translation](https://mlsanity.com/paper/1902.01313/) (2019)
- [Filtering and Mining Parallel Data in a Joint Multilingual Space](https://mlsanity.com/paper/1805.09822/) (2018)
- [A Multilingual View of Unsupervised Machine Translation](https://mlsanity.com/paper/2002.02955/) (2020)
- [Harnessing Multilinguality in Unsupervised Machine Translation for Rare Languages](https://mlsanity.com/paper/2009.11201/) (2020)
- [Multilingual Unsupervised Neural Machine Translation with Denoising Adapters](https://mlsanity.com/paper/2110.10472/) (2021)
- [Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining](https://mlsanity.com/paper/2105.10419/) (2021)

## Abstract (arXiv)

Even with the latest developments in deep learning and large-scale language modeling, the task of machine translation (MT) of low-resource languages remains a challenge. Neural MT systems can be trained in an unsupervised way without any translation resources but the quality lags behind, especially in truly low-resource conditions. We propose a training strategy that relies on pseudo-parallel sentence pairs mined from monolingual corpora in addition to synthetic sentence pairs back-translated from monolingual corpora. We experiment with different training schedules and reach an improvement of up to 14.5 BLEU points (English to Ukrainian) over a baseline trained on back-translated data only.