# On the Copying Problem of Unsupervised NMT: A Training Schedule with a Language Discriminator Loss

arXiv: https://arxiv.org/abs/2305.17182 · HTML: https://mlsanity.com/paper/2305.17182/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 26 May 2023.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [Multi30K: Multilingual English-German Image Descriptions](https://mlsanity.com/paper/1605.00459/) (2016)
- [Unsupervised Neural Machine Translation](https://mlsanity.com/paper/1710.11041/) (2017)
- [Rapid Adaptation of Neural Machine Translation to New Languages](https://mlsanity.com/paper/1808.04189/) (2018)
- [Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description](https://mlsanity.com/paper/1710.07177/) (2017)
- [Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations](https://mlsanity.com/paper/1906.01181/) (2019)
- [Code-switching pre-training for neural machine translation](https://mlsanity.com/paper/2009.08088/) (2020)
- [When Does Unsupervised Machine Translation Work?](https://mlsanity.com/paper/2004.05516/) (2020)
- [When and Why is Unsupervised Neural Machine Translation Useless?](https://mlsanity.com/paper/2004.10581/) (2020)
- [Improving Multilingual Translation by Representation and Gradient Regularization](https://mlsanity.com/paper/2109.04778/) (2021)
- [Language Tags Matter for Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/2106.07930/) (2021)
- [On the Copying Behaviors of Pre-Training for Neural Machine Translation](https://mlsanity.com/paper/2107.08212/) (2021)
- [Improving the Lexical Ability of Pretrained Language Models for Unsupervised Neural Machine Translation](https://mlsanity.com/paper/2103.10531/) (2021)
- [Flow-Adapter Architecture for Unsupervised Machine Translation](https://mlsanity.com/paper/2204.12225/) (2022)
- [On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translation](https://mlsanity.com/paper/2305.10930/) (2023)

## Built on by (most cited first, 1 of 1)

- [CoVoSwitch: Machine Translation of Synthetic Code-Switched Text Based on Intonation Units](https://mlsanity.com/paper/2407.14295/) (2024)

## Abstract (arXiv)

Although unsupervised neural machine translation (UNMT) has achieved success in many language pairs, the copying problem, i.e., directly copying some parts of the input sentence as the translation, is common among distant language pairs, especially when low-resource languages are involved. We find this issue is closely related to an unexpected copying behavior during online back-translation (BT). In this work, we propose a simple but effective training schedule that incorporates a language discriminator loss. The loss imposes constraints on the intermediate translation so that the translation is in the desired language. By conducting extensive experiments on different language pairs, including similar and distant, high and low-resource languages, we find that our method alleviates the copying problem, thus improving the translation performance on low-resource languages.