# Joint Training for Neural Machine Translation Models with Monolingual Data

arXiv: https://arxiv.org/abs/1803.00353 · HTML: https://mlsanity.com/paper/1803.00353/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2018.

## Built on by (most cited first, 20 of 20)

- [Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning](https://mlsanity.com/paper/2402.06619/) (2024)
- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [Non-Parametric Unsupervised Domain Adaptation for Neural Machine Translation](https://mlsanity.com/paper/2109.06604/) (2021)
- [Self-Training Sampling with Monolingual Data Uncertainty for Neural Machine Translation](https://mlsanity.com/paper/2106.00941/) (2021)
- [A Survey on Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2107.04239/) (2021)
- [The Volctrans GLAT System: Non-autoregressive Translation Meets WMT21](https://mlsanity.com/paper/2109.11247/) (2021)
- [Non-Parametric Domain Adaptation for End-to-End Speech Translation](https://mlsanity.com/paper/2205.11211/) (2022)
- [Simple and Scalable Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2302.12188/) (2023)
- [Non-Parametric Online Learning from Human Feedback for Neural Machine Translation](https://mlsanity.com/paper/2109.11136/) (2021)
- [Federated Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2302.12211/) (2023)
- [Simplifying Translations for Children: Iterative Simplification Considering Age of Acquisition with LLMs](https://mlsanity.com/paper/2408.04217/) (2024)
- [Integrating Unsupervised Data Generation into Self-Supervised Neural Machine Translation for Low-Resource Languages](https://mlsanity.com/paper/2107.08772/) (2021)
- [Semi-Supervised Image Captioning by Adversarially Propagating Labeled Data](https://mlsanity.com/paper/2301.11174/) (2023)
- [GTAE: Graph-Transformer based Auto-Encoders for Linguistic-Constrained Text Style Transfer](https://mlsanity.com/paper/2102.00769/) (2021)
- [Constructing Emotion Consensus and Utilizing Unpaired Data for Empathetic Dialogue Generation](https://mlsanity.com/paper/2109.07779/) (2021)
- [Improving Logical-Level Natural Language Generation with Topic-Conditioned Data Augmentation and Logical Form Generation](https://mlsanity.com/paper/2112.06240/) (2021)
- [Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Data](https://mlsanity.com/paper/2212.09982/) (2022)
- [Shared Latent Space by Both Languages in Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2305.03511/) (2023)
- [End-to-End Training for Back-Translation with Categorical Reparameterization Trick](https://mlsanity.com/paper/2202.08465/) (2022)
- [Bridging the Domain Gaps in Context Representations for k-Nearest Neighbor Neural Machine Translation](https://mlsanity.com/paper/2305.16599/) (2023)

## Abstract (arXiv)

Monolingual data have been demonstrated to be helpful in improving translation quality of both statistical machine translation (SMT) systems and neural machine translation (NMT) systems, especially in resource-poor or domain adaptation tasks where parallel data are not rich enough. In this paper, we propose a novel approach to better leveraging monolingual data for neural machine translation by jointly learning source-to-target and target-to-source NMT models for a language pair with a joint EM optimization method. The training process starts with two initial NMT models pre-trained on parallel data for each direction, and these two models are iteratively updated by incrementally decreasing translation losses on training data. In each iteration step, both NMT models are first used to translate monolingual data from one language to the other, forming pseudo-training data of the other NMT model. Then two new NMT models are learnt from parallel data together with the pseudo training data. Both NMT models are expected to be improved and better pseudo-training data can be generated in next step. Experiment results on Chinese-English and English-German translation tasks show that our approach can simultaneously improve translation quality of source-to-target and target-to-source models, significantly outperforming strong baseline systems which are enhanced with monolingual data for model training including back-translation.