# Unsupervised Multi-modal Neural Machine Translation

arXiv: https://arxiv.org/abs/1811.11365 · HTML: https://mlsanity.com/paper/1811.11365/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2018.

## Built on by (most cited first, 15 of 15)

- [VLP: A Survey on Vision-Language Pre-training](https://mlsanity.com/paper/2202.09061/) (2022)
- [VALHALLA: Visual Hallucination for Machine Translation](https://mlsanity.com/paper/2206.00100/) (2022)
- [Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucination](https://mlsanity.com/paper/2305.12256/) (2023)
- [Self-Supervised Multimodal Opinion Summarization](https://mlsanity.com/paper/2105.13135/) (2021)
- [A Survey on Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2107.04239/) (2021)
- [CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation](https://mlsanity.com/paper/2308.15226/) (2023)
- [Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-training](https://mlsanity.com/paper/2108.11119/) (2021)
- [A Survey on Multi-modal Machine Translation: Tasks, Methods and Challenges](https://mlsanity.com/paper/2405.12669/) (2024)
- [Video-Helpful Multimodal Machine Translation](https://mlsanity.com/paper/2310.20201/) (2023)
- [Towards Zero-Shot Multimodal Machine Translation](https://mlsanity.com/paper/2407.13579/) (2024)
- [IITP at WAT 2021: System description for English-Hindi Multimodal Translation Task](https://mlsanity.com/paper/2107.01656/) (2021)
- [Iterative Adversarial Attack on Image-guided Story Ending Generation](https://mlsanity.com/paper/2305.13208/) (2023)
- [ConECT Dataset: Overcoming Data Scarcity in Context-Aware E-Commerce MT](https://mlsanity.com/paper/2506.04929/) (2025)
- [Relevance-guided Neural Machine Translation](https://mlsanity.com/paper/2312.00214/) (2023)
- [An Empirical study of Unsupervised Neural Machine Translation: analyzing NMT output, model's behavior and sentences' contribution](https://mlsanity.com/paper/2312.12588/) (2023)

## Abstract (arXiv)

Unsupervised neural machine translation (UNMT) has recently achieved remarkable results with only large monolingual corpora in each language. However, the uncertainty of associating target with source sentences makes UNMT theoretically an ill-posed problem. This work investigates the possibility of utilizing images for disambiguation to improve the performance of UNMT. Our assumption is intuitively based on the invariant property of image, i.e., the description of the same visual content by different languages should be approximately similar. We propose an unsupervised multi-modal machine translation (UMNMT) framework based on the language translation cycle consistency loss conditional on the image, targeting to learn the bidirectional multi-modal translation simultaneously. Through an alternate training between multi-modal and uni-modal, our inference model can translate with or without the image. On the widely used Multi30K dataset, the experimental results of our approach are significantly better than those of the text-only UNMT on the 2016 test dataset.