# Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting

arXiv: https://arxiv.org/abs/2005.03119 · HTML: https://mlsanity.com/paper/2005.03119/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 May 2020.

## Built on by (most cited first, 22 of 22)

- [UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training](https://mlsanity.com/paper/2104.00332/) (2021)
- [IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages](https://mlsanity.com/paper/2201.11732/) (2022)
- [Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis](https://mlsanity.com/paper/2107.13669/) (2021)
- [Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models](https://mlsanity.com/paper/2103.08849/) (2021)
- [Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment](https://mlsanity.com/paper/2406.19255/) (2024)
- [VALHALLA: Visual Hallucination for Machine Translation](https://mlsanity.com/paper/2206.00100/) (2022)
- [Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucination](https://mlsanity.com/paper/2305.12256/) (2023)
- [LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation](https://mlsanity.com/paper/2210.15461/) (2022)
- [Self-Supervised Multimodal Opinion Summarization](https://mlsanity.com/paper/2105.13135/) (2021)
- [A Survey on Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2107.04239/) (2021)
- [Cross2StrA: Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignment](https://mlsanity.com/paper/2305.12260/) (2023)
- [CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation](https://mlsanity.com/paper/2308.15226/) (2023)
- [Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-training](https://mlsanity.com/paper/2108.11119/) (2021)
- [CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations](https://mlsanity.com/paper/2207.02185/) (2022)
- [Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translation](https://mlsanity.com/paper/2212.10313/) (2022)
- [A Survey on Multi-modal Machine Translation: Tasks, Methods and Challenges](https://mlsanity.com/paper/2405.12669/) (2024)
- [Towards Zero-Shot Multimodal Machine Translation](https://mlsanity.com/paper/2407.13579/) (2024)
- [XDBERT: Distilling Visual Information to BERT from Cross-Modal Systems to Improve Language Understanding](https://mlsanity.com/paper/2204.07316/) (2022)
- [Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunities](https://mlsanity.com/paper/2210.16960/) (2022)
- [How effective is Multi-source pivoting for Translation of Low Resource Indian Languages?](https://mlsanity.com/paper/2406.13332/) (2024)
- [GIIFT: Graph-guided Inductive Image-free Multimodal Machine Translation](https://mlsanity.com/paper/2507.18562/) (2025)
- [TransMeme: A Multi-Agent Framework for Cross-Cultural Meme Transcreation](https://mlsanity.com/paper/2608.27127/) (2026)

## Abstract (arXiv)

Unsupervised machine translation (MT) has recently achieved impressive results with monolingual corpora only. However, it is still challenging to associate source-target sentences in the latent space. As people speak different languages biologically share similar visual systems, the potential of achieving better alignment through visual content is promising yet under-explored in unsupervised multimodal MT (MMT). In this paper, we investigate how to utilize visual content for disambiguation and promoting latent space alignment in unsupervised MMT. Our model employs multimodal back-translation and features pseudo visual pivoting in which we learn a shared multilingual visual-semantic embedding space and incorporate visually-pivoted captioning as additional weak supervision. The experimental results on the widely used Multi30K dataset show that the proposed model significantly improves over the state-of-the-art methods and generalizes well when the images are not available at the testing time.